Ravindra BagaleCourses & study guides मराठी Track your progress

30. Statistics for Analysts

30.7 T.TEST and an A/B Test Walkthrough: Did the New Checkout Page Raise Basket Value?

By Last updated:

Example first. The product team tested a new checkout page (version B) against the old one (A) on 30 fictional customers, 15 each. Average basket: ₹411.67 for A and ₹446.67 for B, a difference of ₹35. Is that real, or could it be luck? =T.TEST(A2:A16,B2:B16,2,3) returns 0.0115. A difference this large would appear by chance only about 1.1% of the time if the pages were really the same. This lesson walks through the whole A/B test: the question, the right test type, the formula, the ToolPak output and the sentence you write for the team.

T.TEST for an A/B test — in 30 seconds

What: A t-test checks whether the difference between two averages is bigger than random noise. T.TEST returns the p-value: the probability of seeing a difference at least this big if the true averages were equal.

Why: A/B tests, before/after comparisons and city-vs-city questions all need it. Without it, teams launch changes because of differences that are only luck.

Tiny example: =T.TEST(A2:A16,B2:B16,2,3) = 0.0115: two tails (2) and two separate groups with possibly different spread (3). Below 0.05, so B's higher basket is statistically significant.

First look at the data (before → after):

T.TEST for an A/B test: before and after tables, fictional Blinkit Maharashtra data

Expected result — A/B result

Measure Result
Mean A 411.67
Mean B 446.67
Difference B − A 35.00
p-value (two-tailed, type 3) 0.0115
Decision at 5% Significant
Before table and formula as text

Before — Basket value (₹), first 5 customers

Basket A Basket B
388 432
425 478
402 401
356 456
471 495
… …

Formula / click path

=T.TEST(A2:A16,B2:B16,2,3)
=AVERAGE(B2:B16)-AVERAGE(A2:A16)
Data › Data Analysis › t-Test: Two-Sample Assuming Unequal Variances

Do it step by step:

  1. Write the question before looking at the data: "Is the average basket different between A and B?" Choose 5% (0.05) as the limit.
  2. Put A's baskets in A2:A16 and B's in B2:B16.
  3. In D2 type =AVERAGE(B2:B16)-AVERAGE(A2:A16) for the size of the difference (35).
  4. In D3 type =T.TEST(A2:A16,B2:B16,2,3): 2 tails, type 3 because the customers in A and B are different people.
  5. Compare: 0.0115 is below 0.05, so the difference is statistically significant.
  6. Write the result with size and uncertainty: "B raised the average basket by ₹35 (95% range about ₹9 to ₹61), p = 0.011."

The data (fictional basket values in ₹)

Customer 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
A 388 425 402 356 471 410 395 447 362 430 418 379 455 401 436
B 432 478 401 456 495 413 449 382 468 510 425 441 487 404 459

Step 1: the question and the limit. The null hypothesis is "A and B have the same average basket". The usual limit (alpha) is 0.05. Decide it before you look at the result, and decide whether you care about any difference (two-tailed, tails = 2) or only "B is higher" (one-tailed, tails = 1). Two-tailed is the safe default.

Step 2: choose the type

type Test Use when p on this data
1 Paired The same customers or stores measured twice (before and after) 0.0082 (wrong here: the rows are not pairs)
2 Two-sample, equal variance Two separate groups with similar spread 0.0114
3 Two-sample, unequal variance (Welch) Two separate groups; the safe default 0.0115

Type 3 does not assume the two groups have the same spread, and gives almost the same answer as type 2 when they do. Using type 1 on unpaired data gives a wrong p-value (0.0082 here) because it pairs customer 1 of A with customer 1 of B for no reason.

Step 3: the size of the effect and its range. A p-value says "probably not luck"; it does not say "big". Always report the difference and a 95% range:

  • Difference: =AVERAGE(B2:B16)-AVERAGE(A2:A16) = ₹35.
  • Half-width: =T.INV.2T(0.05,28)*SQRT(VAR.S(A2:A16)/15+VAR.S(B2:B16)/15) = 26.47.
  • 95% range: ₹8.53 to ₹61.47. B probably adds between about ₹9 and ₹61 per basket.

Step 4: the ToolPak output, line by line. Data › Data Analysis › t-Test: Two-Sample Assuming Unequal Variances, Variable 1 Range A1:A16, Variable 2 Range B1:B16, Hypothesized Mean Difference 0, Labels, Alpha 0.05:

Line Basket A Basket B Meaning
Mean 411.6667 446.6667 Average basket
Variance 1105.2381 1399.5238 Sample variance (VAR.S)
Observations 15 15 Customers per version
Hypothesized Mean Difference 0 — We test "no difference"
df 28 — Welch degrees of freedom 27.62, rounded by the ToolPak
t Stat -2.7085 — Negative because A − B is negative
P(T<=t) one-tail 0.0057 — One-tailed p-value
t Critical one-tail 1.7011 — =T.INV(0.95,28)
P(T<=t) two-tail 0.0114 — Two-tailed p-value: the one to report
t Critical two-tail 2.0484 — =T.INV.2T(0.05,28)

Why the ToolPak p (0.0114) differs slightly from T.TEST (0.0115). Microsoft's documentation explains it: for unequal variances the ToolPak rounds the degrees of freedom to a whole number (27.62 → 28), while T.TEST uses the unrounded value. Both lead to the same decision. Report the T.TEST value.

Step 5: the sentence. "On 30 customers, version B raised the average basket from ₹411.67 to ₹446.67 (+₹35, 95% range ₹9 to ₹61; Welch t-test p = 0.011). Recommend a larger test before full launch, because 15 customers per version is a small sample."

Things a good A/B test needs (no formula fixes these)

  1. Random assignment: customers must be split by chance, not "Pune got B, Solapur got A".
  2. Sample size decided in advance: stopping the test the moment p dips below 0.05 inflates false wins.
  3. One main metric: if you test 20 metrics, about one will look "significant" by luck (lesson 30.9).
  4. Same period: run A and B at the same time so weekends and festivals affect both.

Every number on this page was computed with Excel's documented formulas and re-checked twice: in LibreOffice Calc 25.2 with the same formulas, and in Python with scipy and statsmodels.

Microsoft sources (checked 7 October 2026): T.TEST function; Use the Analysis ToolPak to perform complex data analysis.

Practice task (try it before you open the answer):

  1. Run =T.TEST(A2:A16,B2:B16,1,3). What do you get and how does it relate to the two-tailed value?
  2. A colleague used type 2. Does the decision change?
  3. Suppose the p-value had been 0.21. Write the result sentence.
Answer
  1. 0.0057, exactly half of the two-tailed 0.0115. Use one-tailed only if you decided before the test that only "B higher" matters.
  2. No. Type 2 gives 0.0114, also below 0.05. The spreads of A and B are similar, so types 2 and 3 agree.
  3. For example: "B's average basket was higher, but the difference could easily be chance (p = 0.21). We cannot conclude that B raises basket value; a larger test is needed."

Ravindra Bagale's Tip

Report every A/B test as three numbers: the difference, its 95% range and the p-value. "Significant" alone tells your manager nothing about whether the change is worth the effort.

Loading practice tasks...

  1. Old page mean (Easy): Average of A2:A11.
  2. New page mean (Easy): Average of B2:B11.
  3. Lift in dollars (Easy): Average of B2:B11 minus average of A2:A11.
  4. Lift % (Medium): Lift ÷ average of A2:A11, rounded to 3 decimals.
  5. p-value (equal variance) (Medium): T.TEST of A2:A11 and B2:B11, 2 tails, type 2, rounded to 4 decimals.
  6. p-value (unequal variance) (Medium): T.TEST with 2 tails and type 3, rounded to 4 decimals.
  7. Significant at 5%? (Medium): Is the type 3 two-tailed p-value below 0.05? (TRUE or FALSE)
  8. Paired p-value (Hard): T.TEST with 2 tails and type 1 (paired), rounded to 5 decimals.
  9. Pairs where new won (Hard): How many rows have New page above Old page?
  10. Verdict text (Hard): Show "Launch" if the type 3 p-value is below 0.05, else "Keep testing".

See all practice tasks in the Task Hub →