Milk First, or Tea First?

There are three kinds of lies: lies, damned lies, and statistics.

This saying is often understood as a damning indictment of statistics, but Mark Twain didn’t really mean it that way. Mark Twain, in his old age, acknowledged he was slowing down. According to the metric of words per day, he was clearly writing less. However, he then noted he wrote fewer hours. Thus, from the perspective of words per hour, he was still consistently writing the same amount. The statistics do not lie, but can be used to lie.

Beginning in 2010—every five years on October 20th—the UN observes World Statistics Day. Once, when the decennial census was the only general exposure to statistics, this cadence made sense.

But in the modern world, statistics are more pervasive—and thus more important—than ever. From public health findings, politics, climate, economics, and finance—the applications of statistics ever expand. They form the quintessential backbone of scientific research and development.

And with that, the significance of statistical literacy evolves in tandem. Without at least a passing understanding of statistics, census counts, policy figures, inflation, unemployment, and epidemiological findings—the numbers governments use to make policy and citizens use to act—can be misread, misinterpreted, or worse, weaponised against you.

These are statistical products. Their credibility depends on methodological rigour and clear communication of findings, and their comprehension depends on wisdom, critical thinking, and background knowledge—helping us separate precision from propaganda.

A hundred years ago, nobody had a reliable way to tell a real result from a lucky guess—tools had to be invented. One such tale of discovery is today’s topic—a story about a statistician, a cup of tea served with tea poured first, and a lady declining the tea that led to the foundations of designing experiments and statistical testing.

The Afternoon Tea at Rothamsted

Early 1920s. At Rothamsted Experimental Station in Harpenden, UK.

At afternoon tea, Ronald Fisher—described as a genius who almost single-handedly created the foundations for modern statistical science—poured a cup and offered it to a colleague, Muriel Bristol. Fisher was a statistician for the agricultural research station; Bristol was an algologist who had joined Rothamsted’s Institute of Plant Pathology to study soil algae.

Bristol declined the tea offered—she preferred the milk poured into the cup before the tea. “I can tell when tea is poured first, and when milk is poured first,” she said, “I can taste which way it had been done.”

Fisher found the idea preposterous—impossible even. The two liquids end up chemically identical, regardless of pouring order! No one should have the ability to differentiate between the resulting types of tea.

But Dr Bristol held her ground. She could tell the difference. The biochemist William Roach—Bristol’s fiancé—working in Rothamsted’s antiseptics and insecticides lab at the time, said the favourite words of any scientist, Let’s test her!

The Hypothesis

In Chapter 2 of his The Design of Experiments (1935), Fisher opens with a psycho-physical experiment: The lady tasting tea. Using clever generalisations, Fisher establishes the principles of experimentation by designing the tea test for his colleague.

Formalising the null hypothesis is an important first step in experimentation. The null hypothesis is the claim an experiment tries to disprove. It should be exact, free of vagueness and ambiguity. For this thought experiment, the null hypothesis is,

The subject has no ability to distinguish the teas.

That is, the lady cannot discriminate between tea prepared by pouring milk first and tea prepared by pouring tea first.

Thus, if she distinguishes the teas correctly to a significant extent, the null hypothesis is rejected. Fisher stopped there and set up no alternative hypothesis. The Neyman-Pearson approach did that—rejecting the null meant asserting the alternative hypothesis (the lady can distinguish the teas).

The Experiment

The next step is to set up the design. In this experiment, we begin mixing eight cups of tea—four prepared by pouring milk first, and four by pouring tea first. The subject is presented with the prepared teacups in random order. The subject is also told in advance that her task is to divide the 8 cups into two sets of 4 and identify which were milk first and which were tea first.

What are the chances?

To understand the chances involved for possible selections, we can calculate the exact probabilities.

Assuming the null hypothesis is true, we can compute the distribution of possible numbers of successes using combinations. There are 70 different ways to divide eight cups into two groups of four. So, if the lady was simply guessing, she had a 1-in-70 chance of landing on the one grouping that matched how the cups were prepared. Because she was informed of the split, her possible answers are exactly these 70 groupings, which allows the odds to be counted exactly.

Let X be the number of milk-first cups among the 4 she picks. Only one of the 70 possible groupings gives a perfect score of X = 4, so the probability of selecting all 4 correctly is P(X = 4) = 1/70 = 0.014, or about 1.4%. Getting 3 out of 4 correct is far more common: it can happen in 16 different ways (4 ways to pick the 3 right cups, times 4 ways to pick the 1 wrong cup), so P(X = 3) = 16/70 = 0.23, or about 23%.

The probability of getting at least 3 correct is P(X = 3 or 4) = 17/70, or about 24%, purely by chance. That is far too likely to count as evidence of anything. The most common outcome is exactly 2 correct, which can happen in 36 ways (6 ways to pick the 2 right cups, times 6 ways to pick the 2 wrong cups), so P(X = 2) = 36/70 = 0.51, or about 51%. So, about half of all random guesses end with exactly 2 correct.

The Significance

Picture that same guessing experiment repeated thousands of times, by thousands of different guessers with no actual ability. Spread out across all those attempts, the results are expected to form a pattern—about half the time, someone lands on exactly 2 correct. About a quarter of the time, someone gets 3 or 4 right. A perfect score turns up in roughly 1 out of every 70 attempts. That spread is the entire universe of what guessing alone can produce.

The lady only sat for the test once, and produced one actual result. The real question is where that one result falls inside this spread of what chance alone could have produced—and how rare it has to be before random chance stops being a believable explanation. Fisher proposed that if a result could occur by chance less than 1 in 20 times, or 5% of the time, then it is statistically significant. This threshold is a widely followed standard for statistical significance. Thus, if and only if the lady correctly categorised all 8 cups, Fisher was willing to reject the null hypothesis—effectively acknowledging the lady’s ability of distinguishing tea, since guessing alone would produce a perfect score only 1.4% of the time. Getting three cups right wasn’t enough: a blind guess would do that 14% of the time.

The Results

This hypothetical experiment shows how one can perform a simple experiment and test of significance from start to finish. Once this design is generalised, it is known as Fisher’s exact test.

Thirty years earlier, nobody had a tool built for a problem this small. Pearson’s methods worked for large samples, not eight cups. Gosset’s t-test, the first real small-sample tool, was still over a decade away—and wasn’t the best test for this experiment. A scientist facing the lady’s claim before Fisher would have had no rigorous way to test this dilemma—only to make a judgment call and no way to say how impressive that judgment is.

Through this psycho-physical experiment describing a hypothetical lady and a hypothetical row of cups, he formalises the null hypothesis, the logic of a significance test, and the necessity of randomisation in experiments. This work also contributes to developing permutation & randomisation tests, causal inference, and experimental power.

Hypothesis testing has driven countless research findings, including the discovery of the Higgs boson at CERN, the link between smoking and lung cancer, and the 1954 Salk polio vaccine trial, to name a few.

By understanding the experimental process, we can elucidate phenomena and learn to question and distinguish good science from lies.

Speaking of which. How did the Lady—Dr Muriel Bristol—perform in the experiment?

Once the 8 cups of tea were presented before her—she began her work.

Sipping.

Placing.

Sipping.

Sorting.

A crowd gathered—other researchers lounging in the break hall—as she sipped and placed another cup.

One. Two. Three. Four. Five. Six. Seven. Eight. How many did she get right?

According to historical record, she grouped all the teas correctly—going eight for eight.

Masud "Mass" Hoque is a statistician whose curiosity extends beyond data driven research, into the myriads of cultures, literature, science, and the stories that shape them. Find more of his work on his Substack Mass o Menos.