Probability Theory: From Coin Flips to Statistical Inference

Quantifying the Unknown

Before probability theory was formalized, gambling winnings were disputed and epidemic disease was attributed to divine will. The same mathematical framework that emerged from 17th-century French correspondence between Blaise Pascal and Pierre de Fermat — sparked by a gambling dispute — now underpins quantum mechanics, machine learning, insurance pricing, and clinical trial design. Probability theory is the mathematical language of uncertainty.

The formal axiomatization came in 1933 when Andrey Kolmogorov published his foundational monograph, establishing probability on rigorous set-theoretic grounds. Before Kolmogorov, competing intuitions about probability coexisted uneasily. His three axioms unified them into a coherent framework that remains the basis of modern probability and statistics.

Kolmogorov's Axioms

Kolmogorov defined a probability space as a triple (Ω, ℱ, P), where:

• Ω (sample space): The set of all possible outcomes of an experiment (e.g., {H, T} for a coin flip)
• ℱ (event space): A collection of subsets of Ω representing events (e.g., "at least one head in three flips")
• P (probability measure): A function assigning a number to each event, satisfying three axioms

The three axioms are: (1) Non-negativity: P(A) ≥ 0 for any event A. (2) Normalization: P(Ω) = 1 — the probability that something happens equals 1. (3) Countable additivity: For mutually exclusive events A₁, A₂, ..., P(A₁ ∪ A₂ ∪ ...) = P(A₁) + P(A₂) + ...

All other results in probability theory follow from these three axioms. The fact that P(not-A) = 1 − P(A), that P(A or B) = P(A) + P(B) − P(A and B), and that impossible events have probability zero all derive from axioms without additional assumptions. This deductive structure makes probability theory genuinely mathematical rather than merely intuitive.

Conditional Probability and Independence

Many of probability theory's most important results concern relationships between events. Conditional probability — the probability of event A given that event B has occurred — is defined as:

P(A|B) = P(A ∩ B) / P(B), provided P(B) > 0

This definition captures the intuition that knowing B has occurred restricts the sample space to outcomes within B. Two events are independent if P(A ∩ B) = P(A) × P(B), equivalently if P(A|B) = P(A) — knowing B tells you nothing new about A.

Bayes' theorem follows immediately from the definition of conditional probability:

P(A|B) = P(B|A) × P(A) / P(B)

This simple equation is the engine of Bayesian inference. It describes how to update the probability of hypothesis A (the prior) after observing evidence B, using the likelihood P(B|A). The result is the posterior probability P(A|B).

Random Variables and Distributions

A random variable is a function that assigns a numerical value to each outcome in the sample space. Random variables are either discrete (taking countable values, like the number of heads in 10 flips) or continuous (taking values in an interval, like the height of a randomly chosen adult).

The normal distribution occupies a central position partly because of the Central Limit Theorem and partly because many natural phenomena — heights, measurement errors, the sum of many independent effects — empirically follow approximately normal distributions. Its bell-shaped probability density function is characterized by mean μ and standard deviation σ. Approximately 68% of the distribution lies within one σ of the mean, 95% within two σ, and 99.7% within three σ.

The Law of Large Numbers and the Central Limit Theorem

Two theorems anchor classical probability theory and justify statistics as a discipline:

The Law of Large Numbers states that as the number of independent trials increases, the sample mean converges to the true expected value. In strong form (Borel's law, 1909): with probability 1, the sample average of n i.i.d. random variables converges to the population mean as n → ∞. This provides the theoretical guarantee that empirical frequencies in large samples approximate true probabilities.

The Central Limit Theorem (CLT) is more remarkable: if you sum n independent identically distributed random variables with finite mean and variance — regardless of their original distribution — the distribution of the standardized sum converges to the standard normal distribution as n → ∞. The precise statement: if X₁, X₂, ..., Xₙ are i.i.d. with mean μ and variance σ², then (X̄ₙ − μ)/(σ/√n) converges in distribution to N(0,1).

From Probability Theory to Statistical Inference

Statistical inference reverses the probabilistic question. Probability asks: given a known model, what outcomes are likely? Inference asks: given observed data, what model (or parameter values) are most consistent with the data?

• Frequentist inference treats model parameters as fixed unknowns and uses sampling distributions to construct confidence intervals and hypothesis tests. The p-value — misunderstood perhaps more than any other statistical concept — is the probability of observing a test statistic at least as extreme as the observed value, assuming the null hypothesis is true.
• Bayesian inference treats parameters as random variables with prior distributions, uses Bayes' theorem to update to posterior distributions after observing data, and makes predictions by integrating over parameter uncertainty. It explicitly quantifies uncertainty about parameter values rather than relying on asymptotic sampling arguments.
• Maximum likelihood estimation (MLE) finds the parameter values that maximize the probability of the observed data — the most common estimation method in applied statistics and machine learning.

Probability in Science and Technology

Quantum mechanics is, at its foundation, a probability theory — but not Kolmogorov's. Wave functions encode probability amplitudes whose squares give measurement probabilities, and interference between amplitudes (impossible in classical probability) enables quantum computation. Machine learning uses probability theory for model specification (generative models, Bayesian networks), training (maximum likelihood, cross-entropy loss), and uncertainty quantification. Financial mathematics relies on stochastic processes — random processes evolving in time — to price derivatives, manage risk, and simulate market scenarios. The reach of probability theory extends wherever uncertainty must be reasoned about systematically.