Dark Energy and the Cosmological Constant: Hubble Tension and DESI 2024

In 1998, Two Independent Teams Discovered the Universe Is Tearing Itself Apart

The discovery was a shock. Saul Perlmutter leading the Supernova Cosmology Project and Brian Schmidt and Adam Riess leading the High-Z Supernova Search Team were both using Type Ia supernovae as "standard candles" — objects of known intrinsic luminosity — to measure the universe's expansion rate. Both groups expected to measure how much gravity was slowing the expansion that began with the Big Bang. Instead, both independently found that distant supernovae were dimmer than expected — meaning they were farther away than the expansion rate should have placed them. The universe's expansion is not slowing down. It is speeding up. Both teams rushed to publish in 1998 and 1999, confirming each other's findings. Perlmutter, Schmidt, and Riess shared the Nobel Prize in Physics in 2011 for this discovery.

What Dark Energy Is (and Isn't)

Dark energy is the name given to whatever is causing cosmic acceleration. It comprises approximately 68% of the total energy density of the universe, with ordinary matter at 5% and dark matter at 27%. Despite this dominant share, dark energy remains one of the least understood phenomena in physics. It is not dark matter — dark matter clusters gravitationally while dark energy appears to be uniformly distributed and acts as a repulsive pressure on the largest scales.

The simplest mathematical description is Einstein's cosmological constant, Λ (lambda) — a term Einstein added to his field equations in 1917 to allow for a static universe, then removed when Hubble showed the universe was expanding. Einstein called adding Λ his "greatest mistake." Ironically, the discovery of cosmic acceleration rehabilitated Λ as the leading description of dark energy.

The Cosmological Constant Problem

The worst prediction in physics. Quantum field theory predicts that the vacuum of space should have an enormous energy density due to zero-point fluctuations of quantum fields. When calculated, this vacuum energy is approximately 10^120 times larger than the observed value of dark energy. This discrepancy — 120 orders of magnitude — is the largest known gap between theoretical prediction and experimental observation in all of science. No solution exists. Proposed explanations range from supersymmetry (cancels most vacuum energy contributions) to the anthropic principle (we observe a small Λ because only a small Λ permits galaxies and stars and observers to form).

The Hubble Tension

Two methods of measuring the universe's current expansion rate — the Hubble constant (H₀) — give incompatible answers. This is the Hubble tension, and it has intensified as both measurement techniques have improved beyond systematic error excuses.

• Early-universe method: Analyzing the cosmic microwave background (CMB) and baryon acoustic oscillations (BAO) using the Planck satellite gives H₀ = 67.4 ± 0.5 km/s/Mpc
• Late-universe method: The distance ladder — using Cepheid variable stars and Type Ia supernovae calibrated by Adam Riess's SHOES team — gives H₀ = 73.0 ± 1.0 km/s/Mpc
• The discrepancy is now at approximately 5σ statistical significance — far beyond the 3σ threshold conventionally considered strong evidence for a real effect
• If not a systematic error, the tension may indicate new physics beyond the standard ΛCDM cosmological model

Phantom Energy and the Big Rip

The cosmological constant has equation of state w = −1. If dark energy is something else — a dynamic field called quintessence — w could differ from −1. If w < −1, dark energy is called "phantom energy," and its density would grow over time, leading to a catastrophic end state called the Big Rip. In the Big Rip scenario, dark energy's repulsive force eventually overcomes all other forces: galaxy clusters dissociate, then galaxies, then solar systems, then planets, then atoms themselves are torn apart. The Big Rip would occur approximately 20–22 billion years from now if w ≈ −1.5, though no observational evidence favors w < −1.

DESI 2024: A Crack in the Cosmological Constant?

The Dark Energy Spectroscopic Instrument (DESI) at Kitt Peak Observatory published its first-year results in April 2024, measuring baryon acoustic oscillations in the spectra of 6 million galaxies spanning the past 11 billion years of cosmic history. The preliminary results suggested w may be slightly different from −1 and possibly evolving over time — a hint of dynamical dark energy inconsistent with a pure cosmological constant. The statistical significance was approximately 2.5σ — suggestive but not definitive. DESI's full five-year dataset will encompass 40 million galaxy spectra. If the deviation from w = −1 persists with higher statistical significance, it would require a fundamental revision of the standard ΛCDM cosmological model and force physicists to construct entirely new theories of dark energy.