How the Speed Lab tests are run

What a Speed Lab test is

A Speed Lab test asks one narrow question, such as "what does loading="lazy" cost on the main image?" It answers with a page built for the purpose, never a customer's site. The variants differ in one thing. Everything else, including the files, their sizes and the order of the markup, stays the same. Each test article shows the markup of the variants, the results and the request timelines, so you can see why a number moved and repeat the test.

The setup

Three ways to slow the network down

Lighthouse has two ways to slow a test down. Its default, called simulate, loads the page once at full speed and then calculates what a slow phone would see. The other, called devtools, makes Chrome itself slow down the network and the processor while the page loads. Most tests here use devtools throttling, so Chrome's real behavior shows up: how it queues requests, which priority it gives each one, and how many connections it opens. A few tests use the simulated method instead, and say so. See Lighthouse: what it tests and how to run it for the settings.

The delay of 562.5 ms is 150 ms of round trip time multiplied by 3.75, which is how Lighthouse converts its network settings for devtools throttling. It is a flat delay on every request, not a model of a real connection. That has three consequences, and the test of three ways to slow a test down measures them: a chain of requests looks longer in the lab than on a real connection, opening a connection costs nothing, and the order in which a server sends files cannot show.

For the tests where those limits matter, a small proxy sits between the browser and the server. It gives every byte a round trip of 150 ms, shares one link of 1,638.4 Kbps between all connections and lets each connection start slowly. Lighthouse then applies only the processor slowdown. The test of many small files has the code, and every test that uses it says so.

The rules

1. One change at a time. If two things differ, the test cannot say which one caused the result.
2. Nine runs for each variant. One run tells you little, because the same page gives a slightly different number each time. See why PageSpeed scores change.
3. Variants are taken in turns. The order is A, B, C, then A again, so a slow minute on the machine hits every variant alike.
4. Report the median and the range. The median is the middle run, so one odd run does not move it. The lowest and highest runs show how much the result wobbles.
5. Look at the timeline. A number says that something changed. The Lighthouse report also lists when each request started and ended, and that shows why.
6. Say what the test cannot show, and report a result even when nothing changed.

How to read the tables

Each result shows the median first and the lowest and highest run in brackets, for example 2,491 ms (2,478 to 2,501). In my tests on small pages the lowest and highest runs of a variant were usually within 20 to 50 ms of each other. So a gap of 5 or 10 ms between two variants is noise, and a gap of 300 ms with ranges that do not overlap is a real difference in this setup. When two ranges overlap, treat the variants as equal.

The metrics come from Lighthouse's own audits: First Contentful Paint, Largest Contentful Paint, Total Blocking Time, Cumulative Layout Shift and Speed Index. The request timelines come from the same runs. All times are in milliseconds.

What the tests cannot show

• Your visitors. This is lab data from one simulated phone. See lab data and field data for the difference.
• A real network. The server is on the same machine. The usual delay is flat, and even the proxy has no packet loss. A real connection has distance, loss and shared bandwidth.
• Your page. The test pages are small and simple, so the size of an effect on a real page will differ. The direction and the cause usually carry over, and the numbers do not.
• Other browsers. Every test ran in Chrome.
• Other versions. Lighthouse and Chrome change. A different version can give different numbers.
• Other machines. The processor slowdown is relative to the speed of the machine, so the same test on a faster or slower computer gives different absolute times.

Run it yourself

Save this as lab-runner.mjs. It serves a folder that holds one folder per variant, runs Lighthouse on each in turn and prints the median and the range of the metrics you name. Install Lighthouse with npm install lighthouse (Node 22 or later) and run, for example, node lab-runner.mjs ./variants 9 largest-contentful-paint,total-blocking-time. Add GZIP=1 to send the text files compressed, and THROTTLING=simulate for Lighthouse's own model. The lazy-loading test has a shorter script for one metric.

// lab-runner.mjs: node lab-runner.mjs <folder holding one folder per variant> [runs] [metrics]
//   runs     runs per variant, default 9
//   metrics  comma-separated Lighthouse audit ids, default largest-contentful-paint,first-contentful-paint;
//            style-and-layout reads the main-thread work breakdown
//   GZIP=1   send html, css and js compressed with gzip
//   THROTTLING=simulate   use Lighthouse's simulated throttling instead of the applied (devtools) one
// Serves the folders over plain HTTP/1.1, runs Lighthouse on each variant in turns and prints the median with the lowest and highest run.
import { createServer } from 'node:http';
import { gzipSync } from 'node:zlib';
import { readFileSync, existsSync, readdirSync, statSync } from 'node:fs';
import { join, extname, resolve } from 'node:path';
import { execFile } from 'node:child_process';
import { promisify } from 'node:util';

const root = resolve(process.argv[2]);
const runs = Number(process.argv[3] || 9);
const metrics = (process.argv[4] || 'largest-contentful-paint,first-contentful-paint').split(',');
const types = { '.html': 'text/html', '.css': 'text/css', '.js': 'text/javascript', '.jpg': 'image/jpeg', '.png': 'image/png', '.woff2': 'font/woff2' };
const variants = readdirSync(root).filter((d) => statSync(join(root, d)).isDirectory()).sort();

const server = createServer((req, res) => {
  let path = decodeURIComponent(new URL(req.url, 'http://x').pathname);
  if (path.endsWith('/')) path += 'index.html';
  const file = join(root, path);
  if (!file.startsWith(root) || !existsSync(file)) { res.writeHead(404); return res.end(); }
  const type = types[extname(file)] || 'application/octet-stream';
  const packed = process.env.GZIP === '1' && type.startsWith('text/');
  const body = packed ? gzipSync(readFileSync(file), { level: 6 }) : readFileSync(file);
  res.writeHead(200, { 'content-type': type, ...(packed && { 'content-encoding': 'gzip' }), 'cache-control': 'no-store', 'content-length': body.length });
  res.end(body);
});
await new Promise((done) => server.listen(0, '127.0.0.1', done));
const base = `http://127.0.0.1:${server.address().port}`;

const read = (audits, id) => id === 'style-and-layout'
  ? audits['mainthread-work-breakdown'].details.items.find((item) => item.group === 'styleLayout').duration
  : audits[id].numericValue;
const results = Object.fromEntries(variants.map((v) => [v, []]));
for (let i = 0; i < runs; i++) {
  for (const v of variants) {
    // The server lives in this process, so use the asynchronous execFile. A blocking call would freeze it.
    const { stdout } = await promisify(execFile)('npx', ['lighthouse', `${base}/${v}/`, '--only-categories=performance', `--throttling-method=${process.env.THROTTLING || 'devtools'}`,
      '--output=json', '--quiet', '--chrome-flags=--headless=new'], { maxBuffer: 1 << 28 });   // add --no-sandbox to the Chrome flags if you run as root in a container
    const audits = JSON.parse(stdout).audits;
    results[v].push(Object.fromEntries(metrics.map((id) => [id, read(audits, id)])));
  }
}
server.close();

const median = (xs) => [...xs].sort((a, b) => a - b)[xs.length >> 1];
const show = (id, n) => (id.includes('layout-shift') ? n.toFixed(3) : `${Math.round(n)} ms`);
for (const v of variants) {
  for (const id of metrics) {
    const xs = results[v].map((r) => r[id]);
    console.log(v.padEnd(20), id.padEnd(26), show(id, median(xs)), `(${show(id, Math.min(...xs))} to ${show(id, Math.max(...xs))})`);
  }
}

Keep the computer otherwise idle while it runs, use an odd number of runs, and compare the variants with each other, not with the numbers here. The tests that compare protocols need more than this: see the test of many small files.

Check your own page

Run the free audit on your page, make one change, and run it again. Then repeat both a few times before you decide that the change helped. See how to test speed changes safely.

Sources

• Throttling in the Lighthouse documentation: simulated and applied throttling, the mobile preset and the processor slowdown.
• Variability in the Lighthouse documentation: why repeated runs differ and how to reduce it.
• The Lighthouse 13.5.0 configuration and its throttling constants, as installed for these tests.