NVMe, SATA and the speeds you will never see

Drive marketing quotes one number: sequential read throughput at a high queue depth. It is a real measurement and it describes a workload almost nobody has. Copying one enormous file is sequential. Booting, launching applications, compiling, loading a game, running a database — none of them are.

Queue depth is the trick in the number

Storage benchmarks specify how many requests are outstanding at once. The headline figure usually assumes a deep queue — dozens of simultaneous requests — because that is what saturates the controller's parallelism.

Ordinary desktop work generates a queue depth of roughly 1 to 4. A program asks for something, waits, and asks for the next thing based on what it got. At that depth, NVMe's advantage shrinks dramatically, because its advantage was never raw speed so much as the ability to keep many operations in flight at once.

This is why the upgrade from a hard disk to any SSD is transformative and the upgrade from SATA SSD to NVMe often is not. The first removes seek latency measured in milliseconds; the second improves a figure that was already measured in tens of microseconds.

Random 4K is the number that predicts the feel

If you only look at one figure, make it random 4K at queue depth 1. That is closest to what an operating system does all day.

# random 4K reads, shallow queue, the realistic case
fio --name=random-read --ioengine=libaio --direct=1 \
    --rw=randread --bs=4k --iodepth=1 --numjobs=1 \
    --size=1G --runtime=30 --time_based \
    --directory=/var/tmp --output-format=normal | grep -E 'IOPS|lat \(usec|clat percentiles'

Compare that against the same test at --iodepth=32. The gap between the two is the gap between the marketing and your experience.

Watch latency rather than throughput: it is latency a person perceives. A drive answering a 4K read in 80 microseconds feels instant; one answering in 400 does not, even if both can move gigabytes per second under load.

The SLC cache, and what happens when it runs out

Consumer SSDs store multiple bits per cell — three or four — which is slow to write. To hide this, they treat part of the drive as fast single-bit cache, absorb writes there at full speed, and migrate them later.

That cache is typically a few percent of free capacity. While a write fits inside it, the drive performs as advertised. When it fills, write speed falls to the native rate of the underlying flash — sometimes to a fraction of a SATA drive's.

You can see the cliff directly:

# write continuously and watch throughput over time
fio --name=cache-cliff --ioengine=libaio --direct=1 \
    --rw=write --bs=1M --size=100G --runtime=300 --time_based \
    --directory=/var/tmp --write_bw_log=bw --log_avg_msec=1000 \
    --output-format=normal | grep -E 'bw=|WRITE:'

Plot the log and the shape tells the story: a flat fast plateau, then a step down. Where that step lands is the figure that matters for anyone writing large amounts of data — video, backups, large dataset work — and it is never on the box.

Two practical consequences. A fuller drive has a smaller cache, because the cache is carved from free space, so a drive at 90% capacity behaves markedly worse than the same drive at 40%. And sustained-write workloads should size the drive generously for that reason alone.

DRAM-less drives and the mapping table

An SSD keeps a table mapping logical addresses to physical flash locations. Drives with onboard DRAM hold it in that DRAM. Cheaper drives do not, and instead borrow host memory over the PCIe bus, or read the table from flash as needed.

Sequential benchmarks barely notice. Random access over a large working set notices a great deal, because every lookup may cost an extra round trip. The effect grows as the drive fills, since more of the table is in play.

Drives rarely advertise this. The absence of a DRAM figure in the specification is itself the signal, and for a boot drive or anything doing database work it is worth paying to avoid.

Where NVMe genuinely wins

None of the above argues for SATA. It argues against buying on the headline number. NVMe is clearly better when:

• The queue is deep — virtualisation hosts, databases under concurrent load, anything serving many requests at once.
• Large sequential transfers are routine — video work, moving datasets, large backups.
• Many small files are accessed in parallel — compiling a large codebase, where the build system reads thousands of files across many cores at once.
• The form factor matters — M.2 removes two cables and a bay.

And where the difference is small: booting, launching applications, ordinary document and browser work, games. All are shallow-queue and largely random, and both drive types are already far past the point where storage is the limit.

Things that cost you more than the interface

Thermal throttling. High-end NVMe drives throttle under sustained load without a heatsink, which turns a fast drive into an erratic one. A motherboard heatsink is usually sufficient and usually free.

The PCIe lanes you actually have. A Gen4 drive in a Gen3 slot runs at Gen3. A second M.2 slot sometimes shares lanes with the chipset or disables SATA ports when populated — the motherboard manual states this and nobody reads it until something disappears.

Capacity. Smaller drives in the same family are often slower, having fewer flash dies to write across in parallel. The 500 GB version of a drive is frequently not the 2 TB version's performance at a quarter of the size.

How to choose

1. Decide whether your workload is deep-queue or shallow. Desktop and gaming are shallow; servers, VMs and databases are deep.
2. Compare random 4K at QD1 and latency, not sequential throughput.
3. Check for DRAM on anything that will be a boot drive or hold a database.
4. Find the sustained write rate after cache exhaustion if you write large files regularly.
5. Buy more capacity than you need — it enlarges the cache, keeps the drive off its slow region, and extends endurance.
6. Check lane sharing in the motherboard manual before adding a second drive.

For most desktops the honest answer is that any decent SSD is enough, capacity matters more than interface, and the money saved is better spent on memory. For a server handling concurrent load, the NVMe advantage is real and large — which is exactly the case the marketing number happens to describe.