Squeezing another 10% out of iPhone photos, losslessly
I wrote a tool that makes iPhone photos and Live Photo videos about 10% smaller and gives back the exact original file on the way out. General-purpose compressors manage about 3% on the same files. This is how it works, and what I tried that didn't.
Why these files don't compress
A HEIC photo is about 93% HEVC video data, and a Live Photo video is about 91%. The rest is container boxes, metadata and, in the video, a short uncompressed audio track.
That HEVC data has already been through CABAC, the arithmetic coder at the end of every HEVC encoder. CABAC is adaptive and close to the entropy of what it codes, so to xz or zstd its output looks like random bytes. On my test files, xz -9e saved 2.95% and zstd at its maximum saved 2.43%. Part of even that came from the audio and metadata rather than the video.
So the only way to do much better is to stop treating the file as bytes and start treating it as HEVC.
The idea
CABAC codes a long run of binary decisions, or bins: is this coefficient non-zero, is this block split, which intra mode was chosen. Each bin is coded with a probability from a small adaptive state, picked by a fixed rule in the standard. Those rules are deliberately simple, because a hardware decoder has to run them in real time. A model that can afford to look at more context should predict the same bins better.
So hevcpress does this:
- Parse the HEVC bitstream and decode every CABAC bin, without reconstructing any pixels.
- Code each bin again with a stronger model and a plain binary arithmetic coder.
- To decompress, decode the bins from that stream and feed them to a CABAC encoder, which rebuilds the original slice data bit for bit.
None of this is new in principle. Dropbox did the same for JPEG with Lepton, which saves about 22% because JPEG's Huffman coding leaves a lot of slack. They also tried it for H.264 with avrecode, now abandoned. I couldn't find anything public that does it for HEVC.
A parser that has to be exactly right
The hard part is that nothing can be approximate. If the parser picks the wrong context for a single bin, every bin after it decodes as garbage. So I wrote a full HEVC slice-data parser in Rust: parameter sets, slice headers, the coding tree, prediction units, transform trees and residual coding, for I, P and B slices. It reads bins and stops there. There is no intra prediction, motion compensation, transform or loop filter.
The nice thing about CABAC is that it tells you when you've got it wrong. A slice is only parsed correctly if the decoder reaches the end-of-slice flag exactly at the last coded block, within the last byte of the data. A bug almost never survives that. A second check: adding up the cost of every bin under CABAC's own probabilities should give the slice's actual size. On my files it came within 0.05%.
Apple's encoder threw one curve ball. It uses wavefront parallel processing, where each row of 32x32 blocks is a separate arithmetic-coded substream whose state is inherited from the row above. Once that was handled, all 799 slices in the first six files parsed to the exact byte.
Decompression needs the reverse: a CABAC encoder that, given the same bins, writes the same bytes the iPhone wrote. That isn't guaranteed, since an encoder has some freedom in how it ends each substream. So before compressing a slice, hevcpress replays its bins through that encoder and compares. Every slice from Apple's encoder has reproduced exactly so far. One that didn't would simply stay as it is.
The model
Each bin is predicted by mixing up to fourteen hashed contexts with a small logistic mixer, in the style of the PAQ compressors. The contexts are things CABAC's fixed rules don't look at: coefficient levels already decoded in this block and in the blocks to the left and above, the matching chroma or luma block, the current QP, the intra mode, the neighbouring blocks' sizes, SAO settings and so on. Each context keeps a fast and a slow probability, and the mixer learns which ones to trust.
One rule matters more than any context: the model is integer-only. The decompressor has to compute every probability exactly as the compressor did, on any machine and any compiler. Floating point, or a log() from the system maths library, can differ in the last bit, and then the stream silently decodes to garbage. So the squash and stretch functions are fixed lookup tables in the source, and all the arithmetic is integer.
Here is where the bits go in one of the Live Photo videos, and how much the model saves on each kind of bin:
| syntax element | share of the bits | model saving |
|---|---|---|
| significant-coefficient flags | 25.8% | 8.2% |
| coefficient signs | 13.8% | 2.2% |
| coefficient levels (remainder) | 11.3% | 5.8% |
| last-coefficient position | 10.0% | 9.8% |
| greater-than-one flags | 8.8% | 9.5% |
| QP deltas | 3.8% | 14.9% |
| coded-block flags | 4.8% | 18-19% |
Coefficient data is about 70% of everything, and that is where savings are hardest. Signs are close to coin flips. The small syntax elements save more in percentage terms but don't add up to much. Across the whole video, slice data shrinks by about 8.5%, and photos, which are all intra-coded, by about 6%.
The audio in Live Photos is uncompressed 16-bit PCM, so it gets its own path: a two-stage NLMS predictor whose residual goes through the same model. It comes out 15-27% smaller than FLAC at its highest setting.
Results
Photos come out 8.1-8.9% smaller and videos 10.0-13.5%. Videos do better because P and B frames carry more syntax for the model to work on, and because of the audio. Every file comes back byte-identical. One photo and video pair was added after I'd finished tuning, as a check that the model wasn't fitted to the others; it landed in the same range.
paq8px, one of the strongest general-purpose compressors there is, was too slow to run on everything at about 2 KB/s. On one photo it saved 3.97% in 590 seconds. hevcpress saved 8.91% on the same file in about 6.5 seconds. Since paq8px is close to the best generic modelling can do, the difference comes from understanding the format.
On a 2019 MacBook Pro, a 1.5 MB photo compresses in about 6.5 seconds and restores in 2.5. That includes a full decompression to check the result before anything is written.
What didn't work
Some of the ideas I was most confident about made no difference at all.
- A bigger or smarter mixer. Dual-rate probabilities, a second weight set selected by the bin's own context, learning-rate tuning: a few tenths of a percent between them. The engine wasn't the limit.
- Warm-starting from other files. I expected a prior trained on earlier photos to help small files. It saved about 0.1%, because the model learns the statistics within the first few tiles of a photo anyway.
- Context from the previous frame. Where the previous picture had busy coefficients, the next one probably will too. It did nothing measurable; the neighbouring blocks in the same picture already say the same thing.
- Decoding the pixels. This was the big one. The obvious next step was a full HEVC decoder, so the model could look at the actual image around each block. Before writing thousands of lines for it, I faked it: ffmpeg decoded the pictures, and I fed the model the texture and edge direction of the already-decoded neighbourhood. The features were genuinely informative, correlating at 0.48 with coefficient size. They saved 0.05-0.12%. The coefficients of the neighbouring blocks were already telling the model the same thing.
That last experiment took an afternoon and saved weeks. A full decoder might still add a point or two through sign prediction and template-matched intra modes, but not enough to be worth it.
Where it ends up
About 10% looks like roughly what these files have to give. Apple's encoder is good, and CABAC is close to the limit of what you can do with the information inside the bitstream.
Whether that's worth having depends on scale. For one person's photo library it mostly isn't: 10% of 500 GB is a few dollars of storage a month, and a .hvp file has to be unpacked before you can look at it. For something storing photos by the petabyte, the sums look different, which is exactly why Lepton exists.
If you want to make photos smaller and still be able to open them, the answer is a lossy format, and it costs something real. I measured that too. At 30% smaller, JPEG XL scores about 82 on SSIMULACRA2 (around 90 is visually lossless). It drops the HDR gain map and depth data unless you go out of your way to keep them, and the loss is permanent.
The code is on GitHub at 47-Cloud/hevcpress, under MIT or Apache-2.0, with a macOS binary in the first release. It verifies every file before writing it and carries a checksum, but it's young: keep your originals.