Zstandard (.zst)

Fast modern compression format.

Overview

A zstd frame begins with magic 28 B5 2F FD and a frame header (window descriptor, dictionary ID, content size, checksum flag), followed by blocks (raw, RLE or compressed) and an optional XXH64 checksum. Compressed blocks use Huffman-coded literals and FSE-coded sequences. Skippable frames allow metadata.

Where it is used

  • Linux packages and kernel images, filesystem compression (Btrfs, ZFS, SquashFS).
  • Backups and databases.

History

  • Developed by Yann Collet at Facebook (2015-2016); standardized in RFC 8878.

Status today

  • Rapidly growing adoption; actively maintained.

Structures read by the implementation

Offsets are in bytes from the start of the buffer named in each heading. Field names are taken from the implementation and are not an official specification. Fixed-width integers are shown as u/i (unsigned/signed) followed by bit width and byte order.

ZstdDecoder.Decompress buffer src ยท ZstdDecoder.cs

OffsetSizeTypeFieldNotes
pos4u32 LEAgic
pos + 44u32 LESkip

Other fields read

Read inOffsetSizeTypeFieldNotes
ZstdDecoder0x04u32 LEโ€“

Constants

General Zstd

NameValueTypeNotes
Magic[0x28, 0xB5, 0x2F, 0xFD]byte[]

—- Literals ——————————————————————————- Zstd

NameValueTypeNotes
jt6int

General ZstdDecoder

NameValueTypeNotes
FrameMagic0xFD2FB528uint
LiteralBase[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 22, 24, 28, 32, 40, …int[]
LiteralBits[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 3, 3, 4, 6, 7, 8, 9,…byte[]

General ZstdArchiveModule

NameValueTypeNotes
Magic[0x28, 0xB5, 0x2F, 0xFD]byte[]

Implementation limits reader

ClassNameValueNotes
ZstdDecoderMaxLiterals128 * 1024
Structures and constants were extracted from: Carving/Plugins/Zstd/Zstd.cs, IO/Compression/ZstdDecoder.cs, Modules/FileSystems/ZstdArchiveModule.cs. Prose sections are a summary from public sources; verify details before relying on them.