ZFS (Zettabyte File System)

ZFS combines a volume manager and a copy-on-write file system in one design with checksums, snapshots and pooled storage. It was created at Sun for Solaris (2005) and lives on in OpenZFS on FreeBSD, Linux, illumos and storage appliances such as TrueNAS. It is actively developed.

Overview

A ZFS pool is a tree of virtual devices. Each device carries four 256 KiB labels holding an XDR-encoded name/value list that describes the pool, and an uberblock array. Blocks are addressed by block pointers that contain checksums, so the pool is self-validating. Datasets are described in the DSL, and file system contents in object sets (DMU) with ZAP directories and dnodes.

History

  • Developed at Sun Microsystems from 2001 (Jeff Bonwick, Matt Ahrens); shipped in Solaris 10 in 2006.
  • OpenZFS became the common open source codebase from 2013 after the illumos fork.

Structures

Offsets are in bytes from the start of the structure named in each heading. Field names are descriptive and are not an official specification. Fixed-width integers are shown as u/i (unsigned/signed) followed by bit width and byte order.

Read Dataset

OffsetSizeTypeFieldNotes
0x308u64 LE/BE–
0x388u64 LE/BE–
0x488u64 LE/BE–
0x708u64 LE/BE–
0x80128bytesPointer

Walk Directory

OffsetSizeTypeFieldNotes
0x88u64 LE/BEHead
0x208u64 LE/BEChild Zap

Parse Sa

OffsetSizeTypeFieldNotes
0x04u32 BE–
0x42u16 LE/BEInfo
lengthOffset2u16 LE/BELength

Parse Znode Phys

OffsetSizeTypeFieldNotes
0x016bytesAccessed
0x1016bytesModified
0x2016bytesChanged
0x3016bytesCreated
0x408u64 LE/BEGeneration
0x488u64 LE/BEMode
0x508u64 LE/BESize
0x588u64 LE/BEParent
0x608u64 LE/BELinks
0x788u64 LE/BEFlags
0x808u64 LE/BEUid
0x888u64 LE/BEGid

Time

OffsetSizeTypeFieldNotes
0x08u64 LE/BESeconds
0x88u64 LE/BENanoseconds

Parse List

OffsetSizeTypeFieldNotes
(cursor + 4)4i32 BECount
(cursor + i * 8)8u64 BE–
(position + 4)4i32 BEDecoded Size
cursor4i32 BEType
cursor4u32 BEValue
cursor8u64 BEValue
position4i32 BEEncoded Size

Read Leaf

OffsetSizeTypeFieldNotes
position + 11u8Integer Length
position + 42u16 LE/BEName Chunk
position + 62u16 LE/BEName Length
position + 82u16 LE/BEValue Chunk
position + 102u16 LE/BEValue Count

Read Zap

OffsetSizeTypeFieldNotes
0x08u64 LE/BEBlock Type
position8u64 LE/BE–
position + 1450bytesName

Read Zap

OffsetSizeTypeFieldNotes
0x08u64 LE/BE–
0x184u32 LE/BE–

Zfs Block Pointer

OffsetSizeTypeFieldNotes
0x01u8Is Hole
0x01u8Cache Key
0x11u8–
0x11u8Cache Key
0x61u8Prop
0xA1u8Birth
0xA1u8Cache Key
0xB1u8Fill
0xC4bytesExpected Checksum

Fletcher2

OffsetSizeTypeFieldNotes
i8u64 LE/BE–
i + 88u64 LE/BE–

Sha256

OffsetSizeTypeFieldNotes
0x08u64 BE–
0x88u64 BE–
0x108u64 BE–
0x188u64 BE–

Verify Embedded

OffsetSizeTypeFieldNotes
tail8u64 BE–
tail + 8 + i * 88u64 LE/BE–

Parse

OffsetSizeTypeFieldNotes
0x01u8Type
0x11u8Indirect Shift
0x21u8Levels
0x31u8Pointer Count
0x41u8Bonus Type
0x71u8Flags
0x82u16 LE/BEBlock Size
0xA2u16 LE/BEBonus Length
0xC1u8Extra
0x108u64 LE/BEMax Block Id
0x188u64 LE/BEUsed
size – 128128bytes–

Parse Uberblock

OffsetSizeTypeFieldNotes
0x08u64 BE–
0x88u64 LE/BEVersion
0x108u64 LE/BETxg
0x208u64 LE/BETimestamp
0x28128bytesRoot

Other fields read

Read inOffsetSizeTypeFieldNotes
Decompress Lz40x04u32 BELength
Add Snapshots0x208u64 LE/BESnap Zap
Parse Znode Phys0x08u64 LE/BE–
Read Stringposition4i32 BELength
Open0x2C08u64 LE/BE–
Read Arrayoffset + 222u16 LE/BEChunk
Assemble Gang512 – 408u64 LELittle Endian
Zfs Block Pointeri * 88u64 LE/BE–
Fletcher4i4u32 BE–

Constants

Zfs Compression

NameValueTypeNotes
Inherit0int
On1int
Off2int
Lzjb3int
Empty4int
Gzip15int
Gzip913int
Zle14int
Lz415int
Zstd16int
MatchBits6int
MatchMin3int

Zfs Zpl

NameValueTypeNotes
SaMagic0x2F505Auint

Zfs Nv List

NameValueTypeNotes
TypeBoolean1int
TypeByte2int
TypeInt163int
TypeUInt164int
TypeInt325int
TypeUInt326int
TypeInt647int
TypeUInt648int
TypeString9int
TypeUInt64Array16int
TypeHrTime18int
TypeNvList19int
TypeNvListArray20int
TypeBooleanValue21int
TypeInt822int
TypeUInt823int

Zfs Const

NameValueTypeNotes
LabelSize256 * 1024int
LabelNvListOffset16 * 1024int
LabelNvListSize112 * 1024int
LabelUberblockOffset128 * 1024int
LabelUberblockSize128 * 1024int
DataStart4L * 1024 * 1024long
UberblockMagic0x00BAB10Culong
EckMagic0x0210DA7AB10C7A11ulong
DnodeSize512int
BlockPointerSize128int

Checksum algorithms (zio_checksum)

NameValueTypeNotes
ChecksumFletcher26int
ChecksumFletcher47int
ChecksumSha2568int

Object set types

NameValueTypeNotes
ObjsetMeta1ulong
ObjsetZfs2ulong
ObjsetZvol3ulong

Bonus types

NameValueTypeNotes
BonusZnode17byte
BonusSa44byte

Zfs Dnode

NameValueTypeNotes
FlagSpillPointer0x4byte

Zfs Module

NameValueTypeNotes
RegistryKey“zfs:registry”string

Implementation limits

ClassNameValueNotes
Zfs DslMaxDepth64
Zfs DslMaxDatasets100_000
Zfs File SystemMaxOlderTxgs16
Zfs File SystemMaxLostEntries100_000
Zfs File SystemMaxVisitedPerTxg200_000
Zfs File SystemMaxSymlinkBytes4096
Zfs Nv ListMaxDepth16
Zfs PoolCacheLimitBytes64L * 1024 * 1024
Zfs PoolMaxGangDepth4
Zfs Object SetMaxZapEntries10_000_000
Zfs DnodeFlagUsedBytes0x1
Zfs ModuleMaxSnapshotsPerDataset64
Zfs ModuleMaxMosAttempts8
Zfs ModuleMaxDestroyedScanTxgs16
Structures and constants were extracted from the implementation. Prose sections are a summary from public sources; verify details before relying on them.