1.. SPDX-License-Identifier: GPL-2.0
2
3.. _fsverity:
4
5=======================================================
6fs-verity: read-only file-based authenticity protection
7=======================================================
8
9Introduction
10============
11
12fs-verity (``fs/verity/``) is a support layer that filesystems can
13hook into to support transparent integrity and authenticity protection
14of read-only files.  Currently, it is supported by the ext4, f2fs, and
15btrfs filesystems.  Like fscrypt, not too much filesystem-specific
16code is needed to support fs-verity.
17
18fs-verity is similar to `dm-verity
19<https://www.kernel.org/doc/Documentation/device-mapper/verity.txt>`_
20but works on files rather than block devices.  On regular files on
21filesystems supporting fs-verity, userspace can execute an ioctl that
22causes the filesystem to build a Merkle tree for the file and persist
23it to a filesystem-specific location associated with the file.
24
25After this, the file is made readonly, and all reads from the file are
26automatically verified against the file's Merkle tree.  Reads of any
27corrupted data, including mmap reads, will fail.
28
29Userspace can use another ioctl to retrieve the root hash (actually
30the "fs-verity file digest", which is a hash that includes the Merkle
31tree root hash) that fs-verity is enforcing for the file.  This ioctl
32executes in constant time, regardless of the file size.
33
34fs-verity is essentially a way to hash a file in constant time,
35subject to the caveat that reads which would violate the hash will
36fail at runtime.
37
38Use cases
39=========
40
41By itself, the base fs-verity feature only provides integrity
42protection, i.e. detection of accidental (non-malicious) corruption.
43
44However, because fs-verity makes retrieving the file hash extremely
45efficient, it's primarily meant to be used as a tool to support
46authentication (detection of malicious modifications) or auditing
47(logging file hashes before use).
48
49Trusted userspace code (e.g. operating system code running on a
50read-only partition that is itself authenticated by dm-verity) can
51authenticate the contents of an fs-verity file by using the
52`FS_IOC_MEASURE_VERITY`_ ioctl to retrieve its hash, then verifying a
53digital signature of it.
54
55A standard file hash could be used instead of fs-verity.  However,
56this is inefficient if the file is large and only a small portion may
57be accessed.  This is often the case for Android application package
58(APK) files, for example.  These typically contain many translations,
59classes, and other resources that are infrequently or even never
60accessed on a particular device.  It would be slow and wasteful to
61read and hash the entire file before starting the application.
62
63Unlike an ahead-of-time hash, fs-verity also re-verifies data each
64time it's paged in.  This ensures that malicious disk firmware can't
65undetectably change the contents of the file at runtime.
66
67fs-verity does not replace or obsolete dm-verity.  dm-verity should
68still be used on read-only filesystems.  fs-verity is for files that
69must live on a read-write filesystem because they are independently
70updated and potentially user-installed, so dm-verity cannot be used.
71
72The base fs-verity feature is a hashing mechanism only; actually
73authenticating the files may be done by:
74
75* Userspace-only
76
77* Builtin signature verification + userspace policy
78
79  fs-verity optionally supports a simple signature verification
80  mechanism where users can configure the kernel to require that
81  all fs-verity files be signed by a key loaded into a keyring;
82  see `Built-in signature verification`_.
83
84* Integrity Measurement Architecture (IMA)
85
86  IMA supports including fs-verity file digests and signatures in the
87  IMA measurement list and verifying fs-verity based file signatures
88  stored as security.ima xattrs, based on policy.
89
90
91User API
92========
93
94FS_IOC_ENABLE_VERITY
95--------------------
96
97The FS_IOC_ENABLE_VERITY ioctl enables fs-verity on a file.  It takes
98in a pointer to a struct fsverity_enable_arg, defined as
99follows::
100
101    struct fsverity_enable_arg {
102            __u32 version;
103            __u32 hash_algorithm;
104            __u32 block_size;
105            __u32 salt_size;
106            __u64 salt_ptr;
107            __u32 sig_size;
108            __u32 __reserved1;
109            __u64 sig_ptr;
110            __u64 __reserved2[11];
111    };
112
113This structure contains the parameters of the Merkle tree to build for
114the file, and optionally contains a signature.  It must be initialized
115as follows:
116
117- ``version`` must be 1.
118- ``hash_algorithm`` must be the identifier for the hash algorithm to
119  use for the Merkle tree, such as FS_VERITY_HASH_ALG_SHA256.  See
120  ``include/uapi/linux/fsverity.h`` for the list of possible values.
121- ``block_size`` must be the Merkle tree block size.  Currently, this
122  must be equal to the system page size, which is usually 4096 bytes.
123  Other sizes may be supported in the future.  This value is not
124  necessarily the same as the filesystem block size.
125- ``salt_size`` is the size of the salt in bytes, or 0 if no salt is
126  provided.  The salt is a value that is prepended to every hashed
127  block; it can be used to personalize the hashing for a particular
128  file or device.  Currently the maximum salt size is 32 bytes.
129- ``salt_ptr`` is the pointer to the salt, or NULL if no salt is
130  provided.
131- ``sig_size`` is the size of the signature in bytes, or 0 if no
132  signature is provided.  Currently the signature is (somewhat
133  arbitrarily) limited to 16128 bytes.  See `Built-in signature
134  verification`_ for more information.
135- ``sig_ptr``  is the pointer to the signature, or NULL if no
136  signature is provided.
137- All reserved fields must be zeroed.
138
139FS_IOC_ENABLE_VERITY causes the filesystem to build a Merkle tree for
140the file and persist it to a filesystem-specific location associated
141with the file, then mark the file as a verity file.  This ioctl may
142take a long time to execute on large files, and it is interruptible by
143fatal signals.
144
145FS_IOC_ENABLE_VERITY checks for write access to the inode.  However,
146it must be executed on an O_RDONLY file descriptor and no processes
147can have the file open for writing.  Attempts to open the file for
148writing while this ioctl is executing will fail with ETXTBSY.  (This
149is necessary to guarantee that no writable file descriptors will exist
150after verity is enabled, and to guarantee that the file's contents are
151stable while the Merkle tree is being built over it.)
152
153On success, FS_IOC_ENABLE_VERITY returns 0, and the file becomes a
154verity file.  On failure (including the case of interruption by a
155fatal signal), no changes are made to the file.
156
157FS_IOC_ENABLE_VERITY can fail with the following errors:
158
159- ``EACCES``: the process does not have write access to the file
160- ``EBADMSG``: the signature is malformed
161- ``EBUSY``: this ioctl is already running on the file
162- ``EEXIST``: the file already has verity enabled
163- ``EFAULT``: the caller provided inaccessible memory
164- ``EINTR``: the operation was interrupted by a fatal signal
165- ``EINVAL``: unsupported version, hash algorithm, or block size; or
166  reserved bits are set; or the file descriptor refers to neither a
167  regular file nor a directory.
168- ``EISDIR``: the file descriptor refers to a directory
169- ``EKEYREJECTED``: the signature doesn't match the file
170- ``EMSGSIZE``: the salt or signature is too long
171- ``ENOKEY``: the fs-verity keyring doesn't contain the certificate
172  needed to verify the signature
173- ``ENOPKG``: fs-verity recognizes the hash algorithm, but it's not
174  available in the kernel's crypto API as currently configured (e.g.
175  for SHA-512, missing CONFIG_CRYPTO_SHA512).
176- ``ENOTTY``: this type of filesystem does not implement fs-verity
177- ``EOPNOTSUPP``: the kernel was not configured with fs-verity
178  support; or the filesystem superblock has not had the 'verity'
179  feature enabled on it; or the filesystem does not support fs-verity
180  on this file.  (See `Filesystem support`_.)
181- ``EPERM``: the file is append-only; or, a signature is required and
182  one was not provided.
183- ``EROFS``: the filesystem is read-only
184- ``ETXTBSY``: someone has the file open for writing.  This can be the
185  caller's file descriptor, another open file descriptor, or the file
186  reference held by a writable memory map.
187
188FS_IOC_MEASURE_VERITY
189---------------------
190
191The FS_IOC_MEASURE_VERITY ioctl retrieves the digest of a verity file.
192The fs-verity file digest is a cryptographic digest that identifies
193the file contents that are being enforced on reads; it is computed via
194a Merkle tree and is different from a traditional full-file digest.
195
196This ioctl takes in a pointer to a variable-length structure::
197
198    struct fsverity_digest {
199            __u16 digest_algorithm;
200            __u16 digest_size; /* input/output */
201            __u8 digest[];
202    };
203
204``digest_size`` is an input/output field.  On input, it must be
205initialized to the number of bytes allocated for the variable-length
206``digest`` field.
207
208On success, 0 is returned and the kernel fills in the structure as
209follows:
210
211- ``digest_algorithm`` will be the hash algorithm used for the file
212  digest.  It will match ``fsverity_enable_arg::hash_algorithm``.
213- ``digest_size`` will be the size of the digest in bytes, e.g. 32
214  for SHA-256.  (This can be redundant with ``digest_algorithm``.)
215- ``digest`` will be the actual bytes of the digest.
216
217FS_IOC_MEASURE_VERITY is guaranteed to execute in constant time,
218regardless of the size of the file.
219
220FS_IOC_MEASURE_VERITY can fail with the following errors:
221
222- ``EFAULT``: the caller provided inaccessible memory
223- ``ENODATA``: the file is not a verity file
224- ``ENOTTY``: this type of filesystem does not implement fs-verity
225- ``EOPNOTSUPP``: the kernel was not configured with fs-verity
226  support, or the filesystem superblock has not had the 'verity'
227  feature enabled on it.  (See `Filesystem support`_.)
228- ``EOVERFLOW``: the digest is longer than the specified
229  ``digest_size`` bytes.  Try providing a larger buffer.
230
231FS_IOC_READ_VERITY_METADATA
232---------------------------
233
234The FS_IOC_READ_VERITY_METADATA ioctl reads verity metadata from a
235verity file.  This ioctl is available since Linux v5.12.
236
237This ioctl allows writing a server program that takes a verity file
238and serves it to a client program, such that the client can do its own
239fs-verity compatible verification of the file.  This only makes sense
240if the client doesn't trust the server and if the server needs to
241provide the storage for the client.
242
243This is a fairly specialized use case, and most fs-verity users won't
244need this ioctl.
245
246This ioctl takes in a pointer to the following structure::
247
248   #define FS_VERITY_METADATA_TYPE_MERKLE_TREE     1
249   #define FS_VERITY_METADATA_TYPE_DESCRIPTOR      2
250   #define FS_VERITY_METADATA_TYPE_SIGNATURE       3
251
252   struct fsverity_read_metadata_arg {
253           __u64 metadata_type;
254           __u64 offset;
255           __u64 length;
256           __u64 buf_ptr;
257           __u64 __reserved;
258   };
259
260``metadata_type`` specifies the type of metadata to read:
261
262- ``FS_VERITY_METADATA_TYPE_MERKLE_TREE`` reads the blocks of the
263  Merkle tree.  The blocks are returned in order from the root level
264  to the leaf level.  Within each level, the blocks are returned in
265  the same order that their hashes are themselves hashed.
266  See `Merkle tree`_ for more information.
267
268- ``FS_VERITY_METADATA_TYPE_DESCRIPTOR`` reads the fs-verity
269  descriptor.  See `fs-verity descriptor`_.
270
271- ``FS_VERITY_METADATA_TYPE_SIGNATURE`` reads the signature which was
272  passed to FS_IOC_ENABLE_VERITY, if any.  See `Built-in signature
273  verification`_.
274
275The semantics are similar to those of ``pread()``.  ``offset``
276specifies the offset in bytes into the metadata item to read from, and
277``length`` specifies the maximum number of bytes to read from the
278metadata item.  ``buf_ptr`` is the pointer to the buffer to read into,
279cast to a 64-bit integer.  ``__reserved`` must be 0.  On success, the
280number of bytes read is returned.  0 is returned at the end of the
281metadata item.  The returned length may be less than ``length``, for
282example if the ioctl is interrupted.
283
284The metadata returned by FS_IOC_READ_VERITY_METADATA isn't guaranteed
285to be authenticated against the file digest that would be returned by
286`FS_IOC_MEASURE_VERITY`_, as the metadata is expected to be used to
287implement fs-verity compatible verification anyway (though absent a
288malicious disk, the metadata will indeed match).  E.g. to implement
289this ioctl, the filesystem is allowed to just read the Merkle tree
290blocks from disk without actually verifying the path to the root node.
291
292FS_IOC_READ_VERITY_METADATA can fail with the following errors:
293
294- ``EFAULT``: the caller provided inaccessible memory
295- ``EINTR``: the ioctl was interrupted before any data was read
296- ``EINVAL``: reserved fields were set, or ``offset + length``
297  overflowed
298- ``ENODATA``: the file is not a verity file, or
299  FS_VERITY_METADATA_TYPE_SIGNATURE was requested but the file doesn't
300  have a built-in signature
301- ``ENOTTY``: this type of filesystem does not implement fs-verity, or
302  this ioctl is not yet implemented on it
303- ``EOPNOTSUPP``: the kernel was not configured with fs-verity
304  support, or the filesystem superblock has not had the 'verity'
305  feature enabled on it.  (See `Filesystem support`_.)
306
307FS_IOC_GETFLAGS
308---------------
309
310The existing ioctl FS_IOC_GETFLAGS (which isn't specific to fs-verity)
311can also be used to check whether a file has fs-verity enabled or not.
312To do so, check for FS_VERITY_FL (0x00100000) in the returned flags.
313
314The verity flag is not settable via FS_IOC_SETFLAGS.  You must use
315FS_IOC_ENABLE_VERITY instead, since parameters must be provided.
316
317statx
318-----
319
320Since Linux v5.5, the statx() system call sets STATX_ATTR_VERITY if
321the file has fs-verity enabled.  This can perform better than
322FS_IOC_GETFLAGS and FS_IOC_MEASURE_VERITY because it doesn't require
323opening the file, and opening verity files can be expensive.
324
325Accessing verity files
326======================
327
328Applications can transparently access a verity file just like a
329non-verity one, with the following exceptions:
330
331- Verity files are readonly.  They cannot be opened for writing or
332  truncate()d, even if the file mode bits allow it.  Attempts to do
333  one of these things will fail with EPERM.  However, changes to
334  metadata such as owner, mode, timestamps, and xattrs are still
335  allowed, since these are not measured by fs-verity.  Verity files
336  can also still be renamed, deleted, and linked to.
337
338- Direct I/O is not supported on verity files.  Attempts to use direct
339  I/O on such files will fall back to buffered I/O.
340
341- DAX (Direct Access) is not supported on verity files, because this
342  would circumvent the data verification.
343
344- Reads of data that doesn't match the verity Merkle tree will fail
345  with EIO (for read()) or SIGBUS (for mmap() reads).
346
347- If the sysctl "fs.verity.require_signatures" is set to 1 and the
348  file is not signed by a key in the fs-verity keyring, then opening
349  the file will fail.  See `Built-in signature verification`_.
350
351Direct access to the Merkle tree is not supported.  Therefore, if a
352verity file is copied, or is backed up and restored, then it will lose
353its "verity"-ness.  fs-verity is primarily meant for files like
354executables that are managed by a package manager.
355
356File digest computation
357=======================
358
359This section describes how fs-verity hashes the file contents using a
360Merkle tree to produce the digest which cryptographically identifies
361the file contents.  This algorithm is the same for all filesystems
362that support fs-verity.
363
364Userspace only needs to be aware of this algorithm if it needs to
365compute fs-verity file digests itself, e.g. in order to sign files.
366
367.. _fsverity_merkle_tree:
368
369Merkle tree
370-----------
371
372The file contents is divided into blocks, where the block size is
373configurable but is usually 4096 bytes.  The end of the last block is
374zero-padded if needed.  Each block is then hashed, producing the first
375level of hashes.  Then, the hashes in this first level are grouped
376into 'blocksize'-byte blocks (zero-padding the ends as needed) and
377these blocks are hashed, producing the second level of hashes.  This
378proceeds up the tree until only a single block remains.  The hash of
379this block is the "Merkle tree root hash".
380
381If the file fits in one block and is nonempty, then the "Merkle tree
382root hash" is simply the hash of the single data block.  If the file
383is empty, then the "Merkle tree root hash" is all zeroes.
384
385The "blocks" here are not necessarily the same as "filesystem blocks".
386
387If a salt was specified, then it's zero-padded to the closest multiple
388of the input size of the hash algorithm's compression function, e.g.
38964 bytes for SHA-256 or 128 bytes for SHA-512.  The padded salt is
390prepended to every data or Merkle tree block that is hashed.
391
392The purpose of the block padding is to cause every hash to be taken
393over the same amount of data, which simplifies the implementation and
394keeps open more possibilities for hardware acceleration.  The purpose
395of the salt padding is to make the salting "free" when the salted hash
396state is precomputed, then imported for each hash.
397
398Example: in the recommended configuration of SHA-256 and 4K blocks,
399128 hash values fit in each block.  Thus, each level of the Merkle
400tree is approximately 128 times smaller than the previous, and for
401large files the Merkle tree's size converges to approximately 1/127 of
402the original file size.  However, for small files, the padding is
403significant, making the space overhead proportionally more.
404
405.. _fsverity_descriptor:
406
407fs-verity descriptor
408--------------------
409
410By itself, the Merkle tree root hash is ambiguous.  For example, it
411can't a distinguish a large file from a small second file whose data
412is exactly the top-level hash block of the first file.  Ambiguities
413also arise from the convention of padding to the next block boundary.
414
415To solve this problem, the fs-verity file digest is actually computed
416as a hash of the following structure, which contains the Merkle tree
417root hash as well as other fields such as the file size::
418
419    struct fsverity_descriptor {
420            __u8 version;           /* must be 1 */
421            __u8 hash_algorithm;    /* Merkle tree hash algorithm */
422            __u8 log_blocksize;     /* log2 of size of data and tree blocks */
423            __u8 salt_size;         /* size of salt in bytes; 0 if none */
424            __le32 __reserved_0x04; /* must be 0 */
425            __le64 data_size;       /* size of file the Merkle tree is built over */
426            __u8 root_hash[64];     /* Merkle tree root hash */
427            __u8 salt[32];          /* salt prepended to each hashed block */
428            __u8 __reserved[144];   /* must be 0's */
429    };
430
431Built-in signature verification
432===============================
433
434With CONFIG_FS_VERITY_BUILTIN_SIGNATURES=y, fs-verity supports putting
435a portion of an authentication policy (see `Use cases`_) in the
436kernel.  Specifically, it adds support for:
437
4381. At fs-verity module initialization time, a keyring ".fs-verity" is
439   created.  The root user can add trusted X.509 certificates to this
440   keyring using the add_key() system call, then (when done)
441   optionally use keyctl_restrict_keyring() to prevent additional
442   certificates from being added.
443
4442. `FS_IOC_ENABLE_VERITY`_ accepts a pointer to a PKCS#7 formatted
445   detached signature in DER format of the file's fs-verity digest.
446   On success, this signature is persisted alongside the Merkle tree.
447   Then, any time the file is opened, the kernel will verify the
448   file's actual digest against this signature, using the certificates
449   in the ".fs-verity" keyring.
450
4513. A new sysctl "fs.verity.require_signatures" is made available.
452   When set to 1, the kernel requires that all verity files have a
453   correctly signed digest as described in (2).
454
455fs-verity file digests must be signed in the following format, which
456is similar to the structure used by `FS_IOC_MEASURE_VERITY`_::
457
458    struct fsverity_formatted_digest {
459            char magic[8];                  /* must be "FSVerity" */
460            __le16 digest_algorithm;
461            __le16 digest_size;
462            __u8 digest[];
463    };
464
465fs-verity's built-in signature verification support is meant as a
466relatively simple mechanism that can be used to provide some level of
467authenticity protection for verity files, as an alternative to doing
468the signature verification in userspace or using IMA-appraisal.
469However, with this mechanism, userspace programs still need to check
470that the verity bit is set, and there is no protection against verity
471files being swapped around.
472
473Filesystem support
474==================
475
476fs-verity is supported by several filesystems, described below.  The
477CONFIG_FS_VERITY kconfig option must be enabled to use fs-verity on
478any of these filesystems.
479
480``include/linux/fsverity.h`` declares the interface between the
481``fs/verity/`` support layer and filesystems.  Briefly, filesystems
482must provide an ``fsverity_operations`` structure that provides
483methods to read and write the verity metadata to a filesystem-specific
484location, including the Merkle tree blocks and
485``fsverity_descriptor``.  Filesystems must also call functions in
486``fs/verity/`` at certain times, such as when a file is opened or when
487pages have been read into the pagecache.  (See `Verifying data`_.)
488
489ext4
490----
491
492ext4 supports fs-verity since Linux v5.4 and e2fsprogs v1.45.2.
493
494To create verity files on an ext4 filesystem, the filesystem must have
495been formatted with ``-O verity`` or had ``tune2fs -O verity`` run on
496it.  "verity" is an RO_COMPAT filesystem feature, so once set, old
497kernels will only be able to mount the filesystem readonly, and old
498versions of e2fsck will be unable to check the filesystem.  Moreover,
499currently ext4 only supports mounting a filesystem with the "verity"
500feature when its block size is equal to PAGE_SIZE (often 4096 bytes).
501
502ext4 sets the EXT4_VERITY_FL on-disk inode flag on verity files.  It
503can only be set by `FS_IOC_ENABLE_VERITY`_, and it cannot be cleared.
504
505ext4 also supports encryption, which can be used simultaneously with
506fs-verity.  In this case, the plaintext data is verified rather than
507the ciphertext.  This is necessary in order to make the fs-verity file
508digest meaningful, since every file is encrypted differently.
509
510ext4 stores the verity metadata (Merkle tree and fsverity_descriptor)
511past the end of the file, starting at the first 64K boundary beyond
512i_size.  This approach works because (a) verity files are readonly,
513and (b) pages fully beyond i_size aren't visible to userspace but can
514be read/written internally by ext4 with only some relatively small
515changes to ext4.  This approach avoids having to depend on the
516EA_INODE feature and on rearchitecturing ext4's xattr support to
517support paging multi-gigabyte xattrs into memory, and to support
518encrypting xattrs.  Note that the verity metadata *must* be encrypted
519when the file is, since it contains hashes of the plaintext data.
520
521Currently, ext4 verity only supports the case where the Merkle tree
522block size, filesystem block size, and page size are all the same.  It
523also only supports extent-based files.
524
525f2fs
526----
527
528f2fs supports fs-verity since Linux v5.4 and f2fs-tools v1.11.0.
529
530To create verity files on an f2fs filesystem, the filesystem must have
531been formatted with ``-O verity``.
532
533f2fs sets the FADVISE_VERITY_BIT on-disk inode flag on verity files.
534It can only be set by `FS_IOC_ENABLE_VERITY`_, and it cannot be
535cleared.
536
537Like ext4, f2fs stores the verity metadata (Merkle tree and
538fsverity_descriptor) past the end of the file, starting at the first
53964K boundary beyond i_size.  See explanation for ext4 above.
540Moreover, f2fs supports at most 4096 bytes of xattr entries per inode
541which wouldn't be enough for even a single Merkle tree block.
542
543Currently, f2fs verity only supports a Merkle tree block size of 4096.
544Also, f2fs doesn't support enabling verity on files that currently
545have atomic or volatile writes pending.
546
547btrfs
548-----
549
550btrfs supports fs-verity since Linux v5.15.  Verity-enabled inodes are
551marked with a RO_COMPAT inode flag, and the verity metadata is stored
552in separate btree items.
553
554Implementation details
555======================
556
557Verifying data
558--------------
559
560fs-verity ensures that all reads of a verity file's data are verified,
561regardless of which syscall is used to do the read (e.g. mmap(),
562read(), pread()) and regardless of whether it's the first read or a
563later read (unless the later read can return cached data that was
564already verified).  Below, we describe how filesystems implement this.
565
566Pagecache
567~~~~~~~~~
568
569For filesystems using Linux's pagecache, the ``->read_folio()`` and
570``->readahead()`` methods must be modified to verify pages before they
571are marked Uptodate.  Merely hooking ``->read_iter()`` would be
572insufficient, since ``->read_iter()`` is not used for memory maps.
573
574Therefore, fs/verity/ provides a function fsverity_verify_page() which
575verifies a page that has been read into the pagecache of a verity
576inode, but is still locked and not Uptodate, so it's not yet readable
577by userspace.  As needed to do the verification,
578fsverity_verify_page() will call back into the filesystem to read
579Merkle tree pages via fsverity_operations::read_merkle_tree_page().
580
581fsverity_verify_page() returns false if verification failed; in this
582case, the filesystem must not set the page Uptodate.  Following this,
583as per the usual Linux pagecache behavior, attempts by userspace to
584read() from the part of the file containing the page will fail with
585EIO, and accesses to the page within a memory map will raise SIGBUS.
586
587fsverity_verify_page() currently only supports the case where the
588Merkle tree block size is equal to PAGE_SIZE (often 4096 bytes).
589
590In principle, fsverity_verify_page() verifies the entire path in the
591Merkle tree from the data page to the root hash.  However, for
592efficiency the filesystem may cache the hash pages.  Therefore,
593fsverity_verify_page() only ascends the tree reading hash pages until
594an already-verified hash page is seen, as indicated by the PageChecked
595bit being set.  It then verifies the path to that page.
596
597This optimization, which is also used by dm-verity, results in
598excellent sequential read performance.  This is because usually (e.g.
599127 in 128 times for 4K blocks and SHA-256) the hash page from the
600bottom level of the tree will already be cached and checked from
601reading a previous data page.  However, random reads perform worse.
602
603Block device based filesystems
604~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
605
606Block device based filesystems (e.g. ext4 and f2fs) in Linux also use
607the pagecache, so the above subsection applies too.  However, they
608also usually read many pages from a file at once, grouped into a
609structure called a "bio".  To make it easier for these types of
610filesystems to support fs-verity, fs/verity/ also provides a function
611fsverity_verify_bio() which verifies all pages in a bio.
612
613ext4 and f2fs also support encryption.  If a verity file is also
614encrypted, the pages must be decrypted before being verified.  To
615support this, these filesystems allocate a "post-read context" for
616each bio and store it in ``->bi_private``::
617
618    struct bio_post_read_ctx {
619           struct bio *bio;
620           struct work_struct work;
621           unsigned int cur_step;
622           unsigned int enabled_steps;
623    };
624
625``enabled_steps`` is a bitmask that specifies whether decryption,
626verity, or both is enabled.  After the bio completes, for each needed
627postprocessing step the filesystem enqueues the bio_post_read_ctx on a
628workqueue, and then the workqueue work does the decryption or
629verification.  Finally, pages where no decryption or verity error
630occurred are marked Uptodate, and the pages are unlocked.
631
632On many filesystems, files can contain holes.  Normally,
633``->readahead()`` simply zeroes holes and sets the corresponding pages
634Uptodate; no bios are issued.  To prevent this case from bypassing
635fs-verity, these filesystems use fsverity_verify_page() to verify hole
636pages.
637
638Filesystems also disable direct I/O on verity files, since otherwise
639direct I/O would bypass fs-verity.
640
641Userspace utility
642=================
643
644This document focuses on the kernel, but a userspace utility for
645fs-verity can be found at:
646
647	https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/fsverity-utils.git
648
649See the README.md file in the fsverity-utils source tree for details,
650including examples of setting up fs-verity protected files.
651
652Tests
653=====
654
655To test fs-verity, use xfstests.  For example, using `kvm-xfstests
656<https://github.com/tytso/xfstests-bld/blob/master/Documentation/kvm-quickstart.md>`_::
657
658    kvm-xfstests -c ext4,f2fs,btrfs -g verity
659
660FAQ
661===
662
663This section answers frequently asked questions about fs-verity that
664weren't already directly answered in other parts of this document.
665
666:Q: Why isn't fs-verity part of IMA?
667:A: fs-verity and IMA (Integrity Measurement Architecture) have
668    different focuses.  fs-verity is a filesystem-level mechanism for
669    hashing individual files using a Merkle tree.  In contrast, IMA
670    specifies a system-wide policy that specifies which files are
671    hashed and what to do with those hashes, such as log them,
672    authenticate them, or add them to a measurement list.
673
674    IMA supports the fs-verity hashing mechanism as an alternative
675    to full file hashes, for those who want the performance and
676    security benefits of the Merkle tree based hash.  However, it
677    doesn't make sense to force all uses of fs-verity to be through
678    IMA.  fs-verity already meets many users' needs even as a
679    standalone filesystem feature, and it's testable like other
680    filesystem features e.g. with xfstests.
681
682:Q: Isn't fs-verity useless because the attacker can just modify the
683    hashes in the Merkle tree, which is stored on-disk?
684:A: To verify the authenticity of an fs-verity file you must verify
685    the authenticity of the "fs-verity file digest", which
686    incorporates the root hash of the Merkle tree.  See `Use cases`_.
687
688:Q: Isn't fs-verity useless because the attacker can just replace a
689    verity file with a non-verity one?
690:A: See `Use cases`_.  In the initial use case, it's really trusted
691    userspace code that authenticates the files; fs-verity is just a
692    tool to do this job efficiently and securely.  The trusted
693    userspace code will consider non-verity files to be inauthentic.
694
695:Q: Why does the Merkle tree need to be stored on-disk?  Couldn't you
696    store just the root hash?
697:A: If the Merkle tree wasn't stored on-disk, then you'd have to
698    compute the entire tree when the file is first accessed, even if
699    just one byte is being read.  This is a fundamental consequence of
700    how Merkle tree hashing works.  To verify a leaf node, you need to
701    verify the whole path to the root hash, including the root node
702    (the thing which the root hash is a hash of).  But if the root
703    node isn't stored on-disk, you have to compute it by hashing its
704    children, and so on until you've actually hashed the entire file.
705
706    That defeats most of the point of doing a Merkle tree-based hash,
707    since if you have to hash the whole file ahead of time anyway,
708    then you could simply do sha256(file) instead.  That would be much
709    simpler, and a bit faster too.
710
711    It's true that an in-memory Merkle tree could still provide the
712    advantage of verification on every read rather than just on the
713    first read.  However, it would be inefficient because every time a
714    hash page gets evicted (you can't pin the entire Merkle tree into
715    memory, since it may be very large), in order to restore it you
716    again need to hash everything below it in the tree.  This again
717    defeats most of the point of doing a Merkle tree-based hash, since
718    a single block read could trigger re-hashing gigabytes of data.
719
720:Q: But couldn't you store just the leaf nodes and compute the rest?
721:A: See previous answer; this really just moves up one level, since
722    one could alternatively interpret the data blocks as being the
723    leaf nodes of the Merkle tree.  It's true that the tree can be
724    computed much faster if the leaf level is stored rather than just
725    the data, but that's only because each level is less than 1% the
726    size of the level below (assuming the recommended settings of
727    SHA-256 and 4K blocks).  For the exact same reason, by storing
728    "just the leaf nodes" you'd already be storing over 99% of the
729    tree, so you might as well simply store the whole tree.
730
731:Q: Can the Merkle tree be built ahead of time, e.g. distributed as
732    part of a package that is installed to many computers?
733:A: This isn't currently supported.  It was part of the original
734    design, but was removed to simplify the kernel UAPI and because it
735    wasn't a critical use case.  Files are usually installed once and
736    used many times, and cryptographic hashing is somewhat fast on
737    most modern processors.
738
739:Q: Why doesn't fs-verity support writes?
740:A: Write support would be very difficult and would require a
741    completely different design, so it's well outside the scope of
742    fs-verity.  Write support would require:
743
744    - A way to maintain consistency between the data and hashes,
745      including all levels of hashes, since corruption after a crash
746      (especially of potentially the entire file!) is unacceptable.
747      The main options for solving this are data journalling,
748      copy-on-write, and log-structured volume.  But it's very hard to
749      retrofit existing filesystems with new consistency mechanisms.
750      Data journalling is available on ext4, but is very slow.
751
752    - Rebuilding the Merkle tree after every write, which would be
753      extremely inefficient.  Alternatively, a different authenticated
754      dictionary structure such as an "authenticated skiplist" could
755      be used.  However, this would be far more complex.
756
757    Compare it to dm-verity vs. dm-integrity.  dm-verity is very
758    simple: the kernel just verifies read-only data against a
759    read-only Merkle tree.  In contrast, dm-integrity supports writes
760    but is slow, is much more complex, and doesn't actually support
761    full-device authentication since it authenticates each sector
762    independently, i.e. there is no "root hash".  It doesn't really
763    make sense for the same device-mapper target to support these two
764    very different cases; the same applies to fs-verity.
765
766:Q: Since verity files are immutable, why isn't the immutable bit set?
767:A: The existing "immutable" bit (FS_IMMUTABLE_FL) already has a
768    specific set of semantics which not only make the file contents
769    read-only, but also prevent the file from being deleted, renamed,
770    linked to, or having its owner or mode changed.  These extra
771    properties are unwanted for fs-verity, so reusing the immutable
772    bit isn't appropriate.
773
774:Q: Why does the API use ioctls instead of setxattr() and getxattr()?
775:A: Abusing the xattr interface for basically arbitrary syscalls is
776    heavily frowned upon by most of the Linux filesystem developers.
777    An xattr should really just be an xattr on-disk, not an API to
778    e.g. magically trigger construction of a Merkle tree.
779
780:Q: Does fs-verity support remote filesystems?
781:A: So far all filesystems that have implemented fs-verity support are
782    local filesystems, but in principle any filesystem that can store
783    per-file verity metadata can support fs-verity, regardless of
784    whether it's local or remote.  Some filesystems may have fewer
785    options of where to store the verity metadata; one possibility is
786    to store it past the end of the file and "hide" it from userspace
787    by manipulating i_size.  The data verification functions provided
788    by ``fs/verity/`` also assume that the filesystem uses the Linux
789    pagecache, but both local and remote filesystems normally do so.
790
791:Q: Why is anything filesystem-specific at all?  Shouldn't fs-verity
792    be implemented entirely at the VFS level?
793:A: There are many reasons why this is not possible or would be very
794    difficult, including the following:
795
796    - To prevent bypassing verification, pages must not be marked
797      Uptodate until they've been verified.  Currently, each
798      filesystem is responsible for marking pages Uptodate via
799      ``->readahead()``.  Therefore, currently it's not possible for
800      the VFS to do the verification on its own.  Changing this would
801      require significant changes to the VFS and all filesystems.
802
803    - It would require defining a filesystem-independent way to store
804      the verity metadata.  Extended attributes don't work for this
805      because (a) the Merkle tree may be gigabytes, but many
806      filesystems assume that all xattrs fit into a single 4K
807      filesystem block, and (b) ext4 and f2fs encryption doesn't
808      encrypt xattrs, yet the Merkle tree *must* be encrypted when the
809      file contents are, because it stores hashes of the plaintext
810      file contents.
811
812      So the verity metadata would have to be stored in an actual
813      file.  Using a separate file would be very ugly, since the
814      metadata is fundamentally part of the file to be protected, and
815      it could cause problems where users could delete the real file
816      but not the metadata file or vice versa.  On the other hand,
817      having it be in the same file would break applications unless
818      filesystems' notion of i_size were divorced from the VFS's,
819      which would be complex and require changes to all filesystems.
820
821    - It's desirable that FS_IOC_ENABLE_VERITY uses the filesystem's
822      transaction mechanism so that either the file ends up with
823      verity enabled, or no changes were made.  Allowing intermediate
824      states to occur after a crash may cause problems.
825