1============================ 2XZ data compression in Linux 3============================ 4 5Introduction 6============ 7 8XZ is a general purpose data compression format with high compression 9ratio and relatively fast decompression. The primary compression 10algorithm (filter) is LZMA2. Additional filters can be used to improve 11compression ratio even further. E.g. Branch/Call/Jump (BCJ) filters 12improve compression ratio of executable data. 13 14The XZ decompressor in Linux is called XZ Embedded. It supports 15the LZMA2 filter and optionally also BCJ filters. CRC32 is supported 16for integrity checking. The home page of XZ Embedded is at 17<https://tukaani.org/xz/embedded.html>, where you can find the 18latest version and also information about using the code outside 19the Linux kernel. 20 21For userspace, XZ Utils provide a zlib-like compression library 22and a gzip-like command line tool. XZ Utils can be downloaded from 23<https://tukaani.org/xz/>. 24 25XZ related components in the kernel 26=================================== 27 28The xz_dec module provides XZ decompressor with single-call (buffer 29to buffer) and multi-call (stateful) APIs. The usage of the xz_dec 30module is documented in include/linux/xz.h. 31 32The xz_dec_test module is for testing xz_dec. xz_dec_test is not 33useful unless you are hacking the XZ decompressor. xz_dec_test 34allocates a char device major dynamically to which one can write 35.xz files from userspace. The decompressed output is thrown away. 36Keep an eye on dmesg to see diagnostics printed by xz_dec_test. 37See the xz_dec_test source code for the details. 38 39For decompressing the kernel image, initramfs, and initrd, there 40is a wrapper function in lib/decompress_unxz.c. Its API is the 41same as in other decompress_*.c files, which is defined in 42include/linux/decompress/generic.h. 43 44scripts/xz_wrap.sh is a wrapper for the xz command line tool found 45from XZ Utils. The wrapper sets compression options to values suitable 46for compressing the kernel image. 47 48For kernel makefiles, two commands are provided for use with 49$(call if_needed). The kernel image should be compressed with 50$(call if_needed,xzkern) which will use a BCJ filter and a big LZMA2 51dictionary. It will also append a four-byte trailer containing the 52uncompressed size of the file, which is needed by the boot code. 53Other things should be compressed with $(call if_needed,xzmisc) 54which will use no BCJ filter and 1 MiB LZMA2 dictionary. 55 56Notes on compression options 57============================ 58 59Since the XZ Embedded supports only streams with no integrity check or 60CRC32, make sure that you don't use some other integrity check type 61when encoding files that are supposed to be decoded by the kernel. With 62liblzma, you need to use either LZMA_CHECK_NONE or LZMA_CHECK_CRC32 63when encoding. With the xz command line tool, use --check=none or 64--check=crc32. 65 66Using CRC32 is strongly recommended unless there is some other layer 67which will verify the integrity of the uncompressed data anyway. 68Double checking the integrity would probably be waste of CPU cycles. 69Note that the headers will always have a CRC32 which will be validated 70by the decoder; you can only change the integrity check type (or 71disable it) for the actual uncompressed data. 72 73In userspace, LZMA2 is typically used with dictionary sizes of several 74megabytes. The decoder needs to have the dictionary in RAM, thus big 75dictionaries cannot be used for files that are intended to be decoded 76by the kernel. 1 MiB is probably the maximum reasonable dictionary 77size for in-kernel use (maybe more is OK for initramfs). The presets 78in XZ Utils may not be optimal when creating files for the kernel, 79so don't hesitate to use custom settings. Example:: 80 81 xz --check=crc32 --lzma2=dict=512KiB inputfile 82 83An exception to above dictionary size limitation is when the decoder 84is used in single-call mode. Decompressing the kernel itself is an 85example of this situation. In single-call mode, the memory usage 86doesn't depend on the dictionary size, and it is perfectly fine to 87use a big dictionary: for maximum compression, the dictionary should 88be at least as big as the uncompressed data itself. 89 90Future plans 91============ 92 93Creating a limited XZ encoder may be considered if people think it is 94useful. LZMA2 is slower to compress than e.g. Deflate or LZO even at 95the fastest settings, so it isn't clear if LZMA2 encoder is wanted 96into the kernel. 97 98Support for limited random-access reading is planned for the 99decompression code. I don't know if it could have any use in the 100kernel, but I know that it would be useful in some embedded projects 101outside the Linux kernel. 102 103Conformance to the .xz file format specification 104================================================ 105 106There are a couple of corner cases where things have been simplified 107at expense of detecting errors as early as possible. These should not 108matter in practice all, since they don't cause security issues. But 109it is good to know this if testing the code e.g. with the test files 110from XZ Utils. 111 112Reporting bugs 113============== 114 115Before reporting a bug, please check that it's not fixed already 116at upstream. See <https://tukaani.org/xz/embedded.html> to get the 117latest code. 118 119Report bugs to <lasse.collin@tukaani.org> or visit #tukaani on 120Freenode and talk to Larhzu. I don't actively read LKML or other 121kernel-related mailing lists, so if there's something I should know, 122you should email to me personally or use IRC. 123 124Don't bother Igor Pavlov with questions about the XZ implementation 125in the kernel or about XZ Utils. While these two implementations 126include essential code that is directly based on Igor Pavlov's code, 127these implementations aren't maintained nor supported by him. 128