Revision tags: v6.6.25, v6.6.24, v6.6.23, v6.6.16, v6.6.15, v6.6.14, v6.6.13, v6.6.12, v6.6.11, v6.6.10, v6.6.9, v6.6.8, v6.6.7, v6.6.6, v6.6.5, v6.6.4, v6.6.3, v6.6.2, v6.5.11, v6.6.1, v6.5.10, v6.6, v6.5.9, v6.5.8, v6.5.7, v6.5.6, v6.5.5, v6.5.4, v6.5.3, v6.5.2, v6.1.51, v6.5.1, v6.1.50, v6.5, v6.1.49, v6.1.48, v6.1.46, v6.1.45, v6.1.44, v6.1.43, v6.1.42, v6.1.41, v6.1.40 |
|
#
3f339920 |
| 22-Jul-2023 |
Jingbo Xu <jefflexu@linux.alibaba.com> |
erofs: update on-disk format for xattr name filter
The xattr name bloom filter feature is going to be introduced to speed up the negative xattr lookup, e.g. system.posix_acl_[access|default] lookup
erofs: update on-disk format for xattr name filter
The xattr name bloom filter feature is going to be introduced to speed up the negative xattr lookup, e.g. system.posix_acl_[access|default] lookup when running "ls -lR" workload.
There are some commonly used extended attributes (n) and the total number of these is approximately 30.
trusted.overlay.opaque trusted.overlay.redirect trusted.overlay.origin trusted.overlay.impure trusted.overlay.nlink trusted.overlay.upper trusted.overlay.metacopy trusted.overlay.protattr user.overlay.opaque user.overlay.redirect user.overlay.origin user.overlay.impure user.overlay.nlink user.overlay.upper user.overlay.metacopy user.overlay.protattr security.evm security.ima security.selinux security.SMACK64 security.SMACK64IPIN security.SMACK64IPOUT security.SMACK64EXEC security.SMACK64TRANSMUTE security.SMACK64MMAP security.apparmor security.capability system.posix_acl_access system.posix_acl_default user.mime_type
Given the number of bits of the bloom filter (m) is 32, the optimal value for the number of the hash functions (k) is 1 (ln2 * m/n = 0.74).
The single hash function is implemented as:
xxh32(name, strlen(name), EROFS_XATTR_FILTER_SEED + index)
where `index` represents the index of corresponding predefined short name prefix, while `name` represents the name string after stripping the above predefined name prefix.
The constant magic number EROFS_XATTR_FILTER_SEED, i.e. 0x25BBE08F, is used to give a better spread when mapping these 30 extended attributes into 32-bit bloom filter as:
bit 0: security.ima bit 1: bit 2: trusted.overlay.nlink bit 3: bit 4: user.overlay.nlink bit 5: trusted.overlay.upper bit 6: user.overlay.origin bit 7: trusted.overlay.protattr bit 8: security.apparmor bit 9: user.overlay.protattr bit 10: user.overlay.opaque bit 11: security.selinux bit 12: security.SMACK64TRANSMUTE bit 13: security.SMACK64 bit 14: security.SMACK64MMAP bit 15: user.overlay.impure bit 16: security.SMACK64IPIN bit 17: trusted.overlay.redirect bit 18: trusted.overlay.origin bit 19: security.SMACK64IPOUT bit 20: trusted.overlay.opaque bit 21: system.posix_acl_default bit 22: bit 23: user.mime_type bit 24: trusted.overlay.impure bit 25: security.SMACK64EXEC bit 26: user.overlay.redirect bit 27: user.overlay.upper bit 28: security.evm bit 29: security.capability bit 30: system.posix_acl_access bit 31: trusted.overlay.metacopy, user.overlay.metacopy
h_name_filter is introduced to the on-disk per-inode xattr header to place the corresponding xattr name filter, where bit value 1 indicates non-existence for compatibility.
This feature is indicated by EROFS_FEATURE_COMPAT_XATTR_FILTER compatible feature bit.
Reserve one byte in on-disk superblock as the on-disk format for xattr name filter may change in the future. With this flag we don't need bothering these compatible bits again at that time.
Suggested-by: Alexander Larsson <alexl@redhat.com> Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com> Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com> Reviewed-by: Chao Yu <chao@kernel.org> Link: https://lore.kernel.org/r/20230722094538.11754-2-jefflexu@linux.alibaba.com Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
show more ...
|
#
ffa09b3b |
| 10-Aug-2023 |
Gao Xiang <hsiangkao@linux.alibaba.com> |
erofs: DEFLATE compression support
Add DEFLATE compression as the 3rd supported algorithm.
DEFLATE is a popular generic-purpose compression algorithm for quite long time (many advanced formats like
erofs: DEFLATE compression support
Add DEFLATE compression as the 3rd supported algorithm.
DEFLATE is a popular generic-purpose compression algorithm for quite long time (many advanced formats like gzip, zlib, zip, png are all based on that) as Apple documentation written "If you require interoperability with non-Apple devices, use COMPRESSION_ZLIB. [1]".
Due to its popularity, there are several hardware on-market DEFLATE accelerators, such as (s390) DFLTCC, (Intel) IAA/QAT, (HiSilicon) ZIP accelerator, etc. In addition, there are also several high-performence IP cores and even open-source FPGA approches available for DEFLATE. Therefore, it's useful to support DEFLATE compression in order to find a way to utilize these accelerators for asynchronous I/Os and get benefits from these later.
Besides, it's a good choice to trade off between compression ratios and performance compared to LZ4 and LZMA. The DEFLATE core format is simple as well as easy to understand, therefore the code size of its decompressor is small even for the bootloader use cases. The runtime memory consumption is quite limited too (e.g. 32K + ~7K for each zlib stream). As usual, EROFS ourperforms similar approaches too.
Alternatively, DEFLATE could still be used for some specific files since EROFS supports multiple compression algorithms in one image.
[1] https://developer.apple.com/documentation/compression/compression_algorithm Reviewed-by: Chao Yu <chao@kernel.org> Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com> Link: https://lore.kernel.org/r/20230810154859.118330-1-hsiangkao@linux.alibaba.com
show more ...
|
Revision tags: v6.1.39, v6.1.38, v6.1.37, v6.1.36, v6.4, v6.1.35, v6.1.34, v6.1.33, v6.1.32, v6.1.31, v6.1.30, v6.1.29, v6.1.28, v6.1.27, v6.1.26, v6.3, v6.1.25 |
|
#
745ed7d7 |
| 14-Apr-2023 |
Gao Xiang <hsiangkao@linux.alibaba.com> |
erofs: cleanup i_format-related stuffs
Switch EROFS_I_{VERSION,DATALAYOUT}_BITS into EROFS_I_{VERSION,DATALAYOUT}_MASK.
Also avoid erofs_bitrange() since its functionality is simple enough.
Signed
erofs: cleanup i_format-related stuffs
Switch EROFS_I_{VERSION,DATALAYOUT}_BITS into EROFS_I_{VERSION,DATALAYOUT}_MASK.
Also avoid erofs_bitrange() since its functionality is simple enough.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com> Reviewed-by: Chao Yu <chao@kernel.org> Link: https://lore.kernel.org/r/20230414083027.12307-2-hsiangkao@linux.alibaba.com
show more ...
|
Revision tags: v6.1.24 |
|
#
6a318ccd |
| 07-Apr-2023 |
Jingbo Xu <jefflexu@linux.alibaba.com> |
erofs: enable long extended attribute name prefixes
Let's enable long xattr name prefix feature. Old kernels will just ignore / skip such extended attributes. In addition, in case you don't want t
erofs: enable long extended attribute name prefixes
Let's enable long xattr name prefix feature. Old kernels will just ignore / skip such extended attributes. In addition, in case you don't want to mount such images, add another incompatible feature as an option for this.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com> Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com> Acked-by: Chao Yu <chao@kernel.org> Link: https://lore.kernel.org/r/20230407222808.19670-1-jefflexu@linux.alibaba.com [ Gao Xiang: minor commit message fix. ] Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
show more ...
|
#
b3bfcb9d |
| 07-Apr-2023 |
Jingbo Xu <jefflexu@linux.alibaba.com> |
erofs: introduce on-disk format for long xattr name prefixes
Besides the predefined xattr name prefixes, introduces long xattr name prefixes, which work similarly as the predefined name prefixes, ex
erofs: introduce on-disk format for long xattr name prefixes
Besides the predefined xattr name prefixes, introduces long xattr name prefixes, which work similarly as the predefined name prefixes, except that they are user specified.
It is especially useful for use cases together with overlayfs like Composefs model, which introduces diverse xattr values with only a few common xattr names (trusted.overlay.redirect, trusted.overlay.digest, and maybe more in the future). That makes the existing predefined prefixes ineffective in both image size and runtime performance.
When a user specified long xattr name prefix is used, only the trailing part of the xattr name apart from the long xattr name prefix will be stored in erofs_xattr_entry.e_name. e_name is empty if the xattr name matches exactly as the long xattr name prefix. All long xattr prefixes are stored in the packed or meta inode, which depends if fragments feature is enabled or not.
For each long xattr name prefix, the on-disk format is kept as the same as the unique metadata format: ALIGN({__le16 len, data}, 4), where len represents the total size of struct erofs_xattr_long_prefix, followed by data of struct erofs_xattr_long_prefix itself.
Each erofs_xattr_long_prefix keeps predefined prefixes (base_index) and the remaining prefix string without the trailing '\0'.
Two fields are introduced to the on-disk superblock, where xattr_prefix_count represents the total number of the long xattr name prefixes recorded, and xattr_prefix_start represents the start offset of recorded name prefixes in the packed/meta inode divided by 4.
When referring to a long xattr name prefix, the highest bit (bit 7) of erofs_xattr_entry.e_name_index is set, while the lower bits (bit 0-6) as a whole represents the index of the referred long name prefix among all long xattr name prefixes.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com> Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com> Acked-by: Chao Yu <chao@kernel.org> Link: https://lore.kernel.org/r/20230407141710.113882-5-jefflexu@linux.alibaba.com Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
show more ...
|
Revision tags: v6.1.23 |
|
#
1c7f49a7 |
| 31-Mar-2023 |
Gao Xiang <hsiangkao@linux.alibaba.com> |
erofs: tidy up EROFS on-disk naming
- Get rid of all "vle" (variable-length extents) expressions since they only expand overall name lengths unnecessarily; - Rename COMPRESSION_LEGACY to COMPRE
erofs: tidy up EROFS on-disk naming
- Get rid of all "vle" (variable-length extents) expressions since they only expand overall name lengths unnecessarily; - Rename COMPRESSION_LEGACY to COMPRESSED_FULL; - Move on-disk directory definitions ahead of compression; - Drop unused extended attribute definitions; - Move inode ondisk union `i_u` out as `union erofs_inode_i_u`.
No actual logical change.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com> Reviewed-by: Yue Hu <huyue2@coolpad.com> Reviewed-by: Chao Yu <chao@kernel.org> Link: https://lore.kernel.org/r/20230331063149.25611-1-hsiangkao@linux.alibaba.com
show more ...
|
Revision tags: v6.1.22, v6.1.21, v6.1.20 |
|
#
d3c4bdcc |
| 13-Mar-2023 |
Jingbo Xu <jefflexu@linux.alibaba.com> |
erofs: set block size to the on-disk block size
Set the block size to that specified in on-disk superblock.
Also remove the hard constraint of PAGE_SIZE block size for the uncompressed device backe
erofs: set block size to the on-disk block size
Set the block size to that specified in on-disk superblock.
Also remove the hard constraint of PAGE_SIZE block size for the uncompressed device backend. This constraint is temporarily remained for compressed device and fscache backend, as there is more work needed to handle the condition where the block size is not equal to PAGE_SIZE.
It is worth noting that the on-disk block size is read prior to erofs_superblock_csum_verify(), as the read block size is needed in the latter.
Besides, later we are going to make erofs refer to tar data blobs (which is 512-byte aligned) for OCI containers, where the block size is 512 bytes. In this case, the 512-byte block size may not be adequate for a directory to contain enough dirents. To fix this, we are also going to introduce directory block size independent on the block size.
Due to we have already supported block size smaller than PAGE_SIZE now, disable all these images with such separated directory block size until we supported this feature later.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com> Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com> Reviewed-by: Yue Hu <huyue2@coolpad.com> Reviewed-by: Chao Yu <chao@kernel.org> Link: https://lore.kernel.org/r/20230313135309.75269-3-jefflexu@linux.alibaba.com [ Gao Xiang: update documentation. ] Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
show more ...
|
Revision tags: v6.1.19, v6.1.18, v6.1.17, v6.1.16, v6.1.15, v6.1.14, v6.1.13, v6.2, v6.1.12, v6.1.11, v6.1.10, v6.1.9, v6.1.8, v6.1.7, v6.1.6, v6.1.5, v6.0.19, v6.0.18, v6.1.4, v6.1.3, v6.0.17, v6.1.2, v6.0.16, v6.1.1, v6.0.15, v6.0.14, v6.0.13, v6.1, v6.0.12, v6.0.11, v6.0.10, v5.15.80, v6.0.9, v5.15.79, v6.0.8, v5.15.78, v6.0.7, v5.15.77, v5.15.76, v6.0.6, v6.0.5, v5.15.75, v6.0.4, v6.0.3, v6.0.2, v5.15.74, v5.15.73, v6.0.1, v5.15.72, v6.0, v5.15.71, v5.15.70 |
|
#
5c2a6425 |
| 22-Sep-2022 |
Gao Xiang <hsiangkao@linux.alibaba.com> |
erofs: introduce partial-referenced pclusters
Due to deduplication for compressed data, pclusters can be partially referenced with their prefixes.
Together with the user-space implementation, it en
erofs: introduce partial-referenced pclusters
Due to deduplication for compressed data, pclusters can be partially referenced with their prefixes.
Together with the user-space implementation, it enables EROFS variable-length global compressed data deduplication with rolling hash.
Link: https://lore.kernel.org/r/20220923014915.4362-1-hsiangkao@linux.alibaba.com Reviewed-by: Yue Hu <huyue2@coolpad.com> Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
show more ...
|
#
b15b2e30 |
| 22-Sep-2022 |
Yue Hu <huyue2@coolpad.com> |
erofs: support on-disk compressed fragments data
Introduce on-disk compressed fragments data feature.
This approach adds a new field called `h_fragmentoff' in the per-file compression header to ind
erofs: support on-disk compressed fragments data
Introduce on-disk compressed fragments data feature.
This approach adds a new field called `h_fragmentoff' in the per-file compression header to indicate the fragment offset of each tail pcluster or the whole file in the special packed inode.
Similar to ztailpacking, it will also find and record the 'headlcn' of the tail pcluster when initializing per-inode zmap for making follow-on requests more easy.
Signed-off-by: Yue Hu <huyue2@coolpad.com> Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com> Link: https://lore.kernel.org/r/YzHKxcFTlHGgXeH9@B-P7TQMD6M-0146.local Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
show more ...
|
#
fdffc091 |
| 22-Sep-2022 |
Yue Hu <huyue2@coolpad.com> |
erofs: support interlaced uncompressed data for compressed files
Currently, uncompressed data is all handled in the shifted way, which means we have to shift the whole on-disk plain pcluster to get
erofs: support interlaced uncompressed data for compressed files
Currently, uncompressed data is all handled in the shifted way, which means we have to shift the whole on-disk plain pcluster to get the logical data. However, since we are also using in-place I/O for uncompressed data, data copy will be reduced a lot if pcluster is recorded in the interlaced way as illustrated below: _______________________________________________________________ | | | |_ tail part |_ head part _| |<- blk0 ->| .. |<- blkn-2 ->|<- blkn-1 ->|
The logical data then becomes: ________________________________________________________ |_ head part _|_ blk0 _| .. |_ blkn-2 _|_ tail part _|
In addition, non-4k plain pclusters are also survived by the interlaced way, which can be used for non-4k lclusters as well.
However, it's almost impossible to de-duplicate uncompressed data in the interlaced way, therefore shifted uncompressed data is still useful.
Signed-off-by: Yue Hu <huyue2@coolpad.com> Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com> Link: https://lore.kernel.org/r/8369112678604fdf4ef796626d59b1fdd0745a53.1663898962.git.huyue2@coolpad.com Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
show more ...
|
Revision tags: v5.15.69, v5.15.68, v5.15.67, v5.15.66, v5.15.65, v5.15.64, v5.15.63, v5.15.62, v5.15.61, v5.15.60, v5.15.59, v5.19, v5.15.58, v5.15.57, v5.15.56, v5.15.55, v5.15.54, v5.15.53, v5.15.52, v5.15.51, v5.15.50, v5.15.49, v5.15.48, v5.15.47, v5.15.46, v5.15.45, v5.15.44, v5.15.43, v5.15.42, v5.18, v5.15.41, v5.15.40, v5.15.39 |
|
#
ba73eadd |
| 12-May-2022 |
Jeffle Xu <jefflexu@linux.alibaba.com> |
erofs: scan devices from device table
When "-o device" mount option is not specified, scan the device table and instantiate the devices if there's any in the device table. In this case, the tag fiel
erofs: scan devices from device table
When "-o device" mount option is not specified, scan the device table and instantiate the devices if there's any in the device table. In this case, the tag field of each device slot uniquely specifies a device.
Signed-off-by: Jeffle Xu <jefflexu@linux.alibaba.com> Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com> Link: https://lore.kernel.org/r/20220512055601.106109-1-jefflexu@linux.alibaba.com Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
show more ...
|
Revision tags: v5.15.38 |
|
#
2833f4bb |
| 06-May-2022 |
Gao Xiang <hsiangkao@linux.alibaba.com> |
erofs: refine on-disk definition comments
Fix some outdated comments and typos, hopefully helpful.
Link: https://lore.kernel.org/r/20220506194612.117120-3-hsiangkao@linux.alibaba.com Reviewed-by: Y
erofs: refine on-disk definition comments
Fix some outdated comments and typos, hopefully helpful.
Link: https://lore.kernel.org/r/20220506194612.117120-3-hsiangkao@linux.alibaba.com Reviewed-by: Yue Hu <huyue2@coolpad.com> Reviewed-by: Chao Yu <chao@kernel.org> Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
show more ...
|
Revision tags: v5.15.37, v5.15.36, v5.15.35, v5.15.34, v5.15.33, v5.15.32, v5.15.31, v5.17, v5.15.30 |
|
#
a1108dcd |
| 17-Mar-2022 |
David Anderson <dvander@google.com> |
erofs: rename ctime to mtime
EROFS images should inherit modification time rather than change time, since users and host tooling have no easy way to control change time.
To reflect the new timestam
erofs: rename ctime to mtime
EROFS images should inherit modification time rather than change time, since users and host tooling have no easy way to control change time.
To reflect the new timestamp meaning, i_ctime and i_ctime_nsec are renamed to i_mtime and i_mtime_nsec.
Link: https://lore.kernel.org/r/20220311041829.3109511-1-dvander@google.com # v1 Signed-off-by: David Anderson <dvander@google.com> [ Gao Xiang: update document as well. ] Reviewed-by: Chao Yu <chao@kernel.org> Link: https://lore.kernel.org/r/20220317114959.106787-1-hsiangkao@linux.alibaba.com # v2 Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
show more ...
|
Revision tags: v5.15.29, v5.15.28, v5.15.27, v5.15.26, v5.15.25, v5.15.24, v5.15.23, v5.15.22, v5.15.21, v5.15.20, v5.15.19, v5.15.18, v5.15.17, v5.4.173, v5.15.16, v5.15.15, v5.16 |
|
#
ab92184f |
| 27-Dec-2021 |
Yue Hu <huyue2@yulong.com> |
erofs: add on-disk compressed tail-packing inline support
Introduces erofs compressed tail-packing inline support.
This approach adds a new field called `h_idata_size' in the per-file compression h
erofs: add on-disk compressed tail-packing inline support
Introduces erofs compressed tail-packing inline support.
This approach adds a new field called `h_idata_size' in the per-file compression header to indicate the encoded size of each tail-packing pcluster.
At runtime, it will find the start logical offset of the tail pcluster when initializing per-inode zmap and record such extent (headlcn, idataoff) information to the in-memory inode. Therefore, follow-on requests can directly recognize if one pcluster is a tail-packing inline pcluster or not.
Link: https://lore.kernel.org/r/20211228054604.114518-6-hsiangkao@linux.alibaba.com Reviewed-by: Chao Yu <chao@kernel.org> Signed-off-by: Yue Hu <huyue2@yulong.com> Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
show more ...
|
Revision tags: v5.15.10, v5.15.9, v5.15.8, v5.15.7 |
|
#
7acc3d1a |
| 06-Dec-2021 |
Gao Xiang <hsiangkao@linux.alibaba.com> |
erofs: Replace zero-length array with flexible-array member
There is a regular need in the kernel to provide a way to declare having a dynamically sized set of trailing elements in a structure.
Ker
erofs: Replace zero-length array with flexible-array member
There is a regular need in the kernel to provide a way to declare having a dynamically sized set of trailing elements in a structure.
Kernel code should always use `flexible array members' [1] for these cases. The older style of one-element or zero-length arrays should no longer be used [2].
[1] https://en.wikipedia.org/wiki/Flexible_array_member [2] https://www.kernel.org/doc/html/v5.15/process/deprecated.html#zero-length-and-one-element-arrays
Link: https://lore.kernel.org/r/20211206121702.221331-1-hsiangkao@linux.alibaba.com Reported-by: Abaci Robot <abaci@linux.alibaba.com> Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
show more ...
|
Revision tags: v5.15.6, v5.15.5, v5.15.4, v5.15.3 |
|
#
7e508f2c |
| 12-Nov-2021 |
Huang Jianan <huangjianan@oppo.com> |
erofs: rename lz4_0pading to zero_padding
Renaming lz4_0padding to zero_padding globally since LZMA and later algorithms also need that.
Link: https://lore.kernel.org/r/20211112160935.19394-1-jnhua
erofs: rename lz4_0pading to zero_padding
Renaming lz4_0padding to zero_padding globally since LZMA and later algorithms also need that.
Link: https://lore.kernel.org/r/20211112160935.19394-1-jnhuang95@gmail.com Reviewed-by: Chao Yu <chao@kernel.org> Signed-off-by: Huang Jianan <huangjianan@oppo.com> Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
show more ...
|
Revision tags: v5.15.2, v5.15.1, v5.15, v5.14.14, v5.14.13, v5.14.12 |
|
#
622ceadd |
| 10-Oct-2021 |
Gao Xiang <hsiangkao@linux.alibaba.com> |
erofs: lzma compression support
Add MicroLZMA support in order to maximize compression ratios for specific scenarios. For example, it's useful for low-end embedded boards and as a secondary algorith
erofs: lzma compression support
Add MicroLZMA support in order to maximize compression ratios for specific scenarios. For example, it's useful for low-end embedded boards and as a secondary algorithm in a file for specific access patterns.
MicroLZMA is a new container format for raw LZMA1, which was created by Lasse Collin aiming to minimize old LZMA headers and get rid of unnecessary EOPM (end of payload marker) as well as to enable fixed-sized output compression, especially for 4KiB pclusters.
Similar to LZ4, inplace I/O approach is used to minimize runtime memory footprint when dealing with I/O. Overlapped decompression is handled with 1) bounced buffer for data under processing or 2) extra short-lived pages from the on-stack pagepool which will be shared in the same read request (128KiB for example).
Link: https://lore.kernel.org/r/20211010213145.17462-8-xiang@kernel.org Acked-by: Chao Yu <chao@kernel.org> Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
show more ...
|
#
72bb5262 |
| 17-Oct-2021 |
Gao Xiang <hsiangkao@linux.alibaba.com> |
erofs: introduce the secondary compression head
Previously, for each HEAD lcluster, it can be either HEAD or PLAIN lcluster to indicate whether the whole pcluster is compressed or not.
In this patc
erofs: introduce the secondary compression head
Previously, for each HEAD lcluster, it can be either HEAD or PLAIN lcluster to indicate whether the whole pcluster is compressed or not.
In this patch, a new HEAD2 head type is introduced to specify another compression algorithm other than the primary algorithm for each compressed file, which can be used for upcoming LZMA compression and LZ4 range dictionary compression for various data patterns.
It has been stayed in the EROFS roadmap for years. Complete it now!
Link: https://lore.kernel.org/r/20211017165721.2442-1-xiang@kernel.org Reviewed-by: Yue Hu <huyue2@yulong.com> Reviewed-by: Chao Yu <chao@kernel.org> Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
show more ...
|
#
dfeab2e9 |
| 14-Oct-2021 |
Gao Xiang <hsiangkao@linux.alibaba.com> |
erofs: add multiple device support
In order to support multi-layer container images, add multiple device feature to EROFS. Two ways are available to use for now:
- Devices can be mapped into 32-bi
erofs: add multiple device support
In order to support multi-layer container images, add multiple device feature to EROFS. Two ways are available to use for now:
- Devices can be mapped into 32-bit global block address space; - Device ID can be specified with the chunk indexes format.
Note that it assumes no extent would cross device boundary and mkfs should take care of it seriously.
In the future, a dedicated device manager could be introduced then thus extra devices can be automatically scanned by UUID as well.
Link: https://lore.kernel.org/r/20211014081010.43485-1-hsiangkao@linux.alibaba.com Reviewed-by: Chao Yu <chao@kernel.org> Reviewed-by: Liu Bo <bo.liu@linux.alibaba.com> Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
show more ...
|
Revision tags: v5.14.11, v5.14.10, v5.14.9, v5.14.8, v5.14.7, v5.14.6, v5.10.67, v5.10.66, v5.14.5, v5.14.4, v5.10.65, v5.14.3, v5.10.64, v5.14.2, v5.10.63, v5.14.1, v5.10.62, v5.14, v5.10.61 |
|
#
2a9dc7a8 |
| 20-Aug-2021 |
Gao Xiang <hsiangkao@linux.alibaba.com> |
erofs: introduce chunk-based file on-disk format
Currently, uncompressed data except for tail-packing inline is consecutive on disk.
In order to support chunk-based data deduplication, add a new co
erofs: introduce chunk-based file on-disk format
Currently, uncompressed data except for tail-packing inline is consecutive on disk.
In order to support chunk-based data deduplication, add a new corresponding inode data layout.
In the future, the data source of chunks can be either (un)compressed.
Link: https://lore.kernel.org/r/20210820100019.208490-1-hsiangkao@linux.alibaba.com Reviewed-by: Liu Bo <bo.liu@linux.alibaba.com> Reviewed-by: Chao Yu <chao@kernel.org> Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
show more ...
|
Revision tags: v5.10.60, v5.10.53, v5.10.52, v5.10.51, v5.10.50, v5.10.49, v5.13, v5.10.46, v5.10.43, v5.10.42 |
|
#
c5fcb511 |
| 02-Jun-2021 |
Gao Xiang <hsiangkao@linux.alibaba.com> |
erofs: clean up file headers & footers
- Remove my outdated misleading email address;
- Get rid of all unnecessary trailing newline by accident.
Link: https://lore.kernel.org/r/20210602160634.10
erofs: clean up file headers & footers
- Remove my outdated misleading email address;
- Get rid of all unnecessary trailing newline by accident.
Link: https://lore.kernel.org/r/20210602160634.10757-1-xiang@kernel.org Reviewed-by: Chao Yu <yuchao0@huawei.com> Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
show more ...
|
Revision tags: v5.10.41, v5.10.40, v5.10.39, v5.4.119, v5.10.36, v5.10.35, v5.10.34, v5.4.116, v5.10.33, v5.12, v5.10.32, v5.10.31, v5.10.30 |
|
#
8e6c8fa9 |
| 06-Apr-2021 |
Gao Xiang <hsiangkao@redhat.com> |
erofs: enable big pcluster feature
Enable COMPR_CFGS and BIG_PCLUSTER since the implementations are all settled properly.
Link: https://lore.kernel.org/r/20210407043927.10623-11-xiang@kernel.org Ac
erofs: enable big pcluster feature
Enable COMPR_CFGS and BIG_PCLUSTER since the implementations are all settled properly.
Link: https://lore.kernel.org/r/20210407043927.10623-11-xiang@kernel.org Acked-by: Chao Yu <yuchao0@huawei.com> Signed-off-by: Gao Xiang <hsiangkao@redhat.com>
show more ...
|
#
5404c330 |
| 06-Apr-2021 |
Gao Xiang <hsiangkao@redhat.com> |
erofs: add big physical cluster definition
Big pcluster indicates the size of compressed data for each physical pcluster is no longer fixed as block size, but could be more than 1 block (more accura
erofs: add big physical cluster definition
Big pcluster indicates the size of compressed data for each physical pcluster is no longer fixed as block size, but could be more than 1 block (more accurately, 1 logical pcluster)
When big pcluster feature is enabled for head0/1, delta0 of the 1st non-head lcluster index will keep block count of this pcluster in lcluster size instead of 1. Or, the compressed size of pcluster should be 1 lcluster if pcluster has no non-head lcluster index.
Also note that BIG_PCLUSTER feature reuses COMPR_CFGS feature since it depends on COMPR_CFGS and will be released together.
Link: https://lore.kernel.org/r/20210407043927.10623-6-xiang@kernel.org Acked-by: Chao Yu <yuchao0@huawei.com> Signed-off-by: Gao Xiang <hsiangkao@redhat.com>
show more ...
|
#
9f6cc76e |
| 06-Apr-2021 |
Gao Xiang <hsiangkao@redhat.com> |
erofs: introduce physical cluster slab pools
Since multiple pcluster sizes could be used at once, the number of compressed pages will become a variable factor. It's necessary to introduce slab pools
erofs: introduce physical cluster slab pools
Since multiple pcluster sizes could be used at once, the number of compressed pages will become a variable factor. It's necessary to introduce slab pools rather than a single slab cache now.
This limits the pclustersize to 1M (Z_EROFS_PCLUSTER_MAX_SIZE), and get rid of the obsolete EROFS_FS_CLUSTER_PAGE_LIMIT, which has no use now.
Link: https://lore.kernel.org/r/20210407043927.10623-4-xiang@kernel.org Acked-by: Chao Yu <yuchao0@huawei.com> Signed-off-by: Gao Xiang <hsiangkao@redhat.com>
show more ...
|
#
54e0b6c8 |
| 06-Apr-2021 |
Gao Xiang <hsiangkao@redhat.com> |
erofs: reserve physical_clusterbits[]
Formal big pcluster design is actually more powerful / flexable than the previous thought whose pclustersize was fixed as power-of-2 blocks, which was obviously
erofs: reserve physical_clusterbits[]
Formal big pcluster design is actually more powerful / flexable than the previous thought whose pclustersize was fixed as power-of-2 blocks, which was obviously inefficient and space-wasting. Instead, pclustersize can now be set independently for each pcluster, so various pcluster sizes can also be used together in one file if mkfs wants (for example, according to data type and/or compression ratio).
Let's get rid of previous physical_clusterbits[] setting (also notice that corresponding on-disk fields are still 0 for now). Therefore, head1/2 can be used for at most 2 different algorithms in one file and again pclustersize is now independent of these.
Link: https://lore.kernel.org/r/20210407043927.10623-2-xiang@kernel.org Acked-by: Chao Yu <yuchao0@huawei.com> Signed-off-by: Gao Xiang <hsiangkao@redhat.com>
show more ...
|