1# eMMC Storage Design
2
3Author: Adriana Kobylak < anoo! >
4
5Primary assignee: Adriana Kobylak
6
7Other contributors: Joel Stanley < shenki! >,
8                    Milton Miller
9
10Created: 2019-06-20
11
12## Problem Description
13Proposal to define an initial storage design for an eMMC device. This includes
14filesystem type, partitioning, volume management, boot options and
15initialization, etc.
16
17## Background and References
18OpenBMC currently supports raw flash such as the SPI NOR found in the systems
19based on AST2400 and AST2500, but there is no design for managed NAND.
20
21## Requirements
22- Security: Ability to enforce read-only, verification of official/signed
23  images for production.
24
25- Updatable: Ensure that the filesystem design allows for an effective and
26  simple update mechanism to be implemented.
27
28- Simplicity: Make the system easy to understand, so that it is easy to
29  develop, test, use, and recover.
30
31- Code reuse: Try to use something that already exists instead of re-inventing
32  the wheel.
33
34## Proposed Design
35- The eMMC image layout and characteristics are specified in a meta layer. This
36  allows OpenBMC to support different layouts and configurations. The tarball to
37  perform a code update is still built by image_types_phosphor, so a separate
38  IMAGE_TYPES would need to be created to support a different filesystem type.
39
40- Code update: Support two versions on flash. This allows a known good image to
41  be retained and a new image to be validated.
42
43- GPT partitioning for the eMMC User Data Area: This is chosen over dynamic
44  partitioning due to the lack of offline tools to build an LVM image (see
45  Logical Volumes in the Alternatives section below).
46
47- Initramfs: An initramfs is needed to run sgdisk on first boot to move the
48  secondary GPT to the end of the device where it belongs, since the yocto wic
49  tool does not currently support building an image of a specified size and
50  therefore the generated image may not be exactly the size of the device that
51  is flashed into.
52
53- Read-only and read-write filesystem: ext4. This is a stable and widely used
54  filesystem for eMMC.
55
56- Filesystem layout: The root filesystem is hosted in a read-only volume. The
57  /var directory is mounted in a read-write volume that persists through code
58  updates. The /home directory needs to be writable to store user data such as
59  ssh keys, so it is a bind mount to a directory in the read-write volume. A
60  bind mount is more reliable than an overlay, and has been around longer. Since
61  there are no contents delivered by the image in the /home directory, a bind
62  mount can be used. On the other hand, the /etc directory has content delivered
63  by the image, so it is an overlayfs to have the ability to restore its
64  configuration content on a factory reset.
65
66        +------------------+ +-----------------------------+
67        | Read-only volume | | Read-write volume           |
68        |------------------| |-----------------------------|
69        |                  | |                             |
70        | / (rootfs)       | | /var                        |
71        |                  | |                             |
72        | /etc  +------------->/var/etc-work/  (overlayfs) |
73        |                  | |                             |
74        | /home +------------->/var/home-work/ (bind mount)|
75        |                  | |                             |
76        |                  | |                             |
77        +------------------+ +-----------------------------+
78
79- Provisioning: OpenBMC will produce as a build artifact a flashable eMMC image
80  as it currently does for NOR chips.
81
82## Alternatives Considered
83- Store U-Boot and the Linux kernel in a separate SPI NOR flash device, since
84  SOCs such as the AST2500 do not support executing U-Boot from an eMMC. In
85  addition, having the Linux kernel on the NOR saves from requiring U-Boot
86  support for the eMMC. The U-Boot and kernel are less than 10MB in size, so a
87  fairly small chip such as a 32MB one would suffice. Therefore, in order to
88  support two firmware versions, the kernel for each version would need to be
89  stored in the NOR. A second NOR device could be added as redundancy in case
90  U-Boot or the kernel failed to run.
91
92  Format the NOR as it is currently done for a system that supports UBI: a fixed
93  MTD partition for U-Boot, one for its environment, and a UBI volume spanning
94  the remaining of the flash. Store the dual kernel volumes in the UBI partition.
95  This approach allows the re-use of the existing code update interfaces, since
96  the static approach does not currently support storing two kernel images.
97  Selection of the desired kernel image would be done with the existing U-Boot
98  environment approach.
99
100  Static MTD partitions could be created to store the kernel images, but
101  additional work would be required to introduce a new method to select the
102  desired kernel image, because the static layout does not currently have dual
103  image support.
104
105  The AST2600 supports executing U-Boot from the eMMC, so that provides the
106  flexibility of just having the eMMC chip on a system, or still have U-Boot in
107  a separate chip for recovery in cases where the eMMC goes bad.
108
109- Filesystem: f2fs (Flash-Friendly File System). The f2fs is an up-and-coming
110  filesystem, and therefore it may be seen as less mature and stable than the
111  ext4 filesystem, although it is unknown how any of the two would perform in an
112  OpenBMC environment.
113
114  A suitable alternative would be btrfs, which has checksums for both metadata
115  and data in the filesystem, and therefore provides stronger guarantees on the
116  data integrity.
117
118- All Code update artifacts combined into a single image.
119
120  This provides simple code maintenance where an image is intact or not, and
121  works or not, with no additional fragments lying around. U-Boot has one choice
122  to make - which image to load, and one piece of information to forward to the
123  kernel.
124
125  To reduce boot time by limiting IO reading unneeded sectors into memory, a
126  small FS is placed at the beginning of the partition to contain any artifacts
127  that must be accessed by U-Boot.
128
129  This file system will be selected from ext2, FAT12, and cramfs, as these are
130  all supported in both the Linux kernel and U-Boot. (If we desire the U-Boot
131  environment to be per-side, then choose one of ext2 or FAT12 (squashfs support
132  has not been merged, it was last updated in 2018 -- two years ago).
133
134- No initramfs: It may be possible to boot the rootfs by passing the UUID of the
135  logical volume to the kernel, although a [pre-init script][] will likely still
136  be needed. Therefore, having an initramfs would offer a more standard
137  implementation for initialization.
138
139- FAT MBR partitioning: FAT is a simple and well understood partition table
140  format. There is space for 4 independent partitions. Alternatively one slot
141  can be chained into extended partitions, but each partition in the chan
142  depends on the prior partition. Four partitions may be sufficient to meet the
143  initial demand for a shared (single) boot filesystem design (boot, rofs-a,
144  rofs-b, and read-write). Additional partitions would be needed for a dual boot
145  volume design.
146
147  If common space is needed for the U-Boot environment, is is redundantly stored
148  as file in partition 1. The U-Boot SPL will be located here. If this is not
149  needed, partition 1 can remain unallocated.
150
151  The two code sides are created in slots 2 and 3.
152
153  The read-write filesystem occupies partition 4.
154
155  If in the future there is demand for additional partitions, partition can be
156  moved into an extended partition in a future code update.
157
158- Device Mapper: The eMMC is divided using the device-mapper linear target,
159  which allows for the expansion of devices if necessary without having to
160  physically repartition since the device-mapper devices expose logical blocks.
161  This is achieved by changing the device-mapper configuration table entries
162  provided to the kernel to append unused physical blocks.
163
164- Logical Volumes:
165
166  - Volume management: LVM. This allows for dynamic partition/removal, similar
167    to the current UBI implementation. LVM support increases the size of the
168    kernel by ~100kB, but the increase in size is worth the ability of being
169    able to resize the partition if needed. In addition, UBI volume management
170    works in a similar way, so it would not be complex to implement LVM
171    management in the code update application.
172
173  - Partitioning: If the eMMC is used to store the boot loader, a ext4 (or vfat)
174    partition would hold the FIT image containing the kernel, initrd and device
175    tree. This volume would be mounted as /boot. This allows U-Boot to load the
176    kernel since it doesn't have support for LVM. After the boot partition,
177    assign the remaining eMMC flash as a single physical volume containing
178    logical volumes, instead of fixed-size partitions. This provides flexibility
179    for cases where the contents of a partition outgrow a fixed size. This also
180    means that other firmware images, such as BIOS and PSU, can be stored in
181    volumes in the single eMMC device.
182
183  - Initramfs: Use an initramfs, which is the default in OpenBMC, to boot the
184    rootfs from a logical volume. An initramfs allows for flexibility if
185    additional boot actions are needed, such as mounting overlays. It also
186    provides a point of departure (environment) to provision and format the eMMC
187    volume(s). To boot the rootfs, the initramfs would search for the desired
188    rootfs volume to be mounted, instead of using the U-Boot environments.
189
190  - Mount points: For firmware images such as BIOS that currently reside in
191    separate SPI NOR modules, the logical volume in the eMMC would be mounted in
192    the same paths as to prevent changes to the applications that rely on the
193    location of that data.
194
195  - Provisioning: Since the LVM userspace tools don't offer an offline
196    mode, it's not straightforward to assemble an LVM disk image from a bitbake
197    task. Therefore, have the initramfs create the LVM volume and fetch the
198    rootfs file into tmpfs from an external source to flash the volume. The
199    rootfs file can be fetched using DHCP, UART, USB key, etc. An alternative
200    option include to build the image from QEMU, this would require booting QEMU
201    as part of the build process to setup the LVM volume and create the image
202    file.
203
204## Impacts
205This design would impact the OpenBMC build process and code update
206internal implementations but should not affect the external interfaces.
207
208- openbmc/linux: Kernel changes to support the eMMC chip and its filesystem.
209- openbmc/openbmc: Changes to create an eMMC image.
210- openbmc/openpower-pnor-code-mgmt: Changes to support updating the new
211  filesystem.
212- openbmc/phosphor-bmc-code-mgmt: Changes to support updating the new
213  filesystem.
214
215## Testing
216Verify OpenBMC functionality in a system containing an eMMC. This system could
217be added to the CI pool.
218
219[pre-init script]: https://github.com/openbmc/openbmc/blob/master/meta-phosphor/recipes-phosphor/preinit-mounts/preinit-mounts/init
220