1# eMMC Storage Design
2
3Author: Adriana Kobylak < anoo! >
4
5Other contributors: Joel Stanley < shenki! >,
6                    Milton Miller
7
8Created: 2019-06-20
9
10## Problem Description
11Proposal to define an initial storage design for an eMMC device. This includes
12filesystem type, partitioning, volume management, boot options and
13initialization, etc.
14
15## Background and References
16OpenBMC currently supports raw flash such as the SPI NOR found in the systems
17based on AST2400 and AST2500, but there is no design for managed NAND.
18
19## Requirements
20- Security: Ability to enforce read-only, verification of official/signed
21  images for production.
22
23- Updatable: Ensure that the filesystem design allows for an effective and
24  simple update mechanism to be implemented.
25
26- Simplicity: Make the system easy to understand, so that it is easy to
27  develop, test, use, and recover.
28
29- Code reuse: Try to use something that already exists instead of re-inventing
30  the wheel.
31
32## Proposed Design
33- The eMMC image layout and characteristics are specified in a meta layer. This
34  allows OpenBMC to support different layouts and configurations. The tarball to
35  perform a code update is still built by image_types_phosphor, so a separate
36  IMAGE_TYPES would need to be created to support a different filesystem type.
37
38- Code update: Support two versions on flash. This allows a known good image to
39  be retained and a new image to be validated.
40
41- GPT partitioning for the eMMC User Data Area: This is chosen over dynamic
42  partitioning due to the lack of offline tools to build an LVM image (see
43  Logical Volumes in the Alternatives section below).
44
45- Initramfs: An initramfs is needed to run sgdisk on first boot to move the
46  secondary GPT to the end of the device where it belongs, since the yocto wic
47  tool does not currently support building an image of a specified size and
48  therefore the generated image may not be exactly the size of the device that
49  is flashed into.
50
51- Read-only and read-write filesystem: ext4. This is a stable and widely used
52  filesystem for eMMC.
53
54- Filesystem layout: The root filesystem is hosted in a read-only volume. The
55  /var directory is mounted in a read-write volume that persists through code
56  updates. The /home directory needs to be writable to store user data such as
57  ssh keys, so it is a bind mount to a directory in the read-write volume. A
58  bind mount is more reliable than an overlay, and has been around longer. Since
59  there are no contents delivered by the image in the /home directory, a bind
60  mount can be used. On the other hand, the /etc directory has content delivered
61  by the image, so it is an overlayfs to have the ability to restore its
62  configuration content on a factory reset.
63
64        +------------------+ +-----------------------------+
65        | Read-only volume | | Read-write volume           |
66        |------------------| |-----------------------------|
67        |                  | |                             |
68        | / (rootfs)       | | /var                        |
69        |                  | |                             |
70        | /etc  +------------->/var/etc-work/  (overlayfs) |
71        |                  | |                             |
72        | /home +------------->/var/home-work/ (bind mount)|
73        |                  | |                             |
74        |                  | |                             |
75        +------------------+ +-----------------------------+
76
77- Provisioning: OpenBMC will produce as a build artifact a flashable eMMC image
78  as it currently does for NOR chips.
79
80## Alternatives Considered
81- Store U-Boot and the Linux kernel in a separate SPI NOR flash device, since
82  SOCs such as the AST2500 do not support executing U-Boot from an eMMC. In
83  addition, having the Linux kernel on the NOR saves from requiring U-Boot
84  support for the eMMC. The U-Boot and kernel are less than 10MB in size, so a
85  fairly small chip such as a 32MB one would suffice. Therefore, in order to
86  support two firmware versions, the kernel for each version would need to be
87  stored in the NOR. A second NOR device could be added as redundancy in case
88  U-Boot or the kernel failed to run.
89
90  Format the NOR as it is currently done for a system that supports UBI: a fixed
91  MTD partition for U-Boot, one for its environment, and a UBI volume spanning
92  the remaining of the flash. Store the dual kernel volumes in the UBI partition.
93  This approach allows the re-use of the existing code update interfaces, since
94  the static approach does not currently support storing two kernel images.
95  Selection of the desired kernel image would be done with the existing U-Boot
96  environment approach.
97
98  Static MTD partitions could be created to store the kernel images, but
99  additional work would be required to introduce a new method to select the
100  desired kernel image, because the static layout does not currently have dual
101  image support.
102
103  The AST2600 supports executing U-Boot from the eMMC, so that provides the
104  flexibility of just having the eMMC chip on a system, or still have U-Boot in
105  a separate chip for recovery in cases where the eMMC goes bad.
106
107- Filesystem: f2fs (Flash-Friendly File System). The f2fs is an up-and-coming
108  filesystem, and therefore it may be seen as less mature and stable than the
109  ext4 filesystem, although it is unknown how any of the two would perform in an
110  OpenBMC environment.
111
112  A suitable alternative would be btrfs, which has checksums for both metadata
113  and data in the filesystem, and therefore provides stronger guarantees on the
114  data integrity.
115
116- All Code update artifacts combined into a single image.
117
118  This provides simple code maintenance where an image is intact or not, and
119  works or not, with no additional fragments lying around. U-Boot has one choice
120  to make - which image to load, and one piece of information to forward to the
121  kernel.
122
123  To reduce boot time by limiting IO reading unneeded sectors into memory, a
124  small FS is placed at the beginning of the partition to contain any artifacts
125  that must be accessed by U-Boot.
126
127  This file system will be selected from ext2, FAT12, and cramfs, as these are
128  all supported in both the Linux kernel and U-Boot. (If we desire the U-Boot
129  environment to be per-side, then choose one of ext2 or FAT12 (squashfs support
130  has not been merged, it was last updated in 2018 -- two years ago).
131
132- No initramfs: It may be possible to boot the rootfs by passing the UUID of the
133  logical volume to the kernel, although a [pre-init script][] will likely still
134  be needed. Therefore, having an initramfs would offer a more standard
135  implementation for initialization.
136
137- FAT MBR partitioning: FAT is a simple and well understood partition table
138  format. There is space for 4 independent partitions. Alternatively one slot
139  can be chained into extended partitions, but each partition in the chan
140  depends on the prior partition. Four partitions may be sufficient to meet the
141  initial demand for a shared (single) boot filesystem design (boot, rofs-a,
142  rofs-b, and read-write). Additional partitions would be needed for a dual boot
143  volume design.
144
145  If common space is needed for the U-Boot environment, is is redundantly stored
146  as file in partition 1. The U-Boot SPL will be located here. If this is not
147  needed, partition 1 can remain unallocated.
148
149  The two code sides are created in slots 2 and 3.
150
151  The read-write filesystem occupies partition 4.
152
153  If in the future there is demand for additional partitions, partition can be
154  moved into an extended partition in a future code update.
155
156- Device Mapper: The eMMC is divided using the device-mapper linear target,
157  which allows for the expansion of devices if necessary without having to
158  physically repartition since the device-mapper devices expose logical blocks.
159  This is achieved by changing the device-mapper configuration table entries
160  provided to the kernel to append unused physical blocks.
161
162- Logical Volumes:
163
164  - Volume management: LVM. This allows for dynamic partition/removal, similar
165    to the current UBI implementation. LVM support increases the size of the
166    kernel by ~100kB, but the increase in size is worth the ability of being
167    able to resize the partition if needed. In addition, UBI volume management
168    works in a similar way, so it would not be complex to implement LVM
169    management in the code update application.
170
171  - Partitioning: If the eMMC is used to store the boot loader, a ext4 (or vfat)
172    partition would hold the FIT image containing the kernel, initrd and device
173    tree. This volume would be mounted as /boot. This allows U-Boot to load the
174    kernel since it doesn't have support for LVM. After the boot partition,
175    assign the remaining eMMC flash as a single physical volume containing
176    logical volumes, instead of fixed-size partitions. This provides flexibility
177    for cases where the contents of a partition outgrow a fixed size. This also
178    means that other firmware images, such as BIOS and PSU, can be stored in
179    volumes in the single eMMC device.
180
181  - Initramfs: Use an initramfs, which is the default in OpenBMC, to boot the
182    rootfs from a logical volume. An initramfs allows for flexibility if
183    additional boot actions are needed, such as mounting overlays. It also
184    provides a point of departure (environment) to provision and format the eMMC
185    volume(s). To boot the rootfs, the initramfs would search for the desired
186    rootfs volume to be mounted, instead of using the U-Boot environments.
187
188  - Mount points: For firmware images such as BIOS that currently reside in
189    separate SPI NOR modules, the logical volume in the eMMC would be mounted in
190    the same paths as to prevent changes to the applications that rely on the
191    location of that data.
192
193  - Provisioning: Since the LVM userspace tools don't offer an offline
194    mode, it's not straightforward to assemble an LVM disk image from a bitbake
195    task. Therefore, have the initramfs create the LVM volume and fetch the
196    rootfs file into tmpfs from an external source to flash the volume. The
197    rootfs file can be fetched using DHCP, UART, USB key, etc. An alternative
198    option include to build the image from QEMU, this would require booting QEMU
199    as part of the build process to setup the LVM volume and create the image
200    file.
201
202## Impacts
203This design would impact the OpenBMC build process and code update
204internal implementations but should not affect the external interfaces.
205
206- openbmc/linux: Kernel changes to support the eMMC chip and its filesystem.
207- openbmc/openbmc: Changes to create an eMMC image.
208- openbmc/openpower-pnor-code-mgmt: Changes to support updating the new
209  filesystem.
210- openbmc/phosphor-bmc-code-mgmt: Changes to support updating the new
211  filesystem.
212
213## Testing
214Verify OpenBMC functionality in a system containing an eMMC. This system could
215be added to the CI pool.
216
217[pre-init script]: https://github.com/openbmc/openbmc/blob/master/meta-phosphor/recipes-phosphor/preinit-mounts/preinit-mounts/init
218