1====================== 2Firmware-Assisted Dump 3====================== 4 5July 2011 6 7The goal of firmware-assisted dump is to enable the dump of 8a crashed system, and to do so from a fully-reset system, and 9to minimize the total elapsed time until the system is back 10in production use. 11 12- Firmware assisted dump (fadump) infrastructure is intended to replace 13 the existing phyp assisted dump. 14- Fadump uses the same firmware interfaces and memory reservation model 15 as phyp assisted dump. 16- Unlike phyp dump, fadump exports the memory dump through /proc/vmcore 17 in the ELF format in the same way as kdump. This helps us reuse the 18 kdump infrastructure for dump capture and filtering. 19- Unlike phyp dump, userspace tool does not need to refer any sysfs 20 interface while reading /proc/vmcore. 21- Unlike phyp dump, fadump allows user to release all the memory reserved 22 for dump, with a single operation of echo 1 > /sys/kernel/fadump_release_mem. 23- Once enabled through kernel boot parameter, fadump can be 24 started/stopped through /sys/kernel/fadump_registered interface (see 25 sysfs files section below) and can be easily integrated with kdump 26 service start/stop init scripts. 27 28Comparing with kdump or other strategies, firmware-assisted 29dump offers several strong, practical advantages: 30 31- Unlike kdump, the system has been reset, and loaded 32 with a fresh copy of the kernel. In particular, 33 PCI and I/O devices have been reinitialized and are 34 in a clean, consistent state. 35- Once the dump is copied out, the memory that held the dump 36 is immediately available to the running kernel. And therefore, 37 unlike kdump, fadump doesn't need a 2nd reboot to get back 38 the system to the production configuration. 39 40The above can only be accomplished by coordination with, 41and assistance from the Power firmware. The procedure is 42as follows: 43 44- The first kernel registers the sections of memory with the 45 Power firmware for dump preservation during OS initialization. 46 These registered sections of memory are reserved by the first 47 kernel during early boot. 48 49- When a system crashes, the Power firmware will save 50 the low memory (boot memory of size larger of 5% of system RAM 51 or 256MB) of RAM to the previous registered region. It will 52 also save system registers, and hardware PTE's. 53 54 NOTE: 55 The term 'boot memory' means size of the low memory chunk 56 that is required for a kernel to boot successfully when 57 booted with restricted memory. By default, the boot memory 58 size will be the larger of 5% of system RAM or 256MB. 59 Alternatively, user can also specify boot memory size 60 through boot parameter 'crashkernel=' which will override 61 the default calculated size. Use this option if default 62 boot memory size is not sufficient for second kernel to 63 boot successfully. For syntax of crashkernel= parameter, 64 refer to Documentation/admin-guide/kdump/kdump.rst. If any offset is 65 provided in crashkernel= parameter, it will be ignored 66 as fadump uses a predefined offset to reserve memory 67 for boot memory dump preservation in case of a crash. 68 69- After the low memory (boot memory) area has been saved, the 70 firmware will reset PCI and other hardware state. It will 71 *not* clear the RAM. It will then launch the bootloader, as 72 normal. 73 74- The freshly booted kernel will notice that there is a new 75 node (ibm,dump-kernel) in the device tree, indicating that 76 there is crash data available from a previous boot. During 77 the early boot OS will reserve rest of the memory above 78 boot memory size effectively booting with restricted memory 79 size. This will make sure that the second kernel will not 80 touch any of the dump memory area. 81 82- User-space tools will read /proc/vmcore to obtain the contents 83 of memory, which holds the previous crashed kernel dump in ELF 84 format. The userspace tools may copy this info to disk, or 85 network, nas, san, iscsi, etc. as desired. 86 87- Once the userspace tool is done saving dump, it will echo 88 '1' to /sys/kernel/fadump_release_mem to release the reserved 89 memory back to general use, except the memory required for 90 next firmware-assisted dump registration. 91 92 e.g.:: 93 94 # echo 1 > /sys/kernel/fadump_release_mem 95 96Please note that the firmware-assisted dump feature 97is only available on Power6 and above systems with recent 98firmware versions. 99 100Implementation details: 101----------------------- 102 103During boot, a check is made to see if firmware supports 104this feature on that particular machine. If it does, then 105we check to see if an active dump is waiting for us. If yes 106then everything but boot memory size of RAM is reserved during 107early boot (See Fig. 2). This area is released once we finish 108collecting the dump from user land scripts (e.g. kdump scripts) 109that are run. If there is dump data, then the 110/sys/kernel/fadump_release_mem file is created, and the reserved 111memory is held. 112 113If there is no waiting dump data, then only the memory required 114to hold CPU state, HPTE region, boot memory dump and elfcore 115header, is usually reserved at an offset greater than boot memory 116size (see Fig. 1). This area is *not* released: this region will 117be kept permanently reserved, so that it can act as a receptacle 118for a copy of the boot memory content in addition to CPU state 119and HPTE region, in the case a crash does occur. Since this reserved 120memory area is used only after the system crash, there is no point in 121blocking this significant chunk of memory from production kernel. 122Hence, the implementation uses the Linux kernel's Contiguous Memory 123Allocator (CMA) for memory reservation if CMA is configured for kernel. 124With CMA reservation this memory will be available for applications to 125use it, while kernel is prevented from using it. With this fadump will 126still be able to capture all of the kernel memory and most of the user 127space memory except the user pages that were present in CMA region:: 128 129 o Memory Reservation during first kernel 130 131 Low memory Top of memory 132 0 boot memory size | 133 | | |<--Reserved dump area -->| | 134 V V | Permanent Reservation | V 135 +-----------+----------/ /---+---+----+-----------+----+------+ 136 | | |CPU|HPTE| DUMP |ELF | | 137 +-----------+----------/ /---+---+----+-----------+----+------+ 138 | ^ 139 | | 140 \ / 141 ------------------------------------------- 142 Boot memory content gets transferred to 143 reserved area by firmware at the time of 144 crash 145 Fig. 1 146 147 o Memory Reservation during second kernel after crash 148 149 Low memory Top of memory 150 0 boot memory size | 151 | |<------------- Reserved dump area ----------- -->| 152 V V V 153 +-----------+----------/ /---+---+----+-----------+----+------+ 154 | | |CPU|HPTE| DUMP |ELF | | 155 +-----------+----------/ /---+---+----+-----------+----+------+ 156 | | 157 V V 158 Used by second /proc/vmcore 159 kernel to boot 160 Fig. 2 161 162Currently the dump will be copied from /proc/vmcore to a 163a new file upon user intervention. The dump data available through 164/proc/vmcore will be in ELF format. Hence the existing kdump 165infrastructure (kdump scripts) to save the dump works fine with 166minor modifications. 167 168The tools to examine the dump will be same as the ones 169used for kdump. 170 171How to enable firmware-assisted dump (fadump): 172---------------------------------------------- 173 1741. Set config option CONFIG_FA_DUMP=y and build kernel. 1752. Boot into linux kernel with 'fadump=on' kernel cmdline option. 176 By default, fadump reserved memory will be initialized as CMA area. 177 Alternatively, user can boot linux kernel with 'fadump=nocma' to 178 prevent fadump to use CMA. 1793. Optionally, user can also set 'crashkernel=' kernel cmdline 180 to specify size of the memory to reserve for boot memory dump 181 preservation. 182 183NOTE: 184 1. 'fadump_reserve_mem=' parameter has been deprecated. Instead 185 use 'crashkernel=' to specify size of the memory to reserve 186 for boot memory dump preservation. 187 2. If firmware-assisted dump fails to reserve memory then it 188 will fallback to existing kdump mechanism if 'crashkernel=' 189 option is set at kernel cmdline. 190 3. if user wants to capture all of user space memory and ok with 191 reserved memory not available to production system, then 192 'fadump=nocma' kernel parameter can be used to fallback to 193 old behaviour. 194 195Sysfs/debugfs files: 196-------------------- 197 198Firmware-assisted dump feature uses sysfs file system to hold 199the control files and debugfs file to display memory reserved region. 200 201Here is the list of files under kernel sysfs: 202 203 /sys/kernel/fadump_enabled 204 This is used to display the fadump status. 205 206 - 0 = fadump is disabled 207 - 1 = fadump is enabled 208 209 This interface can be used by kdump init scripts to identify if 210 fadump is enabled in the kernel and act accordingly. 211 212 /sys/kernel/fadump_registered 213 This is used to display the fadump registration status as well 214 as to control (start/stop) the fadump registration. 215 216 - 0 = fadump is not registered. 217 - 1 = fadump is registered and ready to handle system crash. 218 219 To register fadump echo 1 > /sys/kernel/fadump_registered and 220 echo 0 > /sys/kernel/fadump_registered for un-register and stop the 221 fadump. Once the fadump is un-registered, the system crash will not 222 be handled and vmcore will not be captured. This interface can be 223 easily integrated with kdump service start/stop. 224 225 /sys/kernel/fadump_release_mem 226 This file is available only when fadump is active during 227 second kernel. This is used to release the reserved memory 228 region that are held for saving crash dump. To release the 229 reserved memory echo 1 to it:: 230 231 echo 1 > /sys/kernel/fadump_release_mem 232 233 After echo 1, the content of the /sys/kernel/debug/powerpc/fadump_region 234 file will change to reflect the new memory reservations. 235 236 The existing userspace tools (kdump infrastructure) can be easily 237 enhanced to use this interface to release the memory reserved for 238 dump and continue without 2nd reboot. 239 240Here is the list of files under powerpc debugfs: 241(Assuming debugfs is mounted on /sys/kernel/debug directory.) 242 243 /sys/kernel/debug/powerpc/fadump_region 244 This file shows the reserved memory regions if fadump is 245 enabled otherwise this file is empty. The output format 246 is:: 247 248 <region>: [<start>-<end>] <reserved-size> bytes, Dumped: <dump-size> 249 250 e.g. 251 Contents when fadump is registered during first kernel:: 252 253 # cat /sys/kernel/debug/powerpc/fadump_region 254 CPU : [0x0000006ffb0000-0x0000006fff001f] 0x40020 bytes, Dumped: 0x0 255 HPTE: [0x0000006fff0020-0x0000006fff101f] 0x1000 bytes, Dumped: 0x0 256 DUMP: [0x0000006fff1020-0x0000007fff101f] 0x10000000 bytes, Dumped: 0x0 257 258 Contents when fadump is active during second kernel:: 259 260 # cat /sys/kernel/debug/powerpc/fadump_region 261 CPU : [0x0000006ffb0000-0x0000006fff001f] 0x40020 bytes, Dumped: 0x40020 262 HPTE: [0x0000006fff0020-0x0000006fff101f] 0x1000 bytes, Dumped: 0x1000 263 DUMP: [0x0000006fff1020-0x0000007fff101f] 0x10000000 bytes, Dumped: 0x10000000 264 : [0x00000010000000-0x0000006ffaffff] 0x5ffb0000 bytes, Dumped: 0x5ffb0000 265 266NOTE: 267 Please refer to Documentation/filesystems/debugfs.txt on 268 how to mount the debugfs filesystem. 269 270 271TODO: 272----- 273 - Need to come up with the better approach to find out more 274 accurate boot memory size that is required for a kernel to 275 boot successfully when booted with restricted memory. 276 - The fadump implementation introduces a fadump crash info structure 277 in the scratch area before the ELF core header. The idea of introducing 278 this structure is to pass some important crash info data to the second 279 kernel which will help second kernel to populate ELF core header with 280 correct data before it gets exported through /proc/vmcore. The current 281 design implementation does not address a possibility of introducing 282 additional fields (in future) to this structure without affecting 283 compatibility. Need to come up with the better approach to address this. 284 285 The possible approaches are: 286 287 1. Introduce version field for version tracking, bump up the version 288 whenever a new field is added to the structure in future. The version 289 field can be used to find out what fields are valid for the current 290 version of the structure. 291 2. Reserve the area of predefined size (say PAGE_SIZE) for this 292 structure and have unused area as reserved (initialized to zero) 293 for future field additions. 294 295 The advantage of approach 1 over 2 is we don't need to reserve extra space. 296 297Author: Mahesh Salgaonkar <mahesh@linux.vnet.ibm.com> 298 299This document is based on the original documentation written for phyp 300 301assisted dump by Linas Vepstas and Manish Ahuja. 302