1*e310dd91SPatrick Williams# `ipkdbg`: Generate gdb environments from opkg package archives 2c7a446e2SAndrew Jeffery 3c7a446e2SAndrew Jeffery`ipkdbg` automates the process of generating `gdb` environments for interactive 4c7a446e2SAndrew Jefferydebugging and coredump analysis of split-debug binaries by exploiting bitbake's 5c7a446e2SAndrew Jefferyruntime package management support outside the context of the BMC. 6c7a446e2SAndrew Jeffery 7*e310dd91SPatrick Williams## Use of `ipkdbg` 8c7a446e2SAndrew Jeffery 9c7a446e2SAndrew JefferyAssuming the presence of the [appropriate integration](#theory-of-integration), 10c7a446e2SAndrew Jefferyusing `ipkdbg` requires minimal fuss. 11c7a446e2SAndrew Jeffery 12*e310dd91SPatrick Williams```text 13c7a446e2SAndrew JefferySYNOPSIS 14c7a446e2SAndrew Jeffery ipkdbg [-q] RELEASE FILE CORE [PACKAGE...] 15c7a446e2SAndrew Jeffery``` 16c7a446e2SAndrew Jeffery 17c7a446e2SAndrew JefferyIt handles the following use-cases: 18c7a446e2SAndrew Jeffery 19*e310dd91SPatrick Williams### Core Dump Analysis 20c7a446e2SAndrew Jeffery 21c7a446e2SAndrew JefferyAfter extracting a core file from the BMC an interactive `gdb` session can be 22c7a446e2SAndrew Jefferyset up against it with: 23c7a446e2SAndrew Jeffery 24*e310dd91SPatrick Williams```sh 25c7a446e2SAndrew Jeffery$ ipkdbg 2.11.0 - my-application.core 26c7a446e2SAndrew Jeffery... 27c7a446e2SAndrew Jeffery(gdb) 28c7a446e2SAndrew Jeffery``` 29c7a446e2SAndrew Jeffery 30c7a446e2SAndrew JefferyWhere: 31c7a446e2SAndrew Jeffery 32c7a446e2SAndrew Jeffery1. `ipkdbg` is the script 33c7a446e2SAndrew Jeffery2. `2.11.0` is the OpenBMC version tag that locates the appropriate `opkg.conf` 34c7a446e2SAndrew Jeffery3. `-` is the FILE parameter. `-` specifies it must be extracted from the core 35c7a446e2SAndrew Jeffery4. `my-application.core` is a core dump generated by `/usr/bin/my-application` 36c7a446e2SAndrew Jeffery 37c7a446e2SAndrew JefferyNote that for convenience `ipkdbg` automatically runs `bt` in the `gdb` session 38c7a446e2SAndrew Jefferyonce the core has been loaded to provide a backtrace. 39c7a446e2SAndrew Jeffery 40*e310dd91SPatrick WilliamsNote that we don't need to specify any packages to install into the rootfs _if_ 41c7a446e2SAndrew Jefferythe image package database is available. `ipkdbg` will perform a reverse-lookup 42c7a446e2SAndrew Jefferyto map the absolute binary path extracted from the core to the package that 43c7a446e2SAndrew Jefferyinstalled it. From there `opkg` will resolve all the dependencies. Additionally, 44c7a446e2SAndrew Jeffery`ipkdbg` will identify the associated debug symbol and source packages and 45c7a446e2SAndrew Jefferyinstall them as well. 46c7a446e2SAndrew Jeffery 47c7a446e2SAndrew JefferyIf an image package database is not available then `ipkdbg` will require that 48c7a446e2SAndrew Jefferythe necessary packages are listed on the command-line. 49c7a446e2SAndrew Jeffery 50*e310dd91SPatrick Williams### Scripted Backtrace Extraction 51c7a446e2SAndrew Jeffery 52c7a446e2SAndrew JefferyFor use in scripting environments to automate backtrace extraction `ipkdbg` has 53c7a446e2SAndrew Jefferythe `-q` option to quit `gdb` immediately after the backtrace is printed. The 54c7a446e2SAndrew Jefferyinvocation looks like: 55c7a446e2SAndrew Jeffery 56*e310dd91SPatrick Williams```sh 57*e310dd91SPatrick Williamsipkdbg -q 2.11.0 - my-application.core 58c7a446e2SAndrew Jeffery``` 59c7a446e2SAndrew Jeffery 60*e310dd91SPatrick Williams### Interactive debug 61c7a446e2SAndrew Jeffery 62c7a446e2SAndrew JefferyInteractive debugging is handled by attaching `gdbserver` on the BMC to the 63c7a446e2SAndrew Jefferyrunning instance of `/usr/bin/my-application` on the BMC. `ipkdbg` is then used 64c7a446e2SAndrew Jefferyto generate the `gdb` environment: 65c7a446e2SAndrew Jeffery 66*e310dd91SPatrick Williams```sh 67c7a446e2SAndrew Jeffery$ ipkdbg 2.11.0 /usr/bin/my-application - 68c7a446e2SAndrew Jeffery... 69c7a446e2SAndrew Jeffery(gdb) 70c7a446e2SAndrew Jeffery``` 71c7a446e2SAndrew Jeffery 72c7a446e2SAndrew JefferyWhere: 73c7a446e2SAndrew Jeffery 74c7a446e2SAndrew Jeffery1. `ipkdbg` is the script 75c7a446e2SAndrew Jeffery2. `2.11.0` is the OpenBMC version tag that locates the appropriate `opkg.conf` 76c7a446e2SAndrew Jeffery3. `/usr/bin/my-application` is the absolute path of the application to debug 77c7a446e2SAndrew Jeffery4. `-` is the CORE parameter. `-` specifies that there is no core file 78c7a446e2SAndrew Jeffery 79*e310dd91SPatrick WilliamsNote that we don't need to specify any packages to install into the rootfs _if_ 80c7a446e2SAndrew Jefferythe image package database is available. `ipkdbg` will perform a reverse-lookup 81c7a446e2SAndrew Jefferyto map the absolute binary path provided on the command-line to the package that 82c7a446e2SAndrew Jefferyinstalled it. From there `opkg` will resolve all the dependencies. Additionally, 83c7a446e2SAndrew Jeffery`ipkdbg` will identify the associated debug symbol and source packages and 84c7a446e2SAndrew Jefferyinstall them as well. 85c7a446e2SAndrew Jeffery 86c7a446e2SAndrew JefferyIf an image package database is not available then `ipkdbg` will require that 87c7a446e2SAndrew Jefferythe necessary packages are listed on the command-line. 88c7a446e2SAndrew Jeffery 89c7a446e2SAndrew JefferyOnce the `(gdb)` prompt is reached the usual remote debugging command applies: 90c7a446e2SAndrew Jeffery 91*e310dd91SPatrick Williams```text 92c7a446e2SAndrew Jeffery(gdb) target remote $BMC:1234 93c7a446e2SAndrew Jeffery``` 94c7a446e2SAndrew Jeffery 95*e310dd91SPatrick Williams### Whoops, I Forgot A Package 96c7a446e2SAndrew Jeffery 97c7a446e2SAndrew JefferyTo save exiting the `gdb` session and re-invoking `ipkdbg` to add a package to 98c7a446e2SAndrew Jefferythe rootfs environment, `ipkdbg` installs an `opkg` helper into the PATH of the 99c7a446e2SAndrew Jeffery`gdb` process. With this, you can drop to a shell from `gdb` via the `sh` 100c7a446e2SAndrew Jefferycommand, and then run `opkg install ...` to install the necessary packages. Exit 101c7a446e2SAndrew Jefferythe shell subprocess to return to your `gdb` prompt with the new package 102c7a446e2SAndrew Jefferyincluded in the `gdb` environment. 103c7a446e2SAndrew Jeffery 104*e310dd91SPatrick Williams## Theory of Maintenance 105c7a446e2SAndrew Jeffery 106c7a446e2SAndrew Jeffery`ipkdbg` is intended to operate with minimal assumptions about its environment. 107c7a446e2SAndrew JefferyThe assumptions it does make are: 108c7a446e2SAndrew Jeffery 109c7a446e2SAndrew Jeffery1. It's running in a POSIX environment with access to 110c7a446e2SAndrew Jeffery 1. `coreutils` 111c7a446e2SAndrew Jeffery 2. `awk` 112c7a446e2SAndrew Jeffery 3. `tar` 113c7a446e2SAndrew Jeffery 4. `gzip` 114c7a446e2SAndrew Jeffery 5. `wget` 115c7a446e2SAndrew Jeffery2. A multi-arch capable `gdb` is installed 116c7a446e2SAndrew Jeffery 117c7a446e2SAndrew JefferyHowever, `ipkdbg` also relies on `opkg` to get its job done. As `opkg` is itself 118c7a446e2SAndrew Jefferya package manager, it tends not to be available as a package via distro package 119c7a446e2SAndrew Jefferymanagers. A conflicting requirement is that we want `ipkdbg` to be able to run 120c7a446e2SAndrew Jefferyalmost anywhere without fuss. To resolve these issues `ipkdbg` packages `opkg` 121c7a446e2SAndrew Jefferybinaries inside itself as a self-extracting shell-script. What you see in this 122c7a446e2SAndrew Jefferydirectory is the input script ([`ipkdbg.in`](ipkdbg.in)) and build 123c7a446e2SAndrew Jefferyinfrastructure ([`Makefile`](Makefile)) for generating an `ipkdbg` executable 124c7a446e2SAndrew Jefferythat packages one or more `opkg` binaries. 125c7a446e2SAndrew Jeffery 126c7a446e2SAndrew Jeffery`ipkdbg` supports multiple distros and architectures by defining a directory 127c7a446e2SAndrew Jefferyscheme to contain `opkg` binaries. This hierarchy is archived and compressed 128c7a446e2SAndrew Jefferyusing `tar` and `gzip`, and the archive embedded in the `ipkdbg` shell script by 129c7a446e2SAndrew Jefferythe [`Makefile`](Makefile). The `opkg` path for a given architecture, distro and 130c7a446e2SAndrew Jefferyrelease combination is defined by the following directory hierarchy in terms of 131c7a446e2SAndrew Jeffery[`os-release(5)`][os-release]: 132c7a446e2SAndrew Jeffery 133*e310dd91SPatrick Williams[os-release]: 134*e310dd91SPatrick Williams https://www.man7.org/linux/man-pages/man5/os-release.5.html#OPTIONS 135c7a446e2SAndrew Jeffery 136*e310dd91SPatrick Williams```sh 137c7a446e2SAndrew Jeffery$(uname -m)/${ID}/${VERSION_ID}/opkg 138c7a446e2SAndrew Jeffery``` 139c7a446e2SAndrew Jeffery 140c7a446e2SAndrew JefferyThis hierarchy must be placed under a `bin/` directory alongside the 141c7a446e2SAndrew Jeffery[`Makefile`](Makefile). 142c7a446e2SAndrew Jeffery 143c7a446e2SAndrew JefferyTo avoid licensing and distribution legal issues openbmc-tools does not provide 144c7a446e2SAndrew Jefferypre-built `opkg` binaries. You must build your own and integrate them into the 145*e310dd91SPatrick Williamshierarchy under `bin/` outlined above. To this end, 146*e310dd91SPatrick Williams[the `build-opkg` script is provided](build-opkg). It probably won't work 147*e310dd91SPatrick Williamswithout tinkering for your target distro, but it might get you 90% of the way 148*e310dd91SPatrick Williamsthere. 149c7a446e2SAndrew Jeffery 150*e310dd91SPatrick Williams## Maintenance in Practice 151c7a446e2SAndrew Jeffery 152c7a446e2SAndrew JefferyOnce you have built the required `opkg` binaries and integrated them into the 153c7a446e2SAndrew Jefferyhierarchy under `bin/` alongside the `Makefile`, run `make` to generate the 154c7a446e2SAndrew Jefferyfinal `ipkdbg` script. The output script can then be freely copied between 155c7a446e2SAndrew Jefferymachines supported by the embedded `opkg` binaries. The build system will handle 156c7a446e2SAndrew Jefferystripping the `opkg` binaries to ensure their size is reduced as much as 157c7a446e2SAndrew Jefferypractical prior to archiving and compression. It is worth committing the 158c7a446e2SAndrew Jefferybinaries under `bin/` to ensure that the build for your output `ipkdbg` script 159c7a446e2SAndrew Jefferyis reproducible. 160c7a446e2SAndrew Jeffery 161*e310dd91SPatrick Williams## Theory of Integration 162c7a446e2SAndrew Jeffery 163c7a446e2SAndrew JefferyFor `ipkdbg` to do its job via `opkg` it must be possible for it to acquire an 164c7a446e2SAndrew Jeffery`opkg.conf` that describes the location of the package archive for your target 165c7a446e2SAndrew Jefferyenvironment. The `IPKDBG_CONF_*` variables help describe a `wget`-compatible URL 166c7a446e2SAndrew Jefferywhere this configuration lives. `ipkdbg` will bootstrap by fetching this 167c7a446e2SAndrew Jeffery`opkg.conf` and then passing it as a parameter to all invocations of `opkg`, 168c7a446e2SAndrew Jefferywhere these invocations of `opkg` populate a rootfs used as your `gdb` debugging 169c7a446e2SAndrew Jefferyenvironment. 170c7a446e2SAndrew Jeffery 171c7a446e2SAndrew JefferyA consequence of this is you must also host `ipk` package archives whose URLs 172c7a446e2SAndrew Jefferycan be described in the acquired `opkg.conf`. You will need one complete package 173c7a446e2SAndrew Jefferyarchive per BMC target environment (tag or release). 174c7a446e2SAndrew Jeffery 175c7a446e2SAndrew JefferyThe packages can be extracted from the bitbake build tree by archiving 176c7a446e2SAndrew Jeffery`tmp/deploy/ipk` once `bitbake obmc-phosphor-image && bitbake package-index` has 177c7a446e2SAndrew Jefferyexited successfully. 178c7a446e2SAndrew Jeffery 179*e310dd91SPatrick WilliamsFinally, `ipkdbg` works best when it has access to the image's installed package 180*e310dd91SPatrick Williamsdatabase. This can be captured from the build tree when the build is configured 181*e310dd91SPatrick Williamswith [`INC_IPK_IMAGE_GEN = "1"`][incremental-builds] by archiving the directory 182*e310dd91SPatrick Williamshierarchy under `./tmp/work/*/obmc-phosphor-image/1.0-r0/temp/saved` with the 183*e310dd91SPatrick Williamsfollowing incantation: 184c7a446e2SAndrew Jeffery 185*e310dd91SPatrick Williams```sh 186*e310dd91SPatrick Williamstar -cJf opkg-database.tar.xz \ 187c7a446e2SAndrew Jeffery -C ./tmp/work/*/obmc-phosphor-image/1.0-r0/temp/saved/target/ \ 188c7a446e2SAndrew Jeffery info lists status 189c7a446e2SAndrew Jeffery``` 190c7a446e2SAndrew Jeffery 191c7a446e2SAndrew Jeffery`ipkdbg` assumes that the appropriate `opkg-database.tar.xz` can be fetched 192c7a446e2SAndrew Jefferyusing `wget` and that its URL can be generated in the same manner as that for 193c7a446e2SAndrew Jeffery`opkg.conf`. 194c7a446e2SAndrew Jeffery 195*e310dd91SPatrick Williams[incremental-builds]: 196*e310dd91SPatrick Williams https://git.openembedded.org/openembedded-core/commit/?id=adf587e55c0f9bc74f0bef415273c937401baebb 197