1*c7a446e2SAndrew Jeffery## `ipkdbg`: Generate gdb environments from opkg package archives 2*c7a446e2SAndrew Jeffery 3*c7a446e2SAndrew Jeffery`ipkdbg` automates the process of generating `gdb` environments for interactive 4*c7a446e2SAndrew Jefferydebugging and coredump analysis of split-debug binaries by exploiting bitbake's 5*c7a446e2SAndrew Jefferyruntime package management support outside the context of the BMC. 6*c7a446e2SAndrew Jeffery 7*c7a446e2SAndrew Jeffery### Use of `ipkdbg` 8*c7a446e2SAndrew Jeffery 9*c7a446e2SAndrew JefferyAssuming the presence of the [appropriate integration](#theory-of-integration), 10*c7a446e2SAndrew Jefferyusing `ipkdbg` requires minimal fuss. 11*c7a446e2SAndrew Jeffery 12*c7a446e2SAndrew Jeffery``` 13*c7a446e2SAndrew JefferySYNOPSIS 14*c7a446e2SAndrew Jeffery ipkdbg [-q] RELEASE FILE CORE [PACKAGE...] 15*c7a446e2SAndrew Jeffery``` 16*c7a446e2SAndrew Jeffery 17*c7a446e2SAndrew JefferyIt handles the following use-cases: 18*c7a446e2SAndrew Jeffery 19*c7a446e2SAndrew Jeffery#### Core Dump Analysis 20*c7a446e2SAndrew Jeffery 21*c7a446e2SAndrew JefferyAfter extracting a core file from the BMC an interactive `gdb` session can be 22*c7a446e2SAndrew Jefferyset up against it with: 23*c7a446e2SAndrew Jeffery 24*c7a446e2SAndrew Jeffery``` 25*c7a446e2SAndrew Jeffery$ ipkdbg 2.11.0 - my-application.core 26*c7a446e2SAndrew Jeffery... 27*c7a446e2SAndrew Jeffery(gdb) 28*c7a446e2SAndrew Jeffery``` 29*c7a446e2SAndrew Jeffery 30*c7a446e2SAndrew JefferyWhere: 31*c7a446e2SAndrew Jeffery 32*c7a446e2SAndrew Jeffery1. `ipkdbg` is the script 33*c7a446e2SAndrew Jeffery2. `2.11.0` is the OpenBMC version tag that locates the appropriate `opkg.conf` 34*c7a446e2SAndrew Jeffery3. `-` is the FILE parameter. `-` specifies it must be extracted from the core 35*c7a446e2SAndrew Jeffery4. `my-application.core` is a core dump generated by `/usr/bin/my-application` 36*c7a446e2SAndrew Jeffery 37*c7a446e2SAndrew JefferyNote that for convenience `ipkdbg` automatically runs `bt` in the `gdb` session 38*c7a446e2SAndrew Jefferyonce the core has been loaded to provide a backtrace. 39*c7a446e2SAndrew Jeffery 40*c7a446e2SAndrew JefferyNote that we don't need to specify any packages to install into the rootfs *if* 41*c7a446e2SAndrew Jefferythe image package database is available. `ipkdbg` will perform a reverse-lookup 42*c7a446e2SAndrew Jefferyto map the absolute binary path extracted from the core to the package that 43*c7a446e2SAndrew Jefferyinstalled it. From there `opkg` will resolve all the dependencies. Additionally, 44*c7a446e2SAndrew Jeffery`ipkdbg` will identify the associated debug symbol and source packages and 45*c7a446e2SAndrew Jefferyinstall them as well. 46*c7a446e2SAndrew Jeffery 47*c7a446e2SAndrew JefferyIf an image package database is not available then `ipkdbg` will require that 48*c7a446e2SAndrew Jefferythe necessary packages are listed on the command-line. 49*c7a446e2SAndrew Jeffery 50*c7a446e2SAndrew Jeffery#### Scripted Backtrace Extraction 51*c7a446e2SAndrew Jeffery 52*c7a446e2SAndrew JefferyFor use in scripting environments to automate backtrace extraction `ipkdbg` has 53*c7a446e2SAndrew Jefferythe `-q` option to quit `gdb` immediately after the backtrace is printed. The 54*c7a446e2SAndrew Jefferyinvocation looks like: 55*c7a446e2SAndrew Jeffery 56*c7a446e2SAndrew Jeffery``` 57*c7a446e2SAndrew Jeffery$ ipkdbg -q 2.11.0 - my-application.core 58*c7a446e2SAndrew Jeffery``` 59*c7a446e2SAndrew Jeffery 60*c7a446e2SAndrew Jeffery#### Interactive debug 61*c7a446e2SAndrew Jeffery 62*c7a446e2SAndrew JefferyInteractive debugging is handled by attaching `gdbserver` on the BMC to the 63*c7a446e2SAndrew Jefferyrunning instance of `/usr/bin/my-application` on the BMC. `ipkdbg` is then used 64*c7a446e2SAndrew Jefferyto generate the `gdb` environment: 65*c7a446e2SAndrew Jeffery 66*c7a446e2SAndrew Jeffery``` 67*c7a446e2SAndrew Jeffery$ ipkdbg 2.11.0 /usr/bin/my-application - 68*c7a446e2SAndrew Jeffery... 69*c7a446e2SAndrew Jeffery(gdb) 70*c7a446e2SAndrew Jeffery``` 71*c7a446e2SAndrew Jeffery 72*c7a446e2SAndrew JefferyWhere: 73*c7a446e2SAndrew Jeffery 74*c7a446e2SAndrew Jeffery1. `ipkdbg` is the script 75*c7a446e2SAndrew Jeffery2. `2.11.0` is the OpenBMC version tag that locates the appropriate `opkg.conf` 76*c7a446e2SAndrew Jeffery3. `/usr/bin/my-application` is the absolute path of the application to debug 77*c7a446e2SAndrew Jeffery4. `-` is the CORE parameter. `-` specifies that there is no core file 78*c7a446e2SAndrew Jeffery 79*c7a446e2SAndrew JefferyNote that we don't need to specify any packages to install into the rootfs *if* 80*c7a446e2SAndrew Jefferythe image package database is available. `ipkdbg` will perform a reverse-lookup 81*c7a446e2SAndrew Jefferyto map the absolute binary path provided on the command-line to the package that 82*c7a446e2SAndrew Jefferyinstalled it. From there `opkg` will resolve all the dependencies. Additionally, 83*c7a446e2SAndrew Jeffery`ipkdbg` will identify the associated debug symbol and source packages and 84*c7a446e2SAndrew Jefferyinstall them as well. 85*c7a446e2SAndrew Jeffery 86*c7a446e2SAndrew JefferyIf an image package database is not available then `ipkdbg` will require that 87*c7a446e2SAndrew Jefferythe necessary packages are listed on the command-line. 88*c7a446e2SAndrew Jeffery 89*c7a446e2SAndrew JefferyOnce the `(gdb)` prompt is reached the usual remote debugging command applies: 90*c7a446e2SAndrew Jeffery 91*c7a446e2SAndrew Jeffery``` 92*c7a446e2SAndrew Jeffery(gdb) target remote $BMC:1234 93*c7a446e2SAndrew Jeffery``` 94*c7a446e2SAndrew Jeffery 95*c7a446e2SAndrew Jeffery#### Whoops, I Forgot A Package 96*c7a446e2SAndrew Jeffery 97*c7a446e2SAndrew JefferyTo save exiting the `gdb` session and re-invoking `ipkdbg` to add a package to 98*c7a446e2SAndrew Jefferythe rootfs environment, `ipkdbg` installs an `opkg` helper into the PATH of the 99*c7a446e2SAndrew Jeffery`gdb` process. With this, you can drop to a shell from `gdb` via the `sh` 100*c7a446e2SAndrew Jefferycommand, and then run `opkg install ...` to install the necessary packages. Exit 101*c7a446e2SAndrew Jefferythe shell subprocess to return to your `gdb` prompt with the new package 102*c7a446e2SAndrew Jefferyincluded in the `gdb` environment. 103*c7a446e2SAndrew Jeffery 104*c7a446e2SAndrew Jeffery### Theory of Maintenance 105*c7a446e2SAndrew Jeffery 106*c7a446e2SAndrew Jeffery`ipkdbg` is intended to operate with minimal assumptions about its environment. 107*c7a446e2SAndrew JefferyThe assumptions it does make are: 108*c7a446e2SAndrew Jeffery 109*c7a446e2SAndrew Jeffery1. It's running in a POSIX environment with access to 110*c7a446e2SAndrew Jeffery 1. `coreutils` 111*c7a446e2SAndrew Jeffery 2. `awk` 112*c7a446e2SAndrew Jeffery 3. `tar` 113*c7a446e2SAndrew Jeffery 4. `gzip` 114*c7a446e2SAndrew Jeffery 5. `wget` 115*c7a446e2SAndrew Jeffery2. A multi-arch capable `gdb` is installed 116*c7a446e2SAndrew Jeffery 117*c7a446e2SAndrew JefferyHowever, `ipkdbg` also relies on `opkg` to get its job done. As `opkg` is itself 118*c7a446e2SAndrew Jefferya package manager, it tends not to be available as a package via distro package 119*c7a446e2SAndrew Jefferymanagers. A conflicting requirement is that we want `ipkdbg` to be able to run 120*c7a446e2SAndrew Jefferyalmost anywhere without fuss. To resolve these issues `ipkdbg` packages `opkg` 121*c7a446e2SAndrew Jefferybinaries inside itself as a self-extracting shell-script. What you see in this 122*c7a446e2SAndrew Jefferydirectory is the input script ([`ipkdbg.in`](ipkdbg.in)) and build 123*c7a446e2SAndrew Jefferyinfrastructure ([`Makefile`](Makefile)) for generating an `ipkdbg` executable 124*c7a446e2SAndrew Jefferythat packages one or more `opkg` binaries. 125*c7a446e2SAndrew Jeffery 126*c7a446e2SAndrew Jeffery`ipkdbg` supports multiple distros and architectures by defining a directory 127*c7a446e2SAndrew Jefferyscheme to contain `opkg` binaries. This hierarchy is archived and compressed 128*c7a446e2SAndrew Jefferyusing `tar` and `gzip`, and the archive embedded in the `ipkdbg` shell script by 129*c7a446e2SAndrew Jefferythe [`Makefile`](Makefile). The `opkg` path for a given architecture, distro and 130*c7a446e2SAndrew Jefferyrelease combination is defined by the following directory hierarchy in terms of 131*c7a446e2SAndrew Jeffery[`os-release(5)`][os-release]: 132*c7a446e2SAndrew Jeffery 133*c7a446e2SAndrew Jeffery[os-release]: https://www.man7.org/linux/man-pages/man5/os-release.5.html#OPTIONS 134*c7a446e2SAndrew Jeffery 135*c7a446e2SAndrew Jeffery``` 136*c7a446e2SAndrew Jeffery$(uname -m)/${ID}/${VERSION_ID}/opkg 137*c7a446e2SAndrew Jeffery``` 138*c7a446e2SAndrew Jeffery 139*c7a446e2SAndrew JefferyThis hierarchy must be placed under a `bin/` directory alongside the 140*c7a446e2SAndrew Jeffery[`Makefile`](Makefile). 141*c7a446e2SAndrew Jeffery 142*c7a446e2SAndrew JefferyTo avoid licensing and distribution legal issues openbmc-tools does not provide 143*c7a446e2SAndrew Jefferypre-built `opkg` binaries. You must build your own and integrate them into the 144*c7a446e2SAndrew Jefferyhierarchy under `bin/` outlined above. To this end, [the 145*c7a446e2SAndrew Jeffery`build-opkg` script is provided](build-opkg). It probably won't work without 146*c7a446e2SAndrew Jefferytinkering for your target distro, but it might get you 90% of the way there. 147*c7a446e2SAndrew Jeffery 148*c7a446e2SAndrew Jeffery### Maintenance in Practice 149*c7a446e2SAndrew Jeffery 150*c7a446e2SAndrew JefferyOnce you have built the required `opkg` binaries and integrated them into the 151*c7a446e2SAndrew Jefferyhierarchy under `bin/` alongside the `Makefile`, run `make` to generate the 152*c7a446e2SAndrew Jefferyfinal `ipkdbg` script. The output script can then be freely copied between 153*c7a446e2SAndrew Jefferymachines supported by the embedded `opkg` binaries. The build system will handle 154*c7a446e2SAndrew Jefferystripping the `opkg` binaries to ensure their size is reduced as much as 155*c7a446e2SAndrew Jefferypractical prior to archiving and compression. It is worth committing the 156*c7a446e2SAndrew Jefferybinaries under `bin/` to ensure that the build for your output `ipkdbg` script 157*c7a446e2SAndrew Jefferyis reproducible. 158*c7a446e2SAndrew Jeffery 159*c7a446e2SAndrew Jeffery### Theory of Integration 160*c7a446e2SAndrew Jeffery 161*c7a446e2SAndrew JefferyFor `ipkdbg` to do its job via `opkg` it must be possible for it to acquire an 162*c7a446e2SAndrew Jeffery`opkg.conf` that describes the location of the package archive for your target 163*c7a446e2SAndrew Jefferyenvironment. The `IPKDBG_CONF_*` variables help describe a `wget`-compatible URL 164*c7a446e2SAndrew Jefferywhere this configuration lives. `ipkdbg` will bootstrap by fetching this 165*c7a446e2SAndrew Jeffery`opkg.conf` and then passing it as a parameter to all invocations of `opkg`, 166*c7a446e2SAndrew Jefferywhere these invocations of `opkg` populate a rootfs used as your `gdb` debugging 167*c7a446e2SAndrew Jefferyenvironment. 168*c7a446e2SAndrew Jeffery 169*c7a446e2SAndrew JefferyA consequence of this is you must also host `ipk` package archives whose URLs 170*c7a446e2SAndrew Jefferycan be described in the acquired `opkg.conf`. You will need one complete package 171*c7a446e2SAndrew Jefferyarchive per BMC target environment (tag or release). 172*c7a446e2SAndrew Jeffery 173*c7a446e2SAndrew JefferyThe packages can be extracted from the bitbake build tree by archiving 174*c7a446e2SAndrew Jeffery`tmp/deploy/ipk` once `bitbake obmc-phosphor-image && bitbake package-index` has 175*c7a446e2SAndrew Jefferyexited successfully. 176*c7a446e2SAndrew Jeffery 177*c7a446e2SAndrew JefferyFinally, `ipkdbg` works best when it has access to the image's installed 178*c7a446e2SAndrew Jefferypackage database. This can be captured from the build tree when the build is 179*c7a446e2SAndrew Jefferyconfigured with [`INC_IPK_IMAGE_GEN = "1"`][incremental-builds] by archiving 180*c7a446e2SAndrew Jefferythe directory hierarchy under 181*c7a446e2SAndrew Jeffery`./tmp/work/*/obmc-phosphor-image/1.0-r0/temp/saved` with the following 182*c7a446e2SAndrew Jefferyincantation: 183*c7a446e2SAndrew Jeffery 184*c7a446e2SAndrew Jeffery``` 185*c7a446e2SAndrew Jeffery$ tar -cJf opkg-database.tar.xz \ 186*c7a446e2SAndrew Jeffery -C ./tmp/work/*/obmc-phosphor-image/1.0-r0/temp/saved/target/ \ 187*c7a446e2SAndrew Jeffery info lists status 188*c7a446e2SAndrew Jeffery``` 189*c7a446e2SAndrew Jeffery 190*c7a446e2SAndrew Jeffery`ipkdbg` assumes that the appropriate `opkg-database.tar.xz` can be fetched 191*c7a446e2SAndrew Jefferyusing `wget` and that its URL can be generated in the same manner as that for 192*c7a446e2SAndrew Jeffery`opkg.conf`. 193*c7a446e2SAndrew Jeffery 194*c7a446e2SAndrew Jeffery[incremental-builds]: https://git.openembedded.org/openembedded-core/commit/?id=adf587e55c0f9bc74f0bef415273c937401baebb 195