openbmc-tools/ipkdbg/README.md

# `ipkdbg`: Generate gdb environments from opkg package archives

`ipkdbg` automates the process of generating `gdb` environments for interactive
debugging and coredump analysis of split-debug binaries by exploiting bitbake's
runtime package management support outside the context of the BMC.

## Use of `ipkdbg`

Assuming the presence of the [appropriate integration](#theory-of-integration),
using `ipkdbg` requires minimal fuss.

```text
SYNOPSIS
    ipkdbg [-q] RELEASE FILE CORE [PACKAGE...]
```

It handles the following use-cases:

### Core Dump Analysis

After extracting a core file from the BMC an interactive `gdb` session can be
set up against it with:

```sh
$ ipkdbg 2.11.0 - my-application.core
...
(gdb)
```

Where:

1. `ipkdbg` is the script
2. `2.11.0` is the OpenBMC version tag that locates the appropriate `opkg.conf`
3. `-` is the FILE parameter. `-` specifies it must be extracted from the core
4. `my-application.core` is a core dump generated by `/usr/bin/my-application`

Note that for convenience `ipkdbg` automatically runs `bt` in the `gdb` session
once the core has been loaded to provide a backtrace.

Note that we don't need to specify any packages to install into the rootfs _if_
the image package database is available. `ipkdbg` will perform a reverse-lookup
to map the absolute binary path extracted from the core to the package that
installed it. From there `opkg` will resolve all the dependencies. Additionally,
`ipkdbg` will identify the associated debug symbol and source packages and
install them as well.

If an image package database is not available then `ipkdbg` will require that
the necessary packages are listed on the command-line.

### Scripted Backtrace Extraction

For use in scripting environments to automate backtrace extraction `ipkdbg` has
the `-q` option to quit `gdb` immediately after the backtrace is printed. The
invocation looks like:

```sh
ipkdbg -q 2.11.0 - my-application.core
```

### Interactive debug

Interactive debugging is handled by attaching `gdbserver` on the BMC to the
running instance of `/usr/bin/my-application` on the BMC. `ipkdbg` is then used
to generate the `gdb` environment:

```sh
$ ipkdbg 2.11.0 /usr/bin/my-application -
...
(gdb)
```

Where:

1. `ipkdbg` is the script
2. `2.11.0` is the OpenBMC version tag that locates the appropriate `opkg.conf`
3. `/usr/bin/my-application` is the absolute path of the application to debug
4. `-` is the CORE parameter. `-` specifies that there is no core file

Note that we don't need to specify any packages to install into the rootfs _if_
the image package database is available. `ipkdbg` will perform a reverse-lookup
to map the absolute binary path provided on the command-line to the package that
installed it. From there `opkg` will resolve all the dependencies. Additionally,
`ipkdbg` will identify the associated debug symbol and source packages and
install them as well.

If an image package database is not available then `ipkdbg` will require that
the necessary packages are listed on the command-line.

Once the `(gdb)` prompt is reached the usual remote debugging command applies:

```text
(gdb) target remote $BMC:1234
```

### Whoops, I Forgot A Package

To save exiting the `gdb` session and re-invoking `ipkdbg` to add a package to
the rootfs environment, `ipkdbg` installs an `opkg` helper into the PATH of the
`gdb` process. With this, you can drop to a shell from `gdb` via the `sh`
command, and then run `opkg install ...` to install the necessary packages. Exit
the shell subprocess to return to your `gdb` prompt with the new package
included in the `gdb` environment.

## Theory of Maintenance

`ipkdbg` is intended to operate with minimal assumptions about its environment.
The assumptions it does make are:

1. It's running in a POSIX environment with access to
   1. `coreutils`
   2. `awk`
   3. `tar`
   4. `gzip`
   5. `wget`
2. A multi-arch capable `gdb` is installed

However, `ipkdbg` also relies on `opkg` to get its job done. As `opkg` is itself
a package manager, it tends not to be available as a package via distro package
managers. A conflicting requirement is that we want `ipkdbg` to be able to run
almost anywhere without fuss. To resolve these issues `ipkdbg` packages `opkg`
binaries inside itself as a self-extracting shell-script. What you see in this
directory is the input script ([`ipkdbg.in`](ipkdbg.in)) and build
infrastructure ([`Makefile`](Makefile)) for generating an `ipkdbg` executable
that packages one or more `opkg` binaries.

`ipkdbg` supports multiple distros and architectures by defining a directory
scheme to contain `opkg` binaries. This hierarchy is archived and compressed
using `tar` and `gzip`, and the archive embedded in the `ipkdbg` shell script by
the [`Makefile`](Makefile). The `opkg` path for a given architecture, distro and
release combination is defined by the following directory hierarchy in terms of
[`os-release(5)`][os-release]:

[os-release]:
  https://www.man7.org/linux/man-pages/man5/os-release.5.html#OPTIONS

```sh
$(uname -m)/${ID}/${VERSION_ID}/opkg
```

This hierarchy must be placed under a `bin/` directory alongside the
[`Makefile`](Makefile).

To avoid licensing and distribution legal issues openbmc-tools does not provide
pre-built `opkg` binaries. You must build your own and integrate them into the
hierarchy under `bin/` outlined above. To this end,
[the `build-opkg` script is provided](build-opkg). It probably won't work
without tinkering for your target distro, but it might get you 90% of the way
there.

## Maintenance in Practice

Once you have built the required `opkg` binaries and integrated them into the
hierarchy under `bin/` alongside the `Makefile`, run `make` to generate the
final `ipkdbg` script. The output script can then be freely copied between
machines supported by the embedded `opkg` binaries. The build system will handle
stripping the `opkg` binaries to ensure their size is reduced as much as
practical prior to archiving and compression. It is worth committing the
binaries under `bin/` to ensure that the build for your output `ipkdbg` script
is reproducible.

## Theory of Integration

For `ipkdbg` to do its job via `opkg` it must be possible for it to acquire an
`opkg.conf` that describes the location of the package archive for your target
environment. The `IPKDBG_CONF_*` variables help describe a `wget`-compatible URL
where this configuration lives. `ipkdbg` will bootstrap by fetching this
`opkg.conf` and then passing it as a parameter to all invocations of `opkg`,
where these invocations of `opkg` populate a rootfs used as your `gdb` debugging
environment.

A consequence of this is you must also host `ipk` package archives whose URLs
can be described in the acquired `opkg.conf`. You will need one complete package
archive per BMC target environment (tag or release).

The packages can be extracted from the bitbake build tree by archiving
`tmp/deploy/ipk` once `bitbake obmc-phosphor-image && bitbake package-index` has
exited successfully.

Finally, `ipkdbg` works best when it has access to the image's installed package
database. This can be captured from the build tree when the build is configured
with [`INC_IPK_IMAGE_GEN = "1"`][incremental-builds] by archiving the directory
hierarchy under `./tmp/work/*/obmc-phosphor-image/1.0-r0/temp/saved` with the
following incantation:

```sh
tar -cJf opkg-database.tar.xz \
    -C ./tmp/work/*/obmc-phosphor-image/1.0-r0/temp/saved/target/ \
    info lists status
```

`ipkdbg` assumes that the appropriate `opkg-database.tar.xz` can be fetched
using `wget` and that its URL can be generated in the same manner as that for
`opkg.conf`.

[incremental-builds]:
  https://git.openembedded.org/openembedded-core/commit/?id=adf587e55c0f9bc74f0bef415273c937401baebb