1.. SPDX-License-Identifier: GPL-2.0
2
3=================
4Boot-time tracing
5=================
6
7:Author: Masami Hiramatsu <mhiramat@kernel.org>
8
9Overview
10========
11
12Boot-time tracing allows users to trace boot-time process including
13device initialization with full features of ftrace including per-event
14filter and actions, histograms, kprobe-events and synthetic-events,
15and trace instances.
16Since kernel command line is not enough to control these complex features,
17this uses bootconfig file to describe tracing feature programming.
18
19Options in the Boot Config
20==========================
21
22Here is the list of available options list for boot time tracing in
23boot config file [1]_. All options are under "ftrace." or "kernel."
24prefix. See kernel parameters for the options which starts
25with "kernel." prefix [2]_.
26
27.. [1] See :ref:`Documentation/admin-guide/bootconfig.rst <bootconfig>`
28.. [2] See :ref:`Documentation/admin-guide/kernel-parameters.rst <kernelparameters>`
29
30Ftrace Global Options
31---------------------
32
33Ftrace global options have "kernel." prefix in boot config, which means
34these options are passed as a part of kernel legacy command line.
35
36kernel.tp_printk
37   Output trace-event data on printk buffer too.
38
39kernel.dump_on_oops [= MODE]
40   Dump ftrace on Oops. If MODE = 1 or omitted, dump trace buffer
41   on all CPUs. If MODE = 2, dump a buffer on a CPU which kicks Oops.
42
43kernel.traceoff_on_warning
44   Stop tracing if WARN_ON() occurs.
45
46kernel.fgraph_max_depth = MAX_DEPTH
47   Set MAX_DEPTH to maximum depth of fgraph tracer.
48
49kernel.fgraph_filters = FILTER[, FILTER2...]
50   Add fgraph tracing function filters.
51
52kernel.fgraph_notraces = FILTER[, FILTER2...]
53   Add fgraph non-tracing function filters.
54
55
56Ftrace Per-instance Options
57---------------------------
58
59These options can be used for each instance including global ftrace node.
60
61ftrace.[instance.INSTANCE.]options = OPT1[, OPT2[...]]
62   Enable given ftrace options.
63
64ftrace.[instance.INSTANCE.]tracing_on = 0|1
65   Enable/Disable tracing on this instance when starting boot-time tracing.
66   (you can enable it by the "traceon" event trigger action)
67
68ftrace.[instance.INSTANCE.]trace_clock = CLOCK
69   Set given CLOCK to ftrace's trace_clock.
70
71ftrace.[instance.INSTANCE.]buffer_size = SIZE
72   Configure ftrace buffer size to SIZE. You can use "KB" or "MB"
73   for that SIZE.
74
75ftrace.[instance.INSTANCE.]alloc_snapshot
76   Allocate snapshot buffer.
77
78ftrace.[instance.INSTANCE.]cpumask = CPUMASK
79   Set CPUMASK as trace cpu-mask.
80
81ftrace.[instance.INSTANCE.]events = EVENT[, EVENT2[...]]
82   Enable given events on boot. You can use a wild card in EVENT.
83
84ftrace.[instance.INSTANCE.]tracer = TRACER
85   Set TRACER to current tracer on boot. (e.g. function)
86
87ftrace.[instance.INSTANCE.]ftrace.filters
88   This will take an array of tracing function filter rules.
89
90ftrace.[instance.INSTANCE.]ftrace.notraces
91   This will take an array of NON-tracing function filter rules.
92
93
94Ftrace Per-Event Options
95------------------------
96
97These options are setting per-event options.
98
99ftrace.[instance.INSTANCE.]event.GROUP.EVENT.enable
100   Enable GROUP:EVENT tracing.
101
102ftrace.[instance.INSTANCE.]event.GROUP.enable
103   Enable all event tracing within GROUP.
104
105ftrace.[instance.INSTANCE.]event.enable
106   Enable all event tracing.
107
108ftrace.[instance.INSTANCE.]event.GROUP.EVENT.filter = FILTER
109   Set FILTER rule to the GROUP:EVENT.
110
111ftrace.[instance.INSTANCE.]event.GROUP.EVENT.actions = ACTION[, ACTION2[...]]
112   Set ACTIONs to the GROUP:EVENT.
113
114ftrace.[instance.INSTANCE.]event.kprobes.EVENT.probes = PROBE[, PROBE2[...]]
115   Defines new kprobe event based on PROBEs. It is able to define
116   multiple probes on one event, but those must have same type of
117   arguments. This option is available only for the event which
118   group name is "kprobes".
119
120ftrace.[instance.INSTANCE.]event.synthetic.EVENT.fields = FIELD[, FIELD2[...]]
121   Defines new synthetic event with FIELDs. Each field should be
122   "type varname".
123
124Note that kprobe and synthetic event definitions can be written under
125instance node, but those are also visible from other instances. So please
126take care for event name conflict.
127
128
129When to Start
130=============
131
132All boot-time tracing options starting with ``ftrace`` will be enabled at the
133end of core_initcall. This means you can trace the events from postcore_initcall.
134Most of the subsystems and architecture dependent drivers will be initialized
135after that (arch_initcall or subsys_initcall). Thus, you can trace those with
136boot-time tracing.
137If you want to trace events before core_initcall, you can use the options
138starting with ``kernel``. Some of them will be enabled eariler than the initcall
139processing (for example,. ``kernel.ftrace=function`` and ``kernel.trace_event``
140will start before the initcall.)
141
142
143Examples
144========
145
146For example, to add filter and actions for each event, define kprobe
147events, and synthetic events with histogram, write a boot config like
148below::
149
150  ftrace.event {
151        task.task_newtask {
152                filter = "pid < 128"
153                enable
154        }
155        kprobes.vfs_read {
156                probes = "vfs_read $arg1 $arg2"
157                filter = "common_pid < 200"
158                enable
159        }
160        synthetic.initcall_latency {
161                fields = "unsigned long func", "u64 lat"
162                actions = "hist:keys=func.sym,lat:vals=lat:sort=lat"
163        }
164        initcall.initcall_start {
165                actions = "hist:keys=func:ts0=common_timestamp.usecs"
166        }
167        initcall.initcall_finish {
168                actions = "hist:keys=func:lat=common_timestamp.usecs-$ts0:onmatch(initcall.initcall_start).initcall_latency(func,$lat)"
169        }
170  }
171
172Also, boot-time tracing supports "instance" node, which allows us to run
173several tracers for different purpose at once. For example, one tracer
174is for tracing functions starting with "user\_", and others tracing
175"kernel\_" functions, you can write boot config as below::
176
177  ftrace.instance {
178        foo {
179                tracer = "function"
180                ftrace.filters = "user_*"
181        }
182        bar {
183                tracer = "function"
184                ftrace.filters = "kernel_*"
185        }
186  }
187
188The instance node also accepts event nodes so that each instance
189can customize its event tracing.
190
191With the trigger action and kprobes, you can trace function-graph while
192a function is called. For example, this will trace all function calls in
193the pci_proc_init()::
194
195  ftrace {
196        tracing_on = 0
197        tracer = function_graph
198        event.kprobes {
199                start_event {
200                        probes = "pci_proc_init"
201                        actions = "traceon"
202                }
203                end_event {
204                        probes = "pci_proc_init%return"
205                        actions = "traceoff"
206                }
207        }
208  }
209
210
211This boot-time tracing also supports ftrace kernel parameters via boot
212config.
213For example, following kernel parameters::
214
215 trace_options=sym-addr trace_event=initcall:* tp_printk trace_buf_size=1M ftrace=function ftrace_filter="vfs*"
216
217This can be written in boot config like below::
218
219  kernel {
220        trace_options = sym-addr
221        trace_event = "initcall:*"
222        tp_printk
223        trace_buf_size = 1M
224        ftrace = function
225        ftrace_filter = "vfs*"
226  }
227
228Note that parameters start with "kernel" prefix instead of "ftrace".
229