1.. SPDX-License-Identifier: GPL-2.0
2
3=================
4Boot-time tracing
5=================
6
7:Author: Masami Hiramatsu <mhiramat@kernel.org>
8
9Overview
10========
11
12Boot-time tracing allows users to trace boot-time process including
13device initialization with full features of ftrace including per-event
14filter and actions, histograms, kprobe-events and synthetic-events,
15and trace instances.
16Since kernel command line is not enough to control these complex features,
17this uses bootconfig file to describe tracing feature programming.
18
19Options in the Boot Config
20==========================
21
22Here is the list of available options list for boot time tracing in
23boot config file [1]_. All options are under "ftrace." or "kernel."
24prefix. See kernel parameters for the options which starts
25with "kernel." prefix [2]_.
26
27.. [1] See :ref:`Documentation/admin-guide/bootconfig.rst <bootconfig>`
28.. [2] See :ref:`Documentation/admin-guide/kernel-parameters.rst <kernelparameters>`
29
30Ftrace Global Options
31---------------------
32
33Ftrace global options have "kernel." prefix in boot config, which means
34these options are passed as a part of kernel legacy command line.
35
36kernel.tp_printk
37   Output trace-event data on printk buffer too.
38
39kernel.dump_on_oops [= MODE]
40   Dump ftrace on Oops. If MODE = 1 or omitted, dump trace buffer
41   on all CPUs. If MODE = 2, dump a buffer on a CPU which kicks Oops.
42
43kernel.traceoff_on_warning
44   Stop tracing if WARN_ON() occurs.
45
46kernel.fgraph_max_depth = MAX_DEPTH
47   Set MAX_DEPTH to maximum depth of fgraph tracer.
48
49kernel.fgraph_filters = FILTER[, FILTER2...]
50   Add fgraph tracing function filters.
51
52kernel.fgraph_notraces = FILTER[, FILTER2...]
53   Add fgraph non-tracing function filters.
54
55
56Ftrace Per-instance Options
57---------------------------
58
59These options can be used for each instance including global ftrace node.
60
61ftrace.[instance.INSTANCE.]options = OPT1[, OPT2[...]]
62   Enable given ftrace options.
63
64ftrace.[instance.INSTANCE.]tracing_on = 0|1
65   Enable/Disable tracing on this instance when starting boot-time tracing.
66   (you can enable it by the "traceon" event trigger action)
67
68ftrace.[instance.INSTANCE.]trace_clock = CLOCK
69   Set given CLOCK to ftrace's trace_clock.
70
71ftrace.[instance.INSTANCE.]buffer_size = SIZE
72   Configure ftrace buffer size to SIZE. You can use "KB" or "MB"
73   for that SIZE.
74
75ftrace.[instance.INSTANCE.]alloc_snapshot
76   Allocate snapshot buffer.
77
78ftrace.[instance.INSTANCE.]cpumask = CPUMASK
79   Set CPUMASK as trace cpu-mask.
80
81ftrace.[instance.INSTANCE.]events = EVENT[, EVENT2[...]]
82   Enable given events on boot. You can use a wild card in EVENT.
83
84ftrace.[instance.INSTANCE.]tracer = TRACER
85   Set TRACER to current tracer on boot. (e.g. function)
86
87ftrace.[instance.INSTANCE.]ftrace.filters
88   This will take an array of tracing function filter rules.
89
90ftrace.[instance.INSTANCE.]ftrace.notraces
91   This will take an array of NON-tracing function filter rules.
92
93
94Ftrace Per-Event Options
95------------------------
96
97These options are setting per-event options.
98
99ftrace.[instance.INSTANCE.]event.GROUP.EVENT.enable
100   Enable GROUP:EVENT tracing.
101
102ftrace.[instance.INSTANCE.]event.GROUP.enable
103   Enable all event tracing within GROUP.
104
105ftrace.[instance.INSTANCE.]event.enable
106   Enable all event tracing.
107
108ftrace.[instance.INSTANCE.]event.GROUP.EVENT.filter = FILTER
109   Set FILTER rule to the GROUP:EVENT.
110
111ftrace.[instance.INSTANCE.]event.GROUP.EVENT.actions = ACTION[, ACTION2[...]]
112   Set ACTIONs to the GROUP:EVENT.
113
114ftrace.[instance.INSTANCE.]event.kprobes.EVENT.probes = PROBE[, PROBE2[...]]
115   Defines new kprobe event based on PROBEs. It is able to define
116   multiple probes on one event, but those must have same type of
117   arguments. This option is available only for the event which
118   group name is "kprobes".
119
120ftrace.[instance.INSTANCE.]event.synthetic.EVENT.fields = FIELD[, FIELD2[...]]
121   Defines new synthetic event with FIELDs. Each field should be
122   "type varname".
123
124Note that kprobe and synthetic event definitions can be written under
125instance node, but those are also visible from other instances. So please
126take care for event name conflict.
127
128Ftrace Histogram Options
129------------------------
130
131Since it is too long to write a histogram action as a string for per-event
132action option, there are tree-style options under per-event 'hist' subkey
133for the histogram actions. For the detail of the each parameter,
134please read the event histogram document [3]_.
135
136.. [3] See :ref:`Documentation/trace/histogram.rst <histogram>`
137
138ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]keys = KEY1[, KEY2[...]]
139  Set histogram key parameters. (Mandatory)
140  The 'N' is a digit string for the multiple histogram. You can omit it
141  if there is one histogram on the event.
142
143ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]values = VAL1[, VAL2[...]]
144  Set histogram value parameters.
145
146ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]sort = SORT1[, SORT2[...]]
147  Set histogram sort parameter options.
148
149ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]size = NR_ENTRIES
150  Set histogram size (number of entries).
151
152ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]name = NAME
153  Set histogram name.
154
155ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]var.VARIABLE = EXPR
156  Define a new VARIABLE by EXPR expression.
157
158ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]<pause|continue|clear>
159  Set histogram control parameter. You can set one of them.
160
161ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]onmatch.[M.]event = GROUP.EVENT
162  Set histogram 'onmatch' handler matching event parameter.
163  The 'M' is a digit string for the multiple 'onmatch' handler. You can omit it
164  if there is one 'onmatch' handler on this histogram.
165
166ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]onmatch.[M.]trace = EVENT[, ARG1[...]]
167  Set histogram 'trace' action for 'onmatch'.
168  EVENT must be a synthetic event name, and ARG1... are parameters
169  for that event. Mandatory if 'onmatch.event' option is set.
170
171ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]onmax.[M.]var = VAR
172  Set histogram 'onmax' handler variable parameter.
173
174ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]onchange.[M.]var = VAR
175  Set histogram 'onchange' handler variable parameter.
176
177ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]<onmax|onchange>.[M.]save = ARG1[, ARG2[...]]
178  Set histogram 'save' action parameters for 'onmax' or 'onchange' handler.
179  This option or below 'snapshot' option is mandatory if 'onmax.var' or
180  'onchange.var' option is set.
181
182ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]<onmax|onchange>.[M.]snapshot
183  Set histogram 'snapshot' action for 'onmax' or 'onchange' handler.
184  This option or above 'save' option is mandatory if 'onmax.var' or
185  'onchange.var' option is set.
186
187ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.filter = FILTER_EXPR
188  Set histogram filter expression. You don't need 'if' in the FILTER_EXPR.
189
190Note that this 'hist' option can conflict with the per-event 'actions'
191option if the 'actions' option has a histogram action.
192
193
194When to Start
195=============
196
197All boot-time tracing options starting with ``ftrace`` will be enabled at the
198end of core_initcall. This means you can trace the events from postcore_initcall.
199Most of the subsystems and architecture dependent drivers will be initialized
200after that (arch_initcall or subsys_initcall). Thus, you can trace those with
201boot-time tracing.
202If you want to trace events before core_initcall, you can use the options
203starting with ``kernel``. Some of them will be enabled eariler than the initcall
204processing (for example,. ``kernel.ftrace=function`` and ``kernel.trace_event``
205will start before the initcall.)
206
207
208Examples
209========
210
211For example, to add filter and actions for each event, define kprobe
212events, and synthetic events with histogram, write a boot config like
213below::
214
215  ftrace.event {
216        task.task_newtask {
217                filter = "pid < 128"
218                enable
219        }
220        kprobes.vfs_read {
221                probes = "vfs_read $arg1 $arg2"
222                filter = "common_pid < 200"
223                enable
224        }
225        synthetic.initcall_latency {
226                fields = "unsigned long func", "u64 lat"
227                hist {
228                        keys = func.sym, lat
229                        values = lat
230                        sort = lat
231                }
232        }
233        initcall.initcall_start.hist {
234                keys = func
235                var.ts0 = common_timestamp.usecs
236        }
237        initcall.initcall_finish.hist {
238                keys = func
239                var.lat = common_timestamp.usecs - $ts0
240                onmatch {
241                        event = initcall.initcall_start
242                        trace = initcall_latency, func, $lat
243                }
244        }
245  }
246
247Also, boot-time tracing supports "instance" node, which allows us to run
248several tracers for different purpose at once. For example, one tracer
249is for tracing functions starting with "user\_", and others tracing
250"kernel\_" functions, you can write boot config as below::
251
252  ftrace.instance {
253        foo {
254                tracer = "function"
255                ftrace.filters = "user_*"
256        }
257        bar {
258                tracer = "function"
259                ftrace.filters = "kernel_*"
260        }
261  }
262
263The instance node also accepts event nodes so that each instance
264can customize its event tracing.
265
266With the trigger action and kprobes, you can trace function-graph while
267a function is called. For example, this will trace all function calls in
268the pci_proc_init()::
269
270  ftrace {
271        tracing_on = 0
272        tracer = function_graph
273        event.kprobes {
274                start_event {
275                        probes = "pci_proc_init"
276                        actions = "traceon"
277                }
278                end_event {
279                        probes = "pci_proc_init%return"
280                        actions = "traceoff"
281                }
282        }
283  }
284
285
286This boot-time tracing also supports ftrace kernel parameters via boot
287config.
288For example, following kernel parameters::
289
290 trace_options=sym-addr trace_event=initcall:* tp_printk trace_buf_size=1M ftrace=function ftrace_filter="vfs*"
291
292This can be written in boot config like below::
293
294  kernel {
295        trace_options = sym-addr
296        trace_event = "initcall:*"
297        tp_printk
298        trace_buf_size = 1M
299        ftrace = function
300        ftrace_filter = "vfs*"
301  }
302
303Note that parameters start with "kernel" prefix instead of "ftrace".
304