1=============
2Event Tracing
3=============
4
5:Author: Theodore Ts'o
6:Updated: Li Zefan and Tom Zanussi
7
81. Introduction
9===============
10
11Tracepoints (see Documentation/trace/tracepoints.rst) can be used
12without creating custom kernel modules to register probe functions
13using the event tracing infrastructure.
14
15Not all tracepoints can be traced using the event tracing system;
16the kernel developer must provide code snippets which define how the
17tracing information is saved into the tracing buffer, and how the
18tracing information should be printed.
19
202. Using Event Tracing
21======================
22
232.1 Via the 'set_event' interface
24---------------------------------
25
26The events which are available for tracing can be found in the file
27/sys/kernel/debug/tracing/available_events.
28
29To enable a particular event, such as 'sched_wakeup', simply echo it
30to /sys/kernel/debug/tracing/set_event. For example::
31
32	# echo sched_wakeup >> /sys/kernel/debug/tracing/set_event
33
34.. Note:: '>>' is necessary, otherwise it will firstly disable all the events.
35
36To disable an event, echo the event name to the set_event file prefixed
37with an exclamation point::
38
39	# echo '!sched_wakeup' >> /sys/kernel/debug/tracing/set_event
40
41To disable all events, echo an empty line to the set_event file::
42
43	# echo > /sys/kernel/debug/tracing/set_event
44
45To enable all events, echo ``*:*`` or ``*:`` to the set_event file::
46
47	# echo *:* > /sys/kernel/debug/tracing/set_event
48
49The events are organized into subsystems, such as ext4, irq, sched,
50etc., and a full event name looks like this: <subsystem>:<event>.  The
51subsystem name is optional, but it is displayed in the available_events
52file.  All of the events in a subsystem can be specified via the syntax
53``<subsystem>:*``; for example, to enable all irq events, you can use the
54command::
55
56	# echo 'irq:*' > /sys/kernel/debug/tracing/set_event
57
582.2 Via the 'enable' toggle
59---------------------------
60
61The events available are also listed in /sys/kernel/debug/tracing/events/ hierarchy
62of directories.
63
64To enable event 'sched_wakeup'::
65
66	# echo 1 > /sys/kernel/debug/tracing/events/sched/sched_wakeup/enable
67
68To disable it::
69
70	# echo 0 > /sys/kernel/debug/tracing/events/sched/sched_wakeup/enable
71
72To enable all events in sched subsystem::
73
74	# echo 1 > /sys/kernel/debug/tracing/events/sched/enable
75
76To enable all events::
77
78	# echo 1 > /sys/kernel/debug/tracing/events/enable
79
80When reading one of these enable files, there are four results:
81
82 - 0 - all events this file affects are disabled
83 - 1 - all events this file affects are enabled
84 - X - there is a mixture of events enabled and disabled
85 - ? - this file does not affect any event
86
872.3 Boot option
88---------------
89
90In order to facilitate early boot debugging, use boot option::
91
92	trace_event=[event-list]
93
94event-list is a comma separated list of events. See section 2.1 for event
95format.
96
973. Defining an event-enabled tracepoint
98=======================================
99
100See The example provided in samples/trace_events
101
1024. Event formats
103================
104
105Each trace event has a 'format' file associated with it that contains
106a description of each field in a logged event.  This information can
107be used to parse the binary trace stream, and is also the place to
108find the field names that can be used in event filters (see section 5).
109
110It also displays the format string that will be used to print the
111event in text mode, along with the event name and ID used for
112profiling.
113
114Every event has a set of ``common`` fields associated with it; these are
115the fields prefixed with ``common_``.  The other fields vary between
116events and correspond to the fields defined in the TRACE_EVENT
117definition for that event.
118
119Each field in the format has the form::
120
121     field:field-type field-name; offset:N; size:N;
122
123where offset is the offset of the field in the trace record and size
124is the size of the data item, in bytes.
125
126For example, here's the information displayed for the 'sched_wakeup'
127event::
128
129	# cat /sys/kernel/debug/tracing/events/sched/sched_wakeup/format
130
131	name: sched_wakeup
132	ID: 60
133	format:
134		field:unsigned short common_type;	offset:0;	size:2;
135		field:unsigned char common_flags;	offset:2;	size:1;
136		field:unsigned char common_preempt_count;	offset:3;	size:1;
137		field:int common_pid;	offset:4;	size:4;
138		field:int common_tgid;	offset:8;	size:4;
139
140		field:char comm[TASK_COMM_LEN];	offset:12;	size:16;
141		field:pid_t pid;	offset:28;	size:4;
142		field:int prio;	offset:32;	size:4;
143		field:int success;	offset:36;	size:4;
144		field:int cpu;	offset:40;	size:4;
145
146	print fmt: "task %s:%d [%d] success=%d [%03d]", REC->comm, REC->pid,
147		   REC->prio, REC->success, REC->cpu
148
149This event contains 10 fields, the first 5 common and the remaining 5
150event-specific.  All the fields for this event are numeric, except for
151'comm' which is a string, a distinction important for event filtering.
152
1535. Event filtering
154==================
155
156Trace events can be filtered in the kernel by associating boolean
157'filter expressions' with them.  As soon as an event is logged into
158the trace buffer, its fields are checked against the filter expression
159associated with that event type.  An event with field values that
160'match' the filter will appear in the trace output, and an event whose
161values don't match will be discarded.  An event with no filter
162associated with it matches everything, and is the default when no
163filter has been set for an event.
164
1655.1 Expression syntax
166---------------------
167
168A filter expression consists of one or more 'predicates' that can be
169combined using the logical operators '&&' and '||'.  A predicate is
170simply a clause that compares the value of a field contained within a
171logged event with a constant value and returns either 0 or 1 depending
172on whether the field value matched (1) or didn't match (0)::
173
174	  field-name relational-operator value
175
176Parentheses can be used to provide arbitrary logical groupings and
177double-quotes can be used to prevent the shell from interpreting
178operators as shell metacharacters.
179
180The field-names available for use in filters can be found in the
181'format' files for trace events (see section 4).
182
183The relational-operators depend on the type of the field being tested:
184
185The operators available for numeric fields are:
186
187==, !=, <, <=, >, >=, &
188
189And for string fields they are:
190
191==, !=, ~
192
193The glob (~) accepts a wild card character (\*,?) and character classes
194([). For example::
195
196  prev_comm ~ "*sh"
197  prev_comm ~ "sh*"
198  prev_comm ~ "*sh*"
199  prev_comm ~ "ba*sh"
200
2015.2 Setting filters
202-------------------
203
204A filter for an individual event is set by writing a filter expression
205to the 'filter' file for the given event.
206
207For example::
208
209	# cd /sys/kernel/debug/tracing/events/sched/sched_wakeup
210	# echo "common_preempt_count > 4" > filter
211
212A slightly more involved example::
213
214	# cd /sys/kernel/debug/tracing/events/signal/signal_generate
215	# echo "((sig >= 10 && sig < 15) || sig == 17) && comm != bash" > filter
216
217If there is an error in the expression, you'll get an 'Invalid
218argument' error when setting it, and the erroneous string along with
219an error message can be seen by looking at the filter e.g.::
220
221	# cd /sys/kernel/debug/tracing/events/signal/signal_generate
222	# echo "((sig >= 10 && sig < 15) || dsig == 17) && comm != bash" > filter
223	-bash: echo: write error: Invalid argument
224	# cat filter
225	((sig >= 10 && sig < 15) || dsig == 17) && comm != bash
226	^
227	parse_error: Field not found
228
229Currently the caret ('^') for an error always appears at the beginning of
230the filter string; the error message should still be useful though
231even without more accurate position info.
232
2335.3 Clearing filters
234--------------------
235
236To clear the filter for an event, write a '0' to the event's filter
237file.
238
239To clear the filters for all events in a subsystem, write a '0' to the
240subsystem's filter file.
241
2425.3 Subsystem filters
243---------------------
244
245For convenience, filters for every event in a subsystem can be set or
246cleared as a group by writing a filter expression into the filter file
247at the root of the subsystem.  Note however, that if a filter for any
248event within the subsystem lacks a field specified in the subsystem
249filter, or if the filter can't be applied for any other reason, the
250filter for that event will retain its previous setting.  This can
251result in an unintended mixture of filters which could lead to
252confusing (to the user who might think different filters are in
253effect) trace output.  Only filters that reference just the common
254fields can be guaranteed to propagate successfully to all events.
255
256Here are a few subsystem filter examples that also illustrate the
257above points:
258
259Clear the filters on all events in the sched subsystem::
260
261	# cd /sys/kernel/debug/tracing/events/sched
262	# echo 0 > filter
263	# cat sched_switch/filter
264	none
265	# cat sched_wakeup/filter
266	none
267
268Set a filter using only common fields for all events in the sched
269subsystem (all events end up with the same filter)::
270
271	# cd /sys/kernel/debug/tracing/events/sched
272	# echo common_pid == 0 > filter
273	# cat sched_switch/filter
274	common_pid == 0
275	# cat sched_wakeup/filter
276	common_pid == 0
277
278Attempt to set a filter using a non-common field for all events in the
279sched subsystem (all events but those that have a prev_pid field retain
280their old filters)::
281
282	# cd /sys/kernel/debug/tracing/events/sched
283	# echo prev_pid == 0 > filter
284	# cat sched_switch/filter
285	prev_pid == 0
286	# cat sched_wakeup/filter
287	common_pid == 0
288
2895.4 PID filtering
290-----------------
291
292The set_event_pid file in the same directory as the top events directory
293exists, will filter all events from tracing any task that does not have the
294PID listed in the set_event_pid file.
295::
296
297	# cd /sys/kernel/debug/tracing
298	# echo $$ > set_event_pid
299	# echo 1 > events/enable
300
301Will only trace events for the current task.
302
303To add more PIDs without losing the PIDs already included, use '>>'.
304::
305
306	# echo 123 244 1 >> set_event_pid
307
308
3096. Event triggers
310=================
311
312Trace events can be made to conditionally invoke trigger 'commands'
313which can take various forms and are described in detail below;
314examples would be enabling or disabling other trace events or invoking
315a stack trace whenever the trace event is hit.  Whenever a trace event
316with attached triggers is invoked, the set of trigger commands
317associated with that event is invoked.  Any given trigger can
318additionally have an event filter of the same form as described in
319section 5 (Event filtering) associated with it - the command will only
320be invoked if the event being invoked passes the associated filter.
321If no filter is associated with the trigger, it always passes.
322
323Triggers are added to and removed from a particular event by writing
324trigger expressions to the 'trigger' file for the given event.
325
326A given event can have any number of triggers associated with it,
327subject to any restrictions that individual commands may have in that
328regard.
329
330Event triggers are implemented on top of "soft" mode, which means that
331whenever a trace event has one or more triggers associated with it,
332the event is activated even if it isn't actually enabled, but is
333disabled in a "soft" mode.  That is, the tracepoint will be called,
334but just will not be traced, unless of course it's actually enabled.
335This scheme allows triggers to be invoked even for events that aren't
336enabled, and also allows the current event filter implementation to be
337used for conditionally invoking triggers.
338
339The syntax for event triggers is roughly based on the syntax for
340set_ftrace_filter 'ftrace filter commands' (see the 'Filter commands'
341section of Documentation/trace/ftrace.rst), but there are major
342differences and the implementation isn't currently tied to it in any
343way, so beware about making generalizations between the two.
344
345Note: Writing into trace_marker (See Documentation/trace/ftrace.rst)
346     can also enable triggers that are written into
347     /sys/kernel/tracing/events/ftrace/print/trigger
348
3496.1 Expression syntax
350---------------------
351
352Triggers are added by echoing the command to the 'trigger' file::
353
354  # echo 'command[:count] [if filter]' > trigger
355
356Triggers are removed by echoing the same command but starting with '!'
357to the 'trigger' file::
358
359  # echo '!command[:count] [if filter]' > trigger
360
361The [if filter] part isn't used in matching commands when removing, so
362leaving that off in a '!' command will accomplish the same thing as
363having it in.
364
365The filter syntax is the same as that described in the 'Event
366filtering' section above.
367
368For ease of use, writing to the trigger file using '>' currently just
369adds or removes a single trigger and there's no explicit '>>' support
370('>' actually behaves like '>>') or truncation support to remove all
371triggers (you have to use '!' for each one added.)
372
3736.2 Supported trigger commands
374------------------------------
375
376The following commands are supported:
377
378- enable_event/disable_event
379
380  These commands can enable or disable another trace event whenever
381  the triggering event is hit.  When these commands are registered,
382  the other trace event is activated, but disabled in a "soft" mode.
383  That is, the tracepoint will be called, but just will not be traced.
384  The event tracepoint stays in this mode as long as there's a trigger
385  in effect that can trigger it.
386
387  For example, the following trigger causes kmalloc events to be
388  traced when a read system call is entered, and the :1 at the end
389  specifies that this enablement happens only once::
390
391	  # echo 'enable_event:kmem:kmalloc:1' > \
392	      /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/trigger
393
394  The following trigger causes kmalloc events to stop being traced
395  when a read system call exits.  This disablement happens on every
396  read system call exit::
397
398	  # echo 'disable_event:kmem:kmalloc' > \
399	      /sys/kernel/debug/tracing/events/syscalls/sys_exit_read/trigger
400
401  The format is::
402
403      enable_event:<system>:<event>[:count]
404      disable_event:<system>:<event>[:count]
405
406  To remove the above commands::
407
408	  # echo '!enable_event:kmem:kmalloc:1' > \
409	      /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/trigger
410
411	  # echo '!disable_event:kmem:kmalloc' > \
412	      /sys/kernel/debug/tracing/events/syscalls/sys_exit_read/trigger
413
414  Note that there can be any number of enable/disable_event triggers
415  per triggering event, but there can only be one trigger per
416  triggered event. e.g. sys_enter_read can have triggers enabling both
417  kmem:kmalloc and sched:sched_switch, but can't have two kmem:kmalloc
418  versions such as kmem:kmalloc and kmem:kmalloc:1 or 'kmem:kmalloc if
419  bytes_req == 256' and 'kmem:kmalloc if bytes_alloc == 256' (they
420  could be combined into a single filter on kmem:kmalloc though).
421
422- stacktrace
423
424  This command dumps a stacktrace in the trace buffer whenever the
425  triggering event occurs.
426
427  For example, the following trigger dumps a stacktrace every time the
428  kmalloc tracepoint is hit::
429
430	  # echo 'stacktrace' > \
431		/sys/kernel/debug/tracing/events/kmem/kmalloc/trigger
432
433  The following trigger dumps a stacktrace the first 5 times a kmalloc
434  request happens with a size >= 64K::
435
436	  # echo 'stacktrace:5 if bytes_req >= 65536' > \
437		/sys/kernel/debug/tracing/events/kmem/kmalloc/trigger
438
439  The format is::
440
441      stacktrace[:count]
442
443  To remove the above commands::
444
445	  # echo '!stacktrace' > \
446		/sys/kernel/debug/tracing/events/kmem/kmalloc/trigger
447
448	  # echo '!stacktrace:5 if bytes_req >= 65536' > \
449		/sys/kernel/debug/tracing/events/kmem/kmalloc/trigger
450
451  The latter can also be removed more simply by the following (without
452  the filter)::
453
454	  # echo '!stacktrace:5' > \
455		/sys/kernel/debug/tracing/events/kmem/kmalloc/trigger
456
457  Note that there can be only one stacktrace trigger per triggering
458  event.
459
460- snapshot
461
462  This command causes a snapshot to be triggered whenever the
463  triggering event occurs.
464
465  The following command creates a snapshot every time a block request
466  queue is unplugged with a depth > 1.  If you were tracing a set of
467  events or functions at the time, the snapshot trace buffer would
468  capture those events when the trigger event occurred::
469
470	  # echo 'snapshot if nr_rq > 1' > \
471		/sys/kernel/debug/tracing/events/block/block_unplug/trigger
472
473  To only snapshot once::
474
475	  # echo 'snapshot:1 if nr_rq > 1' > \
476		/sys/kernel/debug/tracing/events/block/block_unplug/trigger
477
478  To remove the above commands::
479
480	  # echo '!snapshot if nr_rq > 1' > \
481		/sys/kernel/debug/tracing/events/block/block_unplug/trigger
482
483	  # echo '!snapshot:1 if nr_rq > 1' > \
484		/sys/kernel/debug/tracing/events/block/block_unplug/trigger
485
486  Note that there can be only one snapshot trigger per triggering
487  event.
488
489- traceon/traceoff
490
491  These commands turn tracing on and off when the specified events are
492  hit. The parameter determines how many times the tracing system is
493  turned on and off. If unspecified, there is no limit.
494
495  The following command turns tracing off the first time a block
496  request queue is unplugged with a depth > 1.  If you were tracing a
497  set of events or functions at the time, you could then examine the
498  trace buffer to see the sequence of events that led up to the
499  trigger event::
500
501	  # echo 'traceoff:1 if nr_rq > 1' > \
502		/sys/kernel/debug/tracing/events/block/block_unplug/trigger
503
504  To always disable tracing when nr_rq  > 1::
505
506	  # echo 'traceoff if nr_rq > 1' > \
507		/sys/kernel/debug/tracing/events/block/block_unplug/trigger
508
509  To remove the above commands::
510
511	  # echo '!traceoff:1 if nr_rq > 1' > \
512		/sys/kernel/debug/tracing/events/block/block_unplug/trigger
513
514	  # echo '!traceoff if nr_rq > 1' > \
515		/sys/kernel/debug/tracing/events/block/block_unplug/trigger
516
517  Note that there can be only one traceon or traceoff trigger per
518  triggering event.
519
520- hist
521
522  This command aggregates event hits into a hash table keyed on one or
523  more trace event format fields (or stacktrace) and a set of running
524  totals derived from one or more trace event format fields and/or
525  event counts (hitcount).
526
527  See Documentation/trace/histogram.txt for details and examples.
528