1============= 2Event Tracing 3============= 4 5:Author: Theodore Ts'o 6:Updated: Li Zefan and Tom Zanussi 7 81. Introduction 9=============== 10 11Tracepoints (see Documentation/trace/tracepoints.rst) can be used 12without creating custom kernel modules to register probe functions 13using the event tracing infrastructure. 14 15Not all tracepoints can be traced using the event tracing system; 16the kernel developer must provide code snippets which define how the 17tracing information is saved into the tracing buffer, and how the 18tracing information should be printed. 19 202. Using Event Tracing 21====================== 22 232.1 Via the 'set_event' interface 24--------------------------------- 25 26The events which are available for tracing can be found in the file 27/sys/kernel/debug/tracing/available_events. 28 29To enable a particular event, such as 'sched_wakeup', simply echo it 30to /sys/kernel/debug/tracing/set_event. For example:: 31 32 # echo sched_wakeup >> /sys/kernel/debug/tracing/set_event 33 34.. Note:: '>>' is necessary, otherwise it will firstly disable all the events. 35 36To disable an event, echo the event name to the set_event file prefixed 37with an exclamation point:: 38 39 # echo '!sched_wakeup' >> /sys/kernel/debug/tracing/set_event 40 41To disable all events, echo an empty line to the set_event file:: 42 43 # echo > /sys/kernel/debug/tracing/set_event 44 45To enable all events, echo ``*:*`` or ``*:`` to the set_event file:: 46 47 # echo *:* > /sys/kernel/debug/tracing/set_event 48 49The events are organized into subsystems, such as ext4, irq, sched, 50etc., and a full event name looks like this: <subsystem>:<event>. The 51subsystem name is optional, but it is displayed in the available_events 52file. All of the events in a subsystem can be specified via the syntax 53``<subsystem>:*``; for example, to enable all irq events, you can use the 54command:: 55 56 # echo 'irq:*' > /sys/kernel/debug/tracing/set_event 57 582.2 Via the 'enable' toggle 59--------------------------- 60 61The events available are also listed in /sys/kernel/debug/tracing/events/ hierarchy 62of directories. 63 64To enable event 'sched_wakeup':: 65 66 # echo 1 > /sys/kernel/debug/tracing/events/sched/sched_wakeup/enable 67 68To disable it:: 69 70 # echo 0 > /sys/kernel/debug/tracing/events/sched/sched_wakeup/enable 71 72To enable all events in sched subsystem:: 73 74 # echo 1 > /sys/kernel/debug/tracing/events/sched/enable 75 76To enable all events:: 77 78 # echo 1 > /sys/kernel/debug/tracing/events/enable 79 80When reading one of these enable files, there are four results: 81 82 - 0 - all events this file affects are disabled 83 - 1 - all events this file affects are enabled 84 - X - there is a mixture of events enabled and disabled 85 - ? - this file does not affect any event 86 872.3 Boot option 88--------------- 89 90In order to facilitate early boot debugging, use boot option:: 91 92 trace_event=[event-list] 93 94event-list is a comma separated list of events. See section 2.1 for event 95format. 96 973. Defining an event-enabled tracepoint 98======================================= 99 100See The example provided in samples/trace_events 101 1024. Event formats 103================ 104 105Each trace event has a 'format' file associated with it that contains 106a description of each field in a logged event. This information can 107be used to parse the binary trace stream, and is also the place to 108find the field names that can be used in event filters (see section 5). 109 110It also displays the format string that will be used to print the 111event in text mode, along with the event name and ID used for 112profiling. 113 114Every event has a set of ``common`` fields associated with it; these are 115the fields prefixed with ``common_``. The other fields vary between 116events and correspond to the fields defined in the TRACE_EVENT 117definition for that event. 118 119Each field in the format has the form:: 120 121 field:field-type field-name; offset:N; size:N; 122 123where offset is the offset of the field in the trace record and size 124is the size of the data item, in bytes. 125 126For example, here's the information displayed for the 'sched_wakeup' 127event:: 128 129 # cat /sys/kernel/debug/tracing/events/sched/sched_wakeup/format 130 131 name: sched_wakeup 132 ID: 60 133 format: 134 field:unsigned short common_type; offset:0; size:2; 135 field:unsigned char common_flags; offset:2; size:1; 136 field:unsigned char common_preempt_count; offset:3; size:1; 137 field:int common_pid; offset:4; size:4; 138 field:int common_tgid; offset:8; size:4; 139 140 field:char comm[TASK_COMM_LEN]; offset:12; size:16; 141 field:pid_t pid; offset:28; size:4; 142 field:int prio; offset:32; size:4; 143 field:int success; offset:36; size:4; 144 field:int cpu; offset:40; size:4; 145 146 print fmt: "task %s:%d [%d] success=%d [%03d]", REC->comm, REC->pid, 147 REC->prio, REC->success, REC->cpu 148 149This event contains 10 fields, the first 5 common and the remaining 5 150event-specific. All the fields for this event are numeric, except for 151'comm' which is a string, a distinction important for event filtering. 152 1535. Event filtering 154================== 155 156Trace events can be filtered in the kernel by associating boolean 157'filter expressions' with them. As soon as an event is logged into 158the trace buffer, its fields are checked against the filter expression 159associated with that event type. An event with field values that 160'match' the filter will appear in the trace output, and an event whose 161values don't match will be discarded. An event with no filter 162associated with it matches everything, and is the default when no 163filter has been set for an event. 164 1655.1 Expression syntax 166--------------------- 167 168A filter expression consists of one or more 'predicates' that can be 169combined using the logical operators '&&' and '||'. A predicate is 170simply a clause that compares the value of a field contained within a 171logged event with a constant value and returns either 0 or 1 depending 172on whether the field value matched (1) or didn't match (0):: 173 174 field-name relational-operator value 175 176Parentheses can be used to provide arbitrary logical groupings and 177double-quotes can be used to prevent the shell from interpreting 178operators as shell metacharacters. 179 180The field-names available for use in filters can be found in the 181'format' files for trace events (see section 4). 182 183The relational-operators depend on the type of the field being tested: 184 185The operators available for numeric fields are: 186 187==, !=, <, <=, >, >=, & 188 189And for string fields they are: 190 191==, !=, ~ 192 193The glob (~) accepts a wild card character (\*,?) and character classes 194([). For example:: 195 196 prev_comm ~ "*sh" 197 prev_comm ~ "sh*" 198 prev_comm ~ "*sh*" 199 prev_comm ~ "ba*sh" 200 2015.2 Setting filters 202------------------- 203 204A filter for an individual event is set by writing a filter expression 205to the 'filter' file for the given event. 206 207For example:: 208 209 # cd /sys/kernel/debug/tracing/events/sched/sched_wakeup 210 # echo "common_preempt_count > 4" > filter 211 212A slightly more involved example:: 213 214 # cd /sys/kernel/debug/tracing/events/signal/signal_generate 215 # echo "((sig >= 10 && sig < 15) || sig == 17) && comm != bash" > filter 216 217If there is an error in the expression, you'll get an 'Invalid 218argument' error when setting it, and the erroneous string along with 219an error message can be seen by looking at the filter e.g.:: 220 221 # cd /sys/kernel/debug/tracing/events/signal/signal_generate 222 # echo "((sig >= 10 && sig < 15) || dsig == 17) && comm != bash" > filter 223 -bash: echo: write error: Invalid argument 224 # cat filter 225 ((sig >= 10 && sig < 15) || dsig == 17) && comm != bash 226 ^ 227 parse_error: Field not found 228 229Currently the caret ('^') for an error always appears at the beginning of 230the filter string; the error message should still be useful though 231even without more accurate position info. 232 2335.3 Clearing filters 234-------------------- 235 236To clear the filter for an event, write a '0' to the event's filter 237file. 238 239To clear the filters for all events in a subsystem, write a '0' to the 240subsystem's filter file. 241 2425.3 Subsystem filters 243--------------------- 244 245For convenience, filters for every event in a subsystem can be set or 246cleared as a group by writing a filter expression into the filter file 247at the root of the subsystem. Note however, that if a filter for any 248event within the subsystem lacks a field specified in the subsystem 249filter, or if the filter can't be applied for any other reason, the 250filter for that event will retain its previous setting. This can 251result in an unintended mixture of filters which could lead to 252confusing (to the user who might think different filters are in 253effect) trace output. Only filters that reference just the common 254fields can be guaranteed to propagate successfully to all events. 255 256Here are a few subsystem filter examples that also illustrate the 257above points: 258 259Clear the filters on all events in the sched subsystem:: 260 261 # cd /sys/kernel/debug/tracing/events/sched 262 # echo 0 > filter 263 # cat sched_switch/filter 264 none 265 # cat sched_wakeup/filter 266 none 267 268Set a filter using only common fields for all events in the sched 269subsystem (all events end up with the same filter):: 270 271 # cd /sys/kernel/debug/tracing/events/sched 272 # echo common_pid == 0 > filter 273 # cat sched_switch/filter 274 common_pid == 0 275 # cat sched_wakeup/filter 276 common_pid == 0 277 278Attempt to set a filter using a non-common field for all events in the 279sched subsystem (all events but those that have a prev_pid field retain 280their old filters):: 281 282 # cd /sys/kernel/debug/tracing/events/sched 283 # echo prev_pid == 0 > filter 284 # cat sched_switch/filter 285 prev_pid == 0 286 # cat sched_wakeup/filter 287 common_pid == 0 288 2895.4 PID filtering 290----------------- 291 292The set_event_pid file in the same directory as the top events directory 293exists, will filter all events from tracing any task that does not have the 294PID listed in the set_event_pid file. 295:: 296 297 # cd /sys/kernel/debug/tracing 298 # echo $$ > set_event_pid 299 # echo 1 > events/enable 300 301Will only trace events for the current task. 302 303To add more PIDs without losing the PIDs already included, use '>>'. 304:: 305 306 # echo 123 244 1 >> set_event_pid 307 308 3096. Event triggers 310================= 311 312Trace events can be made to conditionally invoke trigger 'commands' 313which can take various forms and are described in detail below; 314examples would be enabling or disabling other trace events or invoking 315a stack trace whenever the trace event is hit. Whenever a trace event 316with attached triggers is invoked, the set of trigger commands 317associated with that event is invoked. Any given trigger can 318additionally have an event filter of the same form as described in 319section 5 (Event filtering) associated with it - the command will only 320be invoked if the event being invoked passes the associated filter. 321If no filter is associated with the trigger, it always passes. 322 323Triggers are added to and removed from a particular event by writing 324trigger expressions to the 'trigger' file for the given event. 325 326A given event can have any number of triggers associated with it, 327subject to any restrictions that individual commands may have in that 328regard. 329 330Event triggers are implemented on top of "soft" mode, which means that 331whenever a trace event has one or more triggers associated with it, 332the event is activated even if it isn't actually enabled, but is 333disabled in a "soft" mode. That is, the tracepoint will be called, 334but just will not be traced, unless of course it's actually enabled. 335This scheme allows triggers to be invoked even for events that aren't 336enabled, and also allows the current event filter implementation to be 337used for conditionally invoking triggers. 338 339The syntax for event triggers is roughly based on the syntax for 340set_ftrace_filter 'ftrace filter commands' (see the 'Filter commands' 341section of Documentation/trace/ftrace.rst), but there are major 342differences and the implementation isn't currently tied to it in any 343way, so beware about making generalizations between the two. 344 345Note: Writing into trace_marker (See Documentation/trace/ftrace.rst) 346 can also enable triggers that are written into 347 /sys/kernel/tracing/events/ftrace/print/trigger 348 3496.1 Expression syntax 350--------------------- 351 352Triggers are added by echoing the command to the 'trigger' file:: 353 354 # echo 'command[:count] [if filter]' > trigger 355 356Triggers are removed by echoing the same command but starting with '!' 357to the 'trigger' file:: 358 359 # echo '!command[:count] [if filter]' > trigger 360 361The [if filter] part isn't used in matching commands when removing, so 362leaving that off in a '!' command will accomplish the same thing as 363having it in. 364 365The filter syntax is the same as that described in the 'Event 366filtering' section above. 367 368For ease of use, writing to the trigger file using '>' currently just 369adds or removes a single trigger and there's no explicit '>>' support 370('>' actually behaves like '>>') or truncation support to remove all 371triggers (you have to use '!' for each one added.) 372 3736.2 Supported trigger commands 374------------------------------ 375 376The following commands are supported: 377 378- enable_event/disable_event 379 380 These commands can enable or disable another trace event whenever 381 the triggering event is hit. When these commands are registered, 382 the other trace event is activated, but disabled in a "soft" mode. 383 That is, the tracepoint will be called, but just will not be traced. 384 The event tracepoint stays in this mode as long as there's a trigger 385 in effect that can trigger it. 386 387 For example, the following trigger causes kmalloc events to be 388 traced when a read system call is entered, and the :1 at the end 389 specifies that this enablement happens only once:: 390 391 # echo 'enable_event:kmem:kmalloc:1' > \ 392 /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/trigger 393 394 The following trigger causes kmalloc events to stop being traced 395 when a read system call exits. This disablement happens on every 396 read system call exit:: 397 398 # echo 'disable_event:kmem:kmalloc' > \ 399 /sys/kernel/debug/tracing/events/syscalls/sys_exit_read/trigger 400 401 The format is:: 402 403 enable_event:<system>:<event>[:count] 404 disable_event:<system>:<event>[:count] 405 406 To remove the above commands:: 407 408 # echo '!enable_event:kmem:kmalloc:1' > \ 409 /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/trigger 410 411 # echo '!disable_event:kmem:kmalloc' > \ 412 /sys/kernel/debug/tracing/events/syscalls/sys_exit_read/trigger 413 414 Note that there can be any number of enable/disable_event triggers 415 per triggering event, but there can only be one trigger per 416 triggered event. e.g. sys_enter_read can have triggers enabling both 417 kmem:kmalloc and sched:sched_switch, but can't have two kmem:kmalloc 418 versions such as kmem:kmalloc and kmem:kmalloc:1 or 'kmem:kmalloc if 419 bytes_req == 256' and 'kmem:kmalloc if bytes_alloc == 256' (they 420 could be combined into a single filter on kmem:kmalloc though). 421 422- stacktrace 423 424 This command dumps a stacktrace in the trace buffer whenever the 425 triggering event occurs. 426 427 For example, the following trigger dumps a stacktrace every time the 428 kmalloc tracepoint is hit:: 429 430 # echo 'stacktrace' > \ 431 /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger 432 433 The following trigger dumps a stacktrace the first 5 times a kmalloc 434 request happens with a size >= 64K:: 435 436 # echo 'stacktrace:5 if bytes_req >= 65536' > \ 437 /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger 438 439 The format is:: 440 441 stacktrace[:count] 442 443 To remove the above commands:: 444 445 # echo '!stacktrace' > \ 446 /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger 447 448 # echo '!stacktrace:5 if bytes_req >= 65536' > \ 449 /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger 450 451 The latter can also be removed more simply by the following (without 452 the filter):: 453 454 # echo '!stacktrace:5' > \ 455 /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger 456 457 Note that there can be only one stacktrace trigger per triggering 458 event. 459 460- snapshot 461 462 This command causes a snapshot to be triggered whenever the 463 triggering event occurs. 464 465 The following command creates a snapshot every time a block request 466 queue is unplugged with a depth > 1. If you were tracing a set of 467 events or functions at the time, the snapshot trace buffer would 468 capture those events when the trigger event occurred:: 469 470 # echo 'snapshot if nr_rq > 1' > \ 471 /sys/kernel/debug/tracing/events/block/block_unplug/trigger 472 473 To only snapshot once:: 474 475 # echo 'snapshot:1 if nr_rq > 1' > \ 476 /sys/kernel/debug/tracing/events/block/block_unplug/trigger 477 478 To remove the above commands:: 479 480 # echo '!snapshot if nr_rq > 1' > \ 481 /sys/kernel/debug/tracing/events/block/block_unplug/trigger 482 483 # echo '!snapshot:1 if nr_rq > 1' > \ 484 /sys/kernel/debug/tracing/events/block/block_unplug/trigger 485 486 Note that there can be only one snapshot trigger per triggering 487 event. 488 489- traceon/traceoff 490 491 These commands turn tracing on and off when the specified events are 492 hit. The parameter determines how many times the tracing system is 493 turned on and off. If unspecified, there is no limit. 494 495 The following command turns tracing off the first time a block 496 request queue is unplugged with a depth > 1. If you were tracing a 497 set of events or functions at the time, you could then examine the 498 trace buffer to see the sequence of events that led up to the 499 trigger event:: 500 501 # echo 'traceoff:1 if nr_rq > 1' > \ 502 /sys/kernel/debug/tracing/events/block/block_unplug/trigger 503 504 To always disable tracing when nr_rq > 1:: 505 506 # echo 'traceoff if nr_rq > 1' > \ 507 /sys/kernel/debug/tracing/events/block/block_unplug/trigger 508 509 To remove the above commands:: 510 511 # echo '!traceoff:1 if nr_rq > 1' > \ 512 /sys/kernel/debug/tracing/events/block/block_unplug/trigger 513 514 # echo '!traceoff if nr_rq > 1' > \ 515 /sys/kernel/debug/tracing/events/block/block_unplug/trigger 516 517 Note that there can be only one traceon or traceoff trigger per 518 triggering event. 519 520- hist 521 522 This command aggregates event hits into a hash table keyed on one or 523 more trace event format fields (or stacktrace) and a set of running 524 totals derived from one or more trace event format fields and/or 525 event counts (hitcount). 526 527 See Documentation/trace/histogram.rst for details and examples. 528