1=============
2Event Tracing
3=============
4
5:Author: Theodore Ts'o
6:Updated: Li Zefan and Tom Zanussi
7
81. Introduction
9===============
10
11Tracepoints (see Documentation/trace/tracepoints.rst) can be used
12without creating custom kernel modules to register probe functions
13using the event tracing infrastructure.
14
15Not all tracepoints can be traced using the event tracing system;
16the kernel developer must provide code snippets which define how the
17tracing information is saved into the tracing buffer, and how the
18tracing information should be printed.
19
202. Using Event Tracing
21======================
22
232.1 Via the 'set_event' interface
24---------------------------------
25
26The events which are available for tracing can be found in the file
27/sys/kernel/tracing/available_events.
28
29To enable a particular event, such as 'sched_wakeup', simply echo it
30to /sys/kernel/tracing/set_event. For example::
31
32	# echo sched_wakeup >> /sys/kernel/tracing/set_event
33
34.. Note:: '>>' is necessary, otherwise it will firstly disable all the events.
35
36To disable an event, echo the event name to the set_event file prefixed
37with an exclamation point::
38
39	# echo '!sched_wakeup' >> /sys/kernel/tracing/set_event
40
41To disable all events, echo an empty line to the set_event file::
42
43	# echo > /sys/kernel/tracing/set_event
44
45To enable all events, echo ``*:*`` or ``*:`` to the set_event file::
46
47	# echo *:* > /sys/kernel/tracing/set_event
48
49The events are organized into subsystems, such as ext4, irq, sched,
50etc., and a full event name looks like this: <subsystem>:<event>.  The
51subsystem name is optional, but it is displayed in the available_events
52file.  All of the events in a subsystem can be specified via the syntax
53``<subsystem>:*``; for example, to enable all irq events, you can use the
54command::
55
56	# echo 'irq:*' > /sys/kernel/tracing/set_event
57
582.2 Via the 'enable' toggle
59---------------------------
60
61The events available are also listed in /sys/kernel/tracing/events/ hierarchy
62of directories.
63
64To enable event 'sched_wakeup'::
65
66	# echo 1 > /sys/kernel/tracing/events/sched/sched_wakeup/enable
67
68To disable it::
69
70	# echo 0 > /sys/kernel/tracing/events/sched/sched_wakeup/enable
71
72To enable all events in sched subsystem::
73
74	# echo 1 > /sys/kernel/tracing/events/sched/enable
75
76To enable all events::
77
78	# echo 1 > /sys/kernel/tracing/events/enable
79
80When reading one of these enable files, there are four results:
81
82 - 0 - all events this file affects are disabled
83 - 1 - all events this file affects are enabled
84 - X - there is a mixture of events enabled and disabled
85 - ? - this file does not affect any event
86
872.3 Boot option
88---------------
89
90In order to facilitate early boot debugging, use boot option::
91
92	trace_event=[event-list]
93
94event-list is a comma separated list of events. See section 2.1 for event
95format.
96
973. Defining an event-enabled tracepoint
98=======================================
99
100See The example provided in samples/trace_events
101
1024. Event formats
103================
104
105Each trace event has a 'format' file associated with it that contains
106a description of each field in a logged event.  This information can
107be used to parse the binary trace stream, and is also the place to
108find the field names that can be used in event filters (see section 5).
109
110It also displays the format string that will be used to print the
111event in text mode, along with the event name and ID used for
112profiling.
113
114Every event has a set of ``common`` fields associated with it; these are
115the fields prefixed with ``common_``.  The other fields vary between
116events and correspond to the fields defined in the TRACE_EVENT
117definition for that event.
118
119Each field in the format has the form::
120
121     field:field-type field-name; offset:N; size:N;
122
123where offset is the offset of the field in the trace record and size
124is the size of the data item, in bytes.
125
126For example, here's the information displayed for the 'sched_wakeup'
127event::
128
129	# cat /sys/kernel/tracing/events/sched/sched_wakeup/format
130
131	name: sched_wakeup
132	ID: 60
133	format:
134		field:unsigned short common_type;	offset:0;	size:2;
135		field:unsigned char common_flags;	offset:2;	size:1;
136		field:unsigned char common_preempt_count;	offset:3;	size:1;
137		field:int common_pid;	offset:4;	size:4;
138		field:int common_tgid;	offset:8;	size:4;
139
140		field:char comm[TASK_COMM_LEN];	offset:12;	size:16;
141		field:pid_t pid;	offset:28;	size:4;
142		field:int prio;	offset:32;	size:4;
143		field:int success;	offset:36;	size:4;
144		field:int cpu;	offset:40;	size:4;
145
146	print fmt: "task %s:%d [%d] success=%d [%03d]", REC->comm, REC->pid,
147		   REC->prio, REC->success, REC->cpu
148
149This event contains 10 fields, the first 5 common and the remaining 5
150event-specific.  All the fields for this event are numeric, except for
151'comm' which is a string, a distinction important for event filtering.
152
1535. Event filtering
154==================
155
156Trace events can be filtered in the kernel by associating boolean
157'filter expressions' with them.  As soon as an event is logged into
158the trace buffer, its fields are checked against the filter expression
159associated with that event type.  An event with field values that
160'match' the filter will appear in the trace output, and an event whose
161values don't match will be discarded.  An event with no filter
162associated with it matches everything, and is the default when no
163filter has been set for an event.
164
1655.1 Expression syntax
166---------------------
167
168A filter expression consists of one or more 'predicates' that can be
169combined using the logical operators '&&' and '||'.  A predicate is
170simply a clause that compares the value of a field contained within a
171logged event with a constant value and returns either 0 or 1 depending
172on whether the field value matched (1) or didn't match (0)::
173
174	  field-name relational-operator value
175
176Parentheses can be used to provide arbitrary logical groupings and
177double-quotes can be used to prevent the shell from interpreting
178operators as shell metacharacters.
179
180The field-names available for use in filters can be found in the
181'format' files for trace events (see section 4).
182
183The relational-operators depend on the type of the field being tested:
184
185The operators available for numeric fields are:
186
187==, !=, <, <=, >, >=, &
188
189And for string fields they are:
190
191==, !=, ~
192
193The glob (~) accepts a wild card character (\*,?) and character classes
194([). For example::
195
196  prev_comm ~ "*sh"
197  prev_comm ~ "sh*"
198  prev_comm ~ "*sh*"
199  prev_comm ~ "ba*sh"
200
201If the field is a pointer that points into user space (for example
202"filename" from sys_enter_openat), then you have to append ".ustring" to the
203field name::
204
205  filename.ustring ~ "password"
206
207As the kernel will have to know how to retrieve the memory that the pointer
208is at from user space.
209
210You can convert any long type to a function address and search by function name::
211
212  call_site.function == security_prepare_creds
213
214The above will filter when the field "call_site" falls on the address within
215"security_prepare_creds". That is, it will compare the value of "call_site" and
216the filter will return true if it is greater than or equal to the start of
217the function "security_prepare_creds" and less than the end of that function.
218
219The ".function" postfix can only be attached to values of size long, and can only
220be compared with "==" or "!=".
221
222Cpumask fields or scalar fields that encode a CPU number can be filtered using
223a user-provided cpumask in cpulist format. The format is as follows::
224
225  CPUS{$cpulist}
226
227Operators available to cpumask filtering are:
228
229& (intersection), ==, !=
230
231For example, this will filter events that have their .target_cpu field present
232in the given cpumask::
233
234  target_cpu & CPUS{17-42}
235
2365.2 Setting filters
237-------------------
238
239A filter for an individual event is set by writing a filter expression
240to the 'filter' file for the given event.
241
242For example::
243
244	# cd /sys/kernel/tracing/events/sched/sched_wakeup
245	# echo "common_preempt_count > 4" > filter
246
247A slightly more involved example::
248
249	# cd /sys/kernel/tracing/events/signal/signal_generate
250	# echo "((sig >= 10 && sig < 15) || sig == 17) && comm != bash" > filter
251
252If there is an error in the expression, you'll get an 'Invalid
253argument' error when setting it, and the erroneous string along with
254an error message can be seen by looking at the filter e.g.::
255
256	# cd /sys/kernel/tracing/events/signal/signal_generate
257	# echo "((sig >= 10 && sig < 15) || dsig == 17) && comm != bash" > filter
258	-bash: echo: write error: Invalid argument
259	# cat filter
260	((sig >= 10 && sig < 15) || dsig == 17) && comm != bash
261	^
262	parse_error: Field not found
263
264Currently the caret ('^') for an error always appears at the beginning of
265the filter string; the error message should still be useful though
266even without more accurate position info.
267
2685.2.1 Filter limitations
269------------------------
270
271If a filter is placed on a string pointer ``(char *)`` that does not point
272to a string on the ring buffer, but instead points to kernel or user space
273memory, then, for safety reasons, at most 1024 bytes of the content is
274copied onto a temporary buffer to do the compare. If the copy of the memory
275faults (the pointer points to memory that should not be accessed), then the
276string compare will be treated as not matching.
277
2785.3 Clearing filters
279--------------------
280
281To clear the filter for an event, write a '0' to the event's filter
282file.
283
284To clear the filters for all events in a subsystem, write a '0' to the
285subsystem's filter file.
286
2875.4 Subsystem filters
288---------------------
289
290For convenience, filters for every event in a subsystem can be set or
291cleared as a group by writing a filter expression into the filter file
292at the root of the subsystem.  Note however, that if a filter for any
293event within the subsystem lacks a field specified in the subsystem
294filter, or if the filter can't be applied for any other reason, the
295filter for that event will retain its previous setting.  This can
296result in an unintended mixture of filters which could lead to
297confusing (to the user who might think different filters are in
298effect) trace output.  Only filters that reference just the common
299fields can be guaranteed to propagate successfully to all events.
300
301Here are a few subsystem filter examples that also illustrate the
302above points:
303
304Clear the filters on all events in the sched subsystem::
305
306	# cd /sys/kernel/tracing/events/sched
307	# echo 0 > filter
308	# cat sched_switch/filter
309	none
310	# cat sched_wakeup/filter
311	none
312
313Set a filter using only common fields for all events in the sched
314subsystem (all events end up with the same filter)::
315
316	# cd /sys/kernel/tracing/events/sched
317	# echo common_pid == 0 > filter
318	# cat sched_switch/filter
319	common_pid == 0
320	# cat sched_wakeup/filter
321	common_pid == 0
322
323Attempt to set a filter using a non-common field for all events in the
324sched subsystem (all events but those that have a prev_pid field retain
325their old filters)::
326
327	# cd /sys/kernel/tracing/events/sched
328	# echo prev_pid == 0 > filter
329	# cat sched_switch/filter
330	prev_pid == 0
331	# cat sched_wakeup/filter
332	common_pid == 0
333
3345.5 PID filtering
335-----------------
336
337The set_event_pid file in the same directory as the top events directory
338exists, will filter all events from tracing any task that does not have the
339PID listed in the set_event_pid file.
340::
341
342	# cd /sys/kernel/tracing
343	# echo $$ > set_event_pid
344	# echo 1 > events/enable
345
346Will only trace events for the current task.
347
348To add more PIDs without losing the PIDs already included, use '>>'.
349::
350
351	# echo 123 244 1 >> set_event_pid
352
353
3546. Event triggers
355=================
356
357Trace events can be made to conditionally invoke trigger 'commands'
358which can take various forms and are described in detail below;
359examples would be enabling or disabling other trace events or invoking
360a stack trace whenever the trace event is hit.  Whenever a trace event
361with attached triggers is invoked, the set of trigger commands
362associated with that event is invoked.  Any given trigger can
363additionally have an event filter of the same form as described in
364section 5 (Event filtering) associated with it - the command will only
365be invoked if the event being invoked passes the associated filter.
366If no filter is associated with the trigger, it always passes.
367
368Triggers are added to and removed from a particular event by writing
369trigger expressions to the 'trigger' file for the given event.
370
371A given event can have any number of triggers associated with it,
372subject to any restrictions that individual commands may have in that
373regard.
374
375Event triggers are implemented on top of "soft" mode, which means that
376whenever a trace event has one or more triggers associated with it,
377the event is activated even if it isn't actually enabled, but is
378disabled in a "soft" mode.  That is, the tracepoint will be called,
379but just will not be traced, unless of course it's actually enabled.
380This scheme allows triggers to be invoked even for events that aren't
381enabled, and also allows the current event filter implementation to be
382used for conditionally invoking triggers.
383
384The syntax for event triggers is roughly based on the syntax for
385set_ftrace_filter 'ftrace filter commands' (see the 'Filter commands'
386section of Documentation/trace/ftrace.rst), but there are major
387differences and the implementation isn't currently tied to it in any
388way, so beware about making generalizations between the two.
389
390.. Note::
391     Writing into trace_marker (See Documentation/trace/ftrace.rst)
392     can also enable triggers that are written into
393     /sys/kernel/tracing/events/ftrace/print/trigger
394
3956.1 Expression syntax
396---------------------
397
398Triggers are added by echoing the command to the 'trigger' file::
399
400  # echo 'command[:count] [if filter]' > trigger
401
402Triggers are removed by echoing the same command but starting with '!'
403to the 'trigger' file::
404
405  # echo '!command[:count] [if filter]' > trigger
406
407The [if filter] part isn't used in matching commands when removing, so
408leaving that off in a '!' command will accomplish the same thing as
409having it in.
410
411The filter syntax is the same as that described in the 'Event
412filtering' section above.
413
414For ease of use, writing to the trigger file using '>' currently just
415adds or removes a single trigger and there's no explicit '>>' support
416('>' actually behaves like '>>') or truncation support to remove all
417triggers (you have to use '!' for each one added.)
418
4196.2 Supported trigger commands
420------------------------------
421
422The following commands are supported:
423
424- enable_event/disable_event
425
426  These commands can enable or disable another trace event whenever
427  the triggering event is hit.  When these commands are registered,
428  the other trace event is activated, but disabled in a "soft" mode.
429  That is, the tracepoint will be called, but just will not be traced.
430  The event tracepoint stays in this mode as long as there's a trigger
431  in effect that can trigger it.
432
433  For example, the following trigger causes kmalloc events to be
434  traced when a read system call is entered, and the :1 at the end
435  specifies that this enablement happens only once::
436
437	  # echo 'enable_event:kmem:kmalloc:1' > \
438	      /sys/kernel/tracing/events/syscalls/sys_enter_read/trigger
439
440  The following trigger causes kmalloc events to stop being traced
441  when a read system call exits.  This disablement happens on every
442  read system call exit::
443
444	  # echo 'disable_event:kmem:kmalloc' > \
445	      /sys/kernel/tracing/events/syscalls/sys_exit_read/trigger
446
447  The format is::
448
449      enable_event:<system>:<event>[:count]
450      disable_event:<system>:<event>[:count]
451
452  To remove the above commands::
453
454	  # echo '!enable_event:kmem:kmalloc:1' > \
455	      /sys/kernel/tracing/events/syscalls/sys_enter_read/trigger
456
457	  # echo '!disable_event:kmem:kmalloc' > \
458	      /sys/kernel/tracing/events/syscalls/sys_exit_read/trigger
459
460  Note that there can be any number of enable/disable_event triggers
461  per triggering event, but there can only be one trigger per
462  triggered event. e.g. sys_enter_read can have triggers enabling both
463  kmem:kmalloc and sched:sched_switch, but can't have two kmem:kmalloc
464  versions such as kmem:kmalloc and kmem:kmalloc:1 or 'kmem:kmalloc if
465  bytes_req == 256' and 'kmem:kmalloc if bytes_alloc == 256' (they
466  could be combined into a single filter on kmem:kmalloc though).
467
468- stacktrace
469
470  This command dumps a stacktrace in the trace buffer whenever the
471  triggering event occurs.
472
473  For example, the following trigger dumps a stacktrace every time the
474  kmalloc tracepoint is hit::
475
476	  # echo 'stacktrace' > \
477		/sys/kernel/tracing/events/kmem/kmalloc/trigger
478
479  The following trigger dumps a stacktrace the first 5 times a kmalloc
480  request happens with a size >= 64K::
481
482	  # echo 'stacktrace:5 if bytes_req >= 65536' > \
483		/sys/kernel/tracing/events/kmem/kmalloc/trigger
484
485  The format is::
486
487      stacktrace[:count]
488
489  To remove the above commands::
490
491	  # echo '!stacktrace' > \
492		/sys/kernel/tracing/events/kmem/kmalloc/trigger
493
494	  # echo '!stacktrace:5 if bytes_req >= 65536' > \
495		/sys/kernel/tracing/events/kmem/kmalloc/trigger
496
497  The latter can also be removed more simply by the following (without
498  the filter)::
499
500	  # echo '!stacktrace:5' > \
501		/sys/kernel/tracing/events/kmem/kmalloc/trigger
502
503  Note that there can be only one stacktrace trigger per triggering
504  event.
505
506- snapshot
507
508  This command causes a snapshot to be triggered whenever the
509  triggering event occurs.
510
511  The following command creates a snapshot every time a block request
512  queue is unplugged with a depth > 1.  If you were tracing a set of
513  events or functions at the time, the snapshot trace buffer would
514  capture those events when the trigger event occurred::
515
516	  # echo 'snapshot if nr_rq > 1' > \
517		/sys/kernel/tracing/events/block/block_unplug/trigger
518
519  To only snapshot once::
520
521	  # echo 'snapshot:1 if nr_rq > 1' > \
522		/sys/kernel/tracing/events/block/block_unplug/trigger
523
524  To remove the above commands::
525
526	  # echo '!snapshot if nr_rq > 1' > \
527		/sys/kernel/tracing/events/block/block_unplug/trigger
528
529	  # echo '!snapshot:1 if nr_rq > 1' > \
530		/sys/kernel/tracing/events/block/block_unplug/trigger
531
532  Note that there can be only one snapshot trigger per triggering
533  event.
534
535- traceon/traceoff
536
537  These commands turn tracing on and off when the specified events are
538  hit. The parameter determines how many times the tracing system is
539  turned on and off. If unspecified, there is no limit.
540
541  The following command turns tracing off the first time a block
542  request queue is unplugged with a depth > 1.  If you were tracing a
543  set of events or functions at the time, you could then examine the
544  trace buffer to see the sequence of events that led up to the
545  trigger event::
546
547	  # echo 'traceoff:1 if nr_rq > 1' > \
548		/sys/kernel/tracing/events/block/block_unplug/trigger
549
550  To always disable tracing when nr_rq  > 1::
551
552	  # echo 'traceoff if nr_rq > 1' > \
553		/sys/kernel/tracing/events/block/block_unplug/trigger
554
555  To remove the above commands::
556
557	  # echo '!traceoff:1 if nr_rq > 1' > \
558		/sys/kernel/tracing/events/block/block_unplug/trigger
559
560	  # echo '!traceoff if nr_rq > 1' > \
561		/sys/kernel/tracing/events/block/block_unplug/trigger
562
563  Note that there can be only one traceon or traceoff trigger per
564  triggering event.
565
566- hist
567
568  This command aggregates event hits into a hash table keyed on one or
569  more trace event format fields (or stacktrace) and a set of running
570  totals derived from one or more trace event format fields and/or
571  event counts (hitcount).
572
573  See Documentation/trace/histogram.rst for details and examples.
574
5757. In-kernel trace event API
576============================
577
578In most cases, the command-line interface to trace events is more than
579sufficient.  Sometimes, however, applications might find the need for
580more complex relationships than can be expressed through a simple
581series of linked command-line expressions, or putting together sets of
582commands may be simply too cumbersome.  An example might be an
583application that needs to 'listen' to the trace stream in order to
584maintain an in-kernel state machine detecting, for instance, when an
585illegal kernel state occurs in the scheduler.
586
587The trace event subsystem provides an in-kernel API allowing modules
588or other kernel code to generate user-defined 'synthetic' events at
589will, which can be used to either augment the existing trace stream
590and/or signal that a particular important state has occurred.
591
592A similar in-kernel API is also available for creating kprobe and
593kretprobe events.
594
595Both the synthetic event and k/ret/probe event APIs are built on top
596of a lower-level "dynevent_cmd" event command API, which is also
597available for more specialized applications, or as the basis of other
598higher-level trace event APIs.
599
600The API provided for these purposes is describe below and allows the
601following:
602
603  - dynamically creating synthetic event definitions
604  - dynamically creating kprobe and kretprobe event definitions
605  - tracing synthetic events from in-kernel code
606  - the low-level "dynevent_cmd" API
607
6087.1 Dyamically creating synthetic event definitions
609---------------------------------------------------
610
611There are a couple ways to create a new synthetic event from a kernel
612module or other kernel code.
613
614The first creates the event in one step, using synth_event_create().
615In this method, the name of the event to create and an array defining
616the fields is supplied to synth_event_create().  If successful, a
617synthetic event with that name and fields will exist following that
618call.  For example, to create a new "schedtest" synthetic event::
619
620  ret = synth_event_create("schedtest", sched_fields,
621                           ARRAY_SIZE(sched_fields), THIS_MODULE);
622
623The sched_fields param in this example points to an array of struct
624synth_field_desc, each of which describes an event field by type and
625name::
626
627  static struct synth_field_desc sched_fields[] = {
628        { .type = "pid_t",              .name = "next_pid_field" },
629        { .type = "char[16]",           .name = "next_comm_field" },
630        { .type = "u64",                .name = "ts_ns" },
631        { .type = "u64",                .name = "ts_ms" },
632        { .type = "unsigned int",       .name = "cpu" },
633        { .type = "char[64]",           .name = "my_string_field" },
634        { .type = "int",                .name = "my_int_field" },
635  };
636
637See synth_field_size() for available types.
638
639If field_name contains [n], the field is considered to be a static array.
640
641If field_names contains[] (no subscript), the field is considered to
642be a dynamic array, which will only take as much space in the event as
643is required to hold the array.
644
645Because space for an event is reserved before assigning field values
646to the event, using dynamic arrays implies that the piecewise
647in-kernel API described below can't be used with dynamic arrays.  The
648other non-piecewise in-kernel APIs can, however, be used with dynamic
649arrays.
650
651If the event is created from within a module, a pointer to the module
652must be passed to synth_event_create().  This will ensure that the
653trace buffer won't contain unreadable events when the module is
654removed.
655
656At this point, the event object is ready to be used for generating new
657events.
658
659In the second method, the event is created in several steps.  This
660allows events to be created dynamically and without the need to create
661and populate an array of fields beforehand.
662
663To use this method, an empty or partially empty synthetic event should
664first be created using synth_event_gen_cmd_start() or
665synth_event_gen_cmd_array_start().  For synth_event_gen_cmd_start(),
666the name of the event along with one or more pairs of args each pair
667representing a 'type field_name;' field specification should be
668supplied.  For synth_event_gen_cmd_array_start(), the name of the
669event along with an array of struct synth_field_desc should be
670supplied. Before calling synth_event_gen_cmd_start() or
671synth_event_gen_cmd_array_start(), the user should create and
672initialize a dynevent_cmd object using synth_event_cmd_init().
673
674For example, to create a new "schedtest" synthetic event with two
675fields::
676
677  struct dynevent_cmd cmd;
678  char *buf;
679
680  /* Create a buffer to hold the generated command */
681  buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL);
682
683  /* Before generating the command, initialize the cmd object */
684  synth_event_cmd_init(&cmd, buf, MAX_DYNEVENT_CMD_LEN);
685
686  ret = synth_event_gen_cmd_start(&cmd, "schedtest", THIS_MODULE,
687                                  "pid_t", "next_pid_field",
688                                  "u64", "ts_ns");
689
690Alternatively, using an array of struct synth_field_desc fields
691containing the same information::
692
693  ret = synth_event_gen_cmd_array_start(&cmd, "schedtest", THIS_MODULE,
694                                        fields, n_fields);
695
696Once the synthetic event object has been created, it can then be
697populated with more fields.  Fields are added one by one using
698synth_event_add_field(), supplying the dynevent_cmd object, a field
699type, and a field name.  For example, to add a new int field named
700"intfield", the following call should be made::
701
702  ret = synth_event_add_field(&cmd, "int", "intfield");
703
704See synth_field_size() for available types. If field_name contains [n]
705the field is considered to be an array.
706
707A group of fields can also be added all at once using an array of
708synth_field_desc with add_synth_fields().  For example, this would add
709just the first four sched_fields::
710
711  ret = synth_event_add_fields(&cmd, sched_fields, 4);
712
713If you already have a string of the form 'type field_name',
714synth_event_add_field_str() can be used to add it as-is; it will
715also automatically append a ';' to the string.
716
717Once all the fields have been added, the event should be finalized and
718registered by calling the synth_event_gen_cmd_end() function::
719
720  ret = synth_event_gen_cmd_end(&cmd);
721
722At this point, the event object is ready to be used for tracing new
723events.
724
7257.2 Tracing synthetic events from in-kernel code
726------------------------------------------------
727
728To trace a synthetic event, there are several options.  The first
729option is to trace the event in one call, using synth_event_trace()
730with a variable number of values, or synth_event_trace_array() with an
731array of values to be set.  A second option can be used to avoid the
732need for a pre-formed array of values or list of arguments, via
733synth_event_trace_start() and synth_event_trace_end() along with
734synth_event_add_next_val() or synth_event_add_val() to add the values
735piecewise.
736
7377.2.1 Tracing a synthetic event all at once
738-------------------------------------------
739
740To trace a synthetic event all at once, the synth_event_trace() or
741synth_event_trace_array() functions can be used.
742
743The synth_event_trace() function is passed the trace_event_file
744representing the synthetic event (which can be retrieved using
745trace_get_event_file() using the synthetic event name, "synthetic" as
746the system name, and the trace instance name (NULL if using the global
747trace array)), along with an variable number of u64 args, one for each
748synthetic event field, and the number of values being passed.
749
750So, to trace an event corresponding to the synthetic event definition
751above, code like the following could be used::
752
753  ret = synth_event_trace(create_synth_test, 7, /* number of values */
754                          444,             /* next_pid_field */
755                          (u64)"clackers", /* next_comm_field */
756                          1000000,         /* ts_ns */
757                          1000,            /* ts_ms */
758                          smp_processor_id(),/* cpu */
759                          (u64)"Thneed",   /* my_string_field */
760                          999);            /* my_int_field */
761
762All vals should be cast to u64, and string vals are just pointers to
763strings, cast to u64.  Strings will be copied into space reserved in
764the event for the string, using these pointers.
765
766Alternatively, the synth_event_trace_array() function can be used to
767accomplish the same thing.  It is passed the trace_event_file
768representing the synthetic event (which can be retrieved using
769trace_get_event_file() using the synthetic event name, "synthetic" as
770the system name, and the trace instance name (NULL if using the global
771trace array)), along with an array of u64, one for each synthetic
772event field.
773
774To trace an event corresponding to the synthetic event definition
775above, code like the following could be used::
776
777  u64 vals[7];
778
779  vals[0] = 777;                  /* next_pid_field */
780  vals[1] = (u64)"tiddlywinks";   /* next_comm_field */
781  vals[2] = 1000000;              /* ts_ns */
782  vals[3] = 1000;                 /* ts_ms */
783  vals[4] = smp_processor_id();   /* cpu */
784  vals[5] = (u64)"thneed";        /* my_string_field */
785  vals[6] = 398;                  /* my_int_field */
786
787The 'vals' array is just an array of u64, the number of which must
788match the number of field in the synthetic event, and which must be in
789the same order as the synthetic event fields.
790
791All vals should be cast to u64, and string vals are just pointers to
792strings, cast to u64.  Strings will be copied into space reserved in
793the event for the string, using these pointers.
794
795In order to trace a synthetic event, a pointer to the trace event file
796is needed.  The trace_get_event_file() function can be used to get
797it - it will find the file in the given trace instance (in this case
798NULL since the top trace array is being used) while at the same time
799preventing the instance containing it from going away::
800
801       schedtest_event_file = trace_get_event_file(NULL, "synthetic",
802                                                   "schedtest");
803
804Before tracing the event, it should be enabled in some way, otherwise
805the synthetic event won't actually show up in the trace buffer.
806
807To enable a synthetic event from the kernel, trace_array_set_clr_event()
808can be used (which is not specific to synthetic events, so does need
809the "synthetic" system name to be specified explicitly).
810
811To enable the event, pass 'true' to it::
812
813       trace_array_set_clr_event(schedtest_event_file->tr,
814                                 "synthetic", "schedtest", true);
815
816To disable it pass false::
817
818       trace_array_set_clr_event(schedtest_event_file->tr,
819                                 "synthetic", "schedtest", false);
820
821Finally, synth_event_trace_array() can be used to actually trace the
822event, which should be visible in the trace buffer afterwards::
823
824       ret = synth_event_trace_array(schedtest_event_file, vals,
825                                     ARRAY_SIZE(vals));
826
827To remove the synthetic event, the event should be disabled, and the
828trace instance should be 'put' back using trace_put_event_file()::
829
830       trace_array_set_clr_event(schedtest_event_file->tr,
831                                 "synthetic", "schedtest", false);
832       trace_put_event_file(schedtest_event_file);
833
834If those have been successful, synth_event_delete() can be called to
835remove the event::
836
837       ret = synth_event_delete("schedtest");
838
8397.2.2 Tracing a synthetic event piecewise
840-----------------------------------------
841
842To trace a synthetic using the piecewise method described above, the
843synth_event_trace_start() function is used to 'open' the synthetic
844event trace::
845
846       struct synth_event_trace_state trace_state;
847
848       ret = synth_event_trace_start(schedtest_event_file, &trace_state);
849
850It's passed the trace_event_file representing the synthetic event
851using the same methods as described above, along with a pointer to a
852struct synth_event_trace_state object, which will be zeroed before use and
853used to maintain state between this and following calls.
854
855Once the event has been opened, which means space for it has been
856reserved in the trace buffer, the individual fields can be set.  There
857are two ways to do that, either one after another for each field in
858the event, which requires no lookups, or by name, which does.  The
859tradeoff is flexibility in doing the assignments vs the cost of a
860lookup per field.
861
862To assign the values one after the other without lookups,
863synth_event_add_next_val() should be used.  Each call is passed the
864same synth_event_trace_state object used in the synth_event_trace_start(),
865along with the value to set the next field in the event.  After each
866field is set, the 'cursor' points to the next field, which will be set
867by the subsequent call, continuing until all the fields have been set
868in order.  The same sequence of calls as in the above examples using
869this method would be (without error-handling code)::
870
871       /* next_pid_field */
872       ret = synth_event_add_next_val(777, &trace_state);
873
874       /* next_comm_field */
875       ret = synth_event_add_next_val((u64)"slinky", &trace_state);
876
877       /* ts_ns */
878       ret = synth_event_add_next_val(1000000, &trace_state);
879
880       /* ts_ms */
881       ret = synth_event_add_next_val(1000, &trace_state);
882
883       /* cpu */
884       ret = synth_event_add_next_val(smp_processor_id(), &trace_state);
885
886       /* my_string_field */
887       ret = synth_event_add_next_val((u64)"thneed_2.01", &trace_state);
888
889       /* my_int_field */
890       ret = synth_event_add_next_val(395, &trace_state);
891
892To assign the values in any order, synth_event_add_val() should be
893used.  Each call is passed the same synth_event_trace_state object used in
894the synth_event_trace_start(), along with the field name of the field
895to set and the value to set it to.  The same sequence of calls as in
896the above examples using this method would be (without error-handling
897code)::
898
899       ret = synth_event_add_val("next_pid_field", 777, &trace_state);
900       ret = synth_event_add_val("next_comm_field", (u64)"silly putty",
901                                 &trace_state);
902       ret = synth_event_add_val("ts_ns", 1000000, &trace_state);
903       ret = synth_event_add_val("ts_ms", 1000, &trace_state);
904       ret = synth_event_add_val("cpu", smp_processor_id(), &trace_state);
905       ret = synth_event_add_val("my_string_field", (u64)"thneed_9",
906                                 &trace_state);
907       ret = synth_event_add_val("my_int_field", 3999, &trace_state);
908
909Note that synth_event_add_next_val() and synth_event_add_val() are
910incompatible if used within the same trace of an event - either one
911can be used but not both at the same time.
912
913Finally, the event won't be actually traced until it's 'closed',
914which is done using synth_event_trace_end(), which takes only the
915struct synth_event_trace_state object used in the previous calls::
916
917       ret = synth_event_trace_end(&trace_state);
918
919Note that synth_event_trace_end() must be called at the end regardless
920of whether any of the add calls failed (say due to a bad field name
921being passed in).
922
9237.3 Dyamically creating kprobe and kretprobe event definitions
924--------------------------------------------------------------
925
926To create a kprobe or kretprobe trace event from kernel code, the
927kprobe_event_gen_cmd_start() or kretprobe_event_gen_cmd_start()
928functions can be used.
929
930To create a kprobe event, an empty or partially empty kprobe event
931should first be created using kprobe_event_gen_cmd_start().  The name
932of the event and the probe location should be specified along with one
933or args each representing a probe field should be supplied to this
934function.  Before calling kprobe_event_gen_cmd_start(), the user
935should create and initialize a dynevent_cmd object using
936kprobe_event_cmd_init().
937
938For example, to create a new "schedtest" kprobe event with two fields::
939
940  struct dynevent_cmd cmd;
941  char *buf;
942
943  /* Create a buffer to hold the generated command */
944  buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL);
945
946  /* Before generating the command, initialize the cmd object */
947  kprobe_event_cmd_init(&cmd, buf, MAX_DYNEVENT_CMD_LEN);
948
949  /*
950   * Define the gen_kprobe_test event with the first 2 kprobe
951   * fields.
952   */
953  ret = kprobe_event_gen_cmd_start(&cmd, "gen_kprobe_test", "do_sys_open",
954                                   "dfd=%ax", "filename=%dx");
955
956Once the kprobe event object has been created, it can then be
957populated with more fields.  Fields can be added using
958kprobe_event_add_fields(), supplying the dynevent_cmd object along
959with a variable arg list of probe fields.  For example, to add a
960couple additional fields, the following call could be made::
961
962  ret = kprobe_event_add_fields(&cmd, "flags=%cx", "mode=+4($stack)");
963
964Once all the fields have been added, the event should be finalized and
965registered by calling the kprobe_event_gen_cmd_end() or
966kretprobe_event_gen_cmd_end() functions, depending on whether a kprobe
967or kretprobe command was started::
968
969  ret = kprobe_event_gen_cmd_end(&cmd);
970
971or::
972
973  ret = kretprobe_event_gen_cmd_end(&cmd);
974
975At this point, the event object is ready to be used for tracing new
976events.
977
978Similarly, a kretprobe event can be created using
979kretprobe_event_gen_cmd_start() with a probe name and location and
980additional params such as $retval::
981
982  ret = kretprobe_event_gen_cmd_start(&cmd, "gen_kretprobe_test",
983                                      "do_sys_open", "$retval");
984
985Similar to the synthetic event case, code like the following can be
986used to enable the newly created kprobe event::
987
988  gen_kprobe_test = trace_get_event_file(NULL, "kprobes", "gen_kprobe_test");
989
990  ret = trace_array_set_clr_event(gen_kprobe_test->tr,
991                                  "kprobes", "gen_kprobe_test", true);
992
993Finally, also similar to synthetic events, the following code can be
994used to give the kprobe event file back and delete the event::
995
996  trace_put_event_file(gen_kprobe_test);
997
998  ret = kprobe_event_delete("gen_kprobe_test");
999
10007.4 The "dynevent_cmd" low-level API
1001------------------------------------
1002
1003Both the in-kernel synthetic event and kprobe interfaces are built on
1004top of a lower-level "dynevent_cmd" interface.  This interface is
1005meant to provide the basis for higher-level interfaces such as the
1006synthetic and kprobe interfaces, which can be used as examples.
1007
1008The basic idea is simple and amounts to providing a general-purpose
1009layer that can be used to generate trace event commands.  The
1010generated command strings can then be passed to the command-parsing
1011and event creation code that already exists in the trace event
1012subsystem for creating the corresponding trace events.
1013
1014In a nutshell, the way it works is that the higher-level interface
1015code creates a struct dynevent_cmd object, then uses a couple
1016functions, dynevent_arg_add() and dynevent_arg_pair_add() to build up
1017a command string, which finally causes the command to be executed
1018using the dynevent_create() function.  The details of the interface
1019are described below.
1020
1021The first step in building a new command string is to create and
1022initialize an instance of a dynevent_cmd.  Here, for instance, we
1023create a dynevent_cmd on the stack and initialize it::
1024
1025  struct dynevent_cmd cmd;
1026  char *buf;
1027  int ret;
1028
1029  buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL);
1030
1031  dynevent_cmd_init(cmd, buf, maxlen, DYNEVENT_TYPE_FOO,
1032                    foo_event_run_command);
1033
1034The dynevent_cmd initialization needs to be given a user-specified
1035buffer and the length of the buffer (MAX_DYNEVENT_CMD_LEN can be used
1036for this purpose - at 2k it's generally too big to be comfortably put
1037on the stack, so is dynamically allocated), a dynevent type id, which
1038is meant to be used to check that further API calls are for the
1039correct command type, and a pointer to an event-specific run_command()
1040callback that will be called to actually execute the event-specific
1041command function.
1042
1043Once that's done, the command string can by built up by successive
1044calls to argument-adding functions.
1045
1046To add a single argument, define and initialize a struct dynevent_arg
1047or struct dynevent_arg_pair object.  Here's an example of the simplest
1048possible arg addition, which is simply to append the given string as
1049a whitespace-separated argument to the command::
1050
1051  struct dynevent_arg arg;
1052
1053  dynevent_arg_init(&arg, NULL, 0);
1054
1055  arg.str = name;
1056
1057  ret = dynevent_arg_add(cmd, &arg);
1058
1059The arg object is first initialized using dynevent_arg_init() and in
1060this case the parameters are NULL or 0, which means there's no
1061optional sanity-checking function or separator appended to the end of
1062the arg.
1063
1064Here's another more complicated example using an 'arg pair', which is
1065used to create an argument that consists of a couple components added
1066together as a unit, for example, a 'type field_name;' arg or a simple
1067expression arg e.g. 'flags=%cx'::
1068
1069  struct dynevent_arg_pair arg_pair;
1070
1071  dynevent_arg_pair_init(&arg_pair, dynevent_foo_check_arg_fn, 0, ';');
1072
1073  arg_pair.lhs = type;
1074  arg_pair.rhs = name;
1075
1076  ret = dynevent_arg_pair_add(cmd, &arg_pair);
1077
1078Again, the arg_pair is first initialized, in this case with a callback
1079function used to check the sanity of the args (for example, that
1080neither part of the pair is NULL), along with a character to be used
1081to add an operator between the pair (here none) and a separator to be
1082appended onto the end of the arg pair (here ';').
1083
1084There's also a dynevent_str_add() function that can be used to simply
1085add a string as-is, with no spaces, delimiters, or arg check.
1086
1087Any number of dynevent_*_add() calls can be made to build up the string
1088(until its length surpasses cmd->maxlen).  When all the arguments have
1089been added and the command string is complete, the only thing left to
1090do is run the command, which happens by simply calling
1091dynevent_create()::
1092
1093  ret = dynevent_create(&cmd);
1094
1095At that point, if the return value is 0, the dynamic event has been
1096created and is ready to use.
1097
1098See the dynevent_cmd function definitions themselves for the details
1099of the API.
1100