Kernel Tracing with Ftrace
Ftrace is a tracing utility in the the Linux kernel, designed to help out developers of the system to find what is going on inside the kernel.
Ftrace was developed by Steven Rostedt and has been included in the kernel since version 2.6.27.
It can be used for debugging or analyzing latencies and performance issues that take place outside of user-space.
Ftrace uses the debugfs file system to hold the control files as well as the files to display output.
To mount this directory, you can add to your /etc/fstab file:
tracefs /sys/kernel/debug/tracing tracefs defaults 0 0
Or you can mount it at run time with:
mount -t tracefs nodev /sys/kernel/debug/tracing
cd /sys/kernel/debug/tracing
Some of the key files are
ls /sys/kernel/debug/tracing
available_tracers – available tracing programs
tracing_on - 0 into this file to disable the tracer or 1 to enable it
trace - holds the output of the trace
current_tracer – display the current tracer that is configured
Current tracers that may be configured
cat /sys/kernel/debug/tracing/available_tracers
function - Function call tracer to trace all kernel functions.
function_graph - Similar to the function tracer except that the function tracer probes the functions on their entry
whereas the function graph tracer traces on both entry and exit of the functions. It then provides the ability to draw a graph of function calls similar to C code source.
blk - The block tracer I/O. The tracer used by the blktrace user application.
hwlat - The Hardware Latency tracer is used to detect if the hardware produces any latency.
irqsoff - Traces the areas that disable interrupts and saves the trace with the longest max latency.
preemptoff - Similar to irqsoff but traces and records the amount of time for which preemption is disabled.
preemptirqsoff - Similar to irqsoff and preemptoff, but traces and records the largest time for which irqs and/or preemption is disabled.
wakeup - Traces and records the max latency that it takes for the highest priority task to get scheduled after it has been woken up.
wakeup_rt - Traces and records the max latency that it takes for just RT tasks (as the current "wakeup" does).
wakeup_dl - Traces and records the max latency that it takes for a SCHED_DEADLINE task to be woken (as the "wakeup" and "wakeup_rt" does).
mmiotrace - A special tracer that is used to trace binary module. It will trace all the calls that a module makes to the hardware. Everything it writes and reads from the I/O as well.
branch - This tracer can be configured when tracing likely/unlikely calls within the kernel. It will trace when a likely and unlikely branch is hit and if it was correct in its prediction of being correct.
nop - This is the "trace nothing" tracer. A way to disable other tracers and can be use with events
Example how to run ftrace with function tracer
sysctl kernel.ftrace_enabled=1
echo function > /sys/kernel/debug/tracing/current_tracer
echo 1 > /sys/kernel/debug/tracing/tracing_on
sleep 1
echo 0 > /sys/kernel/debug/tracing/tracing_on
cat /sys/kernel/debug/tracing/trace
Limiting the traces with setting up the filters
set_ftrace_filter - enabling or disabling specific functions to be traced
set_ftrace_notrace - Any function that is added here will not be traced
set_ftrace-pid - only traces functions executed by a task with the given pid
set_graph_function - Functions listed in this file will cause the function graph tracer to only trace these functions and the functions that they call.
set_graph_notrace - Disable function graph tracing when the function is hit until it exits the function.
Function name filter is very limited show only when enter or exit the function
echo schedule > set_ftrace_filter
Appending the filter
echo '*rcu*' >> set_ftrace_filter
Clear the filter
echo > set_ftrace_filter
Example filter syscalls
echo SyS_read > set_ftrace_filter
Function stack trace with func_stack_trace option
echo schedule > set_ftrace_filter
echo 1 > /sys/kernel/debug/tracing/options/func_stack_trace
echo function > /sys/kernel/debug/tracing/current_tracer
echo 0 > /sys/kernel/debug/tracing/tracing_on
ech0 0 > /sys/kernel/debug/tracing/options/func_stack_trace
Tracing Events
The trace event directory. It holds event tracepoints (also known as static tracepoints) that have been compiled into the kernel. Tracepoints placed in code provides a hook to call a function (probe) that you can provide at runtime can be used without creating custom kernel modules to register probe functions using the event tracing infrastructure.
You can find all events on
cat /sys/kernel/debug/tracing/available_events
Example with events
echo 0 > /sys/kernel/debug/tracing/tracing_on
echo 1 > /sys/kernel/debug/tracing/events/syscalls/enable
echo 1 > /sys/kernel/debug/tracing/events/exceptions/enable
sh -c 'echo $$ > /sys/kernel/debug/tracing/set_event_pid; echo 1 > /sys/kernel/debug/tracing/tracing_on; exec echo ftrace'
cat /sys/kernel/debug/tracing/trace
set_event_pid Events only trace a task with a PID listed in this file
echo 0 > /sys/kernel/debug/tracing/tracing_on
echo 1 > /sys/kernel/debug/tracing/events/syscalls/enable
echo 1 > /sys/kernel/debug/tracing/events/exceptions/enable
echo $$ > /sys/kernel/debug/tracing/set_event_pid
echo 1 > /sys/kernel/debug/tracing/tracing_on;
echo ftrace_events;
echo 0 > /sys/kernel/debug/tracing/tracing_on;
The new interface trace-cmd
Change tracer to function
trace-cmd start -p function
See available filters
trace-cmd list -f
Set ftrace filter
trace-cmd start -p function -l schedule
Set ftrace pid
trace-cmd record -p function -F echo ftrace
Syscall tracing
trace-cmd -p function -g SyS_read
Ftrace no trace (set_ftrace_notrace)
trace -p nop -n schedule
Ftrace function stack trace ( options/func_stack_trace )
trace-cmd start -p function -l schedule --func-stack
Ftrace Events list
trace-cmd list -e
Set Event
trace-cmd start -p -e sched
Set Event pid
trace-cmd record -e syscalls -e exceptions -F echo ftrace
Show output
trace-cmd show
trace-cmd report
source
https://www.kernel.org/doc/Documentation/trace/ftrace.txt
https://www.kernel.org/doc/Documentation/trace/events.txt
https://www.kernel.org/doc/Documentation/trace/tracepoints.txt
https://www.kernel.org/doc/Documentation/trace/ftrace-design.txt
https://kernel-recipes.org/en/2017/talks/understanding-the-linux-kernel-via-ftrace/