This manual is for UST 0.5.
The LTTng Userspace Tracer (UST) is a library accompanied by a set of tools to trace userspace code.
Code may be instrumented with either markers or tracepoints. A highly efficient lockless tracer records these events to a trace buffers. These buffers are reaped by a deamon which writes trace data to disk.
High performance is achieved by the use of lockless buffering algorithms, RCU and per-cpu buffers. In addition, special care is taken to minize cache impact.
The LTTng Userspace Tracer is intended to be linkable to open source software as well as to proprietary applications. This was accomplished by licensing the code that needs to be linked to the traced program as LGPL.
Components licensed as LGPL v2.1:
Components licensed as GPL v2:
UST can currently trace applications running on Linux, on the x86-32, x86-64 and PowerPC 32 architectures.
The LTTng userspace tracer is a library and a set of userspace tools.
The following packages are required:
This contains the tracing library, the ustd daemon, trace control tools and other helper tools.
Repository: http://git.dorsal.polymtl.ca
This is the userspace read-copy update library by Mathieu Desnoyers.
Available in Debian as package liburcu-dev.
Home page: http://lttng.org/urcu
LTTV is a graphical (and text) viewer for LTTng traces.
Home page: http://lttng.org
Liburcu should be installed first. UST may then be compiled and installed. LTTV has no dependency on the other packages; it may therefore be installed on a system which does not have UST installed.
Refer to the README in each of these packages for installation instructions.
First, instrument a program with a marker.
#include <ust/marker.h>
int main(int argc, char **argv)
{
int v;
char *st;
/* ... set values of v and st ... */
/* a marker: */
trace_mark(ust, myevent, "firstarg %d secondarg %s", v, st);
/* a marker without arguments: */
trace_mark(ust, myotherevent, MARK_NOARGS);
return 0;
}
Then compile it in the regular way, linking it with libust. For example:
gcc -o foo -lust foo.c
Run the program with usttrace. The usttrace output says where the trace was written.
usttrace ./foo
Finally, open the trace in LTTV.
lttv-gui -t /path/to/trace
The trace can also be dumped as text in the console:
lttv -m textDump -t /path/to/trace
In order to record a trace of events occurring in a application, the application must be instrumented. Instrumentation points resemble function calls. When the program reaches an instrumentation point, an event is generated.
There are no limitations on the type of code that may be instrumented. Multi-threaded programs may be instrumented without problem. Signal handlers may be instrumented as well.
There are two APIs to instrument programs: markers and tracepoints. Markers are quick to add and are usually used for temporary instrumentation. Tracepoints provide a way to instrument code more cleanly and are suited for permanent instrumentation.
In addition to executable programs, shared libraries may also be instrumented with the methods described in this chapter.
Adding a marker is simply a matter of inserting one line in the program.
#include <ust/marker.h>
int main(int argc, char **argv)
{
int v;
char *st;
/* ... set values of v and st ... */
/* a marker: */
trace_mark(main, myevent, "firstarg %d secondarg %s", v, st);
/* another marker without arguments: */
trace_mark(main, myotherevent, MARK_NOARGS);
return 0;
}
The invocation of the trace_mark() macro requires at least 3 arguments. The first, here "main", is the name of the event category. It is also the name of the channel the event will go in. The second, here "myevent" is the name of the event. The third is a format string that announces the names and the types of the event arguments. Its format resembles that of a printf() format string; it is described thoroughly in Appendix x.
A given Marker may appear more than once in the same program. Other Markers may have the same name and a different format string, although this might induce some confusion at analysis time.
The Tracepoints API uses the Markers, but provides a higher-level abstraction. Whereas the markers API provides limited type checking, the Tracepoints API provides more thorough type checking and discharges from the need to insert format strings directly in the code and to have format strings appear more than once if a given marker is reused.
Note: Although this example uses mychannel as the channel, the only channel name currently supported with early tracing is ust. The usttrace tool always uses the early tracing mode. When using manual mode without early tracing, any channel name may be used.
A function instrumented with a tracepoint looks like this:
#include "tp.h"
void function()
{
int v;
char *st;
/* ... set values of v and st ... */
/* a tracepoint: */
trace_mychannel_myevent(v, st);
}
Another file, here tp.h, contains declarations for the tracepoint.
#include <ust/tracepoint.h>
DECLARE_TRACE(mychannel_myevent, TP_PROTO(int v, char *st),
TP_ARGS(v, st));
A third file, here tp.c, contains definitions for the tracepoint.
#include <ust/marker.h>
#include "tp.h"
DEFINE_TRACE(mychannel_myevent);
void mychannel_myevent_probe(int v, char *st)
{
trace_mark(mychannel, myevent, "v %d st %s", v, st);
}
static void __attribute__((constructor)) init()
{
register_trace_mychannel_myevent(mychannel_myevent_probe);
}
Here, tp.h and tp.c could contain declarations and definitions for other tracepoints. The constructor would contain other register_* calls.
The simplest way to record a trace is to use the usttrace script. An example is given in the quickstart above.
The usttrace script automatically:
Each subdirectory of the save location contains the trace of one process that was generated by the command. The name of a subdirectory consists in the the PID of the process, followed by the timestamp of its creation.
The save location also contains logs of the tracing.
When using usttrace, the early tracing is always active, which means
that the tracing is guaranteed to be started by the time the process enters its
main() function.
Several usttrace's may be run simultaneously without risk of conflict. This facilitates the use of the tracer by idependent users on a system. Each instance of usttrace starts its own daemon which collects the events of the processes it creates.
Instead of using usttrace, a trace may be recorded on an already running process.
First the daemon must be started.
# Make sure the directory for the communication sockets exists.
$ mkdir /tmp/ustsocks
# Make sure the directory where ustd will write the trace exists.
$ mkdir /tmp/trace
# Start the daemon
$ ustd
# We assume the program we want to trace is already running and that
# it has pid 1234.
# List the available markers
$ ustctl --list-markers 1234
# A column indicates 0 for an inactive marker and 1 for an active marker.
# Enable a marker
$ ustctl --enable-marker ust/mymark 1234
# Create a trace
$ ustctl --create-trace 1234
# Start tracing
$ ustctl --start-trace 1234
# Do things...
# Stop tracing
$ ustctl --stop-trace 1234
# Destroy the trace
$ ustctl --destroy-trace 1234
For more information about the manual mode, see the ustctl(1) man page.
Early tracing consists in starting the tracing as early as possible in the
program, so no events are lost between program start and the point where the
command to start the tracing is given. When using early tracing, it is
guaranteed that by the time the traced program enters its main()
function, the tracing will be started.
When using usttrace, the early tracing is always active.
When using the manual mode (ustctl), early tracing is enabled using
environment variables. Setting UST_TRACE to 1, enables early
tracing, while setting UST_AUTOPROBE to 1 enables all markers
automatically.
When a process being traced crashes, the daemon is able to recover all the events in its buffers that were successfully commited. This is possible because the buffers are in a shared memory segment which remains available to the daemon even after the termination of the traced process.
fork() and clone()Tracing across clone() when the CLONE_VM flag is specified is
supported without any particular action.
When clone() is called without CLONE_VM or fork() is
called, a new address space is created and the tracer must be notified to
create new buffers for it.
This can be done automatically, by LD_PRELOAD'ing libinterfork.so.
This library intercepts calls to fork() and informs the tracer it is
being called. When using usttrace, this is accomplied by specifying
the -f command line argument.
Alternatively, the program can call ust_before_fork() before calling
fork() or clone() with CLONE_VM. After the call,
ust_after_fork_parent() must be called in the parent process and
ust_after_fork_child() must be called in the child process.
Some programs need to be traced even though they were not linked to libust either because they were not instrumented or because it was not practical.
An executable that is not instrumented can still yield interesting traces when at least one of its dynamic libraries is instrumented. It is also possible to trace certain function calls by intercepting them with a specially crafted library that is linked with LD_PRELOAD at program start.
In any case, a program that was not linked to libust at compile time must be linked to it at run time with LD_PRELOAD. This can be accomplished with usttrace's -l option. It can also be done by setting the LD_PRELOAD environment variable on the command line. For example:
# Run ls with usttrace, LD_PRELOAD'ing libust
# (assuming one of the libraries used by ls is instrumented).
$ usttrace -l ls
# Run ls, manually adding the LD_PRELOAD.
$ LD_PRELOAD=/usr/local/lib/libust.so.0 ls
Todo.
Traces may be viewed with LTTV. An example of command for launching LTTV is given in the quickstart.
When tracing multi-process applications or several applications simultaneously, more than one trace will be obtained. LTTV can open and display all these traces simultaneously.
In addition to multiple userspace traces, LTTV can open a kernel trace recorded with the LTTng kernel tracer. This provides events that enable the rendering of the Control Flow View and the Resource View.
When doing so, it is necessary to use the same time source for the kernel tracer as well as the userspace tracer. Currently, the recommended method is to use the timestamp counter for both. The TSC can however only be used on architectures where it is synchronized across cores.
The purpose of this section is to give an overview of the resource usage of libust. For a developer, knowing this can be important: because libust is linked with applications, it needs to share some resources with it. Some applications may make some assumptions that are in conflict with libust's usage of resources.
In practice however, libust is designed to be transparent and is compatible with the vast majority of applications. This means no changes are required in the application (or library) being linked to libust.
Libust is initialized by a constructor, which by definition runs before the
main() function of the application starts. This constructor creates a
thread called the listener thread. The listener thread initializes a
named socket and waits for connections for ustd or ustctl.
Libust-specific code may:
malloc() and free()
fork() and clone()
It will not:
The behavior of tracing can be influenced by setting special environment variables in the environment of the traced application. This section describes these variables.
If set to 1, start tracing as soon as the program starts. Tracing is
guaranteed to be started by the time the main() function starts.
If set to 1, enable all markers by the time the main() function starts.
If set to 0, disable notification of daemon on trace start. Useful for
performance tests.
If set to 1, enable overwriting of buffers on overrun.
If set, defines the default number of subbuffers per buffer.
If set, defines the default size of subbuffers, in bytes.
GDB, the GNU Debugger, can use UST markers as GDB tracepoints (note GDB has its own concept of tracepoint). This feature is called GDB Static Tracepoints. When a GDB tracepoint is hit, GDB collects the marker arguments, as well as the state of the registers.
In UST, support for GDB integration is not compiled in by default because of
the cost of saving registers when a marker is hit. To enable it, run the
./configure script with the -DCONFIG_UST_GDB_INTEGRATION flag
in the CFLAGS environment variable. For example:
CFLAGS=-DCONFIG_UST_GDB_INTEGRATION ./configure
As of this writing, GDB Static Tracepoints have been submitted (http://sourceware.org/ml/gdb-patches/2010-06/msg00592.html) to the GDB mailing list.
GDB integration is currently only supported on x86-32 and x86-64.