DTrace is a Dynamic Tracing framework developed by Sun and ported to NetBSD. It enables extensive instrumentation of the kernel and user space. See the DTrace Community Page for more information. Also see DTrace Introduction, Brendan Gregg's DTrace one liners and his notes for DTrace on FreeBSD.

Current status

Supported platforms

DTrace is a work-in-progress effort and it is for x86 systems and some arm boards.

• i386 and amd64
• earm* (evbarm and armv4 based ports)

Supported providers

• DTrace: What to do when a script BEGINs, ENDs, ERRORs
• FBT: Function Boundary Tracing
• IO: Disk I/O
• Lockstat: Kernel Lock Statistics
• Proc: Process and thread related events
• Profile: Time based interrupt event source for Profiling
• SDT: Statically Defined Tracing
• Syscall: System Calls
• Syscall Linux (32bit & 64 bit): System calls via the Linux binary emulation layer
• VFS: Filesystem operations (confined to namecache events at time of writing - 8.99.22)

TODO for netbsd-7

• Measure effect of options KDTRACE_HOOKS on system performance.
• Determine whether the profile module works and list it here.
• Integrate riz's syscall provider patch.

How to use

Building DTrace

You need the following options in your kernel:

options         KDTRACE_HOOKS   # kernel DTrace hooks
options         MODULAR

Optionally:

options         INSECURE   # permit modules to loaded from user space once system has gone multiuser and securelevel has been raised.

A Distribution needs to be built with the options MKDTRACE=yes and MKCTF=yes, this is taken care of automatically and doesn't need to be specified manually. The list of platforms it is applied to automatically is set in src/share/mk/bsd.own.mk

Set the system to load the solaris and dtrace related modules in /etc/modules.conf, for a list of available modules, see /stand/$MACHINE/$VERSION/modules/

For example, add the following to /etc/modules.conf (the file may not exist already on a system):

• solaris
• dtrace
• dtrace_fbt
• dtrace_lockstat
• dtrace_profile
• dtrace_sdt
• dtrace_syscall
• dtrace_syscall_linux

A dtrace device node is created automatically in /dev/dtrace when the modules are loaded into place.

List the dtrace probes

dtrace -l

   ID   PROVIDER            MODULE                          FUNCTION NAME
    1     dtrace                                                     BEGIN
    2     dtrace                                                     END
    3     dtrace                                                     ERROR
    4        fbt            netbsd             AcpiAcquireGlobalLock entry
    5        fbt            netbsd             AcpiAcquireGlobalLock return
    6        fbt            netbsd             AcpiAllocateRootTable entry
    7        fbt            netbsd                    AcpiAttachData entry
    . 
    .
29129        fbt           solaris                   zfs_vop_getattr entry 
29130        fbt           solaris                   zfs_vop_getattr return
29131       proc                                                     create
29132       proc                                                     exec 
    .
    .
29140       proc                                                     lwp_start
29141       proc                                                     lwp_exit

Running hello world

Put the following into the file hello.d:

BEGIN
{
    trace("Hello world");
    exit(0);
}

Run the hello world script:

dtrace -s hello.d

dtrace: script './hello.d' matched 1 probe
CPU     ID                    FUNCTION:NAME
  0      1                           :BEGIN   Hello world

The same script could be executed as a one liner on the shell, using

dtrace -n 'BEGIN { trace("Hello world"); exit(0); }'

A more complex example

The following script traces the execution of a sleep operation in the kernel. Put it in sleep.d:

#pragma D option flowindent

syscall::nanosleep:entry
/execname == "sleep" && guard++ == 0/
{
        self->traceme = 1;
}

fbt:::
/self->traceme/
{}

syscall::nanosleep:return
/self->traceme/
{
        self->traceme = 0;
        exit(0);
}

Start the script running:

dtrace -s sleep.d

This will take a while as the script instruments every function in the kernel. When it's ready, it will print a message like "dtrace: script 'sleep.d' matched 59268 probes". Then execute a "sleep 2" in another shell.

Tools included in base

Starting with NetBSD-8, on builds where MKDTRACE=yes is set, scripts from Brendan Gregg's DTrace toolkit are installed in base as standard.

At present, the following scripts are installed in /usr/sbin:

• dtruss - An implementation of the truss utility in DTrace which traces the system calls made by a process
• execsnoop - snoop on execution of processes as they occur
• opensnoop - snoop on openning of files as they occur
• procsystime - print process system call time details.

Troubleshooting

The Compact C Type Format (CTF) has a 2¹⁵ limit on types which can overflow, this prevents DTrace from working correctly.

Check the number of types using ctfdump e.g

ctfdump -S /netbsd

Note the line which states total number of types, the value should by less than 32768.

If overflow is not an issue, libdtrace(3) can provide some insight into what is going on via an environment variable. Define DTRACE_DEBUG before tracing.

 env DTRACE_DEBUG= execsnoop