I like dtrace. I also like Linux. And thats where the problems begin: there is no proper dtrace for Linux. However, systemtap exists. systemtap is compatible to dtrace probes, however, it comes with it's own scripting language.

For those who don't know what any of these are: both are tools to efficiently trace programs at runtime, with a very low overhead. To allow that, they use so-called "probes" compiled into the program. Your kernel is usually equipped with such probes and Ruby 2.0 or higher also comes with quite a few of them.

However, the whole process is a bit involved and I found no good explanation for the whole setup and the pitfalls. Read on for my attempt. I will use Ubuntu as an example system.

Prerequisites

To use the probes embedded in the Ruby interpreter, you need first need to make sure that you have a kernel that support userspace probes. This is the case since Linux 3.5, patches for older kernels are available and are part of some distributions (e.g. RHEL). According to the systemtap wiki and my own experiences, this rules out any Ubuntu with a version number below 12.10, unless you want to compile your own kernel. So, let's grab a Ubuntu, e.g. using the raring64 Vagrant box from vagrantbox.es:

$ mkdir systemtapvm; cd systemtapvm
$ vagrant init raring64 https://copy.com/OljurIPEsjaX
$ vagrant up
$ vagrant ssh

After connecting to the VM, you need to install multiple packages before installing Ruby:

apt-get install systemtap systemtap-sdt-dev

systemtap is the tool itself, systemtap-sdt-dev contains additional tools to write userspace probes. Especially, it contains the dtrace binary and headers, which we will need later when building Ruby.

apt-get install linux-headers-$(uname -r)

You kernel headers. systemtap needs them to compile it's probes. This is strictly necessary.

Optionally, if you want to trace things included in the kernel, you will need your kernels debug image. The process involves adding an additional repository and can be found here. Be aware that the debug image weights around 600MB and the download servers are rather slow. It took me more than an hour to download.

As a final, also optional step, add your user to the stapdev or stapusr group (both explained here):

$ adduser vagrant stapusr
$ adduser vagrant stapdev

Finally, there is this nice script that checks whether you got everything right.

Aside: the whole thing on Gentoo

$ sudo emerge =systemtap-2.4 --autounmask-write

Follow the instructions from there.

Compiling Ruby

If everything is set up, the only thing to do is to compile Ruby 2.0 or higher on your own. Either get the source, alternatively use rvm or ruby-install. DTrace support will be built automatically if available, but I recommend passing --enable-dtrace to make sure that the build process fails if anything is wrong.

$ ./configure --enable-dtrace
$ make
$ make install

So, let's see what we can do with this! First of all, let's list all available probes:

$ stap -L 'process("ruby").mark("*")'
process("ruby").mark("array__create") $arg1:long $arg2:long $arg3:long
...

Refer to the full list of probes for explanations.

Finally, tracing Ruby!

Instead of implementing DTraces scripting language, systemtap comes with it's own scripting language. The scripts allow us to make sense of the events passed by the probes.

For example, we can count all requires happening (example taken from http://avsej.net/2012/systemtap-and-ruby-20/):

global nmodules = 0;

probe process("ruby").mark("require__entry")
{
    module = kernel_string($arg1)
    file = kernel_string($arg2)
    line = $arg3
    printf("%s(%d) %s %s:%d required file `%s'\n", execname(), pid(), $$name, file, line, module)
    nmodules++;
}

probe end {
    printf("Total files: %d\n", nmodules);
    delete nmodules;
}

Let's run this on a simple Ruby program:

$ stap requires.stp -c 'ruby -e "exit"'
ruby(5191) require__entry ruby:0 required file `enc/encdb.so'
...
Total files: 27

And we will get all requires, neatly ordered (in this case, 27, 3 for the encoding system, the rest for rubygems).

Another interesting example is Aman Guptas method cache script (my fork, due to a bug in the original). It allows us to see when the method cache is cleared and which objects are involved. This probe is only available in Ruby 2.1 or higher, as well as the fine-grained method cache. This finally gives us insight into the what happens when pull certain metaprogramming tricks.

This time, let's try it differently. systemtap can also be attached to processes externally:

$ stap ruby_mcache.stp

Open another console and type:

$ irb

The first console should show lines like this:

irb(5329) method__cache__clear /home/vagrant/.rubies/ruby-2.1.0/lib/ruby/2.1.0/irb /extend-command.rb:198 cleared `Object'

For added fun: require 'ostruct'.

This allows us to attach and detach from currently running processes (or ones probably running in the future). Try closing and restarting stap while keeping the irb session.

Conclusion

I haven't tested systemtap in production, but I like what I see so far. I ran the whole thing on the Padrino codebase, to good results. I will definitely try to use it in development more and maybe move to production after a while.

Still, I found the setup process a bit involved, especially as there are many systems out there that don't come with the necessary kernel features. Check before you buy!

Credits

Code examples and lots of pointers from avsej. One code example from tmm1.