[vox-tech] C profiling tools

[Bill Broadley]

Shwaine wrote:
> Any software profiling gurus out there? I've come across a wee bit of a 
> problem trying to streamline some code. I suspect the bottleneck for my 
> code is rand(), but before going to the hassle of writing a PRNG (or 
> finding a decent open-source one),

Most linux distributions have a quite good PRNG implementation for 
rand/random.  Of course you can trade speed for quality, or quality
for speed if you want.

Also if you need direct access /dev/urandom and /dev/random

> I'd like to confirm that rand() is 
> indeed the bottleneck. So I fired up the profiling tools I know about: 
> gprof, strace and ltrace.

For specification questions like this it's likely a good idea to just
spend a minute or two and write an example and time it.

> strace confirmed that the majority of the execution time was within the 
> program's primary loop (another concern I had was I/O overhead reading the 
> input file and writing report files after the loop, so strace eliminated 
> that possibility). So then I moved on to profiling the primary loop. 
> That's when I ran head-first into a brick wall. The first problem is that 
> gprof can only trace the code about 60-65% of the time due to the time 
> spent within library calls. Which leads to the second problem, ltrace 

I'm not familiar with slackware.  But in general you can usually load the 
development, debugging, or profile enabled libraries that will allow you to 
debug (or profile) into libraries.  Once you install said libraries they are 
usually installed in /usr/lib/debug or something, so make sure you use
those libraries by running.  Either mention them specifically during compile,
tweak your LD_LIBRARY_PATH, or use LD_PRELOAD.

> crashes with SIGPROF and SIGSEGV before it even finishes reading the input 
> file, so I can't use it to get timing on the library calls. I compiled 
> ltrace-0.5-3.1 which seemed to be the newest version available on Debian's 
> website. The system is Slackware 10.2 Pentium 4 2.4GHz, so it seems like 
> ltrace should support it, but no such luck. ltrace does work for small 
> program, e.g. "ltrace ls" worked perfectly fine, but it crashes on my 
> program.

Heh, never heard of ltrace.  It sounds like potentially you want to record 3 
times of things:
* time used by your code in your program
* time used by your code in the libraries
* time used by the kernel in response to system calls

All 3 are rather different, system calls are relatively straight forward, but 
the kernel is tougher since it's not easily tracked who caused the various 
work inside the kernel.  Anything involving the buffer cache, network stack,
or filesystem can lead to an large amount of work and/or latency without being
able to tell who exactly caused a data structure to be walked.

How much pseudo random data did you need?  I've got a 2.4 GHz intel and I get 
about 5.7MB/s

dd if=/dev/urandom of=/dev/null bs=1024 count=102400
102400+0 records in
102400+0 records out
104857600 bytes (105 MB) copied, 18.2676 seconds, 5.7 MB/s

oprofile is my favorite way to get system wide profiling information.  Nothing 
like theorizing what is going on in the system as a result, then actually 
verifying the number of TLB misses or main memory accesses.

Especially with I/O you really need the kernel time to have any idea what is 
going on, iostat can help as well.

A similar but different system that tweaks the kernel to help record more info 
is sysprof.

> Does anyone know of any other open-source profiling tools that would be 
> able to say how long the program is spending not only in my function calls 
> but also the library calls? Or alternatively, does anyone have a clue 
> about how to get ltrace working?

Sounds like debugging libraries will fix part of the problem, and 
sysprof/oprofile will fix the rest.