strace - trace system calls and signals

 strace - trace system calls and signals.  It's a useful tool for diagnosis, debugging processes on Linux. 

To install:
 - root: yum install strace
Loaded plugins: rhnplugin, security
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package strace.x86_64 0:4.5.18-11.el5_8 set to be updated
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
 Package     Arch        Version                Repository                 Size
================================================================================
Installing:
 strace      x86_64      4.5.18-11.el5_8        rhel-x86_64-server-5      177 k
Transaction Summary
================================================================================
Install       1 Package(s)
Upgrade       0 Package(s)
Total download size: 177 k
Is this ok [y/N]: y
Downloading Packages:
strace-4.5.18-11.el5_8.x86_64.rpm                                                                                                                           | 177 kB     00:00    
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing     : strace  
 
  Installed:
    strace.x86_64 0:4.5.18-11.el5_8                                                                                                                                                 
 
Complete!

Some useful parameters:

-tt  If given twice, the time printed will include the microseconds.
-T  Show the time spent in system calls. This records the time difference between the beginning and the end of each system call.
-o  filename : Write the trace output to the file filename rather than to screen (stderr).
-p  PID   Attach to the process with the process ID pid and begin tracing. The trace may be terminated at any time by a keyboard interrupt signal (hit CTRL-C). strace will respond by detaching itself from the traced process(es) leaving it (them) to continue running. Multiple -p options can be used to attach to up to 32 processes in addition to command (which is optional if at least one -p option is given).

-s SIZE  Specify the maximum string size to print (the default is 32).

Examples: 

To trace an oracle process and see what it's doing..
strace -p 28446 -s 100 -o /tmp/orammnl.txt

to trace the open and read system calls
strace -e  trace=open,read -p 28446 -o /tmp/orammnl.debug

to trace the process spends time on system calls
strace -ttT -p 15878 -o /tmp/tracetiming.out

14:54:26.232457 open("/proc/15914/stat", O_RDONLY) = 26 <0.000028>
14:54:26.232584 read(26, "15914 (oracle) S 1 15914 15914 0"..., 999) = 243 <0.000049>
14:54:26.232884 close(26)               = 0 <0.000019>
14:54:26.233002 open("/proc/15916/stat", O_RDONLY) = 26 <0.000027>
14:54:26.233127 read(26, "15916 (oracle) S 1 15916 15916 0"..., 999) = 240 <0.000046>
14:54:26.233269 close(26)               = 0 <0.000019>
...
At 14:54:26.233127, it took 0.46 milisecond to read 240 bytes.