Using the ftrace Utility

1. Once you are using the trace variant of the MRG Realtime kernel, you can set up the ftrace utility. You will need to create a /debug directory and then mount it to use the debugfs file system.

   # mkdir /debug
   
   # mount -t debugfs nodev /debug

2. To check if the ftrace utility is running, use the cat command to view the /debug/tracing/tracing_enabled file. A value of 1 indicates that ftrace is running, and 0 indicates that it is not running.

   # cat /debug/tracing/tracing_enabled
   1

   By default, the tracer is enabled. To turn the tracer on or off, echo the appropriate value to the /debug/tracing/tracing_enabled file.

   # echo 0 > /debug/tracing/tracing_enabled
   	
   # echo 1 > /debug/tracing/tracing_enabled

3. The ftrace utility has a variety of options that allow you to use the utility in a number of different ways. The available options are:

   events	Trace specific events.
   wakeup	Record the time it takes for the highest priority RT task to wake up. If no RT tasks are running, this option will result in no output.
   pre-emptoff	Record the the longest time for which pre-emption is disabled.
   irqsoff	Record the the longest time for which interrupts have been disabled.
   pre-emptirqsoff	Record the the longest time for which either interrupts or pre-emption have been disabled.
   ftrace	Trace functions using mcount.
   sched_switch	Trace tasks as they are switched by the scheduler.
   none	Disable all tracing.

4. To check which options you currently have in the tracer use the cat command on the /debug/tracing/available_tracers file:

   # cat /debug/tracing/available_tracers

5. To set a single option on the tracer, echo the option name to the /debug/tracing/current_tracer file.

   # echo ftrace > /debug/tracing/current_tracer

   Note

   If you use a single > with the echo command, it will override any existing value in the file. If you wish to append the value to the file, use >> instead.
   
   
6. The results of any traces are saved to two files. The contents of /debug/tracing/latency_trace contains detailed information:

   # cat /debug/tracing/latency_trace

   The /debug/tracing/trace uses a simpler format:

   # cat /debug/tracing/trace

7. There are a number of options available for changing the format of the output. These options are stored in /debug/tracing/iter_ctrl:

   print-parent	Show the parent of the functions.
   sym-offset	Add the offset into the function.
   sym-addr	Add the address of a symbol.
   verbose	Increase the verbosity of the tracer output.

   
   
8. Use the cat command to view the current configuration:

   # cat /debug/tracing/iter_ctrl

   To set a single option on the tracer output configuration, echo the option name to the /debug/tracing/iter_ctrl file.

   # echo verbose > /debug/tracing/iter_ctrl

   To disable a single option on the tracing output configuration, echo the option name with the test no before it to the /debug/tracing/iter_ctrl file.

   # echo noverbose > /debug/tracing/iter_ctrl

   Note

   If you use a single > with the echo command, it will override any existing value in the file. If you wish to append the value to the file, use >> instead.
   
   
9. The ftrace utility can be filtered by altering the settings in the /debug/tracing/set_ftrace_filter file. If no filters are specified in the file, all processes are traced. Use the cat to view the current filters:

   # cat /debug/tracing/set_ftrace_filter

10. To change the filters, echo the name of the process to be traced. The filter allows the use of a * wildcard at the beginning or end of a search term. Some examples of filters are:
    • Trace only the schedule process:

      # echo schedule > /debug/tracing/set_ftrace_filter

    • Trace all processes that end with lock:

      # echo *lock > /debug/tracing/set_ftrace_filter

    • Trace all processes that start with spin_:

      # echo spin_* > /debug/tracing/set_ftrace_filter

    • Trace all processes with cpu in the name:

      # echo *cpu* > /debug/tracing/set_ftrace_filter

    Note

    The * wildcard for the tracer filter will only work at the beginning or end of a word. For example: spin_* and *lock will work, but spin_*lock will not.
    
    
    
    

    ---

    1. Document :
       #2
       
       
       Source :
       http://linux.die.net/man/1/ftrace
       
       
       Content :

    ftrace(1) - Linux man page

    Name

    ftrace - Runs the frysk systemcall tracing utility

    Synopsis

    ftrace

    Description

    ftrace Runs the frysk systemcall tracing utility.

    ftrace is a small utility that uses the frysk engine to trace systemcalls in a similar manner to strace.

    Example

    ftrace command

     

    ftrace pid

    See Also

    funit(8) fcatch(1) fcore(1) fstack(1) fstep(1)

    Bugs

    The option parser is greedy when looking for options so running ftrace on a program that uses options can be a problem, use -- to split between ftrace and the program. So change from:

    ~/prefix/bin/ftrace ~/prefix/lib64/frysk/funit --arch 32 frysk.proc.TestAbandon

    to

    ~/prefix/bin/ftrace -- ~/prefix/lib64/frysk/funit --arch 32 frysk.proc.TestAbandon

    Report bugs to [1] http://sources.redhat.com/ftrace/ .

    References

    1. http://sources.redhat.com/ftrace/

    http:/sources.redhat.com/ftrace

    Referenced By

    fhpd(1), fltrace(1), frysk(1), frysk(7)
    
    

    
    
    

    ---

    Document :
    #3
    
    
    Source :
    http://www.mgriepentrog.com/2009/09/09/enable-ftrace-on-ubuntu-linux/
    
    
    Content :
    

     Enable ftrace on Ubuntu Linux

    Posted on September 9, 2009 by Mike

    Introduction

    I was interested in using ftrace for a potential project, but didn’t find much documentation on how to get it working. I hope to change that by providing step-by-step instructions to enable ftrace for use in Ubuntu Linux 9.10 (Karmic Koala). Overview from the kernel documentation:

    Ftrace is an internal tracer designed to help out developers and designers of systems to find what is going on inside the kernel. It can be used for debugging or analyzing latencies and performance issues that take place outside of user-space.

    Given that ftrace is targeted towards developers and designers of systems, I’m assuming you more or less know what you’re doing with these steps. I also recommend virtualizing an instance of linux so that if you do mess up, you aren’t left with an un-bootable machine.

    Steps

    Download and install Ubuntu 9.10 Alpha-3

    Run the following commands to install the necessary dependencies for a custom kernel:

    $ sudo apt-get install git-core fakeroot build-essential makedumpfile kernel-wedge
$ sudo apt-get build-dep linux

    Get the latest version of the Ubuntu kernel:

    $ cd ~/
$ git clone git://kernel.ubuntu.com/ubuntu/ubuntu-karmic.git

    edit “debian/config/config.common.ubuntu” and make the following changes:

    CONFIG_FTRACE_SYSCALLS=y
CONFIG_FUNCTION_TRACER=y

    Next, run the following commands:

    $ sudo debian/rules clean
$ CONCURRENCY_LEVEL=2 AUTOBUILD=1 NOEXTRAS=1 fakeroot debian/rules binary-386

    If you are prompted about enabling other forms of tracing, see ftrace.txt for more information about each type. You will want to enable DYNAMIC_FTRACE, which allows you to only trace functions you are concerned about.

    Once finished, run the following commands from the parent directory :

    $ sudo dpkg -i linux-image-2.6.31-5-386_2.6.31-5.24_i386.deb
$ sudo dpkg -i linux-headers-2.6.31-5-386_2.6.31-5.24_i386.deb
$ sudo update-grub

    Restart your computer, and you should now see the kernel we compiled as a boot option:

    

    The rest is outlined in LWN’s look at ftrace, but here are a few things I noticed:

    • You don’t have to mount a debugfs. There’s already one at /sys/kernel/debug (where it’s recommended to be).
    • I ran into some permission errors when trying to change what the current tracer was. Changing the owner or permissions of that directory should solve this issue.

    To verify that everything is working, when you cat “/sys/kernel/debug/tracing/available_tracers” you should see more options than just “nop”. Enjoy!

    Resources

    • Ubuntu Kernel Compile Page
    • How to compile a custom kernel for Ubuntu Intrepid using git
    • ftrace.txt
    • LWN.net: A look at ftrace

    
     

    ---

    
    
    http://serverfault.com/questions/268037/centos-5-realtime-patch
    
    
    
    
    

    CentOS 5 - realtime patch

    up vote 4 down vote favorite 1 share [fb] share [tw]

    Where can I download precompiled CentOS 5 kernel including realtime patch? Or I need to compile it myself (if yes, how to do that)? linux centos rhel5 patch realtime link|improve this question edited May 15 at 14:39   Iain 20.7k41736 asked May 10 at 6:56   Dima 614 100% accept rate

    2   Have you gone searching through the CentOS forums? A quick Google search yeilded a few hits. Also why do you need the real time patch? Perhaps changing your process scheduler to something like the deadline scheduler might be just as suitable. – Red Tux May 10 at 7:18 @Red Tux: I tried to search and I got only MRG project, but I prefer to use something free. About deadline scheduler: I've never heard of it. I don't know what better for me, I'm implementing server that receiving big number of UDP packets per seconds and I got missing packets even I set receiver thread priority to 99 with SCHED_FIFO. I think, to use real-time path will be enough? – Dima May 10 at 18:06 This is right in the sweet-spot for a realtime kernel. You will notice a difference. The benefit of the solution I posted below is that you will be able to test the realtime kernel without making any major changes to your operating system installation. RHEL MRG is an add-on to the base Enterprise Linux product. – ewwhite May 10 at 18:14 Was this post useful to you?

    1 Answer

    active oldest votes

    up vote 2 down vote accepted

    While I own a few RHEL MRG licenses, I've started using the FREE realtime recompile of Red Hat MRG from Scientific Linux to convert my CentOS 5 servers to realtime. You can view the Scientific Linux MRG distribution at: http://glitesoft.cern.ch/cern/mrg/slc5X/x86_64/RPMS/ In order to install, I create a small repo file named slc5x-mrg-realtime.repo and drop it in /etc/yum.repos.d. The file contains: [slc-mrg-realtime] name = SLC5X - MRG Realtime baseurl = http://linuxsoft.cern.ch/cern/mrg/slc5X/$basearch/RPMS enabled = 1 protect = 0 gpgkey = http://glitesoft.cern.ch/cern/slc44/x86_64/RPM-GPG-KEYs/RPM-GPG-KEY-cern gpgcheck = 1 From there, I can simply run yum grouplist and see the MRG component groups: MRG Realtime MRG Grid MRG Management MRG Messaging A yum groupinstall "MRG Realtime" && yum install kernel-rt-devel gives me everything I need for realtime operation. You can also just pick and choose the files you want from the distro's rpm listing page at: http://glitesoft.cern.ch/cern/mrg/slc5X/x86_64/RPMS/ Another option is to perform a recompile of Red Hat's SRPMS available at: http://ftp.redhat.com/pub/redhat/linux/enterprise/5Server/en/RHEMRG/SRPMS/ It's the standard rpmbuild --rebuild package.srpm process. This takes more time, but I've done this successfully as well. link|improve this answer edited May 15 at 14:33 answered May 10 at 9:58 ewwhite 10.1k1931

    I prefer to use free SW (not commercial MRG) How can I know, which RT path I need for CentOS 5.5 kernel? My Kernel version is: 2.6.18-194.26.1.el5, as I see, the latest version of the RT patch for 2.6.18 is patch-2.6.18-rt7. Is it OK to use use it? – Dima May 15 at 13:19 1   Please reread my comment. I've provided you two free solutions. Scientific Linux IS free! It has a clone of RHEL MRG in the sane manner that CentOS is a clone of RHEL. MRG is a new kernel and a set of utilities, not a patch. – ewwhite May 15 at 14:32