1 Linux Development ToolsKevin Dankwardt, Ph.D. K Computing
2 Overview Tools helpful to developers using C when writing a network based application ... other languages may sometimes benefit, too Free and Open Source tools only Quick look at a number of tools plus info on how to dig deeper We demo and dig deeper with a few tools Objectives: let you know about the existence and use of a wide variety of tools save you time in finding out more
3 Tutorial Files and NotesTo get a copy of these tutorial notes, the examples used, updated slides, and more info:
4 gcc C compiler and “front-end” to the tool chainBe sure to look at the man (about 185 pages) and info (about 527 pages) pages Has about 1911 command line options man gcc |col -b |tr “ “ “\n” |tr “\t” “\n”|sort|uniq|grep -e “^-”|wc -l Some tools (profiling, debugging, code coverage, ...) require special options when compiling and/or linking. Many tools provide inaccurate results if optimization is used Version matters. Read the README with software.
5 Toolchains Matrix from www.kegel.com/crosstoolToolchains don't all work
6 gdb the GNU debugger. ubiquitous info page is about 444 pages longdebug live (step through, breakpoints, ...) and dead (core dump) processes debug remote targets with gdbserver attach to running processes (gdb -p
7 debuginfo RPMs Often needed for stack traces (backtrace) in gdbyum -y --enablerepo=core-debuginfo install glibc-debuginfo.i686 Often needed for stack traces (backtrace) in gdb yum install $(ldd /usr/bin/whatever | cut -d ' ' -f 3 | sort | uniq | xargs rpm -qf --qf '%{SOURCERPM}\n' | sort | uniq | sed -e 's/-[0-9].*$/-debuginfo/g') See: from: Program received signal SIGSEGV, Segmentation fault. 0x004db27a in _IO_vfscanf_internal () from /lib/libc.so.6 (gdb) bt #0 0x004db27a in _IO_vfscanf_internal () from /lib/libc.so.6 #1 0x004e01fb in scanf () from /lib/libc.so.6 #2 0x in get_num () at error.c:7 #3 0x b in main () at error.c:22
8 DDD GNU's Data Display DebuggerFront-end to other debuggers, such as GDB Click on variables in code to display, click on fields in display to draw a linked list ...
9 insight Red Hat/Cygnus project – GUI debugger integrated into GDBInstall from tarball
10 gprof GNU preformance profiler Build with -pg and -g flagsrun the program gprof -l
11 Profiling Profiling is finding "hotspots" in one's codeCode often exhibits the 80/20 rule - 80% of the time is spent in 20% of the code Profiling helps one to find which 20% Profiling is normally done after a program is performing correctly Profiling often times incurs significant overhead
12 oprofile Need vmlinux file (install kernel source and build)Profiles kernel and user space Can use hardware performance counters commands: opcontrol, opreport, opannotate, opgprof
13 LTTng Linux Trace Tool – Next GenerationTrace kernel and user space tracing Low overhead: usable in ISR and even in NMIs Precise enough for real-time measuring GUI viewer – LTTV Need to patch your kernel
14 systemtap Write scripts that extract data from the kernel to help diagnose performance or functional problems. Script is translated into a kernel loadable module Can trace kernel functions by name, system calls, instructions. Can set timers probe for function entry probe for function exits.
15 gcov GNU tool Provides code coverage reportsDon't optimize code when compiling Annotates source with the count for the number of times each line was executed.
16 Mpatrol Dynamically linked library that can be dynamically reconfigured to log dynamic memory operations. Memory allocations listed by size The function that allocated them. List of memory leaks Redefines a large set of memory related functions, not just malloc/free.
17 valgrind Program for debugging and profiling programs. Works on X86Language independent - runs on the executable. Has been used with C, C++, Java, Perl, Python, x86 assembly, Fortan and others. Can start GDB and attach to the program when it detects a problem. GUI front-ends are available. Helps with memory related bugs. ==15308== Invalid read of size 4 ==15308== at 0x : main (lotsa_errs.c:15) ==15308== Address 0x3C01F024 is 0 bytes inside a block of size 4000 free'd ==15308== at 0x3C01B739: free (vg_replace_malloc.c:127) ==15308== by 0x804846D: main (lotsa_errs.c:14)
18 Dmalloc library debug library to replace malloc(), realloc(), calloc(), free() and other memory allocation functions. Helps to detect memory leaks and other problems. No man page, use info page. Include dmalloc.h link -ldmalloc Set environment variable DMALLOC_OPTIONS before running the program
19 Using Dmalloc Put the following function definition in ~/.bashrc and “source” it. function dmalloc { eval `command dmalloc -b $*`; } Run dmalloc function with options like “-l logfile -i 100 high”. This sets the environment variable. Run the program, the file “logfile” will contain the report. $ dmalloc -l logfile -i 100 high $ gcc -g bad_stuff.c -ldmalloc -o bad_stuff $ ./bad_stuff $ cat logfile
20 Electric Fence Provides support for finding code thatreferences beyond the bounds of malloc'd buffers or references within freed buffers Provides both read and write checking Makes use of VM hardware support Ordinarily a part of Fedora Core Uses hardware (MMU) to detect problems so it can be fast. May allocate many extra pages so can consume a lot of memory.
21 Using ElectricFence Compile program with debugging and ElectricFence library cc -g prog.c libefence.a -o prog -lpthread Run program If program segfaults - i.e., ends with "Segmentation Fault" you can use a debugger to find the line that caused the fault Alternatively, you can set up a signal hander for SIGSEGV or run the program from within a debugger There may also be an “ef” program.
22 strace Trace system callsShows parameter list and return value (with errno code when there's an error) Can show timing information Great for helping to debug applications that do a poor job with error messages. Can trace a process that is already running
23 strace output stace -e open date 2>&1 | grep \libopen("/lib/librt.so.1", O_RDONLY) = 3 open("/lib/libc.so.6", O_RDONLY) = 3 open("/lib/libpthread.so.0", O_RDONLY) = 3 open("/usr/lib/locale/locale-archive", O_RDONLY|O_LARGEFILE) = 3
24 ltrace Trace library calls Can also trace system callsShow the system calls that a library call makes Trace a script to trace shell builtins
25 vmstat Report system information about:processes, memory, paging, block IO, traps, and cpu activity procs memory swap io system-- r b swpd free buff cache si so bi bo in cs -----cpu------ us sy id wa st
26 /proc/diskstats An example proc file #reads#reads/writes merged with adjacent #sectors read #writes completed #sectors written #milliseconds spent writing #I/Os currently in progress and more ...
27 iptraf Network statisticsTCP connections, packet/byte counts, traffic breakdown Activity of other hosts
28 Wireshark (Ethereal) Packet Snifferyum install wireshark and gnome-wireshark
29 netstat Network statistics Connections Routing Interface statisticsPorts being listened on
30 free Amount of free and used memory in the systemtotal used free shared buffers cached Mem: -/+ buffers/cache: Swap:
31 frysk fcatch fcore fhpd fparser fstack fstep ftrace
32 pstack pstack output of top command processPrint stack trace of running process pstack output of top command process #0 0x00bb2402 in __kernel_vsyscall () #1 0x00ce909d in ___newselect_nocancel () from /lib/libc.so.6 #2 0x in ?? () #3 0x00c38f2c in __libc_start_main () from /lib/libc.so.6 #4 0x in ?? ()
33 ctags/etags Provides a mechanism to locate the definition of C function declarations ctags file_list etags file_list
34 vim with tags Syntax Highlighting :syntax on :syntax off Tags supportvim -t tag_name - switch to a tag :tag tag_name - switch to a tag Control-] - switch to tag under cursor Control-T - return to previous location :tn - switch to next occurrence of a tag :ts - list all occurrences of a tag
35 cscope Interactive tool for browsing C sourcecodeLocate functions, function calls, macros, variables, preprocessor symbols Functions called and called by Search for extended regular expressions
36 splint Secure Programming Lint Static C program checker A better LintUnused declarations, type inconsistencies, etc. Can get greater checking by adding annotations to the source. Annotations about: whether a pointer can be null, mutability, abstract types and many more
37 doxygen Documentation generatorExtracts information from source file comments Output can be HTML, RTF, PDF, LaTex, Postscript or man pages
38 Summary