Understanding how Linux works under the hood (strace/lsof/proc...)

This page is designed to expand and explore some of the more intresting systems calls and tools avaliable. Which can be of use in troubleshooting and general sys admin tasks.

It is only a drop in the ocean of Understanding how Linux works but it is hopefully an intresting read.

It will bring together information,links to other pages on the site and useful external links.

What happens when you type the free command
File systems
Processes and threads
More reading

What happens when you type the free command

Ever wondered some of the steps when you type a command into linux.
Lets take the free command and follow the typical steps (after you have hit enter):

  • local Alias are checked (run alias to check yourself what is there)
  • you PATH vairable is searched for the command(run echo $PATH to check where it is going to look, which ls will search for you.)
  • free is found and run (in this case a binary) file
  • it queries/run against proc filesystem
  • you get the results

File systems

File system are a large area to cover but in relation to this page...

some user processes/commands will check the local filesystem (i.e storage) i.e cat /etc/sysconfig
some user processes/commands will check the virtual file system (proc and pipefs)

Proc - For an explanation see further down this document.

PipeFS is a unique virtual filesystem. This filesystem is mounted inside the kernel rather than in the userspace. While most filesystems are mounted under the root patition, PipeFS is mounted on "pipe:", making PipeFS its own root .
The purporse is to allow the pipe() system-call to make a new pipe object on the filesystem. Without this filesystem, pipes cannot be made. Also, threads and forks communicate together via pipes. Without PipeFS, processes could not fork and threads could not communicate.

Processes and threads



You cannot talk to the kernel but you can query proc (where the kernel posts information) normally its easyier to get this data through commands such as top,uptime,free ... (which will query files such as /proc/uptime)

Each numbered directory in /proc is a pid within each file the size of the files is zero as the files are updated on the fly (created when read) it can show you a lot of raw information about processes straight from the kernel.

Here a few of the examples of what can be found within the /proc/n/ directory

cmd - what the commands is currently running
cmdline- how the process was called (the command that was issued)
cwd - link to where the process is operating from (current working directory)
environ - environment variables - the file contains the initial environment of the process so for example the following will show you the mysql values (the tr command makes it more readable) cat /proc/`pidof mysqld`/environ | tr \\0 \\n
ls -l /proc/30060/exe # shows which command has been run
ls -l /proc/30060/cwd To find out current working directory of a process or see pdwx
maps -memory mapping information (shared libraries)
statm - memory
fd -file descriptors
root - a link to the root directory of the process
status -the status of the process ( a lot of commands i.e top use this) (it interrupts the stat file)
/proc/sys/kenel/ -allows you to realtime edit/disable kernel parameters, including sysrq

The lsof -p n command can display the information in an easier way

A variety of network information and data is available in the /proc/net/ directory

each of the files within /proc/net contains information about some aspect of Linux networking
The /proc/net/dev file contains information about the configured network interfaces

interesting page on cat /proc/meminfo http://www.embedded-bits.co.uk/2011/cat-procmeminfo-memtotal/ & http://www.centos.org/docs/5/html/5.1/Deployment_Guide/s2-proc-meminfo.html

more details on proc and /proc/sys/vm


Some useful info at

unamed pipes i.e ls -l | less (reduce the need for a temp file in processing)

named pipes (sometimes called a fifo, the order of bytes going in is the same coming out) file that exist in the file system

you can create a named pipe with mkfifo mypipe
doing a ls -l on mypipe will show the filetype as a p

names pipes are useful if multiple programs need to call the same input useful if you want validation for example to pass a passcode to a program (see https://www.youtube.com/watch?v=Zj7Jh1f47do)
or if you want to zip all files passed to it allow totally unrelated programs to communicate with each other. For example, a program that services requests of some sort (print files, access a database) could open the pipe for reading. Then, another process could make a request by opening the pipe and writing a command.

In linux you can show pipes by running
lsof | grep pipe
init 1 root 3r FIFO 0,8 0t0 6373 pipe
init 1 root 4w FIFO 0,8 0t0 6373 pipe
httpd 531 apache 7r FIFO 0,8 0t0 8608212 pipe
httpd 531 apache 8w FIFO 0,8 0t0 8608212 pipe

Pipes self test

presumption its a fresh system i.e no one else is using cat or date if not you may have to change steps 4 and 5
1) create a pipe mkfifo mypipe
2) create the pipe - in terminal 1 run watch "date" >mypipe
3) open the pipe - in terminal 2 run cat <mypipe
4) check 1 end of the pipe - in terminal 3 run lsof -p `ps -ef | grep "watch date" | grep -v grep | awk '{print $2}'` and you should see a line like watch 8543 root 1w FIFO 252,1 0t0 394968 /root/mypipe
5) check 2nd end of the pipe in terminal 4 run lsof -p `ps -ef | grep "cat" | grep -v grep | awk '{print $2}'` and you should see a line like cat 8537 root 0r FIFO 252,1 0t0 394968 /root/mypipe


A socket represents a single connection between two network applications. These two applications nominally run on different computers, but sockets can also be used for interprocess communication on a single computer. Applications can create multiple sockets for communicating with each other. Sockets are bidirectional, meaning that either side of the connection is capable of both sending and receiving data.
It provides a interface for applications to utilize a system resource


An intresting tool pv

Details on using strace and lsof to investigate and troubleshoot processes can be found here https://linuxblog.info/strace/

More reading