Best Linux Command-Line Tools For Network Engineers

These Linux utilities come in handy when designing, implementing or troubleshooting a network.

Trends like open networking and adoption of the Linux operating system by network equipment vendors require network administrators and engineers to have a basic knowledge of Linux-based command-line utilities.

When I worked full-time as a network engineer, my Linux skills helped me with the tasks of design, implementation, and support of enterprise networks. I was able to efficiently collect information needed to do network design, verify routing and availability during configuration changes, and grab troubleshooting data necessary to quickly fix outages that were impacting users and business operations. Here is a list of some of the command-line utilities I recommend to network engineers.

NMAP

Nmap is the network security scanner of choice. It can give you useful information about what’s running on network hosts. It’s also so famous that it has been featured in many movies. With Nmap you can, for example, scan and identify open and filtered TCP/IP ports, check what operating system is running on a remote host, and perform a ping sweep on an IP subnet or range.

List open ports on a host

Knowing which TCP/IP ports of a host are listening for incoming connections is crucial, especially when you’re hardening a server or locking down network equipment. Nmap allows you to quickly verify that; just run the Nmap command followed by the hostname or fully qualified domain name.

Linux 1.png

In this example, we have host 10.1.10.1 with MAC address C4:04:12:BE:1A:2C and open ports 80 and 443.

Some useful options are:

-O                    Enable operating system detection

-p                     Port range (e.g. -p22-123)

-sP                   Ping sweep of a subnet (e.g. 192.168.0.0/24) or range of hosts

Ping sweep on a IPv4 subnet

Ping sweeps are great for creating an inventory list of hosts in a network. Use this technique with caution and don’t simply scan the entire 10.0.0.0/8 subnet. Rather, go subnet per subnet (e.g. 10.1.1.0/24). I used this option many times while replacing the routers at large sites. I would create an IP inventory list before and after my configuration change to make sure that all the hosts would see the new gateways and could reach the outside world.

Linux 2.png

Real-time ping sweeps

Do you want a real-time ping sweep of a subnet? The following bash script will continuously execute a ping sweep to subnet 192.168.1.0/24 every five seconds. To exit the command, just hit CTRL-C.

while [ `clear` ]; do nmap -sP 192.168.1.0/24; sleep 5; done

TCPDUMP

Tcpdump is the tool that you want to use to analyze traffic sourced or destined to your own host or to capture traffic between two or more endpoints (also called sniffing). To sniff traffic, you will need to connect the host running tcpdump to a SPAN port (also called port mirroring), a hub (if you can still find one), or a network tap. This will allow you to intercept and process all captured traffic with tcpdump. Just execute the command with the -i option to select what interface to use (eth0), and the command will print all traffic captured:

tcpdump -i eth0

Tcpdump is a great utility to troubleshoot network and application issues. For example, at remote sites connected with IPsec tunnels back to the main site, I was often able to figure out why some applications would make it through the tunnel and some wouldn’t. Specifically, I noticed that applications using the entire IP payload and also enabling the DF (don't fragment) setting, would fail.

The root cause was that the addition of the IPsec header, required by the VPN tunnel, would cause the overall packet to be larger than the maximum transmission unit (MTU) allowed to pass through the tunnel. As result, the router was discarding these oversized packets and sending back ICMP packets with the “Can't Fragment Error” code. This is something I discovered while listening to the wire with tcpdump.

Here are some basic options that you should know about when using tcpdump:

tcpdump src 192.168.0.1

Capture all traffic from host 192.168.0.1

tcpdump dst 192.168.0.1

Capture all traffic destined to host 192.168.0.1

tcpdump icmp

Capture all ICMP traffic

tcpdump src port 80

Capture all traffic sourced from port 80

tcpdump dst port 80

Capture all traffic destined to port 80

IPERF

Use iperf  to assess the bandwidth available between two computers. You can choose between TCP or UDP traffic and set the destination port, bandwidth rate (if UDP is selected), DSCP marking, and TCP window size. The UDP iperf test can also be used to generate multicast traffic and test your PIM infrastructure.

I’ve used iperf many times to troubleshoot bandwidth issues, verify whether the ISP would honor the DSCP marking, and estimate the jitter value of VoIP traffic.

HPING3

Hping3 is a utility command very similar to ping, with the difference that it can use TCP, UDP, and RAW-IP as transport protocols. Hping3 allows you to not only test whether a specific TCP/IP port is open, but also measure the round-trip time. For example, if you want to test whether google.com has port 80 open and measure the round-trip time, you can type:

Linux 6.png

Here are the options I used:

-S                    Set the SYN tcp flag

-V                    Enable verbose output and display more information about the replies

-p                     Set the TCP/IP destination port

NETCAT

Netcat (nc) is the network engineer’s Swiss Army knife. If you want to be the MacGyver of your network, you must know the basics of netcat. If you use it in client mode, it’s similar to telnet; you can create a TCP connection to a specific port and send anything that you type. You can also use it to open a TCP/IP port and read from standard input. That makes it an easy way to transfer files between two computers. Another use case is testing whether your firewall is blocking certain traffic. For example, execute netcat in server mode on a host behind your firewall and then execute netcat in client mode from outside the firewall. If you can read on the server whatever you type on the client, then the firewall is not filtering the connection.

nc -l -p 1234

This executes netcat in server mode on port 1234 and waits for incoming connections

nc destination_host 1234

This executes netcat in client mode and connects to TCP port 1234 on remote host destination_host

You can also use netcat with pipe commands. For example, you can compress a file before sending it to the remote host with netcat:

tar cfp - /some/dir | compress -c | nc -w 3 othermachine 1234

I hope this blog post provided some useful Linux tricks that will make your life easier. If you have other Linux command line utilities in your toolbox, please feel free to share them in the comment section below.

A tcpdump Tutorial and Primer

Courtesy -  Daniel Miesslwer

Image from securitywizardry.com

 

tcpdump is the premier network analysis tool for information security professionals. Having a solid grasp of this über-powerful application is mandatory for anyone desiring a thorough understanding of TCP/IP. Many prefer to use higher level analysis tools such as Ethereal Wireshark, but I believe this to usually be a mistake.

In a discipline so dependent on a true understanding of concepts vs. rote learning, it's important to stay fluent in the underlying mechanics of the TCP/IP suite. A thorough grasp of these protocols allows one to troubleshoot at a level far beyond the average analyst, but mastery of the protocols is only possible through continued exposure to them.

When using a tool that displays network traffic a more natural (raw) way the burden of analysis is placed directly on the human rather than the application. This approach cultivates continued and elevated understanding of the TCP/IP suite, and for this reason I strongly advocate using tcpdump instead of other tools whenever possible.

15:31:34.079416 IP (tos 0x0, ttl 64, id 20244, offset 0, flags [DF], proto: TCP (6), length: 60) source.35970 > dest.80: S, cksum 0x0ac1 (correct), 2647022145:2647022145(0) win 5840 0x0000: 4500 003c 4f14 4000 4006 7417 0afb 0257 E.. 0x0010: 4815 222a 8c82 0050 9dc6 5a41 0000 0000 H."*...P..ZA.... 0x0020: a002 16d0 0ac1 0000 0204 05b4 0402 080a ................ 0x0030: 14b4 1555 0000 0000 0103 0302 ...U........

Options

Below are a few options (with examples) that will help you greatly when working with the tool. They're easy to forget and/or confuse with other types of filters, i.e. ethereal, so hopefully this page can serve as a reference for you, as it does me.

First off, I like to add a few options to the tcpdump command itself, depending on what I'm looking at. The first of these is -n, which requests that names are not resolved, resulting in the IPs themselves always being displayed. The second is -X, which displays both hex and ascii content within the packet. The final one is -S, which changes the display of sequence numbers to absolute rather than relative. The idea there is that you can't see weirdness in the sequence numbers if they're being hidden from you. Remember, the advantage of using tcpdump vs. another tool is getting manual interaction with the packets.

It's also important to note that tcpdump only takes the first 68 96 bytes of data from a packet by default. If you would like to look at more, add the -s number option to the mix, where number is the number of bytes you want to capture. I recommend using 0 (zero) for a snaplength, which gets everything. Here's a short list of the options I use most:

• -i any : Listen on all interfaces just to see if you're seeing any traffic.
• -n : Don't resolve hostnames.
• -nn : Don't resolve hostnames or port names.
• -X : Show the packet's contents in both hex and ASCII.
• -XX : Same as -X, but also shows the ethernet header.
• -v, -vv, -vvv : Increase the amount of packet information you get back.
• -c : Only get x number of packets and then stop.
• -s : Define the snaplength (size) of the capture in bytes. Use -s0 to get everything, unless you are intentionally capturing less.
• -S : Print absolute sequence numbers.
• -e : Get the ethernet header as well.
• -q : Show less protocol information.
• -E : Decrypt IPSEC traffic by providing an encryption key.

[ The default snaplength as of tcpdump 4.0 has changed from 68 bytes to 96 bytes. While this will give you more of a packet to see, it still won't get everything. Use -s 1514 to get full coverage ]

Basic Usage

So, based on the kind of traffic I'm looking for, I use a different combination of options to tcpdump, as can be seen below:

1. Basic communication // see the basics without many options
   
   # tcpdump -nS


2. Basic communication (very verbose) // see a good amount of traffic, with verbosity and no name help
   
   # tcpdump -nnvvS


3. A deeper look at the traffic // adds -X for payload but doesn't grab any more of the packet
   
   # tcpdump -nnvvXS


4. Heavy packet viewing // the final "s" increases the snaplength, grabbing the whole packet
   
   # tcpdump -nnvvXSs 1514

Here's a capture of exactly two (-c2) ICMP packets (a ping and pong) using some of the options described above. Notice how much we see about each packet.

hermes root # tcpdump -nnvXSs 0 -c2 icmp
tcpdump: listening on eth0, link-type EN10MB (Ethernet), 23:11:10.370321 IP 
(tos 0x20, ttl  48, id 34859, offset 0, flags [none], length: 84) 
69.254.213.43 > 72.21.34.42: icmp 64: echo request seq 0

        0x0000:  4520 0054 882b 0000 3001 7cf5 45fe d52b  E..T.+..0.|.E..+
        0x0010:  4815 222a 0800 3530 272a 0000 25ff d744  H."*..50'*..%..D
        0x0020:  ae5e 0500 0809 0a0b 0c0d 0e0f 1011 1213  .^..............
        0x0030:  1415 1617 1819 1a1b 1c1d 1e1f 2021 2223  .............!"#
        0x0040:  2425 2627 2829 2a2b 2c2d 2e2f 3031 3233  $%&'()*+,-./0123
        0x0050:  3435 3637                                4567
23:11:10.370344 IP (tos 0x20, ttl  64, id 35612, offset 0, flags [none], 
length: 84) 72.21.34.42 > 69.254.213.43: icmp 64: echo reply seq 0
        0x0000:  4520 0054 8b1c 0000 4001 6a04 4815 222a  E..T....@.j.H."*
        0x0010:  45fe d52b 0000 3d30 272a 0000 25ff d744  E..+..=0'*..%..D
        0x0020:  ae5e 0500 0809 0a0b 0c0d 0e0f 1011 1213  .^..............
        0x0030:  1415 1617 1819 1a1b 1c1d 1e1f 2021 2223  .............!"#
        0x0040:  2425 2627 2829 2a2b 2c2d 2e2f 3031 3233  $%&'()*+,-./0123
        0x0050:  3435 3637                                4567
2 packets captured
2 packets received by filter
0 packets dropped by kernel
hermes root # 

Common Syntax

Expressions allow you to trim out various types of traffic and find exactly what you're looking for. Mastering the expressions and learning to combine them creatively is what makes one truly powerful with tcpdump. There are three main types of expression: type, dir, and proto.

Type options are host, net, and port. Direction is indicated by dir, and there you can have src, dst, src or dst, and src and dst. Here are a few that you should definitely be comfortable with:

• host // look for traffic based on IP address (also works with hostname if you're not using -n)
  
  # tcpdump host 1.2.3.4


• src, dst // find traffic from only a source or destination (eliminates one side of a host conversation)
  
  # tcpdump src 2.3.4.5
  # tcpdump dst 3.4.5.6


• net // capture an entire network using CIDR notation
  
  # tcpdump net 1.2.3.0/24


• proto // works for tcp, udp, and icmp. Note that you don't have to type proto
  
  # tcpdump icmp


• port // see only traffic to or from a certain port
  
  # tcpdump port 3389


• src, dst port // filter based on the source or destination port
  
  # tcpdump src port 1025
  # tcpdump dst port 389


• src/dst, port, protocol // combine all three
  
  # tcpdump src port 1025 and tcp
  # tcpdump udp and src port 53

You also have the option to filter by a range of ports instead of declaring them individually, and to only see packets that are above or below a certain size.

• Port Ranges // see traffic to any port in a range
  tcpdump portrange 21-23


• Packet Size Filter // only see packets below or above a certain size (in bytes)
  tcpdump less 32
tcpdump greater 128
[ You can use the symbols for less than, greater than, and less than or equal / greater than or equal signs as well. ]
// filtering for size using symbols
tcpdump > 32
tcpdump <= 128

Writing to a File

tcpdump allows you to send what you're capturing to a file for later use using the -w option, and then to read it back using the -r option. This is an excellent way to capture raw traffic and then run it through various tools later.
The traffic captured in this way is stored in tcpdump format, which is pretty much universal in the network analysis space. This means it can be read in by all sorts of tools, including Wireshark, Snort, etc.

Capture all Port 80 Traffic to a File

# tcpdump -s 1514 port 80 -wcapture_file

Then, at some point in the future, you can then read the traffic back in like so:

Read Captured Traffic back into tcpdump

# tcpdump -r capture_file

Getting Creative

Expressions are nice, but the real magic of tcpdump comes from the ability to combine them in creative ways in order to isolate exactly what you're looking for. There are three ways to do combinations, and if you've studied computers at all they'll be pretty familar to you:

1. AND
   and or &&
2. OR
   or or ||
3. EXCEPT
   not or !

More Examples

# TCP traffic from 10.5.2.3 destined for port 3389

tcpdump -nnvvS and src 10.5.2.3 and dst port 3389

# Traffic originating from the 192.168 network headed for the 10 or 172.16 networks

tcpdump -nvX src net 192.168.0.0/16 and dst net 10.0.0.0/8 or 172.16.0.0/16

# Non-ICMP traffic destined for 192.168.0.2 from the 172.16 network

tcpdump -nvvXSs 1514 dst 192.168.0.2 and src net and not icmp

# Traffic originating from Mars or Pluto that isn't to the SSH port

tcpdump -vv src mars and not dst port 22

As you can see, you can build queries to find just about anything you need. The key is to first figure out precisely what you're looking for and then to build the syntax to isolate that specific type of traffic.

Grouping

Also keep in mind that when you're building complex queries you might have to group your options using single quotes. Single quotes are used in order to tell tcpdump to ignore certain special characters -- in this case the "( )" brackets. This same technique can be used to group using other expressions such as host, port, net, etc. Take a look at the command below:

# Traffic that's from 10.0.2.4 AND destined for ports 3389 or 22 (incorrect)

tcpdump src 10.0.2.4 and (dst port 3389 or 22)

If you tried to run this otherwise very useful command, you'd get an error because of the parenthesis. You can either fix this by escaping the parenthesis (putting a \ before each one), or by putting the entire command within single quotes:

# Traffic that's from 10.0.2.4 AND destined for ports 3389 or 22 (correct)

tcpdump 'src 10.0.2.4 and (dst port 3389 or 22)'

Advanced

You can also filter based on specific portions of a packet, as well as combine multiple conditions into groups. The former is useful when looking for only SYNs or RSTs, for example, and the latter for even more advanced traffic isolation.

[ Hint: An anagram for the TCP flags: Unskilled Attackers Pester Real Security Folk ]

Show me all URGENT (URG) packets...

# tcpdump 'tcp[13] & 32!=0'

Show me all ACKNOWLEDGE (ACK) packets...

# tcpdump 'tcp[13] & 16!=0'

Show me all PUSH (PSH) packets...

# tcpdump 'tcp[13] & 8!=0'

Show me all RESET (RST) packets...

# tcpdump 'tcp[13] & 4!=0'

Show me all SYNCHRONIZE (SYN) packets...

# tcpdump 'tcp[13] & 2!=0'

Show me all FINISH (FIN) packets...

# tcpdump 'tcp[13] & 1!=0'

Show me all SYNCHRONIZE/ACKNOWLEDGE (SYNACK) packets...

# tcpdump 'tcp[13]=18'

[ Note: Only the PSH, RST, SYN, and FIN flags are displayed in tcpdump's flag field output. URGs and ACKs are displayed, but they are shown elsewhere in the output rather than in the flags field ]

Keep in mind the reasons these filters work. The filters above find these various packets because tcp[13] looks at offset 13 in the TCP header, the number represents the location within the byte, and the !=0 means that the flag in question is set to 1, i.e. it's on.
As with most powerful tools, however, there are multiple ways to do things. The example below shows another way to capture packets with specific TCP flags set.

Capture TCP Flags Using the tcpflags Option...

# tcpdump 'tcp[tcpflags] & & tcp-syn != 0'

Specialized Traffic

Finally, there are a few quick recipes you'll want to remember for catching specific and specialized traffic, such as IPv6 and malformed/likely-malicious packets.

 

IPv6 traffic

 

# tcpdump ip6

 

Packets with both the RST and SYN flags set (why?)

 

# tcpdump 'tcp[13] = 6'

 

Traffic with the 'Evil Bit' Set

 

# tcpdump 'ip[6] & 128 != 0'

Conclusion

Well, this primer should get you going strong, but the man page should always be handy for the most advanced and one-off usage scenarios. I truly hope this has been useful to you, and feel free to contact me if you have any questions. ::

Additional Resources and References

¹ http://danielmiessler.com/study/tcpflags/
² http://danielmiessler.com/study/lsof/
³ http://danielmiessler.com/study/find/
⁴ http://danielmiessler.com/study/subnetting/
⁵ http://www.tcpdump.org/tcpdump_man.html

 

A lsof Introduction

lsof is the Linux/Unix über-tool. I use it most for getting network connection related information from a system, but that's just the beginning for this amazing and little-known application. The tool is aptly called lsof because it "lists open files". And remember, in Unix just about everything (including a network socket) is a file.

** lsof is also the Linux/Unix command with the most switches. It has so many it has to use both pluses and minuses.

usage: [-?abhlnNoOPRstUvV] [+|-c c] [+|-d s] [+D D] [+|-f[cgG]]
 [-F [f]] [-g [s]] [-i [i]] [+|-L [l]] [+|-M] [-o [o]]
 [-p s] [+|-r [t]] [-S [t]] [-T [t]] [-u s] [+|-w] [-x [fl]] [--] [names]

As you can see, lsof has a truly staggering number of options. You can use it to get information about devices on your system, what a given user is touching at any given point, or even what files or network connectivity a process is using. lsof replaces my need for both netstat and ps entirely. It has everthing I get from those tools and much, much more.

Show Your Network Connections

Show all connections with -i

lsof -i

COMMAND  PID USER   FD   TYPE DEVICE SIZE NODE NAME
dhcpcd 6061 root 4u IPv4 4510 UDP *:bootpc
sshd 7703 root 3u IPv6  6499 TCP *:ssh (LISTEN)
sshd 7892 root 3u IPv6  6757 TCP 10.10.1.5:ssh->192.168.1.5:49901 (ESTABLISHED)

Show only TCP (works the same for UDP)

lsof -iTCP

COMMAND  PID USER   FD   TYPE DEVICE SIZE NODE NAME
sshd 7703 root 3u IPv6 6499 TCP *:ssh (LISTEN)
sshd 7892 root 3u IPv6 6757 TCP 10.10.1.5:ssh->192.168.1.5:49901 (ESTABLISHED)

-i :port shows all networking related to a given port

lsof -i :22

COMMAND  PID USER   FD   TYPE DEVICE SIZE NODE NAME
sshd 7703 root 3u  IPv6 6499 TCP *:ssh (LISTEN)
sshd 7892 root 3u  IPv6 6757 TCP 10.10.1.5:ssh->192.168.1.5:49901 (ESTABLISHED)

To show connections to a specific host, use @host

lsof -i@192.168.1.5

sshd 7892 root 3u IPv6 6757 TCP 10.10.1.5:ssh->192.168.1.5:49901 (ESTABLISHED)

Show connections based on the host and the port using @host:port

lsof -i@192.168.1.5:22

sshd 7892 root 3u IPv6 6757 TCP 10.10.1.5:ssh->192.168.1.5:49901 (ESTABLISHED)

Grepping for "LISTEN" shows what ports your system is waiting for connections on

lsof -i| grep LISTEN

iTunes     400 daniel   16u  IPv4 0x4575228  0t0 TCP *:daap (LISTEN)

Grepping for "ESTABLISHED" shows current active connections

lsof -i| grep ESTABLISHED

firefox-b 169 daniel  49u IPv4 0t0 TCP 1.2.3.3:1863->1.2.3.4:http (ESTABLISHED)

Working with Users, Processes, and Files

You can also get information on various users, processes, and files on your system using lsof:

Show what a given user has open using -u

lsof -u daniel

-- snipped --
Dock 155 daniel  txt REG   14,2   2798436   823208 /usr/lib/libicucore.A.dylib
Dock 155 daniel  txt REG   14,2   1580212   823126 /usr/lib/libobjc.A.dylib
Dock 155 daniel  txt REG   14,2   2934184   823498 /usr/lib/libstdc++.6.0.4.dylib
Dock 155 daniel  txt REG   14,2    132008   823505 /usr/lib/libgcc_s.1.dylib
Dock 155 daniel  txt REG   14,2    212160   823214 /usr/lib/libauto.dylib
-- snipped --

See what files and network connections a command is using with -c

lsof -c syslog-ng

COMMAND    PID USER   FD   TYPE     DEVICE    SIZE       NODE NAME
syslog-ng 7547 root  cwd    DIR    3,3    4096   2 /
syslog-ng 7547 root  rtd    DIR    3,3    4096   2 /
syslog-ng 7547 root  txt    REG    3,3  113524  1064970 /usr/sbin/syslog-ng
syslog-ng 7547 root  mem    REG    0,0   0 [heap] 
syslog-ng 7547 root  mem    REG    3,3  105435   850412 /lib/libpthread-2.4.so
syslog-ng 7547 root  mem    REG    3,3 1197180   850396 /lib/libc-2.4.so
syslog-ng 7547 root  mem    REG    3,3   59868   850413 /lib/libresolv-2.4.so
syslog-ng 7547 root  mem    REG    3,3   72784   850404 /lib/libnsl-2.4.so
syslog-ng 7547 root  mem    REG    3,3   32040   850414 /lib/librt-2.4.so
syslog-ng 7547 root  mem    REG    3,3  126163   850385 /lib/ld-2.4.so
-- snipped --

Pointing to a file shows what's interacting with that file

lsof /var/log/messages

COMMAND    PID USER   FD   TYPE DEVICE   SIZE   NODE NAME
syslog-ng 7547 root    4w   REG    3,3 217309 834024 /var/log/messages

The -p switch lets you see what a given process ID has open, which is good for learning more about unknown processes

lsof -p 10075

-- snipped --
sshd    10068 root  mem    REG    3,3   34808 850407 /lib/libnss_files-2.4.so
sshd    10068 root  mem    REG    3,3   34924 850409 /lib/libnss_nis-2.4.so
sshd    10068 root  mem    REG    3,3   26596 850405 /lib/libnss_compat-2.4.so
sshd    10068 root  mem    REG    3,3  200152 509940 /usr/lib/libssl.so.0.9.7
sshd    10068 root  mem    REG    3,3   46216 510014 /usr/lib/liblber-2.3
sshd    10068 root  mem    REG    3,3   59868 850413 /lib/libresolv-2.4.so
sshd    10068 root  mem    REG    3,3 1197180 850396 /lib/libc-2.4.so
sshd    10068 root  mem    REG    3,3   22168 850398 /lib/libcrypt-2.4.so
sshd    10068 root  mem    REG    3,3   72784 850404 /lib/libnsl-2.4.so
sshd    10068 root  mem    REG    3,3   70632 850417 /lib/libz.so.1.2.3
sshd    10068 root  mem    REG    3,3    9992 850416 /lib/libutil-2.4.so
-- snipped --

The -t option returns just a PID

lsof -t -c Mail

350

ps aux | grep Mail

daniel 350 0.0 1.5 405980 31452 ?? S  Mon07PM 2:50.28 /Applications/Mail.app

Advanced Usage

Using-a allows you to combine search terms, so the query below says, "show me everything running as daniel connected to 1.1.1.1"

lsof -a -u daniel -i @1.1.1.1

bkdr   1893 daniel 3u  IPv6 3456 TCP 10.10.1.10:1234->1.1.1.1:31337 (ESTABLISHED)

Using the -t and -c options together you can HUP processes

kill -HUP `lsof -t -c sshd`

You can also use the -t with -u to kill everything a user has open

kill -9 `lsof -t -u daniel`

lsof +L1 shows you all open files that have a link count less than 1, often indicative of a cracker trying to hide something

lsof +L1

(hopefully nothing)

References

The lsof man page:
http://www.netadmintools.com/html/lsof.man.html