This webpage suggests various steps to determine if your system has been compromised. System Administrators can use this checklist for intrusion detection.

Most of the time it depends on your Organization's policy to decide whether to disconnect the compromised system from the network or not. Intrusion Detection experts generally draw analogy between the compromised network and a crime scene. Some incidents need immediate disconnection from the network, whereas some incidents need further monitoring to catch the culprit. Both approaches have their own pros and cons. Disconnection from the network may prevent further compromise of the system and may save other systems on the network from compromise but this action may not provide enough number of leads to investigate the crime. On the other hand keeping the compromised system connected to the network involves lot of risk and isolation of this system from others should be considered at the minimum. Never underestimate the scope of any incident.

These are the three situations that might arise while verifying the compromised system.

1. Verification without touching the system
2. Verification by touching the system minimally which should be enough to verify the compromise and at the same time avoids alerting the attacker about the detection
3. Verification by full analysis of the compromised system to gather all the possible evidence to prove that the incident in fact occurred

One very important thing for investigators to keep in mind is the need for documenting everything before proceeding to take any action. Document all the actions and commands that are run on the compromised system. This helps in providing the best evidence for crime investigation and to learn lessons from the incident. Without documentation people may forget what they noticed and what actions they performed on the compromised system after a few days.

Now I am going to discuss the Checklist for Linux/Unix systems first and Windows systems next.

Linux/Unix Intrusion Detection Checklist:

Whatever action you perform like powering off, disconnecting from the network, backing up the system or even doing nothing can change state of the system. It is up to your best judgment to decide which action to take. You should also consider notifying the higher management about the compromised system and proposed action.

• Each investigation warrants a different approach. If the breach is serious, strongly consider duplicating the system. This will provide the snapshot of the file system at the time the compromise was first discovered. The best choice is to make two sound duplicate copies of the original drive. One to refer back in the future and the other copy for immediate analysis by the investigators. So keep one of this copy in a secure place.

  dd, dump, cpio and tar are some of the non-commercial tools on the Linux/Unix operating systems for backup. But 'dd' is the only package that you can use for duplicating or cloning the hard drive. Before using any of the tools or any other commands verify the integrity of the system binaries against the distribution media. Attackers generally replace the original binaries with the rootkits so that the user can not notice the changes made to the compromised system and the associated processes. Some of the binaries which are commonly replaced by the attackers are: ls, ps, netstat, ifconfig, find, du, df, libc, sync, xinetd, telnet, in.telnetd, login, syslogd, dmesg, cmp, diff, sum, fdisk, strace, ltrace, mv, dd, dig, nc (netcat), lsof, strings, last, des, uptime, md5sum, rm, cat, more, less, icat, pcat, tar, gunzip, gzip, compress, uncompress, bash, sh, vi, w, lsmod, modinfo, perl, gdb and syslogd. These trojaned binaries rely on configuration files, which are often found under '/dev' directory. So, look for unexpected ASCII files under this directory.

  In order to duplicate the system we can use the 'dd' command, as I have mentioned earlier. Before proceeding with the 'dd' command we have to verify the authenticity of this binary file. The package that provides 'dd' command can be determined using the following command.

  # rpm -q --whatprovides /bin/dd
  fileutils-4.1.4

  RPM database maintains a record of each file present inside the installed packages. The record contains the file size, ownership, permissions, modification time, and MD5 checksum of each file and other information. With the help from these records, you can determine which files, if any, have been modified since installation of the package. For instance to verify the changes made to the files provided by the fileutils package use the following command.

  # rpm -V fileutils
  S.5....T c /bin/dd

  This tells us if the size, MD5 checksum, modification times of the configuration file 'dd' have changed since the installation of the 'fileutils' package. You can compare against package files from the original distribution media rather than the database, if you prefer. This option is useful if you suspect the database may be corrupt. Before doing that you would like to verify the signature on the package file present in the distribution media with the help of the following command.

  # rpm --checksig /mnt/cdrom/RedHat/RPMS/fileutils-4.1-4.i386.rpm
  fileutils-4.1-4.i386.rpm: md5 OK

  This verified the MD5 checksum of the package file present in the cdrom. Now you can use the following command to verify the installed package on the system with the package on the cdrom.

  # rpm -Vp /mnt/cdrom/RedHat/RPMS/fileutils-4.1-4.i386.rpm
  S.5....T c /bin/dd

  This shows that they differ. So it is not a good idea to proceed with the existing 'dd' command. Uninstall and reinstall the 'fileutils' package.

  'dd' is low-level command designed for copying bits of information from one place to another and it can copy a file or an entire partition even from stdin to stdout. Before imaging anything make sure that you have clean copy of the media. After that you can start imaging into the clean copy of the media.

  # dd if=/dev/zero of=device
  # dd if=device of=device bs=blocksize

  Keep in mind that the output device could a floppy drive, tape drive, ide hard drive or scsi hard drive.

  If the compromised system does not have any removable media or if you want to image it to a forensic workstation connect both the systems using a Fast Ethernet cable. Setup your forensic workstation to accept incoming connections on a TCP port with netcat. Use 'des' command along with the 'dd' and 'nc' commands, which helps to encrypt and run a checksum on the transferred data.

  # dd if=(local backup media) | des -e -c -k somekeyword | nc -w 3 forensichost 7000 (On the compromised workstation)
  # nc -l -p 7000 | des -d -c -k somekeyword | dd of=(local backup media) (On the forensic workstation)

  Write-protect the backup media, to protect it from tampering. This precaution is necessary if you want to produce this backup media as evidence of the attack.

  If you are not sure about the integrity of the kernel itself at the first place before performing above things it is a good idea to boot from the trusted kernel using a boot floppy and use a known clean copy of each tool as explained above. But keep in mind that by rebooting the machine you will be stopping all the normally running processes and running processes that are caused by the attacker. This can cause problems in evidence gathering.

  You can protect the system's kernel by using LIDS tool from http://www.lids.org website. LIDS enhances system security by reducing the root user's power. It also provides the incident detection and response capabilities.

  Also you should consider running 'tripwire' and 'tiger' scripts periodically on your machine to know what all files are modified, added and deleted. While setting up the 'tripwire', carefully select the directories to track for changes as this tool can generate enormous amount of data, which may not be interesting to go through each time. Make sure that the data is conceivable enough.

• Check for any abnormal processes running on your system. For example, if you are not running 'portmap' service (which corresponds to port number 111 (sunrpc)) for sure, but suddenly notice it to be running on the system, you can investigate everything about this process as explained below. But before doing this strongly consider running any investigative command at the console of the compromised system, instead of generating network traffic unnecessarily which may alert the attacker. Use only trusted tools. 'ps' is the command that shows all the running processes on the machine. From here onwards I will show only the output lines that are relevant to our investigation.

  # ps -efl
  S    UID    PID    PPID    C    SZ    STIME    TTY    TIME        CMD
  S     root   1         0           1    353   19:19      ?        00:00:05    init
  ---
  ---
  S     rpc    2163    1          0    389   19:25       ?        00:00:00    portmap
  ---
  ---
  ---

  The above lines shows that 'portmap' service in fact started and is in running status under the UID 'rpc'. The PID is 2163 and the Parent PID is 1(PID of init). That means the 'init' process started the 'portmap' service when the system started in one of the runlevel.

  The running processes can also be viewed using either 'top' or 'gtop' commands. 'gtop' is the graphical version of 'top', but it is better not to use any X windows-based tool because of vulnerabilities associated with them. Some of them allow the attacker to log keystrokes and also they are processor intensive.

  Now that you know that 'portmap' service is running on the machine, you can use 'netstat' command to know what ports are open, what applications are running on those open ports and which hosts are connected to the victim's machine.

  # netstat -nav
  Active Internet connections (servers and established)
  Proto    Recv-Q    Send-Q    Local Address    Foreign Address    State          PID/Program name
  ---
  tcp        0              0              0.0.0.0:111        0.0.0.0:*                 LISTEN    2163/portmap
  ---
  ---
  udp        0              0             0.0.0.0:111         0.0.0.0:*                                 2163/portmap
  ---
  ---

  As shown above 'netstat' and provides in-depth view of the running processes observed using the 'ps' command. The authenticity and location of the foreign host can be estimated using the 'ping', 'traceroute', 'nslookup', 'dig', 'whois' commands.

  The files used by these running processes can be figured out by using 'lsof' command as shown below.

  # lsof
  COMMAND    PID    USER    FD     TYPE     DEVICE   SIZE     NODE                             NAME
  ---
  ---
  ---
  portmap         2163    root       cwd    DIR      3,5             4096         2                                 /
  portmap         2163    root       rtd      DIR      3,5             4096         2                                 /
  portmap         2163    root       txt      REG    3,5              29724      457348                        /sbin/portmap
  portmap         2163    root       mem   REG    3,5              485171    277508                        /lib/ld-2.2.4.so
  portmap         2163    root       mem   REG    3,5              436784    277526                        /lib/nsl-2.2.4.so
  portmap         2163    root       mem   REG    3,5              5772268  408078                        /lib/i686/libc-2.2.4.so
  portmap         2163    root       mem   REG    3,5              261460    277542                        /lib/libnss_files-2.2.4.so
  portmap         2163    root       0u       CHR    1,3                              66720                         /dev/null
  portmap         2163    root       1u       CHR    1,3                              66720                         /dev/null
  portmap         2163    root       2u       CHR    1,3                              66720                         /dev/null
  portmap         2163    root       3u       IPv4    16934                         UDP                          *:sunrpc
  portmap         2163    root       4u       IPv6    16935                         TCP                           *:sunrpc (LISTEN)
  ---
  ---
  ---

  These lines show which files and libraries are used by the 'portmap' service. You should look for any abnormalities associated with these processes like trying to access log files, system configuration files etc. Also watch the size of these files. Usually some of these doubtful processes sniff the traffic and log the details into some files. In that case the size of these files will increase from time to time. To sniff the traffic the network interface card needs to be placed in promiscuous mode. Detection procedure for a sniffing mode on the compromised system is explained later in this document. If you want to look at only the files a specific foreign host is accessing, use '# lsof -i IP_ADDRESS' command.

• Each running process has a subdirectory under the /proc directory that corresponds to its process ID. That means each subdirectory is named with a number corresponding to its process. These particular subdirectories can supply vital information that is needed for investigation. Continuing our discussion about the 'portmap' service, we know that 2163 is the corresponding PID. File listing on this subdirectory reveals what is inside.

  # ls -la /proc/2163
  total 0
  dr-xr-xr-x     3  rpc   rpc    0 Feb 15 20:34  .
  dr-xr-xr-x   45  root  root  0 Feb 15 17:25  ..
  -r--------        1  root  root  0 Feb 15 20:34  cmdline
  lrwxrwxrwx 1  root  root  0 Feb 15 20:34  cwd -> /
  -r--------        1  root  root  0 Feb 15 20:34  environ
  lrwxrwxrwx 1  root  root  0 Feb 15 20:34  exe -> /sbin/portmap
  dr-x------       2 root  root  0 Feb 15 20:34   fd
  -r--------        1  root  root  0 Feb 15 20:34  maps
  -rw-------      1  root  root  0 Feb 15 20:34  mem
  lrwx------     1  root  root  0 Feb 15 20:34  root -> /
  -r--------       1  root  root  0 Feb 15 20:34  stat
  -r--------       1  root  root  0 Feb 15 20:34  statm
  -r--------       1  root  root  0 Feb 15 20:34  status

  From the above file listing the most significant things are the 'exe' link, 'cmdline' file and 'fd' subdirectory.

  The 'exe' link reveals the name of the program, running process is linked to. The 'cmdline' file shows the command-line arguments used to run an application. Attackers use the file to hide their rogue process by replacing the contents of 'cmdline' file with an innocuous process. When the user executes 'ps' command to see the running process, 'cmdline' file contents show up on the screen hiding the actually process. But of course, you can realize this by running the file listing command on the '/proc/pid' directory as 'exe' link and 'cmdline' file content should be the same. You can get some more insight by looking into the 'fd' subdirectory.

  # ls -la /proc/2163/fd
  total 0
  dr-x------    2   root   root     0   Feb   15   21:30 .
  dr-xr-xr-x  3   rpc    rpc       0   Feb   15   21:27 ..
  lrwx------   1   root   root   64   Feb   15   21:30 0 -> /dev/null
  lrwx------   1   root   root   64   Feb   15   21:30 1 -> /dev/null
  lrwx------   1   root   root   64   Feb   15   21:30 2 -> /dev/null
  lrwx------   1   root   root   64   Feb   15   21:30 3 -> socket:[818]
  lrwx------   1   root   root   64   Feb   15   21:30 4 -> socket:[819]

  The 'fd' or 'file descriptor' subdirectory shows all the files a process had open. The last two lines indicates that 'portmap' service opened two network sockets and listening for incoming connections.

• Apart from investigating the spurious processes running on the compromised system, logged in unauthorized users could be detected using the 'w' command. The 'w' command shows the users logged into the system, the systems they are using for logging in and what they are doing.

  # w
  11:16 PM up 3 days, 8:19, 1 user, load average: 0.33, 0.21, 0.09
  USER      TTY    FROM    LOGIN@    IDLE    JCPU    PCPU    WHAT
  sysuser    tty1     -             11:21am     0.00s     0.02s     0.01s       w

  Here, users 'sysuser' is logged into the system. You can dig more into this using 'finger @localhost' command, which gives the full list of users logged into the system. But, be cautious before enabling the 'finger' service on the system as it is prone to lot of vulnerabilities and also gives too many details about the logged in users to the attackers. Always keep this service disabled.

• The attackers some times place the Network Interface Card in the promiscuous mode to capture the traffic. In that case not only this particular system is compromised, safety of other systems on the network is also threatened. Note that detection of promiscuous NIC is not possible if the machine is rebooted or the machine is running in single user mode. Passwords of all the users who use systems on that particular network might have been sniffed by this NIC. The NIC's status could be detected by using either 'ifconfig' or 'cpm' commands. If you don't have 'cpm' package installed on your system, you can download it from http://www.rpmfind.net website. I have showed output of both these commands below.

  # ifconfig -a
  eth0     Link encap: Ethernet  HWaddr 00:10:4B:D5:2B:A9
              inet addr:192.168.80.80  Bcast:192.168.80.255  Mask:255.255.255.0
              UP BROADCAST RUNNING  PROMISC  MULTICAST MTU:1500 Metric:1
              RX packets:2989  errors:0  dropped:0  overruns:0  frame:0
              TX packets:29  errors:0  dropped:0  overruns:0  carrier:0
              collisions:5  txqueuelen:100
              RX bytes: 510549 (498.6 Kb) TX bytes: 2396 (2.3 Kb)
              Interrupt:5  Base address:0xef00

  lo         Link encap: Local  Loopback
              inet addr:127.0.0.1  Mask:255.0.0.0
              UP LOOPBACK  RUNNING  MTU:16436 Metric:1
              RX packets:612  errors:0  dropped:0  overruns:0  frame:0
              TX packets:612  errors:0  dropped:0  overruns:0  carrier:0
              collisions:0  txqueuelen:0
             RX bytes: 888 (888.0 b)   TX bytes: 888 (888.0 b)

  As shown above the interface 'eth0' is placed in promiscuous mode. Now let us see what the 'cpm' command says about this interface.

  # cpm
  2 network interface found:
  eth0:   ***  IN  PROMISCUOUS  MODE   ***
  lo:  Normal
  1 of them is in promiscuous mode.

  This confirms the status of NIC. To change the status of the NIC to normal, use # ifconfig eth0 -promisc command. Usually if one host gets compromised other hosts on the network may also get compromised as I mentioned earlier. This is especially true for networks where NIS is running or where hosts trust each other through the use of '.rhosts' files and/or '/etc/hosts.equiv' files. Network Administrators can use "Antisniff" software from L0pht Heavy Industries, Inc. to unearth any sniffing NICs on the network.

• To list the files that are been accessed, modified and created (say since the last 2 days) can be listed by using the following commands.

  # find / -atime -2 (List of files accessed since the last 2 days)

  # find / -mtime -2 (List of files modified since the last 2 days)

• Programs that run 'setuid' and 'setgid' files, especially ones that run 'setuid' and 'setgid' to root, are prone to security problems. The commands that are distributed with these permissions set faced so many security breaches. These features allow programs to operate with higher privileges than those of the user running the program. 'setuid' and 'setgid' root programs are also prone to privilege-escalation attacks. These features provide backdoor for attackers. Intruders often leave 'setuid' copies of '/bin/sh' or '/bin/time' to allow them root access at a later time. A setuid root copy of '/bin/ksh' is known to provide root privileges to any user who executes it. This is also known by attackers as a 'rootshell'. You can use the following commands to find out the files on which 'setuid' and 'setgid' permissions are set.

  # find / -perm -2000 -print
  # find / -perm -4000 -print

  Instead, you can combine both the commands as shown below into one expression.

  # find / \ ( -perm -4000 -o -perm -2000 \ ) -print

  You can always be aware about these 'setuid' and 'setgid' files by creating a script under 'cron.daily' or 'cron.weekly' directory, so that you can regularly receive an email from the system.

  # find / \ ( -perm -4000 -o -perm -2000 \ ) -print | mail -s 'setuid and setgid files' myname@myorganization.com

• Attackers usually hide their files and directories by naming them with filenames that start with '.' Also attackers usually use innocuous names to their files and directories. A commonly used technique is to place a hidden directory under a user's directory with an unusual name, something like '..' or '.. ' Once again 'find' command is helpful in unearthing these files.

  # find / -name ".. "
  # find / -name ".*"

  Take a closer look at all these files. If you want to look for hidden files in particular directory, do 'cd .{tab, tab}' after getting into that directory.

• System logs keep track of all the events that take place on the system. Reviewing these log files gives an idea of how the system got compromised and what happened after the system got compromised, etc. Below are the list of log files and the corresponding commands to access them. Each of these log files are very informative.
  • messages: The /var/log/messages file contains most of the messages. Some common message types include authorization failures, failed 'su' commands, hardware errors, such as timeouts, login via ftp, ssh and telnet, mail activity, startups and shutdowns of processes and services. You can issue 'less /var/log/messages' command to access this file. Check this file to see what processes are starting when the system boots up.
  • dmesg: When the system boots up, it displays boot messages on the console. These messages are stored in the kernel ring buffer, which is overwritten when full. You can view the kernel ring buffer by issuing 'dmesg' command. To prevent the loss of potentially important boot messages, the initialization scripts save the contents of the kernel ring buffer in the file /var/log/dmesg, the boot message file, which can be viewed by issuing 'less /var/log/dmesg' command or 'dmesg' command.
  • wtmp: Each time a user succesfully logs in, logs out, or your machine reboots, the wtmp file is modified. This is a binary file, so you need to use 'last' command to access these details.
  • utmp: This is also another binary file under the '/var/run' directory. This file contains information about each user currently logged into the system. This file contents can be read using either 'w' command as explained earlier or using 'who' command.
  • lastlog: This file under the '/var/log' directory contains the details of each user's last login time, system from which they logged in and login port for each user listed in '/etc/passwd' file. Contents of this file can be accessed by using the 'lastlog' command.
  • secure: Every time a connection is established with one of the services managed by 'xinetd', a log message will be appended to this file under /var/log directory. That means this file tracks whenever some one access ftp, telnet or any other service from the system. You can access the latest 'secure' file using the 'more /var/log/secure' command.
  • xferlog: If you are having 'ftp' service running on your system, this file under the /var/log directory contains the information about all the ftp transfers taken place. You should read this file to know the details of the ftp transfer, once you notice that some one has accessed the 'ftp' service by looking at the 'secure' file. The latest 'xferlog' file can be accessed using 'more /var/log/xferlog' command.
  • Apart from these log files look at the web access logs if you are running the web server on your machine. Verify '/var/log/messages' file for 'ssh' logins. If you are doubtful about your employees look into the building physical access logs to know if there is any one inside the building during the attack time, if it happened at some odd time.

• If you want to retrieve time stamps on all the files residing on the system, use the file listing command 'ls' with -u switch to sort according to the access time, -c to sort according to modifications time and according to creation time. Save these commands output to a file and find md5 checksum of it and save that value in secured place. You can save everything into a floppy, but be careful to protect floppy's integrity. You can save the output of the three commands to different files or to the same file as shown below. Here I have used 'script' command to output everything into the same file named 'fileaccess' with timestamp applied after each step using the 'date' command.

  # script fileaccess
  script started, file is fileaccess
  # date
  Tue Feb 26 00:11:25 CST 2002
  # ls -alRu
  # date
  # ls -alRc
  # date
  # ls -alR
  # date
  # md5sum fileaccess
  cf161148903f366f13d65ccae54bae541c03 fileaccess
  # ^d     [ctrl + d]
  # script done, file is fileaccess

  You can use this file 'fileaccess' for further analysis. For example, if you think the attack happened on February 27th, then you can use the following command to list all the files that are accessed, created and modified.

  # grep "Feb 27" fileaccess

  To find all the files with word "sniffer" inside you can use the following command. This command will search entire file system, irrespective of the case of "sniffer".

  # grep -r -i sniffer /

• The 'cron' system lets users schedule jobs for automatic execution at specified times and intervals. The jobs run using the owning user's login. The 'cron' jobs can run on daily, weekly or monthly basis. These jobs are specified under the '/etc' directory inside the 'cron.daily', 'cron.weekly' and 'cron.monthly'. Check for any suspicious files under these directories. Also check the jobs run by already existing cron files. The attackers might have replaced the authentic job by some spurious process. By default, the results of these jobs are sent to it's owner via e-mail. So, don't neglect your e-mail for finding clues. Also these things are logged into '/var/log/cron' file. Also verify files run by 'at'.

• The '/etc/inittab' file determines the processes that should be active at each runlevel. The system reads this file when it boots up. Go through this file to find out any unauthorized modifications. Also go through '/etc/rc*' directories to ensure the same.

• The '/etc/password' file contains the details about user accounts. Each line of this file describes about each user's account. 'root' user is allocated UID of '0', which provides superuser level access to this user. Look if any other users also have '0' as their UID, which is a big security breach. Also check for unauthorized accounts present in this file. Administrator should create password to all the users, databases and to any other processes whose UIDs are listed in this file. The passwords will be placed under the '/etc/shadow' file in an encrypted format.

• Check the percentage of free space on each partition of the hard disk. Continuously depleting free space under any particular partition may tip about some unauthorized process running on the system.

• Look for '+' sign entries and inappropriate non-local host names in '/etc/hosts.equiv', '/etc/hosts.lpd', and in all '.rhosts' files on the system. The hosts with '+' sign before them in these files are trusted by the system and they can access the target system without supplying the password. Also, it is recommended to disable all the 'r' services such as rlogin, rsh, and rcp. Keep the '/etc/hosts.equiv' and '.rhosts' files empty.