Steps for Recovering from a UNIX or NT System Compromise

This document is being published jointly by the CERT Coordination Center and AusCERT (Australian Computer Emergency Response Team). It describes suggested steps for responding to a UNIX or NT system compromise. Your response should be carried out in several stages:

Introduction

A. Before you get started

1. Consult your security policy
2. If you do not have a security policy
3. 1. Consult with management
   2. Consult with your legal counsel
   3. Contact law enforcement agencies
   4. Notify others within your organization
4. Document all of the steps you take in recovering

B. Regain control

1. Disconnect compromised system(s) from the network
2. Copy an image of the compromised system(s)

C. Analyze the intrusion

1. Look for modifications made to system software and configuration files
2. Look for modifications to data
3. Look for tools and data left behind by the intruder
4. Review log files
5. Look for signs of a network sniffer
6. Check other systems on your network
7. Check for systems involved or affected at remote sites

D. Contact the relevant CSIRT and other sites involved

1. Incident Reporting
2. Contact AusCERT - Australian Computer Emergency Response Team
3. Contact the CERT Coordination Center
4. Obtain contact information for other sites involved

E. Recover from the intrusion

1. Install a clean version of your operating system
2. Disable unnecessary services
3. Install all vendor security patches
4. Consult AusCERT advisories and external security bulletins
5. Consult CERT advisories and vendor-initiated bulletins
6. Caution use of data from backups
7. Change passwords

F. Improve the security of your system and network

1. Review security using the UNIX or NT configuration guidelines document
2. Install security tools
3. Enable maximal logging
4. Configure firewalls to defend networks

G. Reconnect to the Internet

H. Update your security policy

1. Document lessons learned from being compromised
2. Calculate the cost of this incident
3. Incorporate necessary changes (if any) in your security policy

Document revision history

Introduction

Note that all actions taken during your recovery from a system compromise should be in accordance with your organization's policies and procedures.

1. Before you get started
2. 1. Consult your security policy
   2. If you do not have a security policy
   3. 1. Consult with management

      2. Depending on how your organization is structured, it may be important to notify management in order to facilitate internal coordination of your recovery effort. Also be aware that intrusions may get the attention of the media.

      3. Consult with your legal counsel

      4. Before you get started in your recovery, your organization needs to decide if pursuing a legal investigation is an option.

         Note that the CERT Coordination Center and AusCERT (Australian Computer Emergency Response Team) are involved in providing technical assistance and facilitating communications in response to computer security incidents involving hosts on the Internet. We do not have legal expertise and cannot offer legal advice or opinions. For legal advice, we recommend that you consult with your legal counsel. Your legal counsel can provide you with legal options (both civil and criminal) and courses of action based on you or your organization's needs.

         It is up to you how you wish to pursue this incident. You may wish to secure your systems or to contact law enforcement to investigate the case.

         If you are interested in determining the identity of or pursuing action against the intruder, we suggest that you consult your management and legal counsel to see if any local, state, or federal laws have been violated. Based on that, you could then choose to contact a law enforcement agency and see if they wish to pursue an investigation.

         We encourage you to discuss the root compromise activity with your management and legal counsel to answer the following questions:

         • What is your legal status in terms of your ability to trap intruders or trace connections (i.e., do you have a login banner stating that connections can be tracked or traced? See CERT Advisory CA-1992-19, "Keystroke Logging Banner").
         • What are your legal responsibilities if your site is aware of the activity and does not take steps to prevent it?
         • Have any local, state, or federal laws been violated?
         • Should an investigation be pursued?
         • Should you report the activity to local, state, or national) law enforcement?
      5. Contact law enforcement agencies

      6. In general, if you are interested in pursuing any type of investigation or legal prosecution, we'd encourage you to first discuss the activity with your organization's management and legal counsel and to notify any appropriate law enforcement agencies (in accordance with any policies or guidelines at your site).

         Keep in mind that unless one of the parties involved contacts law enforcement, any efforts to trap or trace the intruder may be to no avail. We suggest you contact law enforcement before attempting to set a trap or tracing an intruder.

         U.S. sites interested in an investigation can contact their local Federal Bureau of Investigation (FBI) field office. To find contact information for your local FBI field office, please consult your local telephone directory or see the FBI's field offices web page available at:

         www.fbi.gov/contact/fo/fo.htm

         U.S. sites and foreign locations involving U.S. assets, interested in an investigation can contact their local U.S. Secret Service (USSS) Field Office. To find contact information for your local USSS Field Office, please consult your local telephone directory or see the USSS web site available at:

         www.secretservice.gov/field_offices.shtml

         To contact the USSS Electronic Crimes Branch please call:

         Phone: +1(202)406-5850

         Fax: +1(202)406-9233

         If your site involves a Department of Defense Contractor, a Department of Defense Entity or any of the U.S. Military Services, and you are interested in an investigation, you may contact the United States Department of Defense Criminal Investigative Service (DCIS), Pittburgh, Pennsylvania at telephone number +1(412)395-6931. For information regarding DCIS please see:

         www.dodig.osd.mil/INV/DCIS/programs.htm

         Non-U.S. sites may want to discuss the activity with their local law enforcement agency to determine the appropriate steps that should be taken with regard to pursuing an investigation.

         To contact the Australian Federal Police:

         Canberra	+61 2 6256 7777	Ask for the Co-ordination centre
         Brisbane	+61 7 3222 1222	Ask for Operations
         Sydney	+61 2 9286 4000	Ask for the Co-ordination centre
         Melbourne	+61 3 9607 7777	Ask for the Co-ordination centre
         Adelaide	+61 8 8419 1811	Ask for the Co-ordination centre
         Perth	+61 8 9320 3444	Ask for the Co-ordination centre
         Darwin	+61 8 8981 1044	Ask for the Co-ordinator

      7. Notify others within your organization

      8. In addition to notifying management and legal counsel at your site, you may also need to notify others within your organization who may be directly affected by your recovery process (e.g., other administrators or users).

   4. Document all of the steps you take in recovering
   5. The importance of documenting every step you take in recovery can not be overstated. Recovering from a system compromise can be a hectic and time-consuming process and hasty decisions are often made. Documenting the steps you take in recovery will help prevent hasty decisions and give you a record of all the steps you took to recover, which you can reference in the future. Documenting the steps you take in recovery also may be useful if there is a legal investigation.
3. Regain control
4. 1. Disconnect compromised system(s) from the network
   2. To regain control, you will need to disconnect all compromised machines from your network including dial in connections. After that you may wish to operate in single user mode in UNIX or as the local administrator in NT to ensure that you have complete control of the machine; however, by rebooting or changing to single user/local administrator mode, you may lose some useful information because all processes executing at the time of discovery will be killed.

      Therefore, you may wish to work through steps in section C.5. Look for signs of a network sniffer to determine if the compromised system is currently running a network sniffer.

      Operating in single user mode on UNIX systems will prevent users, intruders, and intruder processes from accessing or changing state on the compromised machine while you are going through the recovery process.

      If you do not disconnect the compromised machine from the network, you run the risk that the intruder may be connected to your machine and may be undoing your steps as you try to recover the machine.

   3. Copy an image of the compromised system(s)
   4. Before analyzing the intrusion we encourage you to create a backup of your system. This will provide a "snapshot" of the file system at the time that the root compromise was first discovered. You may need to refer back to this backup in the future.

      If you have an available disk which is the same size and model as the disk in the compromised system, you can use the dd command in UNIX to make an exact copy of the compromised system.

      For example, on a Linux system with two SCSI disks, the following command would make an exact replica of the compromised system (/dev/sda) to the disk of the same size and model (/dev/sdb).

                 # dd if=/dev/sda of=/dev/sdb
      

      Please read the dd man page for more information.

      There are many other ways to create a backup of your system. On NT systems there is no built in command like dd, but there are a number of third party applications that will make an image copy of an entire hard drive.

      Creating a low level backup is important in case you ever need to restore the state of the compromised machine when it was first discovered. Also, files may be needed for a legal investigation. Label, sign, and date the backup and keep the backup in a secure location to maintain integrity of the data.

5. Analyze the intrusion
6. With your system disconnected from the network, you can now thoroughly review log files and configuration files for signs of intrusion, intruder modifications, and configuration weaknesses.
   1. Look for modifications made to system software and configuration files
   2. Verify all system binaries and configuration files.

      When looking for modifications of system software and configuration files, keep in mind that any tool you are using on the compromised system to verify the integrity of binaries and configuration files could itself be modified. Also keep in mind that the kernel (operating system) itself could be modified. Because of this, we encourage you to boot from a trusted kernel and obtain a known clean copy of any tool you intend to use in analyzing the intrusion. On UNIX systems you can create a boot disk and make it write protected to obtain a trustworth kernel.

      We urge you to check all of your system binaries thoroughly against distribution media. We have seen an extensive range of Trojan horse binaries that have been installed by intruders.

      Some of the binaries which are commonly replaced by Trojan horses on UNIX systems are: telnet, in.telnetd, login, su, ftp, ls, ps, netstat, ifconfig, find, du, df, libc, sync, inetd, and syslogd. Also check any binaries referenced in /etc/inetd.conf, critical network and system programs, and shared object libraries.

      On NT systems, Trojan horses commonly introduce computer viruses or "remote administration" programs such as Back Orifice and NetBus. There have been cases where the system file that handles internet connectivity was replaced with a Trojan horse.

      Because some Trojan horse programs could have the same timestamps as the original binaries and give the correct sum values, we recommend you use cmp on UNIX systems to make a direct comparison of the binaries and the original distribution media.

      Alternatively, you can check the MD5 results for either UNIX or NT on suspect binaries against a list of MD5 checksums from known good binaries. Ask your vendor if they make MD5 checksums available for their distribution binaries.

      Additionally, verify your configuration files against copies that you know to be unchanged.

      When inspecting your configuration files on UNIX systems, you may want to

      • check your /etc/passwd file for entries that do not belong.
      • check to see if /etc/inetd.conf has been modified.
      • if you allow the "r-commands" (rlogin, rsh, rexec), ensure that there is nothing that does not belong in /etc/hosts.equiv or in any .rhosts files.
      • check for new SUID and SGID files. The following command will print out all SUID and SGID files within your filesystem.

             # find / \( -perm -004000 -o -perm -002000 \) -type f -print
        

      When inspecting NT systems, you may want to

      • check for odd users or group memberships.
      • check for changes to registry entries that start programs at logon or services. (see LISTING 1)
      • check for unauthorized hidden shares with the 'net share' command or Server Manager tool.
      • check for processes that you do not identify using the pulist.exe tool from the NT resource kit or the NT Task Manager.
   3. Look for modifications to data
   4. Data on compromised systems is often modified by intruders. We encourage you to verify the integrity of web pages, ftp archives, files in users' home directories, and any other data files on your system.
   5. Look for tools and data left behind by the intruder
   6. Intruders will commonly install custom-made tools for continued monitoring or for access to a compromised system.

      The common classes of files left behind by intruders are

      • Network Sniffers
      • 
        A network sniffer is a utility which will monitor and log network activity to a file. Intruders commonly use network sniffers to capture username and password data that is passed in cleartext over the network. (see section C.5 below)

        Sniffers are more common on UNIX systems, but on NT systems check for key logging programs.

      • Trojan Horse Programs
      • 
        Trojan horse programs are programs that appear to perform one function while actually performing a different function. Intruders use Trojan horse programs to hide their activity, capture username and password data, and create backdoors for future access to a compromised system. (see section C.1 above)
      • Backdoors
      • 
        Backdoor programs are designed to hide itself inside a target host. The backdoor allows the user that installed it to access the system without using normal authorization or vulnerability exploitation.
      • Vulnerability Exploits
      • 
        A majority of compromises are a result of machines running vulnerable versions of software. Intruders often use tools to exploit known vulnerabilities and gain unauthorized access. These tools are often left behind on the system in "hidden" directories.
      • Other Intruder Tools
      • 
        The intruder tools listed above are not intended to be a conclusive or comprehensive list. There may be other tools left behind by an intruder. Some of the other types of tools you may find are tools to
        
        • probe systems for vulnerabilities
        • launch widespread probes of many other sites
        • launch denial of service attacks
        • use your computing and networking resources
      • Intruder Tool Output
      • 
        You may find log files from any number of intruder tools. These log files may contain information about other sites involved, vulnerabilities of your compromised machine(s), and vulnerabilities at other sites.

      We encourage you to search thoroughly for such tools and output files. Be sure to use a known clean copy of any tool that you use to search for intruder tools.

      When searching for intruder tools on a compromised system

      • Look for unexpected ASCII files in the /dev directory on UNIX systems. Some of the Trojan binaries rely on configuration files which are often found in /dev.
      • Look very carefully for hidden files or directories. If an intruder has created a new account and home directory then there may be hidden files or directories.
      • Look for files or directories with strange names such as "..." (three dots) or ".. " (two dots and some whitespace) [UNIX]. Intruders often try and hide files within such directories. On NT systems, look for files and directories that closely match what may appear as a system file (EXPLORE.EXE, UMGR32.EXE, etc).
   7. Review log files
   8. Reviewing your log files will help you get a better idea of how your machine was compromised, what happened during the compromise, and what remote hosts accessed your machine.

      Keep in mind when reviewing any log files from a compromised machine that any of the logs could have been modified by the intruder.

      On UNIX systems, you may need to look in your /etc/syslog.conf file to find where syslog is logging messages. NT systems generally log everything to one of three logs for NT events, all of which are viewed through the Event Viewer. Other NT applications such as IIS server may log to other locations. IIS by default writes logs to the c:\winnt\system32\logfiles directory.

      Below is a list of some of the more common UNIX log file names, their function, and what to look for in those files. Depending on how your system is configured, you may or may not have the following log files.

      • messages
      • 
        The messages log will contain a wide variety of information. Look for anomalies in this file. Anything out of the ordinary should be inspected. Also, look for events that occurred around the known time of the intrusion.
      • xferlog
      • 
        If the compromised system has a functioning ftp server, xferlog will contain log files for all of the ftp transfers. This may help you discover what intruder tools have been uploaded to your system, as well as what information has been downloaded from your system.
      • utmp
      • 
        This file contains binary information for every user currently logged in. This file is only useful to determine who is currently logged in. One way to access this data is the who command.
      • wtmp
      • 
        Every time a user successfully logs in, logs out, or your machine reboots, the wtmp file is modified. This is a binary file; thus, you need to use a tool to obtain useful information from this file. One such tool is last<">