How To Investigate A Linux Server Compromise

Usually the first signs of a server compromise are high processor load. If you have a good monitoring solution in place this should have alerted you.

Some common tools that can show the compromised processes running are “top” and “ps”.
top can show you which processes are using the most CPU time and memory.
ps will show the names of processes currently running. If you look carefully through the output from “ps fax” you can see parent and child processes. Often the hackers will start a service and name it with a similar name to an existing service – for example Apache or Apache2.

You have two options now – kill the processes or do a quick investigation to capture information about the process before killing it. Here are some steps to capture the compromised commands before we kill them – that way we can then perform some analysis on them.

Create a directory called ”’/root/$DATE1”’ and save all output into there
 $ export DATE1=`date +%d-%b-%Y`
 $ mkdir /root/$DATE1/

Try capture all the info below ”’BEFORE”’ killing the process – once its dead all the vital information will be lost.

Run the commands below and save their output into a file in the /root/$DATE1 directory:
e.g. $ ps fax > /root/$DATE1/psfax

Commands to Run
Show processes, PID, usernames:
$ ps faxuww
  $ ps faxuww > /root/$DATE1/psfaxuww

root      3062  0.0  0.3  20960  6552 ?        Ss   08:28   0:00 /usr/sbin/httpd
apache    3150  0.0  0.1  21092  3456 ?        S    08:28   0:00  \_ /usr/sbin/httpd

Show current time, Process, PID and how long its been running for:
$ date ;  ps -eo pid,cmd,etime
  $ date > /root/$DATE1/ps-eo ;  ps -eo pid,cmd,etime >> /root/$DATE1/ps-eo

Show whats currently listening for traffic:
$ netstat -npl
  $ netstat -npl >  /root/$DATE1/netstat-npl

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
tcp        0      0 :::80                       :::*                        LISTEN      3062/httpd

Show who is currently connected to the server and also outgoing connections:
$ netstat -np
  $ netstat -np >  /root/$DATE1/netstat-np

Now use the above to find the compromised process. Get the PID from that process. PID is the PROCESS ID – it can be found in the first column of the “ps fax”
$ export PID=”insert process PID here”

View all files opened by a process, this will help find the directory the process is running from:
$ lsof -p $PID
 $ lsof -p $PID > /root/$DATE1/lsof-$PID

Show all information related to a process:
$ ls -la /proc/$PID/
  $ ls -la /proc/$PID/ >  /root/$DATE1/proc-$PID

Show the time the process was started. This is a good indication of when the compromise happened:
$ ls -lad /proc/$PID/
  $ ls -lad /proc/$PID/ >  /root/$DATE1/proc-$PID

Using the above steps- now try find the location of the compromised files.
if you can find the files use ”’ls”’ and ”’file”’ on them
-> so we can see who file owner is and what type of file it is.

Kill The Processes
Make sure you ”’kill -9”’ all the compromised processes.

Log Files
Use the dates and times found above to see if the Apache logs etc show any activity, usually they will be POST requests.

This entry was posted in Managed Hosting. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>