Usually the first signs of a server compromise are high processor load. If you have a good monitoring solution in place this should have alerted you.
Some common tools that can show the compromised processes running are “top” and “ps”.
top can show you which processes are using the most CPU time and memory.
ps will show the names of processes currently running. If you look carefully through the output from “ps fax” you can see parent and child processes. Often the hackers will start a service and name it with a similar name to an existing service – for example Apache or Apache2.
You have two options now – kill the processes or do a quick investigation to capture information about the process before killing it. Here are some steps to capture the compromised commands before we kill them – that way we can then perform some analysis on them.
Create a directory called ”’/root/$DATE1”’ and save all output into there
$ export DATE1=`date +%d-%b-%Y`
$ mkdir /root/$DATE1/
Try capture all the info below ”’BEFORE”’ killing the process – once its dead all the vital information will be lost.
Run the commands below and save their output into a file in the /root/$DATE1 directory:
e.g. $ ps fax > /root/$DATE1/psfax
Commands to Run
Show processes, PID, usernames:
$ ps faxuww
$ ps faxuww > /root/$DATE1/psfaxuww
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 3062 0.0 0.3 20960 6552 ? Ss 08:28 0:00 /usr/sbin/httpd
apache 3150 0.0 0.1 21092 3456 ? S 08:28 0:00 \_ /usr/sbin/httpd
Show current time, Process, PID and how long its been running for:
$ date ; ps -eo pid,cmd,etime
$ date > /root/$DATE1/ps-eo ; ps -eo pid,cmd,etime >> /root/$DATE1/ps-eo
Show whats currently listening for traffic:
$ netstat -npl
$ netstat -npl > /root/$DATE1/netstat-npl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 :::80 :::* LISTEN 3062/httpd
Show who is currently connected to the server and also outgoing connections:
$ netstat -np
$ netstat -np > /root/$DATE1/netstat-np
Now use the above to find the compromised process. Get the PID from that process. PID is the PROCESS ID – it can be found in the first column of the “ps fax”
$ export PID=”insert process PID here”
View all files opened by a process, this will help find the directory the process is running from:
$ lsof -p $PID
$ lsof -p $PID > /root/$DATE1/lsof-$PID
Show all information related to a process:
$ ls -la /proc/$PID/
$ ls -la /proc/$PID/ > /root/$DATE1/proc-$PID
Show the time the process was started. This is a good indication of when the compromise happened:
$ ls -lad /proc/$PID/
$ ls -lad /proc/$PID/ > /root/$DATE1/proc-$PID
Using the above steps- now try find the location of the compromised files.
if you can find the files use ”’ls”’ and ”’file”’ on them
-> so we can see who file owner is and what type of file it is.
Kill The Processes
Make sure you ”’kill -9”’ all the compromised processes.
Log Files
Use the dates and times found above to see if the Apache logs etc show any activity, usually they will be POST requests.
