Hacked system -steps to look at

Common causes
Symptoms of a hack
Quick investigation?
Have i been hacked?
Strange processes
restarting a process
Strange/attempted logins
Recently edited files
other (use where relevant)
Next steps
Rootkit checkers
Thinking like an attacker
Checking someone's bash history
Links for hacking

Common causes

  • Poor security i.e bad password
  • Key logger
  • Vulnerability targeted i.e applications not updated

Symptoms of a hack

  • high cpu from a shell script (normally perl)
  • poor system performance
  • cannot access websites pages as per normal
  • reports of spamming
  • cannot start a process as something else is running on the port
  • generally odd behavior that you cannot attribute to anything else

Quick investigation?

1) netstat -an - check for open unexpected ports
2) tcpdump src port x -check what that port is doing
3) use lsof i.e lsof -i:22 to see what files are open
4) run and install rkhunter
5) check logs /i.e /var/log/messages apache logs....

Have i been hacked?

The checks to investigate if you have been hacked, and other useful bits:

-strange processes + restarting a process

-strange/attempted logins

-hidden/odd files

-recently edited files


-root kit checker


-thinking like an attacker

Strange processes

1) run top and look for odd processes i.e a perl script taking up a lot of cpu or another process that is using up a lot of memory… If you find one - use c flag in top to see the full command , run cd /proc/pid ; ls -l and look for files with odd symbolic links I.e files pointing to hidden files. Run lsof -p PID to see other files related to the process. Where are the files located? what user owns them? You might find the source of the attack from this information.

2) Run netstat –tplan | grep LISTEN and look for services you are not expecting, if you find a port in use run lsof –i :port to show all files opened by process any output from the following commands could indicate a comprised system:

run netstat -an | grep 6667 # irc server used by hackers

run netstat -an | grep 1337 # used by leet

lsof -i | grep ircd

restarting a process

for i in `lsof -i tcp:80 | awk '{print $2}' | sed '1 d'`; do kill -9 $i; done && /etc/init.d/httpd start # get pid of anything running on port 80 and kill them & then restart httpd

Strange/attempted logins

1) run last to see who has recently logged in, and narrow down if you have an idea when a system was hacked.

2) run last -i | awk '{ print $3 }'| sort| uniq -c | sort -n# orders logins via ip used to login, follow any ips found (check IP is not from unexpected countries I.e Russia/China/…)

3) run sudo cat /var/log/secure* | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | uniq -c | sort -k2 # find number of failed logins, good to see how well protected a system is, if numbers are big (i.e 50+) even if the system hasn’t been hacked look at fail2ban/iptables/firewall setup.

4) run cat /var/log/secure* | grep 'Failed password' | grep sshd | awk '{print $11,$13}' | sort | uniq -c| sort -n # find usernames of failed logins check against valid users in /etc/passwd

Hidden/odd files

1) if you have a suspected comprised user run ls -la on users home dir and search for hidden files

2) root kit checkers can also check for hidden files

Recently edited files

as a slight side not be aware of

1) find ./ -name "*.*" -type f -mtime -4 -exec ls -al {} \; | less # find files modified up to 4 days ago with . in them

2) stat * | egrep “File|Modify” | grep –B1 "2012" ** # see what has recently been modified, add more flags where needed

3) if you have a suspected comprised user check users .bash_history files has it been deleted/amended or does it contain any odd looking commands.

4) check /var/spool/cron/.. to check for any unsual or strange editions to the cron file

5) stat /bin/ls /bin/top …and check recently modified data (good to check against a file such as stat /lost+found)

6) also check other cron locations (/etc/crontab , /var/spool/cron & /etc/cron.d)

other (use where relevant)

  1. find / -type f -perm +u=s # symbolic notation gives more flexibility , search for suid files (used for escalation of privileges)
  2. check .htaccess if website issues and check which files have recently been edited
  3. check /etc/passwd for dodgy users, would indicate root comprised
  4. use tcpdump/wireshark for advanced investigation
  5. ls -asl /tmp /var/tmp - check whats in the tmp directories (tmp directories are sometimes vunerable
  6. ps -Udaemon -u daemon - check which processes are related to a daemon
  7. crontab -l -u daemon - check what crontab are related to a daemon
  8. atq - check if any at jobs are set
  9. check .ssh authorized key files for unexpected keys
  10. find / -perm -a+w ! -type l >> world_writable.txt : Look at world_writable.txt to see all world writable files and directories. Note l will only show symbolic links
  11. find / -nouser -o -nogroup >> no_owner.txt : Look at no_owner for all files that do not have a user or group associated with them. All files should be owned by a specific user or group to restrict access to them
  12. in mysql run mysql -s <<< 'SELECT CONCAT("SHOW GRANTS FOR ",user,"@",host,";") FROM mysql.user;' | sed -e "s/FOR /&'/" -e "s/@/'&'"/ -e "s/;/'&/" | mysql -s and check which users have the file and ALL permission
  13. check php files for badly configured php upload permissions more info at http://www.acunetix.com/websitesecurity/upload-forms-threat/

Next steps

Remove and clear up then Patch + change passwords or reinstall Os

Rootkit checkers

Rkhunter and chrootkit are good tools to help spot odd/bad things going on, the following pages explain install and how to set them up:





Monitor what you put on the internet

Make sure firewall is configured correctly (both inbound and outbound ports

Patch your systems

Install and use av

Make use of an intrusion detection system i.e snort

Make sure admin/root access is restricted as much as possible

Monitor traffic in and out of network

Run port scans on your network/machines (nmap)

Keep an eye on the logs: logwatch by default sends last days info to an email address (very verbose) .log check is an update verson of logsentry similar to logwatch (tuning needed) .modern linux kernels (2.6.x) come with auditd installed it can record data from the kernel including system calls,user ID &process Id

Thinking like an attacker

Kali (previously backtrack) is a good hacking OS with lots of built in tools

steps in an attack:

1) reconnasissance –google is your friend, check forums for server admin talking about “how do I configure a cisco firewall” means you now the company where the sys admin works uses cisco –the aim research your target, some great google terms (all goolge searches are case insensitive)


intitle:"VNC viewer for Java"

intitle:"speedstream router management interface"

inurl:email.xls you get the idea...

filetype:sql + "identified by"

filetype:sql + "identified by" -cvs

filetype:sql + "identified by" ("Grant * on *" | "create user"


intitle:"Network Print Server" filetype:shtm

intitle:phpmyadmin "Welcome to phpmyadmin" "running on * as root@"

site:mysite.competer white (and yes there is no space between the enter of domain and the search term)

site:mysite.com allintitle:index of to see all open indexes run

inurl:admin - to search for specfic sites that contain words in the url

cache:mydomain.com - search in the google cache use

filetype: - to search for particualr file types run

other tools/ideas:

harvestor (get email address and subdomains)


send an email with a .bat to mail server see if you get av type and email vender

2)scanning (port and vunerability portscanning can flag ids monitors)

a) port scan

b) find targets and vunerabilities

3)exploitation (start the hack after gathering above info) –


Jack the ripper brute force password
nikto is a web server vunerablity scanner, (make sure epel is installed, yum install
nikto, nikto -h localhost or nikto -h localhost -evasion 1 to try and bypass an ids)
more on nikto @ http://www.slashroot.in/website-vulnerability-scanner
metaexploit great allround tool
Netcat – the swiss army knife –can be used to transfer files,conduct port scans,as a instant messanger and even a simple web server.

4) maintaining access (backdoor to system I.e you need to do more than crash a program)

Checking someone's bash history

Bash history is a great place to look at what commands have been run,
bash_history is typically only written to after the user has logged out to check the bash_history or a currently logged in user checkout https://www.wirehive.net/blog/2015/01/08/permanent-data-loss#all

Links for hacking