crond dead but subsys locked

You sometimes get the following error if you try and check the status of a service i.e

service crond status
crond dead but subsys locked

Quite an harmless sounding message but if dealt wrongly it can cause some strange and at times dangerous issues.

Here is an example:

Background
Before
Let cause some chaos
dangerous next steps
Better fix
Notes on services

background

why do init scripts require lock files
what are pid and lock files for

before

so first check cron is working ok

# cat /var/run/crond.pid
1116

And make sure only 1 cron is running

# ps -ef | grep cron
root 1116 1 0 Sep18 ? 00:00:37 crond
root 27198 26991 0 10:51 pts/0 00:00:00 grep cron

And again check what the system thinks is running
#service crond status
crond (pid 1116) is running...

Let cause some chaos

remove the pid file
# rm /var/run/crond.pid

check the status
# service crond status
crond dead but subsys locked

dangerous next steps

don't just restart the service here is an example of the damage that can be caused

# service crond restart
Starting crond: [ OK ]
# service crond status
crond (pid 27250) is running...

now you have 2 cron running
# ps -ef | grep cron
root 1116 1 0 Sep18 ? 00:00:37 crond
root 27303 1 0 10:53 ? 00:00:00 crond

# crontab -l
* * * * * date >> /root/test.log
# tail -f /root/test.log
Thu Oct 22 11:00:01 EDT 2015
Thu Oct 22 11:00:01 EDT 2015
Thu Oct 22 11:01:01 EDT 2015
Thu Oct 22 11:01:01 EDT 2015

Better fix

1) vi /var/run/crond.pid and copy in the pid
2) chmod 644 /var/run/crond.pid
3) service crond status # and check it looks better

Troubleshooting the issue

Centos 5.11 to have more issues than centos 6
/var/log/cron will have details of when cron is stoped/started/restarted
Users bash history should have details of when a change was made
if you have auditd try: (taken from http://www.cyberciti.biz/tips/linux-audit-files-to-see-who-made-changes-to-a-file.html has more examples)

setup auditctl
auditctl -w /var/run/crond.pid -p war -k password-file

search auditctl
ausearch -i -f /var/run/crond.pid

Notes on services

note not all services will log what and when they change status here is an example of when cron logs

condrestart - cron log will tell you if it is has restarted
force-reload - cron log will tell you if it is has restarted
reload - not in log
restart - cron log will tell you if it is has restarted
start - cron log will tell you if it is has restarted
status - not in log
stop - cron log will tell you if it is has restarted
try-restart - cron log will tell you if it is has restarted

the following gives you more details on services
http://refspecs.linuxbase.org/LSB_3.1.0/LSB-Core-generic/LSB-Core-generic/iniscrptact.html

service --status-all shows the status off all services