[NTLUG:Discuss] Challenge....

[Will Senn]

Douglas King wrote:

> OK, I've got a problem.  This past week.....I've had a RedHat 7.3 
> webserver shut down every night somewhere between the hours of 1:30 AM 
> and 3:00 AM.  We have checked ALL the scheduled crons, etc. and find 
> nothing that would be shutting it down "naturally".  The power light 
> on the case remains on, but you cannot SSH into it, nor is the machine 
> functional.  Log files don't indicate a lot to me...although, I did 
> catch a potential hacker 2 nights ago...but he's since been dealt with.
>
> Where do I look now? 
>
> _______________________________________________
> https://ntlug.org/mailman/listinfo/discuss
>
Douglas,

Could be nearly anything, tough to tell from the outside looking in. 
Here are a couple suggestions, for what they're worth, followed by some 
questions to ask (ranging from obvious to simply logical, may be of no 
use to you at all or they might trigger an epiphany, hope for the later).

1. check the logs to see who's logged in when the machine shuts down 
every night and look for a pattern.
2. check the logs to see what processes are logging just prior to the 
shut down every night and look for a pattern.

Here are some things to consider:
If you have a monitor attached to the machine - is the led green, yellow 
or what (ie is the monitor on with signal, on standby, or off)?
Is the screen blank or does the machine only appear to be 'hung' not off?
Where is the machine located? Is it at home or in a lab? Is it 
physically secure?
Is it possible that their was an external power event that caused the 
machine to shutdown?
Is the machine equipped with any external controller such as wake on 
lan, remote shutdown, etc?
Is it possible that their was an internal hardware event that caused the 
machine to shutdown (power supply issue, CPU overheat, etc)?
What kind of machine is it and does it have hardware monitoring 
capabilities - cpu sensors, power sensors, etc?
Is the machine overclocked or insufficiently ventilated (could cause any 
number of chaotic problems)?
Is APM turned off (no standbye or hibernate features enabled for the 
system or monitor)?
Is this a new box or has this box had recent software/hardware/use 
pattern changes?
Is tripwire installed and have you looked into it?

Some possible causes:
Security breach.
Someone shutting the machine down through software (something along the 
lines of 'shutdown -h now' or 'halt').
External power event (brownout - power spike).
Internal power event (powersupply, motherboard, harddrive).
Internal motherboard failure.
Hard disk failure.
Power management failure

Later, hope this helps - surely some guru'll tell you how to turn on 
'uberlog' and it'll just spit out the answer. I eagerly await the 
findings...

Will