Linux server losing local time on reboot

Hi,

You might try the following (with root privelages):

1) ntpdate pool.ntp.org

Just to make sure the clock is currently updated to the right time

2) hwclock –systohc –utc

This will set the system clock to use UTC.  If your setup is then set to whatever timezone you are in, it should display correctly for local time.

I originally found this here (Slackware.com) as I had a similar issue.

HTH,

Alan.

Hmmm, seems counter intuitive to set the hardware clock to UTC, but I'll give it a try. It does seem distro dependent. I never had this issue on Debian/Ubuntu, but only have noticed it on Slackware. It also only happens on an unexpected shutdown (e.g. power failure). That kind of makes sense too because a normal shutdown executes a shutdown script which runs hwclock.

Normal shutdown script /etc/rc.d/rc.0:

# Save the system time to the hardware clock using hwclock --systohc.
if [ -x /sbin/hwclock ]; then
  # Check for a broken motherboard RTC clock (where ioports for rtc are
  # unknown) to prevent hwclock causing a hang:
  if ! grep -q -w rtc /proc/ioports ; then
    CLOCK_OPT="--directisa"
  fi
  if grep -q "^UTC" /etc/hardwareclock 2> /dev/null ; then
    echo "Saving system time to the hardware clock (UTC)."
    /sbin/hwclock $CLOCK_OPT --utc --systohc
  else
    echo "Saving system time to the hardware clock (localtime)."
    /sbin/hwclock  $CLOCK_OPT --localtime --systohc
  fi
fi

Select allOpen in new window

Startup/boot script: /etc/rc.d/rc.S:

# Set the system time from the hardware clock using hwclock --hctosys.
if [ -x /sbin/hwclock ]; then
  # Check for a broken motherboard RTC clock (where ioports for rtc are
  # unknown) to prevent hwclock causing a hang:
  if ! grep -q -w rtc /proc/ioports ; then
    CLOCK_OPT="--directisa"
  fi
  if grep -wq "^UTC" /etc/hardwareclock ; then
    echo -n "Setting system time from the hardware clock (UTC): "
    /sbin/hwclock $CLOCK_OPT --utc --hctosys
  else
    echo -n "Setting system time from the hardware clock (localtime): "
    /sbin/hwclock $CLOCK_OPT --localtime --hctosys
  fi
  date
fi

Select allOpen in new window

ntpd startup script /etc/rc.d/rc.ntpd:

  # The kernel is now mocking around with the the hardware clock if
  # ntpd is running, so if the hardware clock (wall clock) is set to
  # 'localtime' execute hwclock --localtime --systohc to disable the
  # 11 minute mode kernel function:
  if [ -x /sbin/hwclock ]; then
    # Check for a broken motherboard RTC clock (where ioports for rtc are
    # unknown) to prevent hwclock causing a hang:
    if ! grep -q -w rtc /proc/ioports ; then
      CLOCK_OPT="--directisa"
    fi
    if ! grep -q "^UTC" /etc/hardwareclock 2> /dev/null ; then
      echo "Saving system time to the hardware clock (localtime)."
      /sbin/hwclock  $CLOCK_OPT --localtime --systohc
    fi
  fi

Select allOpen in new window

My current /etc/hardwareclock is set to:

# /etc/hardwareclock
#
# Tells how the hardware clock time is stored.
# You should run timeconfig to edit this file.

localtime

Select allOpen in new window

This is trickier than I thought. I ran your suggested `hwclock –systohc –utc` even though these various scripts would detect /etc/hardwareclock set to "localtime" and run `/sbin/hwclock  --localtime --systohc`

I guess I'll have to try various things and see. I'll post back results.

Hi,

I agree with your reasoning - your situation is not the same as the one I had in that you are having the 'unexpected shutdowns', and the time issue does appear to be a symptom of that.

I realise it isn't answering the question you have asked, but if it were me, I would probably be addressing that issue first, and hoping that the time issue is just a symptom of that (as it appears to be).

Do you have, or can you attach, a UPS to your system?  If so, can you then get them to 'talk' and initiate a controlled shutdown after the power fails, but before the batteries run out?

Alan.

Alan3285:

Do you have, or can you attach, a UPS to your system?  If so, can you then get them to 'talk' and initiate a controlled shutdown after the power fails, but before the batteries run out?

Yes, there is a UPS attached, but I suppose the outage was longer than the battery could keep things up.

It is on my "todo" list to get the UPS and server talking!

Meanwhile, I have moved the rc.ntpd startup script to after the DNS startup has been run. Since the ntp.conf specifies pool.ntp.org it stands to reason that DNS is needed to resolve that. I've put that diagnostic script before the netdate command in rc.local and before any DNS initialization to see what' in the hwclock and what another host on the LAN has its time set to (for comparison). Hopefully, these steps will either fix or at least help diagnose the problem.

Hi,

jmarkfoley:

Meanwhile, I have moved the rc.ntpd startup script to after the DNS startup has been run. Since the ntp.conf specifies pool.ntp.org it stands to reason that DNS is needed to resolve that. I've put that diagnostic script before the netdate command in rc.local and before any DNS initialization to see what' in the hwclock and what another host on the LAN has its time set to (for comparison). Hopefully, these steps will either fix or at least help diagnose the problem.

Sounds like a good idea.

Your are certainly correct that you'll need DNS up and running in order to resolve any hostname (e.g. pool.ntp.org).

I actually use geographically local timeservers myself (e.g. nz.pool.ntp.org), but pool.ntp.org should always be good.

Good luck!

Alan.

I'm going to be taking this server down shortly (controlled, not plug-yank) to check out the UPS, so I'll post back whatever I see. Otherwise, this is likely to be a long term experiment and I won't leave the question open that long.

I am going to modify my diagnostic script to email me if the local host and other LAN host's time differ. At least that way I'll be able to catch the problem, if any, before users get hours of mis-timestamped email.

well, we had another power outage -- and I noticed that the server is plugged into 'Surge Protection', not 'Battery Backup', so that's the reason it doesn't stay up!

Anyway, my diagnostic log worked (except it didn't email the message to me). Here's what was logged:

my date Thu Mar 5 07:30:20 EST 2015
webservers date Thu Mar 5 07:30:21 EST 2015
Starting from unexpected shutdown

You will notice that the hwclock setting (my date) is the same as the system time on another computer on the LAN. So, the hwclock was not the culprit. In this boot event, the date and time stayed correct and ntpd was running.

Conclusion: I think moving the ntpd startup script to after the DNS start up script did the trick. Not sure I understand ntpd completely. the ntp.conf has

server pool.ntp.org iburst
server  127.127.1.0     # local clock
fudge   127.127.1.0 stratum 10

which seems to me to indicate that it should check 127.127.1.0 if pool.ntp.org is not available. And what is 127.127.1.0 anyway? I thought my localhost IP was 127.0.0.1.

Last comments?

Thank's for that info on localhost.

Another wrinkle. I did a controlled shutdown in order to plug the computer into the battery-backed side of the UPS. Interesting results in my diagnosic log on reboot:

my date Thu Mar 5 07:30:20 EST 2015
webservers date Thu Mar 5 07:30:21 EST 2015
Starting from unexpected shutdown

my date Sat Mar 7 17:47:06 EST 2015
webservers date Sat Mar 7 12:47:09 EST 2015
my hwclock Sat Mar 7 17:47:07 2015 -0.354118 seconds
Starting from normal shutdown

Select allOpen in new window

The first group is from the power-outage shutdown from March 5, discussed above. The 2nd group is from my reboot which I did at 12:47 on Saturday. Notice that upon startup the hardware clock is set to 17:47!!!! This is with a controlled reboot!

Fortuntately, I have the line `/usr/sbin/netdate tcp 128.138.141.172` (time.nist.gov) later in the startup script, and it logged:

128.138.141.172 -18000.227 Sat Mar  7 12:47:07.000

setting the date back to the correct time. ntpd then started normally. When the time is way off, ntpd won't run.

This leads me to conclude that the hwclock command in the normal shutdown script is doing something bad. Here is my rc.0 shutdown script fragment:

# Save the system time to the hardware clock using hwclock --systohc.
if [ -x /sbin/hwclock ]; then
  # Check for a broken motherboard RTC clock (where ioports for rtc are
  # unknown) to prevent hwclock causing a hang:
  if ! grep -q -w rtc /proc/ioports ; then
    CLOCK_OPT="--directisa"
  fi
  if grep -q "^UTC" /etc/hardwareclock 2> /dev/null ; then
    echo "Saving system time to the hardware clock (UTC)."
    /sbin/hwclock $CLOCK_OPT --utc --systohc
  else
    echo "Saving system time to the hardware clock (localtime)."
    /sbin/hwclock  $CLOCK_OPT --localtime --systohc
  fi
fi

Select allOpen in new window

The `grep -w rtc /proc/ioports` statement does not find rtc and thus CLOCK_OPT does get set to "--directisa". The comment says it's to prevent hwclock causing a hang, but I've never experienced a hang without this.

the hwclock man page says:

      --directisa
              This option is meaningful only on an ISA machine or an Alpha (which implements enough
              of ISA to be, roughly speaking, an ISA machine for hwclock's  purposes).   For  other
              machines,  it  has no effect.  This option tells hwclock to use explicit I/O instruc-
              tions to access the Hardware Clock.  Without this option, hwclock will try to use the
              /dev/rtc  device  (which it assumes to be driven by the RTC device driver).  If it is
              unable to open the device (for reading), it will use the  explicit  I/O  instructions
              anyway.

I check this rtc in /proc/ioports in 4 other computer and none of them have it. Some of the motherboards I checked are quite old, but all are Slackware. I'd be interest to see if you Ubuntu box has it.

According to the man page, it should have no bad effect, but I'm wonder about that. I'm going to comment out this --directisa, try a reboot and see what happens. I definitely didn't see this behavior when I had Ubuntu running on 2 of these same machines last Fall, and then DNS had nothing to do with it.

Hi,

I have to note that you are definitely moving beyond my level of comfort - I have never adjusted those settings myself.  I can;t see that commenting out the --directisa would stop the machine booting up, but beyond that, I wouldn't like to guess.  As long as you can get back in and reinstate it, you should be okay.

For what its worth, my Ubuntu 12.04.5 LTS has the following line in /proc/ioports:

0070-007f : rtc0

There is no other occurrence of the string 'rtc' in the ioports file.

HTH,

Alan.

That's interesting, you do have "rtc" in /proc/ioports. I have nothing among 4 machines.

Anyway, I think I have a work-around including using `netdate  ime.nist.gov` in the rc.local start-up script. It's a bit of a kludge, but at least I get the correct time and ntpd runs. It's hard to imagine this is a Slackware bug. Someone would have caught it by now I would think.

I'll keep my eye on this --directisa option and see if that tells me anything.

I developed a work-around for the problem. Thanks to Alan3285 for working with me.