Linux server losing local time on reboot

I have a new Slackware64 14.1 kernel 3.10.17 host. This host is running Samba4 and is active as our domain's Active Directory/Domain Controller, DNS server, DHCP server, etc. I have a big problem in that when the system reboots unexpectedly (e.g. because of power failure) it comes back up with GMT time, not local time and ntpd does not run.

I originally staged this system on a much older computer and had the same issue, but I thought that was because of the age of the computer and perhaps the CMOS battery was dying or some other such problem. I thought the problem would go away with brand new hardware, but no.

Any idea what the problem could be? This is serious. Our office ran all morning 5 hours ahead until I was able to fix the time.
LVL 1
MarkAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

AlanConsultantCommented:
Hi,

You might try the following (with root privelages):

1) ntpdate pool.ntp.org

Just to make sure the clock is currently updated to the right time

2) hwclock –systohc –utc

This will set the system clock to use UTC.  If your setup is then set to whatever timezone you are in, it should display correctly for local time.

I originally found this here (Slackware.com) as I had a similar issue.

HTH,

Alan.
0
MarkAuthor Commented:
Hmmm, seems counter intuitive to set the hardware clock to UTC, but I'll give it a try. It does seem distro dependent. I never had this issue on Debian/Ubuntu, but only have noticed it on Slackware. It also only happens on an unexpected shutdown (e.g. power failure). That kind of makes sense too because a normal shutdown executes a shutdown script which runs hwclock.

Normal shutdown script /etc/rc.d/rc.0:
# Save the system time to the hardware clock using hwclock --systohc.
if [ -x /sbin/hwclock ]; then
  # Check for a broken motherboard RTC clock (where ioports for rtc are
  # unknown) to prevent hwclock causing a hang:
  if ! grep -q -w rtc /proc/ioports ; then
    CLOCK_OPT="--directisa"
  fi
  if grep -q "^UTC" /etc/hardwareclock 2> /dev/null ; then
    echo "Saving system time to the hardware clock (UTC)."
    /sbin/hwclock $CLOCK_OPT --utc --systohc
  else
    echo "Saving system time to the hardware clock (localtime)."
    /sbin/hwclock  $CLOCK_OPT --localtime --systohc
  fi
fi

Open in new window

Startup/boot script: /etc/rc.d/rc.S:
# Set the system time from the hardware clock using hwclock --hctosys.
if [ -x /sbin/hwclock ]; then
  # Check for a broken motherboard RTC clock (where ioports for rtc are
  # unknown) to prevent hwclock causing a hang:
  if ! grep -q -w rtc /proc/ioports ; then
    CLOCK_OPT="--directisa"
  fi
  if grep -wq "^UTC" /etc/hardwareclock ; then
    echo -n "Setting system time from the hardware clock (UTC): "
    /sbin/hwclock $CLOCK_OPT --utc --hctosys
  else
    echo -n "Setting system time from the hardware clock (localtime): "
    /sbin/hwclock $CLOCK_OPT --localtime --hctosys
  fi
  date
fi

Open in new window

ntpd startup script /etc/rc.d/rc.ntpd:
  # The kernel is now mocking around with the the hardware clock if
  # ntpd is running, so if the hardware clock (wall clock) is set to
  # 'localtime' execute hwclock --localtime --systohc to disable the
  # 11 minute mode kernel function:
  if [ -x /sbin/hwclock ]; then
    # Check for a broken motherboard RTC clock (where ioports for rtc are
    # unknown) to prevent hwclock causing a hang:
    if ! grep -q -w rtc /proc/ioports ; then
      CLOCK_OPT="--directisa"
    fi
    if ! grep -q "^UTC" /etc/hardwareclock 2> /dev/null ; then
      echo "Saving system time to the hardware clock (localtime)."
      /sbin/hwclock  $CLOCK_OPT --localtime --systohc
    fi
  fi

Open in new window

My current /etc/hardwareclock is set to:
# /etc/hardwareclock
#
# Tells how the hardware clock time is stored.
# You should run timeconfig to edit this file.

localtime

Open in new window

This is trickier than I thought. I ran your suggested `hwclock –systohc –utc` even though these various scripts would detect /etc/hardwareclock set to "localtime" and run `/sbin/hwclock  --localtime --systohc`

I guess I'll have to try various things and see. I'll post back results.
0
MarkAuthor Commented:
The problem occurs when there is an unexpected shutdown, which means that the rc.0 shutdown script is doing the right thing, which means that "localtime" in my /etc/hardwareclock must be correct. But what's going on when there is not a normal shutdown? Not seeing the logic here. I'm hesitant to simply change /etc/hardwareclock to UTC.

And why would `hwclock –systohc -–utc` be any different that `hwclock –systohc –-localtime`? The hardware clock doesn't store timezone.

I'm putting in a couple of diagnostic scripts to see what's what. This first one runs in the 'start' function of rc.ntpd, before it tries to run ntpd. It snags what the current time is set to, the current time on another host in the LAN, and the current hwclock. It also checks a flag file to see if the computer rebooted normally. The shutdown script rc.local_shutdown will create the file /root/shutdown if it is run.

JMFLOG=/var/log/ntpdJMF.log

echo my date `/usr/bin/date` >> $JMFLOG
echo webservers date `/usr/local/bin/rdate -p 192.168.0.3` >>$JMFLOG
echo my hwclock `/sbin/hwclock --show` >> $JMFLOG

if [ -e /root/shutdown ]
then
    echo Starting from normal shutdown >>$JMFLOG
    rm /root/shutdown
else
    echo Starting from unexpected shutdown >>$JMFLOG
fi

Open in new window

The following line is in rc.local and execute after all the other init scripts. It will set the host time to time.nist.gov using the old netdate command -- this is in case the time gets set badly by rc.ntpd. I'm using the IP because DNS is not running yet.

/usr/sbin/netdate tcp 128.138.141.172 2>&1 | /usr/bin/tee -a /var/log/netdate.log

Not sure when this will get tested. I hesitate to yank the plug on a production server. I'll give it a couple of days. Maybe there'll be another power failure (that UPS must be shot).

Maybe I should run ntpd after the DNS is up?
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

AlanConsultantCommented:
Hi,

I agree with your reasoning - your situation is not the same as the one I had in that you are having the 'unexpected shutdowns', and the time issue does appear to be a symptom of that.

I realise it isn't answering the question you have asked, but if it were me, I would probably be addressing that issue first, and hoping that the time issue is just a symptom of that (as it appears to be).

Do you have, or can you attach, a UPS to your system?  If so, can you then get them to 'talk' and initiate a controlled shutdown after the power fails, but before the batteries run out?

Alan.
0
MarkAuthor Commented:
Alan3285:
Do you have, or can you attach, a UPS to your system?  If so, can you then get them to 'talk' and initiate a controlled shutdown after the power fails, but before the batteries run out?
Yes, there is a UPS attached, but I suppose the outage was longer than the battery could keep things up.

It is on my "todo" list to get the UPS and server talking!

Meanwhile, I have moved the rc.ntpd startup script to after the DNS startup has been run. Since the ntp.conf specifies pool.ntp.org it stands to reason that DNS is needed to resolve that. I've put that diagnostic script before the netdate command in rc.local and before any DNS initialization to see what' in the hwclock and what another host on the LAN has its time set to (for comparison). Hopefully, these steps will either fix or at least help diagnose the problem.
0
AlanConsultantCommented:
Hi,

jmarkfoley:

Meanwhile, I have moved the rc.ntpd startup script to after the DNS startup has been run. Since the ntp.conf specifies pool.ntp.org it stands to reason that DNS is needed to resolve that. I've put that diagnostic script before the netdate command in rc.local and before any DNS initialization to see what' in the hwclock and what another host on the LAN has its time set to (for comparison). Hopefully, these steps will either fix or at least help diagnose the problem.

Sounds like a good idea.

Your are certainly correct that you'll need DNS up and running in order to resolve any hostname (e.g. pool.ntp.org).

I actually use geographically local timeservers myself (e.g. nz.pool.ntp.org), but pool.ntp.org should always be good.

Good luck!

Alan.
0
MarkAuthor Commented:
I'm going to be taking this server down shortly (controlled, not plug-yank) to check out the UPS, so I'll post back whatever I see. Otherwise, this is likely to be a long term experiment and I won't leave the question open that long.

I am going to modify my diagnostic script to email me if the local host and other LAN host's time differ. At least that way I'll be able to catch the problem, if any, before users get hours of mis-timestamped email.
0
MarkAuthor Commented:
well, we had another power outage -- and I noticed that the server is plugged into 'Surge Protection', not 'Battery Backup', so that's the reason it doesn't stay up!

Anyway, my diagnostic log worked (except it didn't email the message to me). Here's what was logged:

my date Thu Mar 5 07:30:20 EST 2015
webservers date Thu Mar 5 07:30:21 EST 2015
Starting from unexpected shutdown

You will notice that the hwclock setting (my date) is the same as the system time on another computer on the LAN. So, the hwclock was not the culprit. In this boot event, the date and time stayed correct and ntpd was running.

Conclusion: I think moving the ntpd startup script to after the DNS start up script did the trick. Not sure I understand ntpd completely. the ntp.conf has

server pool.ntp.org iburst
server  127.127.1.0     # local clock
fudge   127.127.1.0 stratum 10

which seems to me to indicate that it should check 127.127.1.0 if pool.ntp.org is not available. And what is 127.127.1.0 anyway? I thought my localhost IP was 127.0.0.1.

Last comments?
0
AlanConsultantCommented:
Hi,

Looks like you have it nailed with the swapping to follow DNS - good detective work!

My understanding is that 127.127.1.0 should be localhost (same as 127.0.0.1).

I believe that localhost is usually 127.0.0.0/8 so you can use anything in that range.

I am in front of a Win8.1 machine right now, but if I drop to a command prompt, and type 'route print' (without the quotes), it gives me (amongst other info):

IPv4 Route Table
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
          0.0.0.0          0.0.0.0      192.168.1.1    192.168.1.169     25
        127.0.0.0        255.0.0.0         On-link         127.0.0.1    306
        127.0.0.1  255.255.255.255         On-link         127.0.0.1    306
  127.255.255.255  255.255.255.255         On-link         127.0.0.1    306
      192.168.1.0    255.255.255.0         On-link     192.168.1.169    281
    192.168.1.169  255.255.255.255         On-link     192.168.1.169    281
    192.168.1.255  255.255.255.255         On-link     192.168.1.169    281
        224.0.0.0        240.0.0.0         On-link         127.0.0.1    306
        224.0.0.0        240.0.0.0         On-link     192.168.1.169    281
  255.255.255.255  255.255.255.255         On-link         127.0.0.1    306
  255.255.255.255  255.255.255.255         On-link     192.168.1.169    281
===========================================================================

Open in new window


I would guess I would see something similar on my Ubuntu boxes in the office.

Glad its working, and that you have the power plugged into the battery now :-)

Alan.
0
MarkAuthor Commented:
Thank's for that info on localhost.

Another wrinkle. I did a controlled shutdown in order to plug the computer into the battery-backed side of the UPS. Interesting results in my diagnosic log on reboot:
my date Thu Mar 5 07:30:20 EST 2015
webservers date Thu Mar 5 07:30:21 EST 2015
Starting from unexpected shutdown

my date Sat Mar 7 17:47:06 EST 2015
webservers date Sat Mar 7 12:47:09 EST 2015
my hwclock Sat Mar 7 17:47:07 2015 -0.354118 seconds
Starting from normal shutdown

Open in new window

The first group is from the power-outage shutdown from March 5, discussed above. The 2nd group is from my reboot which I did at 12:47 on Saturday. Notice that upon startup the hardware clock is set to 17:47!!!! This is with a controlled reboot!

Fortuntately, I have the line `/usr/sbin/netdate tcp 128.138.141.172` (time.nist.gov) later in the startup script, and it logged:

128.138.141.172 -18000.227 Sat Mar  7 12:47:07.000

setting the date back to the correct time. ntpd then started normally. When the time is way off, ntpd won't run.

This leads me to conclude that the hwclock command in the normal shutdown script is doing something bad. Here is my rc.0 shutdown script fragment:
# Save the system time to the hardware clock using hwclock --systohc.
if [ -x /sbin/hwclock ]; then
  # Check for a broken motherboard RTC clock (where ioports for rtc are
  # unknown) to prevent hwclock causing a hang:
  if ! grep -q -w rtc /proc/ioports ; then
    CLOCK_OPT="--directisa"
  fi
  if grep -q "^UTC" /etc/hardwareclock 2> /dev/null ; then
    echo "Saving system time to the hardware clock (UTC)."
    /sbin/hwclock $CLOCK_OPT --utc --systohc
  else
    echo "Saving system time to the hardware clock (localtime)."
    /sbin/hwclock  $CLOCK_OPT --localtime --systohc
  fi
fi

Open in new window

The `grep -w rtc /proc/ioports` statement does not find rtc and thus CLOCK_OPT does get set to "--directisa". The comment says it's to prevent hwclock causing a hang, but I've never experienced a hang without this.

the hwclock man page says:
      --directisa
              This option is meaningful only on an ISA machine or an Alpha (which implements enough
              of ISA to be, roughly speaking, an ISA machine for hwclock's  purposes).   For  other
              machines,  it  has no effect.  This option tells hwclock to use explicit I/O instruc-
              tions to access the Hardware Clock.  Without this option, hwclock will try to use the
              /dev/rtc  device  (which it assumes to be driven by the RTC device driver).  If it is
              unable to open the device (for reading), it will use the  explicit  I/O  instructions
              anyway.
I check this rtc in /proc/ioports in 4 other computer and none of them have it. Some of the motherboards I checked are quite old, but all are Slackware. I'd be interest to see if you Ubuntu box has it.

According to the man page, it should have no bad effect, but I'm wonder about that. I'm going to comment out this --directisa, try a reboot and see what happens. I definitely didn't see this behavior when I had Ubuntu running on 2 of these same machines last Fall, and then DNS had nothing to do with it.
0
AlanConsultantCommented:
Hi,

I have to note that you are definitely moving beyond my level of comfort - I have never adjusted those settings myself.  I can;t see that commenting out the --directisa would stop the machine booting up, but beyond that, I wouldn't like to guess.  As long as you can get back in and reinstate it, you should be okay.

For what its worth, my Ubuntu 12.04.5 LTS has the following line in /proc/ioports:

0070-007f : rtc0

There is no other occurrence of the string 'rtc' in the ioports file.

HTH,

Alan.
0
MarkAuthor Commented:
That's interesting, you do have "rtc" in /proc/ioports. I have nothing among 4 machines.

Anyway, I think I have a work-around including using `netdate  ime.nist.gov` in the rc.local start-up script. It's a bit of a kludge, but at least I get the correct time and ntpd runs. It's hard to imagine this is a Slackware bug. Someone would have caught it by now I would think.

I'll keep my eye on this --directisa option and see if that tells me anything.
0
MarkAuthor Commented:
I developed a work-around for the problem. Thanks to Alan3285 for working with me.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux

From novice to tech pro — start learning today.