[Last Call] Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 438
  • Last Modified:

Unix Script add date & time to redirect

Hi,

I run a program (egate, but does not matter in this) and I redirect the output using 1> and 2>

eg.
/pshrmcappkg1/app/egate/client/bin/stccb -rh gbdxgm04 -rs HP3000 -un Administrator -up STC -ln hp3000_cb -ss 1> /psmimcappkg1/app/log/egate/HP3000stdout.log 2> /psmimcappkg1/app/log/egate/HP3000stderr.log

This gives me 2, constantly growing log files, which is the whole idea.

BUT: the output gives NO indication of date/time whatsoever.  so this log file is getting really fustrating if I need to find something.

So I want to add a date/time stamp to this log.

I tryed to do this using unix cron:
00,10,20,30,40,50     *      *       *      *
echo "\r\nDATE & TIME: "`date -u '+%Y%m%d%H%M%S'` >> HP3000stdout.log

BUT: it seems so, that unix keep its fileposition, and overwrites the DATE & TIME again with its next log messages.

Now I am looking to a solution where we redirect to a script that adds date & time before writing the log, but I do not know if that is possible.  

Any other ideas.

Tim
0
thimerion
Asked:
thimerion
  • 5
  • 4
  • 3
  • +1
5 Solutions
 
ahoffmannCommented:
you cannot write this way from 2 sides to the same file
Either your application needs to write a timestam, or you need to pipe it to another application adding the timestamp, somthing like:

/psmimcappkg1/app/log/egate/HP3000stdout.log 2> | perl -MPOSIX -le 'print strftime("%D %T :",localtime()),$_ while <>' >/psmimcappkg1/app/log/egate/HP3000stderr.log
0
 
thimerionAuthor Commented:
Hi,

Thanks for your response, I get following error while trying:

$ /tsmkappkg1/app/egate/client/bin/stccb -rh tsmkappkg1 -rs HP3000 -un Administrator -up STC -ln hp3000_cb -ss 1> | perl -MPOSIX -le 'print strftime("%D %T :",localtime()),$_ while <>' > /tsmkappkg1/app/log/egate/HP3000stdout.log 2>&1
ksh: syntax error: `|' unexpected

Tim
0
 
ahoffmannCommented:
omit the
  >1

do you want to redirect STDERR also to the filter? Then use it like

   stcbb ....... 2>&1 | perl ..... > log
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
NovaDenizenCommented:
Why reinvent syslog?  This is exactly the use for which syslog was intended.

If you don't mind stdout and stderr getting mixed together:
/pshrmcappkg1/app/egate/client/bin/stccb -rh gbdxgm04 -rs HP3000 -un Administrator -up STC -ln hp3000_cb -ss | logger -plocal6.info

This runs stccb, sending all output to syslogd in the 'local6' channel with priority 'info'.  To capture this data to a log file, add a line like this to your /etc/syslog.conf:

local6.*                                  /var/log/HP3000.log

Don't forget to, as root, kill -0 your syslogd process so it re-reads the log config file.  
0
 
NovaDenizenCommented:
I didn't handle the output right in the stccb command.  It should have been:

..../stccb ..... 2>&1 | logger -plocal6.info

0
 
thimerionAuthor Commented:
Hi !!!

@ahoffmann: you gave me  great idea on how to do this!  Unfortunately the perl functions strftime and systime did not compile on my system.  Probably perl is not good initialized because we never use it.

Since I got the idea on how to do this, I could do the same with awk.  And the working thing became:
/tsmkappkg1/app/egate/client/bin/stccb -rh tsmkappkg1 -rs HP3000 -un Administrator -up STC -ln hp3000_cb -ss 2>&1 | awk -F= '{cmd="date -u '+%c'";cmd|getline d; close(cmd);print d, $1}' >> /tsmkappkg1/app/log/egate/HP3000out.log &

The processes it uses however afer looking very strange now, our sysadmin is particually worried about the process called <defunct>, does anyone knows where it comes from, what it does and how we get rid of it if it is harmfull.

egatest  4451  4450  5  May 12  ?         0:00 <defunct>
egatest  4450     1  0  May 12  ?         0:04 awk -F= {cmd="date -u +%c";cmd|getline d; close(cmd);print d, $

@NovaDenizen:
This solution is harder for me to try, because I am not the root administrator on the system.  I have a few questions before I try:
How will the date be formatted in the logfile?
I need multiple logfiles, can I send to other channels as local6 as well?

Thanks!!
Tim
0
 
NovaDenizenCommented:
I think it depends on your particular system, but you can count on the date having the month, day, hour, minute, and second.  I think some syslogd's will report hundredths of a second also.

Each syslog message is assigned a 'facility' and a 'severity level'.  The system-defined facilities are kernel, user, mail, daemon, auth, lpr, news, uucp, and cron.  There are also 8 local facilities, named local0 through local7 that are free for use by user applications.

The severity levels are, in descending order of importance, emerg, alert, crit, err, warning, notice, info, and debug.

The syslogd daemon takes all messages for the same facility, combines them into a single stream, and outputs them according to the configuration in /etc/syslog.conf.  It's possible to log only messages of a certain severity level or above, or log to remote hosts.  Read syslog.conf for the whole story.

Also, the logger command takes a '-t' option that automatically adds a tag to each message logged.  You could use multiple loggers on the same syslog channel, and give them different tags so you can differentiate their messages in the log file.
0
 
TimEliseoCommented:
I agree syslog is the cleanest approach, and the intermediate script the next best, but thought I'd clarify that your original approach *will* work, as long as all writers of the log file open it in append mode. The kernel guarantees the atomicity of the append operation, and the lines will not clobber each other. You just have to change your main log redirections to 1>> and 2>>, respectively. If you want to first clobber any existing file, first use a line like:

   > logfile

Tim
0
 
ahoffmannCommented:
you may use awk too, simply like:
   | gawk '{d=strftime("%T");print d" "$0}'

but keep in mind that this is with gawk only (probably mawk also)

According perl, you probaly just miss the POSIX module, see at http://www.cpan.org/
0
 
thimerionAuthor Commented:
@NovaDenizen : Thanks for the explaination, I have raised a request to the sysadmins to add a line in the syslog.conf.  Curious of the result.

@TimEliseo  : I don't understand what you are saying.

@ahoffmann : Is there any advantage of using gawk instead of awk, or is it just shorter in notation?

Regards,
Tim
0
 
ahoffmannCommented:
gawk is Gnu's version of awk
Most modern UNIX come with both versíons installed (you need to check your manuals or installation).
Even gawk's implementation is different to that of original awk, it is source compatible in most cases but with a lot of usefull extensions (like strftime:)
0
 
TimEliseoCommented:
Using your original cron-based timestamp append method, if you change your application run from:

/pshrmcappkg1/app/egate/client/bin/stccb -rh gbdxgm04 -rs HP3000 -un Administrator -up STC -ln hp3000_cb -ss 1> /psmimcappkg1/app/log/egate/HP3000stdout.log 2> /psmimcappkg1/app/log/egate/HP3000stderr.log

to:

/pshrmcappkg1/app/egate/client/bin/stccb -rh gbdxgm04 -rs HP3000 -un Administrator -up STC -ln hp3000_cb -ss 1>> /psmimcappkg1/app/log/egate/HP3000stdout.log 2>> /psmimcappkg1/app/log/egate/HP3000stderr.log

Then the log it generates will not overwrite the timestamps being appended by the cron job. (This is only guaranteed to work properly if the log file is local, not a network file over NFS.)
0
 
thimerionAuthor Commented:
@ahoffmann :
Cannot use gawk, it is not known and not planning to implement.

Thanks for the helpfull comments.

@NovaDenizen :
syslog is great, only pitty that it creates the log file as root.  I tried to touch it first with the egate user, but then it does not write in it.

Thanks for the helpfull comments to you to

@TimEliseo :
Thanks for the advice, but the other solutions give much more detail in date and time.

I now have to choose between the awk and the logger solution.  I  have running them both, on a different logfile.  So I can review in one week or so.

If someone still has an idea about the <defunct> process I mentioned earlier ... I am getting this with both awk and logger solutions.

Regards,
Tim
0
 
TimEliseoCommented:
The process marked <defunct> is called a zombie process. It should not concern your sysadmin. This is really a separate question, and is a PAQ with many references (that I don't have handy). Just look up "zombie process".
0
 
ahoffmannCommented:
zombies are what the name says: there but not existing
for more discussions and links see http:/Q_21367737.html
0

Featured Post

Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

  • 5
  • 4
  • 3
  • +1
Tackle projects and never again get stuck behind a technical roadblock.
Join Now