sharingsunshine
asked on
Need To Change Centos 6 Timestamp Form In Log File
I have a shell script that I use to analyze my log files and it works great on the access_log file because the timestamp is in this form.
Currently, my error_log file is in this form and I need it to be changed to the same form as the access_log.
Thanks,
03/Jan/2017:18:26:18 -0500
Currently, my error_log file is in this form and I need it to be changed to the same form as the access_log.
[Mon Jan 02 14:18:49.829562 2017]
Please tell me how to get the error_log timestamp the same as the access_log timestamp.Thanks,
ASKER
#
# This is the main Apache HTTP server configuration file. It contains the
# configuration directives that give the server its instructions.
# See <URL:http://httpd.apache.org/docs/2.4/> for detailed information.
# In particular, see
# <URL:http://httpd.apache.org/docs/2.4/mod/directives.html>
# for a discussion of each configuration directive.
#
# Do NOT simply read the instructions in here without understanding
# what they do. They're here only as hints or reminders. If you are unsure
# consult the online docs. You have been warned.
#
# Configuration and logfile names: If the filenames you specify for many
# of the server's control files begin with "/" (or "drive:/" for Win32), the
# server will use that explicit path. If the filenames do *not* begin
# with "/", the value of ServerRoot is prepended -- so 'log/access_log'
# with ServerRoot set to '/www' will be interpreted by the
# server as '/www/log/access_log', where as '/log/access_log' will be
# interpreted as '/log/access_log'.
#
# ServerRoot: The top of the directory tree under which the server's
# configuration, error, and log files are kept.
#
# Do not add a slash at the end of the directory path. If you point
# ServerRoot at a non-local disk, be sure to specify a local disk on the
# Mutex directive, if file-based mutexes are used. If you wish to share the
# same ServerRoot for multiple httpd daemons, you will need to change at
# least PidFile.
#
ServerRoot "/etc/httpd"
#
# Listen: Allows you to bind Apache to specific IP addresses and/or
# ports, instead of the default. See also the <VirtualHost>
# directive.
#
# Change this to Listen on specific IP addresses as shown below to
# prevent Apache from glomming onto all bound IP addresses.
#
#Listen 12.34.56.78:80
Listen 80
#
# Dynamic Shared Object (DSO) Support
#
# To be able to use the functionality of a module which was built as a DSO you
# have to place corresponding `LoadModule' lines at this location so the
# directives contained in it are actually available _before_ they are used.
# Statically compiled modules (those listed by `httpd -l') do not need
# to be loaded here.
#
# Example:
# LoadModule foo_module modules/mod_foo.so
#
Include conf.modules.d/*.conf
#
# If you wish httpd to run as a different user or group, you must run
# httpd as root initially and it will switch.
#
# User/Group: The name (or #number) of the user/group to run httpd as.
# It is usually good practice to create a dedicated user and group for
# running httpd, as with most system services.
#
User apache
Group apache
# 'Main' server configuration
#
# The directives in this section set up the values used by the 'main'
# server, which responds to any requests that aren't handled by a
# <VirtualHost> definition. These values also provide defaults for
# any <VirtualHost> containers you may define later in the file.
#
# All of these directives may appear inside <VirtualHost> containers,
# in which case these default settings will be overridden for the
# virtual host being defined.
#
#
# ServerAdmin: Your address, where problems with the server should be
# e-mailed. This address appears on some server-generated pages, such
# as error documents. e.g. admin@your-domain.com
#
ServerAdmin root@localhost
#
# ServerName gives the name and port that the server uses to identify itself.
# This can often be determined automatically, but we recommend you specify
# it explicitly to prevent problems during startup.
#
# If your host doesn't have a registered DNS name, enter its IP address here.
#
#ServerName www.example.com:80
#
# Deny access to the entirety of your server's filesystem. You must
# explicitly permit access to web content directories in other
# <Directory> blocks below.
#
<Directory />
AllowOverride none
Require all denied
</Directory>
#
# Note that from this point forward you must specifically allow
# particular features to be enabled - so if something's not working as
# you might expect, make sure that you have specifically enabled it
# below.
#
#
# DocumentRoot: The directory out of which you will serve your
# documents. By default, all requests are taken from this directory, but
# symbolic links and aliases may be used to point to other locations.
#
DocumentRoot "/var/www/html"
#
# Relax access to content within /var/www.
#
<Directory "/var/www">
AllowOverride None
# Allow open access:
Require all granted
</Directory>
# Further relax access to the default document root:
<Directory "/var/www/html">
#
# Possible values for the Options directive are "None", "All",
# or any combination of:
# Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews
#
# Note that "MultiViews" must be named *explicitly* --- "Options All"
# doesn't give it to you.
#
# The Options directive is both complicated and important. Please see
# http://httpd.apache.org/docs/2.4/mod/core.html#options
# for more information.
#
Options Indexes FollowSymLinks
#
# AllowOverride controls what directives may be placed in .htaccess files.
# It can be "All", "None", or any combination of the keywords:
# Options FileInfo AuthConfig Limit
#
AllowOverride All
#
# Controls who can get stuff from this server.
#
Require all granted
</Directory>
#
# DirectoryIndex: sets the file that Apache will serve if a directory
# is requested.
#
<IfModule dir_module>
DirectoryIndex index.html
</IfModule>
#
# The following lines prevent .htaccess and .htpasswd files from being
# viewed by Web clients.
#
<Files ".ht*">
Require all denied
</Files>
#
# ErrorLog: The location of the error log file.
# If you do not specify an ErrorLog directive within a <VirtualHost>
# container, error messages relating to that virtual host will be
# logged here. If you *do* define an error logfile for a <VirtualHost>
# container, that host's errors will be logged there and not here.
#
ErrorLog "logs/error_log"
#
# LogLevel: Control the number of messages logged to the error_log.
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
#
LogLevel warn
<IfModule log_config_module>
#
# The following directives define some format nicknames for use with
# a CustomLog directive (see below).
#
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
<IfModule logio_module>
# You need to enable mod_logio.c to use %I and %O
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
</IfModule>
#
# The location and format of the access logfile (Common Logfile Format).
# If you do not define any access logfiles within a <VirtualHost>
# container, they will be logged here. Contrariwise, if you *do*
# define per-<VirtualHost> access logfiles, transactions will be
# logged therein and *not* in this file.
#
#CustomLog "logs/access_log" common
#
# If you prefer a logfile with access, agent, and referer information
# (Combined Logfile Format) you can use the following directive.
#
CustomLog "logs/access_log" combined
</IfModule>
<IfModule alias_module>
#
# Redirect: Allows you to tell clients about documents that used to
# exist in your server's namespace, but do not anymore. The client
# will make a new request for the document at its new location.
# Example:
# Redirect permanent /foo http://www.example.com/bar
#
# Alias: Maps web paths into filesystem paths and is used to
# access content that does not live under the DocumentRoot.
# Example:
# Alias /webpath /full/filesystem/path
#
# If you include a trailing / on /webpath then the server will
# require it to be present in the URL. You will also likely
# need to provide a <Directory> section to allow access to
# the filesystem path.
#
# ScriptAlias: This controls which directories contain server scripts.
# ScriptAliases are essentially the same as Aliases, except that
# documents in the target directory are treated as applications and
# run by the server when requested rather than as documents sent to the
# client. The same rules about trailing "/" apply to ScriptAlias
# directives as to Alias.
#
ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"
</IfModule>
#
# "/var/www/cgi-bin" should be changed to whatever your ScriptAliased
# CGI directory exists, if you have that configured.
#
<Directory "/var/www/cgi-bin">
AllowOverride None
Options None
Require all granted
</Directory>
<IfModule mime_module>
#
# TypesConfig points to the file containing the list of mappings from
# filename extension to MIME-type.
#
TypesConfig /etc/mime.types
#
# AddType allows you to add to or override the MIME configuration
# file specified in TypesConfig for specific file types.
#
#AddType application/x-gzip .tgz
#
# AddEncoding allows you to have certain browsers uncompress
# information on the fly. Note: Not all browsers support this.
#
#AddEncoding x-compress .Z
#AddEncoding x-gzip .gz .tgz
#
# If the AddEncoding directives above are commented-out, then you
# probably should define those extensions to indicate media types:
#
AddType application/x-compress .Z
AddType application/x-gzip .gz .tgz
#
# AddHandler allows you to map certain file extensions to "handlers":
# actions unrelated to filetype. These can be either built into the server
# or added with the Action directive (see below)
#
# To use CGI scripts outside of ScriptAliased directories:
# (You will also need to add "ExecCGI" to the "Options" directive.)
#
#AddHandler cgi-script .cgi
# For type maps (negotiated resources):
#AddHandler type-map var
#
# Filters allow you to process content before it is sent to the client.
#
# To parse .shtml files for server-side includes (SSI):
# (You will also need to add "Includes" to the "Options" directive.)
#
AddType text/html .shtml
AddOutputFilter INCLUDES .shtml
</IfModule>
#
# Specify a default charset for all content served; this enables
# interpretation of all content as UTF-8 by default. To use the
# default browser choice (ISO-8859-1), or to allow the META tags
# in HTML content to override this choice, comment out this
# directive:
#
AddDefaultCharset UTF-8
<IfModule mime_magic_module>
#
# The mod_mime_magic module allows the server to use various hints from the
# contents of the file itself to determine its type. The MIMEMagicFile
# directive tells the module where the hint definitions are located.
#
MIMEMagicFile conf/magic
</IfModule>
#
# Customizable error responses come in three flavors:
# 1) plain text 2) local redirects 3) external redirects
#
# Some examples:
#ErrorDocument 500 "The server made a boo boo."
#ErrorDocument 404 /missing.html
#ErrorDocument 404 "/cgi-bin/missing_handler.pl"
#ErrorDocument 402 http://www.example.com/subscription_info.html
#
#
# EnableMMAP and EnableSendfile: On systems that support it,
# memory-mapping or the sendfile syscall may be used to deliver
# files. This usually improves server performance, but must
# be turned off when serving from networked-mounted
# filesystems or if support for these functions is otherwise
# broken on your system.
# Defaults if commented: EnableMMAP On, EnableSendfile Off
#
#EnableMMAP off
EnableSendfile on
# Supplemental configuration
#
# Load config files in the "/etc/httpd/conf.d" directory, if any.
IncludeOptional conf.d/*.conf
Apache version is
Server version: Apache/2.4.23 (Amazon)
Error log is to be read by eyes and you cannot re-format it in any way. Most log analyzers can handle both timestamps.
Hi sharingsunshine and gheist,
I cannot find a way to get the format exactly like the access_log format, but you can format the error log with the ErrorLogFormat directive. Here is the reference:
https://httpd.apache.org/docs/2.4/mod/core.html#errorlog
I cannot find a way to get the format exactly like the access_log format, but you can format the error log with the ErrorLogFormat directive. Here is the reference:
https://httpd.apache.org/docs/2.4/mod/core.html#errorlog
Wow , i need to refresh for apache 2.4 :)
Just copy relevant pieces from LogFormat then... status code is AP00000 (i.e error/event code) rest may or may not be URL
Centos 6 has apache 2.2 btw.
Just copy relevant pieces from LogFormat then... status code is AP00000 (i.e error/event code) rest may or may not be URL
Centos 6 has apache 2.2 btw.
Oh, I thought you did have 2.4. The httpd.conf file references 2.4 documentation.
ASKER
using httpd -v it shows 2.4.23 as you see above. So I am confident I have 2.4 too.
I am using an Amazon Linux Distribution which is Centos.
I am using an Amazon Linux Distribution which is Centos.
you can easily parse such dates using the shell.
actually the date command will parse it even without specifying a format.
actually the date command will parse it even without specifying a format.
$ LC_ALL=en date +%d/%b/%Y:%H:%M:%S --date 'Mon Jan 02 14:18:49.829562 2017'
02/Jan/2017:14:18:49
ASKER
Hi skullnobrains,
I put in the date command to my shell script but I am not sure if I did it correctly or not. So please clarify what I should put in to make it work.
It seemed it was best to put it at the top of the shell script. If that is the wrong placement please tell me.
I put in the date command to my shell script but I am not sure if I did it correctly or not. So please clarify what I should put in to make it work.
#!/bin/bash
date +"%d/%b/%Y:%H:%M:%S"
It seemed it was best to put it at the top of the shell script. If that is the wrong placement please tell me.
no. you'd need to read each line and call the date command for each of them
for example this one-line will call the date command for each log line and convert from one format to the other
explanation
- the first sed replaces % with %% in the log line
- while reads the input in a loop line by line
- IFS= sets the input separator to ']' so the read command parses the date in the first var ( $date ) and the rest of the line in the second ( $line )
- for each line we call the date command which converts the date. we append the line in the format. this is where replacing % with %% kicks in because this prevents date from inperpreting % signs in the log lines as part of the date format
- ${date#[} is the content of the date variable without the leading '['
you probably can pipe this command into your existing script
for example this one-line will call the date command for each log line and convert from one format to the other
sed 's/%/%%/g' | while IFS=']' read date line ; do LC_ALL=en date +%d/%b/%Y:%H:%M:%S" $line" --date "${date#[}" ; done <input >output
explanation
- the first sed replaces % with %% in the log line
- while reads the input in a loop line by line
- IFS= sets the input separator to ']' so the read command parses the date in the first var ( $date ) and the rest of the line in the second ( $line )
- for each line we call the date command which converts the date. we append the line in the format. this is where replacing % with %% kicks in because this prevents date from inperpreting % signs in the log lines as part of the date format
- ${date#[} is the content of the date variable without the leading '['
you probably can pipe this command into your existing script
sed -e 's/%/%%/g' log_whatevr.log | while IFS=']' read date line ; do LC_ALL=en date +%d/%b/%Y:%H:%M:%S" $line" --date "${date#[}" ; done | sh uour_script.sh
sed 's/^.// ; s/]/\t/' | while read a b c d line ; do echo `LC_ALL=en date +%d/%b/%Y:%H:%M:%S --date "$a $b $c $d"` $line ; done
easier to read
ASKER
I am at a loss how to use the code above. I put it at the top of the shell program but it doesn't do anything when I run the shell it just acts like it is working but provides no output. No error messages are shown either.
the above reads from standard input and writes to standard output
you need to use something like
or possibly integrate the name of the log file and your analysis tool in the script
you need to replace whatever.log with the proper log file and your_existing_script with your actual script
and you need to understand a little shell basics. using things when you don't understand what they do is dangerous
you need to use something like
script.sh <input.log >output.log
or possibly integrate the name of the log file and your analysis tool in the script
sed -e 's/^.// ; s/]/\t/' /var/log/whatever.log| while read a b c d line ; do echo `LC_ALL=en date +%d/%b/%Y:%H:%M:%S --date "$a $b $c $d"` $line ; done | sh your_existing_script.sh
you need to replace whatever.log with the proper log file and your_existing_script with your actual script
and you need to understand a little shell basics. using things when you don't understand what they do is dangerous
ASKER
thanks for the further clarification. Here is what I used
and here is the error I am getting
After the times being stated it is supposed to count the number of errors by IP in a certain time period. Instead it is showing the error_log content. Stating there is a problem reading the file. I checked %d is for the two digit representation of the date.
Regarding knowing something better, I agree but there is only so much time in a day so I am having to prioritize. I am the only person keeping our business going online. Our income is down considerably due to Google's algorithm changes so my focus has been to learn some tools to help us with serp rankings. In the meantime I have bandwidth stealers that need to be blocked or the site will go down due to the memory being overloaded. Also, I lost my wife in November of 40 years so my brain has been scrambled somewhat since I was her primary caregiver for 5 years before she passed.
This is why I use Experts to fill in the gaps for those things I don't know how to do. That being said, I would welcome a suggested tutorial I can study and learn (other than the man pages, I do better with examples rather than just a reference manual). I also have enrolled in several classes in udemy to get myself more up to speed. Admittedly, I don't know the intricacies of changing this log format that's why I posted the question.
Thanks,
sed -e 's/^.// ; s/]/\t/' /var/log/httpd/error_log | while read a b c d line ; do echo `LC_ALL=en date +%d/%b/%Y:%H:%M:%S --date "$a $b $c $d"` $line ; done | sh modified_gawk_errors.sh "09 Jan 2017" 11:00:00 "09 Jan 2017" 11:30:00
and here is the error I am getting
3076] [client 1.1.1.1:22592] PHP Notice: Undefined index: HTTP_HOST in /var/www/html/wp-content/plugins/s2member-pro/includes/syscon.inc.php on line 136
date: invalid date '0/Jan/2017:07:08:43 2017 [:error [pid'
3076] [client 1.1.1.1:22592] PHP Notice: Undefined index: host in /var/www/html/wp-includes/canonical.php on line 444
date: invalid date '0/Jan/2017:07:08:43 2017 [:error [pid'
3076] [client 1.1.1.1:22592] PHP Notice: Undefined index: host in /var/www/html/wp-includes/canonical.php on line 445
date: invalid date '0/Jan/2017:07:08:43 2017 [:error [pid'
3076] [client 1.1.1.1:22592] PHP Notice: Undefined index: host in /var/www/html/wp-includes/canonical.php on line 445
date: invalid date '0/Jan/2017:07:08:43 2017 [:error [pid'
3076] [client 1.1.1.1:22592] PHP Notice: Undefined index: host in /var/www/html/wp-includes/canonical.php on line 446
date: invalid date '0/Jan/2017:07:08:43 2017 [:error [pid'
3076] [client 1.1.1.1:22592] PHP Notice: Undefined index: host in /var/www/html/wp-includes/canonical.php on line 448
Examining from Mon Jan 9 11:00:00 EST 2017 (1483977600)
to Mon Jan 9 11:30:00 EST 2017 (1483979400)
After the times being stated it is supposed to count the number of errors by IP in a certain time period. Instead it is showing the error_log content. Stating there is a problem reading the file. I checked %d is for the two digit representation of the date.
Regarding knowing something better, I agree but there is only so much time in a day so I am having to prioritize. I am the only person keeping our business going online. Our income is down considerably due to Google's algorithm changes so my focus has been to learn some tools to help us with serp rankings. In the meantime I have bandwidth stealers that need to be blocked or the site will go down due to the memory being overloaded. Also, I lost my wife in November of 40 years so my brain has been scrambled somewhat since I was her primary caregiver for 5 years before she passed.
This is why I use Experts to fill in the gaps for those things I don't know how to do. That being said, I would welcome a suggested tutorial I can study and learn (other than the man pages, I do better with examples rather than just a reference manual). I also have enrolled in several classes in udemy to get myself more up to speed. Admittedly, I don't know the intricacies of changing this log format that's why I posted the question.
Thanks,
%d is for the two digit representation of the date.
it should be.
can you test to run just "date +%d" just to check ?
can you post the output of the following please ?
sed -e 's/^.// ; s/]/\t/' /var/log/httpd/error_log | while read a b c d line ; do echo `LC_ALL=en date +%d/%b/%Y:%H:%M:%S --date "$a $b $c $d"` $line ; done | head
and
head /var/log/httpd/error_log
ASKER
[root@ip-172-31-22-236 ~]# date +%d
11
[root@ip-172-31-22-236 ~]# sed -e 's/^.// ; s/]/\t/' /var/log/httpd/error_log | while read a b c d line ; do echo `LC_ALL=en date +%d/%b/%Y:%H:%M:%S --date "$a $b $c $d"` $line ; done | head
08/Jan/2017:03:42:01 2017 [notice] Digest: generating secret for digest authentication ...
08/Jan/2017:03:42:01 2017 [notice] Digest: done
08/Jan/2017:03:42:03 2017 [notice] Apache/2.2.31 (Unix) DAV/2 PHP/5.3.29 mod_ssl/2.2.31 OpenSSL/1.0.1k-fips mod_wsgi/3.2 Python/2.7.12 mod_perl/2.0.7 Perl/v5.16.3 configured -- resuming normal operations
08/Jan/2017:03:59:54 2017 [warn-phpd] mmap cache can't open /data/web/thefrugallife.com/openx/www/delivery/../../var/cache/deliverycache_aab5ee543b24b515ff6e342d00c73601.php - Permission denied (pid 26843)
08/Jan/2017:03:59:54 2017 [warn-phpd] mmap cache can't open /data/web/thefrugallife.com/openx/www/delivery/../../var/cache/deliverycache_aab5ee543b24b515ff6e342d00c73601.php - Permission denied (pid 26843)
08/Jan/2017:03:59:54 2017 [warn-phpd] mmap cache can't open /data/web/thefrugallife.com/openx/www/delivery/../../var/cache/deliverycache_a62501dd22c986875edc5f231e2bb7a9.php - Permission denied (pid 26843)
08/Jan/2017:03:59:54 2017 [warn-phpd] mmap cache can't open /data/web/thefrugallife.com/openx/www/delivery/../../var/cache/deliverycache_a62501dd22c986875edc5f231e2bb7a9.php - Permission denied (pid 26843)
08/Jan/2017:04:10:42 2017 [warn-phpd] mmap cache can't open /data/web/thefrugallife.com/openx/www/delivery/../../var/cache/deliverycache_aab5ee543b24b515ff6e342d00c73601.php - Permission denied (pid 26905)
08/Jan/2017:04:10:42 2017 [warn-phpd] mmap cache can't open /data/web/thefrugallife.com/openx/www/delivery/../../var/cache/deliverycache_aab5ee543b24b515ff6e342d00c73601.php - Permission denied (pid 26905)
08/Jan/2017:04:10:42 2017 [warn-phpd] mmap cache can't open /data/web/thefrugallife.com/openx/www/delivery/../../var/cache/deliverycache_a62501dd22c986875edc5f231e2bb7a9.php - Permission denied (pid 26905)
[root@ip-172-31-22-236 thefrugallife.com]# head /var/log/httpd/error_log
[Sun Jan 08 03:42:01 2017] [notice] Digest: generating secret for digest authentication ...
[Sun Jan 08 03:42:01 2017] [notice] Digest: done
[Sun Jan 08 03:42:03 2017] [notice] Apache/2.2.31 (Unix) DAV/2 PHP/5.3.29 mod_ssl/2.2.31 OpenSSL/1.0.1k-fips mod_wsgi/3.2 Python/2.7.12 mod_perl/2.0.7 Perl/v5.16.3 configured -- resuming normal operations
[Sun Jan 08 03:59:54 2017] [warn-phpd] mmap cache can't open /data/web/thefrugallife.com/openx/www/delivery/../../var/cache/deliverycache_aab5ee543b24b515ff6e342d00c73601.php - Permission denied (pid 26843)
[Sun Jan 08 03:59:54 2017] [warn-phpd] mmap cache can't open /data/web/thefrugallife.com/openx/www/delivery/../../var/cache/deliverycache_aab5ee543b24b515ff6e342d00c73601.php - Permission denied (pid 26843)
[Sun Jan 08 03:59:54 2017] [warn-phpd] mmap cache can't open /data/web/thefrugallife.com/openx/www/delivery/../../var/cache/deliverycache_a62501dd22c986875edc5f231e2bb7a9.php - Permission denied (pid 26843)
[Sun Jan 08 03:59:54 2017] [warn-phpd] mmap cache can't open /data/web/thefrugallife.com/openx/www/delivery/../../var/cache/deliverycache_a62501dd22c986875edc5f231e2bb7a9.php - Permission denied (pid 26843)
[Sun Jan 08 04:10:42 2017] [warn-phpd] mmap cache can't open /data/web/thefrugallife.com/openx/www/delivery/../../var/cache/deliverycache_aab5ee543b24b515ff6e342d00c73601.php - Permission denied (pid 26905)
[Sun Jan 08 04:10:42 2017] [warn-phpd] mmap cache can't open /data/web/thefrugallife.com/openx/www/delivery/../../var/cache/deliverycache_aab5ee543b24b515ff6e342d00c73601.php - Permission denied (pid 26905)
[Sun Jan 08 04:10:42 2017] [warn-phpd] mmap cache can't open /data/web/thefrugallife.com/openx/www/delivery/../../var/cache/deliverycache_a62501dd22c986875edc5f231e2bb7a9.php - Permission denied (pid 26905)
Thanks for sticking with this problem. By the way, there is no openx at that location.
[root@ip-172-31-22-236 thefrugallife.com]# pwd
/data/web/thefrugallife.com
[root@ip-172-31-22-236 thefrugallife.com]# ls
1 maintenance.html
12all manager
BBCblogger PhotoAlbum
BCCblog phpBB3
BCCblog.bak phpinfo.php
BHC plugins_bak
blocklist.txt readme.b457e4cf27ae22e403a24d6ec8a613bf.html
blog removing.html
blogger Resources
captcha rjw
cgi-bin robots.txt
comments ror.xml
comments.bak scanresult_modified.txt
comment_script.zip sendy
confirmsuccessezine.php sendy2
contact_2smf_new.php shtml
contact_2smf_oldFeb102012.php Signup.php
contact_2smf.php sitemap.xml
contact_3smf.php smf
contact copy.php smfconfig.php
contact_oldFeb102012.php smfpost.php
contact.php spell.php
contact.php.bak temp_db
contact_smf.php Templates
CSS test_api.php
error_log test_geo_db.php
evasive_test.pl testgeoip.php
footer.php test_mail.php
formmailer copy.php testregexp.php
formmailer_old.php test_smf.php
formmailer.php TFLblog
formmailer_verify.php TFLblogger
f.txt uebimiau
gallery users
generator vanilla
geoip.inc vdsbackup
geoiptest.php wordfence-waf.php
geotest.php wordpress_test
google31419c7ec89b2d9f.html wp-activate.php
google3b31fec62659cadc.html wp-admin
google6deb0b5fe22fbf9d.html wp-blog-header.php
helpers wp-comments-post.php
html_import wp-config.php
images wp-config-sample.php
index_1.php wp-content
index.php wp-cron.php
index_smf.php wp-includes
INSTALL wp-links-opml.php
Library wp-load.php
license.txt wp-login.php
lists wp-mail.php
lists_old wp-settings.php
login_cron_post.php wp-signup.php
login_form_example.php wp-trackback.php
login_oldFeb102012.php xmlrpc.php
login.php YaBBImages
given the above, the date transformation is more or less ok but days with a single digit end up with a leading zero.
i assume the gawk is unhappy with dates with a leading 0
using %e instead might work but will add a space at the beginning of each line
you can either
- try with %e ( the existing gawk most likely will be more tolerant as the date will end in $1 anyway )
- modify the gawk so it allows such dates
- or remove the zero if it exists manually with an additional sed after the while loop :
--
i assume the gawk is unhappy with dates with a leading 0
using %e instead might work but will add a space at the beginning of each line
you can either
- try with %e ( the existing gawk most likely will be more tolerant as the date will end in $1 anyway )
- modify the gawk so it allows such dates
- or remove the zero if it exists manually with an additional sed after the while loop :
sed -e 's/^.// ; s/]/\t/' /var/log/httpd/error_log | while read a b c d line ; do echo `LC_ALL=en date +%d/%b/%Y:%H:%M:%S --date "$a $b $c $d"` $line ; done | sed 's/^0//' | your awk script'
--
By the way, there is no openx at that location.i have no idea what you're talking about
ASKER
When I use your command above it doesn't execute.
sed -e 's/^.// ; s/]/\t/' /data/log/php_errors_thp.log | while read a b c d line ; do echo `LC_ALL=en date +%d/%b/%Y:%H:%M:%S --date "$a $b $c $d"` $line ; done | sed 's/^0//' | modified_gawk_errors.sh'
>
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Looks like it is ready to be piped. Here is the sed command I used
Here is the log file
sed -e 's/^.// ; s/]/\t/' /var/log/httpd/error_log | while read a b c d e line ; do echo `LC_ALL=en date +%d/%b/%Y:%H:%M:%S' '%z --date "$a $b $c $d $e"` $line ; done
Here is the log file
19/Jan/2017:23:17:36 -0500 [error] [client 1.1.1.1] PHP Notice: Undefined variable: id_currency in /data/web/theherbsplace.com/modules/naturessunshinepayment/controllers/front/validation.php on line 104, referer: https://www.theherbsplace.com/module/naturessunshinepayment/payment
19/Jan/2017:23:21:19 -0500 [error] [client 1.1.1.1] PHP Notice: Undefined variable: id_currency in /data/web/theherbsplace.com/modules/naturessunshinepayment/controllers/front/validation.php on line 104, referer: https://www.theherbsplace.com/module/naturessunshinepayment/payment
20/Jan/2017:02:34:24 -0500 [error] [client 1.1.1.1] rejecting client initiated renegotiation
20/Jan/2017:04:00:01 -0500 [error] [client 1.1.1.1] PHP Notice: Undefined index: search_query in /data/web/theherbsplace.com/cache/smarty/compile/a8/83/ef/a883ef2c0631a5e076346702de2545ef8e4e8c0a.file.search.tpl.php on line 48, referer: http://www.google.com
looks good. does it work when you pipe it to your script ?
ASKER
Yes, it does work. Thanks for all your help.
ASKER
I am having trouble closing the question. I have alerted a moderator.
ASKER
Great help, thanks for your perserverance
Can you provide the output of your current httpd.conf file? Also, what version of apache are you running?