awk command single

zgrep 'MobileDevice' xyz_1.log.20181121.gz|grep 'USER123'| awk '$0>= "2018-Nov-21 00:01" && $0<="2018-Nov-21 23:59"' | awk -F '|' '$24>300 {print}'

zgrep 'MobileDevice' xyz_2.log.20181121.gz|grep 'USER123'| awk '$0>= "2018-Nov-21 00:01" && $0<="2018-Nov-21 23:59"' | awk -F '|' '$24>300 {print}'

zgrep 'MobileDevice' xyz_3.log.20181121.gz|grep 'USER123'| awk '$0>= "2018-Nov-21 00:01" && $0<="2018-Nov-21 23:59"' | awk -F '|' '$24>300 {print}'

i am using above 3 separate greps one after other on 3 types of jvm files

how to combine them as one

zgrep 'MobileDevice' xyz_?.log.20181121.gz|grep 'USER123'| awk '$0>= "2018-Nov-21 00:01" && $0<="2018-Nov-21 23:59"' | awk -F '|' '$24>300 {print}'
i tried above which did not work
i tried below that also did not work


zgrep 'MobileDevice' xyz_*.log.20181121.gz|grep 'USER123'| awk '$0>= "2018-Nov-21 00:01" && $0<="2018-Nov-21 23:59"' | awk -F '|' '$24>300 {print}'
please advise
LVL 7
gudii9Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

nociSoftware EngineerCommented:
Both those command should work...
Did you verify parts of the commands...

zgrep 'MobileDevice' xyz_1.log.20181121.gz|grep 'USER123'  >t.1
zgrep 'MobileDevice' xyz_2.log.20181121.gz|grep 'USER123' >>t.1
zgrep 'MobileDevice' xyz_3.log.20181121.gz|grep 'USER123' >>t.1
sort <t.1 >t.2
zgrep 'MobileDevice' xyz_3.log.20181121.gz|grep 'USER123' | sort >t.3
diff t.2 t.3     # should be the same output...

Open in new window


btw why not compare $1 == "2018-Nov-21" ?
or even: use  zgrep "^2018-Nov-21 " in stead of awk.
arnoldCommented:
For such pattern crunchers, I'd use perl.

In your situation Presumably I by now know the pattern of interest over time, I would either process the data when it is saved into the log file, presumably the data is being added by syslog/rsyslog.
The other is process the log prior to compression during the log file  rotation on the first role.

If properly setup, the processing could be modified to add additional entries, pattern.
Then your automated process has the number of rotation log file count to recapture.

This way the data you are after most of the time would already been processed and made available.

If you store the extracted in a db, it will be searchable and easily accessible.

If this is a regulated entity, you could set the archival, retention policy of the extracted data.
gudii9Author Commented:
Both those command should work...
not working.

all files in same directory though
Get a highly available system for cyber protection

The Acronis SDI Appliance is a new plug-n-play solution with pre-configured Acronis Software-Defined Infrastructure software that gives service providers and enterprises ready access to a fault-tolerant system, which combines universal storage and high-performance virtualization.

Learn More
gudii9Author Commented:
any sample commands like this in any link or resource or book?
arnoldCommented:
Lookup logrotate.conf
Postrotate after the rotation perform the following action/s.
 
Look at passing data from syslog/rsyslog.

You are not including info in your setup nor on what your end goal is.

Your question is always, I have two pieces of wood, and these nails.

In such a limited set of info, the answers you get are similarly limited.
simon3270Commented:
If you have multiple files as arguments to zgrep (same for grep), it puts the filename at the beginnign of each line, and that is messing up your awk looking for the date.

Two ways to avoid this - either combine all of the files into one:

zcat xyz_?.log.20181121.gz | grep MobileDevice | grep USER123  etc....

Open in new window


Or suppress the adding of the filename

  
zgrep -h MobileData xyz_?.log.20181121.gz

Open in new window


There are other improvements you can make: compare the date against 2018-Nov-21 rather than a range, put all of the greps into one statement (assuming that they are always in the same order), and do the grep as part of an awk script, but getting it to work in the first place is more important!
gudii9Author Commented:
zgrep -h MobileData xyz_?.log.20181121.gz

what is -h option means
i have both non zip files
nociSoftware EngineerCommented:
From the man page:

      -h, --no-filename
              Suppress  the  prefixing  of  file names on output.  This is the
              default when there is only one file (or only standard input)  to
              search.

See: https://linux.die.net/man/1/grep

zgrep uses "zcat" before grep..., zcat can read non-compressed files as well.
If no file is  specified,  then  the
       standard input is decompressed if necessary and fed to grep.  Otherwise
       the given files are uncompressed if necessary and fed to grep.
See: https://linux.die.net/man/1/zgrep
gudii9Author Commented:
grep 'search' xyz_?.log.20181121

above fetched results from both
xyz_1.log.20181121
and
xyz_2.log.20181121

similarly
 grep 'search' xyz_*.log.20181121

above fetched results from both
xyz_1.log.20181121
and
xyz_2.log.20181121

when to use * when to use ?

please advise
simon3270Commented:
? matches a single character, * matches any number of characters (including zero).

In your case, there is only one character that varies, so either will find all matches.

What happens if there are more than 9 matching files? If, for example, the name is like xyz_10.log.20181121, then you are better off using the *, so that a single pattern matches xyz_1.log.20181121 and xyz_10.log.20181121.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux

From novice to tech pro — start learning today.