cpwems
asked on
Parsing another programs log file
I've been collecting log files from a program I'm using and 90% of the time I can correctly parse the log with my current script. Unfortunetly I can find no rhyme or reason to the header file they use for the log file, it is never exactly the same characters, bits or end in the same character. The data I want is alwasy in the following form:
11,22,3333,4444,555555,666
Now the problem I am having is that sometimes the header has a comma "," in it and the cut command will keep that line of the header. I've only experienced one line having this problem at the top of the file but would like to check all lines for this common format and remove the lines that do not follow this format. I am new at bash scripting and can not figure out how to do this.
NOTE: the 12th comma deliminated info could contain a comma in it which is also another problem I can't figure out. If there is a comma in the 12th spot it could give me 13,14,15,ect fields if I use the cut command. Is there a way to have the last field just include everything after the last comma or does it already do that? The cut command I've been using is:
cut -f "1 2 3 4 5 6 7 8 9 10 11 12" -d , -s < file.log
Thanks for any help.
11,22,3333,4444,555555,666
Now the problem I am having is that sometimes the header has a comma "," in it and the cut command will keep that line of the header. I've only experienced one line having this problem at the top of the file but would like to check all lines for this common format and remove the lines that do not follow this format. I am new at bash scripting and can not figure out how to do this.
NOTE: the 12th comma deliminated info could contain a comma in it which is also another problem I can't figure out. If there is a comma in the 12th spot it could give me 13,14,15,ect fields if I use the cut command. Is there a way to have the last field just include everything after the last comma or does it already do that? The cut command I've been using is:
cut -f "1 2 3 4 5 6 7 8 9 10 11 12" -d , -s < file.log
Thanks for any help.
ASKER
Below are two examples of the beggining of the log file. The first example the data line starts at '0f,00,00,806050a0' and the second example the data line starts at 'c8,00,00,ff5020ff,' and each data line is seperated by the NUL character '^@'. I have many log files each about 4.5k, so I can send whole files if anyone needs.
Sample 1:
^@^@M^@¹^@^H^Ah^AÈ^A^W^B<8
<8a>
ö
E^K<96>^K^F^Lq^LÄ^L/^M<9a>
^Nz^Nê^N<^O^O^A^Pq^Pá^PQ^
Sample 2:
^@^@S^@<8c>^@à^@6^A<87>^A÷
^HZ^HÉ^H^Y k ¹ &
v
Ò
#^K<93>^Kæ^K7^L§^Lú^LJ^M¹^
This is the script I have written so far:
for i in $log_dir/*; do
if [ -f $i ]; then
# if the file is there
filename=${i#$log_dir/}
tr '\0' '\n' < $log_dir/$filename > temp.log
csplit -s temp.log "/\0/"
if [ -f xx01 ]; then
cut -f1-12 -d , -s xx01 > $clean_dir/$filename
fi
rm -rf temp.log
rm -rf xx*
fi
done
Thanks for the short command Tintin.
Sample 1:
^@^@M^@¹^@^H^Ah^AÈ^A^W^B<8
<8a>
ö
E^K<96>^K^F^Lq^LÄ^L/^M<9a>
^Nz^Nê^N<^O^O^A^Pq^Pá^PQ^
Sample 2:
^@^@S^@<8c>^@à^@6^A<87>^A÷
^HZ^HÉ^H^Y k ¹ &
v
Ò
#^K<93>^Kæ^K7^L§^Lú^LJ^M¹^
This is the script I have written so far:
for i in $log_dir/*; do
if [ -f $i ]; then
# if the file is there
filename=${i#$log_dir/}
tr '\0' '\n' < $log_dir/$filename > temp.log
csplit -s temp.log "/\0/"
if [ -f xx01 ]; then
cut -f1-12 -d , -s xx01 > $clean_dir/$filename
fi
rm -rf temp.log
rm -rf xx*
fi
done
Thanks for the short command Tintin.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
The data does have control characters like ^A and ^B which indicate what color the text is supose to be from that point on.
Instead of your cut, try:
sed "s/[^a-f0-9]*\([a-f0-9][a-
sed "s/[^a-f0-9]*\([a-f0-9][a-
ASKER
Ok that suggestion helped out as long as I piped it with the cut, but I've found other problems, here are some more raw sample files:
sample3:
^@^@B^@<9b>^@^A^AV^A¯^A^B^
sample4:
^@^@q^@â^@3^A×^A^T^Be^B®^B
®
^N^Kj^KÐ^KB^L ^Lø^L{^MÊ^MV^N¼^N^P^Oe^Oº^
The problem with sample4 is that in the header there is a comma and it is left over with:
,
79,03,00,80c0c050,0000008c
79,03,00,80c0c050,0000008d
Tintin thanks again for all your help. I've been trying for weeks to do this myself.
sample3:
^@^@B^@<9b>^@^A^AV^A¯^A^B^
sample4:
^@^@q^@â^@3^A×^A^T^Be^B®^B
®
^N^Kj^KÐ^KB^L ^Lø^L{^MÊ^MV^N¼^N^P^Oe^Oº^
The problem with sample4 is that in the header there is a comma and it is left over with:
,
79,03,00,80c0c050,0000008c
79,03,00,80c0c050,0000008d
Tintin thanks again for all your help. I've been trying for weeks to do this myself.
You should be able to just run:
grep , xx01 | sed "s/[^a-f0-9]*\([a-f0-9][a-
It works for me (note I have changed the control characters to X's and added newlines for the nulls to make it easier to test).
$ cat file
^@^@q^@â^@3^A×^A^T^Be^B®^B
®
XX,YY,ZZ79,03,00,80c0c050,
79,03,00,80c0c050,0000008d
$ grep , file | sed "s/[^a-f0-9]*\([a-f0-9][a-
79,03,00,80c0c050,0000008c
79,03,00,80c0c050,0000008d
grep , xx01 | sed "s/[^a-f0-9]*\([a-f0-9][a-
It works for me (note I have changed the control characters to X's and added newlines for the nulls to make it easier to test).
$ cat file
^@^@q^@â^@3^A×^A^T^Be^B®^B
®
XX,YY,ZZ79,03,00,80c0c050,
79,03,00,80c0c050,0000008d
$ grep , file | sed "s/[^a-f0-9]*\([a-f0-9][a-
79,03,00,80c0c050,0000008c
79,03,00,80c0c050,0000008d
This looks like a job for awk, I wonder why it hasn't been suggested...
ASKER
Well Tintin set me in the right direction in thought for me to solve it myself. Found some huge unix book and just looked through it at all the commands and stumbled on the dd command. So here is the following code that works on my log files. I never realized that the control characters took up two bytes, hence why I couldn't figure out the size of the header. Would still love to know what is in the header but it's not important.
log_dir='/home/tabber/ffxi
clean_dir='/home/tabber/cl
for i in $log_dir/*; do
if [ -f $i ]; then
# if the file is there
filename=${i#$log_dir/}
dd bs=1 skip=100 < $log_dir/$filename | tr '\0' '\n' > $clean_dir/$filename
fi
done
I'm not to sure how to assign points but Tintin you will get them all, please let me know if I do it wrong.
log_dir='/home/tabber/ffxi
clean_dir='/home/tabber/cl
for i in $log_dir/*; do
if [ -f $i ]; then
# if the file is there
filename=${i#$log_dir/}
dd bs=1 skip=100 < $log_dir/$filename | tr '\0' '\n' > $clean_dir/$filename
fi
done
I'm not to sure how to assign points but Tintin you will get them all, please let me know if I do it wrong.
BTW, your current cut command can be reduced to
cut -f1-12 -d, -s <file.log