Solved

# using perl or shell delete column 2, 3, & 5 in text file and make binned rolling averages per month

Posted on 2012-03-16
344 Views
Alright. See complete file attached. The files starts like this below. The first column is the date and the second column is the time which can be deleted.  Columns 3 & 6 are not necessary and can be deleted. The 4th  column is uncorrected data which can also be deleted. Aside from deleting columns 2-4 & 6, I want to create a binned rolling average which will be added in a third column.So the third column won't start until month 12 since 12 data points are needed so rows 1-11 will only have two columns of data and the 3rd column of data begins from row 12.

Oulu Neutron Monitor database query - http://cosmicrays.oulu.fi/
Start: 1979/01/01 00:00 End: 2012/01/31 23:00
Units for date column: fractional years
Columns: #1, #2 -date and time, #3 - fractional date, #4, #5 - uncorrected and corrected (for pressure and efficiency ) count rates [counts/min],
#6 - barometric pressure [mbar]
Effectiveness correction factors:
1 (from 1964-01-01)
1.00674 (from 1985-10-01)
1.01147 (from 1995-01-01)
1.00914 (from 2000-01-01)
1.00813 (from 2003-06-01)
1.0029 (from 2008-08-01)
1.0019 (from 2009-11-01)
Pressure correction factors: -0.0074 (from 1964-01-01)
Resolution: 1 months
Note: query times are adjusted to 1979-01-01 00:00 - 2012-01-01 00:00
1979.01.01 00:00:00 1979.0000000 5612 6179 1013.86
1979.02.01 00:00:00 1979.0849315 5810 6152 1009.25
1979.03.01 00:00:00 1979.1616438 5854 6077 1005.37
cray.txt
0
Question by:libertyforall2

LVL 31

Expert Comment

ID: 37732457
What needs to be done to the non numeric rows?
Could you please give a sample output?
0

LVL 10

Accepted Solution

pfrancois earned 500 total points
ID: 37733420
This shell script gives you the result you want:

``````#!/bin/bash

sed '1,/^Note:/d' cray.txt \
| grep -v seconds \
| cut -d ' ' -f 1,5 \
| awk -F ' ' '{
if (++j > 12) {
j = 1
};
if (++i < 12) {
last12 [j] = \$2;
tot += \$2;
printf ("%s %d\n", \$1, \$2);
} else {
tot -= last12 [j];
last12 [j] = \$2;
tot += \$2;
printf ("%s %d %d\n", \$1, \$2, tot/12);
}
}'
``````
0

Author Closing Comment

ID: 37780019
Works.
0