Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

using perl or shell delete column 2, 3, & 5 in text file and make binned rolling averages per month

Posted on 2012-03-16
3
Medium Priority
?
388 Views
Last Modified: 2012-06-22
Alright. See complete file attached. The files starts like this below. The first column is the date and the second column is the time which can be deleted.  Columns 3 & 6 are not necessary and can be deleted. The 4th  column is uncorrected data which can also be deleted. Aside from deleting columns 2-4 & 6, I want to create a binned rolling average which will be added in a third column.So the third column won't start until month 12 since 12 data points are needed so rows 1-11 will only have two columns of data and the 3rd column of data begins from row 12.




Oulu Neutron Monitor database query - http://cosmicrays.oulu.fi/
Start: 1979/01/01 00:00 End: 2012/01/31 23:00
Units for date column: fractional years
Columns: #1, #2 -date and time, #3 - fractional date, #4, #5 - uncorrected and corrected (for pressure and efficiency ) count rates [counts/min],
#6 - barometric pressure [mbar]
Effectiveness correction factors:
1 (from 1964-01-01)
1.00674 (from 1985-10-01)
1.01147 (from 1995-01-01)
1.00914 (from 2000-01-01)
1.00813 (from 2003-06-01)
1.0029 (from 2008-08-01)
1.0019 (from 2009-11-01)
Pressure correction factors: -0.0074 (from 1964-01-01)
Resolution: 1 months
Note: query times are adjusted to 1979-01-01 00:00 - 2012-01-01 00:00
1979.01.01 00:00:00 1979.0000000 5612 6179 1013.86
1979.02.01 00:00:00 1979.0849315 5810 6152 1009.25
1979.03.01 00:00:00 1979.1616438 5854 6077 1005.37
cray.txt
0
Comment
Question by:libertyforall2
3 Comments
 
LVL 31

Expert Comment

by:farzanj
ID: 37732457
What needs to be done to the non numeric rows?
Could you please give a sample output?
0
 
LVL 10

Accepted Solution

by:
Pierre François earned 2000 total points
ID: 37733420
This shell script gives you the result you want:

#!/bin/bash

sed '1,/^Note:/d' cray.txt \
| grep -v seconds \
| cut -d ' ' -f 1,5 \
| awk -F ' ' '{
	if (++j > 12) {
		j = 1
	};
	if (++i < 12) {
		last12 [j] = $2; 
		tot += $2; 
		printf ("%s %d\n", $1, $2);
	} else {
		tot -= last12 [j]; 
		last12 [j] = $2;
		tot += $2;
		printf ("%s %d %d\n", $1, $2, tot/12);
	}
}'

Open in new window

0
 

Author Closing Comment

by:libertyforall2
ID: 37780019
Works.
0

Featured Post

[Webinar] Cloud and Mobile-First Strategy

Maybe you’ve fully adopted the cloud since the beginning. Or maybe you started with on-prem resources but are pursuing a “cloud and mobile first” strategy. Getting to that end state has its challenges. Discover how to build out a 100% cloud and mobile IT strategy in this webinar.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

We live in a world of interfaces like the one in the title picture. VBA also allows to use interfaces which offers a lot of possibilities. This article describes how to use interfaces in VBA and how to work around their bugs.
If you are a mobile app developer and especially develop hybrid mobile apps then these 4 mistakes you must avoid for hybrid app development to be the more genuine app developer.
Simple Linear Regression
Six Sigma Control Plans

886 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question