Solved

using perl or shell delete column 2, 3, & 5 in text file and make binned rolling averages per month

Posted on 2012-03-16
3
344 Views
Last Modified: 2012-06-22
Alright. See complete file attached. The files starts like this below. The first column is the date and the second column is the time which can be deleted.  Columns 3 & 6 are not necessary and can be deleted. The 4th  column is uncorrected data which can also be deleted. Aside from deleting columns 2-4 & 6, I want to create a binned rolling average which will be added in a third column.So the third column won't start until month 12 since 12 data points are needed so rows 1-11 will only have two columns of data and the 3rd column of data begins from row 12.




Oulu Neutron Monitor database query - http://cosmicrays.oulu.fi/
Start: 1979/01/01 00:00 End: 2012/01/31 23:00
Units for date column: fractional years
Columns: #1, #2 -date and time, #3 - fractional date, #4, #5 - uncorrected and corrected (for pressure and efficiency ) count rates [counts/min],
#6 - barometric pressure [mbar]
Effectiveness correction factors:
1 (from 1964-01-01)
1.00674 (from 1985-10-01)
1.01147 (from 1995-01-01)
1.00914 (from 2000-01-01)
1.00813 (from 2003-06-01)
1.0029 (from 2008-08-01)
1.0019 (from 2009-11-01)
Pressure correction factors: -0.0074 (from 1964-01-01)
Resolution: 1 months
Note: query times are adjusted to 1979-01-01 00:00 - 2012-01-01 00:00
1979.01.01 00:00:00 1979.0000000 5612 6179 1013.86
1979.02.01 00:00:00 1979.0849315 5810 6152 1009.25
1979.03.01 00:00:00 1979.1616438 5854 6077 1005.37
cray.txt
0
Comment
Question by:libertyforall2
3 Comments
 
LVL 31

Expert Comment

by:farzanj
ID: 37732457
What needs to be done to the non numeric rows?
Could you please give a sample output?
0
 
LVL 10

Accepted Solution

by:
pfrancois earned 500 total points
ID: 37733420
This shell script gives you the result you want:

#!/bin/bash

sed '1,/^Note:/d' cray.txt \
| grep -v seconds \
| cut -d ' ' -f 1,5 \
| awk -F ' ' '{
	if (++j > 12) {
		j = 1
	};
	if (++i < 12) {
		last12 [j] = $2; 
		tot += $2; 
		printf ("%s %d\n", $1, $2);
	} else {
		tot -= last12 [j]; 
		last12 [j] = $2;
		tot += $2;
		printf ("%s %d %d\n", $1, $2, tot/12);
	}
}'

Open in new window

0
 

Author Closing Comment

by:libertyforall2
ID: 37780019
Works.
0

Featured Post

Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

Join & Write a Comment

Checking the Alert Log in AWS RDS Oracle can be a pain through their user interface.  I made a script to download the Alert Log, look for errors, and email me the trace files.  In this article I'll describe what I did and share my script.
Although it can be difficult to imagine, someday your child will have a career of his or her own. He or she will likely start a family, buy a home and start having their own children. So, while being a kid is still extremely important, it’s also …
Viewers will learn how to properly install Eclipse with the necessary JDK, and will take a look at an introductory Java program. Download Eclipse installation zip file: Extract files from zip file: Download and install JDK 8: Open Eclipse and …
In this fourth video of the Xpdf series, we discuss and demonstrate the PDFinfo utility, which retrieves the contents of a PDF's Info Dictionary, as well as some other information, including the page count. We show how to isolate the page count in a…

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now