?
Solved

using perl or shell delete column 2, 3, & 5 in text file and make binned rolling averages per month

Posted on 2012-03-16
3
Medium Priority
?
377 Views
Last Modified: 2012-06-22
Alright. See complete file attached. The files starts like this below. The first column is the date and the second column is the time which can be deleted.  Columns 3 & 6 are not necessary and can be deleted. The 4th  column is uncorrected data which can also be deleted. Aside from deleting columns 2-4 & 6, I want to create a binned rolling average which will be added in a third column.So the third column won't start until month 12 since 12 data points are needed so rows 1-11 will only have two columns of data and the 3rd column of data begins from row 12.




Oulu Neutron Monitor database query - http://cosmicrays.oulu.fi/
Start: 1979/01/01 00:00 End: 2012/01/31 23:00
Units for date column: fractional years
Columns: #1, #2 -date and time, #3 - fractional date, #4, #5 - uncorrected and corrected (for pressure and efficiency ) count rates [counts/min],
#6 - barometric pressure [mbar]
Effectiveness correction factors:
1 (from 1964-01-01)
1.00674 (from 1985-10-01)
1.01147 (from 1995-01-01)
1.00914 (from 2000-01-01)
1.00813 (from 2003-06-01)
1.0029 (from 2008-08-01)
1.0019 (from 2009-11-01)
Pressure correction factors: -0.0074 (from 1964-01-01)
Resolution: 1 months
Note: query times are adjusted to 1979-01-01 00:00 - 2012-01-01 00:00
1979.01.01 00:00:00 1979.0000000 5612 6179 1013.86
1979.02.01 00:00:00 1979.0849315 5810 6152 1009.25
1979.03.01 00:00:00 1979.1616438 5854 6077 1005.37
cray.txt
0
Comment
Question by:libertyforall2
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
3 Comments
 
LVL 31

Expert Comment

by:farzanj
ID: 37732457
What needs to be done to the non numeric rows?
Could you please give a sample output?
0
 
LVL 10

Accepted Solution

by:
pfrancois earned 2000 total points
ID: 37733420
This shell script gives you the result you want:

#!/bin/bash

sed '1,/^Note:/d' cray.txt \
| grep -v seconds \
| cut -d ' ' -f 1,5 \
| awk -F ' ' '{
	if (++j > 12) {
		j = 1
	};
	if (++i < 12) {
		last12 [j] = $2; 
		tot += $2; 
		printf ("%s %d\n", $1, $2);
	} else {
		tot -= last12 [j]; 
		last12 [j] = $2;
		tot += $2;
		printf ("%s %d %d\n", $1, $2, tot/12);
	}
}'

Open in new window

0
 

Author Closing Comment

by:libertyforall2
ID: 37780019
Works.
0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this post we will learn how to connect and configure Android Device (Smartphone etc.) with Android Studio. After that we will run a simple Hello World Program.
Q&A with Course Creator, Mark Lassoff, on the importance of HTML5 in the career of a modern-day developer.
In this fourth video of the Xpdf series, we discuss and demonstrate the PDFinfo utility, which retrieves the contents of a PDF's Info Dictionary, as well as some other information, including the page count. We show how to isolate the page count in a…
With the power of JIRA, there's an unlimited number of ways you can customize it, use it and benefit from it. With that in mind, there's bound to be things that I wasn't able to cover in this course. With this summary we'll look at some places to go…
Suggested Courses

777 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question