We have logfiles that are machine data samples. The logs are by hour, and the data they contain are every 10 seconds. Thus, each hourly log file is ~36,000 lines. There is some variance in when the logs start and end as far as time, so there is no set consistency to an exact number of lines or how many lines exist per second or minute. Thus there are about 24 logs in the directory (one for each hour)
I want to sample each log file and pull the first line (field info), the second line (first sample) and every 575th or so line (essentially capturing one sample for every minute in the log file)after that until it runs the end of the file. I want these samples written to a new file, prefixed with "SD-" and then the original filename (SD being Sample Data). So, the resulting hourly file would be ~60 Lines of Sample Data of the 36,000 Line original file.
Please let me know if I can clarify anything. TIA!
This Question has been solved and asker verified All Experts Exchange premium technology solutions are available to subscription members.
Experts Exchange has been collecting answers to technology questions since 1996…3 million and counting! If you have a question, chances are we already have your answer.
If you can't find the exact answer you're looking for, ask our exclusive community of 50,000 experts. You’ll get a personalized answer from a trusted professional.
Thousands of free tech tips, tricks, how-to’s and tutorials are available in our peer reviewed articles section. See for yourself how smart our experts are, no login required.
Save solutions to your questions, answers you’ve discovered through searching plus helpful articles in your personal knowledgebase for easy future access.
Access the answers to your technology questions today.
30-day free trial. Register in 60 seconds.
Members of the expert community talk about why the experience at Experts Exchange is different than what you will find anywhere else.
Try it out and discover for yourself.
30-day free trial. Register in 60 seconds.
Join the community of experts here and help other tech pros by answering question in your area of expertise. You can earn FREE access to all Experts Exchange's premium features and resources.
I don't necessarily have a preference. I would prefer not to install any software on the remote machine that will be doing the processing. I do not mind to include a utility/executable that is needed. I started working on the problem with a sed port to windows, but I'm just not getting where I need to go very quickly.
The easiest to write is in perl.
use Tie::File;
my $file = shift || @ARGV;
die "File not specified" if not $file;
# open an existing file in read-only mode
use Fcntl 'O_RDONLY';
tie my @array, 'Tie::File', $file, mode => O_RDONLY or die $!;
#read first line
my @newArray = ("");
#read first line
$newArray[0] = $array[0];
#read second line
$newArray[1] = $array[1];
#read every 575th
my $startIndex = 574;
my $index = 2;
while ($startIndex <= int(@array)) {
$newArray[$index++] = $array[$startIndex];
$startIndex += 575;
}
untie @array;
#save to file
use File::Basename;
($base, $dir) = fileparse($file);
$file = "$dir" . "SD-" . "$base";
print $file;
open (SAMPLEDATA, ">$file") || die $!;
print SAMPLEDATA join("\n",@newArray);
close SAMPLEDATA;
USAGE: perl thisprog.pl <filename>
You can download a utility called perl2exe what would convert this little script to an exe equivalent :)
Business Accounts
Answer for Membership
by: harris_cPosted on 2007-09-26 at 22:49:18ID: 19969044
What language/script do you want it written?
get first line
get second line
get 575th line
get 1150th line
... and so on.