Hello,
i need to process lots of small text files (1k to ~1M in size) millions per day.
they are syslogs files. the processing logic can be summarized in: filter out unwanted lines.
i did some tests with: cat File | grep -v bla | grep -v foo | grep -v bar > File.2
and i get huge space and clearness benefits. But the piping have some limits, after 20 not good any more :)
i'm not a day to day programmer, but kept playing around over the years.
i know/remember C,C++,Perl, some basic bash, started to learn Python some years back. i see that the fancy thing now is Go :)
do any of the language would offer a performance plus or minus, in this scenario? or the limiting factor for all will be the disk access?
Thank you for your thoughts
Ciprian
Our community of experts have been thoroughly vetted for their expertise and industry experience.
The Distinguished Expert awards are presented to the top veteran and rookie experts to earn the most points in the top 50 topics.