troubleshooting Question

what programing language is best for lots of read/writes of small files? do it count?

Avatar of Ciprian Niculescu
Ciprian NiculescuFlag for Germany asked on
PythonPerlGolangUnix OS
4 Comments2 Solutions233 ViewsLast Modified:
Hello,
i need to process lots of small text files (1k to ~1M in size) millions per day.
they are syslogs files. the processing logic can be summarized in: filter out unwanted lines.
i did some tests with: cat File | grep -v bla | grep -v foo | grep -v bar > File.2
and i get huge space and clearness benefits. But the piping have some limits, after 20 not good any more :)
i'm not a day to day programmer, but kept playing around over the years.
i know/remember C,C++,Perl, some basic bash, started to learn Python some years back. i see that the fancy thing now is Go :)
do any of the language would offer a performance plus or minus, in this scenario? or the limiting factor for all will be the disk access?
Thank you for your thoughts

Ciprian
ASKER CERTIFIED SOLUTION
Join our community to see this answer!
Unlock 2 Answers and 4 Comments.
Start Free Trial
Learn from the best

Network and collaborate with thousands of CTOs, CISOs, and IT Pros rooting for you and your success.

Andrew Hancock - VMware vExpert
See if this solution works for you by signing up for a 7 day free trial.
Unlock 2 Answers and 4 Comments.
Try for 7 days

”The time we save is the biggest benefit of E-E to our team. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange.

-Mike Kapnisakis, Warner Bros