what programing language is best for lots of read/writes of small files? do it count?
Hello,
i need to process lots of small text files (1k to ~1M in size) millions per day.
they are syslogs files. the processing logic can be summarized in: filter out unwanted lines.
i did some tests with: cat File | grep -v bla | grep -v foo | grep -v bar > File.2
and i get huge space and clearness benefits. But the piping have some limits, after 20 not good any more :)
i'm not a day to day programmer, but kept playing around over the years.
i know/remember C,C++,Perl, some basic bash, started to learn Python some years back. i see that the fancy thing now is Go :)
do any of the language would offer a performance plus or minus, in this scenario? or the limiting factor for all will be the disk access?
Thank you for your thoughts
Ciprian
i need to process lots of small text files (1k to ~1M in size) millions per day.
they are syslogs files. the processing logic can be summarized in: filter out unwanted lines.
i did some tests with: cat File | grep -v bla | grep -v foo | grep -v bar > File.2
and i get huge space and clearness benefits. But the piping have some limits, after 20 not good any more :)
i'm not a day to day programmer, but kept playing around over the years.
i know/remember C,C++,Perl, some basic bash, started to learn Python some years back. i see that the fancy thing now is Go :)
do any of the language would offer a performance plus or minus, in this scenario? or the limiting factor for all will be the disk access?
Thank you for your thoughts
Ciprian
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
wilcoxon
I agree with arnold. If you want to do text processing, use perl.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
No comment has been added to this question in more than 21 days, so it is now classified as abandoned.
I have recommended this question be closed as follows:
Split:
-- arnold (https:#a42374520)
-- David Favor (https:#a42374692)
If you feel this question should be closed differently, post an objection and the moderators will review all objections and close it as they feel fit. If no one objects, this question will be closed automatically the way described above.
suhasbharadwaj
Experts-Exchange Cleanup Volunteer
I have recommended this question be closed as follows:
Split:
-- arnold (https:#a42374520)
-- David Favor (https:#a42374692)
If you feel this question should be closed differently, post an objection and the moderators will review all objections and close it as they feel fit. If no one objects, this question will be closed automatically the way described above.
suhasbharadwaj
Experts-Exchange Cleanup Volunteer