pvinodp
asked on
search a pattern in a large file using java
Following is my code in a particular function.
Scanner sc = new Scanner(new File("/exports/nos_issues/
String str="10:37:10.719 [net.jradius.freeradius.Fr
Pattern ptr = Pattern.compile(str);
long toto=0;
try{
while(sc.findWithinHorizon
toto++;
}finally{
sc.close();
}
System.out.println(toto);
I get the following error :
Exception in thread "main" java.lang.OutOfMemoryError
at java.nio.HeapCharBuffer.<i
at java.nio.CharBuffer.alloca
at java.util.Scanner.makeSpac
at java.util.Scanner.readInpu
at java.util.Scanner.findWith
at Cl.main(Cl.java:38)
in the class Cl the line number 38 points to sc.close();
Scanner sc = new Scanner(new File("/exports/nos_issues/
String str="10:37:10.719 [net.jradius.freeradius.Fr
Pattern ptr = Pattern.compile(str);
long toto=0;
try{
while(sc.findWithinHorizon
toto++;
}finally{
sc.close();
}
System.out.println(toto);
I get the following error :
Exception in thread "main" java.lang.OutOfMemoryError
at java.nio.HeapCharBuffer.<i
at java.nio.CharBuffer.alloca
at java.util.Scanner.makeSpac
at java.util.Scanner.readInpu
at java.util.Scanner.findWith
at Cl.main(Cl.java:38)
in the class Cl the line number 38 points to sc.close();
Last Comment
CEHJ
This is confusing - you're looking for what is clearly a timestamped logfile entry. So first of all, why would there be any more than one occurrence?
ASKER
the string pattern is just another example... it can be any string.
OK. The file is a text file containing lines?
ASKER
yes.
What i am trying to do is . divide the file into segments and then search each segment.
I aware of the risk of cutting my pattern while dividing it into segemtns.
What i am trying to do is . divide the file into segments and then search each segment.
I aware of the risk of cutting my pattern while dividing it into segemtns.
OK. The file is a text file containing lines?
yes.In that case, the file makes sense in terms of lines, so i'm wondering why you're trying to span multiple lines ..?
ASKER
I want to create a job to search each segment. and in future allocate job to be executed on separate system.
Segments don't actually have anything to do with a pattern spanning multiple lines. You can think of a segment of a text file as simply a file with fewer lines. The problem is the same (even if the problem space is smaller)
ASKER
but i intend not to do line by line search..
I want to divide the whole file into many portions and then search
I want to divide the whole file into many portions and then search
I want to divide the whole file into many portions and then searchHow?
Take the following file. Divide it equally into two and have each worker thread search each segment for the phrase 'THE CAT SAT ON THE MAT'. Neither will find it
jSx62KM9qB20iMmM1WdlKsM0tKuRbCwEmsyLcZPJ4dSvOHmDovpdoqxe21RpbafryY1
PBga3epRxM3v25usrWwirvVHiNZ28PnguDEFuZo4bKavq2R0T64vi4hnPIUUXMKagtnyKSNUQNyx
URwwKDGcTjfKtcTzw7uHaWSmmZeen51ZuOzmzNe76LnNumzLCkfyIOm9GYA0VsbGD47zkzoku033HSzCtrHHrs0XDBaOHQWL7hXKLfLLLqlpIGYH0kDctda3 lH2XAYfg0J
b THE CAT SAT ON THE MAT
EDGdSmXv18OTJEyeqPFXBCW7ATHsl66SGaFNNYgC5UvtSDPPr4KwDNRYVQDzWsCkzuPKuQ
5y7URHNjEO8eZ1siQUraAvpdZF1WM
ScBL4zBKQwZsrXD
IU
ie7xzxW14C6hV9olbQLHvuO7ZOU3Iva3fa0JqW9UCuK1fpRMeRBCUrQEffbnXDwMP
hzrrG8TJ
ASKER
I think that is because the string is divided across the two parts.
ASKER
I dint understand your question..
HOW?
Yes, so
I think that is because the string is divided across the two parts.
a. how are you going to divide it?
b. are you going to be looking for patterns such as "MAT.*EDGdSmXv18OT"? And if so - why?
ASKER
in my case the pattern is going to appear many times.. and the I just need the number of occurences..
And as it is a log file from a hardware , the number can be huge .
I might get a file of size 1 gb . and if i make a 5 segments of them , i am at a risk of loosing 10 occurences.
But in my case the count can go to 100+ in a single segment.
And as it is a log file from a hardware , the number can be huge .
I might get a file of size 1 gb . and if i make a 5 segments of them , i am at a risk of loosing 10 occurences.
But in my case the count can go to 100+ in a single segment.
You still haven't told me if you're looking for patterns that span lines, and if so, why
ASKER
my pattern is a single word and it cannot span lines.
ASKER CERTIFIED SOLUTION
THIS SOLUTION IS ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
ASKER
i think line by line gives some improvement over using scan.
ASKER
Thanks for your input
:)
Java
Java is a platform-independent, object-oriented programming language and run-time environment, designed to have as few implementation dependencies as possible such that developers can write one set of code across all platforms using libraries. Most devices will not run Java natively, and require a run-time component to be installed in order to execute a Java program.
102K
Questions
--
Followers
--
Top Experts
Get a personalized solution from industry experts
TRUSTED BY
