Solved

Problem with StreamTokenizer

Posted on 2001-07-31
9
616 Views
Last Modified: 2008-03-04
HI All,
I am reading a .csv file(comma separated) using stream tokenizer. My file is as the following:

1000,2000,"Inavesevich wins",5000
1001,2001,"Rafter loses",5001

I am reading it line by line and separating each line into tokens separated by comma.
If the file is in the above format I am not having any problem in reading.

If the file is in the following format:
1000,2000,Inavesevich wins,5000
1001,2001,Rafter loses,5001

i.e if the double quotes are removed from the String having spaces
I am having the problem. My program is taking "Ivanesevich" as one token and "wins" as another token whereas i want "Ivanesevich Wins" as a single token.

I am using the StreamTokenizer. My code goes as follows:

st.eolIsSignificant(true);
st.whitespaceChars(',',',');
st.quoteChar(',');

where st is the streamtokenizer.

How to avoid this problem?

If would be helpful if you can post some sample code.

TIA,
Benaam



0
Comment
Question by:benaam
9 Comments
 
LVL 92

Accepted Solution

by:
objects earned 100 total points
ID: 6336871
You could use StringTokenizer to do what you require.
Something like:

String line = null;
while (null!=(in.readLine()))
{
   StringTokenizer tokens = new StringTokenizer(line, ",\n\r");
   String a = tokens.nextToken();
   String b = tokens.nextToken();
   String c = tokens.nextToken();
   String d = tokens.nextToken();
   ...
}

0
 
LVL 4

Expert Comment

by:Neutron
ID: 6336890
You could try setting that space (' ') is an "ordinary character"

st.ordinaryChar( ' ' )

Greetings,
    Ntr:)
0
 
LVL 7

Expert Comment

by:tomboshell
ID: 6336894
The Stream Tokenizer is designed to seperate into tokens using a white space as the seperator.  You can add further tokens to it so that it will recognize these also as white-spaces, but it will still be white space driven.  To overcome that problem, go with objects suggestion and implement a stringTokenizer.  I agree that it would be your best bet.  It is a simpler tokenizer, located in the utils package.  As parameters pass the string to be parsed, and the deliminator.  There is also a boolean to set if you want it to return the token with the deliminator, seperated from the rest.  

Best of luck,
Tom
0
Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

 
LVL 4

Expert Comment

by:Neutron
ID: 6336895
Anyway, it is better if your string is within quotes, because in that case string can contain comma literals.

Greetings,
    Ntr:)
0
 
LVL 3

Expert Comment

by:rjackman
ID: 6336906
hi
as object said it is better to use the StringTokenizer
try following

public void readAndParse()
{
  String file = "path of ur file";
  try
 {
  BufferedReader reader = new BufferedReader(new FileReader(new File(file)));
 Vector data = new Vector();
 String line = null;
 while((line=reader.readLine())!= null)
 {
    if(!line.trim().length<=0)
   {
     StringTokenizer tokenizer = new StringTokenizer  (line,",");
     while(tokenizer.hasMoreTokens())
    {
      data.add(tokenizer.nextToken());
    }

   }
 }
 }
 catch(Exception ioe)
 {
  ioe.printStackTrace();
 }
}

cheers
RJ
0
 
LVL 3

Expert Comment

by:rjackman
ID: 6336911
oops
chang following
if(!line.trim().length<=0)
to
if(line.trim().length()!=0)
cheers
RJ

0
 

Author Comment

by:benaam
ID: 6339621
Thanx Objects..This way it is working now..
0
 
LVL 1

Expert Comment

by:Moondancer
ID: 7024706
You have 16 open questions today which need your attention.  ADMINISTRATION WILL BE CONTACTING YOU SHORTLY.  Moderators Computer101, Netminder, Mindphaser or I will return to finalize these as soon as possible if after 7 days no response by the Asker.  EXPERTS-->  please post closing recommendations before that time.

Below are your open questions as of today.  Questions which have been inactive for 21 days or longer are considered to be abandoned and for those, your options are:
1. Accept a Comment As Answer (use the button next to the Expert's name).
2. Close the question if the information was not useful to you, but may help others. You must tell the participants why you wish to do this, and allow for Expert response.  This choice will include a refund to you, and will move this question to our PAQ (Previously Asked Question) database.  If you found information outside this question thread, please add it.
3. Ask Community Support to help split points between participating experts, or just comment here with details and we'll respond with the process.
4. Delete the question (if it has no potential value for others).
   --> Post comments for expert of your intention to delete and why
   --> YOU CANNOT DELETE A QUESTION with comments; special handling by a Moderator is required.

For special handling needs, please post a zero point question in the link below and include the URL (question QID/link) that it regards with details.
http://www.experts-exchange.com/jsp/qList.jsp?ta=commspt
 
Please click this link for Help Desk, Guidelines/Member Agreement and the Question/Answer process.  http://www.experts-exchange.com/jsp/cmtyHelpDesk.jsp

Click you Member Profile to view your question history and please keep them updated. If you are a KnowledgePro user, use the Power Search option to find them.  

Questions which are LOCKED with a Proposed Answer but do not help you, should be rejected with comments added.  When you grade the question less than an A, please comment as to why.  This helps all involved, as well as others who may access this item in the future.  PLEASE DO NOT AWARD POINTS TO ME.

To view your open questions, please click the following link(s) and keep them all current with updates.
http://www.experts-exchange.com/questions/Q.20034268.html
http://www.experts-exchange.com/questions/Q.20063886.html
http://www.experts-exchange.com/questions/Q.20071336.html
http://www.experts-exchange.com/questions/Q.20132970.html
http://www.experts-exchange.com/questions/Q.20152566.html
http://www.experts-exchange.com/questions/Q.20160660.html
http://www.experts-exchange.com/questions/Q.20161367.html
http://www.experts-exchange.com/questions/Q.20161370.html
http://www.experts-exchange.com/questions/Q.20166906.html
http://www.experts-exchange.com/questions/Q.20166907.html
http://www.experts-exchange.com/questions/Q.20181828.html
http://www.experts-exchange.com/questions/Q.20182284.html
http://www.experts-exchange.com/questions/Q.20185356.html

To view your locked questions, please click the following link(s) and evaluate the proposed answer.
http://www.experts-exchange.com/questions/Q.20036734.html
http://www.experts-exchange.com/questions/Q.20161368.html
http://www.experts-exchange.com/questions/Q.20161395.html

*****  E X P E R T S    P L E A S E  ******  Leave your closing recommendations.
If you are interested in the cleanup effort, please click this link
http://www.experts-exchange.com/jsp/qManageQuestion.jsp?ta=commspt&qid=20274643 
------> POINTS FOR EXPERTS awaiting comments are listed in the links below
http://www.experts-exchange.com/commspt/Q.20277028.html (Part 1)
http://www.experts-exchange.com/jsp/qShow.jsp?ta=commspt&qid=20295853 (Part 2)
 
Moderators will finalize this question if in @7 days Asker has not responded.  This will be moved to the PAQ (Previously Asked Questions) at zero points, deleted or awarded.
 
Thanks everyone.
Moondancer
Moderator @ Experts Exchange
0
 
LVL 5

Expert Comment

by:Netminder
ID: 7329724
Question abandoned; force-accepted.

Netminder
CS Moderator
0

Featured Post

Announcing the Most Valuable Experts of 2016

MVEs are more concerned with the satisfaction of those they help than with the considerable points they can earn. They are the types of people you feel privileged to call colleagues. Join us in honoring this amazing group of Experts.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
by zero exception 10 52
Is there a simpler dropbox system? 10 34
swing controls 2 16
jsp CRUD operations with and without prepared statement also hibernatge 1 9
An old method to applying the Singleton pattern in your Java code is to check if a static instance, defined in the same class that needs to be instantiated once and only once, is null and then create a new instance; otherwise, the pre-existing insta…
Java contains several comparison operators (e.g., <, <=, >, >=, ==, !=) that allow you to compare primitive values. However, these operators cannot be used to compare the contents of objects. Interface Comparable is used to allow objects of a cl…
Viewers will learn about the regular for loop in Java and how to use it. Definition: Break the for loop down into 3 parts: Syntax when using for loops: Example using a for loop:
The viewer will learn how to implement Singleton Design Pattern in Java.

856 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question