Solved

Problem with StreamTokenizer

Posted on 2001-07-31
9
618 Views
Last Modified: 2008-03-04
HI All,
I am reading a .csv file(comma separated) using stream tokenizer. My file is as the following:

1000,2000,"Inavesevich wins",5000
1001,2001,"Rafter loses",5001

I am reading it line by line and separating each line into tokens separated by comma.
If the file is in the above format I am not having any problem in reading.

If the file is in the following format:
1000,2000,Inavesevich wins,5000
1001,2001,Rafter loses,5001

i.e if the double quotes are removed from the String having spaces
I am having the problem. My program is taking "Ivanesevich" as one token and "wins" as another token whereas i want "Ivanesevich Wins" as a single token.

I am using the StreamTokenizer. My code goes as follows:

st.eolIsSignificant(true);
st.whitespaceChars(',',',');
st.quoteChar(',');

where st is the streamtokenizer.

How to avoid this problem?

If would be helpful if you can post some sample code.

TIA,
Benaam



0
Comment
Question by:benaam
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
9 Comments
 
LVL 92

Accepted Solution

by:
objects earned 100 total points
ID: 6336871
You could use StringTokenizer to do what you require.
Something like:

String line = null;
while (null!=(in.readLine()))
{
   StringTokenizer tokens = new StringTokenizer(line, ",\n\r");
   String a = tokens.nextToken();
   String b = tokens.nextToken();
   String c = tokens.nextToken();
   String d = tokens.nextToken();
   ...
}

0
 
LVL 4

Expert Comment

by:Neutron
ID: 6336890
You could try setting that space (' ') is an "ordinary character"

st.ordinaryChar( ' ' )

Greetings,
    Ntr:)
0
 
LVL 7

Expert Comment

by:tomboshell
ID: 6336894
The Stream Tokenizer is designed to seperate into tokens using a white space as the seperator.  You can add further tokens to it so that it will recognize these also as white-spaces, but it will still be white space driven.  To overcome that problem, go with objects suggestion and implement a stringTokenizer.  I agree that it would be your best bet.  It is a simpler tokenizer, located in the utils package.  As parameters pass the string to be parsed, and the deliminator.  There is also a boolean to set if you want it to return the token with the deliminator, seperated from the rest.  

Best of luck,
Tom
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 4

Expert Comment

by:Neutron
ID: 6336895
Anyway, it is better if your string is within quotes, because in that case string can contain comma literals.

Greetings,
    Ntr:)
0
 
LVL 3

Expert Comment

by:rjackman
ID: 6336906
hi
as object said it is better to use the StringTokenizer
try following

public void readAndParse()
{
  String file = "path of ur file";
  try
 {
  BufferedReader reader = new BufferedReader(new FileReader(new File(file)));
 Vector data = new Vector();
 String line = null;
 while((line=reader.readLine())!= null)
 {
    if(!line.trim().length<=0)
   {
     StringTokenizer tokenizer = new StringTokenizer  (line,",");
     while(tokenizer.hasMoreTokens())
    {
      data.add(tokenizer.nextToken());
    }

   }
 }
 }
 catch(Exception ioe)
 {
  ioe.printStackTrace();
 }
}

cheers
RJ
0
 
LVL 3

Expert Comment

by:rjackman
ID: 6336911
oops
chang following
if(!line.trim().length<=0)
to
if(line.trim().length()!=0)
cheers
RJ

0
 

Author Comment

by:benaam
ID: 6339621
Thanx Objects..This way it is working now..
0
 
LVL 1

Expert Comment

by:Moondancer
ID: 7024706
You have 16 open questions today which need your attention.  ADMINISTRATION WILL BE CONTACTING YOU SHORTLY.  Moderators Computer101, Netminder, Mindphaser or I will return to finalize these as soon as possible if after 7 days no response by the Asker.  EXPERTS-->  please post closing recommendations before that time.

Below are your open questions as of today.  Questions which have been inactive for 21 days or longer are considered to be abandoned and for those, your options are:
1. Accept a Comment As Answer (use the button next to the Expert's name).
2. Close the question if the information was not useful to you, but may help others. You must tell the participants why you wish to do this, and allow for Expert response.  This choice will include a refund to you, and will move this question to our PAQ (Previously Asked Question) database.  If you found information outside this question thread, please add it.
3. Ask Community Support to help split points between participating experts, or just comment here with details and we'll respond with the process.
4. Delete the question (if it has no potential value for others).
   --> Post comments for expert of your intention to delete and why
   --> YOU CANNOT DELETE A QUESTION with comments; special handling by a Moderator is required.

For special handling needs, please post a zero point question in the link below and include the URL (question QID/link) that it regards with details.
http://www.experts-exchange.com/jsp/qList.jsp?ta=commspt
 
Please click this link for Help Desk, Guidelines/Member Agreement and the Question/Answer process.  http://www.experts-exchange.com/jsp/cmtyHelpDesk.jsp

Click you Member Profile to view your question history and please keep them updated. If you are a KnowledgePro user, use the Power Search option to find them.  

Questions which are LOCKED with a Proposed Answer but do not help you, should be rejected with comments added.  When you grade the question less than an A, please comment as to why.  This helps all involved, as well as others who may access this item in the future.  PLEASE DO NOT AWARD POINTS TO ME.

To view your open questions, please click the following link(s) and keep them all current with updates.
http://www.experts-exchange.com/questions/Q.20034268.html
http://www.experts-exchange.com/questions/Q.20063886.html
http://www.experts-exchange.com/questions/Q.20071336.html
http://www.experts-exchange.com/questions/Q.20132970.html
http://www.experts-exchange.com/questions/Q.20152566.html
http://www.experts-exchange.com/questions/Q.20160660.html
http://www.experts-exchange.com/questions/Q.20161367.html
http://www.experts-exchange.com/questions/Q.20161370.html
http://www.experts-exchange.com/questions/Q.20166906.html
http://www.experts-exchange.com/questions/Q.20166907.html
http://www.experts-exchange.com/questions/Q.20181828.html
http://www.experts-exchange.com/questions/Q.20182284.html
http://www.experts-exchange.com/questions/Q.20185356.html

To view your locked questions, please click the following link(s) and evaluate the proposed answer.
http://www.experts-exchange.com/questions/Q.20036734.html
http://www.experts-exchange.com/questions/Q.20161368.html
http://www.experts-exchange.com/questions/Q.20161395.html

*****  E X P E R T S    P L E A S E  ******  Leave your closing recommendations.
If you are interested in the cleanup effort, please click this link
http://www.experts-exchange.com/jsp/qManageQuestion.jsp?ta=commspt&qid=20274643 
------> POINTS FOR EXPERTS awaiting comments are listed in the links below
http://www.experts-exchange.com/commspt/Q.20277028.html (Part 1)
http://www.experts-exchange.com/jsp/qShow.jsp?ta=commspt&qid=20295853 (Part 2)
 
Moderators will finalize this question if in @7 days Asker has not responded.  This will be moved to the PAQ (Previously Asked Questions) at zero points, deleted or awarded.
 
Thanks everyone.
Moondancer
Moderator @ Experts Exchange
0
 
LVL 5

Expert Comment

by:Netminder
ID: 7329724
Question abandoned; force-accepted.

Netminder
CS Moderator
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

INTRODUCTION Working with files is a moderately common task in Java.  For most projects hard coding the file names, using parameters in configuration files, or using command-line arguments is sufficient.   However, when your application has vi…
In this post we will learn how to make Android Gesture Tutorial and give different functionality whenever a user Touch or Scroll android screen.
Viewers will learn about arithmetic and Boolean expressions in Java and the logical operators used to create Boolean expressions. We will cover the symbols used for arithmetic expressions and define each logical operator and how to use them in Boole…
Viewers will learn how to properly install Eclipse with the necessary JDK, and will take a look at an introductory Java program. Download Eclipse installation zip file: Extract files from zip file: Download and install JDK 8: Open Eclipse and …

719 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question