?
Solved

Problem with StreamTokenizer

Posted on 2001-07-31
9
Medium Priority
?
620 Views
Last Modified: 2008-03-04
HI All,
I am reading a .csv file(comma separated) using stream tokenizer. My file is as the following:

1000,2000,"Inavesevich wins",5000
1001,2001,"Rafter loses",5001

I am reading it line by line and separating each line into tokens separated by comma.
If the file is in the above format I am not having any problem in reading.

If the file is in the following format:
1000,2000,Inavesevich wins,5000
1001,2001,Rafter loses,5001

i.e if the double quotes are removed from the String having spaces
I am having the problem. My program is taking "Ivanesevich" as one token and "wins" as another token whereas i want "Ivanesevich Wins" as a single token.

I am using the StreamTokenizer. My code goes as follows:

st.eolIsSignificant(true);
st.whitespaceChars(',',',');
st.quoteChar(',');

where st is the streamtokenizer.

How to avoid this problem?

If would be helpful if you can post some sample code.

TIA,
Benaam



0
Comment
Question by:benaam
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
9 Comments
 
LVL 92

Accepted Solution

by:
objects earned 400 total points
ID: 6336871
You could use StringTokenizer to do what you require.
Something like:

String line = null;
while (null!=(in.readLine()))
{
   StringTokenizer tokens = new StringTokenizer(line, ",\n\r");
   String a = tokens.nextToken();
   String b = tokens.nextToken();
   String c = tokens.nextToken();
   String d = tokens.nextToken();
   ...
}

0
 
LVL 4

Expert Comment

by:Neutron
ID: 6336890
You could try setting that space (' ') is an "ordinary character"

st.ordinaryChar( ' ' )

Greetings,
    Ntr:)
0
 
LVL 7

Expert Comment

by:tomboshell
ID: 6336894
The Stream Tokenizer is designed to seperate into tokens using a white space as the seperator.  You can add further tokens to it so that it will recognize these also as white-spaces, but it will still be white space driven.  To overcome that problem, go with objects suggestion and implement a stringTokenizer.  I agree that it would be your best bet.  It is a simpler tokenizer, located in the utils package.  As parameters pass the string to be parsed, and the deliminator.  There is also a boolean to set if you want it to return the token with the deliminator, seperated from the rest.  

Best of luck,
Tom
0
Get 15 Days FREE Full-Featured Trial

Benefit from a mission critical IT monitoring with Monitis Premium or get it FREE for your entry level monitoring needs.
-Over 200,000 users
-More than 300,000 websites monitored
-Used in 197 countries
-Recommended by 98% of users

 
LVL 4

Expert Comment

by:Neutron
ID: 6336895
Anyway, it is better if your string is within quotes, because in that case string can contain comma literals.

Greetings,
    Ntr:)
0
 
LVL 3

Expert Comment

by:rjackman
ID: 6336906
hi
as object said it is better to use the StringTokenizer
try following

public void readAndParse()
{
  String file = "path of ur file";
  try
 {
  BufferedReader reader = new BufferedReader(new FileReader(new File(file)));
 Vector data = new Vector();
 String line = null;
 while((line=reader.readLine())!= null)
 {
    if(!line.trim().length<=0)
   {
     StringTokenizer tokenizer = new StringTokenizer  (line,",");
     while(tokenizer.hasMoreTokens())
    {
      data.add(tokenizer.nextToken());
    }

   }
 }
 }
 catch(Exception ioe)
 {
  ioe.printStackTrace();
 }
}

cheers
RJ
0
 
LVL 3

Expert Comment

by:rjackman
ID: 6336911
oops
chang following
if(!line.trim().length<=0)
to
if(line.trim().length()!=0)
cheers
RJ

0
 

Author Comment

by:benaam
ID: 6339621
Thanx Objects..This way it is working now..
0
 
LVL 1

Expert Comment

by:Moondancer
ID: 7024706
You have 16 open questions today which need your attention.  ADMINISTRATION WILL BE CONTACTING YOU SHORTLY.  Moderators Computer101, Netminder, Mindphaser or I will return to finalize these as soon as possible if after 7 days no response by the Asker.  EXPERTS-->  please post closing recommendations before that time.

Below are your open questions as of today.  Questions which have been inactive for 21 days or longer are considered to be abandoned and for those, your options are:
1. Accept a Comment As Answer (use the button next to the Expert's name).
2. Close the question if the information was not useful to you, but may help others. You must tell the participants why you wish to do this, and allow for Expert response.  This choice will include a refund to you, and will move this question to our PAQ (Previously Asked Question) database.  If you found information outside this question thread, please add it.
3. Ask Community Support to help split points between participating experts, or just comment here with details and we'll respond with the process.
4. Delete the question (if it has no potential value for others).
   --> Post comments for expert of your intention to delete and why
   --> YOU CANNOT DELETE A QUESTION with comments; special handling by a Moderator is required.

For special handling needs, please post a zero point question in the link below and include the URL (question QID/link) that it regards with details.
http://www.experts-exchange.com/jsp/qList.jsp?ta=commspt
 
Please click this link for Help Desk, Guidelines/Member Agreement and the Question/Answer process.  http://www.experts-exchange.com/jsp/cmtyHelpDesk.jsp

Click you Member Profile to view your question history and please keep them updated. If you are a KnowledgePro user, use the Power Search option to find them.  

Questions which are LOCKED with a Proposed Answer but do not help you, should be rejected with comments added.  When you grade the question less than an A, please comment as to why.  This helps all involved, as well as others who may access this item in the future.  PLEASE DO NOT AWARD POINTS TO ME.

To view your open questions, please click the following link(s) and keep them all current with updates.
http://www.experts-exchange.com/questions/Q.20034268.html
http://www.experts-exchange.com/questions/Q.20063886.html
http://www.experts-exchange.com/questions/Q.20071336.html
http://www.experts-exchange.com/questions/Q.20132970.html
http://www.experts-exchange.com/questions/Q.20152566.html
http://www.experts-exchange.com/questions/Q.20160660.html
http://www.experts-exchange.com/questions/Q.20161367.html
http://www.experts-exchange.com/questions/Q.20161370.html
http://www.experts-exchange.com/questions/Q.20166906.html
http://www.experts-exchange.com/questions/Q.20166907.html
http://www.experts-exchange.com/questions/Q.20181828.html
http://www.experts-exchange.com/questions/Q.20182284.html
http://www.experts-exchange.com/questions/Q.20185356.html

To view your locked questions, please click the following link(s) and evaluate the proposed answer.
http://www.experts-exchange.com/questions/Q.20036734.html
http://www.experts-exchange.com/questions/Q.20161368.html
http://www.experts-exchange.com/questions/Q.20161395.html

*****  E X P E R T S    P L E A S E  ******  Leave your closing recommendations.
If you are interested in the cleanup effort, please click this link
http://www.experts-exchange.com/jsp/qManageQuestion.jsp?ta=commspt&qid=20274643 
------> POINTS FOR EXPERTS awaiting comments are listed in the links below
http://www.experts-exchange.com/commspt/Q.20277028.html (Part 1)
http://www.experts-exchange.com/jsp/qShow.jsp?ta=commspt&qid=20295853 (Part 2)
 
Moderators will finalize this question if in @7 days Asker has not responded.  This will be moved to the PAQ (Previously Asked Questions) at zero points, deleted or awarded.
 
Thanks everyone.
Moondancer
Moderator @ Experts Exchange
0
 
LVL 5

Expert Comment

by:Netminder
ID: 7329724
Question abandoned; force-accepted.

Netminder
CS Moderator
0

Featured Post

Get 15 Days FREE Full-Featured Trial

Benefit from a mission critical IT monitoring with Monitis Premium or get it FREE for your entry level monitoring needs.
-Over 200,000 users
-More than 300,000 websites monitored
-Used in 197 countries
-Recommended by 98% of users

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this post we will learn different types of Android Layout and some basics of an Android App.
In this post we will learn how to connect and configure Android Device (Smartphone etc.) with Android Studio. After that we will run a simple Hello World Program.
This tutorial will introduce the viewer to VisualVM for the Java platform application. This video explains an example program and covers the Overview, Monitor, and Heap Dump tabs.
This tutorial explains how to use the VisualVM tool for the Java platform application. This video goes into detail on the Threads, Sampler, and Profiler tabs.
Suggested Courses
Course of the Month10 days, 8 hours left to enroll

765 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question