Solved

Problem with StreamTokenizer

Posted on 2001-07-31
9
613 Views
Last Modified: 2008-03-04
HI All,
I am reading a .csv file(comma separated) using stream tokenizer. My file is as the following:

1000,2000,"Inavesevich wins",5000
1001,2001,"Rafter loses",5001

I am reading it line by line and separating each line into tokens separated by comma.
If the file is in the above format I am not having any problem in reading.

If the file is in the following format:
1000,2000,Inavesevich wins,5000
1001,2001,Rafter loses,5001

i.e if the double quotes are removed from the String having spaces
I am having the problem. My program is taking "Ivanesevich" as one token and "wins" as another token whereas i want "Ivanesevich Wins" as a single token.

I am using the StreamTokenizer. My code goes as follows:

st.eolIsSignificant(true);
st.whitespaceChars(',',',');
st.quoteChar(',');

where st is the streamtokenizer.

How to avoid this problem?

If would be helpful if you can post some sample code.

TIA,
Benaam



0
Comment
Question by:benaam
9 Comments
 
LVL 92

Accepted Solution

by:
objects earned 100 total points
Comment Utility
You could use StringTokenizer to do what you require.
Something like:

String line = null;
while (null!=(in.readLine()))
{
   StringTokenizer tokens = new StringTokenizer(line, ",\n\r");
   String a = tokens.nextToken();
   String b = tokens.nextToken();
   String c = tokens.nextToken();
   String d = tokens.nextToken();
   ...
}

0
 
LVL 4

Expert Comment

by:Neutron
Comment Utility
You could try setting that space (' ') is an "ordinary character"

st.ordinaryChar( ' ' )

Greetings,
    Ntr:)
0
 
LVL 7

Expert Comment

by:tomboshell
Comment Utility
The Stream Tokenizer is designed to seperate into tokens using a white space as the seperator.  You can add further tokens to it so that it will recognize these also as white-spaces, but it will still be white space driven.  To overcome that problem, go with objects suggestion and implement a stringTokenizer.  I agree that it would be your best bet.  It is a simpler tokenizer, located in the utils package.  As parameters pass the string to be parsed, and the deliminator.  There is also a boolean to set if you want it to return the token with the deliminator, seperated from the rest.  

Best of luck,
Tom
0
 
LVL 4

Expert Comment

by:Neutron
Comment Utility
Anyway, it is better if your string is within quotes, because in that case string can contain comma literals.

Greetings,
    Ntr:)
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 
LVL 3

Expert Comment

by:rjackman
Comment Utility
hi
as object said it is better to use the StringTokenizer
try following

public void readAndParse()
{
  String file = "path of ur file";
  try
 {
  BufferedReader reader = new BufferedReader(new FileReader(new File(file)));
 Vector data = new Vector();
 String line = null;
 while((line=reader.readLine())!= null)
 {
    if(!line.trim().length<=0)
   {
     StringTokenizer tokenizer = new StringTokenizer  (line,",");
     while(tokenizer.hasMoreTokens())
    {
      data.add(tokenizer.nextToken());
    }

   }
 }
 }
 catch(Exception ioe)
 {
  ioe.printStackTrace();
 }
}

cheers
RJ
0
 
LVL 3

Expert Comment

by:rjackman
Comment Utility
oops
chang following
if(!line.trim().length<=0)
to
if(line.trim().length()!=0)
cheers
RJ

0
 

Author Comment

by:benaam
Comment Utility
Thanx Objects..This way it is working now..
0
 
LVL 1

Expert Comment

by:Moondancer
Comment Utility
You have 16 open questions today which need your attention.  ADMINISTRATION WILL BE CONTACTING YOU SHORTLY.  Moderators Computer101, Netminder, Mindphaser or I will return to finalize these as soon as possible if after 7 days no response by the Asker.  EXPERTS-->  please post closing recommendations before that time.

Below are your open questions as of today.  Questions which have been inactive for 21 days or longer are considered to be abandoned and for those, your options are:
1. Accept a Comment As Answer (use the button next to the Expert's name).
2. Close the question if the information was not useful to you, but may help others. You must tell the participants why you wish to do this, and allow for Expert response.  This choice will include a refund to you, and will move this question to our PAQ (Previously Asked Question) database.  If you found information outside this question thread, please add it.
3. Ask Community Support to help split points between participating experts, or just comment here with details and we'll respond with the process.
4. Delete the question (if it has no potential value for others).
   --> Post comments for expert of your intention to delete and why
   --> YOU CANNOT DELETE A QUESTION with comments; special handling by a Moderator is required.

For special handling needs, please post a zero point question in the link below and include the URL (question QID/link) that it regards with details.
http://www.experts-exchange.com/jsp/qList.jsp?ta=commspt
 
Please click this link for Help Desk, Guidelines/Member Agreement and the Question/Answer process.  http://www.experts-exchange.com/jsp/cmtyHelpDesk.jsp

Click you Member Profile to view your question history and please keep them updated. If you are a KnowledgePro user, use the Power Search option to find them.  

Questions which are LOCKED with a Proposed Answer but do not help you, should be rejected with comments added.  When you grade the question less than an A, please comment as to why.  This helps all involved, as well as others who may access this item in the future.  PLEASE DO NOT AWARD POINTS TO ME.

To view your open questions, please click the following link(s) and keep them all current with updates.
http://www.experts-exchange.com/questions/Q.20034268.html
http://www.experts-exchange.com/questions/Q.20063886.html
http://www.experts-exchange.com/questions/Q.20071336.html
http://www.experts-exchange.com/questions/Q.20132970.html
http://www.experts-exchange.com/questions/Q.20152566.html
http://www.experts-exchange.com/questions/Q.20160660.html
http://www.experts-exchange.com/questions/Q.20161367.html
http://www.experts-exchange.com/questions/Q.20161370.html
http://www.experts-exchange.com/questions/Q.20166906.html
http://www.experts-exchange.com/questions/Q.20166907.html
http://www.experts-exchange.com/questions/Q.20181828.html
http://www.experts-exchange.com/questions/Q.20182284.html
http://www.experts-exchange.com/questions/Q.20185356.html

To view your locked questions, please click the following link(s) and evaluate the proposed answer.
http://www.experts-exchange.com/questions/Q.20036734.html
http://www.experts-exchange.com/questions/Q.20161368.html
http://www.experts-exchange.com/questions/Q.20161395.html

*****  E X P E R T S    P L E A S E  ******  Leave your closing recommendations.
If you are interested in the cleanup effort, please click this link
http://www.experts-exchange.com/jsp/qManageQuestion.jsp?ta=commspt&qid=20274643
------> POINTS FOR EXPERTS awaiting comments are listed in the links below
http://www.experts-exchange.com/commspt/Q.20277028.html (Part 1)
http://www.experts-exchange.com/jsp/qShow.jsp?ta=commspt&qid=20295853 (Part 2)
 
Moderators will finalize this question if in @7 days Asker has not responded.  This will be moved to the PAQ (Previously Asked Questions) at zero points, deleted or awarded.
 
Thanks everyone.
Moondancer
Moderator @ Experts Exchange
0
 
LVL 5

Expert Comment

by:Netminder
Comment Utility
Question abandoned; force-accepted.

Netminder
CS Moderator
0

Featured Post

How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

Join & Write a Comment

Suggested Solutions

Title # Comments Views Activity
network + 7 73
compre toata in where clue oracle 4 41
Java - Why doesn't this JFrame work 3 15
Java Timer (static) 9 9
For beginner Java programmers or at least those new to the Eclipse IDE, the following tutorial will show some (four) ways in which you can import your Java projects to your Eclipse workbench. Introduction While learning Java can be done with…
Go is an acronym of golang, is a programming language developed Google in 2007. Go is a new language that is mostly in the C family, with significant input from Pascal/Modula/Oberon family. Hence Go arisen as low-level language with fast compilation…
Viewers learn about the “for” loop and how it works in Java. By comparing it to the while loop learned before, viewers can make the transition easily. You will learn about the formatting of the for loop as we write a program that prints even numbers…
Viewers will learn how to properly install Eclipse with the necessary JDK, and will take a look at an introductory Java program. Download Eclipse installation zip file: Extract files from zip file: Download and install JDK 8: Open Eclipse and …

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

17 Experts available now in Live!

Get 1:1 Help Now