Java parsing, better way to use tokenizer?

Posted on 2007-11-23
Last Modified: 2008-09-20
I'm trying to parse a csv file (probably made in excel, or access).

Each field is separated by a "," (comma) but some fields are empty, so I'm not getting the data correctly.

Below is what I'm doing now. Is there a better way to parse?

This line works fine:
23, something, aTitle, something, Smith

This line gives me wrong values:


Both come from the same table with 4 fields.

String line = in.readLine();
	StringTokenizer st = new StringTokenizer(line,",");
	int fieldCount=0;
		String nextFieldData = st.nextToken();
			case 1: 
			String ID = (nextFieldData);
			case 3: 
			String Title  = (nextFieldData);
			case 5: 
			String Name  = (nextFieldData);

Open in new window

Question by:polkadot
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
LVL 26

Expert Comment

ID: 20339104
there are lot of issues which you have to take care if you do it that ways...

instead try some readily available parsers,

Author Comment

ID: 20339344
csvreader works great, but its a bit bulky, I just wanted some ideas in just parsing it simply ... any other ideas
LVL 27

Expert Comment

ID: 20339554
Then you can't use StringTokenizer, you have to write your own parser.  Read the line character by character, check to see if there's a comma, and parse accordingly.
Optimize your web performance

What's in the eBook?
- Full list of reasons for poor performance
- Ultimate measures to speed things up
- Primary web monitoring types
- KPIs you should be monitoring in order to increase your ROI

LVL 86

Expert Comment

ID: 20339614

I would not use home-brewed parsing. If it were that simple, there would not be any need for classes such as these
LVL 26

Expert Comment

ID: 20339620
>>StringTokenizer st = new StringTokenizer(line,",");


StringTokenizer st = new StringTokenizer( line,",", true );

and discard the odd( which is the comma ) tokens in the traversing process...
LVL 92

Expert Comment

ID: 20340802
you can find a lightweight csv parser here:
LVL 16

Accepted Solution

gnoon earned 500 total points
ID: 20344564
I've surveyed the source code of ST. If delemeters are successive (having blank field(s)), they will be group as a delemeter. For example

data: 1,,3,   will become   1,3,   (ST acts with ,, as ,)
return: {1,3,}

ST is not appropriate to parse CSV file.
If you will use JRE 1.4+, use String.split() instead    String[] fields = line.split(",").
If you will use JRE 1.3-, write your own function to parse the line (never use StringTokenizer) and returns array of fields.
LVL 16

Expert Comment

ID: 20344587
public String[] parse(String s, char delim)
    StringBuffer b = new StringBuffer();
    ArrayList a = new ArrayList();
    char c;
    for(int i=0; i<s.length(); i++)
        c = s.charAt(i);
        if(c == delim)
            b = new StringBuffer();
        else b.append(c);
    String[] r = new String[a.size()];
    for(int i=0; i<r.length; i++)
        r[i] = (String) a.get(i);
    return r;

Featured Post

Get 15 Days FREE Full-Featured Trial

Benefit from a mission critical IT monitoring with Monitis Premium or get it FREE for your entry level monitoring needs.
-Over 200,000 users
-More than 300,000 websites monitored
-Used in 197 countries
-Recommended by 98% of users

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction This article is the second of three articles that explain why and how the Experts Exchange QA Team does test automation for our web site. This article covers the basic installation and configuration of the test automation tools used by…
Introduction This article is the last of three articles that explain why and how the Experts Exchange QA Team does test automation for our web site. This article covers our test design approach and then goes through a simple test case example, how …
Viewers learn about the “while” loop and how to utilize it correctly in Java. Additionally, viewers begin exploring how to include conditional statements within a while loop and avoid an endless loop. Define While Loop: Basic Example: Explanatio…
This video teaches viewers about errors in exception handling.
Suggested Courses

631 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question