Java parsing, better way to use tokenizer?

Posted on 2007-11-23
Last Modified: 2008-09-20
I'm trying to parse a csv file (probably made in excel, or access).

Each field is separated by a "," (comma) but some fields are empty, so I'm not getting the data correctly.

Below is what I'm doing now. Is there a better way to parse?

This line works fine:
23, something, aTitle, something, Smith

This line gives me wrong values:


Both come from the same table with 4 fields.

String line = in.readLine();
	StringTokenizer st = new StringTokenizer(line,",");
	int fieldCount=0;
		String nextFieldData = st.nextToken();
			case 1: 
			String ID = (nextFieldData);
			case 3: 
			String Title  = (nextFieldData);
			case 5: 
			String Name  = (nextFieldData);

Open in new window

Question by:polkadot
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
LVL 26

Expert Comment

ID: 20339104
there are lot of issues which you have to take care if you do it that ways...

instead try some readily available parsers,

Author Comment

ID: 20339344
csvreader works great, but its a bit bulky, I just wanted some ideas in just parsing it simply ... any other ideas
LVL 27

Expert Comment

ID: 20339554
Then you can't use StringTokenizer, you have to write your own parser.  Read the line character by character, check to see if there's a comma, and parse accordingly.
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

LVL 86

Expert Comment

ID: 20339614

I would not use home-brewed parsing. If it were that simple, there would not be any need for classes such as these
LVL 26

Expert Comment

ID: 20339620
>>StringTokenizer st = new StringTokenizer(line,",");


StringTokenizer st = new StringTokenizer( line,",", true );

and discard the odd( which is the comma ) tokens in the traversing process...
LVL 92

Expert Comment

ID: 20340802
you can find a lightweight csv parser here:
LVL 16

Accepted Solution

gnoon earned 500 total points
ID: 20344564
I've surveyed the source code of ST. If delemeters are successive (having blank field(s)), they will be group as a delemeter. For example

data: 1,,3,   will become   1,3,   (ST acts with ,, as ,)
return: {1,3,}

ST is not appropriate to parse CSV file.
If you will use JRE 1.4+, use String.split() instead    String[] fields = line.split(",").
If you will use JRE 1.3-, write your own function to parse the line (never use StringTokenizer) and returns array of fields.
LVL 16

Expert Comment

ID: 20344587
public String[] parse(String s, char delim)
    StringBuffer b = new StringBuffer();
    ArrayList a = new ArrayList();
    char c;
    for(int i=0; i<s.length(); i++)
        c = s.charAt(i);
        if(c == delim)
            b = new StringBuffer();
        else b.append(c);
    String[] r = new String[a.size()];
    for(int i=0; i<r.length; i++)
        r[i] = (String) a.get(i);
    return r;

Featured Post

[Webinar] Code, Load, and Grow

Managing multiple websites, servers, applications, and security on a daily basis? Join us for a webinar on May 25th to learn how to simplify administration and management of virtual hosts for IT admins, create a secure environment, and deploy code more effectively and frequently.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction This article is the last of three articles that explain why and how the Experts Exchange QA Team does test automation for our web site. This article covers our test design approach and then goes through a simple test case example, how …
In this post we will learn how to connect and configure Android Device (Smartphone etc.) with Android Studio. After that we will run a simple Hello World Program.
This tutorial will introduce the viewer to VisualVM for the Java platform application. This video explains an example program and covers the Overview, Monitor, and Heap Dump tabs.
This tutorial explains how to use the VisualVM tool for the Java platform application. This video goes into detail on the Threads, Sampler, and Profiler tabs.
Suggested Courses

734 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question