Solved

Parsing using StreamTokenizer

Posted on 1998-03-05
15
367 Views
Last Modified: 2012-08-13
I'm trying to read tokens from a file using StreamTokenizer. So far, I'm having difficulty in reading this particular data based on the grammar:
      point_list -> point | point COMMA point_list
and the data for example look like this:
      9:8,7:7,1:2
where 9:8 is the point and if there is a comma after is it will then read the next point i.e. 7:7 and so on.
So far my coding looks like this:

void point_list() throws IOException, ParserException {
      this.point();
      if (NextStringToken(",")) {
            this.point_list();
      }
}

void point() throws IOException, ParserException {
      int tok = stok.nextToken();
      if (tok != stok.TT_NUMBER) {
            throw new ParserException("x is not number!");
      }
      else System.out.println("x: " + stok.nval);
            
      this.MatchString(":");
            
      int tok2;
      if ((tok2 = stok.nextToken()) != stok.TT_NUMBER) {
            throw new ParserException("y is not number!");
      }
      else System.out.println("y: " + stok.nval);
}

void MatchString(String s) throws IOException,ParserException {
      int ms = stok.nextToken();
      if (ms == stok.TT_WORD) {
            if (!stok.sval.equals(s)) {
                  throw new ParserException("Doesn't match word!");
            }      
            else System.out.println("Word: " + stok.sval);
      }
      if (( ms != stok.TT_WORD) && (ms != stok.TT_NUMBER)) {
            stok.toString();
            System.out.println(" " + s);
      }
}

boolean NextStringToken(String s) throws IOException, ParserException {
      int nst = stok.nextToken();
      if (nst == stok.TT_WORD) {
            if (stok.sval.equals(s)) {
                  stok.pushBack();
                  return true;      
            }
      }
      if ((nst != stok.TT_WORD) && (nst != stok.TT_NUMBER)) {
            stok.toString();
            stok.pushBack();
            return true;
      }
      stok.pushBack();
      return false;
}

Can you please help me? Thank you.
                        
      
0
Comment
Question by:suhani
  • 8
  • 6
15 Comments
 
LVL 16

Expert Comment

by:imladris
ID: 1233426
It would be helpful if you could indicate more specifically what difficulty you are encountering.

0
 
LVL 1

Expert Comment

by:froderik
ID: 1233427
Generally it is good to check if there are any tokens left with the sending of hasMoreTokens() to the StringTokenizer object before each call to nextToken(). If you don't, you will run into a unwanted runtime exception.

Feel free to reject this answer if it is out of scope. I agree with imladris that more details about the specific error would be beneficial.

Here is some code written just for fun that returns an array of Strings with things between commas:

public String[] someTokens( String inString )
{
  StringTokenixer myTokens = new StringTokenizer( inString, ",");
  Vector results = new Vector();
  while( myTokens.hasMoreTokens() )
    results.addElement( myTokens.nextToken() );

  String[] retArray = new String[ results.size() ];
  for( int i=0; i < results.size(); i++ )
    retArray[ i ] = (String)results.elementAt( i );

  return retArray;    
}
0
 

Author Comment

by:suhani
ID: 1233428
The problem is I cannot read the next point after the comma.
0
Problems using Powershell and Active Directory?

Managing Active Directory does not always have to be complicated.  If you are spending more time trying instead of doing, then it's time to look at something else. For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why

 

Author Comment

by:suhani
ID: 1233429
The problem is I cannot read the next token (i.e. point) after the comma.
0
 
LVL 1

Expert Comment

by:froderik
ID: 1233430
So you can read the first but not the second? Then I would say that there is something wrong with the logic of your code. Subsequent use of nextToken() should return the wanted points. I don't see a construction of a StringTokenizer object in your code. How does your constructor look?

Do you understand the piece of code I wrote in the rejected answer? Are your code working in the same way?
0
 

Author Comment

by:suhani
ID: 1233431
I'm not using StringTokenizer but StreamTokenizer.
0
 
LVL 1

Accepted Solution

by:
froderik earned 40 total points
ID: 1233432
Ok, my apologies for entering stupid comments... Now I make more sense out of your code. In my Java book "Java in a nutshell" it is said about StreanTokenizer that "nextToken() returns the next token in the stream - this is either one of the constants of the class, (...) or a character value." It seems to me that the parsing of the comma would fall into the character category. Your NextStringToken() method doesn't care about the character option though. If the returned int from nextToken() falls outside the four constants it should be a specific character value. Do a cast on it and compare with the incoming string.

Makes sense?
0
 

Author Comment

by:suhani
ID: 1233433
What do you mean by this:
"If the returned int from nextToken() falls outside the four constants it should be a specific character value."
Can you please give me an example how do you cast on it?
0
 

Author Comment

by:suhani
ID: 1233434
What do you mean by this:
"If the returned int from nextToken() falls outside the four
   constants it should be a specific character value."
Could you please give me an example how do you cast on it?
0
 
LVL 1

Expert Comment

by:froderik
ID: 1233435
Ok, here is a method that reads the next token and checks if it is equal to some incoming string.

public boolean compareNextToken( StreamTokenizer tokensStream, String compString )
throws IOException
{
  int nextToken = tokensStream.nextToken();
  if( nextToken == TT_WORD )
    return tokensStream.sval.equals( compString );
  // Add tests for the other constants here.
  else
  {
    char inChar = (char)nextToken;
    return compString.length() == 1 && compString.startsWith( inChar )
  }
}

Something like that. Do you get it? It you would like some more information about type casts just let me know.
0
 

Author Comment

by:suhani
ID: 1233436
How come when I tried it out, it gave me an error:
      Invalid expression statement
Anyway, I would be more grateful if you can give me an example how to read comma based on type casts.

0
 
LVL 1

Expert Comment

by:froderik
ID: 1233437
For which expression did you get that message?

You could use the above method to read a comma like this:

compareNextToken( yourStreamTokenizer, "," );

Returns true if the next token in yourStremTokenizer is a comma.

Personally I find the StringTokenizer easier to use and understand. With a StringTokenizer you can simply define which character(s) you would like to separate the tokens as I have described in previous comments. Consider using the method provided above which takes a String as input and returns an array of Strings extracted from the input String with comma as delimiter character.
0
 

Author Comment

by:suhani
ID: 1233438
There was an error again. This time it was:
      Incompatible type for method. Can't convert char to java.lang.String.
        return compString.length() == 1 && compString.startsWith(inChar);

0
 
LVL 1

Expert Comment

by:froderik
ID: 1233439
I'm sorry. startsWith() takes a String as argument, not a character as I assumed without checking it up. Replace inChar in the argument to startsWith() with String.valueOf(inChar)  and that part should work...

(I have really made all possible mistakes when answering this question.)
0
 

Author Comment

by:suhani
ID: 1233440
It works! Thank you.
0

Featured Post

PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
maven project error 5 56
Opening PDF on button click and fill new document 2 39
Java exception bubble up 2 18
mysql jsp example issue 32 38
For customizing the look of your lightweight component and making it look lucid like it was made of glass. Or: how to make your component more Apple-ish ;) This tip assumes your component to be of rectangular shape and completely opaque. (COD…
Java functions are among the best things for programmers to work with as Java sites can be very easy to read and prepare. Java especially simplifies many processes in the coding industry as it helps integrate many forms of technology and different d…
Viewers will learn about the regular for loop in Java and how to use it. Definition: Break the for loop down into 3 parts: Syntax when using for loops: Example using a for loop:
This tutorial explains how to use the VisualVM tool for the Java platform application. This video goes into detail on the Threads, Sampler, and Profiler tabs.

803 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question