Solved

Parsing using StreamTokenizer

Posted on 1998-03-05
15
370 Views
Last Modified: 2012-08-13
I'm trying to read tokens from a file using StreamTokenizer. So far, I'm having difficulty in reading this particular data based on the grammar:
      point_list -> point | point COMMA point_list
and the data for example look like this:
      9:8,7:7,1:2
where 9:8 is the point and if there is a comma after is it will then read the next point i.e. 7:7 and so on.
So far my coding looks like this:

void point_list() throws IOException, ParserException {
      this.point();
      if (NextStringToken(",")) {
            this.point_list();
      }
}

void point() throws IOException, ParserException {
      int tok = stok.nextToken();
      if (tok != stok.TT_NUMBER) {
            throw new ParserException("x is not number!");
      }
      else System.out.println("x: " + stok.nval);
            
      this.MatchString(":");
            
      int tok2;
      if ((tok2 = stok.nextToken()) != stok.TT_NUMBER) {
            throw new ParserException("y is not number!");
      }
      else System.out.println("y: " + stok.nval);
}

void MatchString(String s) throws IOException,ParserException {
      int ms = stok.nextToken();
      if (ms == stok.TT_WORD) {
            if (!stok.sval.equals(s)) {
                  throw new ParserException("Doesn't match word!");
            }      
            else System.out.println("Word: " + stok.sval);
      }
      if (( ms != stok.TT_WORD) && (ms != stok.TT_NUMBER)) {
            stok.toString();
            System.out.println(" " + s);
      }
}

boolean NextStringToken(String s) throws IOException, ParserException {
      int nst = stok.nextToken();
      if (nst == stok.TT_WORD) {
            if (stok.sval.equals(s)) {
                  stok.pushBack();
                  return true;      
            }
      }
      if ((nst != stok.TT_WORD) && (nst != stok.TT_NUMBER)) {
            stok.toString();
            stok.pushBack();
            return true;
      }
      stok.pushBack();
      return false;
}

Can you please help me? Thank you.
                        
      
0
Comment
Question by:suhani
  • 8
  • 6
15 Comments
 
LVL 16

Expert Comment

by:imladris
ID: 1233426
It would be helpful if you could indicate more specifically what difficulty you are encountering.

0
 
LVL 1

Expert Comment

by:froderik
ID: 1233427
Generally it is good to check if there are any tokens left with the sending of hasMoreTokens() to the StringTokenizer object before each call to nextToken(). If you don't, you will run into a unwanted runtime exception.

Feel free to reject this answer if it is out of scope. I agree with imladris that more details about the specific error would be beneficial.

Here is some code written just for fun that returns an array of Strings with things between commas:

public String[] someTokens( String inString )
{
  StringTokenixer myTokens = new StringTokenizer( inString, ",");
  Vector results = new Vector();
  while( myTokens.hasMoreTokens() )
    results.addElement( myTokens.nextToken() );

  String[] retArray = new String[ results.size() ];
  for( int i=0; i < results.size(); i++ )
    retArray[ i ] = (String)results.elementAt( i );

  return retArray;    
}
0
 

Author Comment

by:suhani
ID: 1233428
The problem is I cannot read the next point after the comma.
0
Active Directory Webinar

We all know we need to protect and secure our privileges, but where to start? Join Experts Exchange and ManageEngine on Tuesday, April 11, 2017 10:00 AM PDT to learn how to track and secure privileged users in Active Directory.

 

Author Comment

by:suhani
ID: 1233429
The problem is I cannot read the next token (i.e. point) after the comma.
0
 
LVL 1

Expert Comment

by:froderik
ID: 1233430
So you can read the first but not the second? Then I would say that there is something wrong with the logic of your code. Subsequent use of nextToken() should return the wanted points. I don't see a construction of a StringTokenizer object in your code. How does your constructor look?

Do you understand the piece of code I wrote in the rejected answer? Are your code working in the same way?
0
 

Author Comment

by:suhani
ID: 1233431
I'm not using StringTokenizer but StreamTokenizer.
0
 
LVL 1

Accepted Solution

by:
froderik earned 40 total points
ID: 1233432
Ok, my apologies for entering stupid comments... Now I make more sense out of your code. In my Java book "Java in a nutshell" it is said about StreanTokenizer that "nextToken() returns the next token in the stream - this is either one of the constants of the class, (...) or a character value." It seems to me that the parsing of the comma would fall into the character category. Your NextStringToken() method doesn't care about the character option though. If the returned int from nextToken() falls outside the four constants it should be a specific character value. Do a cast on it and compare with the incoming string.

Makes sense?
0
 

Author Comment

by:suhani
ID: 1233433
What do you mean by this:
"If the returned int from nextToken() falls outside the four constants it should be a specific character value."
Can you please give me an example how do you cast on it?
0
 

Author Comment

by:suhani
ID: 1233434
What do you mean by this:
"If the returned int from nextToken() falls outside the four
   constants it should be a specific character value."
Could you please give me an example how do you cast on it?
0
 
LVL 1

Expert Comment

by:froderik
ID: 1233435
Ok, here is a method that reads the next token and checks if it is equal to some incoming string.

public boolean compareNextToken( StreamTokenizer tokensStream, String compString )
throws IOException
{
  int nextToken = tokensStream.nextToken();
  if( nextToken == TT_WORD )
    return tokensStream.sval.equals( compString );
  // Add tests for the other constants here.
  else
  {
    char inChar = (char)nextToken;
    return compString.length() == 1 && compString.startsWith( inChar )
  }
}

Something like that. Do you get it? It you would like some more information about type casts just let me know.
0
 

Author Comment

by:suhani
ID: 1233436
How come when I tried it out, it gave me an error:
      Invalid expression statement
Anyway, I would be more grateful if you can give me an example how to read comma based on type casts.

0
 
LVL 1

Expert Comment

by:froderik
ID: 1233437
For which expression did you get that message?

You could use the above method to read a comma like this:

compareNextToken( yourStreamTokenizer, "," );

Returns true if the next token in yourStremTokenizer is a comma.

Personally I find the StringTokenizer easier to use and understand. With a StringTokenizer you can simply define which character(s) you would like to separate the tokens as I have described in previous comments. Consider using the method provided above which takes a String as input and returns an array of Strings extracted from the input String with comma as delimiter character.
0
 

Author Comment

by:suhani
ID: 1233438
There was an error again. This time it was:
      Incompatible type for method. Can't convert char to java.lang.String.
        return compString.length() == 1 && compString.startsWith(inChar);

0
 
LVL 1

Expert Comment

by:froderik
ID: 1233439
I'm sorry. startsWith() takes a String as argument, not a character as I assumed without checking it up. Replace inChar in the argument to startsWith() with String.valueOf(inChar)  and that part should work...

(I have really made all possible mistakes when answering this question.)
0
 

Author Comment

by:suhani
ID: 1233440
It works! Thank you.
0

Featured Post

Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

For customizing the look of your lightweight component and making it look opaque like it was made of plastic.  This tip assumes your component to be of rectangular shape and completely opaque.   (CODE)
Java functions are among the best things for programmers to work with as Java sites can be very easy to read and prepare. Java especially simplifies many processes in the coding industry as it helps integrate many forms of technology and different d…
Viewers will learn about the different types of variables in Java and how to declare them. Decide the type of variable desired: Put the keyword corresponding to the type of variable in front of the variable name: Use the equal sign to assign a v…
This tutorial covers a practical example of lazy loading technique and early loading technique in a Singleton Design Pattern.

828 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question