Go Premium for a chance to win a PS4. Enter to Win

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 391
  • Last Modified:

Parsing using StreamTokenizer

I'm trying to read tokens from a file using StreamTokenizer. So far, I'm having difficulty in reading this particular data based on the grammar:
      point_list -> point | point COMMA point_list
and the data for example look like this:
      9:8,7:7,1:2
where 9:8 is the point and if there is a comma after is it will then read the next point i.e. 7:7 and so on.
So far my coding looks like this:

void point_list() throws IOException, ParserException {
      this.point();
      if (NextStringToken(",")) {
            this.point_list();
      }
}

void point() throws IOException, ParserException {
      int tok = stok.nextToken();
      if (tok != stok.TT_NUMBER) {
            throw new ParserException("x is not number!");
      }
      else System.out.println("x: " + stok.nval);
            
      this.MatchString(":");
            
      int tok2;
      if ((tok2 = stok.nextToken()) != stok.TT_NUMBER) {
            throw new ParserException("y is not number!");
      }
      else System.out.println("y: " + stok.nval);
}

void MatchString(String s) throws IOException,ParserException {
      int ms = stok.nextToken();
      if (ms == stok.TT_WORD) {
            if (!stok.sval.equals(s)) {
                  throw new ParserException("Doesn't match word!");
            }      
            else System.out.println("Word: " + stok.sval);
      }
      if (( ms != stok.TT_WORD) && (ms != stok.TT_NUMBER)) {
            stok.toString();
            System.out.println(" " + s);
      }
}

boolean NextStringToken(String s) throws IOException, ParserException {
      int nst = stok.nextToken();
      if (nst == stok.TT_WORD) {
            if (stok.sval.equals(s)) {
                  stok.pushBack();
                  return true;      
            }
      }
      if ((nst != stok.TT_WORD) && (nst != stok.TT_NUMBER)) {
            stok.toString();
            stok.pushBack();
            return true;
      }
      stok.pushBack();
      return false;
}

Can you please help me? Thank you.
                        
      
0
suhani
Asked:
suhani
  • 8
  • 6
1 Solution
 
imladrisCommented:
It would be helpful if you could indicate more specifically what difficulty you are encountering.

0
 
froderikCommented:
Generally it is good to check if there are any tokens left with the sending of hasMoreTokens() to the StringTokenizer object before each call to nextToken(). If you don't, you will run into a unwanted runtime exception.

Feel free to reject this answer if it is out of scope. I agree with imladris that more details about the specific error would be beneficial.

Here is some code written just for fun that returns an array of Strings with things between commas:

public String[] someTokens( String inString )
{
  StringTokenixer myTokens = new StringTokenizer( inString, ",");
  Vector results = new Vector();
  while( myTokens.hasMoreTokens() )
    results.addElement( myTokens.nextToken() );

  String[] retArray = new String[ results.size() ];
  for( int i=0; i < results.size(); i++ )
    retArray[ i ] = (String)results.elementAt( i );

  return retArray;    
}
0
 
suhaniAuthor Commented:
The problem is I cannot read the next point after the comma.
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
suhaniAuthor Commented:
The problem is I cannot read the next token (i.e. point) after the comma.
0
 
froderikCommented:
So you can read the first but not the second? Then I would say that there is something wrong with the logic of your code. Subsequent use of nextToken() should return the wanted points. I don't see a construction of a StringTokenizer object in your code. How does your constructor look?

Do you understand the piece of code I wrote in the rejected answer? Are your code working in the same way?
0
 
suhaniAuthor Commented:
I'm not using StringTokenizer but StreamTokenizer.
0
 
froderikCommented:
Ok, my apologies for entering stupid comments... Now I make more sense out of your code. In my Java book "Java in a nutshell" it is said about StreanTokenizer that "nextToken() returns the next token in the stream - this is either one of the constants of the class, (...) or a character value." It seems to me that the parsing of the comma would fall into the character category. Your NextStringToken() method doesn't care about the character option though. If the returned int from nextToken() falls outside the four constants it should be a specific character value. Do a cast on it and compare with the incoming string.

Makes sense?
0
 
suhaniAuthor Commented:
What do you mean by this:
"If the returned int from nextToken() falls outside the four constants it should be a specific character value."
Can you please give me an example how do you cast on it?
0
 
suhaniAuthor Commented:
What do you mean by this:
"If the returned int from nextToken() falls outside the four
   constants it should be a specific character value."
Could you please give me an example how do you cast on it?
0
 
froderikCommented:
Ok, here is a method that reads the next token and checks if it is equal to some incoming string.

public boolean compareNextToken( StreamTokenizer tokensStream, String compString )
throws IOException
{
  int nextToken = tokensStream.nextToken();
  if( nextToken == TT_WORD )
    return tokensStream.sval.equals( compString );
  // Add tests for the other constants here.
  else
  {
    char inChar = (char)nextToken;
    return compString.length() == 1 && compString.startsWith( inChar )
  }
}

Something like that. Do you get it? It you would like some more information about type casts just let me know.
0
 
suhaniAuthor Commented:
How come when I tried it out, it gave me an error:
      Invalid expression statement
Anyway, I would be more grateful if you can give me an example how to read comma based on type casts.

0
 
froderikCommented:
For which expression did you get that message?

You could use the above method to read a comma like this:

compareNextToken( yourStreamTokenizer, "," );

Returns true if the next token in yourStremTokenizer is a comma.

Personally I find the StringTokenizer easier to use and understand. With a StringTokenizer you can simply define which character(s) you would like to separate the tokens as I have described in previous comments. Consider using the method provided above which takes a String as input and returns an array of Strings extracted from the input String with comma as delimiter character.
0
 
suhaniAuthor Commented:
There was an error again. This time it was:
      Incompatible type for method. Can't convert char to java.lang.String.
        return compString.length() == 1 && compString.startsWith(inChar);

0
 
froderikCommented:
I'm sorry. startsWith() takes a String as argument, not a character as I assumed without checking it up. Replace inChar in the argument to startsWith() with String.valueOf(inChar)  and that part should work...

(I have really made all possible mistakes when answering this question.)
0
 
suhaniAuthor Commented:
It works! Thank you.
0

Featured Post

Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

  • 8
  • 6
Tackle projects and never again get stuck behind a technical roadblock.
Join Now