Parsing using StreamTokenizer

I'm trying to read tokens from a file using StreamTokenizer. So far, I'm having difficulty in reading this particular data based on the grammar:
      point_list -> point | point COMMA point_list
and the data for example look like this:
      9:8,7:7,1:2
where 9:8 is the point and if there is a comma after is it will then read the next point i.e. 7:7 and so on.
So far my coding looks like this:

void point_list() throws IOException, ParserException {
      this.point();
      if (NextStringToken(",")) {
            this.point_list();
      }
}

void point() throws IOException, ParserException {
      int tok = stok.nextToken();
      if (tok != stok.TT_NUMBER) {
            throw new ParserException("x is not number!");
      }
      else System.out.println("x: " + stok.nval);
            
      this.MatchString(":");
            
      int tok2;
      if ((tok2 = stok.nextToken()) != stok.TT_NUMBER) {
            throw new ParserException("y is not number!");
      }
      else System.out.println("y: " + stok.nval);
}

void MatchString(String s) throws IOException,ParserException {
      int ms = stok.nextToken();
      if (ms == stok.TT_WORD) {
            if (!stok.sval.equals(s)) {
                  throw new ParserException("Doesn't match word!");
            }      
            else System.out.println("Word: " + stok.sval);
      }
      if (( ms != stok.TT_WORD) && (ms != stok.TT_NUMBER)) {
            stok.toString();
            System.out.println(" " + s);
      }
}

boolean NextStringToken(String s) throws IOException, ParserException {
      int nst = stok.nextToken();
      if (nst == stok.TT_WORD) {
            if (stok.sval.equals(s)) {
                  stok.pushBack();
                  return true;      
            }
      }
      if ((nst != stok.TT_WORD) && (nst != stok.TT_NUMBER)) {
            stok.toString();
            stok.pushBack();
            return true;
      }
      stok.pushBack();
      return false;
}

Can you please help me? Thank you.
                        
      
suhaniAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

imladrisCommented:
It would be helpful if you could indicate more specifically what difficulty you are encountering.

0
froderikCommented:
Generally it is good to check if there are any tokens left with the sending of hasMoreTokens() to the StringTokenizer object before each call to nextToken(). If you don't, you will run into a unwanted runtime exception.

Feel free to reject this answer if it is out of scope. I agree with imladris that more details about the specific error would be beneficial.

Here is some code written just for fun that returns an array of Strings with things between commas:

public String[] someTokens( String inString )
{
  StringTokenixer myTokens = new StringTokenizer( inString, ",");
  Vector results = new Vector();
  while( myTokens.hasMoreTokens() )
    results.addElement( myTokens.nextToken() );

  String[] retArray = new String[ results.size() ];
  for( int i=0; i < results.size(); i++ )
    retArray[ i ] = (String)results.elementAt( i );

  return retArray;    
}
0
suhaniAuthor Commented:
The problem is I cannot read the next point after the comma.
0
Cloud Class® Course: Amazon Web Services - Basic

Are you thinking about creating an Amazon Web Services account for your business? Not sure where to start? In this course you’ll get an overview of the history of AWS and take a tour of their user interface.

suhaniAuthor Commented:
The problem is I cannot read the next token (i.e. point) after the comma.
0
froderikCommented:
So you can read the first but not the second? Then I would say that there is something wrong with the logic of your code. Subsequent use of nextToken() should return the wanted points. I don't see a construction of a StringTokenizer object in your code. How does your constructor look?

Do you understand the piece of code I wrote in the rejected answer? Are your code working in the same way?
0
suhaniAuthor Commented:
I'm not using StringTokenizer but StreamTokenizer.
0
froderikCommented:
Ok, my apologies for entering stupid comments... Now I make more sense out of your code. In my Java book "Java in a nutshell" it is said about StreanTokenizer that "nextToken() returns the next token in the stream - this is either one of the constants of the class, (...) or a character value." It seems to me that the parsing of the comma would fall into the character category. Your NextStringToken() method doesn't care about the character option though. If the returned int from nextToken() falls outside the four constants it should be a specific character value. Do a cast on it and compare with the incoming string.

Makes sense?
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
suhaniAuthor Commented:
What do you mean by this:
"If the returned int from nextToken() falls outside the four constants it should be a specific character value."
Can you please give me an example how do you cast on it?
0
suhaniAuthor Commented:
What do you mean by this:
"If the returned int from nextToken() falls outside the four
   constants it should be a specific character value."
Could you please give me an example how do you cast on it?
0
froderikCommented:
Ok, here is a method that reads the next token and checks if it is equal to some incoming string.

public boolean compareNextToken( StreamTokenizer tokensStream, String compString )
throws IOException
{
  int nextToken = tokensStream.nextToken();
  if( nextToken == TT_WORD )
    return tokensStream.sval.equals( compString );
  // Add tests for the other constants here.
  else
  {
    char inChar = (char)nextToken;
    return compString.length() == 1 && compString.startsWith( inChar )
  }
}

Something like that. Do you get it? It you would like some more information about type casts just let me know.
0
suhaniAuthor Commented:
How come when I tried it out, it gave me an error:
      Invalid expression statement
Anyway, I would be more grateful if you can give me an example how to read comma based on type casts.

0
froderikCommented:
For which expression did you get that message?

You could use the above method to read a comma like this:

compareNextToken( yourStreamTokenizer, "," );

Returns true if the next token in yourStremTokenizer is a comma.

Personally I find the StringTokenizer easier to use and understand. With a StringTokenizer you can simply define which character(s) you would like to separate the tokens as I have described in previous comments. Consider using the method provided above which takes a String as input and returns an array of Strings extracted from the input String with comma as delimiter character.
0
suhaniAuthor Commented:
There was an error again. This time it was:
      Incompatible type for method. Can't convert char to java.lang.String.
        return compString.length() == 1 && compString.startsWith(inChar);

0
froderikCommented:
I'm sorry. startsWith() takes a String as argument, not a character as I assumed without checking it up. Replace inChar in the argument to startsWith() with String.valueOf(inChar)  and that part should work...

(I have really made all possible mistakes when answering this question.)
0
suhaniAuthor Commented:
It works! Thank you.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Java

From novice to tech pro — start learning today.