Solved

Parsing using StreamTokenizer

Posted on 1998-03-05
15
360 Views
Last Modified: 2012-08-13
I'm trying to read tokens from a file using StreamTokenizer. So far, I'm having difficulty in reading this particular data based on the grammar:
      point_list -> point | point COMMA point_list
and the data for example look like this:
      9:8,7:7,1:2
where 9:8 is the point and if there is a comma after is it will then read the next point i.e. 7:7 and so on.
So far my coding looks like this:

void point_list() throws IOException, ParserException {
      this.point();
      if (NextStringToken(",")) {
            this.point_list();
      }
}

void point() throws IOException, ParserException {
      int tok = stok.nextToken();
      if (tok != stok.TT_NUMBER) {
            throw new ParserException("x is not number!");
      }
      else System.out.println("x: " + stok.nval);
            
      this.MatchString(":");
            
      int tok2;
      if ((tok2 = stok.nextToken()) != stok.TT_NUMBER) {
            throw new ParserException("y is not number!");
      }
      else System.out.println("y: " + stok.nval);
}

void MatchString(String s) throws IOException,ParserException {
      int ms = stok.nextToken();
      if (ms == stok.TT_WORD) {
            if (!stok.sval.equals(s)) {
                  throw new ParserException("Doesn't match word!");
            }      
            else System.out.println("Word: " + stok.sval);
      }
      if (( ms != stok.TT_WORD) && (ms != stok.TT_NUMBER)) {
            stok.toString();
            System.out.println(" " + s);
      }
}

boolean NextStringToken(String s) throws IOException, ParserException {
      int nst = stok.nextToken();
      if (nst == stok.TT_WORD) {
            if (stok.sval.equals(s)) {
                  stok.pushBack();
                  return true;      
            }
      }
      if ((nst != stok.TT_WORD) && (nst != stok.TT_NUMBER)) {
            stok.toString();
            stok.pushBack();
            return true;
      }
      stok.pushBack();
      return false;
}

Can you please help me? Thank you.
                        
      
0
Comment
Question by:suhani
  • 8
  • 6
15 Comments
 
LVL 16

Expert Comment

by:imladris
ID: 1233426
It would be helpful if you could indicate more specifically what difficulty you are encountering.

0
 
LVL 1

Expert Comment

by:froderik
ID: 1233427
Generally it is good to check if there are any tokens left with the sending of hasMoreTokens() to the StringTokenizer object before each call to nextToken(). If you don't, you will run into a unwanted runtime exception.

Feel free to reject this answer if it is out of scope. I agree with imladris that more details about the specific error would be beneficial.

Here is some code written just for fun that returns an array of Strings with things between commas:

public String[] someTokens( String inString )
{
  StringTokenixer myTokens = new StringTokenizer( inString, ",");
  Vector results = new Vector();
  while( myTokens.hasMoreTokens() )
    results.addElement( myTokens.nextToken() );

  String[] retArray = new String[ results.size() ];
  for( int i=0; i < results.size(); i++ )
    retArray[ i ] = (String)results.elementAt( i );

  return retArray;    
}
0
 

Author Comment

by:suhani
ID: 1233428
The problem is I cannot read the next point after the comma.
0
 

Author Comment

by:suhani
ID: 1233429
The problem is I cannot read the next token (i.e. point) after the comma.
0
 
LVL 1

Expert Comment

by:froderik
ID: 1233430
So you can read the first but not the second? Then I would say that there is something wrong with the logic of your code. Subsequent use of nextToken() should return the wanted points. I don't see a construction of a StringTokenizer object in your code. How does your constructor look?

Do you understand the piece of code I wrote in the rejected answer? Are your code working in the same way?
0
 

Author Comment

by:suhani
ID: 1233431
I'm not using StringTokenizer but StreamTokenizer.
0
 
LVL 1

Accepted Solution

by:
froderik earned 40 total points
ID: 1233432
Ok, my apologies for entering stupid comments... Now I make more sense out of your code. In my Java book "Java in a nutshell" it is said about StreanTokenizer that "nextToken() returns the next token in the stream - this is either one of the constants of the class, (...) or a character value." It seems to me that the parsing of the comma would fall into the character category. Your NextStringToken() method doesn't care about the character option though. If the returned int from nextToken() falls outside the four constants it should be a specific character value. Do a cast on it and compare with the incoming string.

Makes sense?
0
Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

 

Author Comment

by:suhani
ID: 1233433
What do you mean by this:
"If the returned int from nextToken() falls outside the four constants it should be a specific character value."
Can you please give me an example how do you cast on it?
0
 

Author Comment

by:suhani
ID: 1233434
What do you mean by this:
"If the returned int from nextToken() falls outside the four
   constants it should be a specific character value."
Could you please give me an example how do you cast on it?
0
 
LVL 1

Expert Comment

by:froderik
ID: 1233435
Ok, here is a method that reads the next token and checks if it is equal to some incoming string.

public boolean compareNextToken( StreamTokenizer tokensStream, String compString )
throws IOException
{
  int nextToken = tokensStream.nextToken();
  if( nextToken == TT_WORD )
    return tokensStream.sval.equals( compString );
  // Add tests for the other constants here.
  else
  {
    char inChar = (char)nextToken;
    return compString.length() == 1 && compString.startsWith( inChar )
  }
}

Something like that. Do you get it? It you would like some more information about type casts just let me know.
0
 

Author Comment

by:suhani
ID: 1233436
How come when I tried it out, it gave me an error:
      Invalid expression statement
Anyway, I would be more grateful if you can give me an example how to read comma based on type casts.

0
 
LVL 1

Expert Comment

by:froderik
ID: 1233437
For which expression did you get that message?

You could use the above method to read a comma like this:

compareNextToken( yourStreamTokenizer, "," );

Returns true if the next token in yourStremTokenizer is a comma.

Personally I find the StringTokenizer easier to use and understand. With a StringTokenizer you can simply define which character(s) you would like to separate the tokens as I have described in previous comments. Consider using the method provided above which takes a String as input and returns an array of Strings extracted from the input String with comma as delimiter character.
0
 

Author Comment

by:suhani
ID: 1233438
There was an error again. This time it was:
      Incompatible type for method. Can't convert char to java.lang.String.
        return compString.length() == 1 && compString.startsWith(inChar);

0
 
LVL 1

Expert Comment

by:froderik
ID: 1233439
I'm sorry. startsWith() takes a String as argument, not a character as I assumed without checking it up. Replace inChar in the argument to startsWith() with String.valueOf(inChar)  and that part should work...

(I have really made all possible mistakes when answering this question.)
0
 

Author Comment

by:suhani
ID: 1233440
It works! Thank you.
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

For beginner Java programmers or at least those new to the Eclipse IDE, the following tutorial will show some (four) ways in which you can import your Java projects to your Eclipse workbench. Introduction While learning Java can be done with…
Java functions are among the best things for programmers to work with as Java sites can be very easy to read and prepare. Java especially simplifies many processes in the coding industry as it helps integrate many forms of technology and different d…
Viewers learn about the third conditional statement “else if” and use it in an example program. Then additional information about conditional statements is provided, covering the topic thoroughly. Viewers learn about the third conditional statement …
Viewers will learn about arithmetic and Boolean expressions in Java and the logical operators used to create Boolean expressions. We will cover the symbols used for arithmetic expressions and define each logical operator and how to use them in Boole…

863 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

27 Experts available now in Live!

Get 1:1 Help Now