Solved

Integrating flex/bison with other code

Posted on 2009-07-01
17
601 Views
Last Modified: 2012-05-07
I have a program, built with flex/bison/cc that I pipe data to, and then pipe the output to a file.

This feels sort of like an ugly hack, and I'd like to integrate the code produced by flex/bison with te rest of my project.

I've concluded that I should probably redefine the macro YY_INPUT, but I still fail to make the program read from anywhere other than stdin.

As another hack, I've tried directly changing the call to YY_INPUT
YY_INPUT( (&YY_CURRENT_BUFFER_LVALUE->yy_ch_buf[number_to_move]),
         (yy_n_chars), (size_t) num_to_read );

with my own function
nextSymbol( (&YY_CURRENT_BUFFER_LVALUE->yy_ch_buf[number_to_move]), (yy_n_chars), (size_t) num_to_read );

When I try to step into nextSymbol with gdb, the program stops here, and doesn't move on until I type something in stdin.
0
Comment
Question by:letharion
  • 10
  • 6
17 Comments
 
LVL 40

Expert Comment

by:mrjoltcola
ID: 24755840
I may be misunderstanding your question, but that is the expected behavior.

The lexer will consume tokens until end-of-file or until otherwise told so by the parser.

Can you clarify your question?
0
 
LVL 6

Author Comment

by:letharion
ID: 24756675
>I may be misunderstanding your question, but that is the expected behavior.
I understand that this is the expected behaviour normally, but that is why I have replaced the function with my own. (Which for testing purposes just returns 'a'). The question is why this behaviour remains, even without the YY_INPUT macro.
0
 
LVL 53

Expert Comment

by:Infinity08
ID: 24756768
How did you implement YY_INPUT ?


>> This feels sort of like an ugly hack

Chaining programs isn't an ugly hack imo ... it's basically how the Unix world works for a large part (piping the output of one small program to another - that way combining the small programs into something useful).
0
 
LVL 6

Author Comment

by:letharion
ID: 24761194
>>How did you implement YY_INPUT ?
#undef YY_INPUT
#define YY_INPUT(buf,result,max_size) \
   buf[0] = 'a'; \
   result = 1; \

>>Chaining programs isn't an ugly hack imo ... it's basically how the Unix world works for a large part >>(piping the output of one small program to another - that way combining the small programs into >>something useful).
That's ofc completely true, and I do make use of that functionality regularly myself with programs like cat,find, grep,tail and such.

At the moment I don't recall the details of it, but as I recall I was unable to find a standard way of piping from within a program that was guaranteed to work. Which did surprise me a little since piping is so common, but I didn't think to much about it.
And piping has worked quite well actually. But now I can't give up without cracking this problem ;)
0
 
LVL 53

Expert Comment

by:Infinity08
ID: 24761370
You never return eof (YY_NULL), so the program will never end ... (also make sure that yywrap is implemented as you want it)

You also might want to enclose the macro in {}'s to make sure there are no unintended problems when expanding the macro.

I assume you also placed it in the right location ? (ie. in the definitions section of the lex file, between the %{ and %} at the beginning of the file)

If it still doesn't work, could you post your flex (and bison) input files ?
0
 
LVL 6

Author Comment

by:letharion
ID: 24763058
>You never return eof (YY_NULL), so the program will never end ... (also make sure that yywrap is implemented as you want it)

I'm sure that's important, but is that relevant in this case? The function never seems to execute at all? (So I wouldn't even be able to return YY_NULL even if wanted to)

>You also might want to enclose the macro in {}'s to make sure there are no unintended problems when expanding the macro.
Done

>I assume you also placed it in the right location ? (ie. in the definitions section of the lex file, between the %{ and %} at the beginning of the file)
That was it :) I had it in the y file, not lex. Now I got a looong stream of a's :)

Two questions remain:
1) I pass a float* to the start function in the y file, which is the point from where to read data.
How do I make is visible to the lexer? The code below, which is pretty much what I'm doing doesn't work because y is included after lex. If I include the other way around, I get:lex.yy.c(519): error: linkage specification is incompatible with previous "yywrap"
Related to your comment Inifinity on yywrap?

2) I have a float array that I want parsed by the lexer, best way to have YY_INPUT read floats and return as chars?
.y {

  float *inputData;

  start(float *a) { inputData = a; yyparse(); }

}

Open in new window

0
 
LVL 6

Author Comment

by:letharion
ID: 24763249
1) Wasn't any more difficult than to declare it within %{ }%.
0
 
LVL 6

Author Comment

by:letharion
ID: 24763660
I tried the below macro, placed at line 461. It gives me
lex.yy.c(1060): error: a block-scope function may only have extern storage class
lex.yy.c(1061): error: expected a ";"
lex.yy.c(1084): warning: parsing restarts here after previous syntax error
lex.yy.c(1085): warning: missing return statement at end of non-void function "yy_get_next_buffer"
lex.yy.c(347): error: function "yy_get_previous_state" was referenced but not defined

These errors are way way down in the code. Not sure what the top error means, nor what to do about it.
I tried not using YY_INPUT at all, but directly replacing it's use with the code. The resulting code is also below. That compiles, but my input is all wrong. Looking into that.
//Macro:

%{

float *progs;

int done = 0;

#include <stdio.h>

#include "PFtIF.tab.h"
 

#undef YY_INPUT

#define YY_INPUT(buf,result,max_size) { \

   if(done == 1) result = YY_NULL; \

   else { \

      std::ostringstream str; \

      int i = 0; \

      while(progs[i] != 99.0f && i < max_size) \

         str << progs[i++]; \

\

      for(int j = 0; j < i; j++) \

         buf[j] = str.str().c_str()[j]; \

\

      result = i; \

      if(progs[i] == 99.0f) \

         done = 1; \

   }

%}
 
 
 

//Direct replacement

/* Read in more data. */

if(done == 1) (yy_n_chars) = YY_NULL;

else {

  std::ostringstream str;

  int i = 0;

  while(progs[i] != 99.0f && i < num_to_read)

    str << progs[i++];
 

  for(int j = 0; j < i; j++)

    (&(yy_buffer_stack)[(yy_buffer_stack_top)]->yy_ch_buf[number_to_move])[j] = str.str().c_str()[j];
 

  (yy_n_chars) = i;

  if(progs[i] == 99.0f)

    done = 1;

}

Open in new window

0
What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

 
LVL 6

Author Comment

by:letharion
ID: 24763950
The problem with the incorrect result was two-fold.
To begin with, I never inserted any whitespace, so the lexer got on single large "word".
Second, the number of read floats, doesn't not equal the number of read chars, and I never took that into account.

New code below produces nearly perfect result.
The parsing rules change slightly when the last input is detected, and while the parser no longer enters an eternal loop, but exits correctly, it doesn't detect "the end" the way I want any more.
      /* Read in more data. */

      if(done == 1) (yy_n_chars) = YY_NULL;

      else {

        std::ostringstream str;

        int i = 0, j = 0;

        while(progs[i] != 99.0f && str.str().size() < num_to_read)

          str << progs[i++] << " ";
 

        for(;j < str.str().size(); j++)

          (&(yy_buffer_stack)[(yy_buffer_stack_top)]->yy_ch_buf[number_to_move])[j] = str.str().c_str()[j];
 

        (yy_n_chars) = j;

        if(progs[i] == 99.0f)

          done = 1;

      }

Open in new window

0
 
LVL 53

Expert Comment

by:Infinity08
ID: 24763968
>> 2) I have a float array that I want parsed by the lexer, best way to have YY_INPUT read floats and return as chars?

If you have a float array that needs to be passed to the parser one at a time, you don't need flex ... You can simply write the yylex function yourself, and have it return the next float from the array every time it's called.

This makes me wonder too ... Do you actually need a parser ? If it's just for processing an array of floats, can't you just do that in a loop ?

Could you describe with a bit more detail what you're actually doing ?
0
 
LVL 53

Accepted Solution

by:
Infinity08 earned 500 total points
ID: 24763992
In your code in http:#24763660, you seem to be missing the closing } for the YY_INPUT macro.
0
 
LVL 6

Author Comment

by:letharion
ID: 24764021
99 signals "end", and my loop was "off by one".
Correct loop below:

gdb is a god-send :D

So the only question that remains is why I can't make the function-like macro work.
do {

  str << progs[i] << " ";

} while (progs[i++] != 99.0f && str.str().size() < num_to_read);

Open in new window

0
 
LVL 6

Author Comment

by:letharion
ID: 24764098
Infinity08: You are absolutely right, I just noticed too

Wohoo, everything compiles and runs, time for some real world testing :D
0
 
LVL 6

Author Comment

by:letharion
ID: 24764252
Something still gets wrong when used in the "real world", but I think that's me inputing wrong data.
I also realised I have only gotten half work done, since I need to do something with the input too.
Should be easy now though. Probably do that on monday :)
0
 
LVL 53

Expert Comment

by:Infinity08
ID: 24764267
Good luck ;)
0
 
LVL 6

Author Comment

by:letharion
ID: 25230943
I think it's time to close this.
You've been of great help as always Infinity :)
0
 
LVL 53

Expert Comment

by:Infinity08
ID: 25230970
Glad I could be of assistance :)
0

Featured Post

What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

Go is an acronym of golang, is a programming language developed Google in 2007. Go is a new language that is mostly in the C family, with significant input from Pascal/Modula/Oberon family. Hence Go arisen as low-level language with fast compilation…
Since upgrading to Office 2013 or higher installing the Smart Indenter addin will fail. This article will explain how to install it so it will work regardless of the Office version installed.
Video by: Grant
The goal of this video is to provide viewers with basic examples to understand and use nested-loops in the C programming language.
Video by: Grant
The goal of this video is to provide viewers with basic examples to understand and use while-loops in the C programming language.

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now