Integrating flex/bison with other code

I have a program, built with flex/bison/cc that I pipe data to, and then pipe the output to a file.

This feels sort of like an ugly hack, and I'd like to integrate the code produced by flex/bison with te rest of my project.

I've concluded that I should probably redefine the macro YY_INPUT, but I still fail to make the program read from anywhere other than stdin.

As another hack, I've tried directly changing the call to YY_INPUT
YY_INPUT( (&YY_CURRENT_BUFFER_LVALUE->yy_ch_buf[number_to_move]),
         (yy_n_chars), (size_t) num_to_read );

with my own function
nextSymbol( (&YY_CURRENT_BUFFER_LVALUE->yy_ch_buf[number_to_move]), (yy_n_chars), (size_t) num_to_read );

When I try to step into nextSymbol with gdb, the program stops here, and doesn't move on until I type something in stdin.
LVL 6
letharionAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

mrjoltcolaCommented:
I may be misunderstanding your question, but that is the expected behavior.

The lexer will consume tokens until end-of-file or until otherwise told so by the parser.

Can you clarify your question?
0
letharionAuthor Commented:
>I may be misunderstanding your question, but that is the expected behavior.
I understand that this is the expected behaviour normally, but that is why I have replaced the function with my own. (Which for testing purposes just returns 'a'). The question is why this behaviour remains, even without the YY_INPUT macro.
0
Infinity08Commented:
How did you implement YY_INPUT ?


>> This feels sort of like an ugly hack

Chaining programs isn't an ugly hack imo ... it's basically how the Unix world works for a large part (piping the output of one small program to another - that way combining the small programs into something useful).
0
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

letharionAuthor Commented:
>>How did you implement YY_INPUT ?
#undef YY_INPUT
#define YY_INPUT(buf,result,max_size) \
   buf[0] = 'a'; \
   result = 1; \

>>Chaining programs isn't an ugly hack imo ... it's basically how the Unix world works for a large part >>(piping the output of one small program to another - that way combining the small programs into >>something useful).
That's ofc completely true, and I do make use of that functionality regularly myself with programs like cat,find, grep,tail and such.

At the moment I don't recall the details of it, but as I recall I was unable to find a standard way of piping from within a program that was guaranteed to work. Which did surprise me a little since piping is so common, but I didn't think to much about it.
And piping has worked quite well actually. But now I can't give up without cracking this problem ;)
0
Infinity08Commented:
You never return eof (YY_NULL), so the program will never end ... (also make sure that yywrap is implemented as you want it)

You also might want to enclose the macro in {}'s to make sure there are no unintended problems when expanding the macro.

I assume you also placed it in the right location ? (ie. in the definitions section of the lex file, between the %{ and %} at the beginning of the file)

If it still doesn't work, could you post your flex (and bison) input files ?
0
letharionAuthor Commented:
>You never return eof (YY_NULL), so the program will never end ... (also make sure that yywrap is implemented as you want it)

I'm sure that's important, but is that relevant in this case? The function never seems to execute at all? (So I wouldn't even be able to return YY_NULL even if wanted to)

>You also might want to enclose the macro in {}'s to make sure there are no unintended problems when expanding the macro.
Done

>I assume you also placed it in the right location ? (ie. in the definitions section of the lex file, between the %{ and %} at the beginning of the file)
That was it :) I had it in the y file, not lex. Now I got a looong stream of a's :)

Two questions remain:
1) I pass a float* to the start function in the y file, which is the point from where to read data.
How do I make is visible to the lexer? The code below, which is pretty much what I'm doing doesn't work because y is included after lex. If I include the other way around, I get:lex.yy.c(519): error: linkage specification is incompatible with previous "yywrap"
Related to your comment Inifinity on yywrap?

2) I have a float array that I want parsed by the lexer, best way to have YY_INPUT read floats and return as chars?
.y {
  float *inputData;
  start(float *a) { inputData = a; yyparse(); }
}

Open in new window

0
letharionAuthor Commented:
1) Wasn't any more difficult than to declare it within %{ }%.
0
letharionAuthor Commented:
I tried the below macro, placed at line 461. It gives me
lex.yy.c(1060): error: a block-scope function may only have extern storage class
lex.yy.c(1061): error: expected a ";"
lex.yy.c(1084): warning: parsing restarts here after previous syntax error
lex.yy.c(1085): warning: missing return statement at end of non-void function "yy_get_next_buffer"
lex.yy.c(347): error: function "yy_get_previous_state" was referenced but not defined

These errors are way way down in the code. Not sure what the top error means, nor what to do about it.
I tried not using YY_INPUT at all, but directly replacing it's use with the code. The resulting code is also below. That compiles, but my input is all wrong. Looking into that.
//Macro:
%{
float *progs;
int done = 0;
#include <stdio.h>
#include "PFtIF.tab.h"
 
#undef YY_INPUT
#define YY_INPUT(buf,result,max_size) { \
   if(done == 1) result = YY_NULL; \
   else { \
      std::ostringstream str; \
      int i = 0; \
      while(progs[i] != 99.0f && i < max_size) \
         str << progs[i++]; \
\
      for(int j = 0; j < i; j++) \
         buf[j] = str.str().c_str()[j]; \
\
      result = i; \
      if(progs[i] == 99.0f) \
         done = 1; \
   }
%}
 
 
 
//Direct replacement
/* Read in more data. */
if(done == 1) (yy_n_chars) = YY_NULL;
else {
  std::ostringstream str;
  int i = 0;
  while(progs[i] != 99.0f && i < num_to_read)
    str << progs[i++];
 
  for(int j = 0; j < i; j++)
    (&(yy_buffer_stack)[(yy_buffer_stack_top)]->yy_ch_buf[number_to_move])[j] = str.str().c_str()[j];
 
  (yy_n_chars) = i;
  if(progs[i] == 99.0f)
    done = 1;
}

Open in new window

0
letharionAuthor Commented:
The problem with the incorrect result was two-fold.
To begin with, I never inserted any whitespace, so the lexer got on single large "word".
Second, the number of read floats, doesn't not equal the number of read chars, and I never took that into account.

New code below produces nearly perfect result.
The parsing rules change slightly when the last input is detected, and while the parser no longer enters an eternal loop, but exits correctly, it doesn't detect "the end" the way I want any more.
      /* Read in more data. */
      if(done == 1) (yy_n_chars) = YY_NULL;
      else {
        std::ostringstream str;
        int i = 0, j = 0;
        while(progs[i] != 99.0f && str.str().size() < num_to_read)
          str << progs[i++] << " ";
 
        for(;j < str.str().size(); j++)
          (&(yy_buffer_stack)[(yy_buffer_stack_top)]->yy_ch_buf[number_to_move])[j] = str.str().c_str()[j];
 
        (yy_n_chars) = j;
        if(progs[i] == 99.0f)
          done = 1;
      }

Open in new window

0
Infinity08Commented:
>> 2) I have a float array that I want parsed by the lexer, best way to have YY_INPUT read floats and return as chars?

If you have a float array that needs to be passed to the parser one at a time, you don't need flex ... You can simply write the yylex function yourself, and have it return the next float from the array every time it's called.

This makes me wonder too ... Do you actually need a parser ? If it's just for processing an array of floats, can't you just do that in a loop ?

Could you describe with a bit more detail what you're actually doing ?
0
Infinity08Commented:
In your code in http:#24763660, you seem to be missing the closing } for the YY_INPUT macro.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
letharionAuthor Commented:
99 signals "end", and my loop was "off by one".
Correct loop below:

gdb is a god-send :D

So the only question that remains is why I can't make the function-like macro work.
do {
  str << progs[i] << " ";
} while (progs[i++] != 99.0f && str.str().size() < num_to_read);

Open in new window

0
letharionAuthor Commented:
Infinity08: You are absolutely right, I just noticed too

Wohoo, everything compiles and runs, time for some real world testing :D
0
letharionAuthor Commented:
Something still gets wrong when used in the "real world", but I think that's me inputing wrong data.
I also realised I have only gotten half work done, since I need to do something with the input too.
Should be easy now though. Probably do that on monday :)
0
Infinity08Commented:
Good luck ;)
0
letharionAuthor Commented:
I think it's time to close this.
You've been of great help as always Infinity :)
0
Infinity08Commented:
Glad I could be of assistance :)
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
C

From novice to tech pro — start learning today.