Integrating flex/bison with other code

>I may be misunderstanding your question, but that is the expected behavior.
I understand that this is the expected behaviour normally, but that is why I have replaced the function with my own. (Which for testing purposes just returns 'a'). The question is why this behaviour remains, even without the YY_INPUT macro.

How did you implement YY_INPUT ?


>> This feels sort of like an ugly hack

Chaining programs isn't an ugly hack imo ... it's basically how the Unix world works for a large part (piping the output of one small program to another - that way combining the small programs into something useful).

>>How did you implement YY_INPUT ?
#undef YY_INPUT
#define YY_INPUT(buf,result,max_size) \
   buf[0] = 'a'; \
   result = 1; \

>>Chaining programs isn't an ugly hack imo ... it's basically how the Unix world works for a large part >>(piping the output of one small program to another - that way combining the small programs into >>something useful).
That's ofc completely true, and I do make use of that functionality regularly myself with programs like cat,find, grep,tail and such.

At the moment I don't recall the details of it, but as I recall I was unable to find a standard way of piping from within a program that was guaranteed to work. Which did surprise me a little since piping is so common, but I didn't think to much about it.
And piping has worked quite well actually. But now I can't give up without cracking this problem ;)

You never return eof (YY_NULL), so the program will never end ... (also make sure that yywrap is implemented as you want it)

You also might want to enclose the macro in {}'s to make sure there are no unintended problems when expanding the macro.

I assume you also placed it in the right location ? (ie. in the definitions section of the lex file, between the %{ and %} at the beginning of the file)

If it still doesn't work, could you post your flex (and bison) input files ?

>You never return eof (YY_NULL), so the program will never end ... (also make sure that yywrap is implemented as you want it)

I'm sure that's important, but is that relevant in this case? The function never seems to execute at all? (So I wouldn't even be able to return YY_NULL even if wanted to)

>You also might want to enclose the macro in {}'s to make sure there are no unintended problems when expanding the macro.
Done

>I assume you also placed it in the right location ? (ie. in the definitions section of the lex file, between the %{ and %} at the beginning of the file)
That was it :) I had it in the y file, not lex. Now I got a looong stream of a's :)

Two questions remain:
1) I pass a float* to the start function in the y file, which is the point from where to read data.
How do I make is visible to the lexer? The code below, which is pretty much what I'm doing doesn't work because y is included after lex. If I include the other way around, I get:lex.yy.c(519): error: linkage specification is incompatible with previous "yywrap"
Related to your comment Inifinity on yywrap?

2) I have a float array that I want parsed by the lexer, best way to have YY_INPUT read floats and return as chars?

.y {
  float *inputData;
  start(float *a) { inputData = a; yyparse(); }
}

                                              

1:
2:
3:
4:

1) Wasn't any more difficult than to declare it within %{ }%.

I tried the below macro, placed at line 461. It gives me
lex.yy.c(1060): error: a block-scope function may only have extern storage class
lex.yy.c(1061): error: expected a ";"
lex.yy.c(1084): warning: parsing restarts here after previous syntax error
lex.yy.c(1085): warning: missing return statement at end of non-void function "yy_get_next_buffer"
lex.yy.c(347): error: function "yy_get_previous_state" was referenced but not defined

These errors are way way down in the code. Not sure what the top error means, nor what to do about it.
I tried not using YY_INPUT at all, but directly replacing it's use with the code. The resulting code is also below. That compiles, but my input is all wrong. Looking into that.

//Macro:
%{
float *progs;
int done = 0;
#include <stdio.h>
#include "PFtIF.tab.h"
 
#undef YY_INPUT
#define YY_INPUT(buf,result,max_size) { \
   if(done == 1) result = YY_NULL; \
   else { \
      std::ostringstream str; \
      int i = 0; \
      while(progs[i] != 99.0f && i < max_size) \
         str << progs[i++]; \
\
      for(int j = 0; j < i; j++) \
         buf[j] = str.str().c_str()[j]; \
\
      result = i; \
      if(progs[i] == 99.0f) \
         done = 1; \
   }
%}
 
 
 
//Direct replacement
/* Read in more data. */
if(done == 1) (yy_n_chars) = YY_NULL;
else {
  std::ostringstream str;
  int i = 0;
  while(progs[i] != 99.0f && i < num_to_read)
    str << progs[i++];
 
  for(int j = 0; j < i; j++)
    (&(yy_buffer_stack)[(yy_buffer_stack_top)]->yy_ch_buf[number_to_move])[j] = str.str().c_str()[j];
 
  (yy_n_chars) = i;
  if(progs[i] == 99.0f)
    done = 1;
}

                                              

1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:
41:
42:
43:

The problem with the incorrect result was two-fold.
To begin with, I never inserted any whitespace, so the lexer got on single large "word".
Second, the number of read floats, doesn't not equal the number of read chars, and I never took that into account.

New code below produces nearly perfect result.
The parsing rules change slightly when the last input is detected, and while the parser no longer enters an eternal loop, but exits correctly, it doesn't detect "the end" the way I want any more.

      /* Read in more data. */
      if(done == 1) (yy_n_chars) = YY_NULL;
      else {
        std::ostringstream str;
        int i = 0, j = 0;
        while(progs[i] != 99.0f && str.str().size() < num_to_read)
          str << progs[i++] << " ";
 
        for(;j < str.str().size(); j++)
          (&(yy_buffer_stack)[(yy_buffer_stack_top)]->yy_ch_buf[number_to_move])[j] = str.str().c_str()[j];
 
        (yy_n_chars) = j;
        if(progs[i] == 99.0f)
          done = 1;
      }
                                              

1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:

>> 2) I have a float array that I want parsed by the lexer, best way to have YY_INPUT read floats and return as chars?

If you have a float array that needs to be passed to the parser one at a time, you don't need flex ... You can simply write the yylex function yourself, and have it return the next float from the array every time it's called.

This makes me wonder too ... Do you actually need a parser ? If it's just for processing an array of floats, can't you just do that in a loop ?

Could you describe with a bit more detail what you're actually doing ?

99 signals "end", and my loop was "off by one".
Correct loop below:

gdb is a god-send :D

So the only question that remains is why I can't make the function-like macro work.

do {
  str << progs[i] << " ";
} while (progs[i++] != 99.0f && str.str().size() < num_to_read);

                                              

1:
2:
3:

Infinity08: You are absolutely right, I just noticed too

Wohoo, everything compiles and runs, time for some real world testing :D

Something still gets wrong when used in the "real world", but I think that's me inputing wrong data.
I also realised I have only gotten half work done, since I need to do something with the input too.
Should be easy now though. Probably do that on monday :)

Good luck ;)

I think it's time to close this.
You've been of great help as always Infinity :)