Linux Ubuntu: Remove MACROS in C-Code by modifying makefile

The macro's are there for a reason...... they make a LOT of stuff more readable.  So please explain better why you would want to do this. I'll explain the case for macro's below.  also -nostdinc is a gcc specific option, other compilers will have other options.

Then again yes you can get rid of all macro's.  The compiler has a phase called pre-procesing (before actual compilation).

The original compiler had NO macro's, the preprocessor resolved the macro's, modern compiler integrated macro expansion in the compiler for efficiency and more accurate error/warning messages that refer to the actual source, 

All compilers have a flag that allows for producing the "intermediate" result after only preprocessing as a c-source.  

Now the reason for macro's.., there are a lot of compilers and in a several cases there are differences in how their runtime (binaries) libraries handle basic functions.  Some compilers have built ins for comparing extended integers (comparing int 64bit or 32bit on 16bit architectures) others have some RTL code to do the same.   With macro's those implementation differences are hidden. 

Lots of stdio were implemented in a combination of macro's & binary code where the routines were mostly code because those were written in assembly by hand for efficiency, when compilers became a lot better more code ended up in macro's as a compiler then can make a better decision on creating the code as a (lightweight internal) subroutine vs. in-lining the code. This optimization is most prevalent in current hyper-scalar CPU's.  Where the trade of is mostly is this done 1-n times use inline (copies of code) vs.  n+1 - infinite (use subroutine). 

Also maco's allow for differences in platforms and versions of platforms. One can have a macro that hides the difference between say library v X as library v (X+1). Where a new library f.e. has a different arrangement of arguments (and new names of internal routines) where the old API is emulated using macro's for efficiency.

be advised a LOT of the macro's are simply constants (#define EOF (-1) ) ... where a test like if ( read(.....) == EOF )  immediately makes it clear you are testing for end of file  and if (read(....) == -1 ) could be anything... and if a stdio library uses -2 as EOF the first cod would continue to work as advertised and the 2nd example would not.

Preprocessing the code is THE way to get rid of all macro's...., and to create code that is portable to the current version of your system at that point in time. Non portable code that cannot be re-used.

A better way would be to actually USE macro's like they are supposed to do to make your code work on other versions....

using conditionals.. in the code yo

u can make the program behave in the original context as well as a new one...

ok-code1

code-depends on gcc

ok-code2

could be replaced with

#include "configure.h"

ok-code1

#ifdef MY_COMPILER

code-depends on my-compiler

#else

code-depends on gcc

#endif

ok-code2

And add a test to the ./configure sources to test for your compiler.

(The configure scripts btw. depend on the macro's to handle all difference in the fashion i described using conditional pre-processing code (aka a macro).

Macro's are meant to make things portable... you can use the pre-processor of another system to tie it to that environment.

I you mean to port the code from linux/unix => windows please do the porting the correct way, then you can send the changes to the "upstream" developer and hope they are acceptable to them, so your changes gets into future versions, making later reuse a lot easier.

I understand the use of MACROS and have them in the ways you described and in other ways as well.

I used liquid-dsp project as an example. I have a number of other projects that have no documentation and lots of macros. The macros are great for being able to handle multiple platforms, but not so great if they do not accomodate a specific platform.

I want  to be able to read the C-code in order to :

1. understand and document the algorithms without struggling to find out what the macro subtitution represents
2. be able to port the code to a platform not easily covered by the current build.

I created for one file, equalizer_cccf.c, a folder testRemoveMacros, where I put the minimum number of files in it to needed to remove macros.

cpp equalizer_cccf.c   equalizer_cccf.cpp-only    # hard to read with the markers
gcc -E equalizer_cccf.c  > equalizer_cccf.cpp-E   # hard to read with the markers
gcc -E -P equalizer_cccf.c  > equalizer_cccf.cpp-E-P # Easier to read but what is _Complex, liquid_error_fl, etc?
gcc -E -P -nostdinc equalizer_cccf.c  > equalizer_cccf.cpp-E-P-nostdinc  # Easiest to read, _Complex is replaced by complex, but missing function definitions such as liquid_error_fl.

Select allOpen in new window

complex is defined, no doubt, in complex.h (mentioned in one of the output .cpp files), which I hadn't included because there was no error message when performing the above four attempts.

1. Is there a way to get an easy to read .c file that defines things nicely. Header files and those proto.c files that are included would be also nice to remove macros.

2. How can I change the makefile to produce results like above? I tried using combinations o the -E -P -nostdinc options, but make thought they were part of the -I include path due to a mistake on my part.

Attached is the testRemoveMacros.zip file that includes a makefile created in Ubuntu.

I put the four output .cpp files in a sub-folder, Output.

testRemoveMacros.zip

_Complex most probably is a runtime structure (possibly describing a complex number having a real and an imaginary flouting point part)

Convention in C.....    xxxxx is a user level type or name   _xxxxx is the internal name of the RTL for something  __xxxxx is an OS reference....

You use read()..., the rtl may make that an internal one _read()  (not using any buffers f.e.)   which may call upon the os __read()

So the user level complex (which may f.e. refer to a single or double precision float using complex number) to the more explicit type of the RTL: _Complex

complex (due to the -nostdinc) is left "original" ... in stead of being replaced by some complex macro into the internal _Complex.

After running the preprocessor you are left with pure C code..., with a lot of details that are normally hidden, and possibly only documented in "internal" documents or worse peoples heads.

Any comments in the sources & headers also get removed in the same pass  making stuff harder to read and comprehend.

The markers you got rid of  are the markers needed for later passes of the compilation to generate reasonable accurate error messages with references to the correct lines.

You will also see that formalized declarations from header files are not based on default assumptions and declare everything making the source actually harder to read for humans.

The headers have a lot of comments around.. so starting there to get a grasp may make things beter understandable.

First thing if you are porting is to see where things go wrong and then focus on that aspect of the source, of choose some item you known to be simple to fix and fix that step by step.

Al the while reading all relevant parts of it. in headers etc.  been there, done that.

First impression from looking at the liquid.h file..., it implements it's own io-language on top of other stuff.  So you need to get to the details of that,

What does _pop do, what is _print etc. etc.

I hope you are aware there is a site about liquid-dsp with docs and datastructure description?

 https://liquidsdr.org/doc/datastructures/

its API:

 https://liquidsdr.org/api/

So i am not sure you claim there is no documentation around.

BTW be aware of the license... any way you want to use this is ok by the author just ensure  the reference to the original author & library must be kept. 

 (otherwise it becomes code theft, copyright infringement).

Using a preprocessed source does not dissolve the X11/MIT license claim, it may make it worse due to comment removal done in preprocessing

You definitely want to use gcc -E -C to preserve comments. Then you can insert comments to help find your way around the preprocessed code. Discard everything up to but keeping the first line of code after the last #include

You might like to also look at gcc options -save-temps and -dumpdir

I had to check what claim you were referring to.

>> So i am not sure you claim there is no documentation around.

And here is my claim:

I used liquid-dsp project as an example. I have a number of other projects that have no documentation and lots of macros.

My bad about the claim on documentation, i misread that part (english is not my native language), to include liquid-dsp but it refers to the other tools.

The reason for the inclusion of .c files might very well be to exploit the ability of the C compiler to optimize better by combining functions in one compilation.

Otherwise i see no reason to do it. 

Any help available on modifying the makefile to produce the .cpp files?

Hi Paul,
were my last 2 comments not any help with the makefile?

Hi Duncan,

Since I was packing and rushing Sunday night to drive to work for the week, I didn't have time to try them out at home. At work, I am waiting for a CD of my program to be put into my office. (Given post-holiday vacations and catch up, that could take a week. This weekend I'll certainly be able to try your suggestions.

To be clear, I have already tried to add the -E -P options in the makefile, but failed to make it work. The original makefile that builds fine in Ubunti is in the attached zip. My makefile syntax or use of the FLAGS is messing me up.

At work in the MINGW64/msys environment I will look for a makefile and try modifying it if you can show me the change(s) to the makefile line(s) to get the gcc -E -C etc options to work. When I tried, the options got mangled with other options producing nonsense errors.

>> CFLAGS='-P -CC -dumpdir temps/ -save-temps' ./configure 

Great! If it worked for you, I'm sure it will work for me. Looking forward to trying it out.

CFLAGS='-P -CC -dumpdir temps/ -save-temps' ./configure
[copious output] N.B. Add your options to ./configure

Select allOpen in new window

What are the single quotes for in the CFLAGS line?

What is the ./configure at the end of that line. I already did a .configure after the .bootstrap.sh line.

Not sure what options I should add to ./configure or how to do it.

Edited one line in the makefile:

# flags
INCLUDE_CFLAGS   = $(addprefix -I,$(include_dirs))
COVERAGE_FLAGS  =  # dynamic library linker needs separate flag
CONFIG_CFLAGS   = -g -O2  -mavx2 ${COVERAGE_FLAGS}
CPPFLAGS   =  $(INCLUDE_CFLAGS)
CFLAGS='-P -CC -dumpdir temps/ -save-temps' ./configure
CXXFLAGS   = $(CFLAGS) -std=c++14 -Wno-return-type-c-linkage
LDFLAGS      = 
LIBS      = -lm -lc 
PATHSEP      = /

Select allOpen in new window

Did following cmd lines and got no .i files due to the error.

$ make distclean 
$ ./bootstrap.sh 
$ ./configure
$ rm -rf temps; mkdir temp
$ make
gcc '-P -CC -dumpdir temps/ -save-temps' ./configure -I. -Iinclude  -c -o src/libliquid.o src/libliquid.c
gcc: error: unrecognized command line option ‘-P -CC -dumpdir temps/ -save-temps’
make: *** [<builtin>: src/libliquid.o] Error 1

Select allOpen in new window

As an experiment, I removed the single quotes int the CLFAGS line. Then I got 8 .i and 8 .s files. (The temp folder was empty.)

dotprod_cccf.avx.i:13848:1: error: inlining failed in call to always_inline ‘_mm256_setzero_ps’: target specific option mismatch
13848 | _mm256_setzero_ps (void)
      | ^~~~~~~~~~~~~~~~~
dotprod_cccf.avx.i:43826:18: note: called from here
43826 |     __m256 sum = _mm256_setzero_ps(); // load zeros into sum register
      |                  ^~~~~~~~~~~~~~~~~~~
dotprod_cccf.avx.i:13307:1: error: inlining failed in call to always_inline ‘_mm256_add_ps’: target specific option mismatch
13307 | _mm256_add_ps (__m256 __A, __m256 __B)
      | ^~~~~~~~~~~~~

Select allOpen in new window

You should not have done ./configure. Sorry my post was in error - there should have been a timestamp prompt before the CFLAGS ... ./configure line. Then you just need to issue the lines with a timestamp prompt minus the prompt. N.B I have updated that post so it is right now.

CFLAGS='-P -CC -dumpdir temps/ -save-temps' ./configure

Select allOpen in new window

What this does is temporarily export CFLAGS='-P -CC -dumpdir temps/ -save-temps' into the environment of the following command ./configure.
There is absolutely no need to edit anything, so don't do that.

I got .i and .s files. Thanks. Did your files end up in temps/? My temps/ is empty.

Is it possible for the .i files to be nearer to the source .[ch] code? Reason I ask is that I will also be doing a make check, and a make examples, which are in different folders.

Yes temp files were always in temps. You need the trailing / in CFLAGS. I have no idea why your files don't end up there.
In reply to a previous point you raised, ./configure --help will tell you what arguments you can put after ./configure but for your purposes you don't need any.
Don't understand what is LOC and what did you change?

$ rm -rf temps; mkdir temps

Select allOpen in new window

looks to me the same as what I posted except shorter bash prompt

>> You definitely want to use gcc -E -C to preserve comments. 
When I add a -E to the CFLAGS line, i get 

configure: error: C compiler cannot create executables

Select allOpen in new window

How do I find out what the problem is and correct it?

>> Don't understand what is LOC and what did you change? 

I had edited the comment and removed that line. (LOC is Line of Code.)

>> You need the trailing / in CFLAGS.  

I assume you mean the / after temps as shown here:

$ CFLAGS='-P -dumpdir temps/ -save-temps ' ./configure

Select allOpen in new window

The makefile flags section looks like this:

# flags
INCLUDE_CFLAGS   = $(addprefix -I,$(include_dirs))
COVERAGE_FLAGS  =  # dynamic library linker needs separate flag
CONFIG_CFLAGS   = -P -dumpdir temps/ -save-temps   -mavx2 ${COVERAGE_FLAGS}
CPPFLAGS   =  $(INCLUDE_CFLAGS)
CFLAGS      = $(CONFIG_CFLAGS) -Wall -fPIC -Wno-deprecated -Wno-deprecated-declarations
CXXFLAGS   = $(CFLAGS) -std=c++14 -Wno-return-type-c-linkage
LDFLAGS      = 
LIBS      = -lm -lc 
PATHSEP      = /

Select allOpen in new window

Then when I run make, I get this:

$ make
gcc -P -dumpdir temps/   -mavx2  -Wall -fPIC -Wno-deprecated -Wno-deprecated-declarations -I. -Iinclude  -c -o src/libliquid.o src/libliquid.c
gcc -P -dumpdir temps/   -mavx2  -Wall -fPIC -Wno-deprecated -Wno-deprecated-declarations -I. -Iinclude  -c -o src/agc/src/agc_crcf.o src/agc/src/agc_crcf.c
gcc -P -dumpdir temps/   -mavx2  -Wall -fPIC -Wno-deprecated -Wno-deprecated-declarations -I. -Iinclude  -c -o src/agc/src/agc_rrrf.o src/agc/src/agc_rrrf.c
gcc -P -dumpdir temps/   -mavx2  -Wall -fPIC -Wno-deprecated -Wno-deprecated-declarations -I. -Iinclude  -c -o src/audio/src/cvsd.o src/audio/src/cvsd.c
... etc.

Select allOpen in new window

I get .i and .s files in the same folder as the makefile, and the temps folder is empty.

>> You definitely want to use gcc -E -C to preserve comments. 
When I add a -E to the CFLAGS line, I get error:

configure: error: C compiler cannot create executables

Select allOpen in new window

I have also tried having below, but then I get error:

$ CFLAGS='-P -CC -nostdinc -dumpdir temps/ -save-temps' ./configure
configure: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.

Select allOpen in new window

How do I find out where these errors originate from and correct them?

I am thinking that there must be some way to get the -E and -stdnoinc option to work in the makefile (or generated makefile) since I was able to manually use them (as shown in a previous post (repeated here for convenience).

www.experts-exchange.com/dashboard/#/questions/29252674?anchorAnswerId=43496569

cpp equalizer_cccf.c   equalizer_cccf.cpp-only    # hard to read with the markers
gcc -E equalizer_cccf.c  > equalizer_cccf.cpp-E   # hard to read with the markers
gcc -E -P equalizer_cccf.c  > equalizer_cccf.cpp-E-P # Easier to read but no def for liquid_error_fl, etc?
gcc -E -P -nostdinc equalizer_cccf.c  > equalizer_cccf.cpp-E-P-nostdinc  # Easiest to read but missing function definitions such as liquid_error_fl

Select allOpen in new window

You don't need -E with save-temps

OK, won't use -E. Thanks.

How do I get the -nostdinc option? I tried, but then I get error:

$ CFLAGS='-P -CC -nostdinc -dumpdir temps/ -save-temps' ./configure
configure: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.

Select allOpen in new window

You mustn't use -nostdinc with -save-temps. It's only useful with -E, which stops the compilation (no .o file is produced). You could insert a comment after the last #include like maybe

/* ----%< CUT HERE >%---- */

Select allOpen in new window

then edit the .i file to delete up to that line

Thanks for all your help. I found the GCC options to be somewhat confusing because of their interactions with other options. Would need to experiment a lot to get a good feel for them.

>> You could insert a comment after the last #include .

Was tired, but now realize this would require editing too many .c files.

I think you're suggesting that this cannot be done by editing the makefile or using the cmd line: $ CFLAGS='-P -CC ...

Editing the files is the easy bit. It's what to do with the .i files afterwards...
Will post more later

OK here's how I edited the 897 .c files in liquid-dsp:
The q editor is available at https://sourceforge.net/projects/q-editor/
I wrote the following q macros

05:01:42$ cat insert.qm 
# Macros for when developing 'insert.qm'
n202 ^NC^NS^N\q insert.qm^J^NU
n220 ^NC^NS^N\u insert.qm^J^NU

#n2: Insert "Cut Here" comment after last #include
#n3: delete all lines up to "Cut Here" comment from .i files

n062 ^NC^NS^N\fm +g -d -t -l +s^J^ND^N^<1501>^NU
n1501 ^ND^N^<1502>^NS^NLq^J^N^@m-1^J^Z^T/* ----%< CUT HERE >%---- */^Js^Jq^J^N^@
n1502 l ^*#[[:space:]]*include^J^NC^NU^NI^ND^N^<1503>^NU
#n1503 ^[l ^*#[[:space:]]*include^J^NC^N^<1504>^N^@
n1503 ^[l ^*#[[:space:]]*include^J^NC^NU^N^@

#n1504: we are past all the includes, but ae there any endifs?
# NO LONGER USED - there are far too many #if 0 etc.
n1504 ^[l ^*#[[:space:]]*endif^J^NC^NU^N^@


n063 ^NC^NS^N\^ND^N^<1505>^NU
n1505 ^ND^N^<1506>q^J^N^@
n1506 l 'CUT HERE'^J^NC^NU^[d1 - -1^Js^J^NU

Select allOpen in new window

Although q  is essentially an interactive editor, you can run it non-interactively (offline, -o) and I did

make clean
rm -rf temps; mkdir temps
find . -name *.c|xargs q -oniu,insert.qm^J^N2
make -j$(($(nproc)+1))
rm temps/*.s
q -oniu,insert.qm^J^N3 temps/*.i

Select allOpen in new window

That's it.

The results are not perfect. In some files, the last #insert is conditional and not obeyed: the "CUT HERE" comment is then lost. If this affects a file that interests you, move the comment and make.
And that's the question. Why did you start on this in the first place? Can you give an example of a source line for which you'd like to see the expansion?
Also some .c files #include other .c files. The resultant .i flies have multiple "CUT HERE" comments and expanded header files. Hopefully they're not files you wanted to examine

Never heard of the q editor. Will take a look when I get home. Looks like your non-interactive script is very impressive. Your q macros looked a bit tricky. Looks like it would take me many trials and tweaks to get it right.

>> Why did you start on this in the first place?

Many years ago on a mature embedded c project that kept growing over the decades as new platforms and compilers were used, I used an option to show the pre-processor output. It helped a lot in tracking down data types and crazy written (maybe brilliant?) macro functions filled with cryptic macros including macro functions which were also crazy and unintelligible.

The output included the C standard library headers. Even a 50 line file would churn out what seemed to be 100's of pages. Took a long time to even load into an editor on an old solaris workstation.

I may have misunderstood the purpose of the -nostdinc option. I was hoping that only user defined headers would be included and stdio.h and other std library headers would be excluded when creating the .i files.