Linking fortran to c in a makefile

I am getting compiling errors when making a program.  This is the advice from the developer:

This looks like BLACS is having problems getting symbols resolved.

This can happen if scalapack and BLACS are not built correctly (although I though this issue was taken care of in newer versions of mkl): scalapack is written in fortran and BLACS in C and one needs to construct a solver build which allows the two to cross boundaries between the language (the infamous underscore—double-underscore problem).  The default build of the solvers did not include the expanded symbol table that allow a Fortran code to call the mixed-language solver libraries.   You might look into that build to be sure that the symbol tables have the redundancy in them that allow this.  As I said, I though this had been “fixed” in the mkl that Intel ships, but perhaps not.

However, I added the following  option to my makefile and they still don't work.

 -assume 2underscores

and

-fsecond-underscore

Here is my errors:

a(BI_BlacsAbort.o): In function `BI_BlacsAbort':
../../../../scalapack/BLACS/SRC/MPI/INTERNAL/BI_BlacsAbort.c:(.text+0x26): undef                                                                                                 ined reference to `ompi_mpi_comm_world'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.                                                                                                 a(BI_Pack.o): In function `BI_Pack':
../../../../scalapack/BLACS/SRC/MPI/INTERNAL/BI_Pack.c:(.text+0x13): undefined r                                                                                                 eference to `ompi_mpi_byte'
../../../../scalapack/BLACS/SRC/MPI/INTERNAL/BI_Pack.c:(.text+0x6b): undefined r                                                                                                 eference to `ompi_mpi_packed'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.                                                                                                 a(BI_GetMpiGeType.o): In function `BI_GetMpiGeType':
../../../../scalapack/BLACS/SRC/MPI/INTERNAL/BI_GetMpiGeType.c:(.text+0x39): und                                                                                                 efined reference to `ompi_mpi_byte'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.                                                                                                 a(BI_GetMpiTrType.o): In function `BI_GetMpiTrType':
../../../../scalapack/BLACS/SRC/MPI/INTERNAL/BI_GetMpiTrType.c:(.text+0x9f): und                                                                                                 efined reference to `ompi_mpi_byte'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.                                                                                                 a(BI_MPI_C_to_f77_trans_comm.o): In function `BI_MPI_C_to_f77_trans_comm':
../../../../scalapack/BLACS/SRC/MPI/INTERNAL/BI_MPI_C_to_f77_trans_comm.c:(.text                                                                                                 +0x6): undefined reference to `MPI_Comm_c2f'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.                                                                                                 a(sys2blacs_handle_.oo): In function `Csys2blacs_handle':
../../../../scalapack/BLACS/SRC/MPI/sys2blacs_handle_.c:(.text+0x1e): undefined                                                                                                  reference to `ompi_mpi_comm_null'
../../../../scalapack/BLACS/SRC/MPI/sys2blacs_handle_.c:(.text+0xaf): undefined                                                                                                  reference to `ompi_mpi_comm_world'
../../../../scalapack/BLACS/SRC/MPI/sys2blacs_handle_.c:(.text+0x17d): undefined                                                                                                  reference to `ompi_mpi_comm_world'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.                                                                                                 a(BI_MPI_F77_to_c_trans_comm.o): In function `BI_MPI_F77_to_c_trans_comm':
../../../../scalapack/BLACS/SRC/MPI/INTERNAL/BI_MPI_F77_to_c_trans_comm.c:(.text                                                                                                 +0x6): undefined reference to `MPI_Comm_f2c'
make: *** [quest_265c.x] Error 1
ted_yuAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

gheistCommented:
I think you have mismatch between header and object files
You need to either build both C and F with -fopenmp or without.
0
ted_yuAuthor Commented:
thanks, just to clarify, do you mean to build the software...or do you mean building the library that it is referencing?
0
gheistCommented:
I think your fortran program expectc C library to be linked with openmp library (would be nice to hear what compiler you use and what OS...)
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

ted_yuAuthor Commented:
Sorry for the delay.  Here is the makefile.  The OS is redhat.

qvrs = 265c

FC = mpif77
FFLAGS = -O2 -xSSE3 -mcmodel medium -shared-intel

LFLAGS = -mcmodel medium -shared-intel
LIBS =

MKL_LIB = /home/rtorres/intel/mkl/lib/

MATH = -L${MKL_LIB} -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5

PMATH = -L${MKL_LIB} -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64

SUBSU = utl_mpi.o

MVRSTP=lcaomain_tp.o

SUBS=lcaosubs.o
TPSUBS=lcaosubs_tp.o

quest: quest_${qvrs}.x

help:
        @echo "Make SeqQuest program - openmpi parallel"
        @echo "Usage: make <platform>, only platform available: quest"
        @echo "For parallel builds, you will have to specify library locations in makefile"
        @echo ""

clean:
        rm -f *.o quest_${qvrs}.x

quest_${qvrs}.x : ${MVRSTP} ${SUBS} ${TPSUBS} ${SUBSU}
        $(FC) $(LFLAGS) -o $@  ${MVRSTP} ${SUBSU} ${TPSUBS} ${SUBS} ${PMATH} ${MATH}
0
gheistCommented:
Just two builds disagree about their use of openmpi (I was mistaken about openmp)
So if you want to schedule program on multi-machine cluster - build all C parts with openmpi support, if not remove openmpi references from F99 makefile.
0
ted_yuAuthor Commented:
Unfortunately, I am not the author of this program, and I just need to get this thing compiled on my machine.  This program has been successfully compiled on other machines.  Here is the comment from the author of the program about my difficulties:

"This can happen if scalapack and BLACS are not built correctly (although I though this issue was taken care of in newer versions of mkl): scalapack is written in fortran and BLACS in C and one needs to construct a solver build which allows the two to cross boundaries between the language (the infamous underscore—double-underscore problem).  The default build of the solvers did not include the expanded symbol table that allow a Fortran code to call the mixed-language solver libraries.   You might look into that build to be sure that the symbol tables have the redundancy in them that allow this.  As I said, I though this had been “fixed” in the mkl that Intel ships, but perhaps not."

I cannot rewrite the code, so the best way to solve this, as I understand it is to include lines on my makefile such as:
 -assume 2underscores
-fsecond-underscore

If I can understand these things a little better, I can try to fix the problem by changing my makefile.

The mkl math library I am using is by Intel and is already installed in the cluster I am using.  I can try to recompile it as another option.

So, the only thing I can change at the moment are:
1)  the makefile
2)  recompiling the mkl library which contains blacs and scalpack differently.
0
gheistCommented:
Change (i.e comment out):
# PMATH = -L${MKL_LIB} -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64
To (i.e add new line):
PMATH = -L${MKL_LIB} -lmkl_scalapack_lp64

You will lose ability to run product on multi-node cluster.
Ahouls you need to run it on multi-node cluster you need to build blacs with openmpi support, and then build your F99 program using original Makefile..
0
ted_yuAuthor Commented:
Now I'm getting errors that scalapack needs references that are not there (from blacs)

/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_scalapack_lp64.so: undefined reference to `zgesd2d_'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_scalapack_lp64.so: undefined reference to `Cblacs_barrier'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_scalapack_lp64.so: undefined reference to `dtrbs2d_'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_scalapack_lp64.so: undefined reference to `ctrbr2d_'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_scalapack_lp64.so: undefined reference to `dcputime00_'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_scalapack_lp64.so: undefined reference to `Cigsum2d'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_scalapack_lp64.so: undefined reference to `igebs2d_'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_scalapack_lp64.so: undefined reference to `strrv2d_'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_scalapack_lp64.so: undefined reference to `blacs_pnum_'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_scalapack_lp64.so: undefined reference to `MKL_DOUBLE'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_scalapack_lp64.so: undefined reference to `MKL_MAXLOC'
make: *** [quest_265c.x] Error 1

I also need to run this on multi-nodes.  Do you have any idea what it means to look for the  following that was mentioned?  " The default build of the solvers did not include the expanded symbol table that allow a Fortran code to call the mixed-language solver libraries. "  What is this mixed-language solver library and where is it located?
0
gheistCommented:
Take it ready on EL7+EPEL, no need to take long leaps to re-invent the wheel.
http://mirror.uv.es/mirror/fedora-epel/7/x86_64/repoview/blacs-openmpi-devel.html
0
ted_yuAuthor Commented:
OK.  I downloaded that file and put it in the cluster.  I may not have changed the Makefile correctly, because I'm still getting an error.  Here are my changes to the makefile.

BLACS_LIB = /shares/teamyu/usr/lib64/openmpi/lib/

PMATH = -L${MKL_LIB} -lmkl_scalapack_lp64 -L${BLACS_LIB} -lmkl_blacs_openmpi_lp64

This is going back to the original error, so it's still trying to use BLACS from the mkl library.:
../../../../scalapack/BLACS/SRC/MPI/sys2blacs_handle_.c:(.text+0x17d): undefined                                                                                                              reference to `ompi_mpi_comm_world'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.                                                                                                             a(BI_MPI_F77_to_c_trans_comm.o): In function `BI_MPI_F77_to_c_trans_comm':
../../../../scalapack/BLACS/SRC/MPI/INTERNAL/BI_MPI_F77_to_c_trans_comm.c:(.text                                                                                                             +0x6): undefined reference to `MPI_Comm_f2c'
make: *** [quest_265c.x] Error 1
0
gheistCommented:
I was trying to talk you into getting blacs+scalapack as installable RPMs (you know desktop virtualisation ant the stuff for testing)
EPEL RPMs are build with MPI support, so RHEL7+EPEL will sustain your build.
btw no need to target SSE3 - it is so past, --march-native gives bigger bang for the buck.
0
gheistCommented:
Aw pain - you use ICC. You must build both components with same ICC. What I am suggesting is to make POC build with GCC.
0
ted_yuAuthor Commented:
I was advised to use Intel mkl because it is the fastest by a factor of two.  I would like to use it if possible.  Any chance to fix this with the underscore issue?  

 -assume 2underscores
-fsecond-underscore

Is it possible scalapack is not calling blacs correctly and it is just an easy fix of the settings?
0
gheistCommented:
BLACS is not build with openmpi support.
SSE3 is 10 years old. Are you certain GCC does not generate that?
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
ted_yuAuthor Commented:
OK, I've given up on using Intel MKL for this build and am using a different library.  Thanks for your help.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Software

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.