Solved

Linking fortran to c in a makefile

Posted on 2014-10-20
15
355 Views
Last Modified: 2014-11-21
I am getting compiling errors when making a program.  This is the advice from the developer:

This looks like BLACS is having problems getting symbols resolved.

This can happen if scalapack and BLACS are not built correctly (although I though this issue was taken care of in newer versions of mkl): scalapack is written in fortran and BLACS in C and one needs to construct a solver build which allows the two to cross boundaries between the language (the infamous underscore—double-underscore problem).  The default build of the solvers did not include the expanded symbol table that allow a Fortran code to call the mixed-language solver libraries.   You might look into that build to be sure that the symbol tables have the redundancy in them that allow this.  As I said, I though this had been “fixed” in the mkl that Intel ships, but perhaps not.

However, I added the following  option to my makefile and they still don't work.

 -assume 2underscores

and

-fsecond-underscore

Here is my errors:

a(BI_BlacsAbort.o): In function `BI_BlacsAbort':
../../../../scalapack/BLACS/SRC/MPI/INTERNAL/BI_BlacsAbort.c:(.text+0x26): undef                                                                                                 ined reference to `ompi_mpi_comm_world'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.                                                                                                 a(BI_Pack.o): In function `BI_Pack':
../../../../scalapack/BLACS/SRC/MPI/INTERNAL/BI_Pack.c:(.text+0x13): undefined r                                                                                                 eference to `ompi_mpi_byte'
../../../../scalapack/BLACS/SRC/MPI/INTERNAL/BI_Pack.c:(.text+0x6b): undefined r                                                                                                 eference to `ompi_mpi_packed'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.                                                                                                 a(BI_GetMpiGeType.o): In function `BI_GetMpiGeType':
../../../../scalapack/BLACS/SRC/MPI/INTERNAL/BI_GetMpiGeType.c:(.text+0x39): und                                                                                                 efined reference to `ompi_mpi_byte'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.                                                                                                 a(BI_GetMpiTrType.o): In function `BI_GetMpiTrType':
../../../../scalapack/BLACS/SRC/MPI/INTERNAL/BI_GetMpiTrType.c:(.text+0x9f): und                                                                                                 efined reference to `ompi_mpi_byte'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.                                                                                                 a(BI_MPI_C_to_f77_trans_comm.o): In function `BI_MPI_C_to_f77_trans_comm':
../../../../scalapack/BLACS/SRC/MPI/INTERNAL/BI_MPI_C_to_f77_trans_comm.c:(.text                                                                                                 +0x6): undefined reference to `MPI_Comm_c2f'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.                                                                                                 a(sys2blacs_handle_.oo): In function `Csys2blacs_handle':
../../../../scalapack/BLACS/SRC/MPI/sys2blacs_handle_.c:(.text+0x1e): undefined                                                                                                  reference to `ompi_mpi_comm_null'
../../../../scalapack/BLACS/SRC/MPI/sys2blacs_handle_.c:(.text+0xaf): undefined                                                                                                  reference to `ompi_mpi_comm_world'
../../../../scalapack/BLACS/SRC/MPI/sys2blacs_handle_.c:(.text+0x17d): undefined                                                                                                  reference to `ompi_mpi_comm_world'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.                                                                                                 a(BI_MPI_F77_to_c_trans_comm.o): In function `BI_MPI_F77_to_c_trans_comm':
../../../../scalapack/BLACS/SRC/MPI/INTERNAL/BI_MPI_F77_to_c_trans_comm.c:(.text                                                                                                 +0x6): undefined reference to `MPI_Comm_f2c'
make: *** [quest_265c.x] Error 1
0
Comment
Question by:ted_yu
  • 8
  • 7
15 Comments
 
LVL 61

Expert Comment

by:gheist
ID: 40392844
I think you have mismatch between header and object files
You need to either build both C and F with -fopenmp or without.
0
 

Author Comment

by:ted_yu
ID: 40392876
thanks, just to clarify, do you mean to build the software...or do you mean building the library that it is referencing?
0
 
LVL 61

Expert Comment

by:gheist
ID: 40392896
I think your fortran program expectc C library to be linked with openmp library (would be nice to hear what compiler you use and what OS...)
0
 

Author Comment

by:ted_yu
ID: 40420612
Sorry for the delay.  Here is the makefile.  The OS is redhat.

qvrs = 265c

FC = mpif77
FFLAGS = -O2 -xSSE3 -mcmodel medium -shared-intel

LFLAGS = -mcmodel medium -shared-intel
LIBS =

MKL_LIB = /home/rtorres/intel/mkl/lib/

MATH = -L${MKL_LIB} -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5

PMATH = -L${MKL_LIB} -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64

SUBSU = utl_mpi.o

MVRSTP=lcaomain_tp.o

SUBS=lcaosubs.o
TPSUBS=lcaosubs_tp.o

quest: quest_${qvrs}.x

help:
        @echo "Make SeqQuest program - openmpi parallel"
        @echo "Usage: make <platform>, only platform available: quest"
        @echo "For parallel builds, you will have to specify library locations in makefile"
        @echo ""

clean:
        rm -f *.o quest_${qvrs}.x

quest_${qvrs}.x : ${MVRSTP} ${SUBS} ${TPSUBS} ${SUBSU}
        $(FC) $(LFLAGS) -o $@  ${MVRSTP} ${SUBSU} ${TPSUBS} ${SUBS} ${PMATH} ${MATH}
0
 
LVL 61

Expert Comment

by:gheist
ID: 40421021
Just two builds disagree about their use of openmpi (I was mistaken about openmp)
So if you want to schedule program on multi-machine cluster - build all C parts with openmpi support, if not remove openmpi references from F99 makefile.
0
 

Author Comment

by:ted_yu
ID: 40421134
Unfortunately, I am not the author of this program, and I just need to get this thing compiled on my machine.  This program has been successfully compiled on other machines.  Here is the comment from the author of the program about my difficulties:

"This can happen if scalapack and BLACS are not built correctly (although I though this issue was taken care of in newer versions of mkl): scalapack is written in fortran and BLACS in C and one needs to construct a solver build which allows the two to cross boundaries between the language (the infamous underscore—double-underscore problem).  The default build of the solvers did not include the expanded symbol table that allow a Fortran code to call the mixed-language solver libraries.   You might look into that build to be sure that the symbol tables have the redundancy in them that allow this.  As I said, I though this had been “fixed” in the mkl that Intel ships, but perhaps not."

I cannot rewrite the code, so the best way to solve this, as I understand it is to include lines on my makefile such as:
 -assume 2underscores
-fsecond-underscore

If I can understand these things a little better, I can try to fix the problem by changing my makefile.

The mkl math library I am using is by Intel and is already installed in the cluster I am using.  I can try to recompile it as another option.

So, the only thing I can change at the moment are:
1)  the makefile
2)  recompiling the mkl library which contains blacs and scalpack differently.
0
 
LVL 61

Expert Comment

by:gheist
ID: 40421348
Change (i.e comment out):
# PMATH = -L${MKL_LIB} -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64
To (i.e add new line):
PMATH = -L${MKL_LIB} -lmkl_scalapack_lp64

You will lose ability to run product on multi-node cluster.
Ahouls you need to run it on multi-node cluster you need to build blacs with openmpi support, and then build your F99 program using original Makefile..
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 

Author Comment

by:ted_yu
ID: 40422172
Now I'm getting errors that scalapack needs references that are not there (from blacs)

/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_scalapack_lp64.so: undefined reference to `zgesd2d_'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_scalapack_lp64.so: undefined reference to `Cblacs_barrier'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_scalapack_lp64.so: undefined reference to `dtrbs2d_'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_scalapack_lp64.so: undefined reference to `ctrbr2d_'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_scalapack_lp64.so: undefined reference to `dcputime00_'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_scalapack_lp64.so: undefined reference to `Cigsum2d'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_scalapack_lp64.so: undefined reference to `igebs2d_'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_scalapack_lp64.so: undefined reference to `strrv2d_'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_scalapack_lp64.so: undefined reference to `blacs_pnum_'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_scalapack_lp64.so: undefined reference to `MKL_DOUBLE'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_scalapack_lp64.so: undefined reference to `MKL_MAXLOC'
make: *** [quest_265c.x] Error 1

I also need to run this on multi-nodes.  Do you have any idea what it means to look for the  following that was mentioned?  " The default build of the solvers did not include the expanded symbol table that allow a Fortran code to call the mixed-language solver libraries. "  What is this mixed-language solver library and where is it located?
0
 
LVL 61

Expert Comment

by:gheist
ID: 40422221
Take it ready on EL7+EPEL, no need to take long leaps to re-invent the wheel.
http://mirror.uv.es/mirror/fedora-epel/7/x86_64/repoview/blacs-openmpi-devel.html
0
 

Author Comment

by:ted_yu
ID: 40422383
OK.  I downloaded that file and put it in the cluster.  I may not have changed the Makefile correctly, because I'm still getting an error.  Here are my changes to the makefile.

BLACS_LIB = /shares/teamyu/usr/lib64/openmpi/lib/

PMATH = -L${MKL_LIB} -lmkl_scalapack_lp64 -L${BLACS_LIB} -lmkl_blacs_openmpi_lp64

This is going back to the original error, so it's still trying to use BLACS from the mkl library.:
../../../../scalapack/BLACS/SRC/MPI/sys2blacs_handle_.c:(.text+0x17d): undefined                                                                                                              reference to `ompi_mpi_comm_world'
/usr/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.                                                                                                             a(BI_MPI_F77_to_c_trans_comm.o): In function `BI_MPI_F77_to_c_trans_comm':
../../../../scalapack/BLACS/SRC/MPI/INTERNAL/BI_MPI_F77_to_c_trans_comm.c:(.text                                                                                                             +0x6): undefined reference to `MPI_Comm_f2c'
make: *** [quest_265c.x] Error 1
0
 
LVL 61

Expert Comment

by:gheist
ID: 40422756
I was trying to talk you into getting blacs+scalapack as installable RPMs (you know desktop virtualisation ant the stuff for testing)
EPEL RPMs are build with MPI support, so RHEL7+EPEL will sustain your build.
btw no need to target SSE3 - it is so past, --march-native gives bigger bang for the buck.
0
 
LVL 61

Expert Comment

by:gheist
ID: 40422760
Aw pain - you use ICC. You must build both components with same ICC. What I am suggesting is to make POC build with GCC.
0
 

Author Comment

by:ted_yu
ID: 40422988
I was advised to use Intel mkl because it is the fastest by a factor of two.  I would like to use it if possible.  Any chance to fix this with the underscore issue?  

 -assume 2underscores
-fsecond-underscore

Is it possible scalapack is not calling blacs correctly and it is just an easy fix of the settings?
0
 
LVL 61

Accepted Solution

by:
gheist earned 500 total points
ID: 40423339
BLACS is not build with openmpi support.
SSE3 is 10 years old. Are you certain GCC does not generate that?
0
 

Author Comment

by:ted_yu
ID: 40458709
OK, I've given up on using Intel MKL for this build and am using a different library.  Thanks for your help.
0

Featured Post

Threat Intelligence Starter Resources

Integrating threat intelligence can be challenging, and not all companies are ready. These resources can help you build awareness and prepare for defense.

Join & Write a Comment

Suggested Solutions

A list of useful business intelligence software.
In our personal lives, we have well-designed consumer apps to delight us and make even the most complex transactions simple. Many enterprise applications, however, are a bit behind the times. For an enterprise app to be successful in today's tech wo…
This video shows how use content aware, what it’s used for, and when to use it over other tools.
In a previous video, we went over how to export a DynamoDB table into Amazon S3.  In this video, we show how to load the export from S3 into a DynamoDB table.

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

21 Experts available now in Live!

Get 1:1 Help Now