Use Cmake OBJECT libraries to create openfastlib and add option to use downloaded reference lapack and blas sources

[reos-rcrozier]

Feature or improvement description

Updates to the build system to facilitate building on Linux, particularly with Simulink support. OpenFAST cmake files have been modified to properly make use of the cmake OBJECT library type to create the openfast library. In addition, the option to build a local version of the official LAPACK and BLAS sources has been added through the use of the Cmake command ExternalProject_Add. A new boolean variable has been added, USE_LOCAL_STATIC_LAPACK. If set to ON, the latest LAPACK and BLAS soures are downloaded and build locally. The install command always installs these locally build libraries in ${CMAKE_SOURCE_DIR}/install/lib, i.e. regardless of the value of CMAKE_INSTALL_PREFIX to avoid user's accidentally overwriting system LAPACK and BLAS libraries. The purpose of this option is primarily to provide static LAPACK libraries to link to the FAST_Sfunc mex file, to avoid load time issues with shared libraries, but it will also allow testing with the latest LAPACK.

Related issue, if one exists

#924, #556, #482, #926, probably others

Impacted areas of the software

Only cmake files and create_FAST_SFunc.m

Additional supporting information

When building on Linux with Simulink support, there is an issue at load time with libraries being shipped with Matlab incorrectly being loaded when loading openfast as a shared library. These include libraries such as lapack and blas. A solution might be to build the openfast library as a static library, however, the current method of doing that on Linux is broken because the openfast library is created by linking together a bunch of other libraries created during the build. This is ok for shared libraries, but is incorrect on linux for static libraries. On Linux, the linker when creating a static library is ar. ar is just an archiver and doesn't really do a lot more than take a bunch of object files and stuff them in the static library (at least by default with Cmake). ar cannot link an existing library to the static library, one would have to extract the object files from the static library and then use ar to add them. With the current build system, when you try to link to the created static library on Linux, you get undefined references, because ar hasn't actually linked any of the sub-libraries like aerodyn etc.

Cmake have provided a mechanism to deal with this, which is OBJECT libraries. OBJECT libraries allow you to break up libraries into smaller modules as in OpenFAST, but link them correctly at the higher level. With this PR we have modified OpenFAST to use object libraries throughout.

This solves the issue with undefined references due to the missing OpenFAST modules, but, it also means that when later linking to libopenfastlib.a one must remember to link to the gfortran, quadmath, dl, lapack and blas libraries to avoid undefined references from these libraries (which are linked by default if building a shared library). This presents a further problem when linking a mex file on Linux as the Matlab supplied LAPACK and BLAS shared libraries are not the same version as the versions which are linked by default by the mex command. For shared libraries the problem is not apparent at link time, but appears at load time when differences in the interfaces cause errors when running the FAST_SFunc command. The solution is to link to static LAPACK and BLAS libraries when creating the mex file. For this reason the USE_LOCAL_STATIC_LAPACK options described above was created. This builds the LAPACK and BLAS libraries locally, and installs them in a place which is easy for the mex linker to find, in the OpenFAST source tree.

Some basic modifications to create_FAST_SFunc.m to make use of these modification have been made. On non-windows platforms it links to gfortran, lapack, blas, quadmath and dt by default to avoid undefined references. The default install directory is also modified to ../../install as on windows, mainly to allow easy linking to static LAPACK installed in these directories. Therefore on linux, an appropriate cmake command is something like:

cmake -LH /home/username/source/path/openfast -DBUILD_FASTFARM=ON -DBUILD_OPENFAST_CPP_API=ON -DBUILD_OPENFAST_SIMULINK_API=ON -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX=/home/username/source/path/openfast/install -DDOUBLE_PRECISION=ON -DOPENMP=OFF -DUSE_LOCAL_STATIC_LAPACK=ON

[reos-rcrozier]

FYI a backport of these modifications to OpenFAST v3.0.0 is available on our branch here

[reos-rcrozier]

looking at the output of the runs, it seems we might have missed specifying a target dependency for versioninfolib_obj. Probably being picked up because the order of compilation is slightly different than on our test machine.

[rafmudaf]

This already happens in set_fast_fortran() on line 65

[reos-rcrozier]

I also realised my comment was wrong when I went to examine this. I'm not yet sure what the issue really is, but I think it might be something to do with the fact that the cmake files now separately specify the target libraries and object libraries. Possibly it need to be changed so the target 'real' libraries all use the object libraries, so cmake doesn't attempt to build the same object twice.

[rafmudaf]

Do we need to install the new objects? As I understand it, they're only used at compile-time to create the new openfast_prelib_obj.

[reos-rcrozier]

Yes, intuitively they have to be in the install target to be used in other CMAKE libraries, but they are not actually installed like real libraries are.

[rafmudaf]

@reos-rcrozier Thanks for this pull request and my apologies that it has sat for a few months. I think I understand the motivation and how the code changes solve the problem.
Have you worked through the issue with the version lib?

[reos-rcrozier]

@rafmudaf @andrew-platt sorry, but pinging again to see if this can now be considered for merge?

[andrew-platt]

@reos-rcrozier, my apologies for the delay in getting to this. I have a few concerns about what impacts this might have on some of the systems we deploy on, but have not had a chance to test your branch or do a thorough review yet.

I would like to get this into our v3.5.0 release that we plan to release before the end of the month, so we will be looking at this within the next week or two.

@deslaughter, is there anything you wanted to add to the discussion at present?

[reos-rcrozier]

@andrew-platt we have experience cross-compiling to many platforms, so if you need testing perhaps we can help.

For example I have previously cross-compiled FAST for PowerPC and (i think, it was some time ago) arm, although some tweaks were needed to FORTRAN code for powerPC.

[henyau]

Hi all. I'm not sure if this will be helpful to anyone, but to get a working MEX file on my system (Ubuntu 22.04) with USE_LOCAL_STATIC_LAPACK ON, I had to explicitly add a lot of sources to the simulink CMakeLists.txt (NWTC, PDAS, NetLib SLATEC, Supercontroller, RanLux, Openfoam). Otherwise I would get errors like: "undefined symbol: __openfoam_types_MOD_opfm_destroyoutput" when calling FAST_SFunc in Matlab.

[reos-rcrozier]

@henyau Did you do a static or shared build of OpenFAST? Doing a static build (-DBUILD_SHARED_LIBS=OFF as in the example at the top of this thread) should mean you don't have any undefined references.

[deslaughter]

@henyau I'm also having a lot of issues with this combination of flags on Linux. It seems to work fine on my Mac. If you have the chance, could you try -DBLA_STATIC=ON instead of -DUSE_LOCAL_STATIC_LAPACK=ON? This option will find and use a static BLAS/LAPACK from your system instead of building from source. BLA_STATIC works with the Intel MKL library and has been tested on Windows and Linux.

[henyau]

@reos-rcrozier: Yes, was doing a static build (-DBUILD_SHARED_LIBS=OFF).
@deslaughter: I just tried it with -DBLA_STATIC=ON (Cmake finds and uses OpenBLAS). It compiles and links fine, but I get the same undefined symbol error as before when trying to run the Simulink model. Using Intel MKL would have been my next step if I hadn't gotten it to work.

[reos-rcrozier]

@henyau ok, can I first confirm you using create_FAST_SFunc.m to create the mex file? If so, you could also try adding some more -l options to explicitly link to the libraries there which might also solve the problem. Would also be useful to confirm the compiler suite and version you were are using please.

[reos-rcrozier]

Actually, I didn't realize until now that @deslaughter removed most of the changes introduced in this PR. So I probably can't help much.

[reos-rcrozier]

@deslaughter A bit frustrating that all this work we did, specifically to make simulink mex functions work properly on lInux was simply removed without tagging me to see if we could resolve the mac issue. I don't think removing the object libs was the right way to go.

[deslaughter]

I just tried it with -DBLA_STATIC=ON (Cmake finds and uses OpenBLAS). It compiles and links fine, but I get the same undefined symbol error as before when trying to run the Simulink model. Using Intel MKL would have been my next step if I hadn't gotten it to work.

@henyau I was able to reproduce the issue on Linux and tracked it down to the order that the libraries are linked in glue-codes/simulink/CMakeLists.txt. Could you try the following order and let me know if it resolves the issue? If not, please include the full error message from Matlab.

  $<TARGET_FILE:openfast_prelib>
  $<TARGET_FILE:aerodyn14lib>
  $<TARGET_FILE:aerodynlib>
  $<TARGET_FILE:beamdynlib>
  $<TARGET_FILE:elastodynlib>
  $<TARGET_FILE:extptfm_mckflib>
  $<TARGET_FILE:feamlib>
  $<TARGET_FILE:hydrodynlib>
  $<TARGET_FILE:icedynlib>
  $<TARGET_FILE:icefloelib>
  $<TARGET_FILE:maplib>
  $<TARGET_FILE:moordynlib>
  $<TARGET_FILE:orcaflexlib>
  $<TARGET_FILE:sctypeslib>
  $<TARGET_FILE:servodynlib-matlab>
  $<TARGET_FILE:subdynlib>
  $<TARGET_FILE:foamfastlib>
  $<TARGET_FILE:ifwlib>
  $<TARGET_FILE:scfastlib>
  $<TARGET_FILE:foamtypeslib>
  $<TARGET_FILE:versioninfolib>
  $<TARGET_FILE:nwtclibs-matlab>

[henyau]

@reos-rcrozier: I was building the mex using cmake/make. I've tried using the create_FAST_SFunc.m script and it produces a much smaller file (52kB). When testing out Run_Test01_SIG.m, I had to use loadlibrary('libopenfastlib','FAST_Library.h') otherwise I would get:
Invalid MEX-file '/home/xxxx/openfast/install/lib//FAST_SFunc.mexa64': libopenfastlib.so: cannot open shared object file: No such file or directory.
Running the example gives no undefined symbol errors, OpenFAST begins to run (things are written to the console) but Matlab crashes at the first time step.

@deslaughter: I've just tried your reordered libs and it works in the sense that there are no undefined symbols, but it also crashes Matlab at the first time step.

I've tried previously to pinpoint the exact location of the crash, but debugging .mex files has been a horrendous process.

I don't mind testing out these suggestions. However, I hope that it isn't just a case where my system is bizarrely configured and nobody else is experiencing these issues, because like I've said, I've already built a working FAST_SFunc.mex.

Build info:

OpenFAST-v3.5.0-dirty
 Compile Info:
  - Compiler: GCC version 11.3.0
  - Architecture: 64 bit
  - Precision: double
  - OpenMP: No
  - Date: Jul 12 2023
  - Time: 09:53:09
 Execution Info:
  - Date: 07/12/2023
  - Time: 09:55:37-0400
cmake version 3.22.1
Matlab version 9.13.0.2166757 (R2022b) Update 4
Ubuntu 22.04
openBLAS v0.3.23

[deslaughter]

@henyau I really appreciate you trying out the changes. Since you've resolved the issue and have a working MEX file, I don't want to take up any more of your time trying to troubleshoot the build system.

[deslaughter]

@deslaughter A bit frustrating that all this work we did, specifically to make simulink mex functions work properly on lInux was simply removed without tagging me to see if we could resolve the mac issue. I don't think removing the object libs was the right way to go.

@reos-rcrozier I apologize for the way this PR was handled, I certainly don't want to make you feel like your contribution wasn't appreciated. After it was merged, several issues were submitted regarding building on Windows and Mac. It was determined that the Object Libraries weren't compatible with Visual Studio on Windows and that their added complexity would be difficult for users to understand. I tried very hard to fix these issues while keeping the object libraries and they were only removed as a last resort. I should have communicated this information better and will strive to do so in the future.

I like the concepts that were introduced, many of which are part of the current build system, and would like to revisit the use of Object Libraries, once https://gitlab.kitware.com/cmake/cmake/-/issues/18090 is resolved. I agree that they solved several issues with managing the OpenFAST libraries.