Problems linking to API

[Myoldmopar]

Issue overview

While developing some automated package testing code, I encountered a problem while exercising the workflow of linking to the API. Note: Python Plugins workflows work just fine, and properly exercise the APIs. However, when trying to link to the API from Python is causing a problem.

As a bit of a backstory, when I build the EnergyPlusAPI target, I dynamically link to the Python shared library, which is brought into install tree and included in the package. I run the install_name_tool on the EnergyPlusAPI target by using GetPrerequisites on the EnergyPlus(.exe) binary and finding the API target inside there.

• Which means step 1 is: Just execute on the API target directly, EnergyPlus itself should not be dependent on the Python lib anyway. This needs cleanup.

When I run install_name_tool, I am changing the dependency on /some/path/to/PythonLibNameHere to @executable_path/PythonLibNameHere, and then copying PythonLibNameHere to the install tree. After installing EnergyPlus, during a Python Plugin workflow, the executable being run is energyplus(.exe), and so the @executable_path/PythonLibNameHere works just fine. However, trying to link a Python script to the API is problematic, because the executable_path is now the Python interpreter somewhere on the system.

An underlying problem here is that API workflows DO NOT depend on the EnergyPlus python lib packaged up. Calling the EnergyPlusAPI from some other Python should NOT end up causing a dependence on that Python library.

• So this is the second thing. We need to figure out how to do this:

• One option is to break EnergyPlusAPI into a couple pieces. One part that does NOT depend on Python, and can be built apart from the Python lib dependency, and the other part that DOES depend on Python. The EnergyPlus(.exe) binary will depend on both, since it will call both things. But calling the Python EnergyPlus API wrapper would never hit the library that depends on Python, so no problem.

• Another option is to dlopen inside EnergyPlus to the Python dynamic library. This way we wouldn't have an explicitly declared dependency that would fail on calling the API. The dlopen calls would only be made if we aren't running in API/Library mode. This is actually a pretty clean approach.

• Another option is to figure out how to statically link in Python on all platforms. This would result in the most ideal package since it would not include a Python dynamic library at all.

In order to hit this error, I was installing EnergyPlus into a blank machine image, then creating a small Python script that instantiated the API and tried to make function calls. During API instantiation, the Python wrapper goes out and loads in the C API functions, which are in the EnergyPlusAPI target. Unfortunately this depends on the Python library, which is how the following errors are manifested:

OSError: dlopen(/tmp/EnergyPlus-9.3.0-baff08990c-Darwin-x86_64/libenergyplusapi.dylib, 6): Library not loaded: @executable_path/Python
  Referenced from: /tmp/EnergyPlus-9.3.0-baff08990c-Darwin-x86_64/libenergyplusapi.dylib

On Windows, I am providing the API DLL of course, but not the .lib file AND the .lib file. I am now linking to the C API in both eager and lazy modes. Using Python works great as long as you add the E+ install folder to the PATH (and set PYTHONPATH to the E+ install).

Linux is working happily, but it could just be luck -- I'm really not sure yet. In general, we just need to tighten this up substantially.

A nice status summary as of 3/30/20 is available here: Myoldmopar/EPTravisTester#4 (comment)

Useful links:

• https://wincent.com/wiki/%40executable_path%2C_%40load_path_and_%40rpath
• https://gregoryszorc.com/blog/2018/12/18/distributing-standalone-python-applications/
• https://medium.com/@donblas/fun-with-rpath-otool-and-install-name-tool-e3e41ae86172

[Myoldmopar]

@mbadams5 had noted that running energyplus from the symlink on Mac was not working. A simple workaround is to just add the E+ install directory to PATH temporarily while running EnergyPlus, then the embedded Python interpreter can locate the packaged up Python stdlib. Not great, but there is a simple workaround at least until this can be tidied up for the bug fix release.

Before:

[/tmp] $ energyplus /Applications/EnergyPlus-9-3-0/ExampleFiles/1ZoneUncontrolled.idf
EnergyPlus Starting
EnergyPlus, Version 9.3.0-baff08990c, YMD=2020.04.01 20:48
            (SNIP)
Fatal Python error: initfsencoding: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'

[/tmp] $ PATH=/Applications/EnergyPlus-9-3-0/ energyplus /Applications/EnergyPlus-9-3-0/ExampleFiles/1ZoneUncontrolled.idf
EnergyPlus Starting
EnergyPlus, Version 9.3.0-baff08990c, YMD=2020.04.01 20:48
            (SNIP)
EnergyPlus Run Time=00hr 00min  0.30sec
EnergyPlus Completed Successfully.

[Myoldmopar]

Another oddity is if you are calling the API and try to run an EnergyPlus command with the -r arg (or presumably a few others) it doesn't find the ReadVars binary. It tries to find it relative to the current running binary, which is no longer EnergyPlus in that dir. This will require a little more work as we'll need to actually find the running E+, not the currently running binary image.