AeroDyn driver regression test case failing inconsistently #841

rafmudaf · 2021-09-24T17:41:32Z

Bug description
The ad_BAR_CombinedCases AeroDyn module-regression test case is failing inconsistently in the GitHub Actions environment. In one case it fails on Ubuntu with GCC 10, and in another case it fails on macOS with GCC 11. Both of these are compiled in Debug mode. In Release mode, there is no error.

The reported error is also different between jobs.

In the first, we see an error Fortran runtime error: Index '1' of dimension 1 of array 'fileinfo%lines' above upper bound of 0:

44: Test command: /opt/hostedtoolcache/Python/3.7.12/x64/bin/python3 "/home/runner/work/openfast/openfast/reg_tests/executeAerodynRegressionCase.py" "ad_BAR_OLAF" "/home/runner/work/openfast/build/modules/aerodyn/aerodyn_driver" "/home/runner/work/openfast/openfast/reg_tests/.." "/home/runner/work/openfast/build/reg_tests/modules/aerodyn" "0.00001" "Linux" "GNU" "-p"
86
44: Test timeout computed to be: 5400
87
43: At line 428 of file /home/runner/work/openfast/openfast/modules/aerodyn/src/AirfoilInfo.f90
88
43: Fortran runtime error: Index '1' of dimension 1 of array 'fileinfo%lines' above upper bound of 0
89
43: 
90
43: Error termination. Backtrace:
91
43: #0  0x7f1880643700 in ???
92
43: #1  0x7f1880644259 in ???
93
43: #2  0x7f18806448d6 in ???
94
43: #3  0x5570ea5d3ce8 in readaffile
95
43: 	at /home/runner/work/openfast/openfast/modules/aerodyn/src/AirfoilInfo.f90:428
96
43: #4  0x5570ea5e1383 in __airfoilinfo_MOD_afi_init
97
43: 	at /home/runner/work/openfast/openfast/modules/aerodyn/src/AirfoilInfo.f90:142
98
43: #5  0x5570e9dd1afd in init_afiparams
99
43: 	at /home/runner/work/openfast/openfast/modules/aerodyn/src/AeroDyn.f90:2537
100
43: #6  0x5570e9e7de98 in __aerodyn_MOD_ad_init
101
43: 	at /home/runner/work/openfast/openfast/modules/aerodyn/src/AeroDyn.f90:323
102
43: #7  0x5570e9998324 in __aerodyn_driver_subs_MOD_init_aerodyn
103
43: 	at /home/runner/work/openfast/openfast/modules/aerodyn/src/AeroDyn_Driver_Subs.f90:441
104
43: #8  0x5570e999c21e in __aerodyn_driver_subs_MOD_dvr_initcase
105
43: 	at /home/runner/work/openfast/openfast/modules/aerodyn/src/AeroDyn_Driver_Subs.f90:186
106
43: #9  0x5570e990fee4 in aerodyn_driver
107
43: 	at /home/runner/work/openfast/openfast/modules/aerodyn/src/AeroDyn_Driver.f90:46
108
43: #10  0x5570e991033c in main
109
43: 	at /home/runner/work/openfast/openfast/modules/aerodyn/src/AeroDyn_Driver.f90:21

In the other, a stack trace is not reported.

To Reproduce
I have not been able to reproduce this locally. On GitHub Actions, simple run ctest -VV -j7 -R ad_.

It's compiled with the following CMake command:

          cmake \
            -DCMAKE_INSTALL_PREFIX:PATH=${{runner.workspace}}/openfast/install \
            -DCMAKE_Fortran_COMPILER:STRING=${{matrix.FORTRAN_COMPILER}} \
            -DCMAKE_CXX_COMPILER:STRING=${{env.CXX_COMPILER}} \
            -DCMAKE_C_COMPILER:STRING=${{env.C_COMPILER}} \
            -DCMAKE_BUILD_TYPE:STRING=Debug \
            -DBUILD_TESTING:BOOL=ON \
            -DCODECOV:BOOL=ON \
            -DCTEST_PLOT_ERRORS:BOOL=ON \
            ${GITHUB_WORKSPACE}

OpenFAST Version
This is happening on commits on top of 0543df2

System Information (please complete the following information):

OS	Compiler	Compiler Settings
Ubuntu 20.04	GCC 10	Debug mode, `--coverage` flag
macOS 11	GCC 11	Debug mode, `--coverage` flag

The text was updated successfully, but these errors were encountered:

rafmudaf · 2021-09-28T21:50:22Z

For reference, here's a compiler log of AeroDyn Driver with all compiler warnings enabled (-Wall) with GCC.

aerodyn_driver_compiler_warnings.txt

ebranlard · 2021-10-05T18:32:40Z

The combined case re-initializes AeroDyn, and perform several simulations. My guess is that it could be linked to some variables not being cleaned up/deallocated or reallocated.

This type of syntax caused issues in some version of gfortran long ago: old-NWTC/NWTC_Library@976d9bd#diff-9cf80e40ecf0eacb20ef6f7042888d47defca409fb3c4bb7e972be47e93c2e6f Maybe this helps with OpenFAST#841?

bjonkman · 2021-10-18T20:39:48Z

In this case, the ad_MultipleHAWT regression test failed with the same Fortran runtime error: Index '1' of dimension 1 of array 'fileinfo%lines' above upper bound of 0 error, but the ad_BAR_CombinedCases test passed.

I updated some code just now so that we should not see the issue with accessing past the end of the fileinfo%lines array again, but that does not fix the reason that the code thinks that array is size 0.

bjonkman · 2021-10-21T18:30:37Z

@rafmudaf , I've been running these cases through Inspector to try to understand what's going on in these cases.

In the MacOS with GCC 11 case, it seems like the issue could have been related to a couple of FileInfo variables that weren't deallocated between file reads when AD is restarted. I think I have fixed this issue--at least I have not seen it again--but I'd feel more confident if I could confirm the original problem was what I suspect.

In the Ubuntu with GCC 10 case, it seems to be producing a seg fault on the first call to AeroDyn when it is reading airfoil files. I have put a lot of additional error checks in there, but it still seems to be failing intermittently. I'm going to step away from this issue for now.

bjonkman · 2021-10-21T20:01:58Z

Guess I spoke too soon.... in my last commit, everything worked okay on the OpenFAST PR side, but the MacOS with GCC 11 case failed on my fork. Here's the error it gave this time:

Dvr_InitCase:Init_AeroDyn:AD_Init:Init_AFIparams:AFI_Init:ReadAFfile:ParseInVar:

A fatal error occurred when parsing data from
"/Users/runner/work/openfast/openfast/build/reg_tests/modules/aerodyn/ad_BAR_CombinedCases/./../B
AR_Baseline/Airfoils/BAR0_AeroDyn15_Polar_04.dat".
The variable "NumCoords" was not found on line # 9.

It seems like those arrays in the FileInfo type have a problem...

hkross · 2021-12-20T21:49:14Z

UPDATE: I resolved the issue with my branch. The error was caused by the fact that one of the turbines defined in this test case has zero blades, and I was defining the size of a variable based on the number of blades. This commit resolved the issue. I am not sure if this translates over to the other issues mentioned in this thread.

I am seeing similars error on my Buoyancy feature branch. The ad_QuadRotor_OLAF test passes in release mode but fails in debug mode (both macOS and ubuntu) with the following error: Fortran runtime error: Index '1' of dimension 1 of array 'rotdata%bladeprops' above upper bound of 0.

41: Test command: /Users/runner/hostedtoolcache/Python/3.7.12/x64/bin/python3 "/Users/runner/work/openfast/openfast/reg_tests/executeAerodynRegressionCase.py" "ad_QuadRotor_OLAF" "/Users/runner/work/openfast/openfast/build/modules/aerodyn/aerodyn_driver" "/Users/runner/work/openfast/openfast/reg_tests/.." "/Users/runner/work/openfast/openfast/build/reg_tests/modules/aerodyn" "0.00001" "Darwin" "GNU" "-p"
41: Test timeout computed to be: 5400
41: At line 1080 of file /Users/runner/work/openfast/openfast/modules/aerodyn/src/AeroDyn.f90
41: Fortran runtime error: Index '1' of dimension 1 of array 'rotdata%bladeprops' above upper bound of 0
41: Error termination. Backtrace:
41: # 0 0x10a8add3e
41: # 1 0x10a8ae9e5
41: # 2 0x10a8aefb6
41: # 3 0x10974a994
41: # 4 0x1097a1d21
41: # 5 0x109277f3e
41: # 6 0x10927bd7e
41: # 7 0x1091ea674
41: # 8 0x1091eaaf6
41: /Users/runner/work/openfast/openfast/build/modules/aerodyn/aerodyn_driver /Users/runner/work/openfast/openfast/build/reg_tests/modules/aerodyn/ad_QuadRotor_OLAF/ad_driver.dvr > /Users/runner/work/openfast/openfast/build/reg_tests/modules/aerodyn/ad_QuadRotor_OLAF/ad_driver.log
41: COMPLETE with code 2
1/11 Test # 41: ad_QuadRotor_OLAF ................***Failed 3.75 sec

ebranlard · 2022-02-16T18:50:58Z

Hi Hannah, could you cherry-pick this commit and submit a pull request for your fix maybe?
Otherwise we can wait for your branch to be merged.

hkross · 2022-02-16T19:16:13Z

Hi Hannah, could you cherry-pick this commit and submit a pull request for your fix maybe?
Otherwise we can wait for your branch to be merged.

You mean the commit that fixes the allocation of OtherState%ValidPhi, or a different commit? If you are referring to the allocation of OtherState%ValidPhi, this is a single commit on its own branch in my repository (hkross/bug/BEMT_ValidPhi). Pull request #1001 is for merging this branch into dev, so it should be ready to go.

ebranlard · 2022-02-16T19:23:20Z

Maybe I misunderstood, I thought you found a solution to this issue (#841), but your solution only works for the Buoyancy branch it seems. Sorry!

hkross · 2022-02-16T19:24:30Z

Ah. Yes, unfortunately the issue I was seeing was within a buoyancy variable, so not applicable to dev.

bjonkman · 2022-10-20T21:13:07Z

@rafmudaf: I haven't seen an intermittent error with AD driver from the debug modules test recently. Did the problem get fixed? Or are we just not testing for this problem anymore with the updated GitHub Actions from #1203?

rafmudaf · 2022-10-21T14:15:42Z

Unfortunately, the bug persists and its not captured in the tests. I had started to dig into this earlier in the year on a Linux computer, and after recompiling with the latest dev branch I am still able to trigger this issue by running the AD_BAR regression test cases in a loop:

for i in {1..20}; do

    echo $i
    cd build/
    if ! ctest -R ad_BAR -j8 ; then
        cd ../
        # cp -r build build_$i    ## Save the failing build directory for further inspection
        rm -rf reg_tests/modules/aerodyn
    else
        cd ../
    fi

done

bjonkman · 2022-10-25T15:11:31Z

Okay, thanks for the update.

rafmudaf added Type: Bug Help wanted Module: AeroDyn System: Unix System: CMake Type: Testing Compiler: GNU labels Sep 24, 2021

bjonkman mentioned this issue Oct 13, 2021

AeroDyn cleanup #863

Merged

ebranlard mentioned this issue Feb 16, 2022

Remove conditional statement for initialization of BEMT variable #1001

Merged

rafmudaf mentioned this issue May 31, 2022

Move AD module reg tests to a standalone job #1141

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AeroDyn driver regression test case failing inconsistently #841

AeroDyn driver regression test case failing inconsistently #841

rafmudaf commented Sep 24, 2021

rafmudaf commented Sep 28, 2021

ebranlard commented Oct 5, 2021

bjonkman commented Oct 18, 2021

bjonkman commented Oct 21, 2021 •

edited

Loading

bjonkman commented Oct 21, 2021

hkross commented Dec 20, 2021 •

edited

Loading

ebranlard commented Feb 16, 2022

hkross commented Feb 16, 2022 •

edited

Loading

ebranlard commented Feb 16, 2022

hkross commented Feb 16, 2022

bjonkman commented Oct 20, 2022

rafmudaf commented Oct 21, 2022

bjonkman commented Oct 25, 2022

AeroDyn driver regression test case failing inconsistently #841

AeroDyn driver regression test case failing inconsistently #841

Comments

rafmudaf commented Sep 24, 2021

rafmudaf commented Sep 28, 2021

ebranlard commented Oct 5, 2021

bjonkman commented Oct 18, 2021

bjonkman commented Oct 21, 2021 • edited Loading

bjonkman commented Oct 21, 2021

hkross commented Dec 20, 2021 • edited Loading

ebranlard commented Feb 16, 2022

hkross commented Feb 16, 2022 • edited Loading

ebranlard commented Feb 16, 2022

hkross commented Feb 16, 2022

bjonkman commented Oct 20, 2022

rafmudaf commented Oct 21, 2022

bjonkman commented Oct 25, 2022

bjonkman commented Oct 21, 2021 •

edited

Loading

hkross commented Dec 20, 2021 •

edited

Loading

hkross commented Feb 16, 2022 •

edited

Loading