Have one Dockerfile instead of multiples (#364)

* Added first working prototype * added working fenics adapter * remove comments * added working nutils "adapter" * added not working calclulix -adapter * reorder dockerfile * make autopep8 happy * fix print_case_combinations * WIP * WIP * - Introduced logging instead of prints - refactored into exceptions instead of exit(1) - * - Added Nutils - Added Calculix - Added reference results for some cases - Added heat exchanger tutorial - Added logging and more verbose output - Added timing output * Split building docker and running tutorial * adopt to requested chnages: - fallow-arguments vs std-legacy - no nutils adapter, just install nutils * - Adopt .md to .metadata - generate new reference results * - added overview table * added script to only build dockers to warmup cache * Update tools/tests/components.yaml Co-authored-by: Gerasimos Chourdakis <chourdak@in.tum.de> * make autpep8 happy --------- Co-authored-by: Valentin Seitz <valentin.seitz@tum.de> Co-authored-by: Gerasimos Chourdakis <chourdak@in.tum.de>
precice · Sep 5, 2023 · 0cd624e · 0cd624e
1 parent 847a6f0
commit 0cd624e
Show file tree

Hide file tree

Showing 26 changed files with 993 additions and 296 deletions.
diff --git a/flow-over-heated-plate/reference-data/fluid-openfoam_solid-fenics.tar.gz b/flow-over-heated-plate/reference-data/fluid-openfoam_solid-fenics.tar.gz
diff --git a/flow-over-heated-plate/reference-data/fluid-openfoam_solid-nutils.tar.gz b/flow-over-heated-plate/reference-data/fluid-openfoam_solid-nutils.tar.gz
diff --git a/flow-over-heated-plate/reference-data/fluid-openfoam_solid-openfoam.tar.gz b/flow-over-heated-plate/reference-data/fluid-openfoam_solid-openfoam.tar.gz
diff --git a/flow-over-heated-plate/reference_results.md b/flow-over-heated-plate/reference_results.md
diff --git a/flow-over-heated-plate/reference_results.metadata b/flow-over-heated-plate/reference_results.metadata
@@ -0,0 +1,29 @@
+<!---
+This File has been generated by the generate_reference_results.py and should not be modified manually
+-->
+
+# Reference Results
+
+This file contains an overview of the results over the reference results as well as the arguments used to generate them.
+
+## List of files
+
+| name | time | sha256 |
+|------|------|-------|
+| fluid-openfoam_solid-nutils.tar.gz | 2023-09-04 23:41:09 | 8b3157902d3ad78593c5471a3a94dd5559fe6970e4cb54658b22ffd270c00297 |
+| fluid-openfoam_solid-fenics.tar.gz | 2023-09-04 23:41:09 | 2565ffc51c8d80fab06ccdda33f62efceb6cdc03318998f5adc9ed0ac84acac2 |
+| fluid-openfoam_solid-openfoam.tar.gz | 2023-09-04 23:41:09 | c29989e0118ade759e7313330aa8d3d900c660c404eaacd529e979aae6b899c7 |
+
+## List of arguments used to generate the files
+
+| name | value |
+|------|------|
+| PRECICE_REF | v2.5.0 |
+| OPENFOAM_EXECUTABLE | openfoam2306 |
+| OPENFOAM_ADAPTER_REF | v1.2.3 |
+| PYTHON_BINDINGS_REF | v2.5.0.4 |
+| FENICS_ADAPTER_REF | v1.4.0 |
+| TUTORIALS_REF | v202211.0 |
+| PLATFORM | ubuntu_2204 |
+| CALULIX_VERSION | 2.20 |
+| CALULIX_ADAPTER_REF | v2.20.0 |
diff --git a/heat-exchanger-simplified/metadata.yaml b/heat-exchanger-simplified/metadata.yaml
@@ -0,0 +1,29 @@
+name: Heat Exchanger simplified
+path: heat-exchanger-simplified # relative to git repo 
+url: https://precice.org/tutorials-heat-exchanger-simplified.html
+
+participants:
+  - Fluid-Top
+  - Fluid-Bottom
+  - Solid
+
+cases:
+  fluid-btm-openfoam:
+    participant: Fluid-Bottom
+    directory: ./fluid-bottom-openfoam
+    run: ./run.sh 
+    component: openfoam-adapter
+
+  fluid-top-openfoam:
+    participant: Fluid-Top
+    directory: ./fluid-top-openfoam
+    run: ./run.sh 
+    component: openfoam-adapter
+
+  solid-calculix:
+    participant: Solid
+    directory: ./solid-calculix
+    run: ./run.sh 
+    component: calculix-adapter
+
+
diff --git a/perpendicular-flap/reference-data/fluid-openfoam_solid-calculix.tar.gz b/perpendicular-flap/reference-data/fluid-openfoam_solid-calculix.tar.gz
diff --git a/perpendicular-flap/reference_results.metadata b/perpendicular-flap/reference_results.metadata
@@ -0,0 +1,27 @@
+<!---
+This File has been generated by the generate_reference_results.py and should not be modified manually
+-->
+
+# Reference Results
+
+This file contains an overview of the results over the reference results as well as the arguments used to generate them.
+
+## List of files
+
+| name | time | sha256 |
+|------|------|-------|
+| fluid-openfoam_solid-calculix.tar.gz | 2023-09-04 23:41:09 | 596fe1aec3f72fa194c37c485688dea4e308544e3cd773d7c815d6e27c4e55a8 |
+
+## List of arguments used to generate the files
+
+| name | value |
+|------|------|
+| PRECICE_REF | v2.5.0 |
+| OPENFOAM_EXECUTABLE | openfoam2306 |
+| OPENFOAM_ADAPTER_REF | v1.2.3 |
+| PYTHON_BINDINGS_REF | v2.5.0.4 |
+| FENICS_ADAPTER_REF | v1.4.0 |
+| TUTORIALS_REF | v202211.0 |
+| PLATFORM | ubuntu_2204 |
+| CALULIX_VERSION | 2.20 |
+| CALULIX_ADAPTER_REF | v2.20.0 |
diff --git a/tools/tests/build_docker_images.py b/tools/tests/build_docker_images.py
@@ -0,0 +1,96 @@
+
+import argparse
+from pathlib import Path
+from systemtests.SystemtestArguments import SystemtestArguments
+from systemtests.Systemtest import Systemtest, display_systemtestresults_as_table
+from systemtests.TestSuite import TestSuites
+from metadata_parser.metdata import Tutorials
+import logging
+import time
+from paths import PRECICE_TUTORIAL_DIR, PRECICE_TESTS_RUN_DIR, PRECICE_TESTS_DIR
+
+
+def main():
+    parser = argparse.ArgumentParser(description='build docker images')
+
+    # Add an argument for the components
+    parser.add_argument('--suites', type=str,
+                        help='Comma-separated test-suites to execute')
+    parser.add_argument(
+        '--build_args',
+        type=str,
+        help='Comma-separated list of arguments provided to the components like openfoam:2102,pythonbindings:latest')
+    parser.add_argument('--rundir', type=str, help='Directory to run the systemstests in.',
+                        nargs='?', const=PRECICE_TESTS_RUN_DIR, default=PRECICE_TESTS_RUN_DIR)
+
+    parser.add_argument('--log-level', choices=['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'],
+                        default='INFO', help='Set the logging level')
+
+    # Parse the command-line arguments
+    args = parser.parse_args()
+
+    # Configure logging based on the provided log level
+    logging.basicConfig(level=args.log_level, format='%(levelname)s: %(message)s')
+
+    print(f"Using log-level: {args.log_level}")
+
+    systemtests_to_run = []
+    available_tutorials = Tutorials.from_path(PRECICE_TUTORIAL_DIR)
+
+    build_args = SystemtestArguments.from_args(args.build_args)
+    run_directory = Path(args.rundir)
+    if args.suites:
+        test_suites_requested = args.suites.split(',')
+        available_testsuites = TestSuites.from_yaml(
+            PRECICE_TESTS_DIR / "tests.yaml", available_tutorials)
+        test_suites_to_execute = []
+        for test_suite_requested in test_suites_requested:
+            test_suite_found = available_testsuites.get_by_name(
+                test_suite_requested)
+            if not test_suite_found:
+                logging.error(f"Did not find the testsuite with name {test_suite_requested}")
+            else:
+                test_suites_to_execute.append(test_suite_found)
+        if not test_suites_to_execute:
+            raise RuntimeError(
+                f"No matching test suites with names {test_suites_requested} found. Use print_test_suites.py to get an overview")
+        # now convert the test_suites into systemtests
+        for test_suite in test_suites_to_execute:
+            tutorials = test_suite.cases_of_tutorial.keys()
+            for tutorial in tutorials:
+                for case, reference_result in zip(
+                        test_suite.cases_of_tutorial[tutorial], test_suite.reference_results[tutorial]):
+                    systemtests_to_run.append(
+                        Systemtest(tutorial, build_args, case, reference_result))
+
+    if not systemtests_to_run:
+        raise RuntimeError("Did not find any Systemtests to execute.")
+
+    logging.info(f"About to build the images for the following systemtests:\n {systemtests_to_run}")
+
+    results = []
+    for number, systemtest in enumerate(systemtests_to_run):
+        logging.info(f"Started building {systemtest},  {number}/{len(systemtests_to_run)}")
+        t = time.perf_counter()
+        result = systemtest.run_only_build(run_directory)
+        elapsed_time = time.perf_counter() - t
+        logging.info(f"Building image for {systemtest} took {elapsed_time} seconds")
+        results.append(result)
+
+    build_docker_success = True
+    for result in results:
+        if not result.success:
+            logging.error(f"Failed to run {result.systemtest}")
+            build_docker_success = False
+        else:
+            logging.info(f"Success running {result.systemtest}")
+
+    display_systemtestresults_as_table(results)
+    if build_docker_success:
+        exit(0)
+    else:
+        exit(1)
+
+
+if __name__ == '__main__':
+    main()
diff --git a/tools/tests/component-templates/calculix-adapter.yaml b/tools/tests/component-templates/calculix-adapter.yaml
@@ -0,0 +1,16 @@
+build: 
+  context: {{ dockerfile_context }}
+  args:
+    {% for key, value in build_arguments.items() %}
+      - {{key}}={{value}}
+    {% endfor %}
+  target: calculix_adapter
+depends_on:    
+  prepare:
+    condition: service_completed_successfully
+volumes:
+  - {{ run_directory }}:/runs
+command: >
+  /bin/bash -c "id && 
+  cd '/runs/{{ tutorial_folder }}/{{ case_folder }}' &&
+  {{ run }} | tee {{ case_folder }}.log 2>&1"
diff --git a/tools/tests/component-templates/fenics-adapter.yaml b/tools/tests/component-templates/fenics-adapter.yaml
@@ -1,11 +1,14 @@
-image: precice/fenics-adapter:{{ build_arguments["FENICS_ADAPTER_TAG"] }}
-user: ${UID}:${GID}
+build: 
+  context: {{ dockerfile_context }}
+  args:
+    {% for key, value in build_arguments.items() %}
+      - {{key}}={{value}}
+    {% endfor %}
+  target: fenics_adapter
 depends_on:    
   prepare:
     condition: service_completed_successfully
 volumes:
-  - /etc/passwd:/etc/passwd:ro
-  - /etc/group:/etc/group:ro
   - {{ run_directory }}:/runs
 command: >
   /bin/bash -c "id && 

diff --git a/tools/tests/component-templates/nutils-adapter.yaml b/tools/tests/component-templates/nutils-adapter.yaml
@@ -1,11 +1,14 @@
-image: "ghcr.io/precice/openfoam-adapter:{{ build_arguments["openfoam-adapter-ref"] }}" # TODO: Update
-user: ${UID}:${GID}
+build: 
+  context: {{ dockerfile_context }}
+  args:
+    {% for key, value in build_arguments.items() %}
+      - {{key}}={{value}}
+    {% endfor %}
+  target: nutils_adapter
 depends_on:    
   prepare:
     condition: service_completed_successfully
 volumes:
-  - /etc/passwd:/etc/passwd:ro
-  - /etc/group:/etc/group:ro
   - {{ run_directory }}:/runs
 command: >
   /bin/bash -c "id && 

diff --git a/tools/tests/component-templates/openfoam-adapter.yaml b/tools/tests/component-templates/openfoam-adapter.yaml
@@ -1,9 +1,10 @@
 build: 
-  context: https://github.com/precice/openfoam-adapter.git#add-ci-docker:tools/docker
+  context: {{ dockerfile_context }}
   args:
     {% for key, value in build_arguments.items() %}
       - {{key}}={{value}}
     {% endfor %}
+  target: openfoam_adapter
 depends_on:    
   prepare:
     condition: service_completed_successfully

diff --git a/tools/tests/components.yaml b/tools/tests/components.yaml
@@ -2,39 +2,77 @@ openfoam-adapter:
   repository: https://github.com/precice/openfoam-adapter
   template: component-templates/openfoam-adapter.yaml
   build_arguments: # these things mean something to the docker-service
+    PRECICE_REF:
+      description: Version of preCICE to use
+      default: "main"
+    PLATFORM:
+      description: Dockerfile platform used
+      default: "ubuntu_2204"
+    TUTORIALS_REF:
+      description: Tutorial git reference to use
+      default: "master"
     OPENFOAM_EXECUTABLE:
-      options: ["openfoam2112"]
+      options: ["openfoam2306","openfoam2212","openfoam2112"]
       description: exectuable of openfoam to use
-      default: "openfoam2112"
-    PRECICE_TAG:
-      description: Version of preCICE to use
-      default: "latest"
+      default: "openfoam2306"
     OPENFOAM_ADAPTER_REF:
       description: Reference/tag of the actual OpenFOAM adapter
       default: "master"
+
 
 fenics-adapter:
   repository: https://github.com/precice/fenics-adapter
   template: component-templates/fenics-adapter.yaml
   build_arguments:
-    FENICS_ADAPTER_TAG:
-      semnantic: Version of the fenics adapter image to use
-      default: "latest"
-
-calculix-adapter:
-  repository: https://github.com/precice/calculix-adapter
-  template: component-templates/calculix-adapter.yaml
-  build_arguments:
-    PRECICE_TAG:
+    PRECICE_REF:
       description: Version of preCICE to use
-      default: "latest"
-
-
+      default: "main"
+    PLATFORM:
+      description: Dockerfile platform used
+      default: "ubuntu_2204"
+    TUTORIALS_REF:
+      description: Tutorial git reference to use
+      default: "master"
+    PYTHON_BINDINGS_REF:
+      semnantic: Git ref of the pythonbindings to use
+      default: "master"
+    FENICS_ADAPTER_REF:
+      semnantic: Git ref of the fenics adapter to use
+      default: "master"
 
 nutils-adapter:
   repository: https://github.com/precice/nutils-adapter
   template: component-templates/nutils-adapter.yaml
   build_arguments:
-    PRECICE_TAG:
+    PRECICE_REF:
       description: Version of preCICE to use
-      default: "latest"
+      default: "main"
+    PLATFORM:
+      description: Dockerfile platform used
+      default: "ubuntu_2204"
+    TUTORIALS_REF:
+      description: Tutorial git reference to use
+      default: "master"
+    PYTHON_BINDINGS_REF:
+      semnantic: Git ref of the pythonbindings to use
+
+
+calculix-adapter:
+  repository: https://github.com/precice/calculix-adapter
+  template: component-templates/calculix-adapter.yaml
+  build_arguments:
+    PRECICE_REF:
+      description: Version of preCICE to use
+      default: "main"
+    PLATFORM:
+      description: Dockerfile platform used
+      default: "ubuntu_2204"
+    TUTORIALS_REF:
+      description: Tutorial git reference to use
+      default: "master"
+    CALULIX_VERSION:
+      description: Version of Calculix to use
+      default: "2.20"
+    CALULIX_ADAPTER_REF:
+      description: Version of Calculix-Adapter to use
+      default: "master"