Merge branch '97-integrate-changes-to-params_dict-conversion-code-so-…

…params_dict-now-allows-for-time-durations' into 59-set-hard-resource-limits-on-worker-containers

.env-template → .env.template

@@ -4,6 +4,8 @@ POSTGRES_DEV_PORT=6432
 POSTGRES_ORCHESTRATOR_USER_NAME=omotes_orchestrator
 POSTGRES_ORCHESTRATOR_USER_PASSWORD=somepass3
 
+RABBITMQ_PORT=5672
+RABBITMQ_MANAGEMENT_PORT=15672
 RABBITMQ_ROOT_USER=root
 RABBITMQ_ROOT_PASSWORD=5678
 RABBITMQ_OMOTES_USER_NAME=omotes

CHANGELOG.md

@@ -0,0 +1,17 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [mvp.4.RC2] - 2024-0x-xx
+
+### Omotes-system
+* 47: Add README.md and ci scripts for system_tests and example_sdk_client providing easy info and access to using them locally. Also add Legal considerations in root README.md and alter reconnect.py example to fully show reconnect. by @lfse-slafleur in https://github.com/Project-OMOTES/omotes-system/pull/68
+* 71: Only load .env if it exists on setup.sh and exit with error if any env variables from .env-template are missing. by @lfse-slafleur in https://github.com/Project-OMOTES/omotes-system/pull/72
+* 74: Reference .env.template from _load_dot_env.sh relative to the root of the repository instead of relative to current working directory. by @lfse-slafleur in https://github.com/Project-OMOTES/omotes-system/pull/75
+* 77: Update optimizer worker to 2.0.0 to include mesido 0.1.5.2 which also includes influxdb schema changes. by @lfse-slafleur in https://github.com/Project-OMOTES/omotes-system/pull/78
+* 66: Workflow definitions are now shared by orchestrator with the SDK and are updated on any changes. by @MarkTNO  in https://github.com/Project-OMOTES/omotes-system/pull/67
+* 92: Update to SDK 3.0.0 and orchestrator 1.0.0 to integrate single job submission queue and multi-sdk support for sharing workflow definitions from the orchestrator to any SDK. by @lfse-slafleur in https://github.com/Project-OMOTES/omotes-system/pull/93
+* 95: Update optimizer worker to 2.0.2 which includes mesido 0.1.7.0. by @lfse-slafleur in https://github.com/Project-OMOTES/omotes-system/pull/96

README.md

@@ -1,25 +1,60 @@
-# Computation Engine - NWN
+# Omotes System
+
 Setup the required infrastructure and database migrations.
 The infrastructure consists of the following components:
+
 - RabbitMQ message broker
 - Optimization workers
 
 ## Setup infrastructure
-Copy `.env-template` to `.env` and fill with the appropriate values.  
+
+Copy `.env.template` to `.env` and fill with the appropriate values.
+
+### Workflow configuration
+
+The available workflows are configured via a json file, for an example
+see: `/config/workflow_config_example.json`  
+A workflow must have a `workflow_type_name` and `workflow_type_description_name`.
+The `workflow_parameters` are optional, for workflows that need additional information next to the
+ESDL.
+Each workflow parameter must have a `parameter_type` and `key_name`, all others are optional.  
+See `config/workflow_config_example.json` for options for
+the `string`, `boolean`, `integer`, `float` and `datetime` parameter formats.
+
+In `docker-compose.yml` the config file to use is specified in the `orchestrator` service in
+the `volumes` section.
+
+### Start infrastructure
+
 To setup the infrastructure components (for windows run in `Git Bash`):
+
 ```
 ./scripts/setup.sh
 ```
+
 To start the infrastructure components:
+
 ```
 ./scripts/start.sh
 ```
+
 To stop the components:
+
 ```
 ./scripts/stop.sh
 ```
 
+#### Run in dev mode
+
+Alternatively the components can be run using local code. This requires all omotes repositories to
+be located in the same parent folder. This setup will also expose the postgres database:
+
+```
+./scripts/start-dev.sh
+```
+
 ## Update infrastructure
+
 In order to update the infrastructure, you first stop the current system, setup the new system
 and start it. Docker and other tooling will ensure only updated infrastructure components will be
 replaced.
@@ -31,6 +66,7 @@ replaced.
 ```
 
 ## Quickstart example
+
 Once the infrastructure is up and running, an example job may be run to check if the infrastructure
 is behaving as expected. The example may be found in `example_sdk_client/`. To run it:
 
@@ -44,6 +80,7 @@ deactivate
 ```
 
 We expect similar output to:
+
 ```bash
 Will use log level LogLevel.INFO for logger 'omotes_sdk'
 Will use log level LogLevel.INFO for logger 'omotes_sdk_internal'
@@ -65,6 +102,7 @@ Job cc17549b-2d11-4076-904e-bd548b456e57 is done (type: grow_optimizer). Status:
 ```
 
 # Licensing & legal considerations
+
 We have opensourced the OMOTES stack under GPLv3. This ensures that the components of OMOTES are not
 altered and run closed-source but changes to OMOTES components are required to also be opensourced.
 However, the GPLv3 license does not prevent from OMOTES being used in a closed-source, commercial

config/workflow_config_example.json

@@ -0,0 +1,75 @@
+[
+  {
+    "workflow_type_name": "workflow_1",
+    "workflow_type_description_name": "Example workflow without parameters (except the ESDL)."
+  },
+  {
+    "workflow_type_name": "workflow_2",
+    "workflow_type_description_name": "Example workflow with parameters.",
+    "workflow_parameters": [
+      {
+        "parameter_type": "string",
+        "key_name": "string_example",
+        "title": "Optional, will override the 'capitalized, underscore to space', key_name",
+        "description": "Optional description/explanation.",
+        "default": "Optional default value",
+        "enum_options": [
+          {
+            "key_name": "key1",
+            "display_name": "Display name 1"
+          },
+          {
+            "key_name": "key2",
+            "display_name": "Display name 2"
+          }
+        ]
+      },
+      {
+        "parameter_type": "boolean",
+        "key_name": "boolean_example",
+        "title": "Optional, will override the 'capitalized, underscore to space', key_name",
+        "description": "Optional description/explanation.",
+        "default": true
+      },
+      {
+        "parameter_type": "integer",
+        "key_name": "integer_example",
+        "title": "This will override the 'capitalized, underscore to space', key_name",
+        "description": "Description/explanation.",
+        "default": 1,
+        "minimum": 0,
+        "maximum": 10
+      },
+      {
+        "parameter_type": "float",
+        "key_name": "float_example",
+        "title": "This will override the 'capitalized, underscore to space', key_name",
+        "description": "Description/explanation.",
+        "default": 1.5,
+        "minimum": 0.0,
+        "maximum": 9.5
+      },
+      {
+        "parameter_type": "float",
+        "key_name": "float_example_basic"
+      },
+      {
+        "parameter_type": "datetime",
+        "key_name": "datetime_example"
+      },
+      {
+        "parameter_type": "datetime",
+        "key_name": "datetime_example_with_default",
+        "default": "2023-12-31T20:00:00"
+      },
+      {
+        "parameter_type": "duration",
+        "key_name": "duration_example_with_default",
+        "description": "The default is in seconds.",
+        "default": 900.0,
+        "minimum": 0.0,
+        "maximum": 1500.0
+      }
+    ]
+  }
+]

config/workflow_config_nwn.json

@@ -0,0 +1,38 @@
+[
+  {
+    "workflow_type_name": "grow_optimizer_default",
+    "workflow_type_description_name": "Grow Optimizer default workflow"
+  },
+  {
+    "workflow_type_name": "grow_simulator",
+    "workflow_type_description_name": "Grow Simulator"
+  },
+  {
+    "workflow_type_name": "grow_optimizer_no_heat_losses",
+    "workflow_type_description_name": "Grow Optimizer without heat losses"
+  },
+  {
+    "workflow_type_name": "grow_optimizer_with_pressure",
+    "workflow_type_description_name": "Grow Optimizer with pump pressures"
+  },
+  {
+    "workflow_type_name": "simulator",
+    "workflow_type_description_name": "High fidelity simulator",
+    "workflow_parameters": [
+      {
+        "parameter_type": "duration",
+        "key_name": "timestep",
+        "description": "Duration of a timestep in seconds.",
+        "minimum": 0
+      },
+      {
+        "parameter_type": "datetime",
+        "key_name": "start_time"
+      },
+      {
+        "parameter_type": "datetime",
+        "key_name": "end_time"
+      }
+    ]
+  }
+]

docker-compose.override.dev.yml

@@ -0,0 +1,32 @@
+services:
+
+  orchestrator:
+    image: omotes-orchestrator-dev
+    build:
+      context: ..
+      dockerfile: orchestrator/dev.Dockerfile
+
+  grow_worker_optimizer: &grow_worker
+    image: omotes-worker-mesido-dev
+    build:
+      context: ..
+      dockerfile: optimizer-worker/dev.Dockerfile
+
+  grow_worker_simulator:
+    <<: *grow_worker
+
+  grow_worker_optimizer_no_heat_losses:
+    <<: *grow_worker
+
+  grow_worker_optimizer_with_pressure:
+    <<: *grow_worker
+
+  omotes_simulator_worker:
+    image: omotes-worker-simulator-dev
+    build:
+      context: ..
+      dockerfile: simulator-worker/dev.Dockerfile
+
+  orchestrator_postgres_db:
+    ports:
+      - "${POSTGRES_DEV_PORT}:5432"

docker-compose.yml

@@ -23,8 +23,8 @@ services:
       LOG_LEVEL: ${LOG_LEVEL}
     hostname: rabbitmq-nwn
     ports:
-      - '5672:5672'
-      - '15672:15672'
+      - '${RABBITMQ_PORT}:5672'
+      - '${RABBITMQ_MANAGEMENT_PORT}:15672'
     networks:
       - omotes
     #      - mapeditor-net
@@ -86,10 +86,12 @@ services:
       - manual_dev
 
   orchestrator:
-    image: ghcr.io/project-omotes/omotes_orchestrator:0.0.12
+    image: ghcr.io/project-omotes/omotes_orchestrator:1.0.0
     restart: unless-stopped
     networks:
       - omotes
+    volumes:
+      - "./config/workflow_config_nwn.json:/config/workflow_config.json"
     environment:
       CELERY_RABBITMQ_USERNAME: ${RABBITMQ_CELERY_USER_NAME}
       CELERY_RABBITMQ_PASSWORD: ${RABBITMQ_CELERY_USER_PASSWORD}
@@ -123,7 +125,7 @@ services:
         condition: service_healthy
 
   grow_worker_optimizer: &grow_worker
-    image: ghcr.io/project-omotes/omotes-grow-optimizer-worker:1.0.14
+    image: ghcr.io/project-omotes/omotes-grow-optimizer-worker:2.0.3
     restart: unless-stopped
     deploy:
       replicas: 2
@@ -171,7 +173,7 @@ services:
       GROW_TASK_TYPE: grow_optimizer_with_pressure
 
   omotes_simulator_worker:
-    image: ghcr.io/project-omotes/omotes-simulator-worker:0.0.8
+    image: ghcr.io/project-omotes/omotes-simulator-worker:0.0.11
     restart: unless-stopped
     deploy:
       replicas: 2

example_sdk_client/optimizer.py

@@ -1,5 +1,5 @@
-import base64
 import time
+from datetime import timedelta
 
 from omotes_sdk.config import RabbitMQConfig
 from omotes_sdk.omotes_interface import (
@@ -9,7 +9,6 @@
     JobProgressUpdate,
     JobStatusUpdate,
 )
-from omotes_sdk.workflow_type import WorkflowType, WorkflowTypeManager
 
 rabbitmq_config = RabbitMQConfig(username="omotes", password="somepass1", virtual_host="omotes")
 
@@ -44,25 +43,27 @@ def handle_on_progress_update(job: Job, progress_update: JobProgressUpdate):
 
 
 try:
-    workflow_optimizer = WorkflowType("grow_optimizer_default", "some descr")
-    workflow_manager = WorkflowTypeManager([workflow_optimizer])
-
-    omotes_if = OmotesInterface(rabbitmq_config, possible_workflows=workflow_manager)
+    omotes_if = OmotesInterface(rabbitmq_config, "example_sdk")
     omotes_if.start()
 
-    with open('example_esdl_optimizer_poc_tutorial.esdl') as open_file:
+    workflow_optimizer = omotes_if.get_workflow_type_manager().get_workflow_by_name(
+        "grow_optimizer_default"
+    )
+
+    with open("example_esdl_optimizer_poc_tutorial.esdl") as open_file:
         input_esdl = open_file.read()
 
     omotes_if.submit_job(
         esdl=input_esdl,
         params_dict={"key1": "value1", "key2": ["just", "a", "list", "with", "an", "integer", 3]},
         workflow_type=workflow_optimizer,
-        job_timeout=None,
+        job_timeout=timedelta(hours=1),
         callback_on_finished=handle_on_finished,
         callback_on_progress_update=handle_on_progress_update,
         callback_on_status_update=handle_on_status_update,
         auto_disconnect_on_result=True,
     )
     time.sleep(60)
 finally:
+    print("Closing rabbitmq connection")
     omotes_if.stop()

example_sdk_client/optimizer_cancel.py

@@ -1,5 +1,5 @@
-import base64
 import time
+from datetime import timedelta
 
 from omotes_sdk.config import RabbitMQConfig
 from omotes_sdk.omotes_interface import (
@@ -9,7 +9,6 @@
     JobProgressUpdate,
     JobStatusUpdate,
 )
-from omotes_sdk.workflow_type import WorkflowType, WorkflowTypeManager
 
 rabbitmq_config = RabbitMQConfig(username="omotes", password="somepass1", virtual_host="omotes")
 
@@ -44,20 +43,21 @@ def handle_on_progress_update(job: Job, progress_update: JobProgressUpdate):
 
 
 try:
-    workflow_optimizer = WorkflowType("grow_optimizer_default", "some descr")
-    workflow_manager = WorkflowTypeManager([workflow_optimizer])
-
-    omotes_if = OmotesInterface(rabbitmq_config, possible_workflows=workflow_manager)
+    omotes_if = OmotesInterface(rabbitmq_config, "example_sdk")
     omotes_if.start()
 
-    with open('example_esdl_optimizer_poc_tutorial.esdl') as open_file:
+    workflow_optimizer = omotes_if.get_workflow_type_manager().get_workflow_by_name(
+        "grow_optimizer_default"
+    )
+
+    with open("example_esdl_optimizer_poc_tutorial.esdl") as open_file:
         input_esdl = open_file.read()
 
     job = omotes_if.submit_job(
         esdl=input_esdl,
         params_dict={"key1": "value1", "key2": ["just", "a", "list", "with", "an", "integer", 3]},
         workflow_type=workflow_optimizer,
-        job_timeout=None,
+        job_timeout=timedelta(hours=1),
         callback_on_finished=handle_on_finished,
         callback_on_progress_update=handle_on_progress_update,
         callback_on_status_update=handle_on_status_update,