[SMS-278] yaml dict comp (#48)

* tests: base oseomosys construction

* feat: improvement to base abstraction

* feat: add validation for time_defintion

* tests: for time_definition

* add test_otoole_roundtrip pytest

* [SMS-239] cleanup noqa test data (#36)

* fix: rm otoole sample data from tests

* feat: import utils by name

* fix: cleanup some DS store

* fix: import utils

* Refactor root_validator to model_validator (#37)

* tests: otoole_roundtrip

* feat: fitler pandas=3. dep warning

* tests: roundtrip otoole timedefn

* delint

* otoole_roundtrip as pytest

* fix: rename otoole-csv paths

* fix: make long_name and description optional

* tests: skip full otoole construction for now

* tests: test region construction

* feat: make base osemosys data built from args[0]

* fix: rm composable assumptions and targets for now

* fix: accidental rename

* fix: accident rename test case

* tests: commodity construction and compatability

* feat: begin refactor to /compat

* feat: add 'isnumeric' helper util

* refactor: defaults to initial import, don't import pydatnic schemas

* refactor: defaults and compat

* feat: commodity schema

* feat: make data construction more flexible

* feat: build and test impact construction

* tests: impact otoole roundtrip

* fix: some cleanup on commodity testing

* tests: touchup test-impact

* tests: technology construction

* feat: otoole compatability for technology

* feat: add to defaults

* feat: region and impact compat

* feat: validation for technology

* tests: runspec construction and roundtrip

* feat: add defaults for discount rate depreciation method

* feat: instantiate RunSpec on load model

* feat: add a depreciation_method enum

* feat: include reserve margin in commodity and technology defn

* feat: otoole compatability for RunSpec and Technology

* fix: return 'self' from model_validator(mode='after')

* feat: finish RunSpec model

* tests: roundtrip and yaml compatability

* feat: add data casting for OSeMOSYSData

* refactor into composeable OSeMOSYSData class

* refactor: move otoole time_defn to compat

* feat: test commodity composition

* tests: rename composition base test

* feat: add dtype to operating mode vars

* feat: refactor compose

* feat: compose technologies

* feat: compose all schema

* tests: runspec composition

* fix: simplify yaml to start with passing case

* feat: load model yaml

* fix: create timedefinition from yearparts and dayparts

* feat: restore direct dict expression

* feat: add string casting for dict keys

* feat: cast dict keys to str

* fix: delint

---------

Co-authored-by: edwardxtg <edwardxtg@gmail.com>
Co-authored-by: edwardxtg <71764756+edwardxtg@users.noreply.github.com>

feo/osemosys/schemas/base.py

@@ -51,6 +51,18 @@ def cast_osemosysdata_value(val: Any, info: FieldInfo):
             return getattr(OSeMOSYSData, coords).SumOne(data=val)
         elif isnumeric(val):
             return getattr(OSeMOSYSData, coords)(data=val)
+        elif isinstance(val, dict):
+            if validator == "SumOne":
+                return getattr(OSeMOSYSData, coords).SumOne(
+                    data={str(k): v for k, v in val.items()}
+                )
+            elif validator == "Bool":
+                return getattr(OSeMOSYSData, coords).Bool(data={str(k): v for k, v in val.items()})
+            elif validator == "Int":
+                return getattr(OSeMOSYSData, coords).Int(data={str(k): v for k, v in val.items()})
+            elif validator == "DM":
+                return getattr(OSeMOSYSData, coords).DM(data={str(k): v for k, v in val.items()})
+            return getattr(OSeMOSYSData, coords)(data={str(k): v for k, v in val.items()})
 
     return val
 
@@ -273,9 +285,6 @@ def _check_nesting_depth(obj_id: str, data: Any, max_depth: int):
 
 
 def _check_set_membership(obj_id: str, data: Any, sets: Dict[str, List[str]]):
-    print("sets")
-    print(sets)
-
     # cast 'years' to str
     if "years" in sets.keys():
         sets["years"] = [str(yr) for yr in sets["years"]]

feo/osemosys/schemas/model.py

@@ -64,6 +64,7 @@ def compose(self):
     def cast_values(cls, values: Any) -> Any:
         for field, info in cls.model_fields.items():
             field_val = values.get(field)
+
             if field_val is not None:
                 values[field] = cast_osemosysdata_value(field_val, info)
 

tests/test_construction/test_load_from_yaml.py

@@ -45,6 +45,7 @@ def test_expression_parse():
     c: max([4,5,6])
     d: min([4,5,6])
     e: '{k:v for k,v in zip(["x","y","z"],[1,2,3])}'
+    f: '{k:k+1 for k in range(3)}'
     """
     data = yaml.load(blob, Loader=yaml.SafeLoader)
 
@@ -55,6 +56,7 @@ def test_expression_parse():
     assert data["d"] == 4
     assert isinstance(data["e"], dict)
     assert data["e"]["x"] == 1
+    assert data["f"][0] == 1
 
 
 def test_sample_construction():

tests/test_construction/test_technology.py

@@ -9,11 +9,17 @@
         capex=15,
         opex_fixed=1.5,
         operating_modes=[dict(id="mode_1")],
-    )
+    ),
+    with_capex=dict(
+        id="with_capex",
+        operating_life=10,
+        capex={2022: 15, 2023: 20},
+        opex_fixed=1.5,
+        operating_modes=[dict(id="mode_1")],
+    ),
 )
 
 FAILING_TECH_DEFINITIONS = dict(
-    # Technology minimum capacity limit must be < maximum capacity limit
     capacity_min_gt_max=dict(
         id="capacity_min_gt_max",
         operating_life=10,