This repository has been archived by the owner on Feb 3, 2021. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 66
Feature: Models v2 #543
Merged
Merged
Feature: Models v2 #543
Changes from all commits
Commits
Show all changes
53 commits
Select commit
Hold shift + click to select a range
5ec2a7e
Models 2.0 wip
timotheeguerin 5787211
More wip
timotheeguerin 908cbb8
Models
timotheeguerin 290716e
Models
timotheeguerin 1ad23fe
More
timotheeguerin 4834da8
More tests
timotheeguerin f7d5a56
Tweaks
timotheeguerin be4bb84
tweaks
timotheeguerin 636feb0
Merge beginning
timotheeguerin 48b326e
Merge wip
timotheeguerin dc62fe7
Fix tests
timotheeguerin bea0c5a
More for merging
timotheeguerin 2295159
Merge
timotheeguerin 2ad46eb
List works
timotheeguerin 5b4c55a
Test inheritance
timotheeguerin 7828145
update Cluster configuration
timotheeguerin f262204
update Cluster configuration
timotheeguerin 04ffd6c
Secrets use new models
timotheeguerin af72e5f
more
timotheeguerin 30923f9
Fix print
timotheeguerin dfbc815
Fix more issues
timotheeguerin 0c275c5
rename vm_count
timotheeguerin 0646007
more Tests
timotheeguerin acf9ea9
Style
timotheeguerin 07b1874
fix issues
timotheeguerin 9817a32
Fix yaml dump
timotheeguerin a4d3435
Added serialization tests
timotheeguerin 04036c8
Remove prints
timotheeguerin 417af1d
remove eunused code
timotheeguerin daa7210
Merge branch 'master' into feature/models2
timotheeguerin 9d7e37c
More
timotheeguerin 46916a3
Merge branch 'feature/models2' of https://github.com/Azure/aztk into …
timotheeguerin 69cc064
Plugin configuration wip
timotheeguerin c926e31
Updated writing models docs
timotheeguerin 5c9e8b1
Fix pylint
timotheeguerin d8e6f66
Fix tests
timotheeguerin 2731e53
fix pylint issue
timotheeguerin eabd77d
More plugin as functions
timotheeguerin 9c1f9d1
Subnet not required
timotheeguerin 9c19d11
Added more test and fix
timotheeguerin 359558e
update docs
timotheeguerin 5bb1e2e
Added more test and fix
timotheeguerin c819eed
Fix
timotheeguerin c4aa186
update tempalte
timotheeguerin e328299
merge
jafreck e7c851f
Merge branch 'master' into feature/models2
jafreck 1936111
Make user configuration optional
timotheeguerin b94f832
Merge branch 'feature/models2' of https://github.com/Azure/aztk into …
timotheeguerin db21d2f
Size 0 handled
timotheeguerin 7fb9423
Tweak models
timotheeguerin 90e79fb
rename size_low_pri to size_low_priority
timotheeguerin 116d0c7
tweaks
timotheeguerin 751030c
spark config default none
timotheeguerin File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -49,3 +49,6 @@ tmp/ | |
|
||
# Built docs | ||
docs/_build/ | ||
|
||
# PyCharm | ||
.idea/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
from .model import Model | ||
from .fields import String, Integer, Boolean, Float, List, ModelMergeStrategy, ListMergeStrategy |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,241 @@ | ||
import collections | ||
import enum | ||
|
||
from aztk.error import InvalidModelFieldError | ||
from . import validators as aztk_validators | ||
|
||
class ModelMergeStrategy(enum.Enum): | ||
Override = 1 | ||
""" | ||
Override the value with the other value | ||
""" | ||
Merge = 2 | ||
""" | ||
Try to merge value nested | ||
""" | ||
|
||
class ListMergeStrategy(enum.Enum): | ||
Replace = 1 | ||
""" | ||
Override the value with the other value | ||
""" | ||
Append = 2 | ||
""" | ||
Append all the values of the new list | ||
""" | ||
|
||
# pylint: disable=W0212 | ||
class Field: | ||
""" | ||
Base class for all model fields | ||
""" | ||
def __init__(self, *validators, **kwargs): | ||
self.default = kwargs.get('default') | ||
self.required = 'default' not in kwargs | ||
self.validators = [] | ||
|
||
if self.required: | ||
self.validators.append(aztk_validators.Required()) | ||
|
||
self.validators.extend(validators) | ||
|
||
choices = kwargs.get('choices') | ||
if choices: | ||
self.validators.append(aztk_validators.In(choices)) | ||
|
||
def validate(self, value): | ||
for validator in self.validators: | ||
validator(value) | ||
|
||
def __get__(self, instance, owner): | ||
if instance is not None: | ||
value = instance._data.get(self) | ||
if value is None: | ||
return instance._defaults.setdefault(self, self._default(instance)) | ||
return value | ||
|
||
return self | ||
|
||
def __set__(self, instance, value): | ||
instance._data[self] = value | ||
|
||
def merge(self, instance, value): | ||
""" | ||
Method called when merging 2 model together. | ||
This is overriden in some of the fields where merge can be handled differently | ||
""" | ||
if value is not None: | ||
instance._data[self] = value | ||
|
||
def serialize(self, instance): | ||
return self.__get__(instance, None) | ||
|
||
def _default(self, model): | ||
if callable(self.default): | ||
return self.__call_default(model) | ||
|
||
return self.default | ||
|
||
def __call_default(self, *args): | ||
try: | ||
return self.default() | ||
except TypeError as error: | ||
try: | ||
return self.default(*args) | ||
except TypeError: | ||
raise error | ||
|
||
|
||
class String(Field): | ||
""" | ||
Model String field | ||
""" | ||
|
||
def __init__(self, *args, **kwargs): | ||
super().__init__(aztk_validators.String(), *args, **kwargs) | ||
|
||
|
||
class Integer(Field): | ||
""" | ||
Model Integer field | ||
""" | ||
def __init__(self, *args, **kwargs): | ||
super().__init__(aztk_validators.Integer(), *args, **kwargs) | ||
|
||
|
||
class Float(Field): | ||
""" | ||
Model Float field | ||
""" | ||
|
||
def __init__(self, *args, **kwargs): | ||
super().__init__(aztk_validators.Float(), *args, **kwargs) | ||
|
||
|
||
class Boolean(Field): | ||
""" | ||
Model Boolean field | ||
""" | ||
|
||
def __init__(self, *args, **kwargs): | ||
super().__init__(aztk_validators.Boolean(), *args, **kwargs) | ||
|
||
|
||
class List(Field): | ||
""" | ||
Field that should be a list | ||
""" | ||
|
||
def __init__(self, model=None, **kwargs): | ||
self.model = model | ||
kwargs.setdefault('default', list) | ||
self.merge_strategy = kwargs.get('merge_strategy', ListMergeStrategy.Append) | ||
self.skip_none = kwargs.get('skip_none', True) | ||
|
||
super().__init__( | ||
aztk_validators.List(*kwargs.get('inner_validators', [])), **kwargs) | ||
|
||
def __set__(self, instance, value): | ||
if isinstance(value, collections.MutableSequence): | ||
value = self._resolve(value) | ||
if value is None: | ||
value = [] | ||
super().__set__(instance, value) | ||
|
||
def _resolve(self, value): | ||
result = [] | ||
for item in value: | ||
if item is None and self.skip_none: # Skip none values | ||
continue | ||
|
||
if self.model and isinstance(item, collections.MutableMapping): | ||
item = self.model(**item) | ||
result.append(item) | ||
return result | ||
|
||
def merge(self, instance, value): | ||
if value is None: | ||
value = [] | ||
|
||
if self.merge_strategy == ListMergeStrategy.Append: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Shouldn't we have implementations for all defined types of There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The other one is replace which is the default |
||
current = instance._data.get(self) | ||
if current is None: | ||
current = [] | ||
value = current + value | ||
|
||
instance._data[self] = value | ||
|
||
def serialize(self, instance): | ||
items = super().serialize(instance) | ||
output = [] | ||
if items is not None: | ||
for item in items: | ||
if hasattr(item, 'to_dict'): | ||
output.append(item.to_dict()) | ||
else: | ||
output.append(item) | ||
return output | ||
|
||
class Model(Field): | ||
""" | ||
Field is another model | ||
|
||
Args: | ||
model (aztk.core.models.Model): Model object that field should be | ||
merge_strategy (ModelMergeStrategy): When merging models how should the nested model be merged. | ||
Default: `ModelMergeStrategy.merge` | ||
""" | ||
|
||
def __init__(self, model, *args, **kwargs): | ||
super().__init__(aztk_validators.Model(model), *args, **kwargs) | ||
|
||
self.model = model | ||
self.merge_strategy = kwargs.get('merge_strategy', ModelMergeStrategy.Merge) | ||
|
||
def __set__(self, instance, value): | ||
if isinstance(value, collections.MutableMapping): | ||
value = self.model(**value) | ||
|
||
super().__set__(instance, value) | ||
|
||
def merge(self, instance, value): | ||
if self.merge_strategy == ModelMergeStrategy.Merge: | ||
current = instance._data.get(self) | ||
if current is not None: | ||
current.merge(value) | ||
value = current | ||
|
||
instance._data[self] = value | ||
|
||
def serialize(self, instance): | ||
val = super().serialize(instance) | ||
if val is not None: | ||
return val.to_dict() | ||
else: | ||
return None | ||
|
||
class Enum(Field): | ||
""" | ||
Field that should be an enum | ||
""" | ||
def __init__(self, model, *args, **kwargs): | ||
super().__init__(aztk_validators.InstanceOf(model), *args, **kwargs) | ||
|
||
self.model = model | ||
|
||
def __set__(self, instance, value): | ||
if value is not None and not isinstance(value, self.model): | ||
try: | ||
value = self.model(value) | ||
except ValueError: | ||
available = [e.value for e in self.model] | ||
raise InvalidModelFieldError("{0} is not a valid option. Use one of {1}".format(value, available)) | ||
super().__set__(instance, value) | ||
|
||
|
||
def serialize(self, instance): | ||
val = super().serialize(instance) | ||
if val is not None: | ||
return val.value | ||
else: | ||
return None |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is true the production value?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This setting is to have option to run/debug the test one by one in vscode instead of using the cli, this is just for development