YDK bundles

[ghost]

Background

Currently, YDK covers all the YANG model APIs, including openconfig, IETF, Cisco etc under the same python package and this is published as the ydk-py python package. The main reason why bundles are needed is that there needs to be a way to separately generate python packages for YDK model APIs for various YANG module repositories like openconfig, IETF, Cisco XR, Cisco XE etc. This will have more real world uses.

Bundling is meant as a extension of and replacement to the profile mechanism, which ydk-gen currently makes use of. Currently, ydk-gen generates a set of model APIs based on a profile and combined with the runtime YDK components (providers & services), creates a pip installable package (model APIs + YDK providers & services) called ydk-py.

Proposal

With the introduction of bundles, ydk-gen will no longer generate a pip package called ydk-py. Instead the new pip package will only contain the model API's for a given set of YANG models which are specified in the bundle JSON file. For example, ydk-gen can generate a pip installable package (model APIs) called ietf-bundle.

Currently, ydk-gen operates in the below way:

1. Parse profile JSON
2. Download yang models
3. Using pyang, parse YANG files and return modules which are trees of Statement python objects
4. Generate expanded YDK API model Packages from the above Statement trees
5. Pass the ydk Packages to the PythonModulePrinter to print the YDK model API files

Package structure looks like below:

ydk-py
 |
 |-ydk
    |-models
       |
       |- ietf_interface
       | - ietf_system
    |-providers
    |-services
 |-samples

By introducing the concept of bundling, the flow will be like the below:

Resolve bundle (This sub-task is tracked in #26)

1. Parse bundle JSON
2. Download yang models (e.g. openconfig yang models) specified in the bundle plus any dependent YANG models from a different bundle (e.g. IETF yang models) which can also be specified in the main bundle

Build API model for bundle(s)
3. Using pyang, parse YANG files and return modules which are trees of Statement python objects
4. Generate expanded YDK API model Packages from the above Statement trees. Here, each bundle may utilize different code generation strategies, depending on the YANG model structure of the models in the bundle. For example, bundles can make use of choice statement inheritance or collapse containers with single list
5. Assign the list of YDK Packages as children of Bundle objects (by making use of the bundle JSON file)

Print model APIs for bundle(s)
6. Pass the ydk Bundles to the YDK Printer to print the YDK model API files for each bundle and generate the pip installable bundle (can be a python package and in the future, can be C++ library/ Ruby gem)

Package structure will look like below:

ietf-bundle
 |
 |-models
    |
    |- ietf_interface
    | - ietf_system
 |-samples

[ghost]

How the end user will use YDK and bundles

The end user will pip install the bundle which they are interested in. For example, if they are interested in the openconfig bundle:
pip install openconfig-bundle
The bundle will have ydk-py (which consists of the runtime components - services + providers - and is published separately) as a dependency. So, they will pip install ydk-py:
pip install ydk-py
To use the models in apps, the user will import the model they are interested in plus any YDK services/providers:

from openconfig-bundle.models.bgp import Bgp
from ydk.services import CRUDService
from ydk.providers import NetconfServiceProvider

def main():
  bgp_obj = Bgp()
  crud = CRUDService()
  provider  = NetconfServiceProvider("1.2.3.4")
  result = crud.read(provider, bgp_obj)

[einarnn]

One initial question.

If the developer has two different bundles that both augment a common base module, what happens? E.g. If someone installs (openconfig-interfaces, openconfig-if-ip) and then tries to install (openconfig-interfaces, openconfig-if-ethernet, openconfig-if-aggregate, openconfig-lacp)?

[ghost]

Thanks, Einar. I agree we need to come up with a solution for this. So far, I think I had considered bundles to consist of the entire set of models published by IETF or openconfig, for example. But there needs to be some way of addressing smaller, more granular bundle profiles, consisting of fewer models.

[ghost]

Another consideration is to have the Vagrant files as part of the MANIFEST file in the pip packaging for each bundle

[einarnn]

BTW, as we discussed, if we want to start with a smaller scope that is more tractable, that's also fine!

[ghost]

Imported comment from gitlab:

The profile mechanism needs to be extended to first validate the profile to check if all the models referenced by the models in the profile are included in the profile