XP-456: replace CLI argument with a config file

[little-dude]

Summary

replace the CLI arguments with a config file

Reference:

https://xainag.atlassian.net/browse/XP-456
https://xainag.atlassian.net/browse/XP-512

Rationale

The CLI is getting complex so it is worth loading the configuration
from a file instead.

Implementation details

TOML

We decided to use TOML for the following reasons:

• it is human friendly, ie easy to read and write
• our configuration has a pretty flat structure which makes TOML
  quite adapted
• it is well specified and has lots of implementation
• it is well known

The other options we considered:

• INI: it is quite frequent in the Python ecosystem to use INI for
  config files, and the standard library even provides support for
  this. However, INI is not as powerful as TOML and does not have a
  specification
• JSON: it is very popular but is not human friendly. For instance,
  it does not support comments, is very verbose, and breaks
  easily (if a trailing comma is forgotten at the end of a list for
  instance)
• YAML: another popular choice, but is in my opinion more complex
  than TOML.

Validation

We use the third-party schema library to validate the
configuration. It provides a convenient way to:

• declare a schema to validate our config
• leverage third-party libraries to validate some inputs (we use the
  idna library to validate hostnames)
• define our own validators
• transform data after it has been validated: this can be useful to
  turn a relative path into an absolute one for example
• provide user friendly error message when the configuration is
  invalid

The Config class

By default, the schema library returns a dictionary containing a
valid configuration, but that is not convenient to manipulate in
Python. Therefore, we dynamically create a Config class from the
configuration schema, and instantiate a Config object from the data
returned by the schema validator.

Package re-organization

We moved the command line and config file logic into its own config
sub-package, and moved the former xain_fl.cli.main entrypoint into
the xain_fl.__main__ module.

Docker infrastructure

• Cache the xain_fl dependencies. This considerably reduces
  "edit->build-> debug" cycle, since installing the dependencies takes
  about 30 minutes.
• Move all the docker related files into the docker/ directory

Limitations and future work

1. The documentation generated for the ServerConfig,
   AiConfig and StorageConfig classes is wrong. Each attribute is
   documented as "Alias for field number X". This can be fixed by
   having create_class_from_schema() setting the __doc__ attribute
   for each attribute. However, we won't be able to automatically
   document the type of each attribute.

2. When the configuration contains an invalid value, the error message
   we generate does not contain the invalid value in question. I think
   it is possible to enable this in the future but haven't really
   looked into it.

---

Reviewer checklist

Reviewer agreement:

• Reviewers assign themselves at the start of the review.
• Reviewers do not commit or merge the merge request.
• Reviewers have to check and mark items in the checklist.

Merge request checklist

• Conforms to the merge request title naming XP-XXX <a description in imperative form>.
• Each commit conforms to the naming convention XP-XXX <a description in imperative form>.
• Linked the ticket in the merge request title or the references section.
• Added an informative merge request summary.

Code checklist

• Conforms to the branch naming XP-XXX-<a_small_stub>.
• Passed scope checks.
• Added or updated tests if needed.
• Added or updated code documentation if needed.
• Conforms to Google docstring style.
• Conforms to XAIN structlog style.

[Robert-Steiner]

Looks good to me so far. I will tests it tomorrow.

[atymoshchuk]

Please create follow-up tickets to make changes to docker files and any other changes. Would be great to have only changes about configuration in this pr and everything else in other dedicated prs. Also it’s good for history of tickets and changes introduced to keep the scope according to DoD of linked jira ticket.

[little-dude]

Thanks for the review @finiteprods, I applied the changes you suggested.

[atymoshchuk]

Please update this function after updating structlog message.

[little-dude]

@atymoshchuk sorry, I'm not sure what you mean here.

[little-dude]

This conflicts with #237 but I'll rebase after the final review.