XP-227 - Merge v0.3.0 changes into master (#233)

* XP-337 Clean up docs before generation (#188)

* XP-229 Update Readme.md (#189)

* XP-255 update codeowners and authors in setup (#195)

* XP-265 move benchmarks to separate repo (#193)

* XP-243 remove flags from xain-fl codebase

* XP-255 update codeowners and authors in setup (#195)

* XP-257 cleanup cproto dir (#198)

* XP-261 move tests to own dir (#197)

* XP-168 update setup.py (#191)

* DO-35 📰 ✨ Add placeholder for e2e. Update CI for gitflow

* XP-241 remove legacy participant and sdk dir (#199)

* DO-17 🐳 ✨ Add Dockerfiles, dockerignore and docs (#202)

* XP-354 Remove proto files (#200)

* XP-385 Fix docs badge (#204)

* Xp 273 scripts cleanup (#206)

* XP-357 make controller parametrisable (#201)

* XP-384 remove unused files (#209)

* Do 43 docker compose minio (#208)

* XP-374 Clean up docs (#211)

* XP-271 fix pylint issues (#210)

* XP-424 Remove unused packages (#212)

* DO-49 🐳 ✨ Create initial buckets (#213)

* Xp 373 add sdk as dependency in fl (#214)

* XP-433 Fix docker headings (#218)

* XP-119 Fix gRPC testing setup so that it can run on macOS (#217)

* XP-422 ai metrics (#216)

* XP-308: store aggregated weights in S3 buckets (#215)

* XP-436 Reinstate FINISHED heartbeat from Coordinator (#219)

* XP-208 model sum aggregator as a mock for testing

* XP-208 add test integrating P and C

* XP-208 add identity controller as a mock for test

* XP-208 small typo in id-controller docstring

* XP-208 add coordinator test fixture with mocked components

* XP-208 test for top-level function start_participant
* XP-308: print debugging information for spurious failure

* XP-208 join monitor_thread to confirm response to terminate_event

* XP-208 change mock patch string to object called rather than defined

* XP-208 model sum aggregator as a mock for testing

* XP-208 add test integrating P and C

* XP-208 add identity controller as a mock for test

* XP-208 small typo in id-controller docstring

* XP-208 add coordinator test fixture with mocked components

* XP-208 test for top-level function start_participant

* XP-436 fix bug: coordinator not advertising FINISHED state

* XP-436 revert logging level

* XP-208 join monitor_thread to confirm response to terminate_event

* XP-208 change mock patch string to object called rather than defined

* XP-436 suggested documentation fixes from review

* XP-208 mock the store (following rebase)

* XP-208 fix import

* XP-208 test_start_participant: mark as slow, comment on aggregation

* XP-480 revise message names (#222)

* XP-499 Remove conftest, exclude tests folder (#223)

* XP-333 Replace numproto with xain-proto (#220)

* XP-505: docstrings cleanup (#224)

* XP-508 Replace circleci badge (#225)

* XP-510 allow for zero epochs on cli (#227)

* XP-498: more generic shebangs (#229)

* XP-505: cleanup docstrings in xain_fl.coordinator (#228)

* Prepare release of v0.3.0 (matching version with `xain-sdk` and `xain-proto`) (#230)

Co-authored-by: Anastasiia Tymoshchuk <atymoshchuk@icloud.com>
Co-authored-by: Daniel Kravetz <daniel.kravetz@gmail.com>
Co-authored-by: Felix Reichel <31513604+PanicButtonPressed@users.noreply.github.com>
Co-authored-by: Anselmo Sampietro <ans.samp@gmail.com>
Co-authored-by: Robert Steiner <robertt.debug@gmail.com>
Co-authored-by: Corentin Henry <little-dude@users.noreply.github.com>
Co-authored-by: janpetschexain <58227040+janpetschexain@users.noreply.github.com>
Co-authored-by: kwok <kwok.doc@gmail.com>

.dockerignore

@@ -1,143 +1,23 @@
-# Project
-results
-output
-docs
-
-# Git
-.git
-
-# Pantsbuild
-.pants.d
-.pants.workdir.file_lock
-.pants.workdir.file_lock.lock_message
-
-# Byte-compiled / optimized / DLL files
-__pycache__/
-*.py[cod]
-*$py.class
-
-# C extensions
-*.so
-
-# Distribution / packaging
+**/__pycache__
+*,cover
+*.log
+*.pyc
+*.pyd
+*.pyo
 .Python
-build/
-develop-eggs/
-#dist/
-downloads/
-eggs/
-.eggs/
-lib/
-lib64/
-parts/
-sdist/
-var/
-wheels/
-pip-wheel-metadata/
-share/python-wheels/
-*.egg-info/
-.installed.cfg
-*.egg
-MANIFEST
-
-# PyInstaller
-#  Usually these files are written by a python script from a template
-#  before PyInstaller builds the exe, so as to inject date/other infos into it.
-*.manifest
-*.spec
-
-# Installer logs
-pip-log.txt
-pip-delete-this-directory.txt
-
-# Unit test / coverage reports
-htmlcov/
-.tox/
-.nox/
+.cache
 .coverage
 .coverage.*
-.cache
-nosetests.xml
+.git
+.github
+.tox
+Dockerfile
 coverage.xml
-*.cover
-.hypothesis/
-.pytest_cache/
-
-# Translations
-*.mo
-*.pot
-
-# Django stuff:
-*.log
-local_settings.py
-db.sqlite3
-
-# Flask stuff:
-instance/
-.webassets-cache
-
-# Scrapy stuff:
-.scrapy
-
-# Sphinx documentation
-docs/_build/
-
-# PyBuilder
-target/
-
-# Jupyter Notebook
-.ipynb_checkpoints
-
-# IPython
-profile_default/
-ipython_config.py
-
-# pyenv
-.python-version
-
-# pipenv
-#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
-#   However, in case of collaboration, if having platform-specific dependencies or dependencies
-#   having no cross-platform support, pipenv may install dependencies that don’t work, or not
-#   install all needed dependencies.
-#Pipfile.lock
-
-# celery beat schedule file
-celerybeat-schedule
-
-# SageMath parsed files
-*.sage.py
-
-# Environments
-.env
-.venv
-env/
+docs
+env
+nosetests.xml
+pip-delete-this-directory.txt
+pip-log.txt
+scripts
+tests
 venv/
-ENV/
-env.bak/
-venv.bak/
-
-# Spyder project settings
-.spyderproject
-.spyproject
-
-# Rope project settings
-.ropeproject
-
-# mkdocs documentation
-/site
-
-# mypy
-.mypy_cache/
-.dmypy.json
-dmypy.json
-
-# Pyre type checker
-.pyre/
-
-# 
-.DS_Store
-
-# Editors
-.idea
-.vscode

.github/workflows/development-e2e-tests.yml

@@ -0,0 +1,13 @@
+name: Comment-based E2E test
+
+on:
+  pull_request_review_comment:
+    types: [created]
+
+jobs:
+  e2e:
+    runs-on: ubuntu-18.04
+    steps:
+      - name: logging
+        if: github.event.comment.body == '/run e2e'
+        run: echo "$GITHUB_REF"

.github/workflows/development.yml

@@ -0,0 +1,57 @@
+name: Lint and test (development)
+
+on:
+  pull_request:
+    branches:
+      - development
+jobs:
+  development:
+    runs-on: ubuntu-18.04
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+
+      - name: Setup Python 3.6
+        uses: actions/setup-python@v1
+        with:
+          python-version: 3.6
+
+      - name: Cache pip packages
+        uses: actions/cache@v1
+        with:
+          path: ~/.cache/pip
+          key: pip-dev-${{ hashFiles('setup.py') }}
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install --upgrade setuptools
+          pip install -e .[dev]
+
+      - name: black
+        run: |
+          black --line-length 100 --check setup.py xain_fl tests
+
+      - name: isort
+        run: |
+          isort --check-only --indent=4 -rc setup.py xain_fl tests
+
+      - name: pylint
+        run: |
+          pylint --rcfile=.pylintrc xain_fl tests
+
+      - name: mypy
+        continue-on-error: true
+        run: |
+          mypy xain_fl tests
+
+      - name: Sphinx check
+        env:
+          SPHINXOPTS: "-W"
+        run: |
+          cd docs/
+          make docs
+
+      - name: PyTest
+        run: |
+          pytest -v

.github/workflows/main.yml → .github/workflows/master.yml

@@ -1,4 +1,4 @@
-name: Lint and test
+name: Lint and test (master)
 
 on:
   pull_request:
@@ -20,7 +20,7 @@ jobs:
         uses: actions/cache@v1
         with:
           path: ~/.cache/pip
-          key: pip-${{ hashFiles('setup.py') }}
+          key: pip-master-${{ hashFiles('setup.py') }}
 
       - name: Install dependencies
         run: |
@@ -30,22 +30,20 @@ jobs:
 
       - name: black
         run: |
-          black --check --exclude "xain_fl/cproto/.*_pb2.*" \
-          setup.py conftest.py benchmarks examples xain_fl
+          black --line-length 100 --check setup.py xain_fl tests
 
       - name: isort
         run: |
-          isort --check-only --indent=4 -rc setup.py conftest.py benchmarks \
-          examples xain_fl
+          isort --check-only --indent=4 -rc setup.py xain_fl tests
 
       - name: pylint
         run: |
-          pylint --rcfile=pylint.ini benchmarks examples xain_fl
+          pylint --rcfile=.pylintrc xain_fl tests
 
       - name: mypy
         continue-on-error: true
         run: |
-          mypy benchmarks examples/* xain_fl
+          mypy xain_fl tests
 
       - name: Sphinx check
         env:

.gitignore

@@ -1,13 +1,3 @@
-# Project specific files and directories
-config.cfg
-output/*
-results
-
-# Pantsbuild
-.pants.d
-.pants.workdir.file_lock
-.pants.workdir.file_lock.lock_message
-
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[cod]

.isort.cfg

@@ -1,7 +1,8 @@
 [settings]
-multi_line_output=3
-include_trailing_comma=True
 force_grid_wrap=0
+force_sort_within_sections=True
+include_trailing_comma=True
+known_third_party=xain_proto,xain_sdk
+line_length=100
+multi_line_output=3
 use_parentheses=True
-line_length=88
-skip_glob=*_pb2*

.pylintrc

@@ -0,0 +1,19 @@
+[MESSAGES CONTROL]
+
+disable=bad-continuation,redefined-builtin,duplicate-code,fixme
+
+[TYPECHECK]
+
+# List of members which are set dynamically and missed by pylint inference
+# system, and so shouldn't trigger E1101 when accessed. Python regular
+# expressions are accepted.
+# See: https://github.com/pytorch/pytorch/issues/701#issuecomment-378048755
+generated-members=numpy.*,torch.*
+
+# ignore auto generated grpc files
+ignore-patterns=
+
+[MISCELLANEOUS]
+
+# List of note tags to take in consideration, separated by a comma.
+notes=TODO,FIXME

.readthedocs.yml

@@ -14,4 +14,4 @@ python:
       - method: pip
         path: .
         extra_requirements:
-           - docs
+           - docs