Jobs cannot run migrations

[leafty]

Logs:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/app/env/lib/python3.12/site-packages/renku_data_services/background_jobs/main.py", line 53, in <module>
    asyncio.run(main())
  File "/usr/local/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/app/env/lib/python3.12/site-packages/renku_data_services/background_jobs/main.py", line 49, in main
    await args.func()
  File "/app/env/lib/python3.12/site-packages/renku_data_services/background_jobs/main.py", line 19, in short_period_sync
    run_migrations_for_app("common")
  File "/app/env/lib/python3.12/site-packages/renku_data_services/migrations/core.py", line 20, in run_migrations_for_app
    command.upgrade(cfg, revision)
  File "/app/env/lib/python3.12/site-packages/alembic/command.py", line 403, in upgrade
    script.run_env()
  File "/app/env/lib/python3.12/site-packages/alembic/script/base.py", line 583, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/app/env/lib/python3.12/site-packages/alembic/util/pyfiles.py", line 95, in load_python_file
    module = load_module_py(module_id, path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/env/lib/python3.12/site-packages/alembic/util/pyfiles.py", line 113, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/app/env/lib/python3.12/site-packages/renku_data_services/migrations/env.py", line 46, in <module>
    run_migrations(all_metadata)
  File "/app/env/lib/python3.12/site-packages/renku_data_services/migrations/utils.py", line 158, in run_migrations
    run_migrations_online(metadata, sync_sqlalchemy_url)
  File "/app/env/lib/python3.12/site-packages/renku_data_services/migrations/utils.py", line 146, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/app/env/lib/python3.12/site-packages/alembic/runtime/environment.py", line 948, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/app/env/lib/python3.12/site-packages/alembic/runtime/migration.py", line 627, in run_migrations
    step.migration_fn(**kw)
  File "/app/env/lib/python3.12/site-packages/renku_data_services/migrations/versions/f34b87ddd954_migrate_namespaces_to_authzed.py", line 33, in upgrade
    app_config = AppConfig.from_env()
                 ^^^^^^^^^^^^^^^^^^^^
  File "/app/env/lib/python3.12/site-packages/renku_data_services/app_config/config.py", line 445, in from_env
    encryption_key = Path(encryption_key_path).read_bytes()
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/pathlib.py", line 1019, in read_bytes
    with self.open(mode='rb') as f:
         ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/pathlib.py", line 1013, in open
    return io.open(self, mode, buffering, encoding, errors, newline)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/encryption-key'

[leafty]

This means that currently, only data-services can run the f34b87ddd954_migrate_namespaces_to_authzed.py migration.

[leafty]

Suggestion: only run a check in jobs and panic if the database is not up to date.

[olevski]

@leafty this occurs in production? Can you provide some info about how you triggered it or where you saw it?

[leafty]

@leafty this occurs in production? Can you provide some info about how you triggered it or where you saw it?

This can happen if you need to run the f34b87ddd954 migration but data-services does not run (e.g. crashes on start or replicas set to 0). Then the migration is attempted from the cron job and fails.