Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Workflow endpoint of the Processing Server #1083

Merged
merged 97 commits into from
Oct 11, 2023
Merged

Workflow endpoint of the Processing Server #1083

merged 97 commits into from
Oct 11, 2023

Commits on Aug 31, 2023

  1. First version of the workflow endpoint

    @joschrew
    joschrew committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    439dde4 View commit details
    Browse the repository at this point in the history

Commits on Sep 7, 2023

  1. fix locking for cached requests

    @MehmedGIT
    MehmedGIT committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    f84286c View commit details
    Browse the repository at this point in the history

  2. improve code from kba feedback

    @MehmedGIT
    MehmedGIT committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    c1466d9 View commit details
    Browse the repository at this point in the history

  3. improve run_workflow method

    @MehmedGIT
    MehmedGIT committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    8d3ea65 View commit details
    Browse the repository at this point in the history

  4. support page_id in run_workflow

    @MehmedGIT
    MehmedGIT committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    1505445 View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2023

  1. cancel dependent jobs of failed jobs

    @MehmedGIT
    MehmedGIT committed Sep 11, 2023
    Configuration menu
    Copy the full SHA
    aad8604 View commit details
    Browse the repository at this point in the history

  2. Add new job state - cancelled

    @MehmedGIT
    MehmedGIT committed Sep 11, 2023
    Configuration menu
    Copy the full SHA
    26b1253 View commit details
    Browse the repository at this point in the history

  3. return models for update methods

    @MehmedGIT
    MehmedGIT committed Sep 11, 2023
    Configuration menu
    Copy the full SHA
    e9880e8 View commit details
    Browse the repository at this point in the history

  4. include cancelled jobs in db

    @MehmedGIT
    MehmedGIT committed Sep 11, 2023
    Configuration menu
    Copy the full SHA
    eefcfcc View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2023

  1. Merge branch 'master' into workflow-endpoint 

    Merge the latest master state
    @MehmedGIT
    MehmedGIT committed Sep 12, 2023
    Configuration menu
    Copy the full SHA
    fb7b560 View commit details
    Browse the repository at this point in the history

  2. Improve consumed message debug log

    @MehmedGIT
    MehmedGIT committed Sep 12, 2023
    Configuration menu
    Copy the full SHA
    af45bf9 View commit details
    Browse the repository at this point in the history

  3. fix bug: wrong job_id assigned

    @MehmedGIT
    MehmedGIT committed Sep 12, 2023
    Configuration menu
    Copy the full SHA
    0baee30 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' into workflow-endpoint

    @MehmedGIT
    MehmedGIT committed Sep 12, 2023
    Configuration menu
    Copy the full SHA
    adcc683 View commit details
    Browse the repository at this point in the history

  5. improve code: locking/unlocking

    @MehmedGIT
    MehmedGIT committed Sep 12, 2023
    Configuration menu
    Copy the full SHA
    0725ed0 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'master' into workflow-endpoint

    @MehmedGIT
    MehmedGIT committed Sep 12, 2023
    Configuration menu
    Copy the full SHA
    775beb6 View commit details
    Browse the repository at this point in the history

  7. remove unnecessary stuff

    @MehmedGIT
    MehmedGIT committed Sep 12, 2023
    Configuration menu
    Copy the full SHA
    9b250ef View commit details
    Browse the repository at this point in the history

  8. alpha: utilize mets server

    @MehmedGIT
    MehmedGIT committed Sep 12, 2023
    Configuration menu
    Copy the full SHA
    13154c2 View commit details
    Browse the repository at this point in the history

  9. Merge branch 'master' into workflow-endpoint

    @MehmedGIT
    MehmedGIT committed Sep 12, 2023
    Configuration menu
    Copy the full SHA
    551741a View commit details
    Browse the repository at this point in the history

  10. add mets url to run_cli

    @MehmedGIT
    MehmedGIT committed Sep 12, 2023
    Configuration menu
    Copy the full SHA
    2373537 View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2023

  1. Remove already adressed notes

    @joschrew
    joschrew committed Sep 13, 2023
    Configuration menu
    Copy the full SHA
    7eb7e76 View commit details
    Browse the repository at this point in the history

  2. fix: mets server related issues

    @MehmedGIT
    MehmedGIT committed Sep 13, 2023
    Configuration menu
    Copy the full SHA
    6cbdd05 View commit details
    Browse the repository at this point in the history

  3. fix: exception -> info

    @MehmedGIT
    MehmedGIT committed Sep 13, 2023
    Configuration menu
    Copy the full SHA
    c9fd739 View commit details
    Browse the repository at this point in the history

  4. stop mets server when the cache queue is empty

    @MehmedGIT
    MehmedGIT committed Sep 13, 2023
    Configuration menu
    Copy the full SHA
    6c6251c View commit details
    Browse the repository at this point in the history

Commits on Sep 14, 2023

  1. Merge branch 'master' into workflow-endpoint

    @MehmedGIT
    MehmedGIT committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    9e01a25 View commit details
    Browse the repository at this point in the history

  2. refactor process helpers

    @MehmedGIT
    MehmedGIT committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    aa03c96 View commit details
    Browse the repository at this point in the history

  3. remove chmod from mets server deployer

    @MehmedGIT
    MehmedGIT committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    837db61 View commit details
    Browse the repository at this point in the history

  4. fix: do not pass the logger

    @MehmedGIT
    MehmedGIT committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    0096625 View commit details
    Browse the repository at this point in the history

Commits on Sep 15, 2023

  1. implement page-wise processing

    @MehmedGIT
    MehmedGIT committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    0de15ed View commit details
    Browse the repository at this point in the history

  2. PYJobOutput returns more data

    @MehmedGIT
    MehmedGIT committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    51c44db View commit details
    Browse the repository at this point in the history

  3. feature: multiple job dependencies possible

    @MehmedGIT
    MehmedGIT committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    3c3fbb6 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' into workflow-endpoint

    @MehmedGIT
    MehmedGIT committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    3e9e11c View commit details
    Browse the repository at this point in the history

  5. integrate the mets server feature back

    @MehmedGIT
    MehmedGIT committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    a8eb261 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2023

  1. adapt feedback

    @MehmedGIT
    MehmedGIT committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    57be596 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2023

  1. wait before stopping server

    @MehmedGIT
    MehmedGIT committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    d099445 View commit details
    Browse the repository at this point in the history

  2. feature: more intuitive workflow output

    @MehmedGIT
    MehmedGIT committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    33f2d40 View commit details
    Browse the repository at this point in the history

  3. allow page_wise without page_id

    @MehmedGIT
    MehmedGIT committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    206acf6 View commit details
    Browse the repository at this point in the history

  4. fix: notes

    @MehmedGIT
    MehmedGIT committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    1a54563 View commit details
    Browse the repository at this point in the history

  5. change: return PYJobOutput again

    @MehmedGIT
    MehmedGIT committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    22edc3a View commit details
    Browse the repository at this point in the history

  6. utilize dependency cache

    @MehmedGIT
    MehmedGIT committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    caf12f3 View commit details
    Browse the repository at this point in the history

  7. reduce debug logs

    @MehmedGIT
    MehmedGIT committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    5727e82 View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2023

  1. feature: request counter

    @MehmedGIT
    MehmedGIT committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    6733a87 View commit details
    Browse the repository at this point in the history

  2. refactor and fix internal counter

    @MehmedGIT
    MehmedGIT committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    a785995 View commit details
    Browse the repository at this point in the history

  3. adapt from feedback

    @MehmedGIT
    MehmedGIT committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    2fe1e13 View commit details
    Browse the repository at this point in the history

  4. fix: mets server shutdown

    @MehmedGIT
    MehmedGIT committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    f07574f View commit details
    Browse the repository at this point in the history

  5. refactor: page locking/unlocking

    @MehmedGIT
    MehmedGIT committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    1bf124d View commit details
    Browse the repository at this point in the history

  6. OcrdMets.physical_pages should return "proper" str

    @kba
    kba committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    9b026c1 View commit details
    Browse the repository at this point in the history

  7. add TODOs

    @MehmedGIT
    MehmedGIT committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    50c7eb7 View commit details
    Browse the repository at this point in the history

  8. add todo for #1102

    @MehmedGIT
    MehmedGIT committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    df15f8b View commit details
    Browse the repository at this point in the history

  9. fix: stop mets server, get -> delete

    @MehmedGIT
    MehmedGIT committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    941abd0 View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2023

  1. merge 1102

    @MehmedGIT
    MehmedGIT committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    1e558bc View commit details
    Browse the repository at this point in the history

  2. adapt after merging #1102

    @MehmedGIT
    MehmedGIT committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    27700f5 View commit details
    Browse the repository at this point in the history

  3. utilize as a key again

    @MehmedGIT
    MehmedGIT committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    a968798 View commit details
    Browse the repository at this point in the history

  4. separate page locking cache

    @MehmedGIT
    MehmedGIT committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    c8320d2 View commit details
    Browse the repository at this point in the history

  5. fix: all_pages placeholder

    @MehmedGIT
    MehmedGIT committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    5408b75 View commit details
    Browse the repository at this point in the history

  6. separate request cache

    @MehmedGIT
    MehmedGIT committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    6d1d797 View commit details
    Browse the repository at this point in the history

  7. improve: deployer logs separately

    @MehmedGIT
    MehmedGIT committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    be33ffd View commit details
    Browse the repository at this point in the history

  8. log processing server to files

    @MehmedGIT
    MehmedGIT committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    3425121 View commit details
    Browse the repository at this point in the history

  9. separate cache logging

    @MehmedGIT
    MehmedGIT committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    8cb9953 View commit details
    Browse the repository at this point in the history

  10. use direct access to cache

    @MehmedGIT
    MehmedGIT committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    f96dea6 View commit details
    Browse the repository at this point in the history

  11. fix: RMQ consumer set QoS

    @MehmedGIT
    MehmedGIT committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    224e468 View commit details
    Browse the repository at this point in the history

  12. fix: iteration buggy state

    @MehmedGIT
    MehmedGIT committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    973f2de View commit details
    Browse the repository at this point in the history

  13. fix: recent update typo

    @MehmedGIT
    MehmedGIT committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    56dd312 View commit details
    Browse the repository at this point in the history

  14. fix cache: check context switch removals

    @MehmedGIT
    MehmedGIT committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    81bb1ac View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2023

  1. raise exception when missing

    @MehmedGIT
    MehmedGIT committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    64b3563 View commit details
    Browse the repository at this point in the history

  2. fix bug: overwriting FAILED with CANCELLED

    @MehmedGIT
    MehmedGIT committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    52e9a73 View commit details
    Browse the repository at this point in the history

  3. make more compact 

    Update ocrd/ocrd/processor/base.py

Co-authored-by: Konstantin Baierer <kba@users.noreply.github.com>
    @MehmedGIT @kba
    MehmedGIT and kba authored Sep 26, 2023
    Configuration menu
    Copy the full SHA
    23e2318 View commit details
    Browse the repository at this point in the history

  4. Merge pull request #1104 from OCR-D/fix-1103-raise-excp 

    Fix issue #1103
    @MehmedGIT
    MehmedGIT authored Sep 26, 2023
    Configuration menu
    Copy the full SHA
    36cf428 View commit details
    Browse the repository at this point in the history

  5. fix error from PR #1104

    @MehmedGIT
    MehmedGIT committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    0625050 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'master' into workflow-endpoint

    @kba
    kba committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    6683e96 View commit details
    Browse the repository at this point in the history

  7. logging: simulate ocrd_network logging behavior with ocrd_utils.logging

    @kba
    kba committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    7c98254 View commit details
    Browse the repository at this point in the history

  8. fix tests

    @kba
    kba committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    5c520a3 View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2023

  1. Update ocrd/ocrd/mets_server.py 

    Co-authored-by: Mehmed Mustafa <mehmed.n.mustafa@gmail.com>
    @kba @MehmedGIT
    kba and MehmedGIT authored Sep 27, 2023
    Configuration menu
    Copy the full SHA
    a62a5e2 View commit details
    Browse the repository at this point in the history

  2. adapt rabbitmq loggers

    @MehmedGIT
    MehmedGIT committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    2f50ee5 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'workflow-endpoint-logging-2023-09-26' of github.com:OCR… 

    …-D/core into workflow-endpoint-logging-2023-09-26
    @MehmedGIT
    MehmedGIT committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    e911d81 View commit details
    Browse the repository at this point in the history

  4. improve: worker logging suffix

    @MehmedGIT
    MehmedGIT committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    76ffff0 View commit details
    Browse the repository at this point in the history

  5. add uvicorn loggers

    @MehmedGIT
    MehmedGIT committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    eebc5cd View commit details
    Browse the repository at this point in the history

  6. remove deployer log for processor servers

    @MehmedGIT
    MehmedGIT committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    635d909 View commit details
    Browse the repository at this point in the history

  7. call initLogging in network modules

    @MehmedGIT
    MehmedGIT committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    b52a388 View commit details
    Browse the repository at this point in the history

  8. remove test_logging

    @MehmedGIT
    MehmedGIT committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    7642ffb View commit details
    Browse the repository at this point in the history

  9. fix: starting processor server

    @MehmedGIT
    MehmedGIT committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    e666049 View commit details
    Browse the repository at this point in the history

  10. clean old notices

    @MehmedGIT
    MehmedGIT committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    986f9a0 View commit details
    Browse the repository at this point in the history

  11. remove dupl handler

    @MehmedGIT
    MehmedGIT committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    96da69d View commit details
    Browse the repository at this point in the history

  12. synchronize servers impl

    @MehmedGIT
    MehmedGIT committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    f1ee09e View commit details
    Browse the repository at this point in the history

  13. pass log_level to processors

    @MehmedGIT
    MehmedGIT committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    585212c View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2023

  1. Add endpoint to query workflow status

    joschrew committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    e6ce39e View commit details
    Browse the repository at this point in the history

  2. Improve error handling for workflow requests

    joschrew committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    0182352 View commit details
    Browse the repository at this point in the history

Commits on Oct 5, 2023

  1. Don't return processing_job_ids on workflow execution 

    To not generate too much output

Co-authored-by: Mehmed Mustafa <mehmed.n.mustafa@gmail.com>
    @joschrew @MehmedGIT
    joschrew and MehmedGIT authored Oct 5, 2023
    Configuration menu
    Copy the full SHA
    d7e2570 View commit details
    Browse the repository at this point in the history

  2. Add processing_job_ids of faild processor to workflow-status 

    Co-authored-by: Mehmed Mustafa <mehmed.n.mustafa@gmail.com>
    @joschrew @MehmedGIT
    joschrew and MehmedGIT authored Oct 5, 2023
    Configuration menu
    Copy the full SHA
    deb1188 View commit details
    Browse the repository at this point in the history

  3. Update coded of previous 2 commits

    @joschrew
    joschrew committed Oct 5, 2023
    Configuration menu
    Copy the full SHA
    04a326b View commit details
    Browse the repository at this point in the history

  4. Add page_id to workflow status failed jobs

    @joschrew
    joschrew committed Oct 5, 2023
    Configuration menu
    Copy the full SHA
    0e43e72 View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2023

  1. ocrd_utils.redirect_stderr_and_stdout_to_file

    @kba
    kba committed Oct 10, 2023
    Configuration menu
    Copy the full SHA
    00373c9 View commit details
    Browse the repository at this point in the history

  2. implement --log-filename, wip

    @kba
    kba committed Oct 10, 2023
    Configuration menu
    Copy the full SHA
    6fc6060 View commit details
    Browse the repository at this point in the history

  3. Remove deployed_* log files

    @MehmedGIT
    MehmedGIT committed Oct 10, 2023
    Configuration menu
    Copy the full SHA
    1c9f3d7 View commit details
    Browse the repository at this point in the history

  4. Processor stdout/stderr to ocrd_worker_* logs

    @MehmedGIT
    MehmedGIT committed Oct 10, 2023
    Configuration menu
    Copy the full SHA
    b5fa221 View commit details
    Browse the repository at this point in the history

  5. move initLogging to within stdout/stderr redirect

    @kba
    kba committed Oct 10, 2023
    Configuration menu
    Copy the full SHA
    ff2ee9c View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2023

  1. invoke_processor: force initLogging because sys.stdout change

    @kba
    kba committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    d253492 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/workflow-additions' into workflo… 

    …w-endpoint
    @kba
    kba committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    155ea2a View commit details
    Browse the repository at this point in the history