Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kubernetes: Store Kernel/Config/Files/*.pm files in S3 storage #1320

Closed
bschmalhofer opened this issue Oct 14, 2021 · 7 comments
Closed

Kubernetes: Store Kernel/Config/Files/*.pm files in S3 storage #1320

bschmalhofer opened this issue Oct 14, 2021 · 7 comments
Assignees
@bschmalhofer
Labels
enhancement New feature or request
Milestone
OTOBO 10.1.1

Comments

@bschmalhofer
Copy link
Contributor

See the discussion in #1148.
The goal for this issue is to provide a minimal solution. In Kernel::Config::new() we sync the objects in the S3 storage to the Kernel/Config/Files directory. Then go through the features that write *.pm files and make them write into the S3 storage.
@bschmalhofer bschmalhofer added the enhancement New feature or request label Oct 14, 2021
@bschmalhofer bschmalhofer added this to the OTOBO 10.1 milestone Oct 14, 2021
@bschmalhofer bschmalhofer self-assigned this Oct 14, 2021
@bschmalhofer
Copy link
Contributor Author

bschmalhofer commented Oct 20, 2021 •
edited
Loading

For the progress see also #1295.

Another thing to do:

  • Collect the code into an system module like e.g. `Kernel::System::Storage::S3'.

This can be done in the branch for this issue, and then, in #1295, the module ArticleStorageFS can be migrated to the new module.

@bschmalhofer bschmalhofer mentioned this issue Oct 20, 2021
Closed
bschmalhofer added a commit that referenced this issue Oct 21, 2021
@bschmalhofer
Issue #1320: explicitly requires Mojolicious for S3 support
f61ef5e 
Before it only was implicitly.
bschmalhofer added a commit that referenced this issue Oct 21, 2021
@bschmalhofer
Issue #1320: support for syncing ZZZ*.pm files from S3
73e165c
bschmalhofer added a commit that referenced this issue Oct 21, 2021
@bschmalhofer
Issue #1320: check whether Kernel/Config/Files is a dir
13d67a9
bschmalhofer added a commit that referenced this issue Oct 21, 2021
@bschmalhofer
Issue #1320: tidying
3cfe80f 
mostly removing unneeded parenthensis
bschmalhofer added a commit that referenced this issue Oct 21, 2021
@bschmalhofer
Merge pull request #1343 from RotherOSS/issue-#1320-more_S3_support
b61d1d0 
Issue #1320 more s3 support
bschmalhofer added a commit that referenced this issue Oct 21, 2021
@bschmalhofer
Issue #1320: do not include loader files in the image
84c3945 
loader files can exist when building from a running installation
@bschmalhofer
Copy link
Contributor Author

bschmalhofer commented Oct 22, 2021 •
edited
Loading

TODO:

bschmalhofer added a commit that referenced this issue Oct 22, 2021
@bschmalhofer
Issue #1320: update POD
0ee2985
bschmalhofer added a commit that referenced this issue Oct 22, 2021
@bschmalhofer
Issue #1320: remove unused variables
daa18f2
bschmalhofer added a commit that referenced this issue Oct 22, 2021
@bschmalhofer
Issue #1320: Use ResultType as a variable
488c69d 
Not using hash lookups has the advantage that spelling errors in the hash keys cannot occur.
bschmalhofer added a commit that referenced this issue Oct 22, 2021
@bschmalhofer
Issue #1320: do not dump to file after dumping to S3
149673d
bschmalhofer added a commit that referenced this issue Oct 22, 2021
@bschmalhofer
Issue #1320: reduce the level of nesting in ProcessDump()
18449b1
bschmalhofer added a commit that referenced this issue Oct 22, 2021
@bschmalhofer
Issue #1320: do not write ZZZAAuto.pm into the file system
d2f2421 
when S3 is active
bschmalhofer added a commit that referenced this issue Oct 22, 2021
@bschmalhofer
Issue #1320: update the POD
73e119c 
only the ResultType 'FILE' is supported
bschmalhofer added a commit that referenced this issue Oct 22, 2021
@bschmalhofer
Issue #1320: only dump the ACLs into S3
1913229 
do not write the file into the file system when S3 is active
bschmalhofer added a commit that referenced this issue Oct 22, 2021
@bschmalhofer
Merge pull request #1355 from RotherOSS/issue-#1320-s3_only
9b8d567 
Issue #1320 s3 only
@bschmalhofer
Copy link
Contributor Author

Ah well, simply downloading from S3 is problematic. There are different processes reading these files. This means that creation must be atomic and that probably write access must be synchronised.

bschmalhofer added a commit that referenced this issue Oct 23, 2021
@bschmalhofer
Issue #1320: do not bundle *.pm in Kernel/Config/Files
08a87ac 
These exist only when building from a local installation.
bschmalhofer added a commit that referenced this issue Oct 23, 2021
@bschmalhofer
Issue #1320: add code comments
93e00b6 
trailing slashes are tricky
bschmalhofer added a commit that referenced this issue Oct 23, 2021
@bschmalhofer
Issue #1320: make the sync of ZZZ*.pm files actually work
804f034 
make sure that downloads from S3 are not concurrent
check size and mtime again after download
bschmalhofer added a commit that referenced this issue Oct 23, 2021
@bschmalhofer
Merge pull request #1358 from RotherOSS/issue-#1320-working_by_accide…
03f1c03 
…nt_take_2

Issue #1320 working by accident take 2
@bschmalhofer
Copy link
Contributor Author

After some fixes we are down to just a few failures in the test suite:

/opt/otobo/scripts/test/Console/Command/Maint/Config/Rebuild.t (Wstat: 256 Tests: 2 Failed: 1)
Failed test: 2
Non-zero exit status: 1
/opt/otobo/scripts/test/Loader.t (Wstat: 65280 Tests: 7 Failed: 0)
Non-zero exit status: 255
Parse errors: No plan found in TAP output
/opt/otobo/scripts/test/PostMaster/CommunicationLog.t (Wstat: 2048 Tests: 377 Failed: 8)
Failed tests: 26, 62, 151, 175, 214, 250, 339, 363
Non-zero exit status: 8
/opt/otobo/scripts/test/Selenium/Agent/MultiAttachmentUpload/CreateScreens.t (Wstat: 256 Tests: 38 Failed: 1)
Failed test: 38
Non-zero exit status: 1
/opt/otobo/scripts/test/SysConfig/ConfigurationDeploy.t (Wstat: 65280 Tests: 10 Failed: 0)
Non-zero exit status: 255
Parse errors: No plan found in TAP output

@bschmalhofer
Copy link
Contributor Author

Noticed that some parts Kernel::System::SysConfig::ConfigurationDeploy() assume that the current dir is /opt/otobo. Futhermore the code assumes that Kernel/Config exists when it found that Kernel/Config/Files is missing. This should be rectivied.

@bschmalhofer bschmalhofer mentioned this issue Nov 15, 2021
Closed
bschmalhofer added a commit that referenced this issue Nov 15, 2021
@bschmalhofer
Issue #1320: Return Success => 1 in ConfigurationDeploy,
2ccf25b 
when storing the object in S3 was successful.
No need to check existence of Kernel/Config/Files when using S3.
bschmalhofer added a commit that referenced this issue Nov 15, 2021
@bschmalhofer
Issue #1320: set UTF-8 flag on incoming UTF-8 content
73844b7
bschmalhofer added a commit that referenced this issue Nov 15, 2021
@bschmalhofer
Issue #1320: convert to Test2::V0 and use subtests
9595b26
bschmalhofer added a commit that referenced this issue Nov 15, 2021
@bschmalhofer
Issue #1320: tidying
380f236
bschmalhofer added a commit that referenced this issue Nov 15, 2021
@bschmalhofer
Issue #1320: allow for special file name in SyncWithS3()
79d101b 
Used only for unittests.
bschmalhofer added a commit that referenced this issue Nov 15, 2021
@bschmalhofer
Issue #1320: support overridded file name in the S3 case
ffd4979
bschmalhofer added a commit that referenced this issue Nov 15, 2021
@bschmalhofer
Issue #1320: sync the config files
896edce 
from S3 to the file system
bschmalhofer added a commit that referenced this issue Nov 15, 2021
@bschmalhofer
Issue #1320: add check for the perl config cache file
76f0cd4
bschmalhofer added a commit that referenced this issue Nov 15, 2021
@bschmalhofer
Issue #1320: fix test of the deployment ID
5122a54 
should be defined, not undefined
bschmalhofer added a commit that referenced this issue Nov 15, 2021
@bschmalhofer
Issue #1320: use Test2::V0 and subtests
2c6bcd0
bschmalhofer added a commit that referenced this issue Nov 15, 2021
@bschmalhofer
Issue #1320: call SyncWithS3()
b947799 
including a small sleep, in order to have different modification times
bschmalhofer added a commit that referenced this issue Nov 15, 2021
@bschmalhofer
Issue #1320: satisfy the code policy
a2f8248
bschmalhofer added a commit that referenced this issue Nov 15, 2021
@bschmalhofer
Merge pull request #1412 from RotherOSS/issue-#1320-success_1
e8738d6 
Issue #1320 success 1
@bschmalhofer
Copy link
Contributor Author

Some more fixed and test results look a bit better:

Test Summary Report

/opt/otobo/scripts/test/Selenium/Agent/Admin/AdminCustomerGroup.t (Wstat: 513 Tests: 38 Failed: 2)
Failed tests: 37-38
Non-zero exit status: 2
/opt/otobo/scripts/test/Selenium/Agent/AgentTicketSplitInlineImage.t (Wstat: 768 Tests: 33 Failed: 3)
Failed tests: 30, 32-33
Non-zero exit status: 3
/opt/otobo/scripts/test/Selenium/Agent/MultiAttachmentUpload/CreateScreens.t (Wstat: 256 Tests: 38 Failed: 1)
Failed test: 38
Non-zero exit status: 1
/opt/otobo/scripts/test/Selenium/TestingMethods.t (Wstat: 0 Tests: 36 Failed: 0)
TODO passed: 21, 24, 28
/opt/otobo/scripts/test/SysConfig/GlobalEffectiveValueGet.t (Wstat: 256 Tests: 3 Failed: 1)
Failed test: 2
Non-zero exit status: 1
/opt/otobo/scripts/test/WebUserAgent.t (Wstat: 1024 Tests: 121 Failed: 4)
Failed tests: 76-77, 89, 101
Non-zero exit status: 4
Files=956, Tests=81665, 7546 wallclock secs (31.15 usr 3.61 sys + 1962.56 cusr 236.30 csys = 2233.62 CPU)
Result: FAIL
ran tests for product OTOBO 10.1.x on host abb5d5f0a212 .

Some of the failures seem to be sporadic. AgentTicketSplitInlineImage.t and _CreateScreens.t _ seem to be related to S3.

@bschmalhofer
Copy link
Contributor Author

This seems to be working. Closing this issue.

@bschmalhofer bschmalhofer modified the milestones: OTOBO 10.1, OTOBO 10.1.1 Nov 29, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bschmalhofer bschmalhofer

Labels
enhancement New feature or request
Projects
None yet
Milestone
OTOBO 10.1.1
Development

No branches or pull requests
1 participant
@bschmalhofer