Provide a AWS S3 backend for article storage #1295

bschmalhofer · 2021-10-04T13:13:43Z

Currently articles, including attachments, can be stored in the file system and in the database. Allowing storage in S3 would be nice to have. This would also be a good option for running OTOBO in Kubernetes. See also #193 and #1148.

For local development https://hub.docker.com/r/localstack/localstack can be used. There are several S3 modules on CPAN.
Amazon::S3::Thin looks like a sane option, https://metacpan.org/release/AKIYM/Amazon-S3-Thin-0.30/view/lib/Amazon/S3/Thin.pm.

bschmalhofer · 2021-10-07T15:13:42Z

Got started on a S3 backend.

The first finding is that localstack looks usable.

The second finding is that Amazon::S3::Thin has no way to specify the endpoint. The can be worked with some Sub::Override magic. The alternatives to using that module is to use modules that require Moose.

bschmalhofer · 2021-10-07T15:16:26Z

Found an incorrect error message in Kernel/System/Ticket,pm. This can be handled in a separate issue.

bschmalhofer · 2021-10-08T09:05:59Z

Also nice to have: Article backend using OpenStack Swift

bschmalhofer · 2021-10-08T13:15:18Z

The article backend has the feature the the plain text and the attachments of articles are grouped by year, month, and day. This is a nice setup and should be carried over to the S3 backend. This is for consistency than for technical requirements.

…tack

…_service Issue RotherOSS/otobo#1295: add a compose snippet for starting localstack

for different purposes

The prefix needs the trailing delimiter

alos some tidying

bschmalhofer · 2021-10-08T17:08:29Z

https://metacpan.org/release/SCOTTW/Mojolicious-Plugin-AWS-0.21/source/lib/Mojo/AWS.pm looks nice too, but some features are missing.
https://metacpan.org/pod/Paws is a more or less complete SDK, but a bit overwhelming for our purposes

bschmalhofer · 2021-10-13T07:38:31Z

Additional options would be nice to have in the console command Admin::Article::StorageSwitch:

-- source for specifying the source irrespective of the SysConfig
--keep-source for specifying that articles should not be deleted in the source
--update-sysconfig for updating the article storage in the SysConfig

Other inconsistencies might be lurking too when switching storage. Take an attachment that is stored in the source without content disposition. During the switching ArticleAttachment() is called and a default disposition is assigned to the article. This disposition is then stored in the target. All is well as long as the rules for the defaults don't change. If they do change we might get a discrepancy.

Tentatively implement missing methods

bschmalhofer · 2021-11-25T17:05:18Z

Latest run of the test suite still showed the same spurious failures:

Test Summary Report

/opt/otobo/scripts/test/Selenium/Agent/MultiAttachmentUpload/CreateScreens.t (Wstat: 256 Tests: 38 Failed: 1)
Failed test: 38
Non-zero exit status: 1
/opt/otobo/scripts/test/Selenium/TestingMethods.t (Wstat: 0 Tests: 36 Failed: 0)
TODO passed: 21, 24, 28
/opt/otobo/scripts/test/SysConfig/GlobalEffectiveValueGet.t (Wstat: 256 Tests: 3 Failed: 1)
Failed test: 2
Non-zero exit status: 1
/opt/otobo/scripts/test/WebUserAgent.t (Wstat: 1024 Tests: 121 Failed: 4)
Failed tests: 76-77, 89, 101
Non-zero exit status: 4
Files=956, Tests=81627, 7271 wallclock secs (30.34 usr 3.20 sys + 1919.49 cusr 222.26 csys = 2175.29 CPU)
Result: FAIL
ran tests for product OTOBO 10.1.x on host 60b1be1a1c42 .

Might be worthwhile to investigate a bit more. See also #1067.

bschmalhofer · 2021-11-27T13:11:15Z

TODO:

It would be nice if Kernel/System/Ticket/Article/Backend/MIMEBase/ArticleStorageS3.pm would not use internal attributes from the Kernel::System::Storage::S3 object. This includes the attribute S3Object. This might means that Kernel::System::Storage::S3 might need a couple of additional methods.

do not use the S3Object directly

bschmalhofer · 2021-11-28T14:14:11Z

Some new methods have been added to Kernel::System::Storage::S3. These should be covered in scripts/test/Storage/S3.t.

bschmalhofer · 2021-11-28T16:07:25Z

TODO:

fix newly occuring test failures
simplify ArticleAttachmentIndexRaw(), also get FilesizeRaw from the Headers

bschmalhofer · 2021-11-29T11:38:05Z

The test suite is now back to the usual. Merging now.

Test Summary Report

/opt/otobo/scripts/test/Selenium/Agent/MultiAttachmentUpload/CreateScreens.t (Wstat: 256 Tests: 38 Failed: 1)
Failed test: 38
Non-zero exit status: 1
/opt/otobo/scripts/test/Selenium/TestingMethods.t (Wstat: 0 Tests: 36 Failed: 0)
TODO passed: 21, 24, 28
/opt/otobo/scripts/test/SysConfig/GlobalEffectiveValueGet.t (Wstat: 256 Tests: 3 Failed: 1)
Failed test: 2
Non-zero exit status: 1
/opt/otobo/scripts/test/WebUserAgent.t (Wstat: 1024 Tests: 121 Failed: 4)
Failed tests: 76-77, 89, 101
Non-zero exit status: 4
Files=956, Tests=81610, 7403 wallclock secs (30.91 usr 3.27 sys + 1957.41 cusr 229.63 csys = 2221.22 CPU)
Result: FAIL
ran tests for product OTOBO 10.1.x on host 2fbf14cbfc8d .

This still allows parallel processes of the HEAD requests. All interactions with S3 are now wrapped in Kernel::System::Storage::S3. Metadataprefix is now also set up in Kernel/Config.pm

at least kind of, currently only works with '/'

in order to simplify the code

when constructing a S3 key

Issue #1295 hide the s3object

bschmalhofer · 2021-11-29T11:44:01Z

There is still a test failure that needs to be investigated: SysConfig/GlobalEffectiveValueGet.t

bschmalhofer · 2021-11-29T11:48:13Z

S3 article backend is working. New findings get their own issue. Closing this issue.

bschmalhofer added the enhancement New feature or request label Oct 4, 2021

bschmalhofer added this to the OTOBO 10.1 milestone Oct 4, 2021

bschmalhofer self-assigned this Oct 4, 2021

bschmalhofer mentioned this issue Oct 7, 2021

Incorrect error message in Kernel/System/Ticket,pm #1306

Closed

bschmalhofer mentioned this issue Oct 8, 2021

Article backend for OpenStack Swift #1309

Closed

bschmalhofer added a commit to RotherOSS/otobo-docker that referenced this issue Oct 8, 2021

Issue RotherOSS/otobo#1295: add a compose snippet for starting locals…

2abc1d4

…tack

bschmalhofer added a commit to RotherOSS/otobo-docker that referenced this issue Oct 8, 2021

Merge pull request #73 from RotherOSS/issue-rotheross/otobo#1295-edge…

932048d

…_service Issue RotherOSS/otobo#1295: add a compose snippet for starting localstack

bschmalhofer added a commit that referenced this issue Oct 8, 2021

Issue #1295: added a stub for a S3 article storage backend

e6f9c6f

bschmalhofer added a commit that referenced this issue Oct 8, 2021

Issue #1295: tidying, enhance code comments

a9f33a8

bschmalhofer added a commit that referenced this issue Oct 8, 2021

Issue #1295: first version that acutually stores a file

5146f97

bschmalhofer added a commit that referenced this issue Oct 8, 2021

Issue #1295: tidying, do not reuse a variable name

f0f97aa

for different purposes

bschmalhofer added a commit that referenced this issue Oct 8, 2021

Issue #1295: enhance code comments and POD

e27bd21

bschmalhofer added a commit that referenced this issue Oct 8, 2021

Issue #1295: a more consistent set up of $Path

8b8595a

bschmalhofer added a commit that referenced this issue Oct 8, 2021

Issue #1295: start extracting info from the response

f1dd885

The prefix needs the trailing delimiter

bschmalhofer added a commit that referenced this issue Oct 8, 2021

Issue #1295: include the content path in the S3 key

17a1d34

bschmalhofer added a commit that referenced this issue Oct 8, 2021

Issue #1295: enhance code comments

934e23e

bschmalhofer added a commit that referenced this issue Oct 8, 2021

Issue #1295: skip the plain article

d9c2cbe

bschmalhofer added a commit that referenced this issue Oct 8, 2021

Issue #1295: implement ArticlePlain()

bc9a60a

alos some tidying

bschmalhofer added a commit that referenced this issue Oct 12, 2021

Issue #1295: start conversion to Mojo::AWS::S3

90c169c

bschmalhofer added a commit that referenced this issue Oct 12, 2021

Issue #1295: convert ArticleWriteAttachment() to Mojo::AWS::S3

ffc0390

bschmalhofer added a commit that referenced this issue Oct 13, 2021

Merge branch 'rel-10_1' into issue-#1295-aws_s3

e318bad

bschmalhofer added a commit that referenced this issue Oct 13, 2021

Issue #1295: switch to Mojo::AWS::S3

227bc1a

Tentatively implement missing methods

bschmalhofer added a commit that referenced this issue Oct 13, 2021

Issue #1295: remove no longer needed modules

c722011

bschmalhofer added a commit that referenced this issue Oct 13, 2021

Issue #1295: Redis cache might be used too.

ef04d9b

bschmalhofer modified the milestones: OTOBO 10.1, OTOBO 10.1.1 Nov 28, 2021

bschmalhofer mentioned this issue Nov 28, 2021

Cover all methods of Kernel::System::Storage::S3 in test scripts #1457

Closed

bschmalhofer added a commit that referenced this issue Nov 29, 2021

Issue #1295: get config settings later

02749d1

bschmalhofer added a commit that referenced this issue Nov 29, 2021

Issue #1295: introducing the method DiscardObject()

4904c22

bschmalhofer added a commit that referenced this issue Nov 29, 2021

Issue #1295: introducing the method DiscardObjects()

b169332

bschmalhofer added a commit that referenced this issue Nov 29, 2021

Issue #1295: configure MetadataPrefix in Kernel/Config.pm

3e2c65f

bschmalhofer added a commit that referenced this issue Nov 29, 2021

Issue #1295: use end of line condition for 'next'

6e39d47

bschmalhofer added a commit that referenced this issue Nov 29, 2021

Issue #1295: use a more sane test name

1ef1ba7

bschmalhofer added a commit that referenced this issue Nov 29, 2021

Issue #1295: convert test script to Test2::V0

4590f59

bschmalhofer added a commit that referenced this issue Nov 29, 2021

Issue #1295: Fix spelling Filennames becomes Filenames

7623f7d

bschmalhofer added a commit that referenced this issue Nov 29, 2021

Issue #1295: make the delimiter configurable

01624ce

at least kind of, currently only works with '/'

bschmalhofer added a commit that referenced this issue Nov 29, 2021

Issue #1295: store the Filename in the file properties

c0d3e60

bschmalhofer added a commit that referenced this issue Nov 29, 2021

Issue #1295: another round of struggling with trailing slashes

30a625a

bschmalhofer added a commit that referenced this issue Nov 29, 2021

Issue #1295: remove properties that slipped in inadvertently

41933b3

bschmalhofer added a commit that referenced this issue Nov 29, 2021

Issue #1295: fix a copy&paste error

60a6e9e

bschmalhofer added a commit that referenced this issue Nov 29, 2021

Issue #1295: get all properties from the HEAD headers

b275683

in order to simplify the code

bschmalhofer added a commit that referenced this issue Nov 29, 2021

Issue #1295: tidying, post dereferencing

5dbcaf2

bschmalhofer added a commit that referenced this issue Nov 29, 2021

Issue #1295: removed an addition slash

9ab5424

when constructing a S3 key

bschmalhofer added a commit that referenced this issue Nov 29, 2021

Merge pull request #1459 from RotherOSS/issue-#1295-hide_the_s3object

c6af283

Issue #1295 hide the s3object

bschmalhofer mentioned this issue Nov 29, 2021

Fix test failure in SysConfig/GlobalEffectiveValueGet.t #1460

Closed

bschmalhofer closed this as completed Nov 29, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Provide a AWS S3 backend for article storage #1295

Provide a AWS S3 backend for article storage #1295

bschmalhofer commented Oct 4, 2021

bschmalhofer commented Oct 7, 2021

bschmalhofer commented Oct 7, 2021

bschmalhofer commented Oct 8, 2021

bschmalhofer commented Oct 8, 2021

bschmalhofer commented Oct 8, 2021

bschmalhofer commented Oct 13, 2021 •

edited

Loading

bschmalhofer commented Nov 25, 2021 •

edited

Loading

Test Summary Report

bschmalhofer commented Nov 27, 2021 •

edited

Loading

bschmalhofer commented Nov 28, 2021

bschmalhofer commented Nov 28, 2021 •

edited

Loading

bschmalhofer commented Nov 29, 2021

Test Summary Report

bschmalhofer commented Nov 29, 2021

bschmalhofer commented Nov 29, 2021

Provide a AWS S3 backend for article storage #1295

Provide a AWS S3 backend for article storage #1295

Comments

bschmalhofer commented Oct 4, 2021

bschmalhofer commented Oct 7, 2021

bschmalhofer commented Oct 7, 2021

bschmalhofer commented Oct 8, 2021

bschmalhofer commented Oct 8, 2021

bschmalhofer commented Oct 8, 2021

bschmalhofer commented Oct 13, 2021 • edited Loading

bschmalhofer commented Nov 25, 2021 • edited Loading

Test Summary Report

bschmalhofer commented Nov 27, 2021 • edited Loading

bschmalhofer commented Nov 28, 2021

bschmalhofer commented Nov 28, 2021 • edited Loading

bschmalhofer commented Nov 29, 2021

Test Summary Report

bschmalhofer commented Nov 29, 2021

bschmalhofer commented Nov 29, 2021

bschmalhofer commented Oct 13, 2021 •

edited

Loading

bschmalhofer commented Nov 25, 2021 •

edited

Loading

bschmalhofer commented Nov 27, 2021 •

edited

Loading

bschmalhofer commented Nov 28, 2021 •

edited

Loading