File upload to S3 bucket fails #9285
Comments
|
I'm not sure but this may be a ~known issue with Dell's S3 implementation. Working with one of the community members I think we discovered that Dell doesn't correctly handle tags in signed URLs (treating our two tags as one or something like that). I'll try to bring this issue to their attention to get some details and any status on the Dell side. I think we did discuss a possilbe workaround/optional change we could make to support the way Dell currently works but no work has been done afaik. |
|
Hi there .. we tried for literally a year to use OneFS S3 and have still not been able to successfully transfer files to it using direct upload. We worked with a Dell Solutions Specialist and currently have a bug-fix request outstanding waiting for Dell to resolve the current bug that @qqmyers described above. We started on v9.1.0.5 and our first issue was related to bug on the Isilon side where it did not send back the full certificate chain, and then there was another bug where it did not send the correct certificate (the specific cert for our server) but just used the default Isilon cert. Dell sent us simulators to install in a lab environment, and we were able to test with both v 9.1 and v9.2 and see that the bug was present in v9.1 and fixed in v9.2. So, our support team upgraded the Isilon to v9.2.1.0 and we were able to upload files via dataverse successfully using the UI (not direct upload). We could literally see the wrong cert being sent in v9.1 and the correct one sent in v9.2. Next, we realized that when the bucket was configured with direct upload, files would not upload again. After a LOT of testing and making API calls (using the Direct DataFile Upload/Replace API) to examine the errors, with Jim's expert help we identified that the API call to PUT the file to the bucket was not working because Dell was not decoding "%3B" to ";" and the call wouldn't work unless I manually changed the “%3B” to “;” and send the call again .. then the files would upload. The Dell specialist found a related bug bug where the Dell server doesn't percent decode X-Amz-SignedHeaders correctly, which seems to be the related bug. They submitted a bug-fix to Dell but as far as I know they have not yet fixed it. We are upgraded on our OneFS server now to v9.3.0.6 and the problem still exists. Dell recommended that we ask the Dataverse dev team to add code to change the %3B in X-Amz-SignedHeaders with ";" or stop adding x-amz-tagging .. I put an issue in and they were looking into some possible workarounds, but we actually kind of gave up on the Isilon storage and went back to AWS S3. That's our experience .. it doesn't sound like the bugs we were seeing generated the exact same error messages that you are seeing, and you are able to actually copy a file, so I'm not sure if this even matches. We never tried uploading on v9.0 but know that there may be issues if you upgrade. We would be very interested to hear if you are able to successfully utilize the Dell storage! =) |
|
We are currently using OneFS 9.1.0.12. Unfortunately, I am dependent on a third party vendor and do not have the ability to update. But for the future I have mentioned it. @nightowlaz I can confirm your experience with direct upload, decoding in header solves the problem. Problematic for me is still the checksum, which is not generated correctly for a file. If I upload a file directly via the api I get the correct checksum. About the provided link I get a value that I can not explain. I get a sum with consecutive zeros with 32 characters. The checksum is not static and seems to depend on the key/object. Has anyone had any experience with this? |
|
For direct upload, the checksum is calculated by the sender. In the UI (and dvwebloader), that is JavaScript in the browser. For DVUploader, it is in Java. If something is happening in the UI, the browser console may have more info about any error. In both cases, the checksum is sent to Dataverse and the S3 bucket does not have a checksum per se (i.e. fixity is calculated per part in multi-part uploads, etc.) I haven't tried it, but I'd expect it is possible to create a proxy that could decode the signed URL. It also shouldn't be technically hard to provide a work-around in the Dataverse code. |
|
@qqmyers By direct upload I meant uploading directly to an S3 bucket via the signed URL... was perhaps a bit imprecisely expressed. The encoding problem should not be a problem, I think too. |
|
So, you mean, using the API you got the same encoding issue and were able to fix it by changing the code in the response from the server and sending it? Does the direct upload from the UI work? We assumed that the reason the upload from the UI didn't work either when using direct upload was the same issue, and to fix that it has to be fixed in the core code (I think). Or, were you able to fix it and make it work in the UI as well? |
|
I think the approaches work independently. Upload via UI does not work and leads to the initial problem. Direct upload into bucket and direct download from bucket via Dataverse API works (except checksum problem), as long as the |
|
Okay, great, that is the same as our experience. We needed to be able to utilize direct upload via the UI so were never able to actually enable the Isilon storage. We didn't get far enough to experience the checksum issue, so good to know that is a thing, too. |
|
@golsch - The only place a checksum is used is in https://guides.dataverse.org/en/latest/developers/s3-direct-upload-api.html#adding-the-uploaded-file-to-the-dataset and in the API you are responsible for creating it. The only thing related to fixity coming from S3 itself are the ETags in a multipart upload. Are those what are incorrect? |
|
FWIW: Making the UI work could be handled two ways: change the output of the API call to request the URL(s) or edit the fileupload.js script used in the UI (for the single part call and the multipart Ajax calls) . Both are in the Dataverse codebase but the JavaScript change could be done in the expanded war file on disk as a local fix. |
|
@qqmyers would you be willing to provide instructions for doing so? Would be good to test to see if that is the only issue, and if fixing that would indeed resolve the issue in the UI. |
|
I'd suggest replacing with url: this.urls.url.replace('%3B',';'), . I think that should work for the single part case. (I haven't tried this to even see if I have a typo.) I think the multipart case can't be handled this way - it's possible it just works as the tag is added in a call made by Dataverse rather than from JavaScript so the escaping could be correct already. In any case, if the above fixes the single part case, it's useful info. If this really is the only thing, and multipart doesn't work, getting a small fix into 5.13 for Dell could probably be done.
|
|
@qqmyers Uploading a file directly via the UI works fine with the fix. I'm not quite clear how to trigger multipart via the UI. Edit: Multipart via UI seems to work as well, the file is uploaded in parts and can be downloaded. My default Edit: The checksum problem has magically disappeared. Edit: I'm not that familiar with S3, but why is the |
|
Multipart is automatic when the file size is over the part size configured for the store. So any file over 5 MB in your case. (The browser console should let you confirm this.) The x-amz-tagging is used for multipart, but it is done by the Dataverse Java software prior to sending the browser the signed URLs to upload the parts/ for multipart, the Dataverse server makes the calls to start and end the multipart upload and the browser only sends the parts. For Dell, this means it could be that multipart would have a problem with the tag that can't be fixed in the JavaScript, but it is also possible that the Java code is not sending %3B in the first place. |
|
Small addendum to the upload to S3 via Dataverse. Edit: It works! Maybe there is a possibility to include the sum in Dataverse when uploading. |
Great! Closing.
You are very welcome to open a fresh issue for this. |
What steps does it take to reproduce the issue?
I am using a fresh instance and PowerScale OneFS 9.0 as the backend.
The bucket is configured as follows:
As soon as I want to load a text file via HTTP upload, there is an error on the part of Dataverse, but the file is successfully uploaded to the bucket:
Which version of Dataverse are you using?
Dataverse v5.12.1
The text was updated successfully, but these errors were encountered: