[HOLD for payment 2023-09-06] [$1000] Android - Error attempting to download an attachment

[izarutskaya]

If you haven’t already, check out our contributing guidelines for onboarding and email contributors@expensify.com to request to join our Slack channel!

---

Action Performed:

1. Open a random report and send an attachment (image, video, etc).
2. If it's an image, click on the image, and when the image opens in full view, click on the download button. If it's a video or a different type of attachment, directly click the download button.
3. Observe the error, and the attachment doesn't download.

Expected Result:

The attachment downloads successfully to the device without an error.

Actual Result:

An error appears, and the attachment does not get downloaded to device.

Workaround:

Unknown

Platforms:

Which of our officially supported platforms is this issue occurring on?

• Android / native
• Android / Chrome
• iOS / native
• iOS / Safari
• MacOS / Chrome / Safari
• MacOS / Desktop

Version Number: v1.3.55-4

Reproducible in staging?: Y

Reproducible in production?: Y

If this was caught during regression testing, add the test name, ID and link from TestRail:

Email or phone of affected tester (no customers):

Logs: https://stackoverflow.com/c/expensify/questions/4856

Notes/Photos/Videos: Any additional supporting documentation

recording_20230808_175024.mp4

Screen.Recording.20230818.173007.New.Expensify.mp4

Expensify/Expensify Issue URL:

Issue reported by: @NajiNazzal

Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1691506774751679

View all open jobs on GitHub

Upwork Automation - Do Not Edit

• Upwork Job URL: https://www.upwork.com/jobs/~01ea72aefb3af1826d
• Upwork Job ID: 1692654340381872128
• Last Price Increase: 2023-08-18
• Automatic offers:
• dukenv0307 | Contributor | 26261011

[melvin-bot]

Triggered auto assignment to @garrettmknight (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details.

[melvin-bot]

Bug0 Triage Checklist (Main S/O)

• This "bug" occurs on a supported platform (ensure Platforms in OP are ✅)
• This bug is not a duplicate report (check E/App issues and #expensify-bugs)
  • If it is, comment with a link to the original report, close the issue and add any novel details to the original issue instead
• This bug is reproducible using the reproduction steps in the OP. S/O
  • If the reproduction steps are clear and you're unable to reproduce the bug, check with the reporter and QA first, then close the issue.
  • If the reproduction steps aren't clear and you determine the correct steps, please update the OP.
• This issue is filled out as thoroughly and clearly as possible
  • Pay special attention to the title, results, platforms where the bug occurs, and if the bug happens on staging/production.
• I have reviewed and subscribed to the linked Slack conversation to ensure Slack/Github stay in sync

[garrettmknight]

Confirmed in browserstack

[melvin-bot]

Job added to Upwork: https://www.upwork.com/jobs/~01ea72aefb3af1826d

[melvin-bot]

Current assignee @garrettmknight is eligible for the External assigner, not assigning anyone new.

[melvin-bot]

Triggered auto assignment to Contributor-plus team member for initial proposal review - @burczu (External)

[rayane-djouah]

Proposal

Please re-state the problem that we are trying to solve in this issue.

Our application, designed to save files to the "Downloads/Expensify" directory, is experiencing challenges on Android 11+ devices. Specifically, the application struggles to fetch and store the intended files in the Downloads/Expensify directory.

What is the root cause of that problem?

The introduction of Scoped Storage in Android 11 is the primary factor behind this challenge. This feature:

• Enforces stricter access controls to shared storage areas.
• Prioritizes user privacy by restricting applications from having broad access to the file system.
• Mandates applications to use certain APIs, such as MediaStore, for specific storage operations, particularly when accessing shared storage areas like the Downloads directory.

In our existing implementation, we use the RNFetchBlob library to download the file from the attachement URL and save it to the specified path, here:

App/src/libs/fileDownload/index.android.js

Lines 52 to 61 in 03cf0b1

	// Fetching the attachment
	fetchedAttachment = RNFetchBlob.config({
	fileCache: true,
	path: `${path}/${attachmentName}`,
	addAndroidDownloads: {
	useDownloadManager: true,
	notification: false,
	path: `${path}/Expensify/${attachmentName}`,
	},
	}).fetch('GET', url);

when we set a path and enable addAndroidDownloads with useDownloadManager: true, the RNFetchBlob library attempts to employ the Android Download Manager to fetch and directly store the file in the Downloads directory. However, from Android 11 onwards, designating a custom path for the Download Manager especially paths outside of the app's specific directory will result in failures, all thanks to Scoped Storage restrictions.

What changes do you think we should make in order to solve the problem?

Adapt to the Scoped Storage Model by Modifying RNFetchBlob Configuration:
Exclude the custom path instructions and the addAndroidDownloads configurations from our RNFetchBlob configuration by removing this lines:

App/src/libs/fileDownload/index.android.js

Lines 55 to 60 in 03cf0b1

	path: `${path}/${attachmentName}`,
	addAndroidDownloads: {
	useDownloadManager: true,
	notification: false,
	path: `${path}/Expensify/${attachmentName}`,
	},

As a result, the library will inherently fetch and store the file within the application's specific directory, ensuring alignment with Scoped Storage guidelines.

We have already an Integration with the MediaStore API in the rest of the code:
After the file is fetched within the application's directory, we use RNFetchBlob.MediaCollection.copyToMediaStore to save the file in the Downloads directory.

App/src/libs/fileDownload/index.android.js

Lines 73 to 81 in 03cf0b1

	return RNFetchBlob.MediaCollection.copyToMediaStore(
	{
	name: attachmentName,
	parentFolder: 'Expensify',
	mimeType: null,
	},
	'Download',
	attachmentPath,
	);

This technique collaborates effectively with the Android MediaStore API, granting the application the capability to transfer the file to the Downloads shared directory, without contravening Scoped Storage policies.

Result:

Download.File.mp4

What alternative solutions did you explore? (Optional)

N/A

[rayane-djouah]

@burczu please review my proposal

[dukenv0307]

Proposal

Please re-state the problem that we are trying to solve in this issue.

An error appears, and the attachment does not get downloaded to device.

What is the root cause of that problem?

In here, we're appending DateUtils.getDBTime() to the attachment name to make it unique. So the attachment name will look like
rn_image_picker_lib_temp_aa2c7e8d-e497-43e6-8d3b-1b3d047457b7-2023-08-21 06:28:08.305.jpg

It contains colon :, which is an illegal character in file name, so Android wasn't able to save it.

What changes do you think we should make in order to solve the problem?

We need to remove the illegal characters like : before trying to download the attachment.

Replacing : by empty string will work for this issue, but there're more illegal characters, we should use a regex to remove them as well.

What alternative solutions did you explore? (Optional)

We can escape the illegal characters, but it will lead to not-good-looking file name.

Another way is, instead of using DateUtils.getDBTime() to get the unique time to append to the attachment name, we can use other safer time formats like timestamp or YYYY-MM-DD HHmmss instead. Or use other ways of enforcing uniqueness (uuid, ...) that doesn't have special characters.

[burczu]

not overdue - I'm reviewing proposals right now

[burczu]

I wasn't able to reproduce this issue - I've tried Pixel 5 with Android 11 and Pixel 6 Pro with Android 12 (both emulators). On what devices did you test it? @izarutskaya maybe you can help me with it?

[rayane-djouah]

@burczu

For devices with Android 11 and above, I've reproduced this on Pixel 3a with Android API 34, Android 14.

For devices with Android below 11, where the bug is not occurring, I've tested on Pixel 3a with Android API 34, Android 8.

[izarutskaya]

@burczu I used Samsung M 52/ Android 13 and it's still reproducible for me

Screen.Recording.20230821.133615.New.Expensify.mp4

[AmjedNazzal]

@burczu I think it happens on samsung phones which you can't emulate on andriod studio

[burczu]

Thanks @izarutskaya and @rayane-djouah - I'll give your configs a try.

[rayane-djouah]

I think it happens on samsung phones which you can't emulate on andriod studio

I don't think so, I'm able to reproduce on Pixel 3a emulator on android studio

[AmjedNazzal]

I think it happens on samsung phones which you can't emulate on andriod studio

I don't think so, I'm able to reproduce on Pixel 3a emulator on android studio

You're using android 14, is that stable? because as far as I know that hasn't been officially released yet and it's a beta release at the moment. I've tested on pixel 6 android 13 and wasn't able to reproduce.

[burczu]

Ok, I was able to reproduce it with Pixel 3a with Android API 34, Android 14 as @rayane-djouah suggested. I'll try with Android 13 too, to dispel @AmjedNazzal doubts. Thanks

[burczu]

Hmm... @AmjedNazzal was right here - it is not reproducible on Pixel 3a with Android 13 🤔 I've also tried on Galaxy Nexus with Android 13 which is available in Android Studio and it is also not reproducible...

@garrettmknight I've wrote that you've confirmed it on BrowserStack - on what device and with which Android?

[AmjedNazzal]

@burczu I don't think the galaxy nexus would be helpful here. If I'm to guess, the issue is probably occurring because of samsung one ui since it's involved in apps interactions with the native system. We should probably run and test dev on physical samsung phones.

[garrettmknight]

Tested on a Galaxy S22 v12.0 in browserstack.

[melvin-bot]

The solution for this issue has been 🚀 deployed to production 🚀 in version 1.3.58-5 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

• Fix: Android - Error attempting to download an attachment #25742

If no regressions arise, payment will be issued on 2023-09-06. 🎊

After the hold period is over and BZ checklist items are completed, please complete any of the applicable payments for this issue, and check them off once done.

• External issue reporter
• Contributor that fixed the issue
• Contributor+ that helped on the issue and/or PR

For reference, here are some details about the assignees on this issue:

• @burczu does not require payment (Contractor)
• @dukenv0307 requires payment offer (Contributor)
• @NajiNazzal requires payment

As a reminder, here are the bonuses/penalties that should be applied for any External issue:

• Merged PR within 3 business days of assignment - 50% bonus
• Merged PR more than 9 business days after assignment - 50% penalty

[melvin-bot]

BugZero Checklist: The PR fixing this issue has been merged! The following checklist (instructions) will need to be completed before the issue can be closed:

• [@burczu] The PR that introduced the bug has been identified. Link to the PR:
• [@burczu] The offending PR has been commented on, pointing out the bug it caused and why, so the author and reviewers can learn from the mistake. Link to comment:
• [@burczu] A discussion in #expensify-bugs has been started about whether any other steps should be taken (e.g. updating the PR review checklist) in order to catch this type of bug sooner. Link to discussion:
• [@burczu] Determine if we should create a regression test for this bug.
• [@burczu] If we decide to create a regression test for the bug, please propose the regression test steps to ensure the same bug will not reach production again.
• [@garrettmknight] Link the GH issue for creating/updating the regression test once above steps have been agreed upon:

[burczu]

BugZero Checklist: The PR fixing this issue has been merged! The following checklist (instructions) will need to be completed before the issue can be closed:

• [@burczu] The PR that introduced the bug has been identified. Link to the PR: n/a
• [@burczu] The offending PR has been commented on, pointing out the bug it caused and why, so the author and reviewers can learn from the mistake. Link to comment: n/a
• [@burczu] A discussion in #expensify-bugs has been started about whether any other steps should be taken (e.g. updating the PR review checklist) in order to catch this type of bug sooner. Link to discussion: n/a
• [@burczu] Determine if we should create a regression test for this bug.
• [@burczu] If we decide to create a regression test for the bug, please propose the regression test steps to ensure the same bug will not reach production again.
• [@garrettmknight] Link the GH issue for creating/updating the regression test once above steps have been agreed upon:

[aimane-chnaif]

Hmm, I'd say this regression came from #23531 but they couldn't have caught this earlier in PR review since this happens only on android specific devices.

[garrettmknight]

Payment tomorrow!

[garrettmknight]

Summary of payments:

• urgency bonus? Yes
• Reporter: @NajiNazzal - $250 paid via Upwork
• Contributor: @dukenv0307 - $1500 paid via Upwork
• Contributor+: N/A

Upwork Job: https://www.upwork.com/jobs/~01ea72aefb3af1826d

[garrettmknight]

@NajiNazzal I just sent you an offer in Upwork. Accept and I'll pay this out.

[melvin-bot]

@garrettmknight, @robertjchen, @burczu, @dukenv0307 Uh oh! This issue is overdue by 2 days. Don't forget to update your issues!

[melvin-bot]

@garrettmknight, @robertjchen, @burczu, @dukenv0307 Whoops! This issue is 2 days overdue. Let's get this updated quick!

[garrettmknight]

Paid and done!

[aimane-chnaif]

@garrettmknight I was C+ on this issue

[garrettmknight]

Ah, missed that here. Let me spin up a contract, but in the future, make sure you're assigned to the issue!

[garrettmknight]

Offer sent: https://www.upwork.com/nx/wm/pre-hire/c/8577561/offer/26607572

[robertjchen]

Looks like it was accepted, closing this out!

[aimane-chnaif]

@garrettmknight I haven't received payment yet. Can you please check?

[aimane-chnaif]

@garrettmknight I haven't received payment yet. Can you please check?

bump ^

[aimane-chnaif]

@robertjchen can you please reopen the issue? I haven't received payment yet

Edit: Thanks @garrettmknight. Just received payment. No need to re-open.