Ignore images from embedded html when pasting, paste plaintext instead. #21476

qispark · 2023-06-23T23:44:27Z

Details

When pasting some HTML containing images into Composer, it will cause an error modal. This is because of the server's Content Security Policy, which prevents those images from being fetched separately. This PR prevents the images from being fetched, instead it will paste the plaintext that is copied.

Fixed Issues

$ #18583
$ #18583 (comment)

Tests

Copy a web document which contains both text and images. (e.g. https://www.w3schools.com/tags/tryit.asp?filename=tryhtml_image_test)
Log in with any account.
Go to any chat
Paste the contents into the composer
Verify that no image attachments are being created.
Verify that the correct contents are within the composer box.
Verify that no error modals are appearing upon paste.

Verify that no errors appear in the JS console

Offline tests

N/A

QA Steps

Copy a web document which contains both text and image. (e.g. https://www.w3schools.com/tags/tryit.asp?filename=tryhtml_image_test)
Log in
Go to any chat
Paste the contents into the composer
Verify that the correct contents are within the composer box.
Verify that no error modals are appearing upon paste.

Verify that no errors appear in the JS console

PR Author Checklist

Screenshots/Videos

Web

Screen.Recording.2023-06-23.at.9.08.11.pm.mov

Mobile Web - Chrome

Screen.Recording.2023-06-23.at.10.48.26.pm.mov

Mobile Web - Safari

Screen.Recording.2023-06-23.at.9.10.32.pm.mov

Desktop

Screen.Recording.2023-06-23.at.11.17.31.pm.mov

iOS

Screen.Recording.2023-06-23.at.10.21.31.pm.mov

Android

Screen.Recording.2023-06-23.at.10.58.14.pm.mov

melvin-bot · 2023-06-23T23:45:11Z

@parasharrajat Please copy/paste the Reviewer Checklist from here into a new comment on this PR and complete it. If you have the K2 extension, you can simply click: [this button]

parasharrajat · 2023-06-26T15:53:37Z

src/components/Composer/index.js

            if (embeddedImages.length > 0 && embeddedImages[0].src) {
-                // If HTML has emoji, then treat this as plain text.
-                if (embeddedImages[0].dataset && embeddedImages[0].dataset.stringifyType === 'emoji') {
-                    const plainText = event.clipboardData.getData('text/plain');
-                    this.paste(plainText);
-                    return;
-                }
-                fetch(embeddedImages[0].src)
-                    .then((response) => {
-                        if (!response.ok) {
-                            throw Error(response.statusText);
-                        }
-                        return response.blob();
-                    })
-                    .then((x) => {
-                        const extension = IMAGE_EXTENSIONS[x.type];
-                        if (!extension) {
-                            throw new Error(this.props.translate('composer.noExtensionFoundForMimeType'));
-                        }
-
-                        return new File([x], `pasted_image.${extension}`, {});
-                    })
-                    .then(this.props.onPasteFile)
-                    .catch(() => {
-                        const errorDesc = this.props.translate('composer.problemGettingImageYouPasted');
-                        Growl.error(errorDesc);
-
-                        /*
-                         * Since we intercepted the user-triggered paste event to check for attachments,
-                         * we need to manually set the value and call the `onChangeText` handler.
-                         * Synthetically-triggered paste events do not affect the document's contents.
-                         * See https://developer.mozilla.org/en-US/docs/Web/API/Element/paste_event for more details.
-                         */
-                        this.handlePastedHTML(pastedHTML);
-                    });
+                this.handlePastePlainText(event);


Why are we not pasting the HTML? embeddedImages[0].dataset && embeddedImages[0].dataset.stringifyType === 'emoji' require pasting text but other than this we can use html pasting...

@parasharrajat If you read my proposal carefully, I explain the caveat of why we should paste plaintext instead of pasting the HTML (#18583 (comment))

If any images have links attached, and we try to paste them, it leads to formatting issues in the markdown that is generated.

Please let me know what is the desired behaviour in this instance as I thought you already approved my proposal.

I did approve the removal of fetching of images but the whole formatting. Can you share a screenshot/video of formatting issues?

This is what happens when I set it to paste HTML:

Screen.Recording.2023-06-27.at.8.40.55.am.mov

Apologies for the delay. But I think we need to keep formatting. It is strange that image tags are converted to links. I am not sure where that happens in out parser.

@qispark If images are wrapped in a link then it is fine if they are parsed as links in the pasted code.

But if the image itself is being converted to links then that is a problem. If we want to solve this problem, then we should be modifying the link rules in ExpensiMark to filter out images.

@parasharrajat Let me know if I should action on anything based on your feedback.

@techievivek cc

@qispark Based on Rajat's comment, If images are wrapped in a link then it is fine if they are parsed as links in the pasted code. I think the image was wrapped in a link in the recording above, right? (Can you confirm this by inspecting that element?) and when we are pasting it is being rendered as links.

@parasharrajat

But if the image itself is being converted to links then that is a problem

I don't think that is happening in this case. @qispark can you please confirm this?

And now for the last part

Why are we not pasting the HTML? embeddedImages[0].dataset && embeddedImages[0].dataset.stringifyType === 'emoji' require pasting text but other than this we can use html pasting.

@qispark What do you think can be done to fix this? Do we need to update the ExpensiMark logic? I think that is very much doable, so if that's the case please push a PR in the expensify-common repo and let's keep this moving.

@techievivek Your understanding is correct.
When the image is wrapped in a link, it gets rendered as an empty link.
When the image is not wrapped in a link, it gets rendered as its alt text.

I think we can update the ExpensiMark logic based on what I mentioned in my last comment. Happy to make a PR there to reflect this functionality. I will share an update when this is done.

parasharrajat · 2023-07-04T10:41:14Z

I think we are fixing the empty link parsing in Expensify/expensify-common#555. That might solve our challenge where we see an empty title link being parsed as MD style link for links wrapping images in HTML.

qispark · 2023-07-04T11:11:48Z

OK, @parasharrajat I'm happy to wait until that PR is merged before we test this again - we might not need further changes in that case. Is that ok?

parasharrajat · 2023-07-04T14:16:22Z

src/components/Composer/index.js

-                if (embeddedImages[0].dataset && embeddedImages[0].dataset.stringifyType === 'emoji') {
-                    const plainText = event.clipboardData.getData('text/plain');
-                    this.paste(plainText);
-                    return;
-                }
-                fetch(embeddedImages[0].src)


We still need to revert the logic for .stringifyType === 'emoji' and for the rest pasteHTML. let's do that.

qispark · 2023-07-05T00:29:53Z

@parasharrajat I have just updated the PR to reinstate the logic for emojis. Please let me know if any other changes are required.

techievivek · 2023-07-07T05:19:28Z

Expensify/expensify-common#555 PR is merged, so I think we can move forward with thoroughly testing this PR now, right?

qispark · 2023-07-07T05:32:59Z

@techievivek I've just tested it. Unfortunately, it appears to have the same behaviour, still creating an empty link. I'm using a test branch based on main with this PR's updates.

Do we need to revise ExpensiMark again? @parasharrajat

Just examining the PR to expensify-common, it looks like changes were only made to email links, so I wonder if we need to expand the changes to all links.

parasharrajat · 2023-07-07T05:42:35Z

Yes. We need to find a solution. Please propose the changes here first which you are planning to do to ExpensiMark

qispark · 2023-07-07T05:46:34Z

In lib/ExpensiMark.js:

{
                name: 'link',
                process: (textToProcess, replacement) => this.modifyTextForUrlLinks(MARKDOWN_LINK_REGEX, textToProcess, replacement),
                replacement: (match, g1, g2) => {
                    if (g1.match(CONST.REG_EXP.EMOJIS) || !g1.trim()) {
                        return match;
                    }
                    return `<a href="${Str.sanitizeURL(g2)}" target="_blank" rel="noreferrer noopener">${g1.trim()}</a>`;
                },
            },

If this is the relevant rule, I think we need to update the part !g1.trim() to strip new lines as well in addition to just whitespaces. It's the new line causing the empty link to appear, I believe.

parasharrajat · 2023-07-07T07:46:36Z

Can you share the sample HTML that you tried pasting? It will help me review.

qispark · 2023-07-07T07:52:24Z

@parasharrajat I'm OOO until Monday - but essentially you can recreate it on any website with an image wrapped in a link.

qispark · 2023-07-10T05:35:40Z

@parasharrajat
Here's an example I found, which creates an empty link, even without a new line:

<meta charset='utf-8'><p style="color: rgb(0, 0, 0); font-family: Times; font-size: medium; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">The image below is a link. Try to click on it.</p><a href="https://www.w3schools.com/html/default.asp" style="font-family: Times; font-size: medium; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal;"><img src="https://www.w3schools.com/html/smiley.gif" alt="HTML tutorial" style="width: 42px; height: 42px;"></a>

You can copy the contents from here: https://www.w3schools.com/html/tryit.asp?filename=tryhtml_links_image

This is the result:

parasharrajat · 2023-07-10T16:42:05Z

Thanks

qispark · 2023-07-12T09:01:40Z

Hi @parasharrajat, it wasn't clear to me, but did you want me to make a PR in expensify-common? Or shall I wait until you have tested it?

parasharrajat · 2023-07-18T15:27:15Z

Sorry for the delay here. Once I lost momentum on a PR, it is harder to get back on it. I will try to close this PR as soon as possible.

techievivek · 2023-07-24T03:52:40Z

@parasharrajat, any updates here?

parasharrajat · 2023-07-25T18:44:55Z

My bad, I wasn't able to focus on this. Continuing review so that I can get this merged asap.

techievivek · 2023-07-27T08:28:43Z

@parasharrajat Gentle bump to keep this moving, also QA reported that this was not reproducible #18583 (comment) 🤯, haven't verified it myself tho.

techievivek · 2023-07-31T05:15:46Z

@parasharrajat, Can you please prioritize this PR, as it has been open for some time now? Thanks

src/components/Composer/index.js

Co-authored-by: Rajat Parashar <parasharrajat@users.noreply.github.com>

parasharrajat · 2023-08-04T00:01:58Z

So, image parsing is not related to this issue. I noticed that HTML does have anchor tags wrapping image tags. Thus they are being parsed as [](link) which is correct because images are not parsed in our parser.

parasharrajat · 2023-08-04T00:05:43Z

Screenshots

🔲 iOS / native

Screen.Recording.2023-08-04.at.5.42.09.AM.mov

🔲 iOS / Safari

Screen.Recording.2023-08-04.at.5.29.24.AM.mov

🔲 MacOS / Desktop

Screen.Recording.2023-08-04.at.5.23.45.AM.mov

🔲 MacOS / Chrome

Screen.Recording.2023-08-04.at.5.22.20.AM.mov

🔲 Android / Chrome

Screen.Recording.2023-08-04.at.5.32.58.AM.mov

🔲 Android / native

parasharrajat

Reviewer Checklist

I have verified the author checklist is complete (all boxes are checked off).
I verified the correct issue is linked in the ### Fixed Issues section above
I verified testing steps are clear and they cover the changes made in this PR
- I verified the steps for local testing are in the Tests section
- I verified the steps for Staging and/or Production testing are in the QA steps section
- I verified the steps cover any possible failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
- I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
I checked that screenshots or videos are included for tests on all platforms
I included screenshots or videos for tests on all platforms
I verified tests pass on all platforms & I tested again on:
- Android / native
- Android / Chrome
- iOS / native
- iOS / Safari
- MacOS / Chrome / Safari
- MacOS / Desktop
If there are any errors in the console that are unrelated to this PR, I either fixed them (preferred) or linked to where I reported them in Slack
I verified proper code patterns were followed (see Reviewing the code)
- I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick).
- I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
- I verified that comments were added to code that is not self explanatory
- I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
- I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method
- I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
- I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is approved by marketing by adding the Waiting for Copy label for a copy review on the original GH to get the correct copy.
- I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
- I verified the JSDocs style guidelines (in STYLE.md) were followed
If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
I verified that this PR follows the guidelines as stated in the Review Guidelines
I verified other components that can be impacted by these changes have been tested, and I retested again (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar have been tested & I retested again)
I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
If a new component is created I verified that:
- A similar component doesn't exist in the codebase
- All props are defined accurately and each prop has a /** comment above it */
- The file is named correctly
- The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
- The only data being stored in the state is data necessary for rendering and nothing else
- For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
- Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
- All JSX used for rendering exists in the render method
- The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions
If any new file was added I verified that:
- The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory
If a new CSS style is added I verified that:
- A similar style doesn't already exist
- The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG)
If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.
I have checked off every checkbox in the PR reviewer checklist, including those that don't apply to this PR.

🎀 👀 🎀 C+ reviewed

parasharrajat · 2023-08-04T00:24:25Z

Not sure what changed. My android build is not starting, stuck at info Starting JS server... step.

But I believe it works as changes are only applied to web-specific files.

parasharrajat · 2023-08-04T09:00:18Z

Bump @techievivek

techievivek · 2023-08-07T04:40:22Z

Sorry, I was ooo on Friday, taking a look now. Thanks.

src/components/Composer/index.js

techievivek · 2023-08-07T05:05:09Z

@qispark Linting seems to be failing, can you please fix that, thanks.

…ges-2' into qispark-composer-paste-ignore-images-2

qispark · 2023-08-07T05:13:44Z

@techievivek Just updated lint.

src/components/Composer/index.js

Co-authored-by: Vivek Kumar <35863227+techievivek@users.noreply.github.com>

techievivek

Great job, thanks for sticking by 🙇

qispark · 2023-08-07T05:48:31Z

@techievivek Just updated again due to prettier failing.

OSBotify · 2023-08-07T06:01:59Z

✋ This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.

OSBotify · 2023-08-07T18:54:52Z

🚀 Deployed to staging by https://github.com/techievivek in version: 1.3.51-0 🚀

platform	result
🤖 android 🤖	success ✅
🖥 desktop 🖥	success ✅
🍎 iOS 🍎	success ✅
🕸 web 🕸	success ✅

OSBotify · 2023-08-09T08:44:00Z

🚀 Deployed to production by https://github.com/Julesssss in version: 1.3.51-2 🚀

platform	result
🤖 android 🤖	success ✅
🖥 desktop 🖥	success ✅
🍎 iOS 🍎	success ✅
🕸 web 🕸	success ✅

OSBotify · 2023-08-09T09:12:59Z

🚀 Deployed to staging by https://github.com/techievivek in version: 1.3.52-0 🚀

platform	result
🤖 android 🤖	success ✅
🖥 desktop 🖥	failure ❌
🍎 iOS 🍎	success ✅
🕸 web 🕸	success ✅

OSBotify · 2023-08-09T09:51:42Z

🚀 Deployed to staging by https://github.com/techievivek in version: 1.3.52-0 🚀

platform	result
🤖 android 🤖	success ✅
🖥 desktop 🖥	success ✅
🍎 iOS 🍎	success ✅
🕸 web 🕸	success ✅

OSBotify · 2023-08-10T20:10:08Z

🚀 Deployed to production by https://github.com/puneetlath in version: 1.3.52-5 🚀

platform	result
🤖 android 🤖	success ✅
🖥 desktop 🖥	success ✅
🍎 iOS 🍎	success ✅
🕸 web 🕸	success ✅

Ignore images from embedded html when pasting, paste plaintext instead.

782b48f

qispark marked this pull request as ready for review June 23, 2023 23:45

qispark requested a review from a team as a code owner June 23, 2023 23:45

melvin-bot bot requested review from parasharrajat and removed request for a team June 23, 2023 23:45

qispark mentioned this pull request Jun 23, 2023

Ignore embedded images when pasting in composer #21370

Closed

57 tasks

parasharrajat reviewed Jun 26, 2023

View reviewed changes

techievivek self-requested a review June 30, 2023 05:26

techievivek assigned qispark Jun 30, 2023

parasharrajat reviewed Jul 4, 2023

View reviewed changes

Reinstate paste plaintext for emoji, and otherwise paste HTML

fe399ad

techievivek mentioned this pull request Jul 25, 2023

[HOLD for payment 2023-08-16] [$1000] Selecting all and pasting into the composer prompts user to send an attachment. #18583

Closed

6 tasks

parasharrajat reviewed Aug 3, 2023

View reviewed changes

src/components/Composer/index.js Outdated Show resolved Hide resolved

parasharrajat reviewed Aug 3, 2023

View reviewed changes

src/components/Composer/index.js Outdated Show resolved Hide resolved

Apply suggestions from code review

4f99a5c

Co-authored-by: Rajat Parashar <parasharrajat@users.noreply.github.com>

parasharrajat approved these changes Aug 4, 2023

View reviewed changes

techievivek reviewed Aug 7, 2023

View reviewed changes

src/components/Composer/index.js Outdated Show resolved Hide resolved

qispark added 2 commits August 7, 2023 15:09

Remove unused variables to pass lint

a6cbd61

Merge remote-tracking branch 'robin/qispark-composer-paste-ignore-ima…

5b99ed4

…ges-2' into qispark-composer-paste-ignore-images-2

techievivek reviewed Aug 7, 2023

View reviewed changes

src/components/Composer/index.js Outdated Show resolved Hide resolved

Update src/components/Composer/index.js

cf43831

Co-authored-by: Vivek Kumar <35863227+techievivek@users.noreply.github.com>

techievivek previously approved these changes Aug 7, 2023

View reviewed changes

Update prettier

ebb3932

qispark dismissed techievivek’s stale review via ebb3932 August 7, 2023 05:48

techievivek approved these changes Aug 7, 2023

View reviewed changes

techievivek merged commit e555022 into Expensify:main Aug 7, 2023
13 checks passed

OSBotify mentioned this pull request Aug 7, 2023

Deploy Checklist: New Expensify 2023-08-07 #24230

Closed

98 tasks

OSBotify mentioned this pull request Aug 9, 2023

Deploy Checklist: New Expensify 2023-08-09 #24307

Closed

87 tasks

Ignore images from embedded html when pasting, paste plaintext instead. #21476

Ignore images from embedded html when pasting, paste plaintext instead. #21476

Conversation

qispark commented Jun 23, 2023

Details

Fixed Issues

Tests

Offline tests

QA Steps

PR Author Checklist

Screenshots/Videos

melvin-bot bot commented Jun 23, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qispark Jul 3, 2023 • edited Loading

Choose a reason for hiding this comment

parasharrajat commented Jul 4, 2023

qispark commented Jul 4, 2023

Choose a reason for hiding this comment

qispark commented Jul 5, 2023

techievivek commented Jul 7, 2023

qispark commented Jul 7, 2023 • edited Loading

parasharrajat commented Jul 7, 2023

qispark commented Jul 7, 2023

parasharrajat commented Jul 7, 2023

qispark commented Jul 7, 2023

qispark commented Jul 10, 2023

parasharrajat commented Jul 10, 2023

qispark commented Jul 12, 2023

parasharrajat commented Jul 18, 2023

techievivek commented Jul 24, 2023

parasharrajat commented Jul 25, 2023

techievivek commented Jul 27, 2023

techievivek commented Jul 31, 2023

parasharrajat commented Aug 4, 2023

parasharrajat commented Aug 4, 2023 • edited Loading

Screenshots

🔲 iOS / native

🔲 iOS / Safari

🔲 MacOS / Desktop

🔲 MacOS / Chrome

🔲 Android / Chrome

🔲 Android / native

parasharrajat left a comment

Choose a reason for hiding this comment

Reviewer Checklist

parasharrajat commented Aug 4, 2023 • edited Loading

parasharrajat commented Aug 4, 2023

techievivek commented Aug 7, 2023

techievivek commented Aug 7, 2023

qispark commented Aug 7, 2023

techievivek left a comment

Choose a reason for hiding this comment

qispark commented Aug 7, 2023

OSBotify commented Aug 7, 2023

OSBotify commented Aug 7, 2023

OSBotify commented Aug 9, 2023

OSBotify commented Aug 9, 2023

OSBotify commented Aug 9, 2023

OSBotify commented Aug 10, 2023

qispark Jul 3, 2023 •

edited

Loading

qispark commented Jul 7, 2023 •

edited

Loading

parasharrajat commented Aug 4, 2023 •

edited

Loading

parasharrajat commented Aug 4, 2023 •

edited

Loading