Move ledger data:* favicons to external files #11754

diracdeltas · 2017-11-01T22:11:22Z

This moves data:* favicon URLs in the ledger appState to external image files during session store cleanup. Reduces the size of my session-store-1 from ~23 MB to 5 MB.

Fix #11582

Test plan:

Copy session-store-1 from a Brave profile that has had payments enabled for a while to your brave-development appData directory.
Start the browser and go to about:preferences#payments. The synopsis table should appear fine.
Close the browser.
In your brave-development appData directory, you should now see a ledger-favicons subdirectory which contains a lot of image files. You should also notice that session-store-1 is now much smaller.
Repeat step 2.

Submitter Checklist:

Submitted a ticket for my issue if one did not already exist.
Used Github auto-closing keywords in the commit message.
Added/updated tests for this change (for new code or code which already has tests).
Ran git rebase -i to squash commits (if needed).
Tagged reviewers and labelled the pull request as needed.

Test Plan:

Reviewer Checklist:

Tests

Adequate test coverage exists to prevent regressions
Tests should be independent and work correctly when run individually or as a suite ref
New files have MPL2 license header

codecov-io · 2017-11-01T22:42:25Z

Codecov Report

Merging #11754 into master will decrease coverage by 8.74%.
The diff coverage is 21.91%.

@@            Coverage Diff             @@
##           master   #11754      +/-   ##
==========================================
- Coverage   55.09%   46.34%   -8.75%     
==========================================
  Files         275      270       -5     
  Lines       26619    25849     -770     
  Branches     4288     4116     -172     
==========================================
- Hits        14667    11981    -2686     
- Misses      11952    13868    +1916

Flag	Coverage Δ
#unittest	`46.34% <21.91%> (-8.75%)`	⬇️

Impacted Files	Coverage Δ
...erer/components/preferences/payment/ledgerTable.js	`87.75% <0%> (-1.78%)`	⬇️
js/about/aboutActions.js	`9.09% <0%> (-0.24%)`	⬇️
app/sessionStore.js	`2.87% <2.04%> (-73.72%)`	⬇️
js/lib/urlutil.js	`86.05% <93.75%> (+0.52%)`	⬆️
app/sessionStoreShutdown.js	`1.26% <0%> (-85.45%)`	⬇️
app/browser/reducers/windowsReducer.js	`3.05% <0%> (-77.49%)`	⬇️
app/browser/api/ledger.js	`1.15% <0%> (-48.11%)`	⬇️
app/browser/api/ledgerNotifications.js	`30.26% <0%> (-44.3%)`	⬇️
app/browser/windows.js	`3.14% <0%> (-34.39%)`	⬇️
... and 20 more

NejcZdovc · 2017-11-02T06:20:20Z

@diracdeltas this is really cool. Maybe one suggestion. Now we have synopsis and about synopsis, but synopsis is parent of about synopsis. So what is in about synopsis is for sure in synopsis. If I understand this code correctly we are create all favicons twice. My suggestion would be to generate favicons for synopsis and then just assign this values to about synopsis as well. wdyt?

NejcZdovc · 2017-11-02T06:28:31Z

In near future we would like to remove this duplicate entries, so that you would only have one synopsis.

bridiver · 2017-11-02T07:49:01Z

app/sessionStore.js

+          )
+          fs.writeFile(faviconPath, parsed.data, 'base64', (err) => {
+            if (err) {
+              console.error(`Error writing file: ${faviconPath} ${err}`)


are we ok with potentially losing these?

iirc the ledger tries to get the favicon whenever the user visits the page, so the icon would be restored on the next visit

@diracdeltas that's not the case. We only get favicon if there is not one

i could add an img onerror in the synopsis table that sets the faviconURL to '' / null if the file does not exist on disk - would that cause the favicon to be re-fetched?

yes if you set it to null it will work. More info here: https://github.com/brave/browser-laptop/blob/master/app/browser/api/ledger.js#L1146

bridiver · 2017-11-02T07:49:17Z

app/sessionStore.js

+              ? `favicon-${index}-${Date.now()}.${parsed.ext}`
+              : `favicon-${index}.${parsed.ext}`
+          )
+          fs.writeFile(faviconPath, parsed.data, 'base64', (err) => {


might be better to use muon.file.writeImportant

fs.writeFile was easier because it supported the base64 encoding parameter

bridiver · 2017-11-02T07:50:04Z

app/sessionStore.js

+  try {
+    const basePath = getStoragePath('ledger-favicons')
+    const fs = require('fs')
+    if (!fs.existsSync(basePath)) {


should use the async versions of these to avoid blocking the UI thread

it needs to be synchnrous because this directory must exist before cleanFavicons is called, and cleanFavicons returns immutableData synchronously. I could add a migration flag that indicates whether this directory was created, which would avoid having to do the synchronous mkdir in future runs.

bsclifton · 2017-11-20T16:53:00Z

For review status, @bridiver @NejcZdovc do you approve / not approve? @diracdeltas was any action needed on your part after the above comments?

diracdeltas · 2017-11-20T22:49:49Z

@bsclifton there's some follow-up enhancements which i'll try to do early this week

bsclifton · 2017-11-28T18:02:19Z

Moving to 0.20.x (as the next 0.19.x release is reserved for Chromium 63)

diracdeltas · 2017-12-02T02:00:53Z

@NejcZdovc i addressed all review comments. however, i thought of something which is that the ledger-favicon/ directory should be deleted whenever the user chooses to delete payments data. do you remember if that's doable via the 'Clear browsing data...' menu item or something else in the UI?

NejcZdovc · 2017-12-02T17:32:40Z

we have issue up for this option #8537, so I think that we should finally implement it 😃 will take a look and probably add it

diracdeltas · 2017-12-02T18:57:32Z

ok, if there's currently no way to clear ledger data from the session store in the UI, i don't think this PR needs to address that. ready for re-review.

This moves data:* URLs in the ledger appState to external image files, which reduces the size of my session-store-1 from ~25 MB to 5 MB. Fix #11582 Test plan: 1. Copy session-store-1 from a Brave profile that has had payments enabled for a while to your brave-development appData directory. 2. Start the browser and go to about:preferences#payments. The synopsis table should appear fine. 3. Close the browser. 4. In your brave-development appData directory, you should now see a ledger-favicons subdirectory which contains a lot of image files. You should also notice that session-store-1 is now much smaller. 5. Repeat step 2.

NejcZdovc

++ this will save a lot of space

Move ledger data:* favicons to external files

diracdeltas · 2017-12-13T21:44:06Z

master: 4e7995d
0.21.x: ca4bc9a
0.20.x: 0b45c87

petemill · 2018-02-01T22:36:43Z

Do we have any sample large profiles with this data in, so we can assess impact and clear this change from any perf regressions we're seeing? @diracdeltas @NejcZdovc

diracdeltas self-assigned this Nov 1, 2017

diracdeltas requested review from bbondy and NejcZdovc November 1, 2017 22:11

diracdeltas added feature/rewards perf needs-triage labels Nov 1, 2017

bbondy added this to the 0.19.x Hotfix 4 (Release channel) milestone Nov 2, 2017

bridiver reviewed Nov 2, 2017

View reviewed changes

diracdeltas mentioned this pull request Nov 2, 2017

reduce size of session-store-1 by removing 'data:' URLs #11582

Closed

bbondy modified the milestones: 0.19.x Hotfix 5 (Release channel), 0.19.x Hotfix 6 (Release channel) Nov 8, 2017

cezaraugusto removed the needs-triage label Nov 8, 2017

luixxiul added the QA/test-plan-specified label Nov 13, 2017

luixxiul added the PR/work-in-progress ⚒ label Nov 21, 2017

bsclifton modified the milestones: 0.19.x Hotfix 6 (Release channel), 0.20.x (Beta Channel) Nov 28, 2017

diracdeltas force-pushed the fix/remove-data-urls branch from f2e02fd to fb28e2e Compare December 2, 2017 01:58

diracdeltas removed the PR/work-in-progress ⚒ label Dec 2, 2017

diracdeltas removed request for bbondy and NejcZdovc December 2, 2017 18:55

diracdeltas requested a review from NejcZdovc December 2, 2017 18:55

diracdeltas force-pushed the fix/remove-data-urls branch from fb28e2e to e17af72 Compare December 6, 2017 00:01

NejcZdovc approved these changes Dec 12, 2017

View reviewed changes

diracdeltas merged commit 4e7995d into master Dec 13, 2017

diracdeltas deleted the fix/remove-data-urls branch December 13, 2017 21:28

diracdeltas added a commit that referenced this pull request Dec 13, 2017

Merge pull request #11754 from brave/fix/remove-data-urls

ca4bc9a

Move ledger data:* favicons to external files

diracdeltas added a commit that referenced this pull request Dec 13, 2017

Merge pull request #11754 from brave/fix/remove-data-urls

0b45c87

Move ledger data:* favicons to external files

petemill mentioned this pull request Feb 1, 2018

If window / tab restore fails, persisted state will be deleted #12984

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move ledger data:* favicons to external files #11754

Move ledger data:* favicons to external files #11754

diracdeltas commented Nov 1, 2017 •

edited

Loading

codecov-io commented Nov 1, 2017 •

edited

Loading

NejcZdovc commented Nov 2, 2017

NejcZdovc commented Nov 2, 2017

bridiver Nov 2, 2017

diracdeltas Nov 2, 2017 •

edited

Loading

NejcZdovc Nov 2, 2017

diracdeltas Nov 2, 2017

NejcZdovc Nov 2, 2017

bridiver Nov 2, 2017

diracdeltas Dec 2, 2017

bridiver Nov 2, 2017

diracdeltas Nov 2, 2017

bsclifton commented Nov 20, 2017 •

edited

Loading

diracdeltas commented Nov 20, 2017

bsclifton commented Nov 28, 2017

diracdeltas commented Dec 2, 2017

NejcZdovc commented Dec 2, 2017

diracdeltas commented Dec 2, 2017

NejcZdovc left a comment

diracdeltas commented Dec 13, 2017

petemill commented Feb 1, 2018

Move ledger data:* favicons to external files #11754

Move ledger data:* favicons to external files #11754

Conversation

diracdeltas commented Nov 1, 2017 • edited Loading

codecov-io commented Nov 1, 2017 • edited Loading

Codecov Report

NejcZdovc commented Nov 2, 2017

NejcZdovc commented Nov 2, 2017

Choose a reason for hiding this comment

diracdeltas Nov 2, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bsclifton commented Nov 20, 2017 • edited Loading

diracdeltas commented Nov 20, 2017

bsclifton commented Nov 28, 2017

diracdeltas commented Dec 2, 2017

NejcZdovc commented Dec 2, 2017

diracdeltas commented Dec 2, 2017

NejcZdovc left a comment

Choose a reason for hiding this comment

diracdeltas commented Dec 13, 2017

petemill commented Feb 1, 2018

diracdeltas commented Nov 1, 2017 •

edited

Loading

codecov-io commented Nov 1, 2017 •

edited

Loading

diracdeltas Nov 2, 2017 •

edited

Loading

bsclifton commented Nov 20, 2017 •

edited

Loading