Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Import onyx state #49255

Merged
merged 60 commits into from
Oct 3, 2024
Merged
Show file tree
Hide file tree
Changes from 53 commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
f1d2200
fix: improve perf of isActionOfType by limiting calls to the includes
adhorodyski Jul 30, 2024
6381821
feat(wip): draft the import-onyx-state perf debug functionality
adhorodyski Aug 5, 2024
f8e758b
chore: add comments describing the whole workflow in steps
adhorodyski Aug 5, 2024
3b33ef2
fix: not iterable items from onyx in persisted requests, added web su…
kubabutkiewicz Aug 8, 2024
69b1767
Merge branch 'main' of github.com:callstack-internal/Expensify-App in…
kubabutkiewicz Aug 8, 2024
caa2573
fix: make the android component more performant to not crash the app …
kubabutkiewicz Aug 9, 2024
1fbe276
Merge branch 'main' of github.com:callstack-internal/Expensify-App in…
kubabutkiewicz Aug 12, 2024
b9e7373
Merge branch 'Expensify:main' into feat/import-onyx-state
adhorodyski Aug 12, 2024
7d403da
Merge branch 'Expensify:main' into feat/import-onyx-state
adhorodyski Aug 13, 2024
5c4d6ab
feat: add returning to original state
kubabutkiewicz Aug 14, 2024
28937eb
Merge branch 'feat/import-onyx-state' of github.com:callstack-interna…
kubabutkiewicz Aug 14, 2024
0fe5895
fix: clear onyx state before applying imported one
kubabutkiewicz Aug 16, 2024
47926c0
fix: exporting onyx state on native
kubabutkiewicz Aug 19, 2024
024b4a2
fix: refreshing page
kubabutkiewicz Aug 21, 2024
02059bf
Merge branch 'main' into feat/import-onyx-state
TMisiukiewicz Sep 2, 2024
df69d08
align import button on web to be a menu item
TMisiukiewicz Sep 2, 2024
cc26c13
align import onyx state button on mobile as menu item
TMisiukiewicz Sep 2, 2024
f2c80c8
adjust transformNumericKeysToArray function usage
TMisiukiewicz Sep 9, 2024
1588340
adjust web implementation
TMisiukiewicz Sep 9, 2024
fb3666b
adjust mobile import, cleanup mobile implementation
TMisiukiewicz Sep 9, 2024
179bdf3
Merge remote-tracking branch 'upstream/main' into feat/import-onyx-state
TMisiukiewicz Sep 9, 2024
71db305
Merge remote-tracking branch 'upstream/main' into feat/import-onyx-state
TMisiukiewicz Sep 10, 2024
291402a
display only file upload on native
TMisiukiewicz Sep 10, 2024
077432c
force offline when importing on mobile
TMisiukiewicz Sep 11, 2024
65983c0
reduce diff across platforms
TMisiukiewicz Sep 11, 2024
3397526
add onyx key to watch if state is imported
TMisiukiewicz Sep 11, 2024
b6f375a
Merge remote-tracking branch 'upstream/main' into feat/import-onyx-state
TMisiukiewicz Sep 11, 2024
21205ae
make sure loader is hidden
TMisiukiewicz Sep 11, 2024
ce1ae7e
code cleanup
TMisiukiewicz Sep 11, 2024
dbc414a
add imported state button for resetting onyx state
TMisiukiewicz Sep 16, 2024
fc84004
Merge remote-tracking branch 'upstream/main' into feat/import-onyx-state
TMisiukiewicz Sep 16, 2024
d8102f9
fix crash when reportID is empty after clearing state
TMisiukiewicz Sep 16, 2024
2dd1e91
fix lint
TMisiukiewicz Sep 16, 2024
e756a8b
code review updates
TMisiukiewicz Sep 16, 2024
9a7a330
fix error when importing the file
TMisiukiewicz Sep 20, 2024
eba5ae0
Merge remote-tracking branch 'upstream/main' into feat/import-onyx-state
TMisiukiewicz Sep 20, 2024
35fd5f1
migrate from withOnyx to useOnyx
TMisiukiewicz Sep 20, 2024
072afdd
fix typecheck
TMisiukiewicz Sep 20, 2024
e55113b
Merge branch 'main' into feat/import-onyx-state
TMisiukiewicz Sep 23, 2024
62be7da
Merge remote-tracking branch 'upstream/main' into feat/import-onyx-state
TMisiukiewicz Sep 24, 2024
778934a
fix lint
TMisiukiewicz Sep 24, 2024
c3853c5
fix displaying messages and composer
TMisiukiewicz Sep 24, 2024
29e7857
Merge remote-tracking branch 'upstream/main' into feat/import-onyx-state
TMisiukiewicz Sep 25, 2024
8d91eb8
update App actions
TMisiukiewicz Sep 25, 2024
10bbee3
fix onyx set error
TMisiukiewicz Sep 25, 2024
80d11b3
don't restore sequential queue when going back from imported state
TMisiukiewicz Sep 25, 2024
6f7ce2a
Merge remote-tracking branch 'upstream/main' into feat/import-onyx-state
TMisiukiewicz Sep 27, 2024
aeb02c3
properly get reportNameValuePairs
TMisiukiewicz Sep 27, 2024
601085b
remove unnecessary rendering condition
TMisiukiewicz Sep 27, 2024
8ed35d9
create base component and display modal when file is invalid
TMisiukiewicz Sep 27, 2024
ca55510
navigate to home on clear, do not import preferred theme
TMisiukiewicz Sep 27, 2024
619414d
default reportID as string
TMisiukiewicz Sep 30, 2024
7265c23
fix linter
TMisiukiewicz Sep 30, 2024
f41d5c5
Merge remote-tracking branch 'upstream/main' into feat/import-onyx-state
TMisiukiewicz Oct 1, 2024
b28e3f2
Merge remote-tracking branch 'upstream/main' into feat/import-onyx-state
TMisiukiewicz Oct 1, 2024
e152883
Merge remote-tracking branch 'upstream/main' into feat/import-onyx-state
TMisiukiewicz Oct 2, 2024
c6ff8e3
disable force offline when in import mode
TMisiukiewicz Oct 2, 2024
6bdc95f
remove betas from omitted keys
TMisiukiewicz Oct 2, 2024
3e305c3
fix data transformation in export
TMisiukiewicz Oct 2, 2024
2790ba4
handle null value when masking data
TMisiukiewicz Oct 3, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion src/ONYXKEYS.ts
Original file line number Diff line number Diff line change
Expand Up @@ -419,6 +419,9 @@ const ONYXKEYS = {
/** Stores the route to open after changing app permission from settings */
LAST_ROUTE: 'lastRoute',

/** Stores the information if user loaded the Onyx state through Import feature */
IS_USING_IMPORTED_STATE: 'isUsingImportedState',

/** Stores the information about the saved searches */
SAVED_SEARCHES: 'nvp_savedSearches',

Expand Down Expand Up @@ -983,9 +986,9 @@ type OnyxValuesMapping = {
[ONYXKEYS.APPROVAL_WORKFLOW]: OnyxTypes.ApprovalWorkflowOnyx;
[ONYXKEYS.IMPORTED_SPREADSHEET]: OnyxTypes.ImportedSpreadsheet;
[ONYXKEYS.LAST_ROUTE]: string;
[ONYXKEYS.IS_USING_IMPORTED_STATE]: boolean;
[ONYXKEYS.SHOULD_SHOW_SAVED_SEARCH_RENAME_TOOLTIP]: boolean;
};

type OnyxValues = OnyxValuesMapping & OnyxCollectionValuesMapping & OnyxFormValuesMapping & OnyxFormDraftValuesMapping;

type OnyxCollectionKey = keyof OnyxCollectionValuesMapping;
Expand Down
23 changes: 10 additions & 13 deletions src/components/AttachmentPicker/index.native.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,7 @@ import CONST from '@src/CONST';
import type {TranslationPaths} from '@src/languages/types';
import type IconAsset from '@src/types/utils/IconAsset';
import launchCamera from './launchCamera/launchCamera';
import type BaseAttachmentPickerProps from './types';

type AttachmentPickerProps = BaseAttachmentPickerProps & {
/** If this value is true, then we exclude Camera option. */
shouldHideCameraOption?: boolean;
};
import type AttachmentPickerProps from './types';

type Item = {
/** The icon associated with the item. */
Expand Down Expand Up @@ -112,7 +107,7 @@ const getDataForUpload = (fileData: FileResponse): Promise<FileObject> => {
* a callback. This is the ios/android implementation
* opening a modal with attachment options
*/
function AttachmentPicker({type = CONST.ATTACHMENT_PICKER_TYPE.FILE, children, shouldHideCameraOption = false}: AttachmentPickerProps) {
function AttachmentPicker({type = CONST.ATTACHMENT_PICKER_TYPE.FILE, children, shouldHideCameraOption = false, shouldHideGalleryOption = false}: AttachmentPickerProps) {
const styles = useThemeStyles();
const [isVisible, setIsVisible] = useState(false);

Expand Down Expand Up @@ -218,17 +213,19 @@ function AttachmentPicker({type = CONST.ATTACHMENT_PICKER_TYPE.FILE, children, s

const menuItemData: Item[] = useMemo(() => {
const data: Item[] = [
{
icon: Expensicons.Gallery,
textTranslationKey: 'attachmentPicker.chooseFromGallery',
pickAttachment: () => showImagePicker(launchImageLibrary),
},
{
icon: Expensicons.Paperclip,
textTranslationKey: 'attachmentPicker.chooseDocument',
pickAttachment: showDocumentPicker,
},
];
if (!shouldHideGalleryOption) {
data.unshift({
icon: Expensicons.Gallery,
textTranslationKey: 'attachmentPicker.chooseFromGallery',
pickAttachment: () => showImagePicker(launchImageLibrary),
});
}
if (!shouldHideCameraOption) {
data.unshift({
icon: Expensicons.Camera,
Expand All @@ -238,7 +235,7 @@ function AttachmentPicker({type = CONST.ATTACHMENT_PICKER_TYPE.FILE, children, s
}

return data;
}, [showDocumentPicker, showImagePicker, shouldHideCameraOption]);
}, [showDocumentPicker, shouldHideGalleryOption, shouldHideCameraOption, showImagePicker]);

const [focusedIndex, setFocusedIndex] = useArrowKeyFocusManager({initialFocusedIndex: -1, maxIndex: menuItemData.length - 1, isActive: isVisible});

Expand Down
4 changes: 4 additions & 0 deletions src/components/AttachmentPicker/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,10 @@ type AttachmentPickerProps = {
type?: ValueOf<typeof CONST.ATTACHMENT_PICKER_TYPE>;

acceptedFileTypes?: Array<ValueOf<typeof CONST.API_ATTACHMENT_VALIDATIONS.ALLOWED_RECEIPT_EXTENSIONS>>;

shouldHideCameraOption?: boolean;

shouldHideGalleryOption?: boolean;
};

export default AttachmentPickerProps;
59 changes: 59 additions & 0 deletions src/components/ImportOnyxState/BaseImportOnyxState.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
import React from 'react';
import type {FileObject} from '@components/AttachmentModal';
import AttachmentPicker from '@components/AttachmentPicker';
import DecisionModal from '@components/DecisionModal';
import * as Expensicons from '@components/Icon/Expensicons';
import MenuItem from '@components/MenuItem';
import useLocalize from '@hooks/useLocalize';
import useResponsiveLayout from '@hooks/useResponsiveLayout';
import useThemeStyles from '@hooks/useThemeStyles';

function BaseImportOnyxState({
onFileRead,
isErrorModalVisible,
setIsErrorModalVisible,
}: {
onFileRead: (file: FileObject) => void;
isErrorModalVisible: boolean;
setIsErrorModalVisible: (value: boolean) => void;
}) {
const {translate} = useLocalize();
const styles = useThemeStyles();
const {isSmallScreenWidth} = useResponsiveLayout();

return (
<>
<AttachmentPicker
acceptedFileTypes={['text']}
shouldHideCameraOption
shouldHideGalleryOption
>
{({openPicker}) => {
return (
<MenuItem
icon={Expensicons.Upload}
title={translate('initialSettingsPage.troubleshoot.importOnyxState')}
wrapperStyle={[styles.sectionMenuItemTopDescription]}
onPress={() => {
openPicker({
onPicked: onFileRead,
});
}}
/>
);
}}
</AttachmentPicker>
<DecisionModal
title={translate('initialSettingsPage.troubleshoot.invalidFile')}
prompt={translate('initialSettingsPage.troubleshoot.invalidFileDescription')}
isSmallScreenWidth={isSmallScreenWidth}
onSecondOptionSubmit={() => setIsErrorModalVisible(false)}
secondOptionText={translate('common.ok')}
isVisible={isErrorModalVisible}
onClose={() => setIsErrorModalVisible(false)}
/>
</>
);
}

export default BaseImportOnyxState;
105 changes: 105 additions & 0 deletions src/components/ImportOnyxState/index.native.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
import React, {useState} from 'react';
import RNFS from 'react-native-fs';
import Onyx from 'react-native-onyx';
import type {FileObject} from '@components/AttachmentModal';
import {KEYS_TO_PRESERVE, setIsUsingImportedState} from '@libs/actions/App';
import {setShouldForceOffline} from '@libs/actions/Network';
import Navigation from '@libs/Navigation/Navigation';
import type {OnyxValues} from '@src/ONYXKEYS';
import ROUTES from '@src/ROUTES';
import BaseImportOnyxState from './BaseImportOnyxState';
import type ImportOnyxStateProps from './types';
import {cleanAndTransformState} from './utils';

const CHUNK_SIZE = 100;

function readFileInChunks(fileUri: string, chunkSize = 1024 * 1024) {
const filePath = decodeURIComponent(fileUri.replace('file://', ''));

return RNFS.exists(filePath)
.then((exists) => {
if (!exists) {
throw new Error('File does not exist');
}
return RNFS.stat(filePath);
})
.then((fileStats) => {
const fileSize = fileStats.size;
let fileContent = '';
const promises = [];

// Chunk the file into smaller parts to avoid memory issues
for (let i = 0; i < fileSize; i += chunkSize) {
promises.push(RNFS.read(filePath, chunkSize, i, 'utf8').then((chunk) => chunk));
}

// After all chunks have been read, join them together
return Promise.all(promises).then((chunks) => {
fileContent = chunks.join('');

return fileContent;
});
});
}

function chunkArray<T>(array: T[], size: number): T[][] {
const result = [];
for (let i = 0; i < array.length; i += size) {
result.push(array.slice(i, i + size));
}
return result;
}

function applyStateInChunks(state: OnyxValues) {
const entries = Object.entries(state);
const chunks = chunkArray(entries, CHUNK_SIZE);

let promise = Promise.resolve();
chunks.forEach((chunk) => {
const partialOnyxState = Object.fromEntries(chunk) as Partial<OnyxValues>;
promise = promise.then(() => Onyx.multiSet(partialOnyxState));
});

return promise;
Comment on lines +57 to +63
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct me if I am wrong but it looks like we are overriding the promise from previous chunk with a promise from next chunk. Is this desired behaviour?

What happens with promises generated from chunks before the very last one?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it creates a new promise that waits for the previous promise to resolve before applying the next chunk, creating a chain of promises. This ensures that the Onyx.multiSet operation for each chunk will execute in sequence, waiting for the previous operation to complete

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh right, that's nice!

Thanks for explanation ❤️

}

export default function ImportOnyxState({setIsLoading, isLoading}: ImportOnyxStateProps) {
const [isErrorModalVisible, setIsErrorModalVisible] = useState(false);

const handleFileRead = (file: FileObject) => {
if (!file.uri) {
return;
}

setIsLoading(true);
readFileInChunks(file.uri)
.then((fileContent) => {
const transformedState = cleanAndTransformState<OnyxValues>(fileContent);
setShouldForceOffline(true);
Onyx.clear(KEYS_TO_PRESERVE).then(() => {
applyStateInChunks(transformedState).then(() => {
setIsUsingImportedState(true);
Navigation.navigate(ROUTES.HOME);
});
});
})
.catch(() => {
setIsErrorModalVisible(true);
})
.finally(() => {
setIsLoading(false);
});

if (isLoading) {
setIsLoading(false);
}
};

return (
<BaseImportOnyxState
onFileRead={handleFileRead}
isErrorModalVisible={isErrorModalVisible}
setIsErrorModalVisible={setIsErrorModalVisible}
/>
);
}
59 changes: 59 additions & 0 deletions src/components/ImportOnyxState/index.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
import React, {useState} from 'react';
import Onyx from 'react-native-onyx';
import type {FileObject} from '@components/AttachmentModal';
import {KEYS_TO_PRESERVE, setIsUsingImportedState} from '@libs/actions/App';
import {setShouldForceOffline} from '@libs/actions/Network';
import Navigation from '@libs/Navigation/Navigation';
import type {OnyxValues} from '@src/ONYXKEYS';
import ROUTES from '@src/ROUTES';
import BaseImportOnyxState from './BaseImportOnyxState';
import type ImportOnyxStateProps from './types';
import {cleanAndTransformState} from './utils';

export default function ImportOnyxState({setIsLoading, isLoading}: ImportOnyxStateProps) {
const [isErrorModalVisible, setIsErrorModalVisible] = useState(false);

const handleFileRead = (file: FileObject) => {
if (!file.uri) {
return;
}

setIsLoading(true);
const blob = new Blob([file as BlobPart]);
const response = new Response(blob);

response
.text()
.then((text) => {
const fileContent = text;
const transformedState = cleanAndTransformState<OnyxValues>(fileContent);
setShouldForceOffline(true);
Onyx.clear(KEYS_TO_PRESERVE).then(() => {
Onyx.multiSet(transformedState)
.then(() => {
setIsUsingImportedState(true);
Navigation.navigate(ROUTES.HOME);
})
.finally(() => {
setIsLoading(false);
});
});
})
.catch(() => {
setIsErrorModalVisible(true);
setIsLoading(false);
});

if (isLoading) {
setIsLoading(false);
}
};

return (
<BaseImportOnyxState
onFileRead={handleFileRead}
isErrorModalVisible={isErrorModalVisible}
setIsErrorModalVisible={setIsErrorModalVisible}
/>
);
}
6 changes: 6 additions & 0 deletions src/components/ImportOnyxState/types.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
type ImportOnyxStateProps = {
isLoading: boolean;
setIsLoading: (isLoading: boolean) => void;
};

export default ImportOnyxStateProps;
53 changes: 53 additions & 0 deletions src/components/ImportOnyxState/utils.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
import cloneDeep from 'lodash/cloneDeep';
import type {UnknownRecord} from 'type-fest';
import ONYXKEYS from '@src/ONYXKEYS';

// List of Onyx keys from the .txt file we want to keep for the local override
const keysToOmit = [ONYXKEYS.ACTIVE_CLIENTS, ONYXKEYS.BETAS, ONYXKEYS.FREQUENTLY_USED_EMOJIS, ONYXKEYS.NETWORK, ONYXKEYS.CREDENTIALS, ONYXKEYS.SESSION, ONYXKEYS.PREFERRED_THEME];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would make sense to export betas as well, since having or not having a beta may be the source of a bug, was this discussed somewhere?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can betas differ across different accounts? If yes, let's definitely import them.

I was checking it on a few accounts of mine and each one of them had the same array of betas, that's why my initial thought was that it's safe to omit it

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah right, it can definitely be different sometimes


function isRecord(value: unknown): value is Record<string, unknown> {
return typeof value === 'object' && !Array.isArray(value) && value !== null;
}

function transformNumericKeysToArray(data: UnknownRecord): UnknownRecord | unknown[] {
const dataCopy = cloneDeep(data);
if (!isRecord(dataCopy)) {
return Array.isArray(dataCopy) ? (dataCopy as UnknownRecord[]).map(transformNumericKeysToArray) : (dataCopy as UnknownRecord);
}

const keys = Object.keys(dataCopy);

if (keys.length === 0) {
return dataCopy;
}
const allKeysAreNumeric = keys.every((key) => !Number.isNaN(Number(key)));
const keysAreSequential = keys.every((key, index) => parseInt(key, 10) === index);
if (allKeysAreNumeric && keysAreSequential) {
return keys.map((key) => transformNumericKeysToArray(dataCopy[key] as UnknownRecord));
}

for (const key in dataCopy) {
if (key in dataCopy) {
dataCopy[key] = transformNumericKeysToArray(dataCopy[key] as UnknownRecord);
}
}

return dataCopy;
}

function cleanAndTransformState<T>(state: string): T {
const parsedState = JSON.parse(state) as UnknownRecord;

Object.keys(parsedState).forEach((key) => {
const shouldOmit = keysToOmit.some((onyxKey) => key.startsWith(onyxKey));

if (shouldOmit) {
delete parsedState[key];
}
});

const transformedState = transformNumericKeysToArray(parsedState) as T;
return transformedState;
}

export {transformNumericKeysToArray, cleanAndTransformState};
Loading
Loading