feat: Add BigInt serialization #228

petarvujovic98 · 2022-09-21T08:46:36Z

Add custom serialization function in utils
Improve types in near-bindgen and use serialization function Add separate serialization for function returns
Add serialization to collections
Add serialization tests
Add serialization build and tests scripts

Addresses #172.

This is just a design I came up with that we can iterate on.

Add custom serialization function in utils Improve types in near-bindgen and use serialization function Add separate serialization for function returns Add serialization to collections Add serialization tests Add serialization build and tests scripts

ailisp · 2022-09-21T09:43:16Z

src/collections/lookup-map.ts

@@ -12,7 +12,7 @@ export class LookupMap<DataType> {

  get(key: Bytes, options?: GetOptions<DataType>): DataType | null {
    const storageKey = this.keyPrefix + key;
-    const value = JSON.parse(near.storageRead(storageKey));
+    const value = deserialize(near.storageRead(storageKey));


It's great to make serialize / desalinize general / independent of the collections! Since we have GetOptions, it's possibly a good idea to make it as part of GetOptions, and have current utils.serialize/utils.deserialize as the default option.

@ailisp I have covered this in the latest commits. Let's discuss the approach I took there. I also added a serializer and deserializer option to the NearBindgen decorator

That looks great!

ailisp · 2022-09-21T09:53:08Z

src/utils.ts

+    if (typeof value === "bigint") {
+      return {
+        value: value.toString(),
+        [BIGINT_KEY]: BIGINT_BRAND,


This is very clever mechanism! A little extension based on your approach can become more general. We don't need both BIGINT_KEY and BIGINT_BRAND to indicate it's a bigint, so, maybe it can be TYPE_KEY and TYPE?

@ailisp Could you perhaps give an example or code snippet of what you are thinking about when you say more general?

For example, if later we also want to support serialize and deserialize Date, we can extend your serialize code for date:

return { value: value.toString(), [TYPE_KEY]: DATE_TYPE, }

And in deserialization, we can always check TYPE_KEY, see what type it is and call bigint / date deserialization correspondingly

@ailisp I added something similar along with an implementation of Date serialization and an accompanying test in this commit

ailisp

I like your solution, it solves the bigint serialization in a clean, idiomatic way. I leave some comments to extend your framework a little bit. /cc @volovyks

Add beforeEach and afterEach instead of before and after Remove unused account import

Add serialization and deserialization to options Add serialization and deserialization options to NearBindgen Extract some common logic to utils Refactor unordered-map types

Add exports to class declarations Remove unused variables Fix `any` types Fix `moduleResolution` in examples Format `jsconfig.json` `includes` entry for CLI

petarvujovic98 · 2022-09-22T09:49:59Z

@volovyks @ailisp We should change examples and tests to use the new serialization, and to use the options we added in #213.
Do we want to do this in this PR or create a separate PR?

Improve types for decorators and babel plugin Improve performance of bindgen exporter and clean up code

Update FT example to remove redundant BigInt calls and use defaultValue

ailisp · 2022-09-22T15:57:45Z

Do we want to do this in this PR or create a separate PR?

I think create a separate PR would be great!

Generalize type branding Add `Date` object serialization and tests

ailisp

Wow you're awesome! I was just mention Date as an imaginary example and you've already made it with ! It will be helpful too, developers will be glad to serialize and deserialize date automatically.

@volovyks If you also agree with the change I think we can merge it

ailisp · 2022-09-26T02:02:38Z

src/utils.ts

+  };
+
+  return JSON.stringify(valueToSerialize, (_, value) => {
+    if (typeof value === "bigint") {


I think it's better to check type isinstance of Date instead of override Date.prototype?

Unfortunately, by the time you get the value argument inside the stringifyreplacer, the toJSON function of the Dateobject is already called so I cannot infer the type of the variable, I tried several ways to make it happen but this was the only one that worked, evem though I don't like it's hacky nature/playing around with prototypes

I see, thanks for clarification!

Also doesn't have luck with value - as you said it's already string, but I found this works: https://stackoverflow.com/a/54037861

Thank you. I just updated the Date serialization function with this suggestion and it seems to work as expected.

ailisp · 2022-09-26T02:03:30Z

tests/__tests__/test-date-serialization.ava.js

+  t.is(afterSet, newDate.toISOString());
+});
+
+test("get date field in milliseconds", async (t) => {


These tests are great 👍

Remove prototype override and add key based type checking Add `validateAccountId` util

Remove unneccesary character escape

volovyks

It was a joy to review it, thanks @petarvujovic98 ! Now we are close to borsh serialization!

feat: Add BigInt serialization

6afa6c1

Add custom serialization function in utils Improve types in near-bindgen and use serialization function Add separate serialization for function returns Add serialization to collections Add serialization tests Add serialization build and tests scripts

petarvujovic98 requested review from ailisp and volovyks as code owners September 21, 2022 08:46

fix: Add brand check on deserialization

ee7e0f6

ailisp reviewed Sep 21, 2022

View reviewed changes

ailisp approved these changes Sep 21, 2022

View reviewed changes

petarvujovic98 added 4 commits September 21, 2022 21:54

chore: Update tests

f56233c

Add beforeEach and afterEach instead of before and after Remove unused account import

feat: Add generalized serialization

a3859dc

Add serialization and deserialization to options Add serialization and deserialization options to NearBindgen Extract some common logic to utils Refactor unordered-map types

chore: Add type definitions for lodash-es

a9cb69d

chore: Fix ESLint warnings

b6a3866

Add exports to class declarations Remove unused variables Fix `any` types Fix `moduleResolution` in examples Format `jsconfig.json` `includes` entry for CLI

petarvujovic98 added 5 commits September 22, 2022 14:08

chore: Add type definitions for @babel/traverse

3b82668

chore: Improve types and reformat exporter

aab75a2

Improve types for decorators and babel plugin Improve performance of bindgen exporter and clean up code

fix: Add affected build files

e5824a7

fix: Fix ESLint errors

91684ae

chore: Update examples

b10bb1d

Update FT example to remove redundant BigInt calls and use defaultValue

ailisp linked an issue Sep 23, 2022 that may be closed by this pull request

BigInt serialization support #172

Closed

feat: Add more generic serialization

d843af6

Generalize type branding Add `Date` object serialization and tests

ailisp approved these changes Sep 26, 2022

View reviewed changes

ailisp reviewed Sep 26, 2022

View reviewed changes

petarvujovic98 added 2 commits September 27, 2022 10:14

feat: Improve date serialization

74159a8

Remove prototype override and add key based type checking Add `validateAccountId` util

fix: Fix lints in utils

c9bb9f1

Remove unneccesary character escape

petarvujovic98 mentioned this pull request Sep 27, 2022

No autocomlition in TS contracts #246

Closed

volovyks approved these changes Sep 27, 2022

View reviewed changes

volovyks merged commit e2776a5 into near:develop Sep 27, 2022

volovyks mentioned this pull request Oct 12, 2022

Add Storage Management to FT Example #253

Merged

ailisp mentioned this pull request Nov 14, 2022

JSON.parse will lead to loss of precision for large integers and interop issues #296

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add BigInt serialization #228

feat: Add BigInt serialization #228

petarvujovic98 commented Sep 21, 2022

ailisp Sep 21, 2022

petarvujovic98 Sep 22, 2022

ailisp Sep 22, 2022

ailisp Sep 21, 2022

petarvujovic98 Sep 22, 2022

ailisp Sep 22, 2022

petarvujovic98 Sep 23, 2022

ailisp left a comment

petarvujovic98 commented Sep 22, 2022

ailisp commented Sep 22, 2022

ailisp left a comment

ailisp Sep 26, 2022

petarvujovic98 Sep 26, 2022

ailisp Sep 27, 2022

ailisp Sep 27, 2022

petarvujovic98 Sep 27, 2022

ailisp Sep 26, 2022

volovyks left a comment •

edited

Loading

feat: Add BigInt serialization #228

feat: Add BigInt serialization #228

Conversation

petarvujovic98 commented Sep 21, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ailisp left a comment

Choose a reason for hiding this comment

petarvujovic98 commented Sep 22, 2022

ailisp commented Sep 22, 2022

ailisp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

volovyks left a comment • edited Loading

Choose a reason for hiding this comment

volovyks left a comment •

edited

Loading