Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Benchmark equals() functions #268

Closed
samchon opened this issue Oct 25, 2022 · 13 comments
Closed

Benchmark equals() functions #268

samchon opened this issue Oct 25, 2022 · 13 comments
Assignees
Labels
enhancement New feature or request

Comments

@samchon
Copy link
Owner

samchon commented Oct 25, 2022

Special benchmark program for TypeBox (@sinclairzx81)

Those strict validation functions would be benchmarked:

  • equals()
  • assertEquals()
  • validateEquals()
@samchon samchon added the enhancement New feature or request label Oct 25, 2022
@samchon samchon self-assigned this Oct 25, 2022
@samchon
Copy link
Owner Author

samchon commented Oct 25, 2022

At first, compared with TypeBox (@sinclairzx81). Other libraries also would be measured soon.

equals

Components typescript-json typebox
object (hierarchical) 3698.603500088386 16875.067120100233
object (recursive) 1324.033552151714 9931.947765311752
object (union, explicit) 1288.4007029876975 3178.8677853586573
object (union, implicit) 399.7441052823981 2224.5449630564062
array (recursive) 118.92945763913505 1015.9369847957502
array (union, explicit) 174.8573532118535 642.3728813559322
array (union, implicit) 123.3600583090379 427.71759432200224
ultimate union 90.92578986039676 196.709323583181

assertEquals

Components typescript-json typebox
object (hierarchical) 3741.0123412627436 637.7304456774656
object (recursive) 4607.495429616088 281.0145193898181
object (union, explicit) 1345.9899981478052 96.90493736182756
object (union, implicit) 1180.2075876879026 70.13081395348837
array (recursive) 319.2737942417018 28.963701335339476
array (union, explicit) 317.6987907658483 13.789195315451456
array (union, implicit) 214.1457331195439 6.40542577241899
ultimate union 163.4788884879105 3.4470246734397683

validateEquals

Components typescript-json typebox
object (hierarchical) 3189.8826979472137 607.4251056791031
object (recursive) 1709.7422774629865 270.8530357480613
object (union, explicit) 1132.990056818182 94.31165462293866
object (union, implicit) 931.6239316239316 68.50335070737155
array (recursive) 190.9741331865713 29.533483822422877
array (union, explicit) 261.09336173233936 14.17910447761194
array (union, implicit) 189.9912203687445 6.2065074290013165
ultimate union 93.9513242662849 3.516564871367759

@sinclairzx81
Copy link
Contributor

sinclairzx81 commented Oct 25, 2022

@samchon Heya, if you are going to setup some additional benchmarks for TypeBox, let's get the validation routines aligned.

So from the documentation.

// ALLOW SUPERFLUOUS PROPERTIES
TSON.assertType<T>(input); // throws exception
TSON.is<T>(input); // returns boolean value
TSON.validate<T>(input); // archives all errors

// DO NOT ALLOW SUPERFLUOUS PROPERTIES
TSON.equals<T>(input); // returns boolean value
TSON.assertEquals<T>(input); // throws exception
TSON.validateEquals<T>(input); // archives all errors

Would translate to the following logic per test.

// ------------------------------------------------------------

is() // returns boolean value

(input) => UltimateUnion.Check(input)

// ------------------------------------------------------------

equals() // returns boolean value

(input) => UltimateUnion.Check(input)

// ------------------------------------------------------------

assertEquals() // throws exception

(input) => { if(!UltimateUnion.Check(input)) throw Error('invalid') }

// ------------------------------------------------------------

validate() // archives all errors

(input) => UltimateUnion.Check(input) ? [] : [...UltimateUnion.Errors(input)]

// ------------------------------------------------------------

validateEquals() // archives all errors

(input) => UltimateUnion.Check(input) ? [] : [...UltimateUnion.Errors(input)]

Happy to setup two sets of schemas for is and equals if you need. Keep me posted!

@samchon
Copy link
Owner Author

samchon commented Oct 26, 2022

(input) => { if(!UltimateUnion.Check(input)) throw Error('invalid') }
export class TypeGuardError extends Error {
    public readonly method: string;
    public readonly path: string | undefined;
    public readonly expected: string;
    public readonly value: any;
}

TypeGuardError thrown by assertType() and assertEquals() functions has type error detailed info like above.

Therefore your suggested code about those functions is not acceptable. If I accept that code, it is not measuring assertType() function, but measuring is() function twice. I think ordinary code is enough reasonable because it does not take full errors, but get only the 1st error element.

UltimateUnion.Errors(input).next().value // GET ONLY ONE

@sinclairzx81
Copy link
Contributor

What about ?

(input) => { if(!UltimateUnion.Check(input)) throw Error(UltimateUnion.Errors(input).next().value.message) }

@samchon
Copy link
Owner Author

samchon commented Oct 26, 2022

I'm not measuring typescript-json and io-ts like below.

Please understand that it is measuring assertType() function's performance. Not for is() function.

(input) => if (!TSON.is(input)) TSON.assertType(input);
(input) => if (!IoTsObjectRecursive.is(input)) IoTsObjectRecursive(input);

@sinclairzx81
Copy link
Contributor

TypeBox doesn't have an assert function, but if it did it would be.

if(!UltimateUnion.Check(input)) throw Error(UltimateUnion.Errors(input).next().value.message)

For reference, the type check implementation for Fastify's type provider https://github.com/fastify/fastify-type-provider-typebox/blob/main/index.ts#L23-L39

@samchon
Copy link
Owner Author

samchon commented Oct 26, 2022

Then why the Errors function returns an iterator typed value instead of array?

@sinclairzx81
Copy link
Contributor

sinclairzx81 commented Oct 26, 2022

Errors are treated as an optional query one would runs "after" a validation failure.

The Error query returns an iterator because the number of errors may be large, and populating large arrays upfront may be unnecessary memory overhead. The iterator allows implementers to select as many errors as necessary, and provides options for those concerned about the allocation of large arrays for validation error cases.

@samchon
Copy link
Owner Author

samchon commented Oct 26, 2022

You mean that Errors function does not find all of type errors at one time. Therefore, UltimateUnion.Errors(input).next().value.message will find only the first type error. Then purpose of that function is exactly same with assertType() function and only whether throwing exception or not is different.

I think there's not any problem. What's the problem? Why only TypeBox must be get benefit from is() function?

@sinclairzx81
Copy link
Contributor

Why only TypeBox must be get benefit from is() function?

How is it benefitting? TypeBox doesn't even have an Assert function :D We're having to implement one for the purposes of this test.

Anyway, why would you run a relatively slow error reporting pass when you can so easily test for the existence of errors with .Check() first? Also, wouldn't if(!condition) { throw } be the most typical implementation of an Assert like function?

@samchon
Copy link
Owner Author

samchon commented Oct 26, 2022

The assert benchmark program is designed to measure error detection function. I repeat that if you add is() condition before, it is not measuring error detection logic speed, but just measuring speed of is() function. Whether throwing exception or not, it is not important.

Please understand that in the assert benchmark program, none of other validator libraries are boosting up performance through is() function check.

@sinclairzx81
Copy link
Contributor

The assert benchmark program is designed to measure error detection logic.

But the .Check() function is the error detector. The .Errors() function is the error reporter.

I think the issue here is that TSON.assertType() combines detection and error reporting into the same function where as TypeBox splits these concerns across two functions and expects the end user to implement .Assert() themselves.

As I say, if I was to implement a .Assert() in TypeBox, it would be implemented as

if(!T.Check(value)) throw Error(T.Errors(value).next().value.message)
//  ^ detect                    ^ report

@samchon
Copy link
Owner Author

samchon commented Oct 26, 2022

I'll change assert benchmark program to really generate the type errored data and re-process it after you implement the Assert() function.

samchon added a commit that referenced this issue Oct 27, 2022
samchon added a commit that referenced this issue Oct 27, 2022
samchon added a commit that referenced this issue Oct 28, 2022
Close #268 - refactored entire benchmark program for TypeBox
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants