Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Numeric inputs have the wrong precision #77

Open
jonathanj opened this issue Jan 5, 2016 · 0 comments
Open

Numeric inputs have the wrong precision #77

jonathanj opened this issue Jan 5, 2016 · 0 comments
Labels

Comments

@jonathanj
Copy link
Member

The internal value for IntegerInput, DecimalInput, and other derived inputs, is stored as a JavaScript number. In most (all?) implementations, numbers are stored as double-precision binary floating point values. However, integer values are stored Python-side as Python ints (or longs), which have exact integer precision, and infinite range (although Axiom has range limits. Also, decimal values are stored as Decimal values on the Python side, which have prec=28 in the default context. Thus, values lose precision when sent from the server to the client; and values entered by the user also lose precision compared with the server-side capability.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant