Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix index-out-of-bounds line glyph crash and run select tests with JIT disabled #683

Merged
merged 9 commits into from
Jan 7, 2019

Conversation

jonmmease
Copy link
Collaborator

@jonmmease jonmmease commented Jan 5, 2019

This PR closes #570 and #646

Based on the helpful debugging of @bardiche98 and @p3trus, I was able to construct a small test case that reproduces the conditions of the crash reported in #570 (See
4f7ae29 and the test_bug_570 test in datashader/tests/test_pandas.py).

As discussed in #570, this did turn out to be an index-out-of-bounds error. But since numba JIT-compiled code does not perform array bounds checking, this resulted in a non-deterministic kernel crash.

To more reliably reproduce and diagnose the issue, I made a few small changes to allow Datashader to operate with the numba jit compilation disabled (See ae920da).

I also added a new tox test configuration to run a subset of the Datashader unit tests with JIT compilation disabled (See 4219fba). With JIT compilation disabled (and without the fix described below), test_bug_570 reliably raises an IndexError and does not crash the kernel.

Finally, I actually fixed #570 in 105d766 by changing int to round in this section of the line glyph's map_onto_pixel function:

        # Note that sx and tx were designed so that
        # x_mapper(xmax) * sx + tx equals the width of the canvas in pixels
        #
        # Likewise, sy and ty were designed so that
        # y_mapper(ymax) * sy + ty equals the height of the canvas in pixels
        #
        # We round these results to integers (rather than casting to integers
        # with the int constructor) to handle cases where floating-point
        # precision errors results in a value just under the integer number
        # of pixels.
        xxmax = round(x_mapper(xmax) * sx + tx)
        yymax = round(y_mapper(ymax) * sy + ty)

In this particular test case, the canvas is 300 pixels in height, so we want yymax to end up as 300. But due to numerical precision subtleties y_mapper(ymax) * sy + ty == 299.99999999999994 and so int(y_mapper(ymax) * sy + ty) == 299 and yymax gets set to 299. By changing this to round, yymax ends up as 300 as it should.

Note: Waiting on #682 for travis test fixes

With jit disabled, this combination of x_range, y_range, width, height, and line
coordinates results in an index out of bounds exception.  With jit enabled,
this results in a non-deterministic segmentation fault.
The crash in GH 570 resulted from:

y_mapper(ymax) * sy + ty == 299.99999999999994

the `int` constructor truncates this down to 299, but we need it to
be rounded to 300.
@jbednar jbednar mentioned this pull request Jan 5, 2019
13 tasks
@philippjfr
Copy link
Member

So happy to see this finally being fixed.

@jbednar
Copy link
Member

jbednar commented Jan 5, 2019

Whew, that looks great! Yes, I'm very excited by the possibility that this is fixed, plus the new non-Numba tests that should help detect similar issues in the future. Can you please look at #259 and see if there are any tests that ensure that these changes don't cause a return to the original problem described there? If so, then this seems ready to merge; if not, then please add such tests.

@jonmmease
Copy link
Collaborator Author

Thanks! I took a look at #259 and the surrounding history and I think this is already locked in by several of the tests in datashader/tests/test_pandas.py. E.g.

https://github.com/pyviz/datashader/blob/34e67266af761736ef6d5c9a0fdad78e494d00a1/datashader/tests/test_pandas.py#L256-L274

But I still added two more simple cases in c3264e1 that would have failed previously based on your original observations in #259.

@jbednar
Copy link
Member

jbednar commented Jan 7, 2019

Perfect, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Datashader crashes python with "noisy" timeseries data.
3 participants