Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make Complex From Objects faster by creating a table and indexing #554

Closed
thcrock opened this issue Dec 17, 2018 · 0 comments
Closed

Make Complex From Objects faster by creating a table and indexing #554

thcrock opened this issue Dec 17, 2018 · 0 comments
Assignees

Comments

@thcrock
Copy link
Contributor

thcrock commented Dec 17, 2018

If a feature's from object is complex enough to take even a few seconds, it can greatly increase the execution time if a lot of as-of-dates are present. This is because everything is parallelized over the as-of-date, so 1000 as-of-dates means the same query gets executed 1000 times, both for initial feature generation and the UNION ALL at the end ( https://github.com/dssg/triage/blob/master/src/triage/component/collate/spacetime.py#L214 ).

We can speed this up by preprocessing to create tables out of these and indexing them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant