Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Default ctx initialization improvements, and custom buffer size #256

Merged
merged 8 commits into from
Jan 3, 2020

Conversation

ihnorton
Copy link
Member

@ihnorton ihnorton commented Jan 3, 2020

  • initialize default_ctx lazily, allowing user control over global per-process settings when necessary (e.g. sm.num_tbb_threads)
    • default_ctx may be initialized with custom settings by calling tiledb.libtiledb.initialize_ctx()
    • TBB uses the thread count set at the first pool creation, so a call to tiledb.Ctx(...) should specify number of threads (if needed) whenever called before an implicit call to default_ctx() (unlikely)
  • make default_ctx a delayed call in all method definitions
  • add support for user-controlled initial buffer sizes for queries with unknown result sizes, using new py.init_buffer_bytes parameter

@ihnorton ihnorton changed the title Default ctx initialize improvements, and custom buffer size Default ctx initialization improvements, and custom buffer size Jan 3, 2020
config = tiledb.Config({'sm.memory_budget': 2 * 1024**2})
# create context with 1 MB memory budget (2 MB total, 1 MB usable)
config = tiledb.Config({'sm.memory_budget': 2 * 1024**2,
'py.init_buffer_bytes': 1024**2 })
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stavrospapadopoulos do you foresee any problem with putting a custom value in the Config object, in order to maintain a consistent API? Seems to work... (alternatively, could have a separate py_config object)

Copy link
Member

@Shelnutt2 Shelnutt2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, the global context changes are going to be nice to fix the tbb issue for multi-process local setups!

Also thanks for the quick turn around on py.init_buffer_bytes. py.init_buffer_bytes is per buffer (attr/coords) right?

@ihnorton
Copy link
Member Author

ihnorton commented Jan 3, 2020

py.init_buffer_bytes is per buffer (attr/coords) right?

yes

@ihnorton ihnorton force-pushed the ihn/init_size_cfg branch 2 times, most recently from 4919e88 to cf0ea27 Compare January 3, 2020 18:48
(see detailed explanation in code comment)
@ihnorton ihnorton merged commit 6f2664d into dev Jan 3, 2020
@ihnorton ihnorton deleted the ihn/init_size_cfg branch January 3, 2020 21:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants