Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Add support for yield to bulk inserts #504

Closed
kxleee opened this issue Sep 17, 2023 · 3 comments
Closed

[FEATURE] Add support for yield to bulk inserts #504

kxleee opened this issue Sep 17, 2023 · 3 comments
Labels
enhancement New feature or request

Comments

@kxleee
Copy link

kxleee commented Sep 17, 2023

The bulk method cannot be used in combination with yield when batch inserts

@github-actions github-actions bot added the untriaged Need triage label Sep 17, 2023
@dtaivpp
Copy link

dtaivpp commented Sep 18, 2023

Hey @kxleee can you provide some additional information here to assist? For example are you using the async or sync client? Do you have some sample code you can share?

@saimedhi saimedhi removed the untriaged Need triage label Sep 19, 2023
@wbeckler wbeckler added the bug Something isn't working label Oct 3, 2023
@dblock dblock changed the title The bulk method cannot be used in combination with yield when performing bulk inserts [FEATURE] Add support for yield to bulk inserts Nov 10, 2023
@dblock
Copy link
Member

dblock commented Nov 10, 2023

@kxleee Do you have an example please?

@dblock dblock added enhancement New feature or request and removed bug Something isn't working labels Nov 10, 2023
@dblock
Copy link
Member

dblock commented Nov 18, 2023

The helpers such as streaming_bulk do support yielding data, e.g. https://github.com/opensearch-project/opensearch-py/blob/main/samples/bulk/bulk-helpers.py#L84

def _generate_data() -> Any:
    for i in range(100):
        yield {"_index": index_name, "_id": i, "value": i}


succeeded = []
failed = []
for success, item in helpers.streaming_bulk(client, actions=_generate_data()):
    if success:
        succeeded.append(item)
    else:
        failed.append(item)

I am going to close this since we didn't hear from @kxleee with an example. Let's reopen if they mean something else.

@dblock dblock closed this as completed Nov 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

5 participants