Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix memory leak when breaking mid-way in _get_objects and _get_ids #266

Merged
merged 2 commits into from
Jun 23, 2023

Conversation

akariv
Copy link
Contributor

@akariv akariv commented Jun 23, 2023

This PR fixes a memory leak that is caused from resources not being freed when using _get_objects and _get_ids and not consuming all of the items in the generator.

Explanation - this is the way these functions are implemented currently:

>>> def objects():
...   try:
...     for i in range(10):
...       yield i
...     print('release mem 1')
...   except Exception as e:
...     print('release mem 2')
...     raise

When iterating the objects, but breaking mid way, the memory is not released:

>>> for x in objects():
...   print('got', x)
...   if x == 5: break
... 
got 0
got 1
got 2
got 3
got 4
got 5
>>> 

If we use finally instead, it simplifies the code and ensures that memory is always released:

>>> def objects():
...   try:
...     for i in range(10):
...       yield i
...   finally:
...     print('release mem 3')

This works when consuming the entire iterator:

>>> for x in objects():
...   print('got', x)
... 
got 0
got 1
...
got 8
got 9
release mem 3
>>> 

And when breaking in the middle:

>>> for x in objects():
...   print('got', x)
...   if x == 5: break
... 
got 0
got 1
got 2
got 3
got 4
got 5
release mem 3

Copy link
Collaborator

@adamjstewart adamjstewart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems correct to me but I'll let @hobu or someone else review too.

@hobu hobu merged commit cd5ac27 into Toblerity:master Jun 23, 2023
36 checks passed
@akariv akariv deleted the fix/mem-leak-generator branch June 24, 2023 18:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants