clean_tables script to prepare a database for import can not handle too many rows in a particular table

[vickyszuchin]

Current Behavior

When running the clean_tables script to prepare a database for import, the process gets killed if there are too many rows in a particular table.

Expected Behavior

The clean_tables script to prepare a database for import, the process is successful regardless of the number of rows in a table.

Steps to Reproduce

1. Execute clean_tables script table with large table

Environment

Staging

Additional Context

Peer Dave Kennedy:

• In order to fix, delete rows should be batched in groups of 1000.
• Incorporate this code directly into the clean_tables script:

def delete_objects_in_batches(model):
total_deleted = 0
while True:
pks = list(model.objects.values_list('pk', flat=True)[:BATCH_SIZE])
if not pks:
break
with transaction.atomic():
deleted, _ = model.objects.filter(pk__in=pks).delete()
total_deleted += deleted
logger.info(f"Deleted {deleted} objects, total deleted: {total_deleted}")
sleep(0.1)
logger.info(f"Finished deleting. Total deleted: {total_deleted}")

Issue Links

No response

[vickyszuchin]

Per approval by Alysia to move this up a sprint since it's already in "In review" status. Moved from sprint 50 to sprint 49.

[vickyszuchin]

Per guideline, all "In review" tickets from Sprint 49 will moved forward at noon EST on Day 4 to current sprint (Sprint 50).