You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
MatrixStore caches the matrix until it's serialized (e.g. to send over the wire for multiprocessing), at which point it clears the matrix out. This comes into play because when we generate the model training and testing tasks, we check the matrix's labels to validate if there is more than one unique value. Before #560 was resolved, right after we did this check we initiated a multiprocess pool, so the MatrixStore was serialized and cache cleared. Now, we actually build the entire Experiment's train/test tasks in one go, which means that all of the train and test matrices stay in memory while we're building the tasks. This can easily result in out of memory errors.
There could be several ways to fix this, but the simplest is probably explicitly clearing the cache during the task-building phase.
The text was updated successfully, but these errors were encountered:
MatrixStore caches the matrix until it's serialized (e.g. to send over the wire for multiprocessing), at which point it clears the matrix out. This comes into play because when we generate the model training and testing tasks, we check the matrix's labels to validate if there is more than one unique value. Before #560 was resolved, right after we did this check we initiated a multiprocess pool, so the MatrixStore was serialized and cache cleared. Now, we actually build the entire Experiment's train/test tasks in one go, which means that all of the train and test matrices stay in memory while we're building the tasks. This can easily result in out of memory errors.
There could be several ways to fix this, but the simplest is probably explicitly clearing the cache during the task-building phase.
The text was updated successfully, but these errors were encountered: