Functionality for partial model loading

[blester125]

The other day @dptam was trying to use Git-Theta to partially load models (only loading/saving a subset of parameters). This is easy to do with git-theta but there isn't a user-facing way to do it. We should add that. The question is mostly what should the API be for that? Should there be a list of parameter names to load, a regex, etc? Thoughts @nkandpa2 @craffel?

Also, the merge script currently loads parameters as needed during merges, but it keeps them around in memory after the merge. Thus, we can merge to really large models if only a few parameters are changed, but not if all the parameters are going to be merged. We should add the ability to save merged parameters to disk (with the ability to clean them up if the merge is aborted) and free parameter memory so enable the merging of really big models.

[craffel]

This seems like it'd be checkpoint format-dependent, right? I.e. whether a given checkpoint format supports reading only a subset of parameters? Git-Theta's native format would allow this, but I'm not sure there would be a clean way to do this from the command line (since the command line interface generally assumes that we are going to operate on a whole checkpoint file). If the goal is just to do it from Python, we could use the new save_to_git/load_from_git functions for this?

[blester125]

Yeah, sorry, I should have been more specific, I was imagining this as something from python. Basically trying to add light-weight functionality that's like "I have my model saved in Git-Theta, what does that get me?"

[craffel]

I see, do you think there would be a lightweight way to add this to the new save_to_git/load_from_git functions?