Add unique log files for each mongo worker

[Cmurilochem]

Aim

Allow mongo workers to write output data to separate files.

Idea

Redirect the output of each subprocess.Popen to unique log files (stdout) for each mongo worker. These files are named mongo-worker_n_output_path.log, where n=0, 1, 2, ..., n is the unique id of the worker and output_path corresponds to the runcard name or OUTPUT if n3fit -o OUTPUT was used.

Notes/Drawbacks

The mongo-worker_n_output_path.log files will only contain data output by any function/method called by fmin, i.e., by ModelTrainer.hyperparametrizable which consists of details of the hyperparameters being evaluated and info about epochs. Other infos and warnings are still printed to standard output (screen), including trial numbers which might not be ideal.

Either way, it appears that with this simple change we can prevent the overwhelming/chaotic information that was previously displayed on the screen.

Tasks

• Add stdout redirection files to start_mongo_workers Popen.
• Add name of the runcard/output to the mongo_worker log file
• Test in snellius

[Radonirinaunimi]

Thanks @Cmurilochem for this. This would be useful indeed to know what happened during a trial.

[Radonirinaunimi]

LGTM! Thanks again.

[Cmurilochem]

LGTM! Thanks again.

Thanks @Radonirinaunimi. Not sure if @RoyStegeman wants to have a look at it, but if not, could you merge it ?
It seems that I do not have permissions to so any more.

[RoyStegeman]

Thanks @Cmurilochem! This looks good to me as well, feel free to merge after making these changes. If you agree with them of course

[Cmurilochem]

Thanks @Cmurilochem! This looks good to me as well, feel free to merge after making these changes. If you agree with them of course

Thanks @RoyStegeman. Suggestions accepted. It seems that I am not authorized to merge this PR. Please, could you do that ?

[RoyStegeman]

Ah, I hadn't realised you were not registered as a member of this organisation