What is our notion of best-fit for generation, prediction, and relaxation?

[sgbaird]

EDIT: see also issues with the "notion of best" label

Relaxation is probably the most straightforward - use some crystal distance. Prediction can be about checking against known allotropes, where we take the lowest crystal distance among the allotropes. Generation is the least straightforward. Perhaps a Pareto hypervolume metric via a fictitious adaptive design campaign (e.g. bulk modulus vs. energy above hull)? Perform hyperparameter optimization and then do DFT as the final validation.

[sgbaird]

Another option that struck me is using a time-split. For example:

1. Split Materials Project into two pieces based on a datetime split
2. unconditionally generate many crystal structures
   1. 10+ million? Maybe check convergence with number of generated structures (a hyperparameter of the metric)
3. check fraction of how many close matches with latter half of Materials Project entries to total number of latter half, with higher fraction--> better performance
   1. match tolerance(s) will be other hyperparameter(s) for the metric

[sgbaird]

Also can take a look at the model accuracy for Matbench task(s) as a way to probe the "quality" of the xtal2png representation from another perspective #50

[sgbaird]

DFT simulations will also be important as a high-cost validation.

[sgbaird]

From mp-time-split:

... MPTS-52 can be used with the metrics introduced in CDVAE's compute_metrics.py script (see txie-93/cdvae#10. ...

[sgbaird]

Having trouble getting CDVAE to run txie-93/cdvae#19, but can probably splice out the compute_metrics.py while that's getting sorted out.

[sgbaird]

compute_metrics.py seems to be tightly integrated with the rest of the codebase. Simplest solution might just be to fork CDVAE, make it pip- and conda-installable, and then include it as a dependency for matbench-genmetrics.

[sgbaird]

Might hold off on CDVAE metrics for now. See txie-93/cdvae#10

[sgbaird]

As an update, matbench-genmetrics runs in a reasonable time now https://github.com/sparks-baird/matbench-genmetrics/blob/main/notebooks/1.0-matbench-genmetrics-basic.ipynb