Risklist module for production

docs/sources/risklist/index.md → docs/sources/predictlist/index.md

@@ -1,85 +1,87 @@
-# Predictlist
-If you would like to generate a list of predictions on already-trained Triage model with new data, you can use the 'Predictlist' module.
-
-# Predict Foward with Existed Model
-Use an existing model object to generate predictions on new data.
+# Retrain and Predict
+Use an existing model group to retrain a new model on all the data up to the current date and then predict forward into the future.
 
 ## Examples
 Both examples assume you have already run a Triage Experiment in the past, and know these two pieces of information:
-1. A `model_id` from a Triage model that you want to use to generate predictions
-2. An `as_of_date` to generate your predictions on.
+1. A `model_group_id` from a Triage model group that you want to use to retrain a model and generate prediction
+2. A `today` to generate your predictions on.
 
 ### CLI
-`triage predictlist <model_id> <as_of_date>`
+`triage retrainpredict <model_group_id> <today>`
 
 Example:
-`triage predictlist 46 2019-05-06`
+`triage retrainpredict 30 2021-04-04`
 
-The predictlist will assume the current path to be the 'project path' to find models and write matrices, but this can be overridden by sending the `--project-path` option.
+The `retrainpredict` will assume the current path to be the 'project path' to train models and write matrices, but this can be overridden by sending the `--project-path` option
 
 ### Python
+The `Retrainer` class from `triage.predictlist` module can be used to retrain a model and predict forward.
 
-The `predict_forward_with_existed_model` function from the `triage.predictlist` module can be used similarly to the CLI, with the addition of the database engine and project storage as inputs.
-```
-from triage.predictlist import generate predict_forward_with_existed_model 
+```python
+from triage.predictlist import Retrainer
 from triage import create_engine
 
-predict_forward_with_existed_model(
+retrainer = Retrainer(
     db_engine=create_engine(<your-db-info>),
     project_path='/home/you/triage/project2'
-    model_id=46,
-    as_of_date='2019-05-06'
+    model_group_id=36,
 )
+retrainer.retrain(today='2021-04-04')
+retrainer.predict(today='2021-04-04')
+
 ```
 
 ## Output
-The Predictlist is stored similarly to the matrices created during an Experiment:
+The retrained model is sotred similariy to the matrices created during an Experiment:
 - Raw Matrix saved to the matrices directory in project storage
+- Raw Model saved to the trained_model directory in project storage
+- Retrained Model info saved in a table (triage_metadata.models) where model_comment = 'retrain_2021-04-04 21:19:09.975112'
 - Predictions saved in a table (triage_production.predictions)
-- Prediction metadata (tiebreaking, random seed) saved in a table (triage_production.prediction_metadata)
+- Prediction metadata (tiebreaking, random seed) saved in a table (triage_produciton.prediction_metadata)
 
-## Notes
-- The cohort and features for the Predictlist are all inferred from the Experiment that trained the given model_id (as defined by the experiment_models table).
-- The feature list ensures that imputation flag columns are present for any columns that either needed to be imputed in the training process, or that needed to be imputed in the predictlist dataset.
 
-# Retrain and Predict
-Use an existing model group to retrain a new model on all the data up to the current date and then predict forward into the future.
+# Predictlist
+If you would like to generate a list of predictions on already-trained Triage model with new data, you can use the 'Predictlist' module.
+
+# Predict Foward with Existed Model
+Use an existing model object to generate predictions on new data.
 
 ## Examples
 Both examples assume you have already run a Triage Experiment in the past, and know these two pieces of information:
-1. A `model_group_id` from a Triage model group that you want to use to retrain a model and generate prediction
-2. A `today` to generate your predictions on.
+1. A `model_id` from a Triage model that you want to use to generate predictions
+2. An `as_of_date` to generate your predictions on.
 
 ### CLI
-`triage retrainpredict <model_group_id> <today>`
+`triage predictlist <model_id> <as_of_date>`
 
 Example:
-`triage retrainpredict 30 2021-04-04`
+`triage predictlist 46 2019-05-06`
 
-The `retrainpredict` will assume the current path to be the 'project path' to train models and write matrices, but this can be overridden by sending the `--project-path` option
+The predictlist will assume the current path to be the 'project path' to find models and write matrices, but this can be overridden by sending the `--project-path` option.
 
 ### Python
-The `Retrainer` class from `triage.predictlist` module can be used to retrain a model and predict forward.
 
-```python
-from triage.predictlist import Retrainer
+The `predict_forward_with_existed_model` function from the `triage.predictlist` module can be used similarly to the CLI, with the addition of the database engine and project storage as inputs.
+```
+from triage.predictlist import generate predict_forward_with_existed_model 
 from triage import create_engine
 
-retrainer = Retrainer(
+predict_forward_with_existed_model(
     db_engine=create_engine(<your-db-info>),
     project_path='/home/you/triage/project2'
-    model_group_id=36,
+    model_id=46,
+    as_of_date='2019-05-06'
 )
-retrainer.retrain(today='2021-04-04')
-retrainer.predict(today='2021-04-04')
 ```
 
 ## Output
-The retrained model is sotred similariy to the matrices created during an Experiment:
+The Predictlist is stored similarly to the matrices created during an Experiment:
 - Raw Matrix saved to the matrices directory in project storage
-- Raw Model saved to the trained_model directory in project storage
-- Retrained Model info saved in a table (triage_metadata.models) where model_comment = 'retrain_2021-04-04'
 - Predictions saved in a table (triage_production.predictions)
-- Prediction metadata (tiebreaking, random seed) saved in a table (triage_produciton.prediction_metadata)
+- Prediction metadata (tiebreaking, random seed) saved in a table (triage_production.prediction_metadata)
+
+## Notes
+- The cohort and features for the Predictlist are all inferred from the Experiment that trained the given model_id (as defined by the experiment_models table).
+- The feature list ensures that imputation flag columns are present for any columns that either needed to be imputed in the training process, or that needed to be imputed in the predictlist dataset.
 
 

src/tests/test_predictlist.py

@@ -86,7 +86,7 @@ def test_retrain_should_write_model(finished_experiment):
         model_group_id=model_group_id,
     )
     retrain_info = retrainer.retrain(prediction_date)
-    model_comment = retrain_info['retrain_model_comment']
+    model_comment = retrain_info['retrained_model_comment']
 
     records = [
         row

src/triage/cli.py

@@ -411,7 +411,7 @@ def __init__(self, parser):
         )
 
         parser.add_argument(
-            "today",
+            "prediction_date",
             type=valid_date,
             help="The date as of which to run features. Format YYYY-MM-DD",
         )
@@ -428,8 +428,8 @@ def __call__(self, args):
             args.project_path,
             args.model_group_id,
         )
-        retrainer.retrain(args.today)
-        retrainer.predict(args.today)
+        retrainer.retrain(args.prediction_date)
+        retrainer.predict(args.prediction_date)
 
 
 @Triage.register

src/triage/component/catwalk/model_trainers.py

@@ -218,11 +218,14 @@ def _write_model_to_db(
             return model_id
         else:
             if retrain:
+                logger.debug("Retrain model...")
                 model = Model(
                     model_group_id=model_group_id,
                     model_hash=model_hash,
                     model_type=class_path,
                     hyperparameters=parameters,
+                    built_by_retrain=self.experiment_hash,
+                    built_in_triage_run=self.run_id,
                     model_size=model_size,
                     **misc_db_parameters,
                 )
@@ -234,7 +237,7 @@ def _write_model_to_db(
                     hyperparameters=parameters,
                     model_group_id=model_group_id,
                     built_by_experiment=self.experiment_hash,
-                    built_in_experiment_run=self.run_id,
+                    built_in_triage_run=self.run_id,
                     model_size=model_size,
                     **misc_db_parameters,
                 )    
@@ -283,13 +286,11 @@ def _train_and_store_model(
         unique_parameters = self.unique_parameters(parameters)
 
 
-        if model_hash is None and retrain and model_group_id:
-            model_hash = self._model_hash(
-                matrix_store.metadata,
-                class_path,
-                parameters,
-                random_seed,
-            )
+        if retrain:
+            # if retrain, use the provided model_group_id
+            if not model_group_id:
+                raise ValueError("model_group_id should be provided when retrain") 
+
         else:
             model_group_id = self.model_grouper.get_model_group_id(
                 class_path, unique_parameters, matrix_store.metadata, self.db_engine

src/triage/component/results_schema/__init__.py

@@ -13,12 +13,14 @@
 from .schema import (
     Base,
     Experiment,
+    Retrain,
     FeatureImportance,
     IndividualImportance,
     ListPrediction,
     ExperimentMatrix,
     Matrix,
     ExperimentModel,
+    RetrainModel,
     ExperimentRun,
     ExperimentRunStatus,
     Model,
@@ -39,11 +41,13 @@
 __all__ = (
     "Base",
     "Experiment",
+    "Retrain",
     "FeatureImportance",
     "IndividualImportance",
     "ListPrediction",
     "ExperimentMatrix",
     "Matrix",
+    "RetrainModel",
     "ExperimentModel",
     "ExperimentRun",
     "ExperimentRunStatus",

src/triage/component/results_schema/alembic/versions/5dd2ba8222b1_add_run_type.py

@@ -0,0 +1,48 @@
+"""add run_type
+
+Revision ID: 5dd2ba8222b1
+Revises: 079a74c15e8b
+Create Date: 2021-07-22 23:53:04.043651
+
+"""
+from alembic import op
+import sqlalchemy as sa
+from sqlalchemy.dialects import postgresql
+
+# revision identifiers, used by Alembic.
+revision = '5dd2ba8222b1'
+down_revision = '079a74c15e8b'
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    op.create_table('retrain',
+        sa.Column('retrain_hash', sa.Text(), nullable=False),
+        sa.Column('config', postgresql.JSONB(astext_type=sa.Text()), nullable=True),
+        sa.Column('prediction_date', sa.DateTime(), nullable=True),
+        sa.PrimaryKeyConstraint('retrain_hash'),
+        schema='triage_metadata',
+    )
+    op.add_column('experiment_runs', sa.Column('run_type', sa.Text(), nullable=True), schema='triage_metadata')
+    op.add_column('experiment_runs', sa.Column('retrain_hash', sa.Text(), nullable=True), schema='triage_metadata')
+    op.alter_column('models', 'built_in_experiment_run', nullable=False, new_column_name='built_in_triage_run', schema='triage_metadata')
+    op.add_column('models', sa.Column('built_by_retrain', sa.Text(), nullable=True), schema='triage_metadata')
+
+    op.create_table('retrain_models',
+        sa.Column('retrain_hash', sa.String(), nullable=False),
+        sa.Column('model_hash', sa.String(), nullable=False),
+        sa.ForeignKeyConstraint(['retrain_hash'], ['triage_metadata.retrain.retrain_hash'], ),
+        sa.PrimaryKeyConstraint('retrain_hash', 'model_hash'),
+        schema='triage_metadata'
+    )
+
+
+def downgrade():
+    op.drop_column('experiment_runs', 'run_type', schema='triage_metadata')
+    op.drop_column('experiment_runs', 'retrain_hash', schema='triage_metadata')
+    op.drop_table('retrain_models', schema='triage_metadata')
+    op.drop_table('retrain', schema='triage_metadata')
+    op.drop_column('models', 'built_by_retrain', schema='triage_metadata')
+    op.alter_column('models', 'built_in_triage_run', nullable=False, new_column_name='built_in_experiment_run', schema='triage_metadata')
+

src/triage/component/results_schema/schema.py

@@ -23,6 +23,7 @@
 from sqlalchemy.orm import relationship
 from sqlalchemy.types import ARRAY, Enum
 from sqlalchemy.sql import func
+from sqlalchemy.ext.hybrid import hybrid_property
 
 # One declarative_base object for each schema created
 Base = declarative_base()
@@ -71,13 +72,23 @@ class Experiment(Base):
     models_needed = Column(Integer)
 
 
+class Retrain(Base):
+    __tablename__ = "retrain"
+    __table_args__ = {"schema": "triage_metadata"}
+
+    retrain_hash = Column(String, primary_key=True)
+    config = Column(JSONB)
+    prediction_date = Column(DateTime)
+
+
 class ExperimentRunStatus(enum.Enum):
     started = 1
     completed = 2
     failed = 3
 
 
 class ExperimentRun(Base):
+# class TriageRun(Base):
 
     __tablename__ = "experiment_runs"
     __table_args__ = {"schema": "triage_metadata"}
@@ -88,10 +99,15 @@ class ExperimentRun(Base):
     git_hash = Column(String)
     triage_version = Column(String)
     python_version = Column(String)
+    run_type = Column(String)
     experiment_hash = Column(
         String,
         ForeignKey("triage_metadata.experiments.experiment_hash")
     )
+    retrain_hash = Column(
+        String,
+        ForeignKey("triage_metadata.retrain.retrain_hash")
+    )
     platform = Column(Text)
     os_user = Column(Text)
     working_directory = Column(Text)
@@ -113,6 +129,11 @@ class ExperimentRun(Base):
     stacktrace = Column(Text)
     random_seed = Column(Integer)
     experiment_rel = relationship("Experiment")
+    retrain_rel = relationship("Retrain")
+
+    @hybrid_property
+    def external_hash(self):
+        return self.experiment_hash or self.retrain_hash
 
 
 class Subset(Base):
@@ -225,8 +246,11 @@ class Model(Base):
     built_by_experiment = Column(
         String, ForeignKey("triage_metadata.experiments.experiment_hash")
     )
-    built_in_experiment_run = Column(
-        Integer, ForeignKey("triage_metadata.experiment_runs.id")
+    built_by_retrain = Column(
+        String, ForeignKey("triage_metadata.retrain.retrain_hash")
+    )
+    built_in_triage_run = Column(
+        Integer, ForeignKey("triage_metadata.experiment_runs.id"), nullable=True
     )
     train_end_time = Column(DateTime)
     test = Column(Boolean)
@@ -261,6 +285,20 @@ class ExperimentModel(Base):
     experiment_rel = relationship("Experiment")
 
 
+class RetrainModel(Base):
+    __tablename__ = "retrain_models"
+    __table_args__ = {"schema": "triage_metadata"}
+
+    retrain_hash = Column(
+        String,
+        ForeignKey("triage_metadata.retrain.retrain_hash"),
+        primary_key=True
+    )
+    model_hash = Column(String, primary_key=True)
+    model_rel = relationship("Model", primaryjoin=(Model.model_hash == model_hash), foreign_keys=model_hash)
+    retrain_rel = relationship("Retrain")
+
+
 class FeatureImportance(Base):
 
     __tablename__ = "feature_importances"