Feature/report time to train #100
Conversation
|
I can pick it up and run |
| def get_ec2_instance_type(): | ||
| """Utility function to get ec2 instance name, or empty string if not possible to get name""" | ||
|
|
||
| instance_type_request = requests.get('http://169.254.169.254/latest/meta-data/instance-type') |
There was a problem hiding this comment.
is that a static ip? 169.254.169.254 I think this might be changing
There was a problem hiding this comment.
That's the aws recommended way of doing it: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
There was a problem hiding this comment.
I see, you are accessing that information from the instance itself not internet. Makes sense, this is a good find btw.
dvc.yaml
Outdated
| @@ -12,7 +12,7 @@ stages: | |||
| - data/processed/science_grants_tagged_title_synopsis.jsonl | |||
| train_tfidf_svm: | |||
| cmd: grants_tagger train data/processed/science_grants_tagged_title_synopsis.jsonl | |||
| models/label_binarizer-2020.05.2.pkl models/tfidf-svm-2020.05.2.pkl --approach tfidf-svm | |||
| models/label_binarizer-2020.05.2.pkl models/tfidf-svm-2020.05.2.pkl --approach tfidf-svm --metrics-path results/tfidf_svm_train.json | |||
There was a problem hiding this comment.
maybe I would name it train_info to be even more clear
|
This is ready to be merged. I need to run Update
|
|
|
Fixes #100.
Note that you'll have to do
dvc repro(I did it for scibert/tf-idf), but that will take ages, so not sure how to proceed.