[Example] Add preformer for precipitation nowcasting #976
+1,126
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your contribution! |
|
|
HydrogenSulfate
requested changes
Aug 20, 2024
|
|
||
| ``` sh | ||
| # 模型训练 | ||
| python examples/preformer/train.py |
There was a problem hiding this comment.
Suggested change
| python examples/preformer/train.py | |
| python train.py |
|
|
||
| ``` sh | ||
| # 模型评估 | ||
| python examples/preformer/train.py mode=eval |
There was a problem hiding this comment.
Suggested change
| python examples/preformer/train.py mode=eval | |
| python train.py mode=eval |
Comment on lines
+27
to
+31
| # set random seed for reproducibility | ||
| ppsci.utils.misc.set_random_seed(cfg.seed) | ||
| # initialize logger | ||
| logger.init_logger("ppsci", osp.join(cfg.output_dir, "train.log"), "info") | ||
|
|
Comment on lines
+73
to
+74
| "num_replicas": NUM_GPUS_PER_NODE, | ||
| "rank": dist.get_rank() % NUM_GPUS_PER_NODE, |
There was a problem hiding this comment.
这两个参数应该不需要，并且paddlescience也没有对应的处理逻辑，默认会根据环境中设置的卡数自动设置
Comment on lines
+137
to
+140
| mon = str("0") + mon | ||
| day = str(self.time_table[idxs].timetuple().tm_mday) | ||
| if len(day) == 1: | ||
| day = str("0") + day |
There was a problem hiding this comment.
str("0")是否可以直接写成"0"？，下同
Comment on lines
+144
to
+155
| r_data = np.load( | ||
| os.path.join(self.file_path, year, "r_" + year + mon + day + hour + ".npy") | ||
| ) | ||
| t_data = np.load( | ||
| os.path.join(self.file_path, year, "t_" + year + mon + day + hour + ".npy") | ||
| ) | ||
| u_data = np.load( | ||
| os.path.join(self.file_path, year, "u_" + year + mon + day + hour + ".npy") | ||
| ) | ||
| v_data = np.load( | ||
| os.path.join(self.file_path, year, "v_" + year + mon + day + hour + ".npy") | ||
| ) |
There was a problem hiding this comment.
可以直接使用f-string化简字符串拼接的写法
Comment on lines
+1
to
+21
| hydra: | ||
| run: | ||
| # dynamic output directory according to running time and override name | ||
| dir: outputs_preformer | ||
| job: | ||
| name: ${mode} # name of logfile | ||
| chdir: false # keep current working directory unchanged | ||
| config: | ||
| override_dirname: | ||
| exclude_keys: | ||
| - TRAIN.checkpoint_path | ||
| - TRAIN.trained_model_path | ||
| - EVAL.trained_model_path | ||
| - mode | ||
| - output_dir | ||
| - log_freq | ||
| sweep: | ||
| # output directory for multirun | ||
| dir: ${hydra.run.dir} | ||
| subdir: ./ | ||
|
|
There was a problem hiding this comment.
Suggested change
| hydra: | |
| run: | |
| # dynamic output directory according to running time and override name | |
| dir: outputs_preformer | |
| job: | |
| name: ${mode} # name of logfile | |
| chdir: false # keep current working directory unchanged | |
| config: | |
| override_dirname: | |
| exclude_keys: | |
| - TRAIN.checkpoint_path | |
| - TRAIN.trained_model_path | |
| - EVAL.trained_model_path | |
| - mode | |
| - output_dir | |
| - log_freq | |
| sweep: | |
| # output directory for multirun | |
| dir: ${hydra.run.dir} | |
| subdir: ./ | |
| defaults: | |
| - ppsci_default | |
| - TRAIN: train_default | |
| - TRAIN/ema: ema_default | |
| - TRAIN/swa: swa_default | |
| - EVAL: eval_default | |
| - INFER: infer_default | |
| - hydra/job/config/override_dirname/exclude_keys: exclude_keys_default | |
| - _self_ | |
| hydra: | |
| run: | |
| # dynamic output directory according to running time and override name | |
| dir: outputs_preformer | |
| job: | |
| name: ${mode} # name of logfile | |
| chdir: false # keep current working directory unchanged | |
| sweep: | |
| # output directory for multirun | |
| dir: ${hydra.run.dir} | |
| subdir: ./ | |
|
|
||
| # model settings | ||
| MODEL: | ||
| afno: |
| afno: | ||
| input_keys: ["input"] | ||
| output_keys: ["output"] | ||
| shape_in: [6, 12, IMG_H, IMG_W] |
There was a problem hiding this comment.
Suggested change
| shape_in: [6, 12, IMG_H, IMG_W] | |
| shape_in: | |
| - 6 | |
| - 12 | |
| - ${IMG_H} | |
| - ${IMG_W} | |
HydrogenSulfate
changed the title
|
@EricKing19 标题已经修改过了，原先的merge code of upstream不太合适 |
liaoxin2
reviewed
Aug 27, 2024
Comment on lines
+110
to
+112
| 案例中使用了预处理的 PEMSD4 和 PEMSD8 数据集。PEMSD4 为旧金山湾区交通数据，选取 29 条道路上 307 个传感器记录的交通数据，时间为 2018 年 1 月至 2 月。PEMSD8 为圣贝纳迪诺 8 条道路上 170 个检测器收集的交通数据，时间为 2016 年 7 月至 8 月。 | ||
|
|
||
| 两个数据集均被保存为 N x T x 1 的矩阵，记录了相应交通节点与时间的流量数据，其中 N 为交通节点数量，T 为时间序列长度。两个数据集分别按照 7:2:1 划分为训练集、验证集，和测试集。案例中预先计算了流量数据的均值与标准差，用于后续的正则化操作。 |
There was a problem hiding this comment.
该案例是关于降水的，这个数据集好像是交通的，数据集与代码不一致
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
Others
PR changes
Others
Describe
add Preformer model for precipitation nowcasting
add docs for Preformer
add examples for Preformer