update #3
Merged
update #3
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Convert modulefiles to Lua. * Fix for orion build. * Fix for cheyenne, but issue with load_any remains. * Bug fix for jet data locations. * Hack for Cheyenne: Emulate load_any with try_load + isloaded combo * Fix NCO mode non-ensemble run dependency issue. * Wcoss2 bug fix thanks to @chan-hoo * Remove conda activation message from wcoss2 modulefile.
…ow.py (#414) The error messaging in the generate_FV3LAM_wflow.py has been overhauled to be more pythonic and more user-friendly. The old bash-like print_err_msg_exit() calls are replaced with proper exception handling, and print_info_msg() calls are replaced with the use of Python's built-in logging module. The latter has the added benefit of giving the same "tee"-like functionality--where messages are printed to screen and written to a log file simultaneously--without the need for messy subprocesses and the confusing error messages that come with them. Errors are now handled in a consistent way, with the errors clearly highlighted both on screen and in the log file, as well as an easy-to-follow exception traceback for those wanting that information. In addition, several error conditions that were previously not caught at the generate step now cause the script to fail with helpful error messages. From the user perspective, functionality will not change, aside from differences when error conditions are present. Successful runs will remain the same, except that some log messaging has been clarified, and the log file contains additional information for each message, as well as some indentation changes for ease of reading.
Restore correct `conda activate` message for Cheyenne
…430) * Remove old, deprecated "update_fork.pl" script * Move contents of test/ to tests/ to consolidate testing directories * Update tests/README.md for new path * Fixed one more new script location * Changes for CI tests from Daniel
* Move fixed-file mapping variables out of config_defaults. Also, move other variables that are unlikely/never changed by user. * Minor bug fix. * Use python script to check python version and packages. * Minor fixes. * Format python files with black. * Add pipefail in strict error checking mode. * Remove debugging comments from UFS_plot * More formatting changes. * Removed unused python utilities.
* created devclean.sh a script to clean all the SRW builds in subdirectories, bin/exec directory * updated devclean.sh Co-authored-by: Natalie Perlin <perlin.natalie@gmail.com>
…440) * set for_IC/LBCS dir for nco mode * add missing -rf to rm_vrfy in make_ic/lbcs * add directory empty check for restart * add intermediate symlink to RESTART for cycled runs * update rocoto ver to 1.3.5 on wcoss2 Co-authored-by: chan-hoo <chan-hoo.jeon@clogin05.cactus.wcoss2.ncep.noaa.gov> Co-authored-by: chan-hoo <chan-hoo.jeon@clogin04.cactus.wcoss2.ncep.noaa.gov> Co-authored-by: chan-hoo <chan-hoo.jeon@clogin01.cactus.wcoss2.ncep.noaa.gov>
* Enable SCHED_NATIVE_CMD on all systems. * Don't export variables for those that use SLURM. * Add some missing task specific modulefiles. * Try with an empty native line on cheyenne
* Bug fix on hera/jet run_vx.local modulefile. * Turn off formatting for some blocks of code.
* update stochastic physics link * add HPC-Stack Intersphinx links * remove hpc-stack submodule in favor of intersphinx links * move tables to 'tables' directory * remove duplicate Glossary terms; update links * update Quickstart * add M. Leukin to mgmt team list; change expt gen command * fix typo * fix typo * add stochastic physics link & troubleshooting tips * separate Build & Run chapters; update crosslinks * minor fix * update file exts in Config Params chapter * update user/platform sections of Config Params Chapter * update file name params * grid_gen, verbose, compiler, etc. params * ConfigWorkflow 1st draft revision * add met docs to intersphinx * fix minor errors * updated table 4.1 * add default values to ConfigWorkflow, other minor fixes * add RRFS and GSI to Glossary * update tables & file exts * add L1 data locations * address comments in ConfigWorkflow * fix table * minor fixes * minor updates * rm reg_wflow references, update tables, minor updates * switch order of steps so python env is loaded first * edit and rearrange pyenv and config BR sections * edit mac/linux/VX config sections * 1st draft of RunSRW * add parameter definitions that were accidentally deleted in .sh --> .yaml switch * remove comments * fix typo * fix link * add info from Mike L. * update build & workflow images * update workflow generation image * update image * update image * fix reviewer comments * update expt gen image * rm reg_wflow refs, switch config.sh to .yaml, update directory structure in intro * rm reg_wflow refs, switch config.sh to .yaml * change templates dir to parm * update GSI glossary entry * fixes based on PR review * remove fixed file mapping params * remove SFC_CLIMO_FIELDS from config_defaults.yaml/ConfigWorkflow.rst * update info on data: section of machine file * minor wording fix * minor wording fix * add note about converting ols .sh file to .yaml * update intro forum link, minor wording/details * update pngs, RRFS note, minor intro fixes * update wflow gen img Co-authored-by: gspetro <gillian.s.petro@gmail.com>
* Update build_cheyenne_gnu.lua remove loading of system python3 * Update build_cheyenne_intel.lua remove loading system python module * Update wflow_cheyenne.lua Load updated miniconda3 and ask to activate regional_workflow enviroment * Update wflow_hera.lua Load an updated miniconda3 and ask to activate regional_workflow environment * Update wflow_jet.lua Update miniconda3 module location and ask to activate regional_workflow * Update wflow_orion.lua Update miniconda3/4.12.0 module location and ask to activate regional_workflow environment * Update load_modules_run_task.sh Run an additional cycle of "conda deactivate" and "conda activate regional_workflow". It ensures that _python3_ binary path from the *regional_workflow* environment becomes prepended to the search $PATH, and is found first, before the _python3_ from miniconda3/4.12.0 from the *base* environment. * Update wflow_cheyenne.lua "conda activate regional_workfow" * Update and rename conda_regional_workflow.lua to miniconda_regional_workflow.lua use new miniconda3/4.12.0 with regional_workflow environment * Update make_grid.local.lua * Update get_extrn_ics.local.lua * Update get_extrn_lbcs.local.lua * Update make_ics.local.lua * Update make_lbcs.local.lua * Update and rename make_orog.hardcoded.lua to make_orog.local.lua * Update run_fcst.local.lua * Update run_vx.local.lua * Create make_sfc_climo.local.lua * Update miniconda_regional_workflow.lua * Update get_obs.local.lua all the requested packages for the python3 are found in regional_workflow environment * Update miniconda_regional_workflow.lua load updated miniconda3/4.12.0 with regional_workflow environment * Update miniconda_regional_workflow.lua Load an updated miniconda3/4.12.0 with the regional_workflow environment * Enable SCHED_NATIVE_CMD on all systems. * Update build_cheyenne_gnu.lua need to have miniconda3 loaded in build module * Update build_cheyenne_intel.lua need to have miniconda3 loaded in the build module * Update build_gaea_intel.lua need to have miniconda3 loaded in build module * Update build_hera_intel.lua need to have miniconda3 loaded in build module * Update build_jet_intel.lua need to have miniconda3 loaded in the build modulefile * Update build_orion_intel.lua need to have miniconda3 loaded in build modulefile * Update load_modules_wflow.sh conda activate command same across the platforms * Don't export variables for those that use SLURM. * Add some missing task specific modulefiles. * Update miniconda_regional_workflow.lua miniconda3 is now loaded in build_<system>_<compiler>, not in *.local files * Update miniconda_regional_workflow.lua miniconda3 is now loaded in build_<system>_<compiler>, not in *.local files * Update miniconda_regional_workflow.lua miniconda3 is now loaded in build_<system>_<compiler>, not in *.local files * Update miniconda_regional_workflow.lua miniconda3 is now loaded in build_<system>_<compiler>, not in *.local files * Update miniconda_regional_workflow.lua * fixes for noaacloud that work with Daniels pr * removed extra lines * removed commented lines * Removed set -x and some commented lines * put module list back in * removed ldd * removed miniconda from build*.lua files * returned conda load to local files * returned python to cheyenne * unload python module before setting up miniconda * added unload python to miniconda_regional_workflow.lua file * added local files for orion Co-authored-by: Natalie Perlin <68030316+natalie-perlin@users.noreply.github.com> Co-authored-by: Daniel Abdi <daniel.abdi@noaa.gov> Co-authored-by: Mark Potts <mpotts@redlineperf.com>
* Add a move step to binaries for NCO purposes. * Pass COMROOT to setpdy. * Correct path of post dependency on forecat. * Remove DATA_SHARED directory. * Remove symlinks and use unique id for run_fcst instead. * Remove forecast directory after post is done. * Use raw workflow_id * Add templated data paths to wcoss2 that use compath.py. * Remove cron relaunch from config.nco * Undo for_ICS/LBCS stuff in NCO mode. * Remove RESTART symlinking because DATAFCST is accessible to all other tasks. * Undo cyc addition to COMIN. * Update devclean. * Follow symlinks when tarring log files in Jenkins. * Add memory otpion for get_extrn_ics/lbcs. * Optimize fundamental test coverage. * Reduce work load on orion. * Add minimum memory requirement to all serial tasks. * Simplify set_predef_grid_params. * Simplify generate_FV3LAM_wflow. * Simplify link_fix. * Simplify to get rid of export_vars from setup. * Import only necessary variables from environment. * Use `eval echo` or `eval` depending on situation. * Add cyc back to COMIN. * Fix calculate_cost. * Fix gaea test list. * Remove a test case from cheyenne that is taking extremely long. * Reduce memory requirement for serial job to 1G. * Create COMINaws ics/lbcs staging directory. * Also symlink gfs ics/lbcs if on disk and in community mode. * Remove failing tests on orion. * Exclude gaea from --mem specification. * Add a run_vx modulefile for orion. * Add separate fundamental list for cheyenne GNU runs. * Bug fix for wcoss2 GFS version * Load prod_util on wcoss2. * Rename AWSROOT to EXTROOT. * Remove leftover COMINaws.
… new archive file name on HPSS (#457) * fix time offset with gfs * add new archive file name and we2e test * add ics_or_lbcs to test_retrieve.py * Add check availability of nemsio and netcdf * Copy file_templates dict before modifying it. * remove native from xml for dev_transfer on wcoss2 * correct data locations on wcoss2 * remove met_verification from fundamental.cheyenne.gnu * replace cyc to hh in data locations * make lbcs condition general Co-authored-by: Daniel Abdi <daniel.abdi@noaa.gov>
* Bug fix forecast directory deletion. * Bug fix in build only mode. * Remove unused sample.nc * Make setup_WE2E work irrespective of conda activation. * Fix the bug fix.
* [develop] Added return code to srw_build.sh script. * Correct issue in sorc/CMakeLists.txt so that the ufs-model will build again. * Restore PID functionality in tests/build.sh for GitHub Actions autoci builds. * Add return code to tests/build.sh and read the code in srw_build.sh.
* Update build_macos_gnu.lua architecture-independent method * Update build_macos_gnu.lua * Update wflow_macos.lua use conda environment and conda activate command * Update lmod-setup.sh updates for MacOS * Update lmod-setup.csh updates for MacOS * Update wflow_macos.lua updated for MacOS
* Update plot_allvars.py Updating a variable name "Geopotential Height" to "Geopotential height". Lower-case is needed for it to work with a newer Python. * Update plot_allvars_diff.py Updating a variable name "Geopotential Height" to "Geopotential height" (lower-case). This is needed to work with newer Python
…nd j-job s… (#465) * Source task specific portions of var_defns.sh file in ex- and j-job scripts. * Simplify source_config. * Put dictionary filtering in a function. * Add missing modulefiles now that python3 is required in all tasks. * Rearrange some config entries. * Move fix directories to platform section. * Move fix files for orography and surface climatology to platform. * Move data store types to platform. * Remove useless NOMADS script & logic and rely on retrieve_data instead. The NOMADS test case works for current date - 7. * Forward arguments from setup_WE2E to run_WE2E. * Don't set OPSROOT default * Move PREDEF_GRID_NAME to workflow section. * Bugfix for cheyenne-gnu issue + forward compiler in setup_WE2E. * Add templating for DATE_FIRST/LAST_CYCL in run_WE2E
…e-offset (#475) * change PDY to yyyymmdd for time-offset * change prod in nco we2e script * modify ex-script rather than machine file * fix typo * Set PDYext cycext Co-authored-by: chan-hoo <chan-hoo.jeon@clogin01.cactus.wcoss2.ncep.noaa.gov>
* new script to run and monitor e2e tests without cron * fixed find command and directed system output to /dev/null * Modify run_WE2E_tests.sh - When on Cheyenne, set use_cron_to_relaunch=false - When use_cron_to_relaunch=false, output a message at the end of the script describing how to run the new run_srw_tests.py script to manage the tests - Regardless of other variables, print a message at the end of the script showing the user where the test directory is * Updated .cicd/scripts/srw_test.sh to work with ush/run_srw_tests.py Co-authored-by: Michael Kavulich, Jr <kavulich@ucar.edu> Co-authored-by: michael.lueken <Michael.Lueken@noaa.gov>
* update build process img * update img * Revert changes This reverts commit b839658. * BuildSRW updates - esp mac/linux * remove Contributor's Guide * update Glossary and RunSRW * update build/run process images * remove outdated images; add new * Components & CCPP updates, misc * ConfigWflow updates * Container updates * update FAQ * graphics updates * I/O updates * Intro updates * Grid ch updates * Grid ch updates * Quickstart Updates * Rocoto edits * WE2E ch * update supported tests * update fix file list * minor update * minor edits * minor edits * misc minor fixes * change /home to /Users/gillianpetro * fix geopotential height * update hpc-stack doc link * nperlin changes * nperlin changes Co-authored-by: gspetro <gillian.s.petro@gmail.com>
* Update build_linux_gnu.lua Use the srw_common module with the standard list of modules * Update wflow_linux.lua * Update build_linux_gnu.lua
* Add level for data: section indicating ics_lbcs. * Add the NaturalEarth shape files. * Add plotting to the workflow. * Fix some deprecation warnings. * Don't plot 500mb vars if it is not available. * Add modulefiles for plot_allvars task. * Add a test case for plotting graphics. * Add diff plotting. * Remove ush/Python folder. * Turn off plotting by default. * Bug fix plot_diff call. * Add missing diff plot script. * Make plotting work in NCO mode. * Avoid positional arguments in the plotting scripts. Some bug fixes. * Make COMOUT_REF a template to compare multiple dates and cycles correctly. * Use logging to capture output from plot scripts in log files. * Turn off debug level, too much information. * Ignore warnings in plotting. * Bug fix diff plotter: mismatch in number of ticks, and ticklabels. * Increase wallclock time, bugfix orion FIXshp. * Add some default values to config.community/nco for convenience.
* add Tutorial chapter * update build process img * update img * Revert changes This reverts commit b839658. * misc. changes * add Indy VX sample case * update VX cases data/config * VXcases run instructions * Updates to Compare section * minor fixes * minor fixes * add clarifications; remove comments * remove accidental changes to plotting files * rm Tutorial ch (for separate PR) * rm accidental v2.1.0 refs in BuildSRW * clarify plotting instructions * minor corrections * minor fixes * add conda activate command * Eddie's edits * change wtime to 5h * Update prerequisites, data locations, headings Co-authored-by: gspetro <gillian.s.petro@gmail.com>
#503) * ensure arbitrary restart_interval * remove wtime change in we2e Co-authored-by: chan-hoo <chan-hoo.jeon@clogin04.cactus.wcoss2.ncep.noaa.gov> Co-authored-by: chan-hoo <chan-hoo.jeon@clogin01.cactus.wcoss2.ncep.noaa.gov>
* Updated modules to include EPIC bin path * updated vx maxtries * updated met and metplus paths Co-authored-by: Edward Snyder <Edward.Snyder@noaa.com>
#506) Co-authored-by: Edward Snyder <Edward.Snyder@noaa.com>
Co-authored-by: gspetro <gillian.s.petro@gmail.com>
* add gfs v16.3 to data list on hpss * fix typo Co-authored-by: chan-hoo jeon <chan-hoo.jeon@dlogin03.dogwood.wcoss2.ncep.noaa.gov>
Global variable use has been removed in setup.py, and reduced in generate_FV3LAM_wflow.py. The use of globals is a carry-over from the bash era of this utility, and does not meet modern coding standards.
* update ConfigWorkflow.rst w/plotting info * add plotting info to RunSRW * add Template Vars chapter * add plotting info to RunSRW; remove Graphics ch * add sphinx_rtd extension * add docutils requirement to render bullet points correctly * reformat plotting section * update requirements to render bullets * fix requirements syntax * update sphinx requirements * asterisk to dash bullets * revert changes to fix bullet points * minor edits to RunSRW plotting section * remove Graphics chapter * remove/update references to Graphics chapter * fix typos * change /Users/gillianpetro to Co-authored-by: gspetro <gillian.s.petro@gmail.com>
Use TEST_PREGEN_BASEDIR to set DOMAIN_PREGEN_BASEDIR if provided for a machine.
* add sphinx_rtd_theme to conf.py * pin sphinx_rtd_theme==0.5.1 in req file * pin docutils==0.17 * pin Pygments-2.13.0 * pinned sphinx_rtd_theme==1.1.1 * pin sphinx==5.3.0 * pin importlib-metadata==5.2.0 * pin pillow==9.3.0 * set docutils==0.16 * comment out unwanted pinnings * remove unnecessary changes for formatting fix * add back sphinx_rtd_theme extension in conf.py Co-authored-by: gspetro <gillian.s.petro@gmail.com>
… rocoto. (#508) * Increase precision of degs_per_radian to 15 digits. * Use generic date util. * Add fake slurm commands for rocoto usage on linux. * Modify machine files for linux and mac. * Modify linux and macos wflow modules. * Fix unittest. * Remove openmpi module loading in linux/mac build modulefile. * Fix sacct. * Fix crontab unspecified USER issue. * Add EXTRN_MDL_DATA_STORES to macos. * Add more states to squeue/sacct. * Add a taskthrottle=1 option for linux/mac. * Don't specifiy number of processes for mpirun. * Get exit code directly instead of from log file. * Set taskthrottle to 1000 by default. * Fix linux lmod path bug. * Set stack size to unlimited for linux/mac. * Fix unittest.
…1, jjob and ex- scripts (#536) * Add j-job and ex- scripts files for online-cmaq * reviewers comments on ufs-srw-app * remove unnecessary commands Co-authored-by: chan-hoo <chan-hoo.jeon@clogin01.cactus.wcoss2.ncep.noaa.gov> Co-authored-by: chan-hoo <chan-hoo.jeon@clogin04.cactus.wcoss2.ncep.noaa.gov>
* Fix plotting bug. * Add plotting test case to fundamental tests in both nco/community mode.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Update develop to head at ufs-community
Use this template to give a detailed message describing the change you want to make to the code.
You may delete any sections labeled "optional" and any instructions within .
If you are unclear on what should be written here, see https://github.com/wrf-model/WRF/wiki/Making-a-good-pull-request-message for some guidance and review the Code Contributor's Guide at https://github.com/ufs-community/ufs-srweather-app/wiki/Code-Manager's-Guide.
Code reviewers will assess the PR based on the criteria laid out in the Code Reviewer's Guide (https://github.com/ufs-community/ufs-srweather-app/wiki/Code-Manager's-Guide).
The title of this pull request should be a brief summary (ideally less than 100 characters) of the changes included in this PR. Please also include the branch to which this PR is being issued (e.g., "[develop]: Updated UFS_UTILS hash").
Use the "Preview" tab to see what your PR will look like when you hit "Create pull request"
--- Delete this line and those above before hitting "Create pull request" ---
DESCRIPTION OF CHANGES:
Type of change
TESTS CONDUCTED:
DEPENDENCIES:
DOCUMENTATION:
ISSUE:
CHECKLIST
LABELS (optional):
A Code Manager needs to add the following labels to this PR:
CONTRIBUTORS (optional):
update