WRF Intermediate File Format Writer

In order for the TDS to return data in the WRF Intermediate File Format, we will need code that will write to the WRF Intermediate File Format. The code to write the WRF Intermediate File Format should take a CoverageCollection object and return...well, up for discussion. See the last paragraph here for some thoughts. Probably a zip or tar file containing the WRF imtermediate Files?

At any rate, in order to make sure we are correctly writing data in the WRF Intermediate File Format, I've put together a collection of example WRF Intermediate File Format data, as well as the original GRIB files used by ungrib to create them. They are located on the THREDDS Team Drive in Google Drive, and are stored in a zip file (~21 GiB). The contents of the zip file are as follows:

full_grib_from_tds/
    GFS_Global_0p25deg_20180816_0600.grib2

input_grib/
    kwbc_20180816_0600_F0.grib2
    kwbc_20180816_0600_F12.grib2
    kwbc_20180816_0600_F15.grib2
    kwbc_20180816_0600_F18.grib2
    kwbc_20180816_0600_F21.grib2
    kwbc_20180816_0600_F3.grib2
    kwbc_20180816_0600_F6.grib2
    kwbc_20180816_0600_F9.grib2

intermediate_format/
    FILE:2018-08-16_06
    FILE:2018-08-16_09
    FILE:2018-08-16_12
    FILE:2018-08-16_15
    FILE:2018-08-16_18
    FILE:2018-08-16_21
    FILE:2018-08-17_00
    FILE:2018-08-17_03

Vtable.GFS
namelist.wps

full_grib_from_tds/GFS_Global_0p25deg_20180816_0600.grib2

This is the actual file we will need netCDF-Java to consume (using CoverageDatasetFactory.openCoverageDataset(...)) to produce the intermediate format files.

input_grib/*

These files were created by ECMWF's grib_api package using the command grib_filter. Basically, grib_filter read through all of the messages stored in full_grib_from_tds/GFS_Global_0p25deg_20180816_0600.grib2 and sorted them out into individual GRIB files, each of which contain all of the GRIB messages for a given valid forecast time. This is required by the ungrib program.

namelist.wps

This file is used by the WRF Preprocessing System, and is included in the zip file for basic informational purposes only (i.e. won't be used in by the writer). Basically, ungrib looks at the start_date, end_date, and interval_seconds parameters under the &share section, and uses those to determine which intermediate format files need to be written.

intermediate_format/*

These files were created by the ungrib program. Each file in this directory corresponds to a file in the input_grib/ directory, and contains data following the WRF intermediate format specifications.

input_grib file	intermediate_format file
kwbc_20180816_0600_F0.grib2	FILE:2018-08-16_06
kwbc_20180816_0600_F3.grib2	FILE:2018-08-16_09
kwbc_20180816_0600_F6.grib2	FILE:2018-08-16_12
kwbc_20180816_0600_F9.grib2	FILE:2018-08-16_15
kwbc_20180816_0600_F12.grib2	FILE:2018-08-16_18
kwbc_20180816_0600_F15.grib2	FILE:2018-08-16_21
kwbc_20180816_0600_F18.grib2	FILE:2018-08-17_00
kwbc_20180816_0600_F21.grib2	FILE:2018-08-17_03

Vtable.GFS

Not all messages in a GRIB file are used by ungrib to create an intermediate format file. The parameters needed are stored in vtable files, and this is the vtable that describes which GRIB messages from the GFS are needed to initialize WRF. The important entries in this table are going to be stored in the last four columns of the file:

|GRIB2|GRIB2|GRIB2|GRIB2|
|Discp|Catgy|Param|Level|

Using the info in a vtable to extract messages from a GRIB file using netCDF-java

When using netCDF-Java to read a GRIB file (CoverageDatasetFactory.openCoverageDataset(...)), these parameters are exposed in the CoverageColletion object as a coverage level metadata attribute named Grib_Variable_Id. For example:

var: u-component_of_wind_maximum_wind
    long_name: u-component of wind @ Isobaric surface
    units: m/s
    abbreviation: UGRD
    missing_value: NaN
    grid_mapping: LatLon_Projection
    coordinates: reftime time2 isobaric lat lon 
    Grib_Variable_Id: VAR_0-2-2_L100
    Grib2_Parameter: 0, 2, 2
    Grib2_Parameter_Discipline: Meteorological products
    Grib2_Parameter_Category: Momentum
    Grib2_Parameter_Name: u-component of wind
    Grib2_Level_Type: 100
    Grib2_Level_Desc: Isobaric surface
    Grib2_Generating_Process_Type: Forecast

The Grib_Variable_Id value can be constructed from the vtable entries using VAR_{Discp}-{Catgy}-{Param}_L{Level}. Netcdf-Java has a nice method for finding a variable in a CoverageCollection based on an attribute:

public Variable findCoverageByAttribute(String attName,
                                        String attValue)

The fun part is that there are multiple "flavors" of GFS output out there (different grid spacing, different variables output, etc.), and the vtable tries to account for that. What this means is that not every parameter in the vtable will be found in the example GRIB files in this zip file, and that's ok. Also, there may be a typo in the vtable - Physical Snow Depth does not have a GRIB2 Param associated with it. However, for the GRIB file used in this example dataset, that value is 11. So, the full Grib_Variable_Id for that field would be VAR_0-1-11_L1.

Notes

The code to write the WRF Intermediate File Format should take a list of CoverageCollection objects. The assumption is that by the time the list of CoverageCollection objects is passed into the converter code, the coverage have already been subset in the variable, spatial, and temporal domains, so the writer code needs to iterate over the list of coverage and write out what is there. The subsetting will be handled by existing code in the NetcdfSubset Service in combination with the new varset feature. Essentially, varset's will take the place of the vtable. However, when developing the writer, you'll need to do the all of subsetting manually. For this sample data, there was no spatial subsetting applied, so that leaves subsetting the variables, and subsetting in the time domain. Since each time step will have its own intermediate file, I would not worry about temporal subletting, as you will be able to compare the output files from the new writer to the intermediate files in zip file from the THREDDS team drive.

For development purposes, the goal would be to use the file input_grib/kwbc_20180816_0600_F0.grib2 and the information in the vtable to produce the equivalent of intermediate_format/FILE:2018-08-16_06. End the end, we want something like this:

String location = "<path>/input_grib/kwbc_20180816_0600_F0.grib2";
Optional<FeatureDatasetCoverage> featureDatasetCoverage = CoverageDatasetFactory.openCoverageDataset(location);
CoverageCollection cc = featureDatasetCoverage.get().getSingleCoverageCollection();
// make a list of coverages we want to be written out into the WRF intermediate file format
List<Coverage> coverageList = new ArrayList();
coverageList.add(cc.findCoverageByAttribute("Grib_Variable_Id", "VAR_0-2-2_L100"))
// add more coverages based on the vtable in the zip file
...
// magic in need of a wizard - this call should produce a zip (or tar?) file containing the equivalent of 
// the intermediate_format files contained in the zip file from the THREDDS Team Drive in Google Drive
// For this specific example, the zip (or tar) file should contain one intermediate file that is the same as 
// intermediate_format/FILE:2018-08-16_06
WrfIntermediateFileFormatWriter.write(outfile, coverageList);

At some point in the future (not part of this work), it would be cool to have a command line interface to the new WrfIntermediateFileFormatWriter that takes a location, a vtable or simple list of variable names, a namelist.wps (to get start/end times, domain size in lat/lon), and then writes out the intermediate files using that information.