Building ERDDAP Datasets

This notebook documents the process of creating XML fragments for nowcast system run results files for inclusion in /results/erddap-datasets/datasets.xml which is symlinked to /opt/tomcat/content/erddap/datasets.xml on the skookum ERDDAP server instance.

The contents are a combination of:

• instructions for using the GenerateDatasetsXml.sh and DasDds.sh tools found in the /opt/tomcat/webapps/erddap/WEB-INF/ directory

• instructions for forcing the server to update the datasets collection via the /results/erddap/flags/ directory

• code and metadata to transform the output of GenerateDatasetsXml.sh into XML fragments that are ready for inclusion in /results/erddap-datasets/datasets.xml

This is a snapshot of the erddap-datasets/ERDDAP_datasets.ipynb notebook that is used to maintain the datasets.xml file. Please see https://bitbucket.org/salishsea/erddap-datasets for the active, version controlled version of this notebook, and the production `datasets.xml file.

from collections import OrderedDict

from lxml import etree

NOTE

The next cell mounts the /results filesystem on skookum locally. It is intended for use if when this notebook is run on a laptop or other non-Waterhole machine that has sshfs installed and a mount point for /results available in its root filesystem.

Don’t execute the cell if that doesn’t describe your situation.

!sshfs skookum:/results /results

The metadata dictionary below contains information for dataset attribute tags whose values need to be changed, or that need to be added for all datasets.

The keys are the dataset attribute names.

The values are dicts containing a required text item and perhaps an optional after item.

The value associated with the text key is the text content for the attribute tag.

When present, the value associated with the after key is the name of the dataset attribute after which a new attribute tag containing the text value is to be inserted.

metadata = OrderedDict([
    ('coverage_content_type', {
        'text': 'modelResult',
        'after': 'cdm_data_type',
    }),
    ('infoUrl', {
        'text':
            'https://salishsea-meopar-docs.readthedocs.io/en/latest/results_server/index.html#salish-sea-model-results',
    }),
    ('institution', {'text': 'UBC EOAS'}),
    ('institution_fullname', {
        'text': 'Earth, Ocean & Atmospheric Sciences, University of British Columbia',
        'after': 'institution',
    }),
    ('license', {
        'text': '''The Salish Sea MEOPAR NEMO model results are copyright 2013-2021
by the Salish Sea MEOPAR Project Contributors and The University of British Columbia.

They are licensed under the Apache License, Version 2.0. http://www.apache.org/licenses/LICENSE-2.0''',
    }),
    ('project', {
        'text':'Salish Sea MEOPAR NEMO Model',
        'after': 'title',
    }),
    ('creator_name', {
        'text': 'Salish Sea MEOPAR Project Contributors',
        'after': 'project',
    }),
    ('creator_email', {
        'text': 'sallen@eos.ubc.ca',
        'after': 'creator_name',
    }),
    ('creator_url', {
        'text': 'https://salishsea-meopar-docs.readthedocs.io/',
        'after': 'creator_email',
    }),
    ('acknowledgement', {
        'text': 'MEOPAR, ONC, Compute Canada',
        'after': 'creator_url',
    }),
    ('drawLandMask', {
        'text': 'over',
        'after': 'acknowledgement',
    }),
])

The datasets dictionary below provides the content for the dataset title and summary attributes.

The title attribute content appears in the the datasets list table (among other places). It should be <80 characters long, and note that only the 1st 40 characters will appear in the table.

The summary attribute content appears (among other places) when a user hovers the cursor over the ? icon beside the title content in the datasets list table. The text that is inserted into the summary attribute tag by code later in this notebook is the title content followed by the summary content, separated by a blank line.

The keys of the datasets dict are the datasetID strings that are used in many places by the ERDDAP server. They are structured as follows:

• ubc to indicate that the dataset was produced at UBC

• SS to indicate that the dataset is a product of the Salish Sea NEMO model

• a few letters to indicate the model runs that produce the dataset:

  • n to indicate that the dataset is from a nowcast run,

  • f for forecast,

  • f2 for forecast2 (aka preliminary forecast),

  • hg for hindcast-green

  • ng for nowcast-green,

  • a for atmospheric forcing,

• a description of the dataset variables; e.g. PointAtkinsonSSH or 3DuVelocity

• the time interval of values in the dataset; e.g. 15m, 1h, 1d

• the dataset version; e.g. V16-10, or V1

Versioning was changed to a CalVer type scheme in Oct-2016. Thereafter versions are of the form Vyymm and indicate the year and month when the dataset entered production.

So:

• ubcSSnPointAtkinsonSSH15mV1 is the version 1 dataset of 15 minute averaged sea surface height values at Point Atkinson from PointAtkinson.nc output files

• ubcSSn3DwVelocity1hV2 is the version 2 dataset of 1 hr averaged vertical (w) velocity values over the entire domain from SalishSea_1h_*_grid_W.nc output files

• ubcSSnSurfaceTracers1dV1 is the version 1 dataset of daily averaged surface tracer values over the entire domain from SalishSea_1d_*_grid_T.nc output files

• ubcSSnBathymetry2V16-07 is the version 16-07 dataset of longitude, latitude, and bathymetry of the Salish Sea NEMO model grid that came into use in Jul-2016. The corresponding NEMO-generated mesh mask variables are in the ubcSSn2DMeshMaskDbo2V16-07 (y, x variables), and the ubcSSn3DMeshMaskDbo2V16-07 (z, y, x variables) datasets.

The dataset version part of the datasetID is used to indicate changes in the variables contained in the dataset. For example, the transition from the ubcSSn3DwVelocity1hV1 to the ubcSSn3DwVelocity1hV2 dataset occurred on 24-Jan-2016 when we started to output vertical eddy viscosity and diffusivity values at the w grid points.

All dataset ids end with their version identifier and their summary ends with a notation about the variables that they contain; e.g.

v1: wVelocity variable

When the a dataset version is incremented a line describing the change is added to the end of its summary; e.g.

v1: wVelocity variable
v2: Added eddy viscosity & diffusivity variables ve_eddy_visc & ve_eddy_diff

datasets = {
    'ubcSSnBathymetry2V1' :{
        'type': 'geolocation bathymetry',
        'title': 'Salish Sea NEMO Model Grid, Geo-location and Bathymetry, v1',
        'summary':'''Longitude, latitude, and bathymetry of the Salish Sea NEMO model grid.
The bathymetry values are those calculated by NEMO from the input bathymetry file.
NEMO modifies the input bathymetry to remove isolated holes, and too-small partial steps.
The model grid includes the Juan de Fuca Strait, the Strait of Georgia, Puget Sound,
and Johnstone Strait on the coasts of Washington State and British Columbia.

v1: longitude, latitude and bathymetry variables
''',
        'fileNameRegex': '.*SalishSea2_NEMO_bathy\.nc$'

    },
    'ubcSSnBathymetry2V16-07' :{
        'type': 'geolocation bathymetry',
        'title': 'Salish Sea NEMO Model Grid, Geo-location and Bathymetry, v16-07',
        'summary':'''Longitude, latitude, and bathymetry of the Salish Sea NEMO model grid.
The bathymetry values are those calculated by NEMO from the input bathymetry file.
NEMO modifies the input bathymetry to remove isolated holes, and too-small partial steps.
The model grid includes the Juan de Fuca Strait, the Strait of Georgia, Puget Sound,
and Johnstone Strait on the coasts of Washington State and British Columbia.

v1: longitude, latitude and bathymetry variables
v16-07: same variables,
        bathymetry uniformly deepened by 1 grid level,
        smoothed at Juan de Fuca & Johnstone Strait open boundaries,
        Fraser River lengthened,
        bathymetry deepened near mouth of Fraser River
''',
        'fileNameRegex': '.*downbyone2_NEMO_bathy\.nc$'

    },

    'ubcSSn2DMeshMask2V1': {
        'type': 'geolocation bathymetry',
        'title': 'Salish Sea NEMO Model Grid, 2D Mesh Mask, v1',
        'summary':'''NEMO grid variable value for the u-v plane of the
Salish Sea NEMO model Arakawa-C grid.
The values are those calculated by NEMO from the input coordinates and bathymetry files.
The variable names are those used by NEMO-3.4,
see the NEMO-3.4 book (http://www.nemo-ocean.eu/Media/Files/NEMO_book_V3_4.pdf) for details,
or the long_name attributes of the variables for succinct descriptions of the variables.
The model grid includes the Juan de Fuca Strait, the Strait of Georgia, Puget Sound,
and Johnstone Strait on the coasts of Washington State and British Columbia.

v1: e1t, e2t, e1u, e2u, e1v, e2v, e1f, e2f, glamt, gphit, glamu, gphiu, glamv, gphiv,
    tmaskutil, umaskutil, vmaskutil, fmaskutil, ff, mbathy variables
''',
        'fileNameRegex': '.*mesh_mask_SalishSea2\.nc$',
    },
    'ubcSSn2DMeshMask2V16-07': {
        'type': 'geolocation bathymetry',
        'title': 'Salish Sea NEMO Model Grid, 2D Mesh Mask, v16-07',
        'summary':'''NEMO grid variable value for the u-v plane of the
Salish Sea NEMO model Arakawa-C grid.
The values are those calculated by NEMO from the input coordinates and bathymetry files.
The variable names are those used by NEMO-3.6,
see the NEMO-3.6 book (http://www.nemo-ocean.eu/Media/Files/NEMO_book_V3_6.pdf) for details,
or the long_name attributes of the variables for succinct descriptions of the variables.
The model grid includes the Juan de Fuca Strait, the Strait of Georgia, Puget Sound,
and Johnstone Strait on the coasts of Washington State and British Columbia.

v1: e1t, e2t, e1u, e2u, e1v, e2v, e1f, e2f, glamt, gphit, glamu, gphiu, glamv, gphiv,
    tmaskutil, umaskutil, vmaskutil, fmaskutil, ff, mbathy variables
v16-07: e1t, e2t, e1u, e2u, e1v, e2v, e1f, e2f, glamt, gphit, glamu, gphiu, glamv, gphiv,
        glamf, gphif, tmaskutil, umaskutil, vmaskutil, fmaskutil, ff, mbathy variables
''',
        'fileNameRegex': '.*mesh_mask_downbyone2\.nc$',
    },

    'ubcSSn3DMeshMask2V1': {
        'type': 'geolocation bathymetry',
        'title': 'Salish Sea NEMO Model Grid, 3D Mesh Mask, v1',
        'summary':'''NEMO grid variable value for the Salish Sea NEMO model Arakawa-C grid.
The values are those calculated by NEMO from the input coordinates and bathymetry files.
The variable names are those used by NEMO-3.4,
see the NEMO-3.4 book (http://www.nemo-ocean.eu/Media/Files/NEMO_book_V3_4.pdf) for details,
or the long_name attributes of the variables for succinct descriptions of the variables.
The model grid includes the Juan de Fuca Strait, the Strait of Georgia, Puget Sound,
and Johnstone Strait on the coasts of Washington State and British Columbia.

v1: e3t, e3u, e3v, e3w, gdept, gdepu, gdepv, gdepw, tmask, umask, vmask, fmask variables
''',
        'fileNameRegex': '.*mesh_mask_SalishSea2\.nc$'
    },
    'ubcSSn3DMeshMask2V16-07': {
        'type': 'geolocation bathymetry',
        'title': 'Salish Sea NEMO Model Grid, 3D Mesh Mask, v16-07',
        'summary':'''NEMO grid variable value for the Salish Sea NEMO model Arakawa-C grid.
The values are those calculated by NEMO from the input coordinates and bathymetry files.
The variable names are those used by NEMO-3.6,
see the NEMO-3.6 book (http://www.nemo-ocean.eu/Media/Files/NEMO_book_V3_6.pdf) for details,
or the long_name attributes of the variables for succinct descriptions of the variables.
The model grid includes the Juan de Fuca Strait, the Strait of Georgia, Puget Sound,
and Johnstone Strait on the coasts of Washington State and British Columbia.

v1: e3t_0, e3u_0, e3v_0, e3w_0, gdept_0, gdepu, gdepv, gdepw_0, tmask, umask, vmask, fmask variables
v16-07: e3t, e3u, e3v, e3w, gdept, gdepu, gdepv, gdepw, tmask, umask, vmask, fmask variables
''',
        'fileNameRegex': '.*mesh_mask_downbyone2\.nc$'
    },

    'ubcSSnPointAtkinsonSSH15mV1': {
        'type': 'tide gauge',
        'title': 'Nowcast, Point Atkinson, Sea Surface Height, 15min, v1',
        'summary': '''Sea surface height values averaged over 15 minute intervals from
Salish Sea NEMO model nowcast runs. The values are calculated at the model grid point
closest to the Point Atkinson tide gauge station on the north side of English Bay,
near Vancouver, British Columbia.
Geo-location and depth data for the Salish Sea NEMO model grid are available in the ubcSSnBathymetry2V1 dataset.

v1: ssh variable
''',
        'fileNameRegex': '.*PointAtkinson\.nc$',
    },

    'ubcSSnCampbellRiverSSH15mV1': {
        'type': 'tide gauge',
        'title': 'Nowcast, Campbell River, Sea Surface Height, 15min, v1',
        'summary': '''Sea surface height values averaged over 15 minutes intervals from
Salish Sea NEMO model nowcast runs. The values are calculated at the model grid point
closest to the Campbell River tide gauge station at the north end of the Strait of Georgia,
near Campbell River, British Columbia.
Geo-location and depth data for the Salish Sea NEMO model grid are available in the ubcSSnBathymetry2V1 dataset.

v1: ssh variable
''',
        'fileNameRegex': '.*CampbellRiver\.nc$',
    },

    'ubcSSnCherryPointSSH15mV1': {
        'type': 'tide gauge',
        'title': 'Nowcast, Cherry Point, Sea Surface Height, 15min, v1',
        'summary': '''Sea surface height values averaged over 15 minutes intervals from
Salish Sea NEMO model nowcast runs. The values are calculated at the model grid point
closest to the Cherry Point tide gauge station in the southern Strait of Georgia,
near Birch Bay, Washington.
Geo-location and depth data for the Salish Sea NEMO model grid are available in the ubcSSnBathymetry2V1 dataset.

v1: ssh variable
''',
        'fileNameRegex': '.*CherryPoint\.nc$',
    },

    'ubcSSnFridayHarborSSH15mV1': {
        'type': 'tide gauge',
        'title': 'Nowcast, Friday Harbor, Sea Surface Height, 15min, v1',
        'summary': '''Sea surface height values averaged over 15 minutes intervals from
Salish Sea NEMO model nowcast runs. The values are calculated at the model grid point
closest to the Friday Harbor tide gauge station at San Juan Island in Haro Strait,
near Friday Harbor, Washington.
Geo-location and depth data for the Salish Sea NEMO model grid are available in the ubcSSnBathymetry2V1 dataset.

v1: ssh variable
''',
        'fileNameRegex': '.*FridayHarbor\.nc$',
    },

    'ubcSSnNanaimoSSH15mV1': {
        'type': 'tide gauge',
        'title': 'Nowcast, Nanaimo, Sea Surface Height, 15min, v1',
        'summary': '''Sea surface height values averaged over 15 minutes intervals from
Salish Sea NEMO model nowcast runs. The values are calculated at the model grid point
closest to the Nanaimo tide gauge station on the west side of the central Strait of Georgia,
near Nanaimo, British Columbia.
Geo-location and depth data for the Salish Sea NEMO model grid are available in the ubcSSnBathymetry2V1 dataset.

v1: ssh variable
''',
        'fileNameRegex': '.*Nanaimo\.nc$',
    },

    'ubcSSnNeahBaySSH15mV1': {
        'type': 'tide gauge',
        'title': 'Nowcast, Neah Bay, Sea Surface Height, 15min, v1',
        'summary': '''Sea surface height values averaged over 15 minutes intervals from
Salish Sea NEMO model nowcast runs. The values are calculated at the model grid point
closest to the Neah Bay tide gauge station on the south side of the west end of the Juan de Fuca Strait,
near Neah Bay, Washington.
Geo-location and depth data for the Salish Sea NEMO model grid are available in the ubcSSnBathymetry2V1 dataset.

v1: ssh variable
''',
        'fileNameRegex': '.*NeahBay\.nc$',
    },

    'ubcSSnVictoriaSSH15mV1': {
        'type': 'tide gauge',
        'title': 'Nowcast, Victoria, Sea Surface Height, 15min, v1',
        'summary': '''Sea surface height values averaged over 15 minutes intervals from
Salish Sea NEMO model nowcast runs. The values are calculated at the model grid point
closest to the Victoria tide gauge station on the north side of the east end of the Juan de Fuca Strait,
in the Victoria Inner Harbour, near Victoria, British Columbia.
Geo-location and depth data for the Salish Sea NEMO model grid are available in the ubcSSnBathymetry2V1 dataset.

v1: ssh variable
''',
        'fileNameRegex': '.*Victoria\.nc$',
    },

    'ubcSSnSandHeadsSSH15mV1': {
        'type': 'tide gauge',
        'title': 'Nowcast, Sand Heads, Sea Surface Height, 15min, v1',
        'summary': '''Sea surface height values averaged over 15 minutes intervals from
Salish Sea NEMO model nowcast runs. The values are calculated at the model grid point
closest to the Sand Heads light station on the east side of the central Strait of Georgia,
near Steveston, British Columbia.
Geo-location and depth data for the Salish Sea NEMO model grid are available in the ubcSSnBathymetry2V1 dataset.

v1: ssh variable
''',
        'fileNameRegex': '.*Sandheads\.nc$',
    },

    'ubcSSn3DTracerFields1hV1': {
        'type': '3d fields',
        'title': 'Nowcast, Salish Sea, 3d Tracer Fields, Hourly, v1',
        'summary': '''3d salinity and water temperature field values averaged over 1 hour intervals
from Salish Sea NEMO model nowcast runs. The values are calculated for the entire model grid
that includes the Juan de Fuca Strait, the Strait of Georgia, Puget Sound,
and Johnstone Strait on the coasts of Washington State and British Columbia.
Geo-location and depth data for the Salish Sea NEMO model grid are available in the ubcSSnBathymetry2V1 dataset.

v1: salinity (practical) and temperature variables
''',
        'fileNameRegex': '.*SalishSea_1h_\d{8}_\d{8}_grid_T\.nc$',
    },

    'ubcSSnSurfaceTracerFields1hV1': {
        'type': 'surface fields',
        'title': 'Nowcast, Salish Sea, Surface Tracer Fields, Hourly, v1',
        'summary': '''2d sea surface height and rainfall rate field values averaged over 1 hour intervals
from Salish Sea NEMO model nowcast runs. The values are calculated for the surface of the model grid
that includes the Juan de Fuca Strait, the Strait of Georgia, Puget Sound,
and Johnstone Strait on the coasts of Washington State and British Columbia.
Geo-location and depth data for the Salish Sea NEMO model grid are available in the ubcSSnBathymetry2V1 dataset.

v1: sea surface height and rainfall rate variables
''',
        'fileNameRegex': '.*SalishSea_1h_\d{8}_\d{8}_grid_T\.nc$',
    },

    'ubcSSn3DuVelocity1hV1': {
        'type': '3d fields',
        'title': 'Nowcast, Salish Sea, 3d u Velocity Field, Hourly, v1',
        'summary': '''3d zonal (u) component velocity field values averaged over 1 hour intervals
from Salish Sea NEMO model nowcast runs. The values are calculated for the entire model grid
that includes the Juan de Fuca Strait, the Strait of Georgia, Puget Sound,
and Johnstone Strait on the coasts of Washington State and British Columbia.
Geo-location and depth data for the Salish Sea NEMO model grid are available in the ubcSSnBathymetry2V1 dataset.

v1: uVelocity variable
''',
        'fileNameRegex': '.*SalishSea_1h_\d{8}_\d{8}_grid_U\.nc$',
    },

    'ubcSSn3DvVelocity1hV1': {
        'type': '3d fields',
        'title': 'Nowcast, Salish Sea, 3d v Velocity Field, Hourly, v1',
        'summary': '''3d meridional (v) component velocity field values averaged over 1 hour intervals
from Salish Sea NEMO model nowcast runs. The values are calculated for the entire model grid
that includes the Juan de Fuca Strait, the Strait of Georgia, Puget Sound,
and Johnstone Strait on the coasts of Washington State and British Columbia.
Geo-location and depth data for the Salish Sea NEMO model grid are available in the ubcSSnBathymetry2V1 dataset.

v1: vVelocity variable
''',
        'fileNameRegex': '.*SalishSea_1h_\d{8}_\d{8}_grid_V\.nc$',
    },

    'ubcSSn3DwVelocity1hV1': {
        'type': '3d fields',
        'title': 'Nowcast, Salish Sea, 3d w Velocity Field, Hourly, v1',
        'summary': '''3d vertical (w) component velocity field values averaged over 1 hour intervals
from Salish Sea NEMO model nowcast runs. The values are calculated for the entire model grid
that includes the Juan de Fuca Strait, the Strait of Georgia, Puget Sound,
and Johnstone Strait on the coasts of Washington State and British Columbia.
Geo-location and depth data for the Salish Sea NEMO model grid are available in the ubcSSnBathymetry2V1 dataset.

v1: wVelocity variable
''',
        'fileNameRegex': '.*SalishSea_1h_\d{8}_\d{8}_grid_W\.nc$',
    },

    'ubcSSaSurfaceAtmosphereFieldsV1': {
        'type': 'surface fields',
        'title': 'HRDPS, Salish Sea, Atmospheric Forcing Fields, Hourly, v1',
        'summary': '''2d hourly atmospheric field values from the
Environment Canada HRDPS atmospheric forcing model that are used to force the Salish Sea NEMO model.
The model grid includes the Juan de Fuca Strait, the Strait of Georgia, Puget Sound,
and Johnstone Strait on the coasts of Washington State and British Columbia.
Geo-location data for the atmospheric forcing grid are available in the ubcSSaAtmosphereGridV1 dataset.
Atmospheric field values are interpolated on to the Salish Sea NEMO model grid (ubcSSnBathymetry2V1 dataset)
on-the-fly by NEMO.

v1: atmospheric pressure, precipitation rate, 2m specific humidity, 2m air temperature,
short-wave radiation flux, long-wave radiation flux, 10m u wind component, 10m v wind component variables
''',
        'fileNameRegex': '.*ops_y\d{4}m\d{2}d\d{2}\.nc$',
    },
}

datasets['ubcSSn3DwVelocity1hV2'] = datasets['ubcSSn3DwVelocity1hV1']
datasets['ubcSSn3DwVelocity1hV2'].update({
    'title': datasets['ubcSSn3DwVelocity1hV1']['title'].replace(', v1', ', v2'),
    'summary': datasets['ubcSSn3DwVelocity1hV1']['summary'] + '''
v2: Added eddy viscosity & diffusivity variables ve_eddy_visc & ve_eddy_diff''',
})
datasets['ubcSSn3DTracerFields1hV16-10'] = datasets['ubcSSn3DTracerFields1hV1']
datasets['ubcSSn3DTracerFields1hV16-10'].update({
    'title': datasets['ubcSSn3DTracerFields1hV1']['title'].replace(', v1', ', v16-10'),
    'summary': datasets['ubcSSn3DTracerFields1hV1']['summary'] + '''
v16-10: NEMO-3.6; ubcSSnBathymetry2V16-07 bathymetry; see infoUrl link for full details.
        Changed salinity variable to reference salinity.
        Made temperature variable explicitly potential temperature.
        Added squared buoyancy frequency variable.''',
})
datasets['ubcSSnSurfaceTracerFields1hV16-10'] = datasets['ubcSSnSurfaceTracerFields1hV1']
datasets['ubcSSnSurfaceTracerFields1hV16-10'].update({
    'title': datasets['ubcSSnSurfaceTracerFields1hV1']['title'].replace(', v1', ', v16-10'),
    'summary': datasets['ubcSSnSurfaceTracerFields1hV1']['summary'] + '''
v16-10: NEMO-3.6; ubcSSnBathymetry2V16-07 bathymetry; see infoUrl link for full details.
        Added mixed layer thickness defined by sigma theta variable.
        Deleted rainfall rate variable.''',
})
datasets['ubcSSn3DuVelocity1hV16-10'] = datasets['ubcSSn3DuVelocity1hV1']
datasets['ubcSSn3DuVelocity1hV16-10'].update({
    'title': datasets['ubcSSn3DuVelocity1hV1']['title'].replace(', v1', ', v16-10'),
    'summary': datasets['ubcSSn3DuVelocity1hV1']['summary'] + '''
v16-10: NEMO-3.6; ubcSSnBathymetry2V16-07 bathymetry; see infoUrl link for full details.'''
})
datasets['ubcSSn3DvVelocity1hV16-10'] = datasets['ubcSSn3DvVelocity1hV1']
datasets['ubcSSn3DvVelocity1hV16-10'].update({
    'title': datasets['ubcSSn3DvVelocity1hV1']['title'].replace(', v1', ', v16-10'),
    'summary': datasets['ubcSSn3DvVelocity1hV1']['summary'] + '''
v16-10: NEMO-3.6; ubcSSnBathymetry2V16-07 bathymetry; see infoUrl link for full details.'''
})

The dataset_vars dictionary below is used to rename variables from the often cryptic NEMO names to the names that appear in the ERDDAP generated files and web content.

The keys are the NEMO variable names to replace.

The values are dicts that map the variable names to use in ERDDAP to the destinationName attribute name.

dataset_vars = {
    'sossheig': {'destinationName': 'ssh'},
    'vosaline': {'destinationName': 'salinity'},
    'votemper': {'destinationName': 'temperature'},
    'vozocrtx': {'destinationName': 'uVelocity'},
    'vomecrty': {'destinationName': 'vVelocity'},
    'vovecrtz': {'destinationName': 'wVelocity'},
}

A few convenient functions to reduce code repetition:

def print_tree(root):
    """Display an XML tree fragment with indentation.
    """
    print(etree.tostring(root, pretty_print=True).decode('ascii'))

def find_att(root, att):
    """Return the dataset attribute element named att
    or raise a ValueError exception if it cannot be found.
    """
    e = root.find('.//att[@name="{}"]'.format(att))
    if e is None:
        raise ValueError('{} attribute element not found'.format(att))
    return e

def replace_yx_with_lonlat(root):
    new_axes = {
        'y': {'sourceName': 'nav_lon', 'destinationName': 'longitude'},
        'x': {'sourceName': 'nav_lat', 'destinationName': 'latitude'},
    }
    for axis in root.findall('.//axisVariable'):
        if axis.find('.//sourceName').text in new_axes:
            key = axis.find('.//sourceName').text
            new_axis = etree.Element('axisVariable')
            etree.SubElement(new_axis, 'sourceName').text = new_axes[key]['sourceName']
            etree.SubElement(new_axis, 'destinationName').text = new_axes[key]['destinationName']
            axis.getparent().replace(axis, new_axis)

Now we’re ready to produce a dataset!!!

Use the /opt/tomcat/webapps/erddap/WEB-INF/GenerateDatasetsXml.sh script generate the initial version of an XML fragment for a dataset:

$ cd /opt/tomcat/webapps/erddap/WEB-INF/
$ bash GenerateDatasetsXml.sh EDDGridFromNcFiles /results/SalishSea/nowcast/

The EDDGridFromNcFiles and /results/SalishSea/nowcast/ arguments tell the script which EDDType and what parent directory to use, avoiding having to type those in answer to prompts. Answer the remaining prompts, for example:

File name regex (e.g., ".*\.nc") (default="")
? .*SalishSea_1h_\d{8}_\d{8}_grid_W\.nc$

Full file name of one file (default="")
? /results/SalishSea/nowcast/28jan16/SalishSea_1h_20160128_20160128_grid_W.nc

ReloadEveryNMinutes (e.g., 10080) (default="")
? 10080

Other examples of file name regex are:

• .*PointAtkinson.nc$

• .*SalishSea_1d_\d{8}_\d{8}_grid_W\.nc$

The output is written to /results/erddap/logs/GenerateDatasetsXml.out

Now, we:

• set the datasetID we want to use

• parse the output of GenerateDatasetsXml.sh into an XML tree data structure

• set the datasetID dataset attribute value

• re-set the fileNameRegex dataset attribute value because it looses its \ characters during parsing(?)

• edit and add dataset attributes from the metadata dict

• set the title and summary dataset attributes from the datasets dict

• set the names of the grid x and y axis variables

• rename data variables as specified in the dataset_vars dict

def update_xml(root, datasetID, metadata, datasets, dataset_vars):
    root.attrib['datasetID'] = datasetID
    root.find('.//fileNameRegex').text = datasets[datasetID]['fileNameRegex']

    title = datasets[datasetID]['title']
    summary = find_att(root, 'summary')
    summary.text = '{0}\n\n{1}'.format(title, datasets[datasetID]['summary'])
    e = etree.Element('att', name='title')
    e.text = title
    summary.addnext(e)

    for att, info in metadata.items():
        e = etree.Element('att', name=att)
        e.text = info['text']
        try:
            root.find('.//att[@name="{}"]'.format(info['after'])).addnext(e)
        except KeyError:
            find_att(root, att).text = info['text']

    for axis_name in root.findall('.//axisVariable/destinationName'):
        if axis_name.text in ('x', 'y'):
            axis_name.text = 'grid{}'.format(axis_name.text.upper())
    if datasets[datasetID]['type'] == 'tide gauge':
        replace_yx_with_lonlat(root)

    for var_name in root.findall('.//dataVariable/destinationName'):
        if var_name.text in dataset_vars:
            var_name.text = dataset_vars[var_name.text]['destinationName']

parser = etree.XMLParser(remove_blank_text=True)
tree = etree.parse('/results/erddap/logs/GenerateDatasetsXml.out', parser)
root = tree.getroot()

datasetID = 'ubcSSn3DvVelocity1hV16-10'

update_xml(root, datasetID, metadata, datasets, dataset_vars)

Inspect the resulting dataset XML fragment below and edit the dicts and code cell above until it is what is required for the dataset:

print_tree(root)

<dataset type="EDDGridFromNcFiles" datasetID="ubcSSn3DvVelocity1hV16-10" active="true">
  <reloadEveryNMinutes>10080</reloadEveryNMinutes>
  <updateEveryNMillis>10000</updateEveryNMillis>
  <fileDir>/results/SalishSea/nowcast-blue/</fileDir>
  <recursive>true</recursive>
  <fileNameRegex>.*SalishSea_1h_\d{8}_\d{8}_grid_V\.nc$</fileNameRegex>
  <metadataFrom>last</metadataFrom>
  <matchAxisNDigits>20</matchAxisNDigits>
  <fileTableInMemory>false</fileTableInMemory>
  <accessibleViaFiles>false</accessibleViaFiles>
  <!-- sourceAttributes>
        <att name="Conventions">CF-1.5</att>
        <att name="description">ocean V grid variables</att>
        <att name="history">Mon Dec  5 11:34:46 2016: ncks -4 -L4 -O SalishSea_1h_20161205_20161205_grid_V.nc SalishSea_1h_20161205_20161205_grid_V.nc</att>
        <att name="name">SalishSea_1h_20161205_20161205</att>
        <att name="NCO">4.4.2</att>
        <att name="production">An IPSL model</att>
        <att name="timeStamp">2016-Dec-05 11:03:34 PST</att>
        <att name="title">ocean V grid variables</att>
    </sourceAttributes -->
  <addAttributes>
    <att name="cdm_data_type">Grid</att>
    <att name="coverage_content_type">modelResult</att>
    <att name="Conventions">CF-1.6, COARDS, ACDD-1.3</att>
    <att name="infoUrl">https://salishsea-meopar-docs.readthedocs.io/en/latest/results_server/index.html#salish-sea-model-results</att>
    <att name="institution">UBC EOAS</att>
    <att name="institution_fullname">Earth, Ocean &amp; Atmospheric Sciences, University of British Columbia</att>
    <att name="keywords">along, axis, circulation, current, currents, data, depthv, grid, j-axis, ocean, oceans,
Oceans &gt; Ocean Circulation &gt; Ocean Currents,
sea, sea_water_y_velocity, seawater, time_counter, v, velocity, vomecrty, water</att>
    <att name="keywords_vocabulary">GCMD Science Keywords</att>
    <att name="license">The Salish Sea MEOPAR NEMO model results are copyright 2013-2021
by the Salish Sea MEOPAR Project Contributors and The University of British Columbia.

They are licensed under the Apache License, Version 2.0. http://www.apache.org/licenses/LICENSE-2.0</att>
    <att name="standard_name_vocabulary">CF Standard Name Table v29</att>
    <att name="summary">Nowcast, Salish Sea, 3d v Velocity Field, Hourly, v16-10

3d meridional (v) component velocity field values averaged over 1 hour intervals
from Salish Sea NEMO model nowcast runs. The values are calculated for the entire model grid
that includes the Juan de Fuca Strait, the Strait of Georgia, Puget Sound,
and Johnstone Strait on the coasts of Washington State and British Columbia.
Geo-location and depth data for the Salish Sea NEMO model grid are available in the ubcSSnBathymetry2V1 dataset.

v1: vVelocity variable

v16-10: NEMO-3.6; ubcSSnBathymetry2V16-07 bathymetry; see infoUrl link for full details.</att>
    <att name="title">Nowcast, Salish Sea, 3d v Velocity Field, Hourly, v16-10</att>
    <att name="project">Salish Sea MEOPAR NEMO Model</att>
    <att name="creator_name">Salish Sea MEOPAR Project Contributors</att>
    <att name="creator_email">sallen@eos.ubc.ca</att>
    <att name="creator_url">https://salishsea-meopar-docs.readthedocs.io/</att>
    <att name="acknowledgement">MEOPAR, ONC, Compute Canada</att>
    <att name="drawLandMask">over</att>
  </addAttributes>
  <axisVariable>
    <sourceName>time_counter</sourceName>
    <destinationName>time</destinationName>
    <!-- sourceAttributes>
            <att name="_ChunkSize" type="int">1</att>
            <att name="axis">T</att>
            <att name="bounds">time_counter_bounds</att>
            <att name="calendar">gregorian</att>
            <att name="long_name">Time axis</att>
            <att name="standard_name">time</att>
            <att name="time_origin">1900-01-01 00:00:00</att>
            <att name="units">seconds since 1900-01-01 00:00:00</att>
        </sourceAttributes -->
    <addAttributes>
      <att name="_ChunkSize">null</att>
      <att name="bounds">null</att>
    </addAttributes>
  </axisVariable>
  <axisVariable>
    <sourceName>depthv</sourceName>
    <destinationName>depth</destinationName>
    <!-- sourceAttributes>
            <att name="_ChunkSize" type="int">40</att>
            <att name="axis">Z</att>
            <att name="bounds">depthv_bounds</att>
            <att name="long_name">Vertical V levels</att>
            <att name="positive">down</att>
            <att name="units">m</att>
        </sourceAttributes -->
    <addAttributes>
      <att name="_ChunkSize">null</att>
      <att name="bounds">null</att>
      <att name="standard_name">depth</att>
    </addAttributes>
  </axisVariable>
  <axisVariable>
    <sourceName>y</sourceName>
    <destinationName>gridY</destinationName>
    <!-- sourceAttributes>
        </sourceAttributes -->
    <addAttributes>
      <att name="long_name">Y</att>
    </addAttributes>
  </axisVariable>
  <axisVariable>
    <sourceName>x</sourceName>
    <destinationName>gridX</destinationName>
    <!-- sourceAttributes>
        </sourceAttributes -->
    <addAttributes>
      <att name="long_name">X</att>
    </addAttributes>
  </axisVariable>
  <dataVariable>
    <sourceName>vomecrty</sourceName>
    <destinationName>vVelocity</destinationName>
    <dataType>float</dataType>
    <!-- sourceAttributes>
            <att name="_ChunkSize" type="intList">1 12 285 126</att>
            <att name="_FillValue" type="float">1.0E20</att>
            <att name="cell_methods">time: mean (interval: 40 s)</att>
            <att name="coordinates">time_centered depthv nav_lon nav_lat</att>
            <att name="interval_operation">40 s</att>
            <att name="interval_write">1 h</att>
            <att name="long_name">ocean current along j-axis</att>
            <att name="missing_value" type="float">1.0E20</att>
            <att name="online_operation">average</att>
            <att name="standard_name">sea_water_y_velocity</att>
            <att name="units">m/s</att>
        </sourceAttributes -->
    <addAttributes>
      <att name="_ChunkSize">null</att>
      <att name="colorBarMaximum" type="double">0.5</att>
      <att name="colorBarMinimum" type="double">-0.5</att>
      <att name="coordinates">null</att>
    </addAttributes>
  </dataVariable>
</dataset>

Extra processing step are required for some types of datasets. See:

• Surface Field Datasets

• Model Grid Geo-location and Bathymetry Datasets

• EC HDRPS Atmospheric Forcing Datasets

Store the XML fragment for the dataset:

with open('/results/erddap-datasets/fragments/{}.xml'.format(datasetID), 'wb') as f:
    f.write(etree.tostring(root, pretty_print=True))

Edit /results/erddap-datasets/datasets.xml to include the XML fragment for the dataset that was stored by the above cell.

That file is symlinked to /opt/tomcat/content/erddap/datasets.xml.

Create a flag file to signal the ERDDAP server process to load the dataset:

$ cd /results/erddap/flag/
$ touch <datasetID>

If the dataset does not appear on https://salishsea.eos.ubc.ca/erddap/info/, check /results/erddap/logs/log.txt for error messages from the dataset load process (they may not be at the end of the file because ERDDAP is pretty chatty).

Once the dataset has been successfully loaded and you are happy with the metadata that ERDDAP is providing for it, commit the changes in /results/erddap-datasets/ and push them to GitHub.

Surface Field Datasets

The /opt/tomcat/webapps/erddap/WEB-INF/GenerateDatasetsXml.sh script produces and XML fragment that uses all of the dimensions that it finds in the sample file it parses, and includes only the variables that have all of those dimensions. To produce an XML fragment for surface fields we need to do some additional work:

• Delete the depth axis

• Delete all of the dataVariable elements

• Add dataVariable elements for the surface variables

for axis in root.findall('.//axisVariable'):
    if axis.find('.//destinationName').text == 'depth':
        axis.getparent().remove(axis)
        break

for var in root.findall('.//dataVariable'):
    var.getparent().remove(var)

var = etree.SubElement(root, 'dataVariable')
etree.SubElement(var, 'sourceName').text = 'sossheig'
etree.SubElement(var, 'destinationName').text = 'ssh'
etree.SubElement(var, 'dataType').text = 'float'
attrs = etree.SubElement(var, 'addAttributes')
etree.SubElement(attrs, 'att', name='_ChunkSize').text = 'null'
etree.SubElement(attrs, 'att', name='coordinates').text = 'null'

var = etree.SubElement(root, 'dataVariable')
etree.SubElement(var, 'sourceName').text = 'rain_rate'
etree.SubElement(var, 'destinationName').text = 'rain_rate'
etree.SubElement(var, 'dataType').text = 'float'
attrs = etree.SubElement(var, 'addAttributes')
etree.SubElement(attrs, 'att', name='_ChunkSize').text = 'null'
etree.SubElement(attrs, 'att', name='coordinates').text = 'null'

find_att(root, 'keywords').text = (
    'model results, height, local, sea, sea surface height, sossheig, source, surface, time_counter')

print_tree(root)

with open('/results/erddap-datasets/fragments/{}.xml'.format(datasetID), 'wb') as f:
    f.write(etree.tostring(root, pretty_print=True))

Model Grid Geo-location and Bathymetry Datasets

Model grid geo-location and bathymetry datasets require a lot of hand editing because they are not model generated. Here is an example of a finished one:

parser = etree.XMLParser(remove_blank_text=True)
tree = etree.parse('/results/erddap-datasets/fragments/ubcSSnBathymetry2V1.xml', parser)
root = tree.getroot()

print_tree(root)

<dataset type="EDDGridFromNcFiles" datasetID="ubcSSnBathymetry2V1" active="true">
  <reloadEveryNMinutes>10080</reloadEveryNMinutes>
  <updateEveryNMillis>10000</updateEveryNMillis>
  <fileDir>/results/nowcast-sys/NEMO-forcing/grid/</fileDir>
  <recursive>false</recursive>
  <fileNameRegex>.*SalishSea2_NEMO_bathy\.nc$</fileNameRegex>
  <metadataFrom>last</metadataFrom>
  <matchAxisNDigits>20</matchAxisNDigits>
  <fileTableInMemory>false</fileTableInMemory>
  <accessibleViaFiles>false</accessibleViaFiles>
  <!-- sourceAttributes>
        <att name="comment">Bathymetry, Latitudes and Longitudes</att>
        <att name="Conventions">CF-1.6</att>
        <att name="history">[2016-02-05 Created]</att>
        <att name="institution">Dept of Earth, Ocean &amp; Atmospheric Sciences, University of British Columbia</att>
        <att name="references">REQUIRED</att>
        <att name="source">https://bitbucket.org/salishsea/analysis/src/tip/Susan/NEMOBathymetryfromMeshMask.ipynb</att>
        <att name="title">Bathymetry after NEMO Processes, SalishSea Bathymetry 2</att>
    </sourceAttributes -->
  <addAttributes>
    <att name="cdm_data_type">Grid</att>
    <att name="coverage_content_type">modelResult</att>
    <att name="Conventions">CF-1.6, COARDS, ACDD-1.3</att>
    <att name="infoUrl">https://salishsea-meopar-tools.readthedocs.org/en/latest/results_server/index.html#salish-sea-model-results</att>
    <att name="institution">UBC EOAS</att>
    <att name="institution_fullname">Earth, Ocean &amp; Atmospheric Sciences, University of British Columbia</att>
    <att name="creator_name">Dept of Earth, Ocean &amp; Atmospheric Sciences, University of British Columbia</att>
    <att name="keywords">bathymetry, bottom, data, model results, depth, floor, latitude, longitude, nemo, ocean, oceans,
Oceans &gt; Bathymetry/Seafloor Topography &gt; Bathymetry, salishsea, sea, sea_floor_depth, seafloor, topography</att>
    <att name="keywords_vocabulary">GCMD Science Keywords</att>
    <att name="license">The Salish Sea MEOPAR NEMO model results are copyright 2013-2021
by the Salish Sea MEOPAR Project Contributors and The University of British Columbia.

They are licensed under the Apache License, Version 2.0. http://www.apache.org/licenses/LICENSE-2.0</att>
    <att name="standard_name_vocabulary">CF Standard Name Table v29</att>
    <att name="summary">Salish Sea NEMO Model Grid, Geo-location and Bathymetry, v1

Longitude, latitude, and bathymetry of the Salish Sea NEMO model grid.
The bathymetry values are those calculated by NEMO from the input bathymetry file.
NEMO modifies the input bathymetry to remove isolated holes, and too-small partial steps.
The model grid includes the Juan de Fuca Strait, the Strait of Georgia, Puget Sound,
and Johnstone Strait on the coasts of Washington State and British Columbia.

v1: longitude, latitude and bathymetry variables</att>
    <att name="title">Salish Sea NEMO Model Grid, Geo-location and Bathymetry, v1</att>
    <att name="project">Salish Sea MEOPAR NEMO Model</att>
    <att name="creator_name">Salish Sea MEOPAR Project Contributors</att>
    <att name="creator_email">sallen@eos.ubc.ca</att>
    <att name="creator_url">https://salishsea-meopar-docs.readthedocs.org/</att>
    <att name="acknowledgement">MEOPAR, ONC, Compute Canada</att>
    <att name="drawLandMask">over</att>
  </addAttributes>
  <axisVariable>
    <sourceName>y</sourceName>
    <destinationName>gridY</destinationName>
    <!-- sourceAttributes>
        </sourceAttributes -->
    <addAttributes>
      <att name="long_name">Y</att>
    </addAttributes>
  </axisVariable>
  <axisVariable>
    <sourceName>x</sourceName>
    <destinationName>gridX</destinationName>
    <!-- sourceAttributes>
        </sourceAttributes -->
    <addAttributes>
      <att name="long_name">X</att>
    </addAttributes>
  </axisVariable>
  <dataVariable>
    <sourceName>longitude</sourceName>
    <destinationName>longitude</destinationName>
    <dataType>float</dataType>
    <!-- sourceAttributes>
            <att name="long_name">Longitude</att>
            <att name="units">degrees_east</att>
        </sourceAttributes -->
    <addAttributes>
      <att name="colorBarMaximum" type="double">180.0</att>
      <att name="colorBarMinimum" type="double">-180.0</att>
      <att name="standard_name">longitude</att>
    </addAttributes>
  </dataVariable>
  <dataVariable>
    <sourceName>latitude</sourceName>
    <destinationName>latitude</destinationName>
    <dataType>float</dataType>
    <!-- sourceAttributes>
            <att name="long_name">Latitude</att>
            <att name="units">degrees_north</att>
        </sourceAttributes -->
    <addAttributes>
      <att name="colorBarMaximum" type="double">90.0</att>
      <att name="colorBarMinimum" type="double">-90.0</att>
      <att name="standard_name">latitude</att>
    </addAttributes>
  </dataVariable>
  <dataVariable>
    <sourceName>bathymetry</sourceName>
    <destinationName>bathymetry</destinationName>
    <dataType>float</dataType>
    <!-- sourceAttributes>
            <att name="coordinates">longitude latitude</att>
            <att name="grid">Salish Sea 2</att>
            <att name="long_name">Depth of Bottom</att>
            <att name="units">m</att>
        </sourceAttributes -->
    <addAttributes>
      <att name="colorBarMaximum" type="double">450.0</att>
      <att name="colorBarMinimum" type="double">0.0</att>
      <att name="colorBarPalette">OceanDepth</att>
      <att name="coordinates">null</att>
      <att name="standard_name">sea_floor_depth</att>
    </addAttributes>
  </dataVariable>
</dataset>

EC HDRPS Atmospheric Forcing Datasets

Atmospheric Forcing Grid Geo-location Dataset

Use the /opt/tomcat/webapps/erddap/WEB-INF/GenerateDatasetsXml.sh script generate the initial version of an XML fragment for the dataset:

$ cd /opt/tomcat/webapps/erddap/WEB-INF/
$ bash GenerateDatasetsXml.sh EDDGridFromNcFiles /results/forcing/atmospheric/GEM2.5/operational/ ops_y\d{4}m\d{2}d\d{2}.nc$ /results/forcing/atmospheric/GEM2.5/operational/ops_y2016m03d07.nc 10080

Like the model grid geo-location and bathymetry dataset, the atmospheric forcing grid dataset requires a lot of hand editing. Here is the finished dataset:

parser = etree.XMLParser(remove_blank_text=True)
tree = etree.parse('/results/erddap-datasets/fragments/ubcSSaAtmosphereGridV1.xml', parser)
root = tree.getroot()

print_tree(root)

<dataset type="EDDGridFromNcFiles" datasetID="ubcSSaAtmosphereGridV1" active="true">
  <reloadEveryNMinutes>10080</reloadEveryNMinutes>
  <updateEveryNMillis>10000</updateEveryNMillis>
  <fileDir>/results/forcing/atmospheric/GEM2.5/operational/</fileDir>
  <recursive>false</recursive>
  <fileNameRegex>ops_y2016m03d07.nc$</fileNameRegex>
  <metadataFrom>last</metadataFrom>
  <matchAxisNDigits>20</matchAxisNDigits>
  <fileTableInMemory>false</fileTableInMemory>
  <accessibleViaFiles>false</accessibleViaFiles>
  <!-- sourceAttributes>
      <att name="Conventions">CF-1.0</att>
      <att name="GRIB2_grid_template" type="int">20</att>
      <att name="History">Mon Mar  7 10:07:34 2016: ncks -4 -L4 -O /results/forcing/atmospheric/GEM2.5/operational/ops_y2016m03d07.nc /results/forcing/atmospheric/GEM2.5/operational/ops_y2016m03d07.nc
  created by wgrib2</att>
      <att name="NCO">4.4.2</att>
  </sourceAttributes -->
  <addAttributes>
    <att name="cdm_data_type">Grid</att>
    <att name="coverage_content_type">modelResult</att>
    <att name="Conventions">CF-1.6, COARDS, ACDD-1.3</att>
    <att name="History">null</att>
    <att name="history">Mon Mar  7 10:07:34 2016: ncks -4 -L4 -O /results/forcing/atmospheric/GEM2.5/operational/ops_y2016m03d07.nc /results/forcing/atmospheric/GEM2.5/operational/ops_y2016m03d07.nc
    created by wgrib2</att>
    <att name="infoUrl">https://salishsea-meopar-tools.readthedocs.org/en/latest/results_server/index.html#salish-sea-model-results</att>
    <att name="institution">UBC EOAS</att>
    <att name="institution_fullname">Earth, Ocean &amp; Atmospheric Sciences, University of British Columbia</att>
    <att name="creator_name">Dept of Earth, Ocean &amp; Atmospheric Sciences, University of British Columbia</att>
    <att name="keywords">atmosphere, model results, HRDPS, latitude, longitude, salishsea</att>
    <att name="keywords_vocabulary">GCMD Science Keywords</att>
    <att name="license">The Salish Sea MEOPAR NEMO model results are copyright 2013-2021
by the Salish Sea MEOPAR Project Contributors and The University of British Columbia.

They are licensed under the Apache License, Version 2.0. http://www.apache.org/licenses/LICENSE-2.0

This dataset is derived from a product of the Environment Canada HRDPS (High Resolution Deterministic Prediction System) model. The Terms and conditions of use of Meteorological Data from Environment Canada are available at http://dd.weather.gc.ca/doc/LICENCE_GENERAL.txt.</att>
    <att name="standard_name_vocabulary">CF Standard Name Table v29</att>
    <att name="summary">HRDPS, Salish Sea, Atmospheric Forcing Grid, Geo-location, v1

Longitude and latitude of the Environment Canada HRDPS atmospheric forcing model grid that is used to for the Salish Sea NEMO model.
The model grid includes the Juan de Fuca Strait, the Strait of Georgia, Puget Sound,
and Johnstone Strait on the coasts of Washington State and British Columbia.

v1: longitude and latitude</att>
    <att name="title">HRDPS, Salish Sea, Atmospheric Forcing Grid, Geo-location, v1</att>
    <att name="project">Salish Sea MEOPAR NEMO Model</att>
    <att name="creator_name">Salish Sea MEOPAR Project Contributors</att>
    <att name="creator_email">sallen@eos.ubc.ca</att>
    <att name="creator_url">https://salishsea-meopar-docs.readthedocs.org/</att>
    <att name="acknowledgement">Environment Canada, MEOPAR, ONC, Compute Canada</att>
    <att name="drawLandMask">over</att>
  </addAttributes>
  <axisVariable>
    <sourceName>y</sourceName>
    <destinationName>gridY</destinationName>
    <addAttributes>
      <att name="long_name">Y</att>
    </addAttributes>
  </axisVariable>
  <axisVariable>
    <sourceName>x</sourceName>
    <destinationName>gridX</destinationName>
    <addAttributes>
      <att name="long_name">X</att>
    </addAttributes>
  </axisVariable>
  <dataVariable>
    <sourceName>nav_lon</sourceName>
    <destinationName>longitude</destinationName>
    <dataType>float</dataType>
    <addAttributes>
      <att name="standard_name">longitude</att>
    </addAttributes>
  </dataVariable>
  <dataVariable>
    <sourceName>nav_lat</sourceName>
    <destinationName>latitude</destinationName>
    <dataType>float</dataType>
    <addAttributes>
      <att name="standard_name">latitude</att>
    </addAttributes>
  </dataVariable>
</dataset>

Atmospheric Forcing Model Fields

• Change the value of the recursive element to false so that the /results/forcing/atmospheric/GEM2.5/operational/fcst/ directory is excluded

• Add Environment Canada acknowledgement and terms & conditions of use to license element

• Add Environment Canada to acknowledgement element

root.find('.//recursive').text = 'false'
find_att(root, 'license').text += '''

This dataset is derived from a product of the Environment Canada HRDPS (High Resolution Deterministic Prediction System)
model. The Terms and conditions of use of Meteorological Data from Environment Canada are available at
http://dd.weather.gc.ca/doc/LICENCE_GENERAL.txt.</att>'''
find_att(root, 'acknowledgement').text += ', Environment Canada'

for axis in root.findall('.//axisVariable'):
    axis_name = axis.find('.//sourceName').text
    if 'time' not in axis_name:
        attrs = axis.find('.//addAttributes')
        etree.SubElement(attrs, 'att', name='grid_spacing').text = 'null'
        etree.SubElement(attrs, 'att', name='units').text = 'null'
        etree.SubElement(attrs, 'att', name='long_name').text = axis_name.upper()
        etree.SubElement(attrs, 'att', name='standard_name').text = axis_name

print_tree(root)

<dataset type="EDDGridFromNcFiles" datasetID="ubcSSaSurfaceAtmosphereFieldsV1" active="true">
  <reloadEveryNMinutes>10080</reloadEveryNMinutes>
  <updateEveryNMillis>10000</updateEveryNMillis>
  <fileDir>/results/forcing/atmospheric/GEM2.5/operational/</fileDir>
  <recursive>false</recursive>
  <fileNameRegex>.*ops_y\d{4}m\d{2}d\d{2}\.nc$</fileNameRegex>
  <metadataFrom>last</metadataFrom>
  <matchAxisNDigits>20</matchAxisNDigits>
  <fileTableInMemory>false</fileTableInMemory>
  <accessibleViaFiles>false</accessibleViaFiles>
  <!-- sourceAttributes>
        <att name="Conventions">CF-1.0</att>
        <att name="GRIB2_grid_template" type="int">20</att>
        <att name="History">Thu Mar 10 10:11:37 2016: ncks -4 -L4 -O /results/forcing/atmospheric/GEM2.5/operational/ops_y2016m03d10.nc /results/forcing/atmospheric/GEM2.5/operational/ops_y2016m03d10.nc
created by wgrib2</att>
        <att name="NCO">4.4.2</att>
    </sourceAttributes -->
  <addAttributes>
    <att name="cdm_data_type">Grid</att>
    <att name="coverage_content_type">modelResult</att>
    <att name="Conventions">CF-1.6, COARDS, ACDD-1.3</att>
    <att name="History">null</att>
    <att name="history">Thu Mar 10 10:11:37 2016: ncks -4 -L4 -O /results/forcing/atmospheric/GEM2.5/operational/ops_y2016m03d10.nc /results/forcing/atmospheric/GEM2.5/operational/ops_y2016m03d10.nc
created by wgrib2</att>
    <att name="infoUrl">https://salishsea-meopar-tools.readthedocs.org/en/latest/results_server/index.html#salish-sea-model-results</att>
    <att name="institution">UBC EOAS</att>
    <att name="institution_fullname">Earth, Ocean &amp; Atmospheric Sciences, University of British Columbia</att>
    <att name="keywords">atmosphere,
Atmosphere &gt; Atmospheric Water Vapor &gt; Humidity,
atmospheric, atmpres, component, data, downward, flux, humidity, level, local, long, long-wave, mean, msl, precipitation, pressure, qair, rad, radiation, rain, rainfall, reduced, sea, seawater, short, short-wave, solar, source, specific, specific_humidity, tair, temperature, therm_rad, time_counter, total, u-component, u_wind, v-component, v_wind, vapor, water, wave, wind</att>
    <att name="keywords_vocabulary">GCMD Science Keywords</att>
    <att name="license">The Salish Sea MEOPAR NEMO model results are copyright 2013-2021
by the Salish Sea MEOPAR Project Contributors and The University of British Columbia.

They are licensed under the Apache License, Version 2.0. http://www.apache.org/licenses/LICENSE-2.0

This dataset is derived from a product of the Environment Canada HRDPS (High Resolution Deterministic Prediction System)
model. The Terms and conditions of use of Meteorological Data from Environment Canada are available at
http://dd.weather.gc.ca/doc/LICENCE_GENERAL.txt.&lt;/att&gt;</att>
    <att name="standard_name_vocabulary">CF Standard Name Table v29</att>
    <att name="summary">HRDPS, Salish Sea, Atmospheric Forcing Fields, Hourly, v1

2d hourly atmospheric field values from the
Environment Canada HRDPS atmospheric forcing model that are used to force the Salish Sea NEMO model.
The model grid includes the Juan de Fuca Strait, the Strait of Georgia, Puget Sound,
and Johnstone Strait on the coasts of Washington State and British Columbia.
Geo-location data for the atmospheric forcing grid are available in the ubcSSaAtmosphereGridV1 dataset.
Atmospheric field values are interpolated on to the Salish Sea NEMO model grid (ubcSSnBathymetry2V1 dataset)
on-the-fly by NEMO.

v1: atmospheric pressure, precipitation rate, 2m specific humidity, 2m air temperature,
short-wave radiation flux, long-wave radiation flux, 10m u wind component, 10m v wind component variables</att>
    <att name="title">HRDPS, Salish Sea, Atmospheric Forcing Fields, Hourly, v1</att>
    <att name="project">Salish Sea MEOPAR NEMO Model</att>
    <att name="creator_name">Salish Sea MEOPAR Project Contributors</att>
    <att name="creator_email">sallen@eos.ubc.ca</att>
    <att name="creator_url">https://salishsea-meopar-docs.readthedocs.org/</att>
    <att name="acknowledgement">MEOPAR, ONC, Compute Canada, Environment Canada</att>
    <att name="drawLandMask">over</att>
  </addAttributes>
  <axisVariable>
    <sourceName>time_counter</sourceName>
    <destinationName>time</destinationName>
    <!-- sourceAttributes>
            <att name="_ChunkSize" type="int">1</att>
            <att name="long_name">verification time generated by wgrib2 function verftime()</att>
            <att name="reference_date">2016.03.09 18:00:00 UTC</att>
            <att name="reference_time" type="double">1.4575464E9</att>
            <att name="reference_time_description">kind of product unclear, reference date is variable, min found reference date is given</att>
            <att name="reference_time_type" type="int">0</att>
            <att name="time_origin">1970-Jan-01 00:00:00</att>
            <att name="time_step" type="double">3600.0</att>
            <att name="time_step_setting">auto</att>
            <att name="units">seconds since 1970-01-01 00:00:00.0 0:00</att>
        </sourceAttributes -->
    <addAttributes>
      <att name="_ChunkSize">null</att>
      <att name="standard_name">time</att>
    </addAttributes>
  </axisVariable>
  <axisVariable>
    <sourceName>y</sourceName>
    <destinationName>gridY</destinationName>
    <!-- sourceAttributes>
            <att name="_ChunkSize" type="int">266</att>
            <att name="grid_spacing" type="double">2500.0</att>
            <att name="long_name">y coordinate of projection</att>
            <att name="standard_name">projection_y_coordinate</att>
            <att name="units">m</att>
        </sourceAttributes -->
    <addAttributes>
      <att name="_ChunkSize">null</att>
      <att name="grid_spacing">null</att>
      <att name="units">null</att>
      <att name="long_name">Y</att>
      <att name="standard_name">y</att>
    </addAttributes>
  </axisVariable>
  <axisVariable>
    <sourceName>x</sourceName>
    <destinationName>gridX</destinationName>
    <!-- sourceAttributes>
            <att name="_ChunkSize" type="int">256</att>
            <att name="grid_spacing" type="double">2500.0</att>
            <att name="long_name">x coordinate of projection</att>
            <att name="standard_name">projection_x_coordinate</att>
            <att name="units">m</att>
        </sourceAttributes -->
    <addAttributes>
      <att name="_ChunkSize">null</att>
      <att name="grid_spacing">null</att>
      <att name="units">null</att>
      <att name="long_name">X</att>
      <att name="standard_name">x</att>
    </addAttributes>
  </axisVariable>
  <dataVariable>
    <sourceName>atmpres</sourceName>
    <destinationName>atmpres</destinationName>
    <dataType>float</dataType>
    <!-- sourceAttributes>
            <att name="_ChunkSize" type="intList">1 266 256</att>
            <att name="_FillValue" type="float">9.999E20</att>
            <att name="coordinates">longitude latitude</att>
            <att name="level">mean sea level</att>
            <att name="long_name">Pressure Reduced to MSL</att>
            <att name="short_name">PRMSL_meansealevel</att>
            <att name="units">Pa</att>
        </sourceAttributes -->
    <addAttributes>
      <att name="_ChunkSize">null</att>
      <att name="coordinates">null</att>
    </addAttributes>
  </dataVariable>
  <dataVariable>
    <sourceName>precip</sourceName>
    <destinationName>precip</destinationName>
    <dataType>float</dataType>
    <!-- sourceAttributes>
            <att name="_ChunkSize" type="intList">1 266 256</att>
            <att name="_FillValue" type="float">9.999E20</att>
            <att name="coordinates">longitude latitude</att>
            <att name="level">surface</att>
            <att name="long_name">Total Precipitation</att>
            <att name="short_name">APCP_surface</att>
            <att name="units">kg/m^2</att>
        </sourceAttributes -->
    <addAttributes>
      <att name="_ChunkSize">null</att>
      <att name="colorBarMaximum" type="double">200.0</att>
      <att name="colorBarMinimum" type="double">0.0</att>
      <att name="coordinates">null</att>
    </addAttributes>
  </dataVariable>
  <dataVariable>
    <sourceName>qair</sourceName>
    <destinationName>qair</destinationName>
    <dataType>float</dataType>
    <!-- sourceAttributes>
            <att name="_ChunkSize" type="intList">1 266 256</att>
            <att name="_FillValue" type="float">9.999E20</att>
            <att name="coordinates">longitude latitude</att>
            <att name="level">2 m above ground</att>
            <att name="long_name">Specific Humidity</att>
            <att name="short_name">SPFH_2maboveground</att>
            <att name="units">kg/kg</att>
        </sourceAttributes -->
    <addAttributes>
      <att name="_ChunkSize">null</att>
      <att name="colorBarMaximum" type="double">128.0</att>
      <att name="colorBarMinimum" type="double">0.0</att>
      <att name="coordinates">null</att>
      <att name="standard_name">specific_humidity</att>
    </addAttributes>
  </dataVariable>
  <dataVariable>
    <sourceName>solar</sourceName>
    <destinationName>solar</destinationName>
    <dataType>float</dataType>
    <!-- sourceAttributes>
            <att name="_ChunkSize" type="intList">1 266 256</att>
            <att name="_FillValue" type="float">9.999E20</att>
            <att name="coordinates">longitude latitude</att>
            <att name="level">surface</att>
            <att name="long_name">Downward Short-Wave Radiation Flux</att>
            <att name="short_name">DSWRF_surface</att>
            <att name="units">W/m^2</att>
        </sourceAttributes -->
    <addAttributes>
      <att name="_ChunkSize">null</att>
      <att name="colorBarMaximum" type="double">500.0</att>
      <att name="colorBarMinimum" type="double">-500.0</att>
      <att name="coordinates">null</att>
    </addAttributes>
  </dataVariable>
  <dataVariable>
    <sourceName>tair</sourceName>
    <destinationName>tair</destinationName>
    <dataType>float</dataType>
    <!-- sourceAttributes>
            <att name="_ChunkSize" type="intList">1 266 256</att>
            <att name="_FillValue" type="float">9.999E20</att>
            <att name="coordinates">longitude latitude</att>
            <att name="level">2 m above ground</att>
            <att name="long_name">Temperature</att>
            <att name="short_name">TMP_2maboveground</att>
            <att name="units">K</att>
        </sourceAttributes -->
    <addAttributes>
      <att name="_ChunkSize">null</att>
      <att name="colorBarMaximum" type="double">313.0</att>
      <att name="colorBarMinimum" type="double">263.0</att>
      <att name="coordinates">null</att>
    </addAttributes>
  </dataVariable>
  <dataVariable>
    <sourceName>therm_rad</sourceName>
    <destinationName>therm_rad</destinationName>
    <dataType>float</dataType>
    <!-- sourceAttributes>
            <att name="_ChunkSize" type="intList">1 266 256</att>
            <att name="_FillValue" type="float">9.999E20</att>
            <att name="coordinates">longitude latitude</att>
            <att name="level">surface</att>
            <att name="long_name">Downward Long-Wave Rad. Flux</att>
            <att name="short_name">DLWRF_surface</att>
            <att name="units">W/m^2</att>
        </sourceAttributes -->
    <addAttributes>
      <att name="_ChunkSize">null</att>
      <att name="coordinates">null</att>
    </addAttributes>
  </dataVariable>
  <dataVariable>
    <sourceName>u_wind</sourceName>
    <destinationName>u_wind</destinationName>
    <dataType>float</dataType>
    <!-- sourceAttributes>
            <att name="_ChunkSize" type="intList">1 266 256</att>
            <att name="_FillValue" type="float">9.999E20</att>
            <att name="coordinates">longitude latitude</att>
            <att name="level">10 m above ground</att>
            <att name="long_name">U-Component of Wind</att>
            <att name="short_name">UGRD_10maboveground</att>
            <att name="units">m/s</att>
        </sourceAttributes -->
    <addAttributes>
      <att name="_ChunkSize">null</att>
      <att name="coordinates">null</att>
    </addAttributes>
  </dataVariable>
  <dataVariable>
    <sourceName>v_wind</sourceName>
    <destinationName>v_wind</destinationName>
    <dataType>float</dataType>
    <!-- sourceAttributes>
            <att name="_ChunkSize" type="intList">1 266 256</att>
            <att name="_FillValue" type="float">9.999E20</att>
            <att name="coordinates">longitude latitude</att>
            <att name="level">10 m above ground</att>
            <att name="long_name">V-Component of Wind</att>
            <att name="short_name">VGRD_10maboveground</att>
            <att name="units">m/s</att>
        </sourceAttributes -->
    <addAttributes>
      <att name="_ChunkSize">null</att>
      <att name="coordinates">null</att>
    </addAttributes>
  </dataVariable>
</dataset>

with open('/results/erddap-datasets/fragments/{}.xml'.format(datasetID), 'wb') as f:
    f.write(etree.tostring(root, pretty_print=True))