Link to External Data Sources

Connect data files and metadata stored in other repositories to your ESS-DIVE dataset. Read more to learn how External Linking works and when to use it.

What is External Linking?

External Linking allows data contributors to connect data files and metadata stored in other repositories to ESS-DIVE datasets. It enables data to be stored where it makes the most sense scientifically and practically, while also following ESS Data Management and Sharing Policy to store searchable metadata on ESS-DIVE.

An external link is easily accessible from an ESS-DIVE dataset landing page and has a clearly defined relationship to the dataset. With this feature, your ESS-DIVE dataset can be linked to data files and metadata that have already been published on another established data repository or to data files that cannot be uploaded to ESS-DIVE.

Not all external data or metadata is suitable for external linking on ESS-DIVE. All externally linked datasets will be reviewed by the ESS-DIVE Team before publication and your external links may not be accepted. In this documentation, we will review the types of external data and metadata that is acceptable to link to as well as how to add external links to your dataset.

Figure 1: All external data files are listed in a new table underneath the existing file table.

External Link Table

Datasets with external links will have an external link table underneath the existing file table. The external link table follows a specific format and each link will have the following three components:

1. A brief but meaningful description for this web page,

2. The relationship type and it's definition, and

3. The URL where the data/metadata resides.

ESS-DIVE uses shared vocabulary from Schema.org to define external linking relationship types. This vocabulary is controlled, machine readable, and can be understood by most major search engines. ESS-DIVE has approved three Schema.org vocabularies for use with external linking. For more information, see the Relationship Types section.

Why Link External Data?

There are numerous reasons why linking out to external data sources may be advantageous for certain dataset publications. In this section, we will review the reasons why external linking may be needed and the common examples in which it is used.

• Store your data where it makes the most sense scientifically or for practical reasons (e.g. project data archive), while also complying with ESS data policy to at least store metadata describing your data on ESS-DIVE.

• Generally enhance the discoverability of your dataset by making it searchable on ESS-DIVE, along with other ESS project data stored on ESS-DIVE.

• Your data file storage volume is greater than 500GB and is too large to upload directly to ESS-DIVE. See more details under the Large Data Files examples.

• Your published data product is complex, or involves analysis tools used on another platform, and can be more easily accessed from the external source (e.g. model code stored on GitHub). See more details under the Data Analysis Platforms examples.

If your data product is not represented in this section, please refer to the Relationship Types section to see if one of the available relationships suit your data product.

Examples

1. Long-Term Data Repositories or Data Systems

External data stored in valid data repositories that provide long term data storage and stewardship of data does not need to be uploaded to ESS-DIVE. But again, if your project is funded by the DOE ESS program, you must at least store metadata describing the data on ESS-DIVE, with links to the published dataset. Some data repositories or data systems that are approved for external linking are:

• Environmental Data Initiative (EDI)

• National Microbiome Data Collaborative (NMDC)

• USGS ScienceBase

If your data is stored in a repository not listed here, please reach out to the ESS-DIVE Support Team at [e-mail] to find out if that repository can be linked to your ESS-DIVE datasets.

2. Project Data Archives

Some ESS projects have archives where they upload and manage their data products. If these data are publicly accessible, you may take advantage of our external linking feature to store metadata on ESS-DIVE, with links to the original project data archive.

Some examples of projects archives are:

• Spruce and Peatland Responses Under Changing Environments (SPRUCE)

• Next-Generation Ecosystem Experiment (NGEE) Arctic

• Next-Generation Ecosystem Experiment (NGEE) Tropics

In cases where the project archive does not issue DOIs or is not a long-term data preservation repository, you will need to upload your data files to your ESS-DIVE datasets.

3. Software and Data Analysis Platforms

Some open-access data sources distribute data in a manner that provides useful scientific context to datasets. In some cases, it may be necessary to use External Linking to associate a dataset to the original data analysis platform. Some examples of such data sources are:

• GitHub

• Google Earth Engine

• KBase

If you are linking to data on a platform that does not issue DOIs, you must upload a copy of your data files to ESS-DIVE

4. Large Data Files

For certain datasets, the associated data files are too large to upload to and/or download from ESS-DIVE. In these cases, you may take one of two options. Either (1) upload your data to an established long term data storage service that can take your large data or (2) work with the ESS-DIVE team to upload this data to an external data source that is managed by the ESS-DIVE Team; we call this Large Data Storage.

Data files greater than 500GB cannot be uploaded or downloaded from ESS-DIVE. Please contact the ESS-DIVE Team at [e-mail] if you have a dataset with more than 500GB of data, or if you have a large number of data files.

Relationship Types

ESS-DIVE currently supports three types of relationships between ESS-DIVE datasets and external data sources. In this section we list the approved relationships for your reference. You can read through the following descriptions in Table 1 to see if your external data can be linked to your ESS-DIVE dataset.

If you feel like your data product falls into one of these three categories, email The ESS-DIVE Team at [e-mail] and we will help associate your data with the appropriate type.

Relationship	Description
same as	Original publication of this dataset (where the data+metadata can be found). The DOI url of this dataset, starting with 'https://doi.org/', redirects to original source.
archived at	Complete copy of the data files in the dataset resides in external source.
has part	One or more files that are part of the dataset held outside of the repository. This could be a link to an individual file or a directory.

Table 1: Lists ESS-DIVE's current external linking relationship types and their descriptions

Don't see a relationship type that is applicable to your use case?

We have the infrastructure to accommodate your needs! Please email the ESS-DIVE Support Team ([e-mail]) and share your use case with us. We can discuss expanding the list of acceptable external data relationships to accommodate your use case.

How to Link Datasets

Make a Request

External Linking is managed by the ESS-DIVE Team. To externally link your dataset to an external data product, please contact [e-mail] to start the process. This section outlines the steps involved in making and completing the request.

1. Email [e-mail] to start a request, with details on why you need external linking and any existing DOI(s), if you have them.

   1. When the request is initiated, The ESS-DIVE Team will ask that you provide a brief but meaningful description for each web page, as described in the External Link Table section.

   2. During this stage, the ESS-DIVE Team will also evaluate whether your external data/metadata is suitable to be externally linked. Not all requests are approved.

2. If there is an existing DOI, the ESS-DIVE Team uses a programmatic tool to transfer the metadata from the original repository (e.g. EDI) to one or more new ESS-DIVE dataset(s).

3. If there is NOT an existing DOI, you will create a new dataset for your data and send the associated ESS-DIVE identifier to the ESS-DIVE Team (e.g. ess-dive-ea25aaddf4b47a3-20211103T152543118).

4. The ESS-DIVE Team will populate the External Links table in your dataset, including the appropriate relationship type(s).

   1. The ESS-DIVE Team will select an appropriate relationship from Table 1 and apply it based on your dataset publication needs.

5. The ESS-DIVE Team will send you the dataset URL and give you ownership.

6. Review the dataset and complete your metadata. Our automated assessment report can help in your review. If you had an existing DOI and transferred your metadata, note that programmatic metadata transfers are not comprehensive and there will be missing information in your dataset.

7. Request to publish the dataset.

Use the API

ESS-DIVE's Package Service API can be used to programmatically and autonomously add external links to your datasets. The process for adding external links to datasets via the API only differs slightly from the standard tutorial for creating or updating datasets with the API. In these instructions, we will reference and expand on our existing tutorials. Detailed instructions on the use of the Package Service API can be found in the Package Service Tutorials.

1. Head to our Package Service Tutorial documentation page to learn how to get started with the API for the first time

2. Select an example page in your preferred coding language (R, Python, or Java)

   1. For the remainder of these instructions, we will provide references to the Python tutorial example. You can find a version of each of the linked Python sections in the R and Java tutorials as well.

3. Follow the Setup instructions

4. Create new or copy your existing dataset JSON-LD (i.e. your dataset metadata)

   1. If you are creating a dataset for the first time, skip to Create Metadata and follow the instructions for creating JSON-LD for your dataset metadata

   2. If you are updating a dataset, you can use the Get a Single Dataset code example to copy the JSON-LD for your existing dataset

5. Once you have your JSON-LD, you can now append the external linking schema onto it. Read through the available relationship types (Table 1) and decide which suits your dataset.

   1. To learn how to format your selected external linking relationship into your JSON-LD, navigate to ESS-DIVE's technical API documentation and locate your selected relationship in the dataset schema (Figure 2); either hasPart, archivedAt, and/or sameAs

   2. An example of each relationship's format is provided in code snippets at the end of this section (all code snippets are in Python)

6. Once you have added your external links, submit your dataset using the API!

   1. If you are submitting a new dataset without data files, skip to Create a Dataset > Metadata Only and copy the code examples

   2. If you are submitting a new dataset with data files, skip to Create a Dataset > Metadata and Data Files or Metadata and Many Data Files and copy the code example

   3. If you are updating an existing dataset metadata (without data files), skip to Update a Dataset > Metadata Only section and copy the code example

7. Head to ESS-DIVE (https://data.ess-dive.lbl.gov/) and request to publish your dataset

   1. During the publication process, your external links will be reviewed by the ESS-DIVE Team for suitability. At this stage, we will help refine your external links as needed or we may determine that your external data/metadata are not suitable for external linking. Not all external links will be approved.

Figure 2: The complete dataset schema can be found under the schema dropdown bar on the Package Service API's technical documentation page.

An example of the has part relationship JSON-LD schema.

hasPart = {
   "@type": "Webpage",
   "name": "Code developed in this GitHub repository",
   "url": "https://github.com"
   }

An example of the archived at relationship JSON-LD schema.

archivedAt = {
   "@type": "Webpage",
   "name": "All data is also available at GoogleEarthEngine",
   "url": "https://earthengine.google.com"
   }

An example of the same as relationship JSON-LD schema.

sameAs = "https://doi.org/10.XXX/NNNNN"

More Information

• All externally linked datasets must have complete metadata on ESS-DIVE.

• Externally linked datasets may or may not have data files attached, depending on your use case and needs.

  • For example, external links from data services or repositories that provide long term storage and stewardship of data will not be required to upload data files to ESS-DIVE.

• When data is originally published elsewhere with a DOI, the DOI url will resolve to the original external data source.

  • By default, when a dataset is originally published on another repository, the standard DOI url (e.g. https://doi.org/10.3334/CDIAC/SPRUCE.042) will not direct them to ESS-DIVE. Please contact the ESS-DIVE Support Team ([e-mail]) if you would like your DOI to resolve to your ESS-DIVE dataset.