# Overview

This website serves as an in-depth guide to discover, manage, and publish high-quality data on ESS-DIVE and is home to all help documentation.

{% hint style="info" icon="envelope" %}
We’re here to help — contact us with any questions you may have. Click the "Contact Us" button or email us directly at <ess-dive-support@lbl.gov>.
{% endhint %}

{% columns %}
{% column width="50%" %}

<p align="center"><a href="https://data.ess-dive.lbl.gov/data" class="button primary" data-icon="server">Search Data</a></p>

<p align="center"><a href="https://ess-dive.lbl.gov/" class="button primary" data-icon="house-chimney-window">ESS-DIVE Home Page</a></p>
{% endcolumn %}

{% column width="50%" %}

<p align="center"><a href="https://data.ess-dive.lbl.gov/submit" class="button primary" data-icon="upload">Submit Data</a></p>

<p align="center"><a href="https://data-sandbox.ess-dive.lbl.gov/data" class="button primary" data-icon="codepen">Sandbox: Test Data Submission</a></p>
{% endcolumn %}
{% endcolumns %}

## Introduction to ESS-DIVE Repository Features

{% embed url="<https://www.youtube.com/watch?v=Tkf72FMRbHM>" %}

## Next Steps

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th><th data-hidden data-card-cover data-type="image">Cover image</th></tr></thead><tbody><tr><td><h4>Search for Data</h4></td><td>Find the data you are looking for with ESS-DIVE's search tools.</td><td><a href="/searching-and-accessing-data/data-access">Search for Data</a></td><td><a href="https://images.unsplash.com/photo-1468779036391-52341f60b55d?crop=entropy&#x26;cs=srgb&#x26;fm=jpg&#x26;ixid=M3wxOTcwMjR8MHwxfHNlYXJjaHwyfHxjb21wdXRlciUyMGZpbGVzfGVufDB8fHx8MTc4MTEzMjU4M3ww&#x26;ixlib=rb-4.1.0&#x26;q=85">https://images.unsplash.com/photo-1468779036391-52341f60b55d?crop=entropy&#x26;cs=srgb&#x26;fm=jpg&#x26;ixid=M3wxOTcwMjR8MHwxfHNlYXJjaHwyfHxjb21wdXRlciUyMGZpbGVzfGVufDB8fHx8MTc4MTEzMjU4M3ww&#x26;ixlib=rb-4.1.0&#x26;q=85</a></td></tr><tr><td><h4>Metadata Requirements</h4></td><td>Understand ESS-DIVE's publication criteria and guidelines for metadata.</td><td><a href="/contributing-data/package-level-metadata">Metadata Requirements</a></td><td><a href="https://images.unsplash.com/photo-1515378791036-0648a3ef77b2?crop=entropy&#x26;cs=srgb&#x26;fm=jpg&#x26;ixid=M3wxOTcwMjR8MHwxfHNlYXJjaHwyfHxmb3JtfGVufDB8fHx8MTc4MTEzMjAwM3ww&#x26;ixlib=rb-4.1.0&#x26;q=85">https://images.unsplash.com/photo-1515378791036-0648a3ef77b2?crop=entropy&#x26;cs=srgb&#x26;fm=jpg&#x26;ixid=M3wxOTcwMjR8MHwxfHNlYXJjaHwyfHxmb3JtfGVufDB8fHx8MTc4MTEzMjAwM3ww&#x26;ixlib=rb-4.1.0&#x26;q=85</a></td></tr><tr><td><h4>Data Reporting Formats</h4></td><td>ESS-DIVE's standardized templates and guidelines for common data types. </td><td><a href="/contributing-data/data-reporting-formats">Data Reporting Formats</a></td><td><a href="https://images.unsplash.com/photo-1492496913980-501348b61469?crop=entropy&#x26;cs=srgb&#x26;fm=jpg&#x26;ixid=M3wxOTcwMjR8MHwxfHNlYXJjaHwxfHxzb2lsfGVufDB8fHx8MTc4MTEzMTk5M3ww&#x26;ixlib=rb-4.1.0&#x26;q=85">https://images.unsplash.com/photo-1492496913980-501348b61469?crop=entropy&#x26;cs=srgb&#x26;fm=jpg&#x26;ixid=M3wxOTcwMjR8MHwxfHNlYXJjaHwxfHxzb2lsfGVufDB8fHx8MTc4MTEzMTk5M3ww&#x26;ixlib=rb-4.1.0&#x26;q=85</a></td></tr><tr><td><h4>Register to Submit Data</h4></td><td>Register an account with ESS-DIVE before uploading or editing data.</td><td><a href="/contributing-data/new-contributor-registration">Register to Submit Data</a></td><td><a href="https://images.unsplash.com/photo-1556155092-490a1ba16284?crop=entropy&#x26;cs=srgb&#x26;fm=jpg&#x26;ixid=M3wxOTcwMjR8MHwxfHNlYXJjaHwyfHxyZWdpc3RlcnxlbnwwfHx8fDE3ODExMzE5ODR8MA&#x26;ixlib=rb-4.1.0&#x26;q=85">https://images.unsplash.com/photo-1556155092-490a1ba16284?crop=entropy&#x26;cs=srgb&#x26;fm=jpg&#x26;ixid=M3wxOTcwMjR8MHwxfHNlYXJjaHwyfHxyZWdpc3RlcnxlbnwwfHx8fDE3ODExMzE5ODR8MA&#x26;ixlib=rb-4.1.0&#x26;q=85</a></td></tr><tr><td><h4>Publish your Dataset</h4></td><td>Datasets must be finalized and reviewed by ESS-DIVE to be approved for publication.</td><td><a href="/publish-data/publish-your-dataset">Publish your Dataset</a></td><td><a href="https://images.unsplash.com/photo-1504711331083-9c895941bf81?crop=entropy&#x26;cs=srgb&#x26;fm=jpg&#x26;ixid=M3wxOTcwMjR8MHwxfHNlYXJjaHwyfHxwdWJsaXNofGVufDB8fHx8MTc4MTEzMTk3MXww&#x26;ixlib=rb-4.1.0&#x26;q=85">https://images.unsplash.com/photo-1504711331083-9c895941bf81?crop=entropy&#x26;cs=srgb&#x26;fm=jpg&#x26;ixid=M3wxOTcwMjR8MHwxfHNlYXJjaHwyfHxwdWJsaXNofGVufDB8fHx8MTc4MTEzMTk3MXww&#x26;ixlib=rb-4.1.0&#x26;q=85</a></td></tr><tr><td><h4>Update your Public Dataset</h4></td><td>Review ESS-DIVE's guidelines and expectations for versioning before updating your public dataset.</td><td></td><td><a href="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2Fsq39kMrDTwcrWds2INnO%2Fglenn-carstens-peters-npxXWgQ33ZQ-unsplash.jpg?alt=media&amp;token=1cc548d3-ea1c-462b-ba31-ef5d3ccd1c26">glenn-carstens-peters-npxXWgQ33ZQ-unsplash.jpg</a></td></tr></tbody></table>


# Frequently Asked Questions

* [General Questions](/faq#general-questions)
  * [Will you take my data?](/faq#will-you-take-my-data)
  * [How do I create an account with ESS-DIVE to upload data?](/faq#how-do-i-create-an-account-with-ess-dive-to-upload-data)
  * [I'm having trouble logging in with my ORCID.](#im-having-trouble-logging-in-with-my-orcid)
* [Search FAQ](/faq#search-faq)
  * [Why can’t I see all the CDIAC data here?](/faq#why-cant-i-see-all-the-cdiac-data-here)
  * [Will the CDIAC data be updated by ESS-DIVE beyond 2016?](/faq#will-the-cdiac-data-be-updated-by-ess-dive-beyond-2016)
  * [Why can’t I see the data I uploaded in the dataset listing?](#why-cant-i-see-the-data-i-uploaded-in-the-dataset-listing)
* [Upload FAQ](/faq#upload-faq)
  * [What are the characteristics of a good dataset?](#what-is-a-good-dataset)
  * [What metadata do you need to submit a dataset?](#what-metadata-do-you-need-to-submit-a-dataset)
  * [What if I want to practice uploading data first?](/faq#what-if-i-want-to-practice-uploading-data-first)
  * [What happens after I submit the data? ](/faq#what-happens-after-i-submit-the-data)
  * [What happens after I publish the data?](/faq#what-happens-after-i-publish-the-data)
  * [What do I do if I have updates to my dataset?](#what-do-i-do-if-i-have-updates-to-my-dataset)
  * [Can ESS-DIVE delete datasets?](#can-ess-dive-delete-datasets)
  * [What if my data upload is failing?](/faq#what-if-my-data-upload-is-failing)

## General Questions

### Will you take my data?

ESS-DIVE is primarily intended to host DOE-sponsored environmental research data, and will take data from projects funded by the DOE Environmental Systems Science Program in the Office of Science. However, we would consider archiving certain datasets that are not funded by ESS, but are of high value to our projects and programs. We will expect that any external data submissions conform to the [ESS-DIVE terms of use](https://ess-dive.lbl.gov/about/terms/). Contact <ess-dive-support@lbl.gov> if you are interested in archiving such data with ESS-DIVE.

### How do I create an account with ESS-DIVE to upload data?

Check out our [New User Registration help documentation](/contributing-data/new-contributor-registration) to get started with uploads.

### I'm having trouble logging in with my ORCID.

If login is not working, please try one or all of our [login troubleshooting tips](https://data.ess-dive.lbl.gov/signin-help). Contact <ess-dive-support@lbl.gov> if problems persist.

## Search FAQ

### Why can’t I see all the CDIAC data here?

ESS-DIVE is working on the transition of CDIAC data, and is hosting an [interim version of the CDIAC website](http://cdiac.ess-dive.lbl.gov/) that provides access to the CDIAC data. If you have any questions about ESS-DIVE or the data transition, contact <ess-dive-support@lbl.gov>.

The new archive for the CDIAC data will be ESS-DIVE, except in the specific cases mentioned below. The Oceanic Trace Gas data have been transitioned to the new Ocean Carbon Data System (OCADS) operated by NOAA’s National Centers for Environmental Information (NCEI) at <https://www.nodc.noaa.gov/ocads/>. The Total Carbon Column Observing Network (TCCON) data are being transitioned to Cal Tech at <http://www.tccon.caltech.edu/>. HIAPER Pole-to-Pole Observations (HIPPO) data are transitioning to the NCAR Earth Observing Laboratory at <https://www.eol.ucar.edu/data-software>.

### Will the CDIAC data be updated by ESS-DIVE beyond 2016?

ESS-DIVE will not be updating any data that are on the CDIAC website. Here are some links to check out for additional or new data.

**International Energy Agency** (<https://www.iea.org/statistics/relateddatabases/co2emissionsfromfuelcombustion/> ) – provides global and national (\~ 140 countries) estimates including estimates by sector (e.g., residential, electricity generation).

**Jos Oliver/Emissions Database for Global Atmospheric Research (EDGAR)** – provides emissions estimates for several atmospheric species including carbon dioxide including global and national time series and gridded (0.1 x 0.1) estimates.

**US Environmental Protection Agency** (<https://www.epa.gov/ghgemissions/sources-greenhouse-gas-emissions> ) – provides detailed emission estimates including sectoral estimates for the United States.

**Global Emission Inventory Activity** (GEIA, <http://eccad.aeris-data.fr/#DatasetPlace> ) – provides gridded emissions estimates for many atmospheric species including carbon dioxide.

**Tom Oda (NASA/GSFC)/ ODIAC (**<http://odiac.org/index.html>) - Odiac (Open-source Data Inventory for Anthropogenic CO2) is a global high-resolution emission dataset for fossil fuel carbon dioxide (CO2) emissions, initially developed by the Japanese Greenhouse gas Observing SATellite (GOSAT) project. Odiac emissions are based on CDIAC estimates and create spatial and temporal distributions using various proxy data such as satellite-observed nighttime lights and power plant profiles.

### Why can’t I see the data I uploaded in the dataset listing?

Check if you are logged in. If you haven’t Published your dataset, it will only be visible privately to you once you are logged in

## Upload FAQ

### What is a good dataset?

To see an example of a good dataset, visit [this package](https://data.ess-dive.lbl.gov/view/doi:10.3334/CDIAC/SPRUCE.024) on ESS-DIVE.\
\
If you would like to reference other datasets that have been reviewed and published by the ESS-DIVE Team, visit <https://data.ess-dive.lbl.gov/data> and explore the list of published datasets.

### What metadata do you need to submit a dataset?

We need metadata about what the dataset is about (e.g. title, description, keywords, funding), its association to a DOE project, and author information. Check out [this page](/contributing-data/submit-data-with-online-form#instructions-to-create-a-new-data-package) to preview the submission form to get a sense of the types of metadata a dataset needs.&#x20;

To understand ESS-DIVE's expectations for metadata content, see our [Package Level Metadata Guide](https://docs.ess-dive.lbl.gov/data-and-metadata-upload/package-level-metadata) for details.

### What if I want to practice uploading data first?

Please visit [**https://data-sandbox.ess-dive.lbl.gov**](https://data-sandbox.ess-dive.lbl.gov) to test the ESS-DIVE system and play around with the upload process. Any data that you save in the test system is temporary, and will not be preserved in ESS-DIVE. You do not need to request permission to upload data to ESS-DIVE before using sandbox.

### What happens after I submit the data?

The dataset will be entered into the ESS-DIVE system, and will only be visible to you privately until you Publish the data. You can continue to Edit the record after you submit.

### What happens after I publish the data?

An ESS-DIVE admin will review the record and respond to you by email. The record will not be published until you respond to the email from the ESS-DIVE admin. If you specified an existing DOI, we will publish the data with the same DOI, otherwise the record will be published with a new DOI.

### What do I do if I have updates to my dataset?

You can make minor updates to your dataset after it is published. If you have major updates, you may need to create a new version with a new DOI. Review ESS-DIVE [guidelines on updating and versioning](/publish-data/update-your-public-dataset) your data.

### Can ESS-DIVE delete datasets?

No. ESS-DIVE cannot delete public datasets. If a dataset has been obsoleted, you will create a new version and link both the original and new datasets in Related References. Review ESS-DIVE [guidelines on updating and versioning](/publish-data/update-your-public-dataset) your data.

### What if my data upload is failing?

Uploading large amounts of data can be really difficult on home internet connections. We recommend taking the following steps:&#x20;

1. Upload files one at a time and save after each upload is complete to ensure that the changes are properly saved.
2. Go to your computer’s battery settings and turn off sleep mode. If your system goes to sleep in the middle of an upload it can stop the process and prevent the changes from saving.

Please contact <ess-dive-support@lbl.gov> should the upload issue persist.

Additionally, you may want to consider using **Globus** to workaround upload errors. Globus is a third-party data transfer tool that ESS-DIVE commonly uses to support large or difficult file uploads. Learn more on our Globus documentation page:

{% content-ref url="/pages/-MX-a6sa8lrluGrgpHq6" %}
[Globus Data Transfer Service](/programmatic-tools/large-data-support)
{% endcontent-ref %}


# Get Started

Use this page to help decide how you will upload your dataset to ESS-DIVE by considering file upload limits and dataset organization.

A condensed summary of our Data Submission Guidelines is available on our website. You can find more details throughout this Guide to Using ESS-DIVE.&#x20;

{% embed url="<https://ess-dive.lbl.gov/archive/>" %}

## Terms of Use and Licensing&#x20;

By becoming an ESS-DIVE data contributor and submitting datasets, you are agreeing to ESS-DIVE's **Terms of Use**.&#x20;

#### Important items to review:&#x20;

1. Visit ESS-DIVE's Terms of Use and complete the ESS-DIVE Data Contributor Checklist (Part A).
2. Review and agree to the data contributor license ([Part B](https://ess-dive.lbl.gov/about/terms/#part_b)), and specify one of the standard ESS-DIVE data usage policies for serving data to the public.&#x20;
3. Review the Confidentiality and Ethics section ([Part C](https://ess-dive.lbl.gov/about/terms/#Confidentiality-and-Ethics)); ensure that your dataset, including all metadata and data files, does not contain any [controlled and prohibited information categories](https://commons.lbl.gov/display/rpm2/Controlled+and+Prohibited+Information+Categories) and that you uphold ethical norms to safeguard sensitive information.

{% embed url="<https://ess-dive.lbl.gov/about/terms/>" %}

## Try out Sandbox

**Sandbox** ([https://data-sandbox.ess-dive.lbl.gov](https://data-sandbox.ess-dive.lbl.gov/data)) is the testing grounds for data contributors to experiment with ESS-DIVE's repository features. We recommend that first time data contributors start out on Sandbox.

It's the perfect place to try out data submissions, teams, sharing, and data portals. Sandbox is routinely purged, so you can play around as much as you like! You can also programmatically submit data to Sandbox using the Dataset API (<https://api-sandbox.ess-dive.lbl.gov>).

**Production** (<https://data.ess-dive.lbl.gov/>) is the ESS-DIVE repository. First time data contributors are welcome to try out searches directly on the production website and with the API. Only use it to create datasets when you're ready to publish real data. <mark style="color:red;">Never submit test datasets to production</mark>.&#x20;

## **File Upload Limits**

Use Table 1 to decide which submission tool is best suited for creating your dataset. The next section summarizes the tools in more detail.

ESS-DIVE has three tools available for uploading data and each tool has a limit to the amount of data that they can upload **at one time**. Consider uploading data files in batches if the sum of the data files in your dataset is more than the given upload limits listed in Table 1.

<table><thead><tr><th width="259">Submission Tool</th><th width="131.33333333333331">Total Upload Size Limit</th><th>Why Use This Tool?</th></tr></thead><tbody><tr><td><a href="#data-submission-form">Data Submission Form</a> </td><td><strong>&#x3C; 5 GB</strong></td><td>Self-managed process using the ESS-DIVE data portal. Easiest for managing small numbers of files and datasets</td></tr><tr><td><a href="#dataset-api">ESS-DIVE's Dataset API</a></td><td><strong>&#x3C; 5 GB</strong></td><td>Self-managed process and most time efficient for uploading many data files at once (to one or multiple datasets) using programmatic tools </td></tr><tr><td><a href="#globus-data-transfer-service">Globus Transfer Service</a></td><td><strong>> 5 GB</strong></td><td>User friendly web service for automated high performance transfers, including support for hierarchical folders and very large datasets </td></tr></tbody></table>

*Table 1: ESS-DIVE's submission tools and their upload limits.* &#x20;

### Large Data Uploads

Dataset submissions with large file volumes can take additional time to upload and publish. Additionally, submission attempts will fail if your data files are greater than ESS-DIVE's file upload limits (Table 1) or greater than your available computational resources. As such, it is important to consider these factors when choosing a submission tool.&#x20;

ESS-DIVE has developed its <mark style="color:green;">**Tier 2**</mark> data storage service to support very large, hierarchical datasets that can be directly accessed from the file system layer. ESS-DIVE uses <mark style="color:green;">**Globus**</mark>, a large data transfer service, to make it easier to upload and publish large data on ESS-DIVE. The Tier 2 and Globus services are setup offline with close assistance from the ESS-DIVE Team, data contributors should get in touch with ESS-DIVE Support at <ess-dive-support@lbl.gov> to begin the process of publishing data with either service.

Visit our documentation on large data uploads for more information on ESS-DIVE's Tier 2 data storage service:

{% content-ref url="/pages/FBB6pXi8rET74QMiuPNT" %}
[Large Data Support](/contributing-data/file-upload-guidance/large-data-support)
{% endcontent-ref %}

{% hint style="warning" %} <mark style="color:orange;">**ESS-DIVE can store datasets with data volumes more than a Terabyte in size.**</mark>&#x20;

If your project needs to upload more than a Terabyte of data, ESS-DIVE will need to prepare to accommodate this. Contact <ess-dive-support@lbl.gov> to inform us of your needs.
{% endhint %}

## Submission Tools

This section gives a brief overview of the submission tools available for creating and editing datasets on ESS-DIVE.

For help troubleshooting common submission issues, visit ESS-DIVE's FAQ page:

{% content-ref url="/pages/-LgjX\_eHSOm9w3sgnPAj" %}
[Frequently Asked Questions](/faq)
{% endcontent-ref %}

### Data Submission Form

The ESS-DIVE data submission web form is the easiest way to submit small datasets. For step-by-step instructions on how to create a dataset refer to the how-to guide linked below and ESS-DIVE's [Tutorial Videos](https://ess-dive.lbl.gov/video-tutorials/). When completing the required metadata fields, use our [Dataset Requirements guide](/contributing-data/package-level-metadata), which was created using the NCEAS [FAIR data standards](https://github.com/NCEAS/metadig-checks) to ensure our repository contains high-quality and useful data.

{% content-ref url="/pages/-Lgjl-usvb9PpYlWIwmN" %}
[Submit Data with Online Form](/contributing-data/submit-data-with-online-form)
{% endcontent-ref %}

### Dataset API

ESS-DIVE's Dataset API allows you to programmatically submit many datasets at once.  Detailed tutorials and example code for dataset submissions with the API are provided both in the Dataset API guide (linked below) and [ESS-DIVE's API Examples GitHub repository](https://github.com/ess-dive/essdive-package-service-examples).  Example code is available in Python, Java, and R. Examples of the expected metadata schema are available at <https://api.ess-dive.lbl.gov/>.&#x20;

{% content-ref url="/pages/-LgikSRqc4GcZi-eCBoK" %}
[ESS-DIVE Dataset API](/programmatic-tools/ess-dive-dataset-api)
{% endcontent-ref %}

### Globus Data Transfer Service

[Globus](https://docs.globus.org/) is a cloud-based data transfer service designed to move significant amounts of data and does not require writing code. Globus can help you upload data to ESS-DIVE, **but it cannot be used to curate metadata**. You'll need to create your dataset and curate metadata via the Submission Form or the Dataset API. When using Globus, it is necessary to work with the ESS-DIVE Team to complete data file uploads.&#x20;

If you have immediate questions about uploading data using Globus OR you are encountering upload issues with data volumes less than 500GB, contact the ESS-DIVE Support Team (<ess-dive-support@lbl.gov>) to discuss your upload options and potentially using Globus.

{% content-ref url="/pages/-MX-a6sa8lrluGrgpHq6" %}
[Globus Data Transfer Service](/programmatic-tools/large-data-support)
{% endcontent-ref %}

## **Organizing Your Dataset**

Datasets on ESS-DIVE contain related data and metadata files. Each dataset should contain all the relevant data and metadata necessary for a general user to be able to understand and reuse the data.&#x20;

All data generated in the scientific process may be worth preservation, including raw data, processed data that has gone through extensive QA/QC and transformations, and results of analyses. DataONE has a good summary of best practices in determining what data to preserve ([https://www.dataone.org/best-practices/decide-what-data-preserve).](https://www.dataone.org/best-practices/decide-what-data-preserve)&#x20;

Our general recommendation is to publish data that has the greatest potential to be scientifically useful to others, and to deep archive the rest of the data for reproducibility of the results. Consider that your data may be reused in other studies, and include enough descriptive information so that others could understand your data in the future. The Digital Curation Centre (DCC) has a list of potential future purposes for data (<http://www.dcc.ac.uk/resources/how-guides/five-steps-decide-what-data-keep#4>), which also may be helpful in determining what data to publish.&#x20;

Additionally, we ask that data contributors adhere to existing data standards or data reporting formats when applicable in order to make the data stored in ESS-DIVE as useful as possible. To learn more about reporting formats, visit the following page in our guide.

{% content-ref url="/pages/-MMRTGsimSdnu3kZh6Cd" %}
[Data Reporting Formats](/contributing-data/data-reporting-formats)
{% endcontent-ref %}

## **Publication Process**

The ESS-DIVE publication process begins with a user gathering files to be included in a dataset and uploading them via the Data Submission Form or Dataset API. The user specifies metadata associated with the dataset, including author and citation information, as well as related references. The user then submits the dataset which saves the metadata and accompanying data files to ESS-DIVE as a private dataset. The dataset can be revised as frequently as needed. We recommend that you use the Automated Quality Reports to verify that your package will pass the ESS-DIVE criteria for publication.

When all the revisions are complete, you can publish the dataset. Published data are made available through the ESS-DIVE and DataONE search catalogs, can be downloaded by the public, and can be modified after publication. Publication requests need to be made by selecting the "Publish" button on the dataset landing page. After a publication request is received, the ESS-DIVE team reviews the dataset and assigns a unique Digital Object Identifier (DOI) if an existing DOI is not provided. The ESS-DIVE team will email you regarding any changes that need to be made to the dataset before publication. Publication times range from a few days to a few weeks, depending on the complexity and quality of the dataset. Incomplete submissions and slow response times will result in delayed publication.


# Register to Submit Data

This page provides instructions for requesting approval to publish data on ESS-DIVE as an ESS-DIVE data contributor.

### Quick Link: [Sign in and Register to Submit Data](https://data.ess-dive.lbl.gov/submit)

{% hint style="info" %}
ESS-DIVE uses [ORCID](https://info.orcid.org/what-is-orcid/) (Open Researcher and Contributor ID) to create ESS-DIVE accounts. **You must login to ESS-DIVE with an ORCID&#x20;**<mark style="color:orange;">**before**</mark> you submit a data contributor request.
{% endhint %}

***

## 1. Create an ORCID

Certain ORCID settings are required to contribute data to ESS-DIVE. Fill out the necessary settings marked below with an <mark style="color:orange;">\*</mark>, in addition to the fields required by ORCID.

1. Go to the ORCID registration page to sign up: <https://orcid.org/register> (Figure 1)
2. Enter a Family Name or Surname<mark style="color:orange;">\*</mark>&#x20;
3. Set your default visibility settings<mark style="color:orange;">\*</mark> to specify who can view your ORCID Record (Figure 2). You must either select:
   * Everyone, or&#x20;
   * Trusted Parties

Finish the remaining creation steps.

{% hint style="info" %}

#### Optional - Link Institutional Account

After you create an ORCID, you have the option to access your account through your institution. Linking to an institution will allow you to conveniently login with your institutional account credentials (Figure 3).
{% endhint %}

<div><figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2F6ruHxxTbFnWMV1W2KH0q%2FORCID-create-account-2024.png?alt=media&amp;token=cf5b8ac7-ae8e-4447-af62-ad348c987717" alt="" width="347"><figcaption><p>Figure 1: ORCID Registration Form, Step 1 - Names and Emails</p></figcaption></figure> <figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2F3Bkvsp4xnDakhp2Sb0dR%2FORCID-visibility-settings.png?alt=media&amp;token=ffc7a224-e2c9-4173-99f3-be1574d84b9e" alt="" width="343"><figcaption><p>Figure 2: ORCID Registration Form, Step 4 - Visibility</p></figcaption></figure> <figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FIyuJCj82YpnQYztT8KHd%2FORCID-login-institution.png?alt=media&amp;token=20ecf3e1-0669-4604-94c9-f9142692be34" alt="" width="338"><figcaption><p>Figure 3: Search for your organization name using the dropdown.</p></figcaption></figure></div>

## 2. Create an ESS-DIVE Account using ORCID

Starting an account with ESS-DIVE is easy!&#x20;

1. Go to <https://data.ess-dive.lbl.gov/data> and click on the “Sign in with Orcid” button in the top right corner (Figure 4).
2. You'll be directed to a sign in window (Figure 5). Use your ORCID to sign into ESS-DIVE.
3. A permission pop-up from ORCID will appear. Select Authorize access<mark style="color:orange;">\*</mark> (Figure 6). This will update your list of trusted organizations and allow ESS-DIVE to view your ORCID Record.

Your ESS-DIVE account will automatically be created.

<div data-full-width="false"><figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2F0OXDknesBaxZ6JUXI9Lp%2FESS-DIVE-sign-in-button.png?alt=media&amp;token=562ccccc-bb9c-4934-9d8f-3ac0665b1321" alt="" width="563"><figcaption><p>Figure 4: ESS-DIVE Main Data Search page</p></figcaption></figure> <figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FyT15Ta0UbAkGL5VmhlyR%2FESS-DIVE-orcid-sign-in-pop-up.png?alt=media&amp;token=49342530-8557-4055-a89e-3c11e00aa144" alt="" width="334"><figcaption><p><br>Figure 5: ORCID sign-in screen</p></figcaption></figure> <figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2F4wbyHEal3QyHMsEefEtv%2FORCID-authorize-access-at-login.png?alt=media&amp;token=46cfbcc4-4c20-418b-b87c-195420b96f87" alt="" width="318"><figcaption><p>Figure 6: Authorize access to trust this trusted organization</p></figcaption></figure></div>

{% hint style="warning" %}

### **Having trouble logging in?**

**Troubleshooting login issues**

If you are having issues logging in to ESS-DIVE, please review our [login troubleshooting tips](https://data.ess-dive.lbl.gov/signin-help). You may need to hard refresh the page or re-open your browser after following these tips. Contact <ess-dive-support@lbl.gov> if problems persist.

![](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FleMyEFwayirNUrGW2Vq9%2Fimage.png?alt=media\&token=6c2aea85-7d53-470a-ad04-7a0c20b267ba) ![](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FNm6IvGEejMfADsogjdxh%2FESS-DIVE-sign-in-help.png?alt=media\&token=a55c1e44-9f84-4fb0-ac48-68758b94e751)
{% endhint %}

## 3. Make sure ESS-DIVE is a Trusted Party

{% hint style="info" %}
Skip this step if your ORCID visibility settings are set to "Everyone"
{% endhint %}

**If you selected "Trusted Parties",** you will need to add ESS-DIVE to your list of trusted organizations.

1. Go to <https://orcid.org/> and sign in
2. Click on your name on the top right-hand corner of the page and navigate to "Trusted Parties" (Figure 7)
3. Make sure that "**DataONE**" and "**DataONE Identity Portal**" are listed as trusted organizations (Figure 8)

Alternatively, you can change your visibility settings to "Everyone" from your ORCID account settings at anytime (Figure 9).

<div><figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2Fn9r9VKrAfuDexpfft4GE%2FORCID-trusted-party-settings.png?alt=media&amp;token=fdbe7882-2927-4568-a8f7-b43f9307b9b0" alt=""><figcaption><p>Figure 7: Trusted Parties settings can be found under your name</p></figcaption></figure> <figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FBffSrffhZQli3OgOQWPg%2FORCID-trusted-organizations-list.png?alt=media&amp;token=71c98246-dd2c-401e-a360-0e146149e7e9" alt="" width="563"><figcaption><p>Figure 8: If you have limited visibility access to your ORCID, you must have these trusted organizations</p></figcaption></figure> <figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FuVNsQqPtX7LA5y8LsSYr%2FORCID-change-visibility-from-settings.png?alt=media&amp;token=cee30e12-5c70-43b3-8b7b-ef3eeb60781e" alt="" width="563"><figcaption><p>Figure 9: Change your visibility settings at any time</p></figcaption></figure></div>

## 4. Request Access to Submit Data

**After you have created your account with ESS-DIVE**, you can send in a request to submit data.&#x20;

1. Go to ESS-DIVE and select the green "Submit Data" button at the top right corner next to your name, or access directly from this link: <https://data.ess-dive.lbl.gov/submit>&#x20;
2. A message will appear on screen stating that you do not have permission to submit data (Figure 10). Click on the “Request Upload Access Button”.&#x20;
3. Fill out the ESS-DIVE new data contributor form to send your request (Figure 11). An ESS-DIVE admin will review your information and contact you at the email provided in the form.

   &#x20;         *Note: This email will be used for all future publication requests*

<div><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FfDKCSXa5k3dExnRFeVbG%2Frequest-upload-access.png?alt=media&amp;token=3c2c74c4-a9f8-4849-b06f-fc9d1b8c3ca7" alt="Figure 10. Select the Request Upload Access button to be directed to the registration form." width="563"> <figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FBfZZ8iNYdBLrdXCwQxEw%2FESS-DIVE-contributor-form.png?alt=media&amp;token=a742291a-e938-466e-a366-a28a593c4f1c" alt="" width="181"><figcaption><p>Figure 11. Preview of the ESS-DIVE data contributor registration form</p></figcaption></figure></div>

## 5. Wait for ESS-DIVE to Respond

Your request will be evaluated by the ESS-DIVE team. Please allow some time for this to occur.

## 6. After Approval: Update your Account Settings

Once approved, you need to add a valid email address to your account. **The Submit button will be disabled until you complete this step.**

Go to "My Settings" by clicking on the link in the email sent by the ESS-DIVE admin. Alternatively, you can log in and click on your Settings (Figure 12). Add your email address and save (Figure 13). This will enable the Submit button and allow you to upload datasets.

<div><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-Mj1PnmZZkS6B6vw6UQ2%2F-Mj1_gDSTCix4YFBPH4o%2Fmain-menu.png?alt=media&amp;token=1a61f3c6-8327-4d0d-ab0e-86c080b19992" alt="Figure 12. Click on the menu under your name to access My settings (first menu item)" width="375"> <figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2Fess-dive-docs%2F-LgjRCJ7AOWfsXbziRcZ%2F-LgjU6DbixO1-VIi2g3w%2F6.png?generation=1559869815110014&amp;alt=media" alt="" width="563"><figcaption><p>Figure 13. My account settings screen where you have to add your email and Save.</p></figcaption></figure></div>


# Data Reporting Formats

Make your data more discoverable and usable by following reporting formats applicable to your data types.

Reporting formats are data standards developed by ESS-DIVE and [partner projects](https://ess-dive.lbl.gov/partner-projects/) to standardize metadata and data files of data types commonly collected by DOE ESS projects. These reporting formats will be used to enable synthesis across datasets on ESS-DIVE. The DOE's Office of Science Biological and Environmental Research program supported the development of the data reporting formats.

**To get started**, click on a documentation link from one of the tables below according to the reporting format of interest.&#x20;

{% hint style="success" %}
If you are submitting a dataset on ESS-DIVE that follows any reporting formats, include the corresponding "Dataset Keyword" within the Keywords dataset metadata field.
{% endhint %}

Each reporting format has documentation hosted on both **GitBook** and **GitHub**. Content is exactly the same on both platforms; however, the **GitBook** interface is generally more user-friendly. Additionally, you can easily access the respective **templates** from the tables below.

{% hint style="info" %}
ESS-DIVE develops reporting formats on GitHub based on community input. Submit your comments and suggestions through ***GitHub issues**.* To view the latest status of the reporting formats, check the reporting formats page here:

<p align="center"><a href="https://ess-dive.lbl.gov/data-reporting-formats/#:~:text=data%20reporting%20format.-,Reporting%20Format,Status,-Dataset%20Metadata" class="button primary" data-icon="check-double">Check the status of the reporting formats</a></p>
{% endhint %}

## File Level Metadata

| Purpose         | Provide metadata for each data file uploaded to ESS-DIVE regardless of file type through the File Level Metadata (FLMD). Provide an explanation of headers for CSV through the Data Dictionary (DD).                                                                                                                                                                                          |
| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Documentation   | [GitHub](https://github.com/ess-dive-community/essdive-file-level-metadata) or [GitBook](https://ess-dive.gitbook.io/file-level-metadata-reporting-format/); [FLMD](https://github.com/ess-dive-community/essdive-file-level-metadata/blob/main/flmd_template.csv) and [DD](https://github.com/ess-dive-community/essdive-file-level-metadata/blob/main/CSV_dd/CSV_dd_template.csv) Templates |
| FLMD standard   | ESS-DIVE FLMD v1                                                                                                                                                                                                                                                                                                                                                                              |
| Dataset Keyword | ESS-DIVE File Level Metadata Reporting Format                                                                                                                                                                                                                                                                                                                                                 |
| Authors         | Terri Velliquette; Ranjeet Devarakonda; Jessica N. Welch; Michael Crow; Susan Heinz                                                                                                                                                                                                                                                                                                           |
| Institution     | Oak Ridge National Laboratory                                                                                                                                                                                                                                                                                                                                                                 |

## CSV File Structure

| Purpose         | Guidance for formatting CSV files submitted to ESS-DIVE                                                                                                                                                                                                         |
| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Documentation   | [GitHub](https://github.com/ess-dive-community/essdive-csv-structure) or [GitBook](https://ess-dive.gitbook.io/csv-file-structure-reporting-format/); [Guidelines](https://github.com/ess-dive-workspace/essdive-csv-structure/blob/main/csv_detailed_guide.md) |
| FLMD standard   | ESS-DIVE CSV v1                                                                                                                                                                                                                                                 |
| Dataset Keyword | ESS-DIVE CSV File Formatting Guidelines Reporting Format                                                                                                                                                                                                        |
| Authors         | Terri Velliquette; Ranjeet Devarakonda; Jessica N. Welch; Michael Crow; Susan Heinz                                                                                                                                                                             |
| Institution     | Oak Ridge National Laboratory                                                                                                                                                                                                                                   |

## Sample ID Metadata

<table data-header-hidden data-search="false"><thead><tr><th>Sample ID Metadata</th><th>Status: READY TO USE</th></tr></thead><tbody><tr><td>Purpose</td><td>Guidelines for assigning biological and environmental sample identifiers, describing sample collection details, and linking samples to related data in ESS-DIVE and elsewhere</td></tr><tr><td>Documentation</td><td><a href="https://github.com/ess-dive-community/essdive-sample-id-metadata">GitHub</a> or <a href="https://ess-dive.gitbook.io/sample-id-and-metadata/">GitBook</a>; <a href="https://github.com/ess-dive-community/essdive-sample-id-metadata/blob/master/sampleTemplate.xls">Template</a></td></tr><tr><td>FLMD standard</td><td>ESS-DIVE Sample v1</td></tr><tr><td>Dataset Keyword</td><td>ESS-DIVE Sample ID and Metadata Reporting Format</td></tr><tr><td>Authors</td><td>Joan Damerow</td></tr><tr><td>Institution</td><td>Lawrence Berkeley National Laboratory</td></tr></tbody></table>

## Soil Respiration

| Purpose         | Guidance for structuring data and metadata for soil respiration observations                                                                                                                                                                           |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Documentation   | [GitHub](https://github.com/ess-dive-community/essdive-soil-respiration) or [GitBook](https://ess-dive.gitbook.io/soil-respiration-reporting-format/); [Templates](https://github.com/ess-dive-community/essdive-soil-respiration/tree/main/templates) |
| FLMD standard   | ESS-DIVE Soil Respiration v1                                                                                                                                                                                                                           |
| Dataset Keyword | ESS-DIVE Soil Respiration Reporting Format                                                                                                                                                                                                             |
| Authors         | Ben Bond-Lamberty and Stephanie Pennington                                                                                                                                                                                                             |
| Institution     | Pacific Northwest National Laboratory                                                                                                                                                                                                                  |

## Leaf-level Gas Exchange

| Purpose         | Guidance for leaf-level gas exchange data and metadata                                                                                                                                                                                           |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Documentation   | [GitHub](https://github.com/ess-dive-community/essdive-leaf-gas-exchange) or [GitBook](https://ess-dive.gitbook.io/leaf-level-gas-exchange/); [Templates](https://github.com/ess-dive-community/essdive-leaf-gas-exchange/tree/master/templates) |
| FLMD standard   | ESS-DIVE Leaf-Level Gas Exchange v1                                                                                                                                                                                                              |
| Dataset Keyword | ESS-DIVE Leaf-Level Gas Exchange Reporting Format                                                                                                                                                                                                |
| Authors         | Kim Ely and Alistair Rogers                                                                                                                                                                                                                      |
| Institution     | Brookhaven National Laboratory                                                                                                                                                                                                                   |

## Hydrologic Monitoring

<table data-header-hidden data-search="false"><thead><tr><th>Hydrologic Monitoring</th><th>Status: READY TO USE</th></tr></thead><tbody><tr><td>Purpose</td><td>Guidance for reporting metrics such as water level, temperature, conductivity and more.</td></tr><tr><td>Documentation</td><td><a href="https://github.com/ess-dive-community/essdive-hydrologic-monitoring">GitHub</a> or <a href="https://ess-dive.gitbook.io/hydrologic-monitoring-data-and-metadata/">GitBook</a>; <a href="https://github.com/ess-dive-community/essdive-hydrologic-monitoring/tree/main/templates">Templates</a></td></tr><tr><td>FLMD standard</td><td>ESS-DIVE Hydrologic Monitoring v1</td></tr><tr><td>Dataset Keyword</td><td>ESS-DIVE Hydrologic Monitoring Reporting Format</td></tr><tr><td>Authors</td><td>Amy Goldman</td></tr><tr><td>Institution</td><td>Pacific Northwest National Laboratory</td></tr></tbody></table>

## Water and Soil Chemistry

| Purpose         | Guidance to format chemical concentration data for water, soil and sediment                                                                                                                                                                                               |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Documentation   | [GitHub](https://github.com/ess-dive-community/essdive-water-soil-sed-chem) or [GitBook](https://ess-dive.gitbook.io/water-soil-sediment-chemistry-reporting-format/); [Templates](https://github.com/ess-dive-community/essdive-water-soil-sed-chem/tree/main/templates) |
| FLMD standard   | ESS-DIVE Water-Soil-Sediment Chem v1                                                                                                                                                                                                                                      |
| Dataset Keyword | ESS-DIVE Water-Soil-Sediment Chemistry Reporting Format                                                                                                                                                                                                                   |
| Authors         | Kristin Boye                                                                                                                                                                                                                                                              |
| Institution     | SLAC National Accelerator Lab                                                                                                                                                                                                                                             |

## 16S Amplicon Sequencing

| Purpose         | Guidance for scientists producing 16S abundance data products                                                                                                                                                                                      |
| --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Documentation   | [GitHub](https://github.com/ess-dive-community/essdive-amplicon) or [GitBook](https://ess-dive.gitbook.io/amplicon-sequencing-reporting-format/); [Templates](https://github.com/ess-dive-community/essdive-amplicon/tree/main/templates_and_maps) |
| FLMD standard   | ESS-DIVE Amplicon v1                                                                                                                                                                                                                               |
| Dataset Keyword | ESS-DIVE Amplicon Sequencing Reporting Format                                                                                                                                                                                                      |
| Authors         | Pamela Weisenhorn                                                                                                                                                                                                                                  |
| Institution     | Argonne National Lab                                                                                                                                                                                                                               |

## Model Data Archiving

| Purpose         | Guidance to structure model data submissions                                                                                                                       |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Documentation   | [GitHub](https://github.com/ess-dive-community/essdive-model-data-archiving-guidelines) or [GitBook](https://ess-dive.gitbook.io/model-data-archiving-guidelines/) |
| FLMD standard   | ESS-DIVE Model Data v1                                                                                                                                             |
| Dataset Keyword | ESS-DIVE Model Data Archiving Guidelines                                                                                                                           |
| Authors         | Maegen Simmons                                                                                                                                                     |
| Institution     | Lawrence Berkeley National Lab                                                                                                                                     |

## Location metadata

<table data-header-hidden data-search="false"><thead><tr><th></th><th width="290"></th></tr></thead><tbody><tr><td>Purpose</td><td>Guidance for providing a minimal set of metadata about locations</td></tr><tr><td>Documentation</td><td><a href="https://github.com/ess-dive-workspace/essdive-location-metadata">GitHub</a> or <a href="https://ess-dive.gitbook.io/location-metadata/">GitBook</a>; <a href="https://github.com/ess-dive-workspace/essdive-location-metadata/tree/main/templates">Templates</a></td></tr><tr><td>FLMD Standard Term</td><td>ESS-DIVE Location v1</td></tr><tr><td>Dataset Keyword</td><td>ESS-DIVE Location Metadata Reporting Format</td></tr><tr><td>Authors</td><td>Rob Crystal-Ornelas</td></tr><tr><td>Institution</td><td>Lawrence Berkeley National Lab</td></tr></tbody></table>

## Unoccupied Aerial System data

<table data-header-hidden><thead><tr><th width="344.70660400390625"></th><th></th></tr></thead><tbody><tr><td>Purpose</td><td>Metadata content and data organization recommendations for UAS</td></tr><tr><td>Documentation</td><td><a href="https://github.com/ess-dive-community/essdive-uas">GitHub</a> or <a href="https://ess-dive.gitbook.io/uas-reporting-format">GitBook</a>; <a href="https://github.com/ess-dive-workspace/essdive-uas/tree/main/templates">Templates</a></td></tr><tr><td>FLMD standard</td><td>ESS-DIVE UAS v1</td></tr><tr><td>Dataset Keyword</td><td>ESS-DIVE Unoccupied Aerial Systems (UAS) Reporting Format</td></tr><tr><td>Authors</td><td>Kim Ely and Shawn Serbin</td></tr><tr><td>Institution</td><td>Brookhaven National Lab</td></tr></tbody></table>


# Metadata Requirements

This page lists all ESS-DIVE dataset metadata fields and their publication requirements. Datasets are reviewed by the ESS-DIVE team before approval for publication using this criteria.

ESS-DIVE’s dataset metadata requirements allow you to fully describe your dataset to FAIR standards so that others can more easily find and use relevant data from dataset searches. Metadata for each dataset submitted should meet the stated guidelines and requirements in the tables below. Metadata completeness will be assessed during the dataset publication process using both **automated** and **manual** review workflows.

<i class="fa-seal-exclamation">:seal-exclamation:</i> Ensuring that your dataset has complete metadata before requesting publication will expedite the publication process.

#### Notice: Controlled and Prohibited Information

ESS-DIVE will not accept any datasets that include [controlled and prohibited information categories](https://commons.lbl.gov/display/rpm2/Controlled+and+Prohibited+Information+Categories) (section D.2), such as Protected Information (i.e., Personally Identifiable Information \[PII] and Protected Health Information \[PHI]).&#x20;

<details>

<summary>Examples of Protected Information (PII and PHI)</summary>

* Social Security numbers
* Driver's license numbers
* Health information with personal identifiers
* The individual's past, present, or future physical or mental health or condition

See [Section F. Definitions/Acronyms](https://commons.lbl.gov/display/rpm2/Controlled+and+Prohibited+Information+Categories#ControlledandProhibitedInformationCategories--1898802862) for a detailed list of examples.&#x20;

</details>

{% hint style="info" %}
**File Publication Requirements**

There are a few general file requirements that the ESS-DIVE Team checks during the publication review process.<a href="/contributing-data/file-upload-guidance#file-publication-requirements" class="button primary medium" data-icon="arrow-right">Click here to see file requirements</a>
{% endhint %}

{% columns %}
{% column %}

#### Submit

*<mark style="color:red;">\*</mark> fields required to create a dataset*

Create and save a new private dataset. Edit and collaborate with other contributors.&#x20;
{% endcolumn %}

{% column %}

#### Publish

*<mark style="color:red;">†</mark> fields* *required to publish a dataset*

The dataset is complete and ready for review and public release.

This requires more fields to be completed.
{% endcolumn %}
{% endcolumns %}

| <p><a href="/contributing-data/package-level-metadata#overview"><strong>Overview</strong></a></p><ul><li><a href="/contributing-data/package-level-metadata#title">Title</a><mark style="color:red;">*†</mark></li><li><a href="#alternate-identifier">Alternate Identifiers</a></li><li><a href="/contributing-data/package-level-metadata#abstract">Abstract </a><mark style="color:red;">*†</mark></li><li><a href="/contributing-data/package-level-metadata#keywords">Keywords</a><mark style="color:red;">*†</mark></li><li><a href="/contributing-data/package-level-metadata#data-variables">Data Variables</a></li><li><a href="/contributing-data/package-level-metadata#publication-date">Publication Date</a><mark style="color:red;">†</mark></li><li><a href="/contributing-data/package-level-metadata#usage-rights">Usage Rights</a><mark style="color:red;">*†</mark></li><li><a href="/contributing-data/package-level-metadata#project">Project</a><mark style="color:red;">*†</mark></li><li><a href="/contributing-data/package-level-metadata#funding-organization">Funding Organization</a><mark style="color:red;">*†</mark></li><li><a href="/contributing-data/package-level-metadata#doe-contracts">DOE Contracts</a></li><li><a href="/contributing-data/package-level-metadata#related-references">Related References</a></li></ul> | <p></p><p><a href="/contributing-data/package-level-metadata#people"><strong>People</strong></a></p><ul><li><a href="/contributing-data/package-level-metadata#contact">Contact</a><mark style="color:red;">*†</mark></li><li><a href="/contributing-data/package-level-metadata#creators">Creators</a><mark style="color:red;">*†</mark></li><li><a href="/contributing-data/package-level-metadata#contributors">Contributors</a></li></ul><p><a href="/contributing-data/package-level-metadata#dates"><strong>Dates</strong></a></p><ul><li><a href="/contributing-data/package-level-metadata#start-date">Start Date</a><mark style="color:red;">†</mark></li><li><a href="/contributing-data/package-level-metadata#end-date">End Date</a></li></ul><p><a href="/contributing-data/package-level-metadata#locations"><strong>Locations</strong></a></p><ul><li><a href="/contributing-data/package-level-metadata#geographic-description">Geographic Description</a><mark style="color:red;">†</mark></li><li><a href="/contributing-data/package-level-metadata#bounding-box-coordinates">Bounding Box Coordinates</a><mark style="color:red;">†</mark></li></ul><p><a href="/contributing-data/package-level-metadata#methods"><strong>Methods</strong></a><mark style="color:red;">†</mark></p><p></p> |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

<details>

<summary>Automated Checks</summary>

A set of **automated checks** are performed whenever a dataset is submitted. The results of these checks are compiled into an [**Assessment Report**](/publish-data/check-dataset-metadata-quality) and used in the review process. Failed automated checks or warnings should be addressed by the dataset submitter before requesting publication.&#x20;

Please note that assessment reports can take minutes, or up to 24 hours, to generate.&#x20;

</details>

<details>

<summary>JSON-LD Fields</summary>

Datasets that are created or edited using the [Dataset API](/programmatic-tools/ess-dive-dataset-api) must use the [JSON-LD](https://json-ld.org/) schema. The JSON-LD rows indicates what each metadata field looks like in the JSON-LD schema.&#x20;

</details>

## Overview

### Title<mark style="color:red;">\*</mark><mark style="color:red;">†</mark>

<table data-header-hidden><thead><tr><th width="192"></th><th></th></tr></thead><tbody><tr><td><strong>Format</strong></td><td>Free Text</td></tr><tr><td><strong>JSON-LD Field</strong></td><td><code>name</code></td></tr><tr><td><strong>Description</strong></td><td>A brief but meaningful title for the data package.</td></tr><tr><td><strong>Guidelines</strong></td><td><ul><li>Title is between 7-40 words. Recommended to keep below 20 unless absolutely necessary. <mark style="color:$info;">(Included in</mark> <a href="/publish-data/check-dataset-metadata-quality#automated-checks"><mark style="color:$info;">Automated Checks</mark></a><mark style="color:$info;">)</mark></li><li>Does not contain any acronyms that are not explained elsewhere (eg. in the abstract).</li><li>Includes data's geographic location.</li><li>Describes data's time frame.</li></ul></td></tr><tr><td><strong>Example</strong></td><td>Raw sapflow and soil moisture data from January 2016-April 2016 in Manaus, Brazil</td></tr><tr><td><strong>Requirement Level</strong></td><td>Needed for both submission and publication.</td></tr></tbody></table>

### Alternate Identifier

<table data-header-hidden><thead><tr><th width="175"></th><th></th></tr></thead><tbody><tr><td><strong>Format</strong></td><td>Free Text</td></tr><tr><td><strong>JSON-LD Field</strong></td><td><code>alternateName</code></td></tr><tr><td><strong>Description</strong></td><td>If this dataset has been previously published elsewhere, enter the DOI or alternate identifier. Identifiers are used to locate the dataset within your project's data management system and can provide pertinent contextual information for users. </td></tr><tr><td><strong>Guidelines</strong></td><td><ul><li>Any DOIs should direct to dataset and not manuscripts. Manuscripts should be put in Related References.</li><li>Ensure the identifier correctly leads to the dataset that you are submitting</li></ul></td></tr><tr><td><strong>Example</strong></td><td>spruce.055</td></tr><tr><td><strong>Requirement Level</strong></td><td>Optional</td></tr></tbody></table>

### Abstract<mark style="color:red;">\*</mark><mark style="color:red;">†</mark>

<table data-header-hidden><thead><tr><th width="160"></th><th width="566"></th></tr></thead><tbody><tr><td><strong>Format</strong></td><td>Free Text</td></tr><tr><td><strong>JSON-LD Field</strong></td><td><code>description</code></td></tr><tr><td><strong>Description</strong></td><td>A concise description of the purpose and content of the data package.</td></tr><tr><td><strong>Guidelines</strong></td><td><ul><li>More than 100 words. <mark style="color:$info;">(Included in</mark> <a href="/publish-data/check-dataset-metadata-quality#automated-checks"><mark style="color:$info;">Automated Checks</mark></a><mark style="color:$info;">)</mark></li><li>Clear and concise, written in complete sentences.</li><li><sup><strong>1</strong></sup>Describe the content of the dataset, and provide all necessary scientific context. Includes adequate detail so that people who have not written related manuscripts could still understand the dataset.</li><li><sup><strong>2</strong></sup>Includes statement about the purpose of the data's generation.</li><li>Includes definitions of acronyms.</li><li><sup><strong>3</strong></sup>Describes the files uploaded, including source of data, software needed to view the files, location descriptions, ecosystem type involved, measurement type, etc.</li><li>Any mentioned publications are cited in <a href="#related-references">Related References</a></li></ul></td></tr><tr><td><strong>Example</strong></td><td>The superscripts located throughout the <a href="#abstract-example">example below</a> demonstrates successful application of applicable criteria listed in the guidelines</td></tr><tr><td><strong>Requirement Level</strong></td><td>Required for submission and publication.</td></tr></tbody></table>

<details>

<summary>Abstract Example</summary>

> This data package contains raw data, data output, and code of photosynthetic temperature response curves from tropical forests<sup>**1**</sup>. Data are used in "Photosynthetic responses to temperature across the tropics: a meta-analytic approach". The research investigates how photosynthetic optimum temperatures and shapes of photosynthetic temperature response curves varies across tropical forest climates and growth conditions<sup>**2**</sup>. Growth climate variables include mean annual temperature, max and max temperature, diurnal temperature range, and aridity index. Meta-analysis combines 18 datasets with representation from Africa, Oceana, North American, and South America. Growth conditions analyzed considers deciduousness, successional status, light conditions, and whether plants are grown in situ or ex situ. All files, except for the raw datafile (Tropical\_MetaAnalysis\_Master\_3.8.22\_Edit.csv), have been processed using R code, which is provided<sup>**3**</sup>.

Carter K ; Cavaleri M ; Atkin O ; Bloomfield K ; Bahar N ; Cheesman A ; Choury Z ; Crous K ; Doughty C ; Dusenge M ; Ely K ; Evans J ; Fonseca da Silva J ; Mau A ; Meir P ; Medlyn B ; Norby R ; Read J ; Reed S ; Reich P ; Rogers A ; Schwartz E ; Serbin S ; Slot M ; Tribuzy E ; Uddling J ; Vårhammar A ; Walker A ; Winter K ; Wood T ; Wu J (2025): Data for Photosynthetic responses to temperature across the tropics: a meta-analytic approach. Effects of Hurricane <mark style="color:$info;">Disturbance and Increased Temperature on Carbon Cycling and Storage of a Puerto Rican Forest: A Mechanistic Investigation of Above- and Belowground Processes, ESS-DIVE repository. Dataset. doi:10.15485/2513897 accessed via <https://data.ess-dive.lbl.gov/datasets/doi:10.15485/2513897> on 2026-06-15</mark>

</details>

### Keywords<mark style="color:red;">\*</mark><mark style="color:red;">†</mark>

<table data-header-hidden><thead><tr><th width="182.8355712890625"></th><th></th></tr></thead><tbody><tr><td><strong>Format</strong></td><td><a href="https://gcmd.earthdata.nasa.gov/KeywordViewer/">GCMD Keywords</a>, <a href="http://cfconventions.org/Data/cf-standard-names/48/src/cf-standard-name-table.xml">CF Variables</a>, OR Free Text</td></tr><tr><td><strong>JSON-LD Field</strong></td><td><code>keywords</code></td></tr><tr><td><strong>Description</strong></td><td>Keywords should be associated with its data package to enable thematic searches. As you begin typing in the web form field, GCMD controlled vocabulary terms will appear in a dropdown list. Selecting from the GCMD controlled keywords where possible is encouraged but not required. You can also enter your own keywords. </td></tr><tr><td><strong>Guidelines</strong></td><td><ul><li>At least three keywords or variables that improve findability <mark style="color:$info;">(Included in</mark> <a href="/publish-data/check-dataset-metadata-quality#automated-checks"><mark style="color:$info;">Automated Checks</mark></a><mark style="color:$info;">)</mark></li><li>Keywords are different from the terms in the dataset title <mark style="color:$info;">(Included in</mark> <a href="/publish-data/check-dataset-metadata-quality#automated-checks"><mark style="color:$info;">Automated Checks</mark></a><mark style="color:$info;">)</mark></li></ul></td></tr><tr><td><strong>Example</strong></td><td><ul><li>Earth Science</li><li>Land Surface</li><li>EARTH SCIENCE > BIOSPHERE > ECOLOGICAL DYNAMICS > ECOSYSTEM FUNCTIONS > RESPIRATION RATE</li><li>EARTH SCIENCE > AGRICULTURE > SOILS > SOIL DEPTH</li></ul></td></tr><tr><td><strong>Requirement Level</strong></td><td>Needed for both submission and publication.</td></tr></tbody></table>

### Data Variables

<table data-header-hidden><thead><tr><th width="171.77801513671875"></th><th></th></tr></thead><tbody><tr><td><strong>Format</strong></td><td><a href="https://gcmd.earthdata.nasa.gov/KeywordViewer/">GCMD Keywords</a> or Free Text</td></tr><tr><td><strong>JSON-LD Field</strong></td><td><code>variableMeasured</code></td></tr><tr><td><strong>Description</strong></td><td>Add variables present in the data package to increase the findability of your dataset in searches. Similarly to the keywords field, selecting variable terms from GCMD controlled vocabulary where possible is encouraged but not required.</td></tr><tr><td><strong>Guidelines</strong></td><td><ul><li>At least three keywords or variables that improve findability <mark style="color:$info;">(Included in</mark> <a href="/publish-data/check-dataset-metadata-quality#automated-checks"><mark style="color:$info;">Automated Checks</mark></a><mark style="color:$info;">)</mark></li><li>Variables are different from the terms in the dataset title</li></ul></td></tr><tr><td><strong>Example</strong></td><td><ul><li>Soil Moisture</li><li>EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC TEMPERATURE > SURFACE TEMPERATURE</li></ul></td></tr><tr><td><strong>Requirement Level</strong></td><td>Optional, combined with Keywords</td></tr></tbody></table>

### Publication Date<mark style="color:red;">†</mark>

<table data-header-hidden><thead><tr><th width="200.005859375"></th><th></th></tr></thead><tbody><tr><td><strong>Format</strong></td><td><em>YYYY or YYYY-MM-DD</em></td></tr><tr><td><strong>JSON-LD Field</strong></td><td><code>datePublished</code></td></tr><tr><td><strong>Description</strong></td><td><p>The year or date that the dataset is published. If this is not specified, it will default to the current date. This metadata field does not influence the actual timeline of publication.</p><p></p><p>If your dataset has been published elsewhere, specify the date or year it was originally published. </p></td></tr><tr><td><strong>Guidelines</strong></td><td><ul><li>Publication date included <mark style="color:$info;">(Included in</mark> <a href="/publish-data/check-dataset-metadata-quality#automated-checks"><mark style="color:$info;">Automated Checks</mark></a><mark style="color:$info;">)</mark></li><li>Date reflects actual year of publication</li><li>For existing publications, matches original publication date</li></ul></td></tr><tr><td><strong>Example</strong></td><td><ul><li>2019 </li><li>2019-04-19</li></ul></td></tr><tr><td><strong>Requirement Level</strong></td><td>Required for publication</td></tr></tbody></table>

### Usage Rights<mark style="color:red;">\*</mark><mark style="color:red;">†</mark>

<table data-header-hidden><thead><tr><th width="187.93756103515625"></th><th></th></tr></thead><tbody><tr><td><strong>Format</strong></td><td>Controlled list</td></tr><tr><td><strong>JSON-LD Field</strong></td><td><code>license</code></td></tr><tr><td><strong>Description</strong></td><td>Choose how you wish your data to be shared and reused. Creative Commons Attribution (CC BY 4.0) requires that the dataset be cited by anyone using the data. Creative Commons Public Domain (CC BY 1.0) dedicates the data to the public domain without restriction. When using the API, enter the URL for the selected CC BY license.</td></tr><tr><td><strong>Guidelines</strong></td><td><ul><li>Select one of the two options</li><li>Usage rights is set to Creative Commons CC-BY license by default <mark style="color:$info;">(Included in</mark> <a href="/publish-data/check-dataset-metadata-quality#automated-checks"><mark style="color:$info;">Automated Checks</mark></a><mark style="color:$info;">)</mark></li></ul></td></tr><tr><td><strong>Example</strong></td><td><ul><li>Creative Commons Attribution (CC BY 4.0)</li><li>Creative Commons Public Domain (CC BY 1.0)</li></ul></td></tr><tr><td><strong>Requirement Level</strong></td><td>Required for submission and publication</td></tr></tbody></table>

### Project<mark style="color:red;">\*</mark><mark style="color:red;">†</mark>

#### Online Form only

<table data-header-hidden><thead><tr><th width="200.9091796875"></th><th></th></tr></thead><tbody><tr><td><strong>Format</strong></td><td>Controlled List</td></tr><tr><td><strong>JSON-LD Field</strong></td><td><code>provider</code></td></tr><tr><td><strong>Description</strong></td><td>The DOE project that funded the generation of this data. Only one project is allowed. If multiple projects were involved, enter the project that had the largest contribution to this dataset. </td></tr><tr><td><strong>Guidelines</strong> </td><td><ul><li>Select the DOE project name from the drop down list, which will appear when you start typing in the project name or Principal Investigator (PI) name.</li><li>The project name in within ESS-DIVE's controlled <a href="https://data.ess-dive.lbl.gov/projects">list</a> <mark style="color:$info;">(Included in</mark> <a href="/publish-data/check-dataset-metadata-quality#automated-checks"><mark style="color:$info;">Automated Checks</mark></a><mark style="color:$info;">)</mark></li></ul></td></tr><tr><td><strong>Example</strong></td><td>Soil Carbon Biogeochemistry</td></tr><tr><td><strong>Requirement Level</strong></td><td>Required for submission and publication</td></tr></tbody></table>

#### API Only

<table data-header-hidden><thead><tr><th width="200.6190185546875"></th><th></th></tr></thead><tbody><tr><td><strong>Format</strong></td><td>Value</td></tr><tr><td><strong>Description</strong></td><td>Enter the project ID into the JSON-LD field. Written project names will not be accepted. Look up your project ID using <a href="https://docs.google.com/spreadsheets/d/179SOyv42wXbP4owWZtUg3RqhW9dPOyENYcVYuUCcqwg/edit?usp=sharing">ESS-DIVE's Project List</a>. If multiple projects were involved, enter the project that had the largest contribution to this dataset. </td></tr><tr><td><strong>Example</strong></td><td><pre><code>1e6d50d3-9532-43fb-a63f-bdcb4350bf0c
</code></pre></td></tr><tr><td><strong>JSON-LD Field</strong></td><td><pre class="language-perl"><code class="lang-perl">provider = {
   "identifier": {
      "@type": "PropertyValue",
      "propertyID": "ess-dive",
      "value": "&#x3C;Project ID>"
}
</code></pre></td></tr></tbody></table>

### Funding Organization<mark style="color:red;">\*</mark><mark style="color:red;">†</mark>

<table data-header-hidden><thead><tr><th width="202.33892822265625"></th><th></th></tr></thead><tbody><tr><td><strong>Format</strong></td><td>Controlled List or Free Text</td></tr><tr><td><strong>JSON-LD Field</strong></td><td><code>funder</code></td></tr><tr><td><strong>Description</strong></td><td>List the organizations that funded the work. </td></tr><tr><td><strong>Guidelines</strong></td><td><ul><li>When using the web form, you can choose from the drop down list as you begin to enter the funding organization.</li><li>Funding organization "U.S. DOE > Office of Science > Biological and Environmental Research (BER)" is present, if applicable <mark style="color:$info;">(Included in</mark> <a href="/publish-data/check-dataset-metadata-quality#automated-checks"><mark style="color:$info;">Automated Checks</mark></a><mark style="color:$info;">)</mark></li></ul></td></tr><tr><td><strong>Example</strong></td><td>U.S. DOE > Office of Science > Biological and Environmental Research (BER) </td></tr><tr><td><strong>Requirement Level</strong></td><td>Required for submission and publication</td></tr></tbody></table>

### DOE Contracts

<table data-header-hidden><thead><tr><th width="185.5546875"></th><th></th></tr></thead><tbody><tr><td><strong>Format</strong></td><td>Controlled List or Free Text</td></tr><tr><td><strong>JSON-LD Field</strong></td><td><code>award</code></td></tr><tr><td><strong>Description</strong></td><td>List the numbers of any DOE contract under which the data in the package was funded. </td></tr><tr><td><strong>Guidelines</strong></td><td><ul><li>Select the relevant DOE contract from the drop down list, which will appear when you start typing in the contract number</li><li>Or write in the applicable DOE contract number</li><li>If the dataset is a result of a joint effort between two or more DOE Site/Facility Management Contractors, etc., additional DOE contract numbers may be entered.</li></ul></td></tr><tr><td><strong>Example</strong></td><td>AC0205CH11231</td></tr><tr><td><strong>Requirement Level</strong></td><td>Optional</td></tr></tbody></table>

### Related References

<table data-header-hidden><thead><tr><th width="165.39874267578125"></th><th></th></tr></thead><tbody><tr><td><strong>Format</strong></td><td>Free Text</td></tr><tr><td><strong>JSON-LD Field</strong></td><td><code>citation</code></td></tr><tr><td><strong>Description</strong></td><td>Include the full citations and DOIs of datasets or publications associated with your dataset and those that are directly mentioned in the metadata (such as in the Abstract or Methods). These related materials allow users to learn more about the dataset, processing methods, or how the data were used.</td></tr><tr><td><strong>Guidelines</strong></td><td><ul><li>Citations to related manuscripts located anywhere in the metadata should be entered in this field</li><li>DOIs should resolve</li><li>Leave placeholders for incomplete manuscripts and ensure they are updated later</li></ul></td></tr><tr><td><strong>Example</strong></td><td>Crystal-Ornelas, R., Varadharajan, C., O’Ryan, D. <em>et al.</em> Enabling FAIR data in Earth and environmental science with community-centric (meta)data reporting formats. <em>Sci Data</em> 9, 700 (2022). <a href="https://doi.org/10.1038/s41597-022-01606-w">https://doi.org/10.1038/s41597-022-01606-w</a></td></tr><tr><td><strong>Requirement Level</strong></td><td>Optional</td></tr></tbody></table>

## People

### Contact<mark style="color:red;">\*</mark><mark style="color:red;">†</mark>

<table data-header-hidden><thead><tr><th width="159.47369384765625"></th><th></th></tr></thead><tbody><tr><td><strong>Format</strong></td><td>Free Text</td></tr><tr><td><strong>JSON-LD Field</strong></td><td><code>editor</code></td></tr><tr><td><strong>Description</strong></td><td><p>List the singular person who should be contacted by users seeking further information for the data. Only one contact is allowed. </p><p>If no contact is provided, the person creating the dataset will be listed as the contact by default.</p></td></tr><tr><td><strong>Guidelines</strong></td><td><ul><li>Contact is present <mark style="color:$info;">(Included in</mark> <a href="/publish-data/check-dataset-metadata-quality#automated-checks"><mark style="color:$info;">Automated Checks</mark></a><mark style="color:$info;">)</mark></li><li>Contact ORCID and email are required <mark style="color:$info;">(Included in</mark> <a href="/publish-data/check-dataset-metadata-quality#automated-checks"><mark style="color:$info;">Automated Checks</mark></a><mark style="color:$info;">)</mark></li></ul></td></tr><tr><td><strong>Example</strong></td><td>John, Doe, Lawrence Berkeley National Laboratory, jdoe@lbl.gov, https://orcid.org/0000-0000-0000-0000</td></tr><tr><td><strong>Requirement Level</strong></td><td>Required for submission and publication</td></tr></tbody></table>

### Creators<mark style="color:red;">\*</mark><mark style="color:red;">†</mark>

<table data-header-hidden><thead><tr><th width="196.48095703125"></th><th></th></tr></thead><tbody><tr><td><strong>Format</strong></td><td>Free Text</td></tr><tr><td><strong>JSON-LD Field</strong></td><td><code>creator</code></td></tr><tr><td><strong>Description</strong></td><td>Include the main researchers involved in producing the data such as authors, owners, originators, and principal investigators.</td></tr><tr><td><strong>Guidelines</strong></td><td><ul><li>Include all researchers involved with the data production  </li><li>Researchers are listed in the order they appear in the citation</li><li>Author emails are required </li><li>Author ORCIDs are highly encouraged, but not required</li></ul></td></tr><tr><td><strong>Example</strong></td><td>John, Doe, Lawrence Berkeley National Laboratory, jdoe@lbl.gov, https://orcid.org/0000-0000-0000-0000</td></tr><tr><td><strong>Requirement Level</strong></td><td>Required for submission and publication</td></tr></tbody></table>

### Contributors

<table data-header-hidden><thead><tr><th width="162.58221435546875"></th><th></th></tr></thead><tbody><tr><td><strong>Format</strong></td><td>Free Text</td></tr><tr><td><strong>JSON-LD Field</strong></td><td><code>contributor</code></td></tr><tr><td><strong>Description</strong></td><td><p>List any additional contributors involved in producing the data. These may include people who assisted in creating the dataset but are not considered authors. </p><p><br>Contributors will <strong>not</strong> appear in the data citation.</p></td></tr><tr><td><strong>Guidelines</strong></td><td><ul><li>Contributor emails are required </li><li>Contributor ORCIDs are highly encouraged, but not required</li></ul></td></tr><tr><td><strong>Example</strong></td><td>John, Doe, Lawrence Berkeley National Laboratory, jdoe@lbl.gov, https://orcid.org/0000-0000-0000-0000</td></tr><tr><td><strong>Requirement Level</strong></td><td>Optional</td></tr></tbody></table>

## Dates

### Start Date<mark style="color:red;">†</mark>

<table data-header-hidden><thead><tr><th width="218.06707763671875"></th><th></th></tr></thead><tbody><tr><td><strong>Format</strong></td><td>YYYY or YYYY-MM-DD</td></tr><tr><td><strong>JSON-LD Field</strong></td><td><code>temporalCoverage</code></td></tr><tr><td><strong>Description</strong></td><td>Earliest date of data collection included in the dataset.</td></tr><tr><td><strong>Guidelines</strong></td><td><ul><li>Start date is present <mark style="color:$info;">(Included in</mark> <a href="/publish-data/check-dataset-metadata-quality#automated-checks"><mark style="color:$info;">Automated Checks</mark></a><mark style="color:$info;">)</mark></li></ul></td></tr><tr><td><strong>Example</strong></td><td>2017-04-16</td></tr><tr><td><strong>Requirement Level</strong></td><td>Required for publication</td></tr></tbody></table>

### End Date

<table data-header-hidden><thead><tr><th width="211.0771484375"></th><th></th></tr></thead><tbody><tr><td><strong>Format</strong></td><td>YYYY or YYYY-MM-DD</td></tr><tr><td><strong>JSON-LD Field</strong></td><td><code>temporalCoverage</code></td></tr><tr><td><strong>Description</strong></td><td>Last date of data collection included in the dataset. </td></tr><tr><td><strong>Guidelines</strong></td><td><ul><li>End date is present <mark style="color:$info;">(Included in</mark> <a href="/publish-data/check-dataset-metadata-quality#automated-checks"><mark style="color:$info;">Automated Checks</mark></a><mark style="color:$info;">)</mark></li><li>This field can be left blank if your dataset is open ended</li></ul></td></tr><tr><td><strong>Example</strong></td><td>2019-07-13</td></tr><tr><td><strong>Requirement Level</strong></td><td>Optional, strongly recommended if relevant</td></tr></tbody></table>

## Locations

### Geographic Description<mark style="color:red;">†</mark>

<table data-header-hidden data-search="false"><thead><tr><th width="174.1577911376953"></th><th></th></tr></thead><tbody><tr><td><strong>Format</strong></td><td>Free Text</td></tr><tr><td><strong>JSON-LD Field</strong></td><td><code>spatialCoverage/description</code></td></tr><tr><td><strong>Description</strong></td><td>A short description of the location(s) where data was collected. A complete geographic description will increase the findability of your dataset, as all terms entered are searchable through the data portal.</td></tr><tr><td><strong>Guidelines</strong></td><td><ul><li>This may include the location name, known identifiers if associated with a specific project (e.g. Ameriflux site name), and ecosystem type involved. </li><li>Multiple geographic descriptions can be added if necessary.</li></ul></td></tr><tr><td><strong>Example</strong></td><td>Br-Ma2, Manaus, Brazil: ZF2 K34 Tower. Eddy covariance site established in 1999 on kilometer 34 of the ZF2 highway. It was later expanded into an atmospheric and soil sampling hub. It is a 1.5m x 2.5 m- section aluminum tower, 50 m tall, on a medium-sized plateau (Araujo et al., 2002).</td></tr><tr><td><strong>Requirement Level</strong></td><td>Required for publication</td></tr></tbody></table>

### Bounding Box Coordinates<mark style="color:red;">†</mark>

<table data-header-hidden><thead><tr><th width="172.44630432128906"></th><th></th></tr></thead><tbody><tr><td><strong>Format</strong></td><td>Latitude and Longitude in WGS 84 decimal degrees</td></tr><tr><td><strong>JSON-LD Field</strong></td><td><code>spatialCoverage</code></td></tr><tr><td><strong>Description</strong></td><td>Latitude (-90 to 90) and Longitude (-180 to 180) of the location(s) this data represent in WGS84 decimal format.   </td></tr><tr><td><strong>Guidelines</strong></td><td><ul><li>Latitude should be between -90 and 90, while Longitude is between -180 and 180.</li><li>At least one coordinate pair present.</li><li>Enter only one coordinate pair for a single point and bounding box coordinates for non-point locations. </li><li>If the data location is better represented by a shape, you may additionally include a KML file in the file uploads.</li><li>Coordinates describing the point location or geographic area of the dataset are present and match the data, if applicable <mark style="color:$info;">(Included in</mark> <a href="/publish-data/check-dataset-metadata-quality#automated-checks"><mark style="color:$info;">Automated Checks</mark></a><mark style="color:$info;">)</mark></li></ul></td></tr><tr><td><strong>Example</strong></td><td>North: 9.28087 degrees<br>South: 9.09918 degrees<br>East: -79.821 degrees<br>West: -79.9747 degrees</td></tr><tr><td><strong>Requirement Level</strong></td><td>Required for publication</td></tr></tbody></table>

## Methods<mark style="color:red;">†</mark>

<table data-header-hidden><thead><tr><th width="205.93716430664062">​Title</th><th>​Title</th></tr></thead><tbody><tr><td><strong>Format</strong></td><td>Free Text</td></tr><tr><td><strong>JSON-LD Field</strong></td><td><code>measurementTechnique</code></td></tr><tr><td><strong>Description</strong></td><td><p>Methods for a dataset should focus on all aspects of dataset production and should be thorough enough for your work to be reproduced. A complete methods section will improve findability of your data, as all text entered into methods will also be searchable for users through the data portal filters. </p><p></p><p>When referring to methods published elsewhere, the dataset methods should include the methodology specific to data collection, processing, and QA/QC.</p></td></tr><tr><td><strong>Guidelines</strong></td><td><ul><li>Include descriptions of the experimental design, laboratory and/or field collection methods (e.g. observations and/or devices used), source data for synthesis studies, data processing, and QA/QC procedures, and known issues or limitations of data where applicable</li><li>More than 7 words <mark style="color:$info;">(Included in</mark> <a href="/publish-data/check-dataset-metadata-quality#automated-checks"><mark style="color:$info;">Automated Checks</mark></a><mark style="color:$info;">)</mark></li><li>Explain all acronyms if not defined elsewhere in the metadata (e.g. abstract)</li><li>Any mentioned publications are cited in <a href="#related-references">Related References</a></li></ul></td></tr><tr><td><strong>Example</strong></td><td><p>An example of a complete methods section can be viewed at: </p><blockquote><p>Del Vecchio J ; Thaler E ; Shelef E ; Fratkin M ; Lathrop E ; Thomas L ; Farley M ; Rowland J (2026): Radiocarbon age at Teller and Kougarok field sites, Seward Peninsula, Alaska. Next-Generation Ecosystem Experiments (NGEE) Arctic, ESS-DIVE repository. Dataset. <a href="https://data.ess-dive.lbl.gov/datasets/doi:10.15485/3363470">doi:10.15485/3363470</a> accessed via <a href="https://data.ess-dive.lbl.gov/datasets/doi:10.15485/3363470">https://data.ess-dive.lbl.gov/datasets/doi:10.15485/3363470</a> on 2026-07-24</p></blockquote></td></tr><tr><td><strong>Requirement Level</strong></td><td>Required for publication</td></tr></tbody></table>


# File Upload Guidance

This page provides all necessary information to determine how to transfer your data to ESS-DIVE.

## Dataset Submission Workflow

{% stepper %}
{% step %}

#### **Always submit your metadata first!**

Use the **Online Submission Form** or **Dataset API** to create and submit your metadata before providing any files. You can finalize your metadata later, it is not necessary to meet all publication requirements right away.&#x20;
{% endstep %}

{% step %}

#### **Pick the** [**Storage Option**](#storage-options) **and** [**Transfer Option**](#data-transfer-options) **that best suits your data**

Continue reading this page to find quick summaries of your options.
{% endstep %}

{% step %}

#### **Read through** [**ESS-DIVE's File Publication Requirements**](#file-requirements)

ESS-DIVE has certain file requirements that are checked during publication review. Review them before transferring your data to expedite the review process.
{% endstep %}

{% step %}

#### **Start uploading your files**

Each transfer option below provides links to the relevant instructions to get started.
{% endstep %}

{% step %}

#### Finalize your dataset metadata

Come back at anytime to finish your metadata! In general, always submit your metadata edits separately from your files.
{% endstep %}
{% endstepper %}

{% hint style="info" %}
**Do you have Large Data?**

On ESS-DIVE, we define "large data" as any individual file that is greater than 10GB or a dataset that contains more than 100 individual files.&#x20;

Large data should be uploaded following the [Large Data Support](/contributing-data/file-upload-guidance/large-data-support) instructions.
{% endhint %}

{% hint style="warning" icon="circle-question" %} <mark style="color:orange;">**Do you have Terabytes of data?**</mark>&#x20;

If your project needs to upload more than a Terabyte of data, ESS-DIVE will need to prepare to accommodate this. Contact <ess-dive-support@lbl.gov> to inform us of your needs.
{% endhint %}

<details>

<summary><strong>Why are the data storage and transfer volumes different?</strong></summary>

An **transfer limit** is the maximum volume of data you can upload *at one time* using one of ESS-DIVE's transfer tools (limit varies per tool). If your data exceeds this limit, you can batch your data into multiple uploads.

The **storage limit per dataset** is the maximum quantity of files and total volume of data that you can store on one dataset (limit varies per storage method). If your dataset volume exceeds the Tier 1 storage limit, you have to store your dataset on Tier 2.

</details>

***

## Storage Options

ESS-DIVE has two storage options available: Tier 1 and Tier 2. Your file size and structure will determine which storage option is right for your data.

<table data-card-size="large" data-view="cards" data-full-width="false"><thead><tr><th align="center"></th><th></th><th>Benefits</th><th>Action Item</th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td align="center"><strong>Tier 1</strong></td><td><ul><li>Folders must be zipped</li><li>Recommended for &#x3C;100 files</li><li>Recommended for &#x3C;100GB total volume</li><li>Upload data using any transfer option</li></ul></td><td><ul><li>Tracks data file views and downloads</li><li>DataONE backups</li><li>Supported by all ESS-DIVE features<br></li></ul></td><td><ul><li>Zip all folders before upload</li><li>Pick the <a href="#data-transfer-options">Data Transfer Option</a> that best suits your data</li></ul></td><td></td></tr><tr><td align="center"><strong>Tier 2</strong></td><td><ul><li>Unzipped, hierarchical folders</li><li>Recommended for >100 files</li><li>Recommended for >100GB total volume</li><li>Upload data using Globus</li></ul></td><td><ul><li>Browsable folder hierarchies</li><li>Public copy available on Globus to enable cloud-based downloads</li><li>Generates manifest with checksums for each file to verify integrity</li></ul></td><td><ul><li><a href="https://app.gitbook.com/o/-LgU7CB5CVrJQMt0aaNu/s/-LgU7GrufgIOpY1Ufd5R/~/edit/~/changes/211/contributing-data/file-upload-guidance/large-data-support/~/comments#how-to-contact-ess-dive-about-publishing-large-data">Contact ESS-DIVE to use Globus</a></li></ul></td><td><a href="/contributing-data/file-upload-guidance/large-data-support#tier-2-storage-for-large-data">Large Data Support</a></td></tr></tbody></table>

## **Data Transfer Options**

ESS-DIVE has three tools available for transferring data: Online Submission Form, Dataset API, and Globus Transfer Service. Your storage method, data size, and/or personal preference will determine which option is best for your data.

If you have large data or need to store your data on Tier 2, you will need to use Globus.

<table data-view="cards" data-full-width="false"><thead><tr><th align="center"></th><th>Volume Limit per Transfer</th><th>Folder Organization</th><th>What is this for?</th><th>What can you submit with this tool?</th><th data-hidden data-card-target data-type="content-ref"></th><th data-hidden data-card-cover data-type="image">Cover image</th></tr></thead><tbody><tr><td align="center"><strong>Online Submission Form</strong> </td><td><strong>&#x3C; 10 GB</strong></td><td>Folders must be zipped before upload</td><td>Quick, one-off datasets</td><td>Metadata + data</td><td><a href="/contributing-data/submit-data-with-online-form">Submit Data with Online Form</a></td><td><a href="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2Fx6DkzUjgu8FMXHTvZGwz%2FScreenshot%202026-08-17%20103357.png?alt=media&amp;token=d79342be-86e8-47d9-8500-ef4b4b237ddd">Screenshot 2026-08-17 103357.png</a></td></tr><tr><td align="center"><strong>Dataset API</strong></td><td><strong>&#x3C; 10 GB</strong></td><td>Folders must be zipped before upload</td><td>Programmatic, bulk dataset management </td><td>Metadata + data</td><td><a href="/contributing-data/submit-data-with-the-dataset-api">Submit Data with the Dataset API</a></td><td><a href="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FUY50LsfzhAVNBBsxGzlm%2FScreenshot%202026-08-17%20103519.png?alt=media&amp;token=0e99e074-cce2-4074-ba16-7e154734294a">Screenshot 2026-08-17 103519.png</a></td></tr><tr><td align="center"><strong>Globus Transfer Service</strong></td><td><strong>No Limit</strong></td><td>Supports unzipped folders</td><td>Default transfer method for files >10GB and Tier 2 data</td><td>Data only</td><td><a href="/contributing-data/file-upload-guidance/large-data-support">Large Data Support</a></td><td><a href="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FvTco6WrXNJU2jhXOlULi%2FScreenshot%202026-08-17%20103529.png?alt=media&amp;token=e899b1e6-bb2b-4ae1-b469-b122c9b595a7">Screenshot 2026-08-17 103529.png</a></td></tr></tbody></table>

## File Publication Requirements

These requirements are checked by the ESS-DIVE Team during publication review. \
**Review these requirements before transferring your data to expedite the review process.**

<table><thead><tr><th width="112.30145263671875"></th><th>Requirements</th></tr></thead><tbody><tr><td><strong>File Name</strong></td><td><ul><li>File is named concisely and descriptively</li><li>Any acronyms used in file names are explained in the abstract</li></ul></td></tr><tr><td><strong>File Extension</strong></td><td><ul><li><p>Use non-proprietary file formats where possible <mark style="color:$info;">(Included in</mark> <a href="/publish-data/check-dataset-metadata-quality#automated-checks"><mark style="color:$info;">Automated Checks</mark></a><mark style="color:$info;">)</mark></p><ul><li>Examples include Comma Separated Values (CSV) files, text (TXT) files, PNG, JPEG or TIFF image files, R or Python scripts, GeoJSON, and NetCDF files, among many others.</li><li>Keeping both the proprietary and converted version of the file with the same name is acceptable</li></ul></li><li>Word and Excel documents are proprietary formats and should be converted into non-proprietary formats (e.g., PDFs and CSVs)</li><li>If specific software is needed to view any uncommon file formats, specify the software in your abstract</li></ul></td></tr><tr><td><strong>Zip File</strong></td><td><ul><li>Describe contents of any zip files in the abstract </li><li><p>Zip files should not be corrupted. Ensure zip files can be opened before upload.</p><ul><li><p><span data-gb-custom-inline data-tag="emoji" data-code="1f4a1">💡</span> Quickly check your zip compression method in your terminal with:</p><pre class="language-ruby"><code class="lang-ruby">unzip -t file_name
</code></pre></li></ul></li><li><p>Use non-proprietary compression methods to zip your files. Open source compression methods allow zip files to be opened on any operating system.</p><ul><li>Non-proprietary compression methods include deflate.</li><li>Proprietary compression methods include <mark style="color:$danger;">deflate64</mark>.</li><li><span data-gb-custom-inline data-tag="emoji" data-code="26a0">⚠️</span> Windows uses a proprietary compression method (deflate64) to zip folders greater than 2GB. </li></ul></li></ul></td></tr></tbody></table>

{% hint style="info" icon="readme" %}
**Metadata Publication Requirements**

The ESS-DIVE Team also checks your metadata during the publication review process<a href="/contributing-data/package-level-metadata" class="button primary medium" data-icon="arrow-right">Click here to see metadata requirements</a>
{% endhint %}

{% hint style="info" icon="table-pivot" %}
**Reporting Format File Requirements**

A series of checks are performed on reporting format files during the publication review process&#x20;

<a href="/publish-data/review-cycle-and-criteria#review-process-for-reporting-formats" class="button primary medium" data-icon="arrow-right">Click here to see reporting format file requirements</a>
{% endhint %}

## Troubleshooting

#### Online Form

**\*Coming Soon\***&#x20;

#### Dataset API

{% content-ref url="/pages/GTgJ2cW6V8fNrNXc5cZC" %}
[Setup and Troubleshoot](/programmatic-tools/ess-dive-dataset-api/setup-and-troubleshoot)
{% endcontent-ref %}

#### Globus

{% content-ref url="/pages/-MXss07PasswM3LQmsGA" %}
[FAQs](/programmatic-tools/large-data-support/faqs)
{% endcontent-ref %}


# Large Data Support

Learn about ESS-DIVE's large data support tools and how they're used to upload and publish large data.

ESS-DIVE now has a <mark style="color:green;">**Tier 2 data storage**</mark> service to support publishing very large, hierarchical datasets that can be directly accessed from our repository. ESS-DIVE uses <mark style="color:green;">**Globus**</mark>, a data transfer service, to make it easier to upload large data to ESS-DIVE. The Tier 2 and Globus services are setup offline with close assistance from the ESS-DIVE Team.

#### How to contact ESS-DIVE about publishing large data:

For large data support, please email us at <ess-dive-support@lbl.gov> with the following information about your data:

> 1. What's the total file volume of your dataset?&#x20;
> 2. Approximately how many files are in your dataset and what's the range of file sizes?
> 3. Is the data structure hierarchical? If yes:
>    1. Can you easily flatten your data structure (i.e. move data out of folders)? Or
>    2. Can you compress the folders into ZIP files or will it be necessary to browse the folder hierarchies?
> 4. Where is your data stored currently (e.g. local desktop, cloud based server, Google Drive)?

## Globus: Upload Large Data

Globus (<https://www.globus.org/>) is a free, cloud-based data transfer service designed to move significant amounts of data. ESS-DIVE uses this service to move data from your local desktop or existing Globus endpoint to ESS-DIVE's storage services. This large data support tool can be used to resolve common upload errors or as the default upload method for data greater than 500GB.&#x20;

Learn more about and how to use Globus for publishing data on ESS-DIVE or resolving upload issues via our Globus documentation page.

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FWE8kLMEHPRdZ6XgWLx5K%2FGlobus-publicshare.png?alt=media&amp;token=830f54f7-ad1a-47dd-8107-520dc7b38b51" alt="" width="375"><figcaption><p>Figure 1: The Globus file manager (pictured) is accessible via <br>browser and is used as the primary interface for transferring <br>data with Globus.  </p></figcaption></figure>

{% content-ref url="/pages/-MX-a6sa8lrluGrgpHq6" %}
[Globus Data Transfer Service](/programmatic-tools/large-data-support)
{% endcontent-ref %}

## Tier 2: Storage for Large Data

Tier 2 (Figure 2) is ESS-DIVE's extended storage resource that is used to store very large, hierarchical datasets, instead of storing the data directly on ESS-DIVE's dataset landing pages, or Tier 1 (Figure 3). Data greater than 500GB in volume will be archived on Tier 2 by default. Additionally, Tier 2 supports the functionality to browse hierarchical folders in your browser prior to download.&#x20;

Data stored on Tier 2 resources can be accessed and downloaded from the Tier 2 landing page (Figure 2). This is separate from ESS-DIVE's dataset landing page (Figure 3). You can choose to publish some or all of your dataset files on Tier 2.

Generally, data should be stored on Tier 1 whenever possible. ESS-DIVE is constantly expanding and improving features on Tier 1 that may not be supported on Tier 2. However data less than 500GB can be published on Tier 2 if necessary.

Any data contributor can take advantage of the Tier 2 service even if your data is less than 500GB.  Please contact ESS-DIVE at <ess-dive-support@lbl.gov> to discuss if your data is suitable for Tier 2 storage.

<div><figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FXqdHt1qoCansxji1WvaB%2FTier2-landingpage.png?alt=media&amp;token=0d29198b-5518-443c-a482-c5cdf084ec19" alt=""><figcaption><p>Figure 2: Tier 2 landing page for large file exploration and download. Access to dataset metadata on Tier 1 is provided via link.</p></figcaption></figure> <figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FiBwqZtiKfA1uueNXT4ju%2FDataset-LinkedTier2-Data.png?alt=media&amp;token=cce70025-adef-4ab7-85bd-f26749bcbc8f" alt=""><figcaption><p>Figure 3: Tier 1 dataset landing page where metadata and data can be discovered and downloaded. Access to files on Tier 2 are provided via external link. </p></figcaption></figure></div>

Data contributors must use the Globus transfer service to upload their data to Tier 2. Once uploaded to Globus, ESS-DIVE will organize the data and add additional file metadata on the Tier 2 landing page. The data contributor will review and approve the data on Tier 2 prior to publication. At the time of publication, the data will be publicly accessible on the Globus "ESS-DIVE Public Share" collection, as well as, on the Tier 2 website. Additionally, external links to both Tier 2 and Globus will be added to the dataset metadata landing page for access and download (demonstrated in Figure 3).&#x20;

### Management and Preservation of Tier 2 Data&#x20;

ESS-DIVE stores redundant copies of data published on Tier 2 resources to preserve and provide long-term access to Environmental Systems Science (ESS) research data.

Please be aware that, at this time, the following features are not available for Tier 2 data:

1. Will not be linked to the DataOne federation,&#x20;
2. Cannot be private, and&#x20;
3. Data downloads and views will not be factored into data package statistics.

### How to Download Tier 2 Data

{% content-ref url="/pages/-MfndH11d-0IoWKMHn9P" %}
[Download Data](/searching-and-accessing-data/accessing-data)
{% endcontent-ref %}


# Submit Data with Online Form

This comprehensive tutorial will guide you through the process of creating and submitting a private dataset to ESS-DIVE.

#### :warning: Remember to review the [ESS-DIVE terms of use](https://ess-dive.lbl.gov/about/terms/) and ensure there is no [Protected Information](https://commons.lbl.gov/display/rpm2/Controlled+and+Prohibited+Information+Categories#ControlledandProhibitedInformationCategories--1898802862) in your dataset.

***

#### **Preferred Browsers**

* Chrome
* Firefox
* May have some issues with Safari and Internet Explorer

{% hint style="warning" %}
**IMPORTANT:** The following instructions assume you have been given access to upload data by an ESS-DIVE admin. If you are logging in for the first time, see documentation on how to setup [your account](/contributing-data/new-contributor-registration) to get access to upload.
{% endhint %}

{% hint style="info" %}
**Working Offline?**\
\
ESS-DIVE's [Offline Metadata Template](https://docs.google.com/document/d/1xeqJREKXkahMqJqCo3DdUK0cryE3-YVeftIaH9x4FtU/) can be used to prepare your dataset metadata prior to submission. We recommend using the template to collaborate with your team members in Google Docs, then copying and pasting the completed fields into the ESS-DIVE Dataset Submission form when you are ready to create your dataset.&#x20;
{% endhint %}

***

## Access the Data Submission Web Form

#### Click the link to the data submission web form: [https://data.ess-dive.lbl.gov](https://data.ess-dive.lbl.gov/#submit)

This link will take you to a sign in screen where you can use your ORCID credentials to sign in (Figure 1).&#x20;

:bulb: [**https://data-sandbox.ess-dive.lbl.gov**](https://data-sandbox.ess-dive.lbl.gov) is a test ESS-DIVE system where you can play around with the upload process. Any data that you submit to the test system is temporary, and will not be preserved in ESS-DIVE.

![Figure 1. Sign in page for data uploads](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-Mj1n1lC9z-X2uIK5hA2%2F-Mj1p4TDb4CvJc0BHoM4%2Fsign-in-to-submit.png?alt=media\&token=d39c8b70-e09d-4383-b8da-515a79fde5d0)

{% hint style="info" %}
**Troubleshooting login issues**

If you are having issues logging in to ESS-DIVE, please review our [login troubleshooting tips](https://data.ess-dive.lbl.gov/signin-help). Contact <ess-dive-support@lbl.gov> if problems persist.
{% endhint %}

Alternatively, if you are on the home page (Figure 2), click on the “Sign in with Orcid” button in the top right corner.

![Figure 2. ESS-DIVE Data portal home screen.](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-Mj1OEOK80Q6bTq1oE40%2F-Mj1PEs0NrU67_K5Us51%2Fess-dive-main-data-portal.png?alt=media\&token=0fbc03b9-182f-4223-81c8-2586ca29092d)

Once you click on the “Sign in with ORCID” button, the system will redirect you to a login screen (Figure 3). Login with your ORCID credentials. You can also link your ORCID with your institutional account if you prefer to login that way. If you do not have an ORCID account, visit [Creating an ORCID](https://app.gitbook.com/o/-LgU7CB5CVrJQMt0aaNu/s/-LgU7GrufgIOpY1Ufd5R/~/edit/~/changes/210/contributing-data/new-contributor-registration#id-1.-create-an-orcid).

![Figure 3. ORCID Default login page.](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2Fess-dive-docs%2F-LgjW5G-qORNMyNs9O3Q%2F-LgjWle3L7aVmebzb6lT%2F2.png?generation=1559870514445019\&alt=media)

## Instructions to Create a New Dataset

1. Once you are logged in, click on the green “Submit Data” button next to the “Sign in” button. If the button is disabled, it means your email address needs to be added to your profile found "My Settings." See the [Account Settings](/contributing-data/new-contributor-registration#account-settings) documentation.
2. Add your files in the top section to start your dataset submission. &#x20;
3. Enter all the fields in the various tabs (Figure 4). You only need to fill in required fields (marked with a \*) to submit a record, but we recommend that you fill as many of the fields as possible so users can more easily locate your dataset, and understand what it contains.

![Figure 4. Section to add files](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2Fess-dive-docs%2F-LgjW5G-qORNMyNs9O3Q%2F-LgjWle4jpAseHiCHJ2o%2F3.png?generation=1559870514444320\&alt=media)

### To review the expectations for content in sections 1-5, refer to our [Package Level Metadata Guide](https://docs.ess-dive.lbl.gov/data-and-metadata-upload/package-level-metadata).

### 1. Overview&#x20;

![Figure 5. Overview Section](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2Fess-dive-docs%2F-LgjW5G-qORNMyNs9O3Q%2F-LgjWle5midJDL5vGSvw%2F4.png?generation=1559870514444879\&alt=media)

### 2. People

![Figure 6. People Section](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2Fess-dive-docs%2F-LgjW5G-qORNMyNs9O3Q%2F-LgjWle6m58JrXFtc31i%2F5.png?generation=1559870514471821\&alt=media)

### 3. Dates&#x20;

![Figure 7. Dates Section](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2Fess-dive-docs%2F-LgjW5G-qORNMyNs9O3Q%2F-LgjWle7-8GO4hltFmEo%2F6.png?generation=1559870514414992\&alt=media)

### 4. Locations

![Figure 8. Locations Section](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2Fess-dive-docs%2F-LgjW5G-qORNMyNs9O3Q%2F-LgjWle8gDGQIfmtvPso%2F7.png?generation=1559870514410672\&alt=media)

### 5. Methods&#x20;

![Figure 9. Methods Section](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2Fess-dive-docs%2F-LgjW5G-qORNMyNs9O3Q%2F-LgjWle9zNj8CnRAsuRk%2F8.png?generation=1559870514398631\&alt=media)

### Submit the Dataset

When complete, hit the submit button at the bottom of your screen (Figure 10). This will save the record as a private dataset. Notify <ess-dive-support@lbl.gov> when you are done with your submission.

![Figure 10. Submit Button](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2Fess-dive-docs%2F-LgjW5G-qORNMyNs9O3Q%2F-LgjWleAFJwTsrnHBvzU%2F9.png?generation=1559870514409772\&alt=media)

#### VIEW YOUR SUBMISSION

If your submission is successful, you will see a confirmation message (Figure 11). Click on the I'm Done button, and verify that your dataset details are entered correctly (Figure 12). If you run into any problems, send an email to <ess-dive-support@lbl.gov>, with the error message you received.&#x20;

![Figure 11. A Successful Submission Message](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FwLDp2PCPiIYaV76QLwwR%2FSubmission-Success-Message-2024-v3.4.png?alt=media\&token=790b44b1-be0d-4a08-872a-49ea19d6e044)

![Figure 12. Dataset Landing Page](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FzKMWssA3xDEh8yYkrews%2FDataset-Landing-Page-UIv4-0-0-submitInstructions%201.png?alt=media\&token=f0a9663f-0b0d-4ace-95ba-d58097b76caa)

## Instructions to Edit an Existing Dataset

Click on the Edit Button, on your dataset View (Figure 13). This will take you to the form where you previously entered your metadata and uploaded data files.

![Figure 13. Submission Form](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2Fess-dive-docs%2F-LgjW5G-qORNMyNs9O3Q%2F-LgjWleDNwGgJel_Hkcc%2F12.png?generation=1559870514473556\&alt=media)


# Submit Data with the Dataset API

## Quick Start

Start submitting test datasets with ESS-DIVE's Dataset API right away using our Jupyter Notebook tutorials! No coding knowledge is necessary to run the notebooks.&#x20;

(1) Visit our GitHub linked below to **launch tutorial notebooks directly in your browser**. (2) Select the **Google Colab** or **Jupyter+R Binder** button. We recommend Google Colab for a smoother experience (*a Google account is required to use Colab*). (3) Then, open the **submit\_data** folder to access tutorials in R and Python.

{% embed url="<https://github.com/ess-dive/essdive-tutorials/tree/main>" %}

***

## About submissions with the Dataset API

Datasets are comprised of two components: (1) data files and (2) metadata which describes the data. To submit data files with the Dataset API, organize the data into one directory and enter the file path into our example code where instructed. To submit metadata, write dataset metadata that meet [ESS-DIVE's Dataset Metadata Requirements](/contributing-data/package-level-metadata) and format the metadata into the JSON-LD schema in your script. Then, submit the data and metadata to ESS-DIVE using the appropriate HTTP method.&#x20;

Continue reading this page for a more detailed recommendations on how to familiarize yourself with the process. In the code examples provided on the next page, we demonstrate how to format JSON-LD and submit requests in three different coding languages: R, Python, and Java.&#x20;

{% hint style="info" %}
**Dataset metadata** refers to the top level information that enables a dataset to be “discoverable” in search results. Examples of top-level metadata include the title, abstract, authors, variables and keywords.&#x20;

[**JSON-LD**](https://json-ld.org/) is a schema to encode linked data using JSON.
{% endhint %}

### 1. Prepare Submission

1. **Review** [**ESS-DIVE's Metadata Criteria**](/contributing-data/package-level-metadata) for descriptions and expectations for each metadata field&#x20;
2. **Review** [**Metadata JSON-LD Formatting**](https://api-sandbox.ess-dive.lbl.gov/) (cURL, HTTP) to learn the expected JSON-LD schema
   * The technical documentation linked above will help you get familiar with the interface, understand the available operations, possible errors and the structure of the JSON-LD produced. **Hint:** Our metadata criteria includes the JSON-LD equivalents for each field.
3. **Prepare Metadata JSON-LD in R, Python or Java** using the [JSON-LD code examples](https://github.com/ess-dive/essdive-package-service-examples/blob/master/data/JSON-LD/example-2.jsonld) as reference.

### 2. Test Dataset Submission on [Sandbox](/contributing-data/get-started#try-out-sandbox)

After you have familiarized yourself with the Dataset API and JSON-LD, test your dataset submissions on **Sandbox** ([https://api-sandbox.ess-dive.lbl.gov](https://api-sandbox.ess-dive.lbl.gov/)).&#x20;

Two of the most common programming languages in the ESS space are Python and R. Either of these languages can be used to write scripts for submitting dataset metadata to be validated.

* ***Python:*** dataset JSON-LD metadata can be submitted using the **`requests`** module
* ***R:*** dataset metadata can be submitted using the **`httr`** and **`jsonlite`** packages
* **Java:** datasets JSON-LD metadata can be submitted using Apache HTTPComponents

{% hint style="warning" %}
Be sure to use ESS-DIVE's test instance, Sandbox ([https://api-sandbox.ess-dive.lbl.gov](https://api-sandbox.ess-dive.lbl.gov/)), while building scripts and making corrections to JSON\_LD dataset submissions.&#x20;
{% endhint %}

### 3. Complete Code and Submit Data to Production

Once you've familiarized yourself with ESS-DIVE's metadata and dataset API schema, use our production domain <https://api.ess-dive.lbl.gov/> to submit datasets to ESS-DIVE for publishing and review.&#x20;

Note that the <https://data.ess-dive.lbl.gov/> domain is for directly interacting with the web user interface.

{% hint style="warning" %}
Only switch to ESS-DIVE's production instance (<https://api.ess-dive.lbl.gov/>) after your scripts are complete.
{% endhint %}

## Learn More:

{% content-ref url="/pages/-LgikSRqc4GcZi-eCBoK" %}
[ESS-DIVE Dataset API](/programmatic-tools/ess-dive-dataset-api)
{% endcontent-ref %}


# Code Examples

## Setup

The following code examples require installation of certain packages and authentication from your ESS-DIVE account. Follow the instructions for setting up the Dataset API for your preferred coding language before trying out the search code examples:

{% content-ref url="/pages/GTgJ2cW6V8fNrNXc5cZC" %}
[Setup and Troubleshoot](/programmatic-tools/ess-dive-dataset-api/setup-and-troubleshoot)
{% endcontent-ref %}

***

## Create Metadata

The metadata example provided here is from the ESS-DIVE sandbox site: <https://data-sandbox.ess-dive.lbl.gov/#view/doi:10.3334/CDIAC/spruce.001>. &#x20;

{% tabs %}
{% tab title="Python" %}

## Format JSON-LD Metadata in Python

Setup the JSON for the “provider”, which includes details about the project. Simply update the "value" to use the desired project identifier, lookup project identifiers via ESS-DIVE's project list: <https://data.ess-dive.lbl.gov/projects>. The project will be listed as the publisher in the citation.

```python
provider_spruce = {
    "identifier": {
        "@type": "PropertyValue",
        "propertyID": "ess-dive",
        "value": "1e6d50d3-9532-43fb-a63f-bdcb4350bf0c"
    }
 }
```

Prepare the dataset authors in the order that you would like them to appear in the citation. Please add the ORCID for all authors, especially the first author, if possible.&#x20;

```python
creators =  [
   {
     "@id": "http://orcid.org/0000-0001-7293-3561",
     "givenName": "Paul J",
     "familyName": "Hanson",
     "affiliation": "Oak Ridge National Laboratory",
     "email": "hansonpj@ornl.gov"
   },
   {
     "givenName": "Jeffrey",
     "familyName": "Riggs",
     "affiliation": "Oak Ridge National Laboratory"
   },
   {
     "givenName": "C",
     "familyName": "Nettles",
     "affiliation": "Oak Ridge National Laboratory"
   },
   {
     "givenName": "William",
     "familyName": "Dorrance",
     "affiliation": "Oak Ridge National Laboratory"
   },
   {
     "givenName": "Les",
     "familyName": "Hook",
     "affiliation": "Oak Ridge National Laboratory"
   }
 ]
```

Create the rest of the JSON-LD object

```python
json_ld = {
 "@context": "http://schema.org/",
 "@type": "Dataset",
 "@id": "http://dx.doi.org/10.3334/CDIAC/spruce.001",
 "name": "SPRUCE S1 Bog Environmental Monitoring Data: 2010-2016",
 "description": [
   "This data set reports selected ambient environmental monitoring data from the S1 bog in Minnesota for the period June 2010 through December 2016. Measurements of the environmental conditions at these stations will serve as a pre-treatment baseline for experimental treatments and provide driver data for future modeling activities.",
   "The site is the S1 bog, a Picea mariana [black spruce] - Sphagnum spp. bog forest in northern Minnesota, 40 km north of Grand Rapids, in the USDA Forest Service Marcell Experimental Forest (MEF). There are/were three monitoring sites located in the bog: Stations 1 and 2 are co-located at the southern end of the bog and Station 3 is located north central and adjacent to an existing U.S. Forest Service monitoring well.",
   "There are eight data files with selected results of ambient environmental monitoring in the S1 bog for the period June 2010 through December 2016. One file has the ",
   "other seven have the available data for a given calendar year. Not all measurements started in June 2010 and EM3 measurements ended in May 2014.",
   "Further details about the data package are in the attached pdf file (SPRUCE_EM_DATA_2010_2016_20170620)."
 ],
 "creator": creators,
 "datePublished": "2015",
 "keywords": [
   "EARTH SCIENCE > BIOSPHERE > VEGETATION",
   "Climate Change"
 ],
 "variableMeasured": [
   "EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC TEMPERATURE > SURFACE TEMPERATURE > AIR TEMPERATURE",
   "EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC WATER VAPOR > WATER VAPOR INDICATORS > HUMIDITY > RELATIVE HUMIDITY",
   "EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC PRESSURE > SEA LEVEL PRESSURE",
   "EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC TEMPERATURE > SURFACE TEMPERATURE > DEW POINT TEMPERATURE > DEWPOINT DEPRESSION",
   "EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC WINDS > SURFACE WINDS > WIND SPEED",
   "EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC WINDS > SURFACE WINDS > WIND DIRECTION",
   "EARTH SCIENCE > BIOSPHERE > VEGETATION > PHOTOSYNTHETICALLY ACTIVE RADIATION",
   "EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC RADIATION > NET RADIATION",
   "EARTH SCIENCE > LAND SURFACE > SURFACE RADIATIVE PROPERTIES > ALBEDO",
   "EARTH SCIENCE > LAND SURFACE > SOILS > SOIL TEMPERATURE",
   "Precipitation (Total)",
   "Irradiance",
   "Groundwater Temperature",
   "Groundwater Level",
   "Volumetric Water Content",
   "surface_albedo"
 ],
 "license": "http://creativecommons.org/licenses/by/4.0/",
 "spatialCoverage": [
   {
     "description": "Site ID: S1 Bog Site name: S1 Bog, Marcell Experimental Forest Description: The site is the 8.1-ha S1 bog, a Picea mariana [black spruce] - Sphagnum spp. ombrotrophic bog forest in northern Minnesota, 40 km north of Grand Rapids, in the USDA Forest Service Marcell Experimental Forest (MEF). The S1 bog was harvested in successive strip cuts in 1969 and 1974 and the cut areas were allowed to naturally regenerate. Stations 1 and 2 are located in a 1974 strip that is characterized by a medium density of 3-5 meter black spruce and larch trees with an open canopy. The area was suitable for siting a monitoring station for representative meteorological conditions on the S1 bog. Station 3 is located in a 1969 harvest strip that is characterized by a higher density of 3-5 meter black spruce and larch trees with a generally closed canopy. Measurements at this station represent conditions in the surrounding stand. Site Photographs are in the attached document",
     "geo": [
       {
         "name": "Northwest",
         "latitude": 47.50285,
         "longitude": -93.48283
       },
       {
         "name": "Southeast",
         "latitude": 47.50285,
         "longitude": -93.48283
       }
     ]
   }
 ],
 "funder": {
   "@id": "http://dx.doi.org/10.13039/100006206",
   "name": "U.S. DOE > Office of Science > Biological and Environmental Research (BER)"
 },
 "temporalCoverage": {
   "startDate": "2010-07-16",
   "endDate": "2016-12-31"
 },
 "editor": {
   "@id": "http://orcid.org/0000-0001-7293-3561",
   "givenName": "Paul J",
   "familyName": "Hanson",
   "email": "hansonpj@ornl.gov"
 },
 "provider": provider_spruce,
 "measurementTechnique": [
   "The stations are equipped with standard sensors for measuring meteorological parameters, solar radiation, soil temperature and moisture, and groundwater temperature and elevation. Note that some sensor locations are relative to nearby vegetation and bog microtopographic features (i.e., hollows and hummocks). See Table 1 in the attached pdf (SPRUCE_EM_DATA_2010_2016_20170620) for a list of measurements and further details. Sensors and data loggers were initially installed and became operational in June, July, and August of 2010. Additional sensors were added in September 2011. Station 3 was removed from service on May 12, 2014.",
   "These data are considered at Quality Level 1. Level 1 indicates an internally consistent data product that has been subjected to quality checks and data management procedures. Established calibration procedures were followed."
 ]
}
```

{% hint style="warning" %}
Please refer to the API documentation to understand the schema and navigate through any errors: [https://api-sandbox.ess-dive.lbl.gov](https://api-sandbox.ess-dive.lbl.gov/)
{% endhint %}
{% endtab %}

{% tab title="R" %}

## Format JSON-LD Metadata in R

Due to R complex JSON-LD support limitations, you need to create a text file of your JSON-LD and add it’s directory in the following read\_file function.

Here’s an example for a [JSON-LD](https://github.com/ess-dive/essdive-package-service-examples/blob/master/data/JSON-LD/example-1.jsonld) file located on our [ESS-DIVE package service examples github repository](https://github.com/ess-dive/essdive-package-service-examples).

{% hint style="warning" %}
To make sure your file is properly saved in the JSON-LD format, consider using the Atom text editor (<https://atom.io>)
{% endhint %}

Download the file and load it into your script.

```r
json_file <- read_file("~/directory/to/your/jsonld/file")
```

{% endtab %}

{% tab title="Java" %}

## Format JSON-LD Metadata in Java

Setup the JSON definitions to build your JSON\_LD.

```java
 //JSON objects variables
  JSONObject provider_spruce_json = new JSONObject();
  JSONObject member = new JSONObject();
  JSONObject funder = new JSONObject();
  JSONObject temporalCoverage = new JSONObject();
  JSONObject editor = new JSONObject();
  JSONObject spatial_coverage_json = new JSONObject();
  JSONObject primary_Creator = new JSONObject();
  JSONObject secondary_Creator = new JSONObject();
  JSONObject geo_northwest = new JSONObject();
  JSONObject geo_southeast = new JSONObject();
  JSONObject JSON_LD = new JSONObject();
  
  JSONArray creators_json = new JSONArray();
  JSONArray spatial_coverage_array = new JSONArray();
  JSONArray geo = new JSONArray();
  JSONArray measurementTechnique = new JSONArray();
  JSONArray JSON_LD_Description = new JSONArray();
  JSONArray keywords = new JSONArray();
  JSONArray variableMeasured = new JSONArray();
```

Now fill the details about the “provider”. This is the details about the project. The project will be listed as the publisher in the citation.

```java
  // JSON_LD member assignment
  member.put("@id","http://orcid.org/0000-0001-7293-3561");
  member.put("givenName","Paul J");
  member.put("familyName","Hanson");
  member.put("email","hansonpj@ornl.gov");
  member.put("jobTitle","Principal Investigator");
 
  // JSON_LD provider spruce assignment
  provider_spruce_json.put("name","SPRUCE");
  provider_spruce_json.put("member",member);
```

Prepare the dataset authors in the order that you would like them to appear in the citation.  Please add the ORCID for all authors, especially the first author, if possible.&#x20;

```java
  // JSON_LD primary creator assignment
  primary_Creator.put("@id","http://orcid.org/0000-0001-7293-3561");
  primary_Creator.put("givenName","Paul J");
  primary_Creator.put("familyName","Hanson");
  primary_Creator.put("affiliation","Oak Ridge National Laboratory");
  primary_Creator.put("email","hansonpj@ornl.gov");
 
  // JSON_LD secondary creator assignment
  secondary_Creator.put("givenName","Jeffrey");
  secondary_Creator.put("familyName","Riggs");
  secondary_Creator.put("affiliation","Oak Ridge National Laboratory");
 
  // Define as many creators as you need into newer JSON Objects and add them to the creators_json_array
  creators_json.add(primary_Creator);
  creators_json.add(secondary_Creator);
```

Initialize JSON\_LD strings

```java
  // JSON_LD Strings arrays
  variableMeasured.add("EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC TEMPERATURE > SURFACE TEMPERATURE > AIR TEMPERATURE");
  variableMeasured.add("EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC WATER VAPOR > WATER VAPOR INDICATORS > HUMIDITY > RELATIVE HUMIDITY");
  variableMeasured.add("EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC PRESSURE > SEA LEVEL PRESSURE");
  variableMeasured.add("EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC TEMPERATURE > SURFACE TEMPERATURE > DEW POINT TEMPERATURE > DEWPOINT DEPRESSION");
  variableMeasured.add("EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC WINDS > SURFACE WINDS > WIND SPEED");
  variableMeasured.add("EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC WINDS > SURFACE WINDS > WIND DIRECTION");
  variableMeasured.add("EARTH SCIENCE > BIOSPHERE > VEGETATION > PHOTOSYNTHETICALLY ACTIVE RADIATION");
  variableMeasured.add("EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC RADIATION > NET RADIATION");
  variableMeasured.add("EARTH SCIENCE > LAND SURFACE > SURFACE RADIATIVE PROPERTIES > ALBEDO");
  variableMeasured.add("EARTH SCIENCE > LAND SURFACE > SOILS > SOIL TEMPERATURE");
  variableMeasured.add("Precipitation (Total)");
  variableMeasured.add("Irradiance");
  variableMeasured.add("Groundwater Temperature");
  variableMeasured.add("Groundwater Level");
  variableMeasured.add("Volumetric Water Content");
  variableMeasured.add("surface_albedo");
  measurementTechnique.add("The stations are equipped with standard sensors for measuring meteorological parameters, solar radiation, soil temperature and moisture, and groundwater temperature and elevation. Note that some sensor locations are relative to nearby vegetation and bog microtopographic features (i.e., hollows and hummocks). See Table 1 in the attached pdf (SPRUCE_EM_DATA_2010_2016_20170620) for a list of measurements and further details. Sensors and data loggers were initially installed and became operational in June, July, and August of 2010. Additional sensors were added in September 2011. Station 3 was removed from service on May 12, 2014.");
  measurementTechnique.add("These data are considered at Quality Level 1. Level 1 indicates an internally consistent data product that has been subjected to quality checks and data management procedures. Established calibration procedures were followed.");
  JSON_LD_Description.add("This data set reports selected ambient environmental monitoring data from the S1 bog in Minnesota for the period June 2010 through December 2016. Measurements of the environmental conditions at these stations will serve as a pre-treatment baseline for experimental treatments and provide driver data for future modeling activities.");
  JSON_LD_Description.add("The site is the S1 bog, a Picea mariana [black spruce] - Sphagnum spp. bog forest in northern Minnesota, 40 km north of Grand Rapids, in the USDA Forest Service Marcell Experimental Forest (MEF). There are/were three monitoring sites located in the bog: Stations 1 and 2 are co-located at the southern end of the bog and Station 3 is located north central and adjacent to an existing U.S. Forest Service monitoring well.");
  JSON_LD_Description.add("There are eight data files with selected results of ambient environmental monitoring in the S1 bog for the period June 2010 through December 2016. One file has the ");
  JSON_LD_Description.add("other seven have the available data for a given calendar year. Not all measurements started in June 2010 and EM3 measurements ended in May 2014.");
  JSON_LD_Description.add("Further details about the data package are in the attached pdf file (SPRUCE_EM_DATA_2010_2016_20170620).");
  keywords.add("EARTH SCIENCE > BIOSPHERE > VEGETATION");
  keywords.add("Climate Change");
```

Add nested information in JSON\_objects.

```java
  // JSON_LD spatial coverage assignment
  spatial_coverage_json.put("description","Site ID: S1 Bog Site name: S1 Bog, Marcell Experimental Forest Description: The site is the 8.1-ha S1 bog, a Picea mariana [black spruce] - Sphagnum spp. ombrotrophic bog forest in northern Minnesota, 40 km north of Grand Rapids, in the USDA Forest Service Marcell Experimental Forest (MEF). The S1 bog was harvested in successive strip cuts in 1969 and 1974 and the cut areas were allowed to naturally regenerate. Stations 1 and 2 are located in a 1974 strip that is characterized by a medium density of 3-5 meter black spruce and larch trees with an open canopy. The area was suitable for siting a monitoring station for representative meteorological conditions on the S1 bog. Station 3 is located in a 1969 harvest strip that is characterized by a higher density of 3-5 meter black spruce and larch trees with a generally closed canopy. Measurements at this station represent conditions in the surrounding stand. Site Photographs are in the attached document");
  spatial_coverage_json.put("geo", geo);

  spatial_coverage_array.add(spatial_coverage_json);
 
  // JSON_LD funder assignment
  funder.put("@id", "http://dx.doi.org/10.13039/100006206");
  funder.put("name", "U.S. DOE > Office of Science > Biological and Environmental Research (BER)");
 
  // JSON_LD temporalCoverage assignment
  temporalCoverage.put("startDate","2010-07-16");
  temporalCoverage.put("endDate","2016-12-31");
  
  // JSON_LD editor assignment
  editor.put("@id", "http://orcid.org/0000-0001-7293-3561");
  editor.put("givenName", "Paul J");
  editor.put("familyName", "Hanson");
  editor.put("email", "hansonpj@ornl.gov");
 
  // JSON_LD geo variables assignments
  geo_northwest.put("name","Northwest");
  geo_northwest.put("latitude",47.50285);
  geo_northwest.put("longitude",-93.48283);
 
  geo_southeast.put("name","Southeast");
  geo_southeast.put("latitude",47.50285);
  geo_southeast.put("longitude",-93.48283);
 
  geo.add(geo_northwest);
  geo.add(geo_southeast);
```

Create the rest of the JSON-LD object

```java
  // Main JSON_LD
  JSON_LD.put("@context","http://schema.org/");
  JSON_LD.put("@type","Dataset");
  JSON_LD.put("@id","http://dx.doi.org/10.3334/CDIAC/spruce.001");
  JSON_LD.put("name","SPRUCE S1 Bog Environmental Monitoring Data: 2010-2016");
  JSON_LD.put("description",JSON_LD_Description);
  JSON_LD.put("creator",creators_json);
  JSON_LD.put("datePublished","2015");
  JSON_LD.put("keywords",keywords);
  JSON_LD.put("variableMeasured",variableMeasured);
  JSON_LD.put("license","http://creativecommons.org/licenses/by/4.0/");
  JSON_LD.put("spatialCoverage",spatial_coverage_array);
  JSON_LD.put("funder",funder);
  JSON_LD.put("temporalCoverage",temporalCoverage);
  JSON_LD.put("editor",editor);
  JSON_LD.put("provider", provider_spruce_json);
  JSON_LD.put("measurementTechnique",measurementTechnique);
```

{% endtab %}
{% endtabs %}

## Submit Dataset

The following lines of code submits and validates JSON-LD metadata for a single dataset.

### Metadata Only

{% tabs %}
{% tab title="Python" %}

## Submit Metadata Only in Python

Submit the JSON-LD object with the Dataset API

```python
post_packages_url = "{}{}".format(base,endpoint)
post_package_response = requests.post(post_packages_url,
                                      headers={"Authorization":header_authorization},
                                      json=json_ld)

if post_package_response.status_code == 201:
    # Success
    response=post_package_response.json()
    print(f"View URL:{response['viewUrl']}")
    print(f"Name:{response['dataset']['name']}")
else:
    # There was an error
    print(post_package_response.text)
```

{% endtab %}

{% tab title="R" %}

## Submit Metadata Only in R

Submit the JSON-LD object with the Dataset API

{% hint style="warning" %}
Make sure to enter your own file path.&#x20;

E.g. `C:\\User\\Files\API_Tutorial.json` for Windows & `/Users/Files/API_Tutorial.json` for Mac
{% endhint %}

```r
call_post_package <- paste(base,endpoint, sep="/")
post_package = POST(call_post_package,
                    body = json_file,
                       add_headers(Authorization=header_authorization,
                       "Content-Type"="application/json"))
```

Review the results

```r
results = content(post_package)
attributes(results)
$names
[1] "id"      "viewUrl" "detail"  "errors"  "dataset"

results$detail
[1] "Data Package created successfully."
results$errors
NULL

results$viewUrl
[1] "https://data-dev.ess-dive.lbl.gov/view/ess-dive-XXXXXXXXXXXX-20190621T175431086775"
```

{% endtab %}

{% tab title="Java" %}

## Submit Metadata Only in Java

Submit the JSON-LD object with the Dataset API

```java
  try{
    String url = base + endpoint;
    HttpPost request = new HttpPost(url);
    StringEntity params = new StringEntity(JSON_LD.toString()); //Setting the JSON-LD Object to the request params
    request.addHeader("content-type", "application/json");
    request.addHeader("Authorization", header_authorization);
    request.setEntity(params);
  
    HttpResponse response;
    response = httpClient.execute(request);
    HttpEntity entity = response.getEntity();
    String responseString = EntityUtils.toString(entity, "UTF-8");

    if(response.getStatusLine().getStatusCode() == 201){
      System.out.println(response.toString());
      System.out.println(responseString);
    } else {
      System.out.println(response.getStatusLine().getReasonPhrase());
      System.out.println(responseString);
    }
  } catch (Exception ex) {
    System.out.print(ex.getMessage().toString());
  }
```

{% endtab %}
{% endtabs %}

### Single Data File

{% tabs %}
{% tab title="Python" %}

## Submit Metadata and Single Data File in Python

To submit the JSON-LD object along with data files, **you need to create a folder** named files and add your desired file to upload inside it.

```python
files_tuples_array = []
upload_file = “path/to/your_file”

files_tuples_array.append((("json-ld", json.dumps(json_ld))))
files_tuples_array.append(("data", open(upload_file ,'rb')))

post_packages_url = "{}{}".format(base,endpoint)
post_package_response = requests.post(post_packages_url,
                                    headers={"Authorization":header_authorization},
                                    files= files_tuples_array)

if post_package_response.status_code == 201:
    # Success
    response=post_package_response.json()
    print(f"View URL:{response['viewUrl']}")
    print(f"Name:{response['dataset']['name']}")
else:
    # There was an error
    print(post_package_response.text)
```

{% hint style="warning" %}
Remember to change the file directories & file names to your actual names. The directory variable can be left blank if your API is already located in the same directory as your file.
{% endhint %}
{% endtab %}

{% tab title="R" %}

## Submit Metadata and Single Data File in R

To submit the JSON-LD object along with data files, **you need to create a folder** named files and add your desired file to upload inside it.

```r
call_post_package <- paste(base,endpoint, sep="/")

post_package = POST(call_post_package, body=list("json-ld"=json_file,      
        data=upload_file("your-directory/your-file","text/csv")),
        add_headers(Authorization=header_authorization, 
                    "Content-Type"="multipart/form-data"))
```

Review the results

```r
content(post_package)$viewUrl
[1] "https://data-dev.ess-dive.lbl.gov/view/ess-dive-XXXXXXXXXXXX-20190621T175431086775"
```

{% endtab %}

{% tab title="Java" %}

## Submit Metadata and Single Data File in Java

To submit the JSON-LD object along with data files, **you need to create a folder** named files and add your desired file to upload inside it.

```java
  try{
    String url = base + endpoint;
    HttpPost uploadFile = new HttpPost(url);
    uploadFile.addHeader("Authorization", header_authorization);
    File file_to_upload = new File("/directory-to-your-file/file");
    
    String content = FileUtils.readFileToString(file_to_upload, "UTF-8");
    
    FormBodyPart bodyPart = FormBodyPartBuilder.create()
    .setName("files")
    .addField("Content-Disposition", "form-data; name=\"data\"; filename=\"<your-file-name>\"")
    .setBody(new StringBody(content, ContentType.TEXT_PLAIN))
    .build();
    
    MultipartEntityBuilder builder = MultipartEntityBuilder.create()
    .setMode(HttpMultipartMode.BROWSER_COMPATIBLE)
    .setContentType(ContentType.MULTIPART_FORM_DATA);
    
    builder.addPart("json-ld", new StringBody(JSON_LD.toString(), ContentType.TEXT_PLAIN));
    builder.addPart(bodyPart);
    
    uploadFile.setEntity(builder.build());
    
    CloseableHttpResponse response = httpClient.execute(uploadFile);
    HttpEntity responseEntity = response.getEntity();
    
    response = httpClient.execute(uploadFile);
    HttpEntity entity = response.getEntity();
    String responseString = EntityUtils.toString(responseEntity, "UTF-8");
    
    if(response.getStatusLine().getStatusCode() == 201){
      System.out.println(response.toString());
      System.out.println(responseString);
    } else {
      System.out.println(response.getStatusLine().getReasonPhrase());
      System.out.println(responseString);
    }
  } catch (Exception ex) {
    System.out.print(ex.getMessage().toString());
  }
```

{% endtab %}
{% endtabs %}

### Many Data Files

{% tabs %}
{% tab title="Python" %}

## Submit Metadata and Many Data Files in Python

In case you have many files to be uploaded, you can place them all inside the files directory and use the following code:

```python
files_tuples_array = []
files_upload_directory = "your_upload_directory/"
files = os.listdir(files_upload_directory)

files_tuples_array.append((("json-ld", json.dumps(json_ld))))

for filename in files:
   file_directory = files_upload_directory + filename
   files_tuples_array.append((("data", open(file_directory, 'rb'))))

post_packages_url = "{}{}".format(base,endpoint)
post_package_response = requests.post(post_packages_url,
                                    headers={"Authorization":header_authorization},
                                    files= files_tuples_array)

if post_package_response.status_code == 201:
   # Success
   response=post_package_response.json()
   print(f"View URL:{response['viewUrl']}")
   print(f"Name:{response['dataset']['name']}")
else:
   # There was an error
   print(post_package_response.text)
```

{% endtab %}

{% tab title="R" %}

## Submit Metadata and Many Data Files in R

See our Dataset API GitHub repository for a complete example in R.&#x20;

{% embed url="<https://github.com/ess-dive/essdive-package-service-examples/blob/main/code/r/submit_metadata_with_multiple_files.r>" %}
{% endtab %}

{% tab title="Java" %}
*Currently there are no examples of this available in Java.*
{% endtab %}
{% endtabs %}

***

## Edit Dataset

### Metadata Only

{% tabs %}
{% tab title="Python" %}

## **Edit Metadata in Python**

Use the PUT function to update the metadata of a dataset.  This example updates the name of a dataset. &#x20;

```python
dataset_id = "<Enter an ESS-DIVE Identifier here>"

put_package_url = "{}{}/{}".format(base,endpoint, dataset_id)

metadata_update_dict = {"name": "Updated Dataset Name"}

put_package_response = requests.put(put_package_url,
                                    headers={"Authorization":header_authorization},
                                    json=metadata_update_dict)
```

Check the results for the changed metadata attribute

```python
# Check for errors
if put_package_response.status_code == 200:
   # Success
   response=put_package_response.json()
   print(f"View URL:{response['viewUrl']}")
   print(f"Name:{response['dataset']['name']}")
else:
   # There was an error
   print(put_package_response.text)
```

{% endtab %}

{% tab title="R" %}

## **Edit Metadata in R**

Use the PUT function to update the metadata of a dataset.  This example updates the name of a dataset. &#x20;

```r
call_put_package <- paste(base,endpoint,get_package_json$id, sep="/")

put_package = PUT(call_put_package,
                  body = "{ \"name\": \"My Tutorial Title\" }",
                  add_headers(Authorization=header_authorization,
                      "Content-Type"="application/json"))
```

{% hint style="info" %}
**NOTE:** The body argument is a string not a file.
{% endhint %}

Transform the result into a data frame. (Ignore the warning message)

```r
put_package_text <- content(put_package, "text")
put_package_json <- fromJSON(put_package_text)
```

Check the results for the changed metadata attribute

```r
# Check for errors
if(!http_error(put_package) ){
  attributes(put_package_json)
  put_package_json$viewUrl
  put_package_json$dataset$name
}else {
  http_status(put_package)
  print(put_package_text)
}

[1] "Data Package updated successfully."
NULL
[1] "https://data-dev.ess-dive.lbl.gov/view/ess-dive-xxx-20190621T185438900719"
[1] "My Tutorial Title"
```

{% endtab %}

{% tab title="Java" %}

## **Edit Metadata in Java**

Use the PUT function to update the metadata of a dataset.  This example updates the name of a dataset.&#x20;

```java
  String id = "<Enter-your-dataset-id>";
  JSONObject JSON_LD_update = new JSONObject();
  JSON_LD_update.put("name","Updated dataset title");        
     try{
          String url = base + endpoint + "/" + id;
          HttpPut request = new HttpPut(url);
          StringEntity params = new StringEntity(JSON_LD_update.toString()); //Setting the JSON-LD Object to the request params
          request.addHeader("content-type", "application/json");
          request.addHeader("Authorization", header_authorization);
          request.setEntity(params);
          
          HttpResponse response;
          response = httpClient.execute(request);
          HttpEntity entity = response.getEntity();
          String responseString = EntityUtils.toString(entity, "UTF-8");
          
          System.out.println(response.getStatusLine().getStatusCode());
          System.out.println(response.toString());
          System.out.println(response.getStatusLine());
          
          if(response.getStatusLine().getStatusCode() == 200){
               System.out.println(response.toString());
               System.out.println("Dataset updated");
               System.out.println(responseString);
          } else {
               System.out.println(response.getStatusLine().getReasonPhrase());
               System.out.println(responseString);
               }
     } catch (Exception ex) {
          System.out.print(ex.getMessage().toString());
     }

```

{% endtab %}
{% endtabs %}

### Metadata and Data

{% tabs %}
{% tab title="Python" %}

## **Edit Metadata and Data in Python**

Use the PUT function to update a dataset.  This example updates the date published to 2019 of a dataset and adds a new data file.

```python
dataset_id = "<Enter an ESS-DIVE Identifier here>"

files_tuples_array = []
upload_file = "path/to/your_file"
files_tuples_array.append((("json-ld", json.dumps(metadata_update_dict))))
files_tuples_array.append(("data", open(upload_file ,'rb')))

put_package_url = "{}{}/{}".format(base,endpoint, dataset_id)



put_package_response = requests.put(put_package_url,
                                   headers={"Authorization":header_authorization},
                                   files= files_tuples_array)
```

Check the results for the changed metadata attribute and newly uploaded file

```python
# Check for errors
if put_package_response.status_code == 200:
    # Success
    response=put_package_response.json()
    print(f"View URL:{response['viewUrl']}")
    print(f"Date Published:{response['dataset']['datePublished']}")
    print(f"Files In Dataset:{response['dataset']['distribution']}")
else:
   # There was an error
   print(put_package_response.text)
```

```python
get_packages_url = "{}{}".format(base,endpoint)
get_packages_response = requests.get(get_packages_url, 
    headers={"Authorization":header_authorization})

if get_packages_response.status_code == 200:
   #Success
   print(get_packages_response.json())
else:
   # There was an error
   print(get_packages_response.text)
```

{% endtab %}

{% tab title="R" %}

## **Edit Metadata and Data in R**

Use the PUT function to update a dataset.  This example updates the date published to 2019 of a dataset and adds a new data file.

```r
call_put_package <- paste(base,endpoint,put_package_json$id, sep="/")
put_package_data = PUT(call_put_package, 
           body=list("json-ld"="{ \"datePublished\": \"2019\" }",
           data=upload_file("your-directory/your-file","text/csv")),
           add_headers(Authorization=header_authorization,
                     "Content-Type"="multipart/form-data"))
```

{% hint style="info" %}
**NOTE:** The body argument is a string not a file.
{% endhint %}

Transform the result into a data frame. (Ignore the warning message)

```r
put_package_data_text <- content(put_package_data, "text")
put_package_data_json <- fromJSON(put_package_data_text)
```

Check the results for the changed metadata attribute and newly uploaded file<br>

```r
# Check for errors
if(!http_error(put_package_data) ){
  attributes(put_package_data_json)
  print(put_package_data_json$detail)
  print(put_package_data_json$errors)
  print(put_package_data_json$viewUrl)
  print(put_package_data_json$dataset$datePublished)
  print(put_package_data_json$dataset$distribution)
}else {
  http_status(put_package_data)
  print(put_package_data_text)
}

[1] "Data Package updated successfully."
NULL
[1] "https://data-dev.ess-dive.lbl.gov/view/ess-dive-XXXX-20190621T191953176893"
[1] "2019"
               name encodingFormat
1 <your first file> text/csv
2 <your second file> text/csv
```

Check for errors and view the data frame on success

```r
# Check for errors
if(!http_error(post_package) ){
  print(get_package_json)
}else {
  http_status(post_package)
}
```

{% endtab %}

{% tab title="Java" %}

## **Edit Metadata and Data in Java**

Use the PUT function to update a dataset.  This example updates the date published to 2019 of a dataset and adds a new data file.

```java
    String id = "<Enter-your-dataset-id>";
    JSONObject JSON_LD_update = new JSONObject();
    JSON_LD_update.put("name","Updated dataset");

    try{
        String url = base + endpoint + "/" + id;
        HttpPut uploadFile = new HttpPut(url);
        uploadFile.addHeader("Authorization", header_authorization);
        File file_to_upload = new File("/directory-to-your-file/file");
  
        String content = FileUtils.readFileToString(file_to_upload, "UTF-8");
        
        
        FormBodyPart bodyPart = FormBodyPartBuilder.create()                    
        .setName("files")
        .addField("Content-Disposition", "form-data; name=\"data\"; filename=\"<your-file-name>\"")
        .setBody(new StringBody(content, ContentType.TEXT_PLAIN))
        .build();

        MultipartEntityBuilder builder = MultipartEntityBuilder.create()
        .setMode(HttpMultipartMode.BROWSER_COMPATIBLE)
        .setContentType(ContentType.MULTIPART_FORM_DATA);
        
        builder.addPart("json-ld", new StringBody(JSON_LD_update.toString(), ContentType.TEXT_PLAIN));
        builder.addPart(bodyPart);


        uploadFile.setEntity(builder.build());
        
        CloseableHttpResponse response = httpClient.execute(uploadFile);
        HttpEntity responseEntity = response.getEntity();

        String responseString = EntityUtils.toString(responseEntity, "UTF-8");
```

Check the results for the changed metadata attribute and newly uploaded file

```java
    String id = "<Enter-your-dataset-id>";
    JSONObject JSON_LD_update = new JSONObject();
    JSON_LD_update.put("name","Updated dataset");

    try{
        String url = base + endpoint + "/" + id;
        HttpPut uploadFile = new HttpPut(url);
        uploadFile.addHeader("Authorization", header_authorization);
        File file_to_upload = new File("/directory-to-your-file/file");
  
        String content = FileUtils.readFileToString(file_to_upload, "UTF-8");
        
        
        FormBodyPart bodyPart = FormBodyPartBuilder.create()                    
        .setName("files")
        .addField("Content-Disposition", "form-data; name=\"data\"; filename=\"<your-file-name>\"")
        .setBody(new StringBody(content, ContentType.TEXT_PLAIN))
        .build();

        MultipartEntityBuilder builder = MultipartEntityBuilder.create()
        .setMode(HttpMultipartMode.BROWSER_COMPATIBLE)
        .setContentType(ContentType.MULTIPART_FORM_DATA);
        
        builder.addPart("json-ld", new StringBody(JSON_LD_update.toString(), ContentType.TEXT_PLAIN));
        builder.addPart(bodyPart);


        uploadFile.setEntity(builder.build());
        
        CloseableHttpResponse response = httpClient.execute(uploadFile);
        HttpEntity responseEntity = response.getEntity();

        String responseString = EntityUtils.toString(responseEntity, "UTF-8");
        // Review the results
        System.out.println(responseString);
    } catch (Exception ex) {
        System.out.print(ex.getMessage().toString());
    }
    if(response.getStatusLine().getStatusCode() == 200){
        System.out.println(response.toString());
        System.out.println("package updated");
        System.out.println(responseString);
    } else {
        System.out.println(response.getStatusLine().getReasonPhrase());
        System.out.println(responseString);
    }
    } catch (Exception ex) {
        System.out.print(ex.getMessage().toString());
    }
```

{% hint style="info" %}
At the end of the document, don’t forget to close the class and main function blocks by adding two curly braces (`}}`) if you hadn’t done that already!
{% endhint %}

To compile the code and run it, make sure you’re on the parent directory where the `essdive.java` file is **not inside the lib folder**.

Now assuming [Java](https://www.java.com/en/download/) is already installed on your machine, we will start by compiling the java code you wrote using the following terminal command:

`javac -cp .:"lib/*" essdive.java`

This will create a new file that has the compiled code where it can run using the following command:

`java -cp .:"lib/*" essdive`
{% endtab %}
{% endtabs %}


# Link to External Data Sources

Connect data files and metadata stored in other repositories to your ESS-DIVE dataset. Read more to learn how External Linking works and when to use it.

## What is External Linking?

External Linking allows data contributors to connect data files and metadata stored in other repositories to ESS-DIVE datasets. It enables data to be stored where it makes the most sense scientifically and practically, while also following ESS Data Management and Sharing Policy to store searchable metadata on ESS-DIVE.

An external link is easily accessible from an ESS-DIVE dataset landing page and has a clearly defined relationship to the dataset. With this feature, your ESS-DIVE dataset can be linked to data files and metadata that have already been published on another established data repository or to data files that cannot be uploaded to ESS-DIVE.&#x20;

Not all external data or metadata is suitable for external linking on ESS-DIVE. All externally linked datasets will be reviewed by the ESS-DIVE Team before publication and your external links may not be accepted. In this documentation, we will review the types of external data and metadata that is acceptable to link to as well as how to add external links to your dataset.&#x20;

![Figure 1: All external data files are listed in a new table underneath the existing file table.](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FAze9kk9coAtHw5PTT4J7%2Fexternally-linked-dataset.png?alt=media\&token=ce8b8fc5-954a-4e9a-adcd-45284c1ec919)

### External Link Table

Datasets with external links will have an external link table underneath the existing file table. The external link table follows a specific format and each link will have the following three components:

1. A brief but meaningful description for this web page,
2. The relationship type and it's definition, and
3. The URL where the data/metadata resides.

{% hint style="info" %}
ESS-DIVE uses shared vocabulary from [Schema.org](https://schema.org/) to define external linking relationship types. This vocabulary is controlled, machine readable, and can be understood by most major search engines. ESS-DIVE has approved three Schema.org vocabularies for use with external linking. For more information, see the [Relationship Types](#relationship-types) section.
{% endhint %}

## Why Link External Data?

There are numerous reasons why linking out to external data sources may be advantageous for certain dataset publications. In this section, we will review the reasons why external linking may be needed and the common examples in which it is used.

* Store your data where it makes the most sense scientifically or for practical reasons (e.g. project data archive), while also complying with ESS data policy to at least store metadata describing your data on ESS-DIVE.
* Generally enhance the discoverability of your dataset by making it searchable on ESS-DIVE, along with other ESS project data stored on ESS-DIVE. &#x20;
* Your data file storage volume is greater than 500GB and is too large to upload directly to ESS-DIVE. See more details under the [Large Data Files examples](#id-4.-large-data-files).
* Your published data product is complex, or involves analysis tools used on another platform, and can be more easily accessed from the external source (e.g. model code stored on GitHub). See more details under the [Data Analysis Platforms examples](#id-3.-software-and-data-analysis-platforms).

*If your data product is not represented in this section, please refer to the* [*Relationship Types*](#relationship-types) *section to see if one of the available relationships suit your data product.*&#x20;

### Examples

#### 1. Long-Term Data Repositories or Data Systems

External data stored in valid data repositories that **provide long term data storage and stewardship of data** does not need to be uploaded to ESS-DIVE. But again, if your project is funded by the DOE ESS program, you must at least store metadata describing the data on ESS-DIVE, with links to the published dataset. Some data repositories or data systems that are approved for external linking are:

* Environmental Data Initiative (EDI)
* National Microbiome Data Collaborative (NMDC)
* USGS ScienceBase

{% hint style="info" %}
If your data is stored in a repository not listed here, please reach out to the ESS-DIVE Support Team at <ess-dive-support@lbl.gov> to find out if that repository can be linked to your ESS-DIVE datasets.
{% endhint %}

#### 2. Project Data Archives

Some ESS projects have archives where they upload and manage their data products. If these data are publicly accessible, you may take advantage of our external linking feature to store metadata on ESS-DIVE, with links to the original project data archive.&#x20;

Some examples of projects archives are:

* Spruce and Peatland Responses Under Changing Environments (SPRUCE)&#x20;
* Next-Generation Ecosystem Experiment (NGEE) Arctic
* Next-Generation Ecosystem Experiment (NGEE) Tropics

{% hint style="warning" %}
In cases where the project archive does not issue DOIs or is not a long-term data preservation repository, you will need to upload your data files to your ESS-DIVE datasets.&#x20;
{% endhint %}

#### 3. Software and Data Analysis Platforms

Some open-access data sources distribute data in a manner that provides useful scientific context to datasets. In some cases, it may be necessary to use External Linking to associate a dataset to the original data analysis platform. Some examples of such data sources are:

* [GitHub](https://github.com/)
* [Google Earth Engine](https://earthengine.google.com/)
* [KBase](https://www.kbase.us/)

{% hint style="warning" %}
If you are linking to data on a platform that does not issue DOIs, you must upload a copy of your data files to ESS-DIVE
{% endhint %}

#### 4. Large Data Files

For certain datasets, the associated data files are too large to upload to and/or download from ESS-DIVE. In these cases, you may take one of two options. Either (1) upload your data to an established long term data storage service that can take your large data or (2) work with the ESS-DIVE team to upload this data to an external data source that is managed by the ESS-DIVE Team; we call this Large Data Storage.&#x20;

{% hint style="info" %}
Data files **greater than 500GB** cannot be uploaded or downloaded from ESS-DIVE. Please contact the ESS-DIVE Team at <ess-dive-support@lbl.gov> if you have a dataset with more than 500GB of data, or if you have a large number of data files.
{% endhint %}

## Relationship Types

ESS-DIVE currently supports three types of relationships between ESS-DIVE datasets and external data sources. In this section we list the approved relationships for your reference. You can read through the following descriptions in Table 1 to see if your external data can be linked to your ESS-DIVE dataset.

**If you feel like your data product falls into one of these three categories, email The ESS-DIVE Team at <ess-dive-support@lbl.gov> and we will help associate your data with the appropriate  type.**

<table><thead><tr><th width="203">Relationship</th><th>Description</th></tr></thead><tbody><tr><td><strong>same as</strong></td><td>Original publication of this dataset (where the data+metadata can be found). The DOI url of this dataset, starting with '<a href="https://doi.org/">https://doi.org/</a>', redirects to original source.</td></tr><tr><td><strong>archived at</strong></td><td>Complete copy of the data files in the dataset resides in external source.</td></tr><tr><td><strong>has part</strong></td><td>One or more files that are part of the dataset held outside of the repository. This could be a link to an individual file or a directory.</td></tr></tbody></table>

***Table 1**: Lists ESS-DIVE's current external linking relationship types and their descriptions*

{% hint style="info" %}

#### Don't see a relationship type that is applicable to your use case?

We have the infrastructure to accommodate your needs! Please email the ESS-DIVE Support Team (<ess-dive-support@lbl.gov>) and share your use case with us. We can discuss expanding the list of acceptable external data relationships to accommodate your use case.
{% endhint %}

## How to Link Datasets

### Make a Request

External Linking is managed by the ESS-DIVE Team. To externally link your dataset to an external data product, please contact <ess-dive-support@lbl.gov> to start the process. This section outlines the steps involved in making and completing the request.&#x20;

1. **Email** [**ess-dive-support@lbl.gov**](mailto:ess-dive-support@lbl.gov) to start a request, with details on why you need external linking and any existing DOI(s), if you have them.&#x20;
   1. When the request is initiated, The ESS-DIVE Team will ask that you provide a brief but meaningful description for each web page, as described in the [External Link Table](#undefined) section.
   2. *During this stage, the ESS-DIVE Team will also evaluate whether your external data/metadata is suitable to be externally linked. Not all requests are approved.*
2. If there is an existing DOI, **the ESS-DIVE Team** uses a programmatic tool to transfer the metadata from the original repository (e.g. EDI) to one or more new ESS-DIVE dataset(s).
3. If there is NOT an existing DOI, **you will create a new dataset** for your data and send the associated ESS-DIVE identifier to the ESS-DIVE Team (*e.g. ess-dive-ea25aaddf4b47a3-20211103T152543118*).
4. **The ESS-DIVE Team** will populate the External Links table in your dataset, including the appropriate relationship type(s).
   1. *The ESS-DIVE Team will select an appropriate relationship from* [*Table 1*](#relationship-types) *and apply it based on your dataset publication needs.*&#x20;
5. **The ESS-DIVE Team** will send you the dataset URL and give you ownership.
6. **Review the dataset and complete your metadata**. Our automated assessment report can help in your review. If you had an existing DOI and transferred your metadata, note that programmatic metadata transfers are not comprehensive and there will be missing information in your dataset.
7. **Request** to publish the dataset.

### Use the API

ESS-DIVE's Package Service API can be used to programmatically and autonomously add external links to your datasets. The process for adding external links to datasets via the API only differs slightly from the standard tutorial for creating or updating datasets with the API. In these instructions, we will reference and expand on our existing tutorials. Detailed instructions on the use of the Package Service API can be found in the [Package Service Tutorials](/programmatic-tools/ess-dive-dataset-api).

1. Head to our [Package Service Tutorial](/programmatic-tools/ess-dive-dataset-api) documentation page to learn how to get started with the API for the first time
2. Select an example page in your preferred coding language ([R](https://www.r-project.org/), [Python](https://www.python.org/), or [Java](https://www.java.com/en/))&#x20;
   1. *For the remainder of these instructions, we will provide references to the Python tutorial example. You can find a version of each of the linked Python sections in the R and Java tutorials as well.*
3. Follow the [Setup](https://app.gitbook.com/o/-LgU7CB5CVrJQMt0aaNu/s/-LgU7GrufgIOpY1Ufd5R/~/edit/~/changes/210/contributing-data/submit-data-with-the-dataset-api/code-examples#setup) instructions
4. Create new or copy your existing dataset [JSON-LD](https://json-ld.org/) (i.e. your dataset metadata)
   1. **If you are creating a dataset** for the first time, skip to [Create Metadata](https://app.gitbook.com/o/-LgU7CB5CVrJQMt0aaNu/s/-LgU7GrufgIOpY1Ufd5R/~/edit/~/changes/210/contributing-data/submit-data-with-the-dataset-api/code-examples#create-metadata) and follow the instructions for creating JSON-LD for your dataset metadata
   2. **If you are updating a dataset**, you can use the [Get a Single Dataset](broken://pages/-LgjD0Fi7ngZ-DwMcOLP#get-a-single-dataset) code example to copy the JSON-LD for your existing dataset
5. Once you have your JSON-LD, you can now append the external linking schema onto it. Read through the available relationship types ([Table 1](/contributing-data/link-to-external-data-sources#relationship-types)) and decide which suits your dataset.&#x20;
   1. To learn how to format your selected external linking relationship into your JSON-LD, navigate to [ESS-DIVE's technical API documentation](https://api.ess-dive.lbl.gov/) and locate your selected relationship in the dataset schema (Figure 2); either `hasPart`, `archivedAt`, and/or `sameAs`
   2. An example of each relationship's format is provided in code snippets at the end of this section (all code snippets are in Python)
6. Once you have added your external links, submit your dataset using the API!
   1. If you are **submitting a new dataset without data files**, skip to Create a Dataset > [Metadata Only](broken://pages/-LgjD0Fi7ngZ-DwMcOLP#metadata-only) and copy the code examples
   2. If you are **submitting a new dataset with data files**, skip to Create a Dataset > [Metadata and Data Files](broken://pages/-LgjD0Fi7ngZ-DwMcOLP#metadata-and-data) or [Metadata and Many Data Files](broken://pages/-LgjD0Fi7ngZ-DwMcOLP#metadata-and-many-data-files) and copy the code example
   3. If you are **updating an existing dataset metadata** (without data files), skip to Update a Dataset > [Metadata Only](broken://pages/-LgjD0Fi7ngZ-DwMcOLP#metadata-only-1) section and copy the code example
7. Head to ESS-DIVE ([https://data.ess-dive.lbl.gov/](https://data.ess-dive.lbl.gov/data)) and **request to publish** your dataset
   1. *During the publication process, your external links will be reviewed by the ESS-DIVE Team for suitability. At this stage, we will help refine your external links as needed or we may determine that your external data/metadata are not suitable for external linking. Not all external links will be approved.*

![Figure 2: The complete dataset schema can be found under the schema dropdown bar on the Package Service API's technical documentation page.](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FdXobQxLXF7QukVgtFX8t%2Fdataset-schema.png?alt=media\&token=3a92aabb-e666-4fe3-825f-9e53eb66bf4c)

*An example of the `has part` relationship JSON-LD schema.*

```python
hasPart = {
   "@type": "Webpage",
   "name": "Code developed in this GitHub repository",
   "url": "https://github.com"
   }
```

*An example of the `archived at` relationship JSON-LD schema.*

```python
archivedAt = {
   "@type": "Webpage",
   "name": "All data is also available at GoogleEarthEngine",
   "url": "https://earthengine.google.com"
   }
```

*An example of the `same as` relationship JSON-LD schema.*

```python
sameAs = "https://doi.org/10.XXX/NNNNN"
```

## More Information

* All externally linked datasets **must have complete metadata on ESS-DIVE**.&#x20;
* Externally linked datasets **may or may not have data files attached**, depending on your use case and needs. &#x20;
  * *For example, external links from data services or repositories that provide long term storage and stewardship of data will not be required to upload data files to ESS-DIVE.*
* When data is originally published elsewhere with a DOI, the DOI url will resolve to the original external data source.

  * \_By default, when a dataset is originally published on another repository, the standard DOI url (e.g. <https://doi.org/_&#x31;0.3334/CDIAC/SPRUCE.042>) *will not direct them to ESS-DIVE. Please contact the ESS-DIVE Support Team (<ess-dive-support@lbl.gov>)  if you would like your DOI to resolve to your ESS-DIVE dataset.*


# Check Dataset Metadata Quality

Assessment Reports provide automated dataset metadata quality checks based on FAIR (Findable, Accessible, Interoperable, Reusable) data principles.

ESS-DIVE’s **Assessment Reports** provide automated feedback on metadata quality and should be used by dataset creators to assess their metadata before requesting publication. Datasets must pass all required checks to meet ESS-DIVE’s publication requirements (Figure 1).

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2F0tTiLlBI6plpKZ6Hga5x%2FAssessmentReport_Fig1_202510.png?alt=media&amp;token=8b528319-0833-4684-a8e6-9b42050e95a8" alt=""><figcaption><p>Figure 1. Assessment Report Score and Checks</p></figcaption></figure>

## Assessment Report Automated Review

Dataset metadata undergo an automated review whenever changes are submitted to ESS-DIVE. The automated review uses a standard suite of checks designed based on FAIR (Findable, Accessible, Interoperable, Reusable) data principles (Table 1). The Metadata Assessment Report displays the check results and compiles a score for each FAIR category (e.g. 93/100/100/50).&#x20;

ESS-DIVE reviewers use the Assessment Report scores during the formal review process to assess the quality of metadata before publication. Not all ESS-DIVE dataset publication requirements are contained within the Assessment Report. For a complete list of both manual and automated metadata checks, see the [Dataset Requirements page](/contributing-data/package-level-metadata).

Before publication, the report is only available to the dataset creator and those who have [shared access to the private dataset](/manage-data/share-data-permissions/share-datasets). The Assessment Report becomes available to the public once the dataset is published.

## Check Score while Drafting Metadata

**Before requesting publication for a dataset**, data contributors should review the Assessment Report and address any failed or warning checks. The report provides data contributors immediate feedback on their metadata quality and instructions to revise specific metadata fields. By improving the FAIR category scores before starting a formal review, data contributors **can expedite the overall publication process**.

Please note that assessment reports can take a few minutes, or up to 24 hours, to generate.&#x20;

To access an assessment report, navigate to a dataset landing page on ESS-DIVE and select the "Assessment Report" button on the right-hand side of the landing page (Figure 2).&#x20;

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2F7m75XkMTYpKx2GdJlmyP%2FScreenshot%202023-03-30%20at%2011.20.21%20AM.png?alt=media&amp;token=c80cc067-3ab3-44ce-a7f4-c183fdc5b763" alt=""><figcaption><p>Figure 2. Assessment Report Button on Dataset Page</p></figcaption></figure>

## Resolve Failed Checks and Warnings

Failed and warning checks will impact Assessment Report scores. All checks will state any requirements and steps to address the issue. Use the provided instructions to revise your metadata.

Failed checks indicate that a **required field** does not follow the automated check criteria (Figure 3).&#x20;

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FXTLSPlLj4BMfahvwROcw%2FAssessmentReport_FailChecks_202510.png?alt=media&amp;token=92715255-0a53-4900-bf41-837c6339c3be" alt=""><figcaption><p>Figure 3. Failed Automated Checks</p></figcaption></figure>

Warning checks indicate that an **optional field** does not follow the automated check criteria (Figure 4).&#x20;

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FysQGCDF5lXBsMEmAYiGw%2FAssessmentReport_WarningChecks_202510.png?alt=media&amp;token=c17dec5d-9a0f-4569-ae45-62e5e950a78c" alt=""><figcaption><p>Figure 4. Warning Automated Checks</p></figcaption></figure>

After revising and resubmitting your dataset, check your score again to ensure you’ve addressed all issues. If you see any message other than the notice that the score is loading (Figure 5), please [contact ESS-DIVE support](https://ess-dive.lbl.gov/contact/).

<figure><img src="https://lh4.googleusercontent.com/5bn17R4TDyFvrGIu0Dv2eev-24mnyaThmb8FY3niSgLg3UPA7P_fERAoRUjUwyDQQSI1MqibAfR0V1xWgf_rK0p_9hMVKiIr63p6vTdS3XajG7MXz-42057sm9V20fukxleAhS7Zz-51UwyY6lbYujo" alt=""><figcaption><p>Figure 5. Assessment Report Loading Page</p></figcaption></figure>

## Automated Checks

The checks below are run on each dataset upon submission as a part of the ESS-DIVE automated check suite. Review the[ Dataset Requirements Page](https://docs.ess-dive.lbl.gov/contributing-data/package-level-metadata) for more detailed descriptions, formatting requirements, and examples for the automated checks.

Informational checks do not impact the Assessment Report score.

| Criteria                                                                                                         | Required/Optional | FAIR Category |
| ---------------------------------------------------------------------------------------------------------------- | ----------------- | ------------- |
| **Title** length between between 7 and 40 words                                                                  | Required          | Findable      |
| **Abstract** length is at least 100 words                                                                        | Required          | Findable      |
| **Publication date** is present                                                                                  | Required          | Findable      |
| **Creators**, at least one is present                                                                            | Required          | Findable      |
| **Dataset Contact**, ensure contact is present                                                                   | Required          | Accessible    |
| **Dataset Contact**, ensure ORCiD is provided                                                                    | Required          | Accessible    |
| **Start and End Dates** are present                                                                              | Required          | Findable      |
| **Project name** is from controlled list                                                                         | Optional          | Findable      |
| **Funding organization** "U.S. DOE > Office of Science > Biological and Environmental Research (BER)" is present | Optional          | Findable      |
| **Geographic Description** is present                                                                            | Optional          | Findable      |
| **Coordinates** describing the point location or geographic area of the dataset are present                      | Optional          | Findable      |
| **Methods** description is more than 7 words in length                                                           | Required          | Reusable      |
| **Data file formats** are non-proprietary                                                                        | Optional          | Reusable      |
| **Usage rights** is set to Creative Commons CC-BY license                                                        | Optional          | Reusable      |
| Informational: Number of contacts with email addresses provided                                                  | Informational     | Findable      |
| Informational: Number of creators with email addresses provided                                                  | Informational     | Findable      |
| Informational: Number of data entities present                                                                   | Informational     | Interoperable |

<p align="center"><em>Table 1: List of automated checks performed by the automated assessment suite</em></p>


# Dataset and DOI Status Badges

Data contributors can review dynamic dataset statuses on their dataset landing pages prior to and after publication. Login is required to view statuses.

Every dataset on ESS-DIVE moves through various statuses during its lifecycle and the key statuses relevant to publication are represented with icons on the top right of dataset landing pages (Figure 1). From creating a new dataset to iterating on drafts and publication, the next steps on your publication management journey depend on the latest status of your dataset. ESS-DIVE status badges let you know exactly where you are in the publication process, so you can efficiently manage your datasets.&#x20;

There are two primary status categories needed to plan your publication: <mark style="color:blue;">**Dataset status**</mark> and <mark style="color:blue;">**DOI status**</mark>.

{% hint style="info" %}
**Statuses can only be viewed by editors or managers of a dataset**. Login is required to view the status badges.\
:bulb: Learn more about [Data Permissions](/manage-data/share-data-permissions)
{% endhint %}

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FFNTvSAlwCvAZrnMq2Ojh%2Fnew-badges-PRIVATE-highlight.png?alt=media&amp;token=016d0961-12d0-40b4-8723-0753026c33db" alt=""><figcaption><p><em>Figure 1: Dataset landing page with status badges pictured; this is the starting status for any newly created dataset.</em></p></figcaption></figure>

## Dataset Statuses

**The dataset status captures where a dataset is in the ESS-DIVE publication process** (Table 1)**.** Once a dataset is reviewed by ESS-DIVE and is published, the data and metadata will be accessible to the public through the ESS-DIVE data search page (<https://data.ess-dive.lbl.gov/>).&#x20;

<table><thead><tr><th width="133">Icon</th><th width="151">Label</th><th>Description</th></tr></thead><tbody><tr><td><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FdkF86alLlhgR7J9g0wwF%2Fbadge-drafting.png?alt=media&amp;token=6a00d52e-5b6f-4538-bf47-9f9e32b62c08" alt=""></td><td>Drafting</td><td>The dataset is in progress and has not yet been submitted for publication review.</td></tr><tr><td><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2Fc61w6JwzA38t3cy26hob%2Fbadge-inReview.png?alt=media&amp;token=ce2acd2c-b660-45d7-9b62-4dd8773cea56" alt=""></td><td>In Review</td><td>The dataset is finalized and currently in publication review. An ESS-DIVE reviewer will reach out with proposed changes via email. Learn more about ESS-DIVE reviews here: <a data-mention href="/publish-data/review-cycle-and-criteria">Review Cycle and Criteria</a></td></tr><tr><td><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FyT2GBicIxSwodBHNkU7m%2Fbadge-published.png?alt=media&amp;token=e1472f9f-3d81-4e3e-a577-130e019718d4" alt=""></td><td>Published</td><td>The dataset has been accepted for publication and is openly accessible on ESS-DIVE.</td></tr></tbody></table>

*Table 1: Descriptions for all possible dataset statuses.*

## DOI Statuses

**The DOI status captures the state of the DOI associated with the dataset** (see Table 2)**.** DOIs can be reserved before publication or assigned upon publication by ESS-DIVE using either a new or existing DOI.

\
While we encourage those who require a DOI before review or before publication to reserve one, it is important to clarify that **the DOI will not direct to your dataset landing page until after the data has been reviewed and published**.

<table data-full-width="false"><thead><tr><th width="133">Icon</th><th width="163">Label</th><th>Description</th></tr></thead><tbody><tr><td><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2F6j0FlkP6kgAh0bt3AHDH%2Fbadge-no-doi-added.png?alt=media&amp;token=d09b896d-2a32-491b-bf81-6f553a350edf" alt=""></td><td>No DOI Added</td><td><p><strong>No DOI is associated with the dataset.</strong><br></p><p>The dataset DOI can be set by publishing a dataset, providing an existing DOI, or reserving a DOI. <a href="/publish-data/publish-your-dataset">Learn more here.</a></p></td></tr><tr><td><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FUJFAvuEUJq8RjDWMsWe0%2Fbadge-reserved.png?alt=media&amp;token=6ce9454b-31b4-4f03-b890-da71cec06fef" alt=""></td><td>Reserved</td><td><p><strong>A DOI has been reserved for this dataset.</strong></p><p></p><p><strong>This DOI will not function</strong> until the dataset has been reviewed and published by ESS-DIVE.</p></td></tr><tr><td><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FK1gy8ipepRmCsdTFLg9H%2Fbadge-active.png?alt=media&amp;token=3dd05c55-59f0-4985-8b33-dd6bd59eb40d" alt=""></td><td>Active</td><td><p><strong>The dataset DOI is live and accessible to the public.</strong><br></p><p>Users can access the ESS-DIVE dataset landing page. ESS-DIVE manages this DOI and regularly updates your DOI record with OSTI.</p></td></tr><tr><td><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2Frs27LjuQ3G11LkUkMa71%2Fbadge-NonESSDIVEDOI.png?alt=media&amp;token=23cd574f-78b5-4236-82a5-fea6f46458cf" alt=""></td><td>Non ESS-DIVE DOI</td><td><p><strong>The DOI associated with this dataset is not managed by ESS-DIVE.</strong></p><p></p><p>This dataset was originally published with a DOI on another repository. The DOI is not managed by ESS-DIVE, therefore no status is available.</p></td></tr></tbody></table>

*Table 2: Descriptions for all possible DOI statuses.*


# Review Cycle and Criteria

The ESS-DIVE Team reviews all datasets before publication to assess the quality of metadata and any reporting format files.

#### :clock1: How much time does it take to publish a dataset?

The amount of time required to create and publish a new dataset will vary. Once you let ESS-DIVE know that your dataset is ready for review by [requesting publication](#id-3.-request-publication) (step 3), the publication process can take as much as 3 weeks on average depending on dataset quality, complexity, and author responsiveness.&#x20;

#### :warning: Remember to review the [ESS-DIVE terms of use](https://ess-dive.lbl.gov/about/terms/) and ensure there is no [Protected Information](https://commons.lbl.gov/display/rpm2/Controlled+and+Prohibited+Information+Categories#ControlledandProhibitedInformationCategories--1898802862) in your dataset.

***

## Review Cycle

{% hint style="info" %}
Communication regarding dataset publication reviews take place via email from **<doi@ess-dive.atlassian.net>.**
{% endhint %}

The ESS-DIVE dataset reviewer will provide direct feedback to data contributors on the quality of their dataset metadata and, if applicable, any [reporting format files](/contributing-data/data-reporting-formats). Dataset quality is evaluated based on a standard set of criteria, detailed below.&#x20;

The review cycle includes the following stages:

1. :bell:     Confirmation that your request was initiated
2. :speaking\_head::pencil: Revision requests and edits, if applicable (repeat as needed)
3. :white\_check\_mark:      Dataset approved and published

**Datasets that do not require revisions and pass the first review will automatically be published**.&#x20;

If you receive revision requests, edit your dataset directly and **communicate with ESS-DIVE when you are done by responding to the confirmation email**. Your response will notify the ESS-DIVE team that your dataset is revised and ready for another review.&#x20;

#### Review Process for Reporting Formats

The ESS-DIVE team evaluates your reporting format files during the standard review cycle. [Reporting Format requirements](/publish-data/review-cycle-and-criteria/reporting-format-requirements) help ensure that datasets will be readable and parsable by current and future advanced search and integration tools, such as the [Deep Dive API](/searching-and-accessing-data/search-with-deep-dive-api).

{% hint style="warning" %}
Your dataset publication will be delayed if you do not relay all revision updates via email.

If you do not receive the confirmation email within 24 hours of initiating your request, **email ESS-DIVE (<ess-dive-support@lbl.gov>) for assistance.**
{% endhint %}


# Reporting Format Requirements

This page provides an overview of the review process for datasets utilizing reporting formats.

ESS-DIVE’s [**Reporting Formats**](/contributing-data/data-reporting-formats) are designed to make data and metadata published on ESS-DIVE more FAIR (Findable, Accessible, Interoperable, Reusable). Consistent formatting of data and metadata enables both machines and humans to better understand and reuse valuable data.

We use reporting formats to enable advanced search within data files. Specifically, the [Fusion Database (Fusion DB)](/searching-and-accessing-data/search-with-deep-dive-api) validates, extracts and indexes data within standardized files.&#x20;

The contents of public data and metadata files successfully parsed by the FusionDB are made searchable by the [Deep Dive API](https://fusion.ess-dive.lbl.gov/), which is separate from the ESS-DIVE main search and [Dataset API](/programmatic-tools/ess-dive-dataset-api). This currently requires the use of the [File Level Metadata (FLMD)](https://ess-dive.gitbook.io/file-level-metadata-reporting-format) and [Comma Separated Values (CSV) Guidelines](https://ess-dive.gitbook.io/csv-file-structure-reporting-format) Reporting Formats. These reporting formats are widely applicable to data types stored on ESS-DIVE and ensure that data files are described through standardized metadata fields and are machine-readable. The Fusion DB provides feedback to the ESS-DIVE Publication Review Team if any requirements are not met. These requirements are outlined below. For more detailed documentation of all Reporting Formats, please [visit the ESS-DIVE Workspace GitHub](https://github.com/ess-dive-workspace).&#x20;

We plan to expand the FusionDB to incorporate data-type specific reporting formats and associated automated validations in the future.

## Reporting Format Checks

A series of checks are performed during the publication review process for datasets using reporting formats. Checks listed as required are necessary for machine readability and parsing, whereas strongly recommended and optional checks are recommended enhancements to metadata.&#x20;

Example datasets that have passed all reporting format checks are available [below](#example-datasets-using-reporting-formats-and-successfully-parsed-by-fusion-database).

<table><thead><tr><th width="142">Check Name</th><th width="129">Requirement Level</th><th>Description</th></tr></thead><tbody><tr><td>File Name</td><td>Required</td><td>File name uses only letters, numbers, and underscores. Do not include spaces and do not start with an underscore or hyphen.</td></tr><tr><td>File Description</td><td>Required</td><td>A brief description (minimum 10 characters) is provided</td></tr><tr><td>Column or Row Name</td><td>Required</td><td>Column or row names use only letters, numbers, hyphens, and underscores. Do not include spaces, and do not start with an underscore, hyphen, or number.</td></tr><tr><td>Unit</td><td>Required</td><td>Unit is present</td></tr><tr><td>Definition</td><td>Required</td><td>Description is present</td></tr><tr><td>Character Set</td><td>Required</td><td>All characters are within US-ASCII character set without extensions or UTF-8</td></tr><tr><td>Delimiter</td><td>Required</td><td>Delimiter used for file is comma and saved as a CSV file</td></tr><tr><td>Data Matrix</td><td>Required</td><td>Contents of the data portion of the file is organized in a logical and readable matrix format</td></tr><tr><td>Column or Row Name Orientation</td><td>Required</td><td>Orientation of the file is either horizontal or vertical </td></tr><tr><td>Consistent Values</td><td>Required</td><td>Text and numeric data are not mixed within the same Column or Row</td></tr><tr><td>Missing Value Codes</td><td>Required</td><td>All cells in the data matrix have a value and missing data are represented with Missing Value Codes</td></tr><tr><td>Temporal Data</td><td>Required</td><td>Date format follows ISO 8601 standard (YYYY-MM-DD, to known precision) and time format following Coordinated Universal Time (UTC) (YYYY-MM-DD hh:mm:ss, to known precision)</td></tr><tr><td>Spatial Data</td><td>Required</td><td>Geographic coordinates are provided in WGS84 decimal format</td></tr><tr><td>File naming conventions for File Level Metadata and Data Dictionary files</td><td>Required</td><td>A file within the dataset contains the following suffixes *_flmd.csv and *_dd.csv.</td></tr><tr><td>Reporting Format Keywords</td><td>Required</td><td>ESS-DIVE <a href="https://github.com/ess-dive-workspace/essdive-file-level-metadata/blob/main/RF_FLMD_Standard_Terms.csv">reporting format keywords</a> are used. The File Level Metadata reporting format keyword is required for the FusionDB to identify, validate and parse your dataset.</td></tr><tr><td>Standard</td><td>Strongly Recommended</td><td>ESS-DIVE <a href="https://github.com/ess-dive-workspace/essdive-file-level-metadata/blob/main/RF_FLMD_Standard_Terms.csv">Standard field terms</a> for reporting formats are used</td></tr><tr><td>Data Orientation</td><td>Optional</td><td>Check whether “horizontal” or “vertical” is provided within File Level Metadata file</td></tr></tbody></table>

## Example Datasets Using Reporting Formats and Successfully Parsed By Fusion Database

* Roley et al., (2023) *Data and scripts associated with "Coupled primary production and respiration in a large river contrasts with smaller rivers and streams."* [doi:10.15485/1985922](http://doi.org/10.15485/1985922)
* Jastrow et al., (2022) *Spatially Averaged Ice Contents of Ice-Wedge Polygon Cross-Sections to 3-m Depth, July 2013, Utqiagvik, Alaska* [doi:10.15485/1876898](http://doi.org/10.15485/1876898)
* Kaufman et al., (2023) *Spatial Study 2022: Water Column, Sediment, and Total Ecosystem Respiration Rates across the Yakima River Basin, Washington, USA* [doi:10.15485/1987520](http://doi.org/10.15485/1987520)
* Gooseff et al., (2023) *Riverbed and Near-Surface Water Quality Data, Hanford Reach, Columbia River, February 2021 - April 2022* [doi:10.15485/2204421](http://doi.org/10.15485/2204421)
* Hassett et al., (2023) *Carbon flux measurements from chambers collected between July to October 2022 at Old Woman Creek, Huron, Ohio* [doi:10.15485/2229438](http://doi.org/10.15485/2229438)
* Stolze et al., (2024) *Aerobic respiration controls on shale weathering, Geochimica et Cosmochimica Acta, 2023: Dataset* [doi:10.15485/1987859](http://doi.org/10.15485/1987859)
* Wang et al., (2024) *Continuous soil temperature measurements from 2019-10-4 to 2020-10-4, Teller road Mile 27, Seward Peninsula, Alaska* [doi:10.15485/2301692](http://doi.org/10.15485/2301692)
* Sala et al., (2024) *Plot and Tree Characteristics from the 2022-2023 field experiment at Game Ridge, Missoula County, Montana, USA* [doi:10.15485/2371850](http://doi.org/10.15485/2371850)
* Williams et al., (2024) *Anion Data for the East River Watershed, Colorado (2014-2023)* [doi:10.15485/1668054](http://doi.org/10.15485/1668054)


# Publish your Dataset

Learn how to initiate a metadata review, receive a DOI, and make your data accessible to the public.

By publishing a dataset, you are making your data publicly available on the ESS-DIVE main search page for search and download. **After publishing a dataset, the metadata and files cannot be private again**.&#x20;

## Manage Publication :star: Button

Every dataset must be **assigned a DOI** and **reviewed by the ESS-DIVE team** before it can be published. Datasets can be published from the dataset landing page using the <mark style="color:blue;">**Manage Publication**</mark> :star: button (Figure 1). Clicking this button opens the Manage Dataset window (Figure 2), which is your hub for requesting publication, reserving DOIs, and getting more information on where you are in the publication process.&#x20;

Your dataset will remain in the default <mark style="color:blue;">**statuses**</mark> (draft with no DOI added) until you trigger a publication action using the Manage Dataset window. The statuses will change as you move through the process.

:bulb: Learn more about [Dataset and DOI Statuses](/publish-data/dataset-and-doi-status-badges)

{% hint style="info" %}
**The publication management tool can only be accessed by managers of a dataset**. Login is required to access the tool.\
:bulb: Learn more about [Data Permissions](/manage-data/share-data-permissions)
{% endhint %}

<div><figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2F7VCL3Fv8Bq9CZgeFjPsE%2Fpub-ui-FAIR-landing-page-private.png?alt=media&amp;token=6cdeba9c-fbec-45ac-9646-1ee71babc5bb" alt=""><figcaption><p>Figure 1: You can manage DOI assignment and review requests from your dataset landing page</p></figcaption></figure> <figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FWSDvpBycS04JgpGLckiY%2Fmanage-pub-FAIR-private.png?alt=media&amp;token=34596f8e-fee1-496e-be1f-1ed91affb9e7" alt=""><figcaption><p>Figure 2: There are two primary publication actions you can choose</p></figcaption></figure></div>

***

## Publication Options

Most datasets follow the default publication process and receive a new DOI after the dataset is reviewed and published by ESS-DIVE. However, to accommodate individual publication timelines and workflows, ESS-DIVE's publication management tool gives you options to:

* Publish with New DOI (**default option**)
* Reserve a New DOI before Publication
* Publish with an Existing DOI&#x20;

### Explore Publication Options

{% tabs %}
{% tab title="Publish with New DOI" %}

### Publish with New DOI

You and your team have finalized your dataset and are ready to request publication and **receive a new DOI once the dataset has been reviewed and approved**. This is our most common use case and is the most straightforward path.

{% hint style="info" %}
Your dataset DOI will be assigned once you have completed the publication review and the dataset has been approved for publication by ESS-DIVE.
{% endhint %}

***

<mark style="color:blue;">**Is this right for my dataset?**</mark>

* You need a new DOI for your dataset, but it is not urgent
* You have finalized your dataset, and are ready for dataset review
* You can make any requested changes before receiving a DOI&#x20;
  * This process is expected to take up to three weeks, depending on responsiveness

***

**Step-by-step instructions:**

{% content-ref url="/pages/ILqmpLPIIiy1tq1vtddf" %}
[Request Publication](/publish-data/publish-your-dataset/request-publication)
{% endcontent-ref %}
{% endtab %}

{% tab title="Reserve a New DOI Before Publication" %}

### Reserve a New DOI Before Publication

You and your team are hard at work drafting your metadata and organizing your data files, but **you urgently need a DOI for your dataset** before finalizing your data and completing the publication review.&#x20;

Your dataset will be assigned a DOI before publication, but this DOI will not function until the dataset has been reviewed published.  It is your responsibility to complete the review process and work with ESS-DIVE to publish your dataset.

{% hint style="danger" %}
**You must complete the publication review process for your reserved DOI to function.**
{% endhint %}

***

<mark style="color:blue;">**Is this right for my dataset?**</mark>

* You need a new DOI for your dataset immediately, before finalizing and completing the publication review process
* You will commit to finalizing and publishing your dataset after you receive the DOI
* You understand the DOI will not function until the dataset has been reviewed and published

***

**Step-by-step instructions:**

{% content-ref url="/pages/wzOQOsBkUKOjdKVFyhdP" %}
[Reserve DOI Before Publication](/publish-data/publish-your-dataset/reserve-doi-before-publication)
{% endcontent-ref %}
{% endtab %}

{% tab title="Publish with Existing DOI" %}

### Publish with Existing DOI

You may have **already published your dataset with a DOI elsewhere** (e.g. a project archive or Zenodo) **and do not want a new DOI**, but now need to publish it on ESS-DIVE. You will need to submit an ESS-DIVE dataset, and provide an existing DOI to indicate that you do not need a new DOI.

ESS-DIVE will evaluate your DOI. Existing DOIs that are appropriate to associate with an ESS-DIVE dataset will be approved and assigned to your dataset during the review. Your dataset will become public with your existing DOI after you complete the publication review and the dataset has been approved for publication by ESS-DIVE.

*An **existing DOI** is also referred to as a **Non ESS-DIVE DOI** in the status badges. Non ESS-DIVE DOIs or existing DOIs are not owned by ESS-DIVE nor do we have authority to automatically manage them.*

***

<mark style="color:blue;">**Is this right for my dataset?**</mark>

* You already published this data on another archive or repository and received a DOI (project archive, Zenodo, Figshare)
* The data you are uploading to ESS-DIVE is the exact same as the data associated with the existing DOI with **no changes that alter the original scope or purpose**
* You have finalized your dataset on ESS-DIVE, and are ready for dataset review
* You can make any metadata changes requested in the review process

***

<mark style="color:blue;">**How can I tell if my existing DOI is suitable?**</mark>

* The data in your ESS-DIVE dataset ***is an exact copy*** of the data published under your existing DOI.
* Your existing DOI does ***not*** point to a manuscript. A manuscript DOI is not a valid data DOI. Related manuscripts should be cited in the related references field.
* Your existing DOI does ***not*** point to a related resource. Related data DOIs should be cited in the related references field.

{% hint style="info" %}
**Connect existing data publications to ESS-DIVE with&#x20;**<mark style="color:blue;">**External Links**</mark>

External data that is *a part of* the data you are publishing on ESS-DIVE, but does not contain all data, should be connected using <mark style="color:blue;">External Links</mark>. External links should also be used when you're copying all external data from *from private repositories or repositories with inadequate data preservation*. Learn more in our external link documentation.

:point\_right: [Link to External Data Sources](/contributing-data/link-to-external-data-sources)&#x20;
{% endhint %}

***

**Step-by-step Instructions:**

{% content-ref url="/pages/UctSLCMnpHqnV5TbvHIK" %}
[Publish with Existing DOI](/publish-data/publish-your-dataset/publish-with-existing-doi)
{% endcontent-ref %}
{% endtab %}
{% endtabs %}

***

## Learn More

<table data-card-size="large" data-view="cards"><thead><tr><th></th><th></th><th align="center"></th><th data-hidden data-card-target data-type="content-ref"></th><th data-hidden data-card-cover data-type="files"></th></tr></thead><tbody><tr><td></td><td></td><td align="center"><strong>Review Cycle and Criteria</strong></td><td><a href="/publish-data/review-cycle-and-criteria">Review Cycle and Criteria</a></td><td><a href="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FXq3WwY9EJ0DEB0XpbZbq%2FESS-DIVE_Header.png?alt=media&amp;token=f6c76b10-284b-4de8-bbd9-a24435001cd5">ESS-DIVE_Header.png</a></td></tr><tr><td></td><td></td><td align="center"><strong>Dataset and DOI Status Badges</strong></td><td><a href="/publish-data/dataset-and-doi-status-badges">Dataset and DOI Status Badges</a></td><td><a href="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FXq3WwY9EJ0DEB0XpbZbq%2FESS-DIVE_Header.png?alt=media&amp;token=f6c76b10-284b-4de8-bbd9-a24435001cd5">ESS-DIVE_Header.png</a></td></tr><tr><td></td><td></td><td align="center"><strong>Check Dataset Metadata Quality</strong></td><td><a href="/publish-data/check-dataset-metadata-quality">Check Dataset Metadata Quality</a></td><td><a href="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FXq3WwY9EJ0DEB0XpbZbq%2FESS-DIVE_Header.png?alt=media&amp;token=f6c76b10-284b-4de8-bbd9-a24435001cd5">ESS-DIVE_Header.png</a></td></tr><tr><td></td><td></td><td align="center"><strong>Link to External Data Sources</strong></td><td><a href="/contributing-data/link-to-external-data-sources">Link to External Data Sources</a></td><td><a href="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FXq3WwY9EJ0DEB0XpbZbq%2FESS-DIVE_Header.png?alt=media&amp;token=f6c76b10-284b-4de8-bbd9-a24435001cd5">ESS-DIVE_Header.png</a></td></tr></tbody></table>


# Request Publication

Request a publication review to make your dataset and DOI available to the public.

## 1. Finalize Dataset

Datasets must complete all required metadata fields and pass all required quality checks before they can be published on ESS-DIVE. Ensuring that your dataset has complete metadata **before** requesting publication will expedite the publication review process.

:bulb:Learn more about [ESS-DIVE's Metadata Requirements](/contributing-data/package-level-metadata).

## 2. Open Manage Publication Window

From your dataset landing page, use the <mark style="color:blue;">**Manage Publication**</mark> :star: button on the right side of the screen below the title and header.

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2F7VCL3Fv8Bq9CZgeFjPsE%2Fpub-ui-FAIR-landing-page-private.png?alt=media&amp;token=6cdeba9c-fbec-45ac-9646-1ee71babc5bb" alt=""><figcaption><p>Figure 1: You can manage when and how to publish your dataset from your dataset landing page</p></figcaption></figure>

This will open the “Manage Dataset” main panel. Click the <mark style="color:blue;">**Start Publication Process**</mark> button.

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FpN2JxYCnlzOWZrP4OObM%2Fmanage-pub-FAIR-regPublication-private.png?alt=media&amp;token=b17057c9-6b47-4150-9b79-c6ef44f1e0ad" alt=""><figcaption><p>Figure 2: There are two primary publication actions you can choose from. Select "Start publication process" to create a request for your dataset to be reviewed. </p></figcaption></figure>

{% hint style="info" %}
In the main panel, you can also see your [dataset status](/publish-data/dataset-and-doi-status-badges#dataset-statuses) and [DOI status](/publish-data/dataset-and-doi-status-badges#doi-statuses).
{% endhint %}

## 3. Review and Commit to Agreements

You will then be asked to **confirm that you’re ready to initiate the review process and make the dataset public** on ESS-DIVE and have reviewed the metadata assessment for any recommended improvements.

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FG1Hwr9kDwBreWp1QhMh2%2FRegularPub-agreements.png?alt=media&amp;token=e91070b7-5485-49c1-b6f3-974502a59432" alt=""><figcaption><p>Figure 3: Before you submit a publication request, please read through the informational warnings and agreements</p></figcaption></figure>

{% hint style="info" %}
Learn more about [metadata and data quality standards](/publish-data/check-dataset-metadata-quality)
{% endhint %}

## 4. Publication Request Confirmation

When you click <mark style="color:blue;">**Request Publication**</mark>, you’ll be presented with a confirmation message that your dataset has been added to the review queue to be reviewed by an ESS-DIVE team member.&#x20;

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FrQSJq71IQkxpqxFoBW48%2Fmanage-pub-FAIR-requestPub-confirmation.png?alt=media&amp;token=2bc4c694-34b1-42b1-a07b-8e5349d2bcc7" alt=""><figcaption><p>Figure 4: A confirmation message will appear when your request has been sent, see message for important details</p></figcaption></figure>

Your dataset status will now change to <mark style="color:blue;">**In Review**</mark>. You will be automatically notified via email within 24 hours that we have received your request.

{% hint style="warning" %}
Contact <ess-dive-support@lbl.gov> if you haven’t received a confirmation email in 24 hours.
{% endhint %}

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FhlyYcRaN77NJ2EG70inP%2Fpub-ui-FAIR-private-inReview.png?alt=media&amp;token=7d70cec4-2f44-4e05-adc7-00d7444a900c" alt=""><figcaption><p>Figure 5: The <strong>Dataset Status</strong> will change to <strong>In Review</strong> after you submit a publication review request</p></figcaption></figure>

## 5. Respond to Revision Requests

An ESS-DIVE team member will review your dataset and reach out via your confirmation email with any revision requests. Learn more about how reviews work:

{% content-ref url="/pages/bh4jaKJAXmpdsVdJRGzS" %}
[Review Cycle and Criteria](/publish-data/review-cycle-and-criteria)
{% endcontent-ref %}

{% hint style="warning" %}
You must respond to the review email when you have completed any requested revisions before your dataset will be published.
{% endhint %}

## 6. Approved & Published

Once approved, your dataset will be assigned a new DOI and made public. Your dataset status will now change to <mark style="color:blue;">**Published**</mark> with an <mark style="color:blue;">**Active**</mark> DOI status (Figure 6). You will be notified via email that your dataset has been approved and published.

{% hint style="info" %}
If you **reserved a DOI before publication**: your dataset will continue to use the DOI that you initially reserved and will be made public at this time. Your dataset status will change to <mark style="color:blue;">**Published**</mark> with an <mark style="color:blue;">**Active**</mark> DOI status (Figure 6).

If you **published with an existing DOI**: your dataset will be assigned with the existing DOI that you provided and will be made public at this time. Your dataset status will change to <mark style="color:blue;">**Published**</mark> with a <mark style="color:blue;">**Non ESS-DIVE DOI**</mark> DOI status (Figure 7).&#x20;
{% endhint %}

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FvrKzdauIMN3TOkdZ4I7Q%2Fnew-badges-published-dataset.png?alt=media&amp;token=6575b6c8-4c9e-45e7-87c6-eb83734615c6" alt=""><figcaption><p>Figure 6: Datasets published with DOIs provided by ESS-DIVE, either through reserving a DOI before or receiving one the time of publication, will have an "Active" DOI status.  </p></figcaption></figure>

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FdVsJtzUEzEb4G5LQtyAD%2FExistingDOI-Landing-Page-Published.png?alt=media&amp;token=6084f299-7a60-4219-9c6b-02f77377b5b4" alt=""><figcaption><p>Figure 7: Datasets published with an existing DOI will have the "Non ESS-DIVE DOI" status badge. The DOI you provided will appear in the dataset header as well as in the "Alternate Identifiers" field.</p></figcaption></figure>


# Reserve DOI Before Publication

Obtain a new DOI for your private dataset prior to publication. You can return at a later date to finalize your dataset and request a publication review to make your data and DOI publicly available.

{% hint style="info" %}
This is not required to publish a dataset
{% endhint %}

## 1. Open Manage Publication Window

From your dataset landing page, use the <mark style="color:blue;">**Manage Publication**</mark> :star: button on the right side of the screen below the title and header section.

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2Fi6bhuzt7opNQLlDNCFff%2Fpub-ui-Templates-landing-page-private.png?alt=media&amp;token=d430214d-d0a2-4c78-8b4e-3452f657e476" alt=""><figcaption><p>Figure 1: You can manage when and how to publish your dataset from your dataset landing page</p></figcaption></figure>

This will open the “Manage Dataset” main panel. Click the <mark style="color:blue;">**Reserve DOI**</mark> button.&#x20;

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2F4zmzGxqxr5zGMboBKHMF%2Fmanage-pub-Templates-reserve.png?alt=media&amp;token=39e9dda4-f1c7-44a4-8ccb-82863e80d03d" alt=""><figcaption><p>Figure 2: There are two primary publication actions you can choose from. Select "Reserve DOI" to reserve a new DOI for your private dataset. Reserved DOIs do not function until datasets are public.</p></figcaption></figure>

{% hint style="info" %}
In the main panel, you can also see your [dataset status](/publish-data/dataset-and-doi-status-badges#dataset-statuses) and [DOI status](/publish-data/dataset-and-doi-status-badges#doi-statuses).
{% endhint %}

## 2. Review and Commit to Agreements

You will then be asked to confirm that you’d like to reserve a DOI. A reserved DOI **will not function until you request and complete the publication review process**. The DOI is in a temporary working state while you complete your dataset.

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FJ7zAhskhPIOyONMBRHw7%2FReserveDOI-agreements.png?alt=media&amp;token=1950a54e-f0b3-4091-ab43-61232963c9e3" alt=""><figcaption><p>Figure 3: Before you reserve a DOI for your data, please read through the informational warnings and agreements</p></figcaption></figure>

## 3. DOI Reserved Confirmation

Once you click Reserve DOI, you’ll be presented with a confirmation message that your new DOI has been reserved. Your DOI status will change to <mark style="color:blue;">**Reserved**</mark> (Figure 5). Your dataset **will remain in the&#x20;**<mark style="color:blue;">**Draft**</mark>**&#x20;status** until you indicate that your dataset is finalized by initiating a publication review.

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FxQkjoHaNoGxW4jqamLts%2FReserveDOI-confirmation.png?alt=media&amp;token=ef211856-cb3f-4a0e-ae56-e71db0e72ba5" alt=""><figcaption><p>Figure 4: A confirmation message will appear when your DOI has been reserved, see message for important details</p></figcaption></figure>

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FGUEBA1bIadhSLXmaN0Hm%2Fpub-ui-Templates-landing-page-reservedWarning.png?alt=media&amp;token=3800e9e9-db29-4f87-9bae-aa45c15dd559" alt=""><figcaption><p>Figure 5: Datasets with reserved DOIs will have the DOI status "Reserved". A red warning banner message will help inform you that your dataset is not complete and your DOI is inactive.</p></figcaption></figure>

## 4. Optional: Obtain Your Citation

After reserving your DOI, you will be able to see and copy your dataset citation with the new DOI using the "Cite this Dataset" button at any time. You can make corrections to your citation by selecting the Edit button and updating the metadata fields used in the citation (author names, author order, publish year, title, and project). ESS-DIVE recommends finalizing your citation as much as possible before referencing it in publications.

If you intend to use this citation for any publications, it is your responsibility to make sure that the citation on ESS-DIVE is accurate and that the DOI becomes accessible to any reviewers and readers as soon as possible.

## 5. Initiate Publication Review

ESS-DIVE will not publish a dataset with a reserved DOI until you have submitted all required metadata,  requested publication, and completed the review process. Click the link below and follow the instructions to start and complete the ESS-DIVE publication review process.

:point\_right: [**I'm ready to publish my dataset**](/publish-data/publish-your-dataset/request-publication)

{% hint style="danger" %}
It is your responsibility as the data provider to complete the dataset publication process. The DOI will not provide access to your dataset until you do so.
{% endhint %}

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2Fhy1CsAErVWb2j3lWF6WD%2Fmanage-pub-Templates-alreadyReserved.png?alt=media&amp;token=e821e6e4-59c1-4c62-825a-b3c8f981ea27" alt=""><figcaption><p>Figure 6: The "Reserve DOI" button is no longer available. Click "Start publication process" to continue your journey.</p></figcaption></figure>


# Publish with Existing DOI

Use a DOI that your data was previously published under and let ESS-DIVE know that you do not want a new DOI.

## 1.  Double Check Existing DOI

Take a moment to review whether or not your existing DOI is appropriate for your ESS-DIVE dataset. The data in your ESS-DIVE dataset should contain **an exact copy** of the data published under your existing DOI.

If you wish to publish existing data on ESS-DIVE and this publication option does not exactly fit you use case, you may be interested in using **external link** feature instead.&#x20;

:bulb: Learn more about [External Links](/contributing-data/link-to-external-data-sources) or contact ESS-DIVE (<ess-dive-support@lbl.gov>) to discuss your options.

{% hint style="info" %}
Some tips for understanding whether your existing DOI is appropriate for your ESS-DIVE dataset are provided the [**overview section**](/publish-data/publish-your-dataset#publish-with-existing-doi-1).
{% endhint %}

## 2. Set Existing DOI

When you open the **Manage Dataset** window, there is no button to indicate that you wish to publish your dataset with an existing DOI. Instead, fill out the **Existing DOI and Alternate Identifiers** metadata field with the DOI you would like to use. Filling out this field will prevent a new DOI from being created.

:bulb: Learn more about the [Existing DOI and Alternate Identifiers](/contributing-data/package-level-metadata#existing-doi-and-alternate-identifier) metadata field with examples.

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FDcEFrJugNBVBjc4JGBDK%2FExistingDOI-edit-altid-field.png?alt=media&amp;token=4621e280-3b0e-44e8-8fd9-f8da5c465ed7" alt=""><figcaption><p>Figure 1: Edit your dataset and enter your existing DOI into the "Existing DOI and Alternate Identifiers" metadata field located on the Overview section</p></figcaption></figure>

## 3. Initiate Publication Review

ESS-DIVE will not publish a dataset using an existing DOI until you have submitted all required metadata, requested publication, and completed the review process. Click the link below and follow the instructions to start and complete the ESS-DIVE publication review process.

**Your DOI status will remain in the&#x20;**<mark style="color:blue;">**No DOI Added**</mark>**&#x20;status** until your dataset has been reviewed and published. You will be notified via email during the review process whether your existing DOI has been accepted or rejected. If your existing DOI has been rejected, ESS-DIVE will provide you with suggested next steps.

:point\_right: [**I'm ready to publish my dataset**](/publish-data/publish-your-dataset/request-publication)

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FlaSihkX2yYFbjGtn06cZ%2FExistingDOI-pub-panel-reservationUnavailable.png?alt=media&amp;token=f58efa2d-de10-4524-a3f1-b772970bf773" alt=""><figcaption><p>Figure 2: The option to reserve a DOI is unavailable because you have already indicated that you want to use a DOI that you have provided. Click "Start publication process" to continue your journey.</p></figcaption></figure>


# Troubleshooting

## General

<details>

<summary>Why are the "Reserve DOI" and/or "Start Publication Process" buttons grayed out?</summary>

Buttons are disabled after they have been activated. If your button is inaccessible, you've already completed the available steps for that process.

Alternatively, if you have not used either button, the "Reserve DOI" button will be grayed out when the **Existing Identifiers** field is filled out. If you want a DOI, remove the content of this field to a more appropriate metadata field.

</details>

***

## Publication Review

<details>

<summary>I requested a publication review. Now what?</summary>

The **dataset submitter & contact will receive an automated email** from ESS-DIVE confirming that we have received your request for publication review and that you should hear from the ESS-DIVE review team shortly.

For the submitter (the last person to edit the dataset), emails will be sent to the email address submitted to ESS-DIVE when they registered as a data contributor.&#x20;

For dataset contacts, emails will be sent to the email address listed in the dataset metadata.

</details>

<details>

<summary>I requested a publication review for my dataset and want to submit an update.</summary>

Updates to datasets under publication review are welcomed and encouraged.&#x20;

To notify ESS-DIVE that you have completed an update on your dataset once the review has begun, **please respond to the email chain** that was created for your publication review and let our reviewers know you've made changes. We will not review changes until you let us know your dataset is ready to be re-reviewed.

</details>

***

## Reserving a DOI

<details>

<summary>I reserved a DOI and the "Reserve DOI" button is disabled. How do I find my reserved DOI on the private landing page?</summary>

Go to "Copy Citation" or look for DOI at top of landing page.

</details>

***

## Existing DOIs

<details>

<summary>Why was my existing DOI rejected?</summary>

There are various reasons why your DOI may have been rejected. Some common examples are that the provided existing DOI:

* Does not resolve (the format may be incorrect)
* Is not a DOI
* Resolves to a manuscript
* Resolves to an incomplete version of the data or points to a related resource&#x20;

</details>


# Update your Public Dataset

ESS-DIVE allows updates to published datasets while retaining the same DOI.&#x20;

In fact, we encourage small updates after publication that enhance the data quality, such as completing related citations for recently published manuscripts, adding related identifiers from subsequent research, or expanding the keywords to improve discoverability. These updates can be made freely without further action.

For updates that modify the content of your data publication beyond this, you may be interested in versioning your publication. As a best practice, versioned publications should clearly describe the data provenance. We recommend reviewing these **guidelines** and **expectations** for versioning datasets before you update your publication.

{% hint style="warning" %}
It's important to consider the impact your change will have on the research products that have previously used and cited your data.
{% endhint %}

## Options for Versioning Public Datasets

There are two options for providing updates after your dataset has been published:&#x20;

1. [Update an existing dataset](#id-1.-update-an-existing-dataset),
2. [Publish a new dataset with a new DOI](#id-2.-publish-a-new-dataset-and-doi).

These both result in a new version of the data. Consider option 1 if you are extending data, adding procedural metadata, or providing non-breaking corrections or usability enhancements to the data. Consider option 2 if you want to retain the original dataset citation or obsolete a prior data publication.

There are many other reasons why you may choose one option over the other. **Ultimately, it is your decision** whether the nature of the change merits a new scientific publication or not. Continue to the linked sections for recommendations on how to version datasets that clearly describe data provenance.&#x20;

<details>

<summary>How will versioning affect my citation?</summary>

Changes made to metadata that appear in the citation (authorship, title, or publishing project) will change the citation. It may be your preference to create a new dataset instead of altering the citation. Or you may find that retaining the same DOI, despite any citation alterations, is valuable for the provenance of your data.

**Regardless of your choice, the DOI in your citation will always remain the same**.

</details>

<details>

<summary>What about updating Large Data on Tier 2?</summary>

Researchers who have published very large datasets on Tier 2 should more carefully consider whether the file volume of the change merits a new publication. Please reach out to ESS-DIVE to discuss the nature of your update.&#x20;

</details>

<details>

<summary>Data Publications cannot be deleted</summary>

Formally deleting or removing public datasets is **not allowed**.

As a long-term data repository, ESS-DIVE cannot delete or remove data once it has been published.  Data can be obsoleted by a new version as necessary, however we are not able to limit public access to open data, even if it is obsolete.&#x20;

Obsoleting or retiring a public dataset on ESS-DIVE requires modifying the old dataset metadata properly and publishing the latest version as a new dataset (option 2).

</details>

***

## 1. Update an Existing Dataset

When updating an existing DOI, we recommend updating your metadata to clearly describe all changes to the publication:

* **Abstract:** Add a brief statement clearly describing all changes, which files were changed/added, and the date they were made.&#x20;
* **Methods:** Add a methods step that describes how the dataset was updated. You may go into detail here. Include the date/year that the dataset was originally published first, followed by subsequent versions.&#x20;
* **Publication Date:** Update to the date or year that you changed the dataset.
* **Change log:** Consider adding a simple changelog file in your dataset recording all specific changes with timestamps.
* (If applicable) **Temporal Coverage**: If adding new dates, update date range to include updated data.
* (If applicable) **Authors**: If applicable, add any new contributors to the author list.
* (If applicable) **Title:** Consider updating any time frames or specific sites mentioned in the title which have been expanded on.
* (Optional) **Version numbers**: You may prefer to include specific version numbers in your **Title** (e.g. “Original dataset title (Version 2.0)”) and/or **File Names** (e.g. "filename\_v2.csv") to quickly identify newer versions.

<details>

<summary>Option 1 Examples</summary>

The most common type of updates you see in these examples are periodic updates that extend timeseries data (e.g., sensor streams or new field seasons). This is particularly useful for long-term or key project data products.

[doi:10.15485/1866836](https://data.ess-dive.lbl.gov/view/doi:10.15485/1866836); [doi:10.15485/1603775](https://data.ess-dive.lbl.gov/view/doi:10.15485/1603775); [doi:10.15485/1660962](https://data.ess-dive.lbl.gov/view/doi:10.15485/1660962)

* **New data**: additional dates or sampling locations; including analysis results as they become available.
* **Improved data quality**: reformatting files with reporting formats; improving data and code usability; providing additional documentation or READMEs; expanding methodology.
* **Corrections**: units; dates; coordinates in data files; typos.&#x20;

</details>

## 2. Publish a New Dataset and DOI

When creating a new dataset and DOI version, we recommend the following:

* **Old Abstract:** Add a statement at the beginning of the abstract in the original dataset to indicate that "A new version of this dataset is available at <https://doi.org....."&#x20>;
* **Old File Names:** Update any obsoleted dataset file names in the original dataset to indicate that they should not be used. Do not delete the files.&#x20;
* **Old Related Reference:** Add a full citation to the new dataset as a related reference in the original dataset.
* **New Related Reference:** Add a full citation to the old dataset as a related reference in the new dataset.&#x20;

<details>

<summary>Option 2 Examples</summary>

The reasons for publishing data under a new DOI are varied and can come down to preference. In general, you should get a new DOI if your changes will alter the publication substantially enough that it impairs the reproducibility of research that has previously used and cited your data.

* Data availability is updated on annual basis and the new data is published under a new DOI ([doi:10.15485/2216951](https://data.ess-dive.lbl.gov/view/doi:10.15485/2216951), [doi:10.15485/2476540](https://data.ess-dive.lbl.gov/view/doi:10.15485/2476540))
* Obsoletes the original data product ([doi:10.15485/1818367](https://data.ess-dive.lbl.gov/view/doi:10.15485/1818367)) by publishing a substantially improved version under a new DOI ([doi:10.15485/1866836](https://data.ess-dive.lbl.gov/view/doi:10.15485/1866836))

</details>

{% hint style="info" %}
:star: **Do you have feedback on how you would prefer versioning to work? Let us know!**  Email <ess-dive-support@lbl.gov> or use our [contact form](https://docs.google.com/forms/d/1L-JuV-On2tNIorffysp2NYPeS_2hmFu9tLXv8cl1U9M/edit).
{% endhint %}


# Register Dataset Citations

You can now manually register your dataset citations in ESS-DIVE and improve your dataset metrics. Read this page to learn how to register your dataset citations.

## Dataset Metrics

ESS-DIVE provides metric visualizations for all archived datasets through our search and discovery platform. Our metrics include live counts of dataset views, downloads, and citations, as well as reports on the total quantity of data and data types stored on ESS-DIVE; these reports can be viewed at <https://data.ess-dive.lbl.gov/profile>. In addition to this, ESS-DIVE's metrics are counted and reported on an individual dataset level and can be accessed via any dataset landing page.

These metrics are automatically collected when possible. However, in the case of dataset citations, you can also manually add citations for any papers that you know have cited your dataset and were not automatically reported.

## Why Register Citations?

Academic journals do not always report data citations referenced in paper publications. Without this information, ESS-DIVE's metrics service cannot pick up these citations and this creates an incomplete citation report on ESS-DIVE. The citation registration feature provides a mechanism for researchers to directly register dataset citations and and improve their citation metrics on ESS-DIVE.

## Types of Citation

When registering citations for ESS-DIVE datasets, you will find two possible options to choose from. These options are described in Table 1 and the following section, [How to Register Citations](#how-to-register-citations), instructs when to use these citation types.

| Type                               | Definition                                                                                                                                                    |
| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Publication cites this dataset** | The publication ideally cites the dataset in the reference section of the paper. Or, the dataset is explicitly identified or linked to somewhere in the text. |
| **Publication uses this dataset**  | The publication uses the dataset, but does not formally cite it.                                                                                              |

***Table 1**: A list of the available dataset citation types and their definitions*

{% hint style="info" %}
**Best practice when citing datasets** is to provide a full citation, including the DOI, in your publication references.&#x20;
{% endhint %}

## How To Register Citations

{% hint style="info" %}

#### Please allow 24 hours for your manual registration to show up in ESS-DIVE metrics

{% endhint %}

You can register dataset citations directly from the dataset landing page.

1. Login to [ESS-DIVE](https://data.ess-dive.lbl.gov/).  *Only* [*registered data contributors*](/contributing-data/new-contributor-registration) *can register citations, make sure you are logged in and registered before attempting to register a citation.*
2. Open the dataset and select the "Citations" button underneath the dataset title to open the citations window (Figure 1).
3. If there are already citations associated with this dataset, they will be listed here. Select the "Register Citation" button (Figure 2).
4. Enter the DOI of the publication that cites the dataset.
5. Select the type of citation used in the publication (Figure 3); Table 1 provides a description of the available citation types.
6. Click "Register" to finish registering this dataset citation.
7. Please allow 24 hours for your manual registration to show up in your dataset metrics.

![Figure 1: Dataset citations are accessible via the dataset landing page](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FvIijql5Wx0NfEMxszJwS%2Fselect-citation.png?alt=media\&token=6675f217-34d5-48a3-9fd0-9c66aab628be)

![Figure 2: Only data contributors who are logged in to ESS-DIVE can select the "Register Citation" button](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FrbGpctIyhKBrajvbtRAy%2Fcitation-view.png?alt=media\&token=9a8f40a5-5dba-4a8f-82b6-ff749b119b99)

![Figure 3: Enter the publication DOI and the citation type to finish registering new dataset citations](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FUbBjv5UKq6VX8Uu97Irx%2Fcitation-types.png?alt=media\&token=c1874467-906f-4713-9e4d-2e11ef29fe12)


# Create Data Portals

You can now easily view related datasets all in one place.

**Table of Contents**

* [<mark style="color:green;">What is a Portal?</mark>](#what-is-a-portal)
* [<mark style="color:green;">Components of a Portal</mark>](#components-of-a-portal)
* [<mark style="color:green;">Portal Uses</mark>](#portal-uses)
  * [<mark style="color:green;">Research Themed Portals</mark>](#research-themed-portals)
  * [<mark style="color:green;">ESS Project Portals</mark>](#ess-project-portals)

## What is a Portal?

A portal is a **collection** of private or public datasets on a unique webpage. This webpage also supports additional **text pages** for describing any aspect of the data collection.

Typically, a research project's website won't be maintained beyond the life of the project and all the information on the website that provides context for the data collection is lost. ESS-DIVE portals can provide a means to preserve information regarding the project's objectives, scopes, and organization and couple this with project datasets so it's clear how to use and interpret the data for years to come.

Portals also leverage ESS-DIVE's [<mark style="color:green;">**metric feature**</mark>](https://data.ess-dive.lbl.gov/profile), which create statistics describing the project's datasets. Information such as number of datasets, total size of data, data collection time periods, views, downloads, and citations are immediately available from the portal webpage.

## Components of a Portal

Click through the sections below to learn about the four types of tabs in data portals.&#x20;

{% tabs %}
{% tab title="Settings" %}

## Settings Tab

The first component of portals is the settings tab. This tab is the first page you will see after initially creating a new portal and sets up important infrastructure for the portal.\
\
On this page you can give the portal a title and assign it a unique url; also referred to as a **portal identifier**. You can add a general description of the portal, upload an icon photo or logo for your data, and upload icon photos from any partner organizations that have contributed to the data.These partner icons will appear in the footer banner on every page in a portal, likewise your portal icon will appear in the header banner.

![](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-M6bpQUIRdopIlH7m_4y%2F-M6bv0NLCdSNTWEaz9rx%2FCDIAC_settings.png?alt=media\&token=9b9049df-0d50-4fb0-a142-9d1646f67752)

{% hint style="info" %}
Every ESS-DIVE portal URL will follow this format: \
*<https://data.ess-dive.lbl.gov/portals/>**\<portal\_identifier>**/*
{% endhint %}
{% endtab %}

{% tab title="Data" %}

## Data Tab

The data page is the most important component of the ESS-DIVE portal system. This is where your chosen dataset collection is displayed. It looks and performs just like [ESS-DIVE's main data repository ](https://data.ess-dive.lbl.gov/data)webpage.

![](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-M78n0dpZNmC4bjhVteP%2F-M7AM0SUptE5LOxn_Vws%2FCDIAC_data.png?alt=media\&token=c713aeb0-63d8-46e5-b9b6-39439defe878)
{% endtab %}

{% tab title="Metrics" %}

## Metrics Tab

Unlike the first two pages, the metrics page cannot be edited or customized. It is a default feature that provides the following information about the datasets within a portal:

* The total number of publicly-available datasets
* The volume (in bytes) of all publicly-available dataset metadata records and data files
* The most recent date the datasets were last updated (metadata and data are treated separately)
* The file types of all publicly-available data
* The years in which data was collected, regardless of upload date

Click [here](https://data.ess-dive.lbl.gov/portals/CDIAC/Metrics) to see a complete example of a portal metrics page.

![](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-M6bpQUIRdopIlH7m_4y%2F-M6bwTVgIY0t2yYeaNLe%2Fmetrics.png?alt=media\&token=c893031a-192a-49aa-a55a-3e1d2c645267)

Please contact [ESS-DIVE Support Team](https://ess-dive.lbl.gov/contact/) about any questions or concerns about the metric page
{% endtab %}

{% tab title="Freeform Pages" %}

## Freeform Tabs

Freeform pages are an optional function provided by ESS-DIVE portals. Here, you can add as much supplementary information as needed using markdown. It is highly recommended to create a descriptive **About page** for your data collection. &#x20;

### Example Freeform Tabs&#x20;

Below are three examples of ways to take advantage of portal freeform pages to tie unique content together with a data collection. Add as many tabs as needed.

The following examples are from the Carbon Dioxide Information Analysis Center's portal, visit [this portal](https://data.ess-dive.lbl.gov/portals/CDIAC) to explore its contents further.

![](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-M78n0dpZNmC4bjhVteP%2F-M7AM5GfxDfvjhCBybTz%2FCDIAC_about.png?alt=media\&token=ba90497a-4813-4800-9448-593a74d84582)

![](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-M78n0dpZNmC4bjhVteP%2F-M7AM9yJHNxD4IguDtAk%2FCDIAC_page.png?alt=media\&token=65a63fb8-ba3f-471c-a144-e6d03874afd3)

![](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-M78n0dpZNmC4bjhVteP%2F-M7AMDvmyrGf8g2K_vtM%2FCDIAC_faq.png?alt=media\&token=3da8044b-67f0-4f41-b923-ebd723cdf21f)
{% endtab %}
{% endtabs %}

## Portal Uses

There are two primary use cases for collecting datasets into data portals:

1. Datasets related to the same research theme
2. All datasets published by an ESS project

***Note**: Only* [*registered Data Contributors* ](/contributing-data/new-contributor-registration)*can create data portals on ESS-DIVE*&#x20;

{% hint style="info" %}
**Terminology check:** A Data Contributor is someone who has prior approval to upload data on ESS-DIVE and has registered themselves as a Data Contributor with the ESS-DIVE Team.
{% endhint %}

### Research Themed Portals&#x20;

Data Contributors can use portals to collect public data and create a research-themed webpage. This curated data collection will have data available for direct download, can provide insights about the significance of the collection, and can be shared with colleagues and broad audiences.

### ESS Project Portals

Projects can collect their project-specific datasets into a unique portal and customize the portal's theme and structure according to the needs of that project. Portals enable projects to **preserve their identity** on ESS-DIVE as well as their data.&#x20;

A project portal can use freeform pages to include information about:

* The project, funding program, and project partners&#x20;
* Purpose of the project data
* Significant paper citations
* Authors bios

For projects, portals can also be used to **report on performance** by leveraging the [metric feature](#metrics) (click [here](https://data.ess-dive.lbl.gov/portals/CDIAC/Metrics) to see an example of portal metrics). Important statistics about the project's data are tracked on the metrics page including total data publications, number of data views and downloads, and the time span of data collection across the project.

{% hint style="warning" %}
Currently, project-based portals can only be linked to an individual's ESS-DIVE account. The current infrastructure does not support project-based or group accounts.

**However, once created, a portal can be shared with project team members to grant view, edit or manage access.** See [Share Portals](/manage-data/share-data-permissions/share-portals) and [Project Teams](/manage-data/manage-project-data/project-teams) for more information.
{% endhint %}

## [Explore Public Data Portals](https://data.ess-dive.lbl.gov/portals)

All public data portals are accessible via ESS-DIVE's Portal page.

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FX4HgBRd2QFdv63YgrJCN%2Fpublic-data-portals.png?alt=media&amp;token=275065eb-7321-41dd-b740-d8c6b814bc62" alt=""><figcaption></figcaption></figure>


# How to Create & Publish Portals

A step-by-step guide on how to navigate ESS-DIVE and create a new portal.

## Portal Tutorial Videos

For video tutorials on how to create your first portal, please visit our DataONE partners at the [Arctic Data Center](https://arcticdata.io/about/).&#x20;

{% embed url="<https://arcticdata.io/data-portals/#instructional-videos-identifier>" %}

<div align="center"><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-MibnkkDh6Xk0U36vZ1Y%2F-Mibp2qsyLLdUNFNFXnK%2Farctic-data-center.png?alt=media&amp;token=3b882cb6-7561-422d-94ae-25d32ffaea06" alt=""></div>

**IMPORTANT NOTE**: new data portals are not immediately made public. Just like ESS-DIVE datasets, portals must be approved by The ESS-DIVE Team before they are publicly available.&#x20;

**Table of Contents**

1. [Create a New Portal](#id-1.-create-a-new-portal)
2. [Configure Settings](#id-2.-configure-settings)
3. [Add Data](#id-3.-add-data-to-a-portal)
   * Define Rules
   * [Portal Metrics](#portal-metrics)
4. [Create Freeform Pages](#id-4.-create-freeform-pages)
5. [Re-order pages](#id-5.-reorder-your-pages)
6. [Delete Pages](#id-6.-delete-pages)
7. [Save your Portal ](#id-7.-save-your-portal)
8. [Find & Edit My Portals](#id-8.-find-and-edit-my-portals)
9. [Identify Private Portals](#id-9.-identify-private-portals)
10. [Share Portals](#id-10.-share-portal-permissions)
11. [Publish Portals](#id-11.-publish-portal)

## 1. Create a New Portal

If you are on [ESS-DIVE's website](https://ess-dive.lbl.gov), select the "Data" tab in the navigation bar and click on "Access Data Portals" from the drop down. This will take you directly to the Portals page on ESS-DIVE's [data repository](https://data.ess-dive.lbl.gov/).

![Figure 1. The ESS-DIVE website: https://ess-dive.lbl.gov/](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FUvZuhIvrGhhPmfQ2wiwA%2FAccess%20Data%20Portals.png?alt=media\&token=c8caf8c0-651c-4db3-a8da-a57bfd540d1b)

Login to ESS-DIVE with your ORCID. If you don't have an ORCID and/or haven't linked your ORCID to ESS-DIVE yet, follow our [Register to Submit Data guide](/contributing-data/new-contributor-registration) for more information on how to get started.&#x20;

Once you login to ESS-DIVE with your ORCID, hover over "Portals" in the navigation bar and select "Access Data Portals".

![Figure 2. Click on the menu under your name to access My settings](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FcAmVEMwNC6vieEx4fnnH%2FAcess%20Data%20Portals.png?alt=media\&token=d0b972ce-9a82-4fcc-9364-f3b64366c460)

On portals page, select the green button "+ New Portal" to start editing a blank data portal.

![Figure 3. On the settings page, click the "My Portals" page to create a new portal](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FtzqptmpipLYnydYPozwJ%2FNew%20Portal.png?alt=media\&token=9f01cebf-b93d-487b-a8e8-9020e27cde57)

## 2. Configure Settings

When you start editing a data portal, the first thing you'll see is the settings page where you’ll be able to set the basic elements of your portal. Fill out each of the following:

* Portal title
* Unique portal identifier&#x20;
  * *This identifier will be used to create the portal URL. If the name is available, a label will indicate it's available and if the name is taken already, it will note that the name is already taken. This feature ensures the portals are unique.*&#x20;
* Portal description
* Partner organization logos

![Figure 4. Configure portal settings](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-M2o3et8hhX8ybX-e6sK%2F-M2uLuAmcf65YG2wSYEC%2FESS-DIVE_settings-page.png?alt=media\&token=7a79cc30-8b8d-462a-9731-54c765aed7fa)

## 3. Add Data to a Portal

By default, the data tab is empty. You will need to define a series of "Rules" to select which datasets you want in your portal collection. After you've defined your collection, you can create pre-set "search filters" which allow people to filter your dataset collection.

![Figure 5. Preview of the data tab used to create your data collection](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FXIeYTME6jLwB2MzUD5xL%2Fportal-data-tab.png?alt=media\&token=deced430-6395-46f8-aa77-b0c206b850e1)

### Define Rules

Rules are a powerful tool for creating or editing a collection of datasets in a portal. Build rules based on dataset metadata to define which datasets should be included in your collection. All public datasets stored in ESS-DIVE are available for selection.

Datasets published on ESS-DIVE in the future that match these rules will also be added to your collection.

Figure 6 shows an example of a data portal rule. This example demonstrates that only 129 published datasets from ESS-DIVE (highlighted in the orange box) meet the criteria defined by the rule.&#x20;

When you create a rule, there's no button to confirm your rule selection. The search results are filtered in real time as your rules are defined, so whatever datasets remain in the return field will be in your data portal. Scroll down to preview your data collection.&#x20;

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FiVEz7WKU23M5rPzRqtVc%2Fportal-example-rule.png?alt=media&amp;token=2e44177e-e33d-4845-87e6-c405d13f65b5" alt=""><figcaption><p>Figure 6: An example of a data portal rule</p></figcaption></figure>

{% hint style="info" %}
If you need assistance assembling portal data using complex rules, please contact the [ESS-DIVE Support Team](https://ess-dive.lbl.gov/contact/).
{% endhint %}

Rules have three parts:

1. **Select one or more metadata fields** (Figure 7)
   * Chose any of the 75+ metadata fields from the drop-down menu
   * Select "Any Metadata Field" to search for any text that appears in a dataset
   * For complex filters, use **Table 1** to help map familiar dataset metadata fields to a rule field
2. **Select an operator** (Figure 8)
   * Choose from equals/does not equal, contains/does not contain, or empty/is not empty
3. **Define the value** (Figure 9)
   * In most cases, this will be a typed word or phrase. For few metadata fields (such as \`is Owner\`, a dropdown will appear.
   * One or more value can be entered in one rule
   * After typing in a value of interest, hit enter to save the entry.

<div><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-Mj1n1lC9z-X2uIK5hA2%2F-Mj1pY0nb0QjLOrjj_L7%2Ffield-selection.png?alt=media&amp;token=e8f03867-8321-4351-85e5-bf0a781107c4" alt="Figure 7. Dropdown selection of available metadata fields"> <figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-MYv3SaGZpPDhQ3EjeiD%2F-MYzUu4bms7Rzhy4dK0d%2Fportal-tutorial-operator.png?alt=media&amp;token=a52f6e42-4b7f-4c06-b859-483c73091238" alt=""><figcaption><p>Figure 8. Screenshot of available operations</p></figcaption></figure> <figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-MYv3SaGZpPDhQ3EjeiD%2F-MYzu_3fcY7Euz-SCtCm%2Fportals-tutorial-value.png?alt=media&amp;token=d25530b3-a391-4b67-9290-df1d74679b72" alt=""><figcaption><p>Figure 9. An example of the free-form text that can be entered</p></figcaption></figure></div>

Additionally, you can define how your rules should include or exclude datasets from your collection (Figure 10).  Choose (1) whether rules should include or exclude datasets, as well as (2) whether datasets should meet all rules (this is an AND boolean operation) or any rules (this is an OR boolean operation).

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FHzyukmRuhX6BDaRBAakx%2Fboolean-inclusion.png?alt=media&amp;token=e4896b36-d6a6-4dd5-8d88-6a040d5b1bb0" alt=""><figcaption><p>Figure 10: Define how your rules should include or exclude datasets from your collection.</p></figcaption></figure>

Finally, you can play around with rules and inclusion operators using **Rule Groups** (Figure 11). These groups can help narrow your data collection to just the desired datasets.

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FE6Hf5Q8GNMIF4JUnxtBG%2Fportal-rule-groups.png?alt=media&amp;token=8e5c9da5-6148-4274-a963-a5dfc74fe2f7" alt=""><figcaption><p>Figure 11: An example of rule groups</p></figcaption></figure>

#### Examples of Common Metadata to use in Rules

* Project name&#x20;
* Author name
* Portion of title (if common phrases are used)
* Keywords
* Area of research (wetlands, greenhouse gas, etc)
  * This option can lead to more dynamic results as the term could appear in the abstract, keywords, methods, or title

#### Metadata Mappings

| Dataset Metadata                                                              | Rule Field                                                                                                   |
| ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| **Title**                                                                     | General > **Title**                                                                                          |
| **Project Title**                                                             | Awards & Funding > **Project**                                                                               |
| **DOI**                                                                       | Identifiers > **Series Identifier**                                                                          |
| **Abstract**                                                                  | General > **Abstract**                                                                                       |
| **Keywords, Data Variables**                                                  | General > **Keywords**                                                                                       |
| **Contact, Creators, Contributors**                                           | People & Organizations > **All Creator Last Names**                                                          |
| **Publication Date**                                                          | Dates > **Date Published**                                                                                   |
| **Start Date, End Date**                                                      | Dates > **Year of Data Collection**                                                                          |
| **Geographic Description**                                                    | Geography > **Site Description**                                                                             |
| <p><strong>Bounding Coordinates</strong></p><p><strong>Northwest</strong></p> | <p>Geography ><br><strong>Northern most Latitude</strong>,</p><p><strong>Western most Longitude</strong></p> |
| <p><strong>Bounding Coordinates</strong></p><p><strong>Southeast</strong></p> | <p>Geography ><br><strong>Southern most Latitude</strong></p><p><strong>Eastern most Longitude</strong></p>  |
| **Files**                                                                     | <p>File Details > <strong>File Name</strong></p><p><em>\*does not search file contents</em></p>              |

Table 1: Mapping of common dataset metadata to rules fields.

### Portal Metrics

The metrics page is a default function provided by ESS-DIVE. **This page cannot be edited and cannot be viewed while editing.** You do have the option to delete the page if you'd like. To delete the page, select the arrow next to the word "Metric" in the tab and choose "Delete" from the dropdown list.&#x20;

![Figure 12. The metrics page during an edit session](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-M7ysF7x077qv1VZJV55%2F-M7yuVPka6zgOKtJNBOj%2Ftutorial_metrics.png?alt=media\&token=900c8c0a-856d-4f0a-8ed5-32637fb99e5f)

Please contact [ESS-DIVE Support Team](https://ess-dive.lbl.gov/contact/) about any questions or concerns about the metric page.

## 4. Create Freeform Pages&#x20;

{% hint style="info" %}
**To watch a tutorial on creating a new freeform page see this video from our DataONE partners at the Arctic Data Center:** [**Creating a Freeform Text Page**](https://arcticdata.io/data-portals/#instructional-videos-identifier)
{% endhint %}

To add a freeform page to a portal, select the "+" tab next to to the data and metric tabs and then choose the freeform option that appears on screen. A freeform page will the populate the page.&#x20;

![Figure 13. Menu options after opening a new freeform page](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-M62iZcsw90iNTKQ_HlM%2F-M6bNcAK5rVIKXh9WWoN%2Ftutorial_metrics.png?alt=media\&token=d5fa9798-2dc4-4995-92a1-6be4264d2168)

Easily customize your banner with a unique image, title, and page description. To change the name of the tab, click on the arrow in the "Untitled" tab and select "Rename" from the dropdown list.&#x20;

![Figure 14. Rename your page and preview changes](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-M7ysF7x077qv1VZJV55%2F-M7yvdiPom_NBp66YWnf%2Ftutorial_untitled.png?alt=media\&token=483d92ea-879d-4b6b-a280-d111e09f56f1)

Below the banner, there is a markdown text box with some examples on how to use the markdown language to customize the text display. As you write, toggle through the Edit and Preview modes in the markdown text box to make sure your information is displaying as intended. Portals are flexible and can accommodate as many additional freeform pages as needed. \
\
Please see these additional resources for help with markdown:

* [Markdown reference](https://commonmark.org/help/)
* [Ten minute tutorial](https://commonmark.org/help/tutorial/)
* For a longer example where you can also preview the results, checkout the [Showdown Live Editor](http://demo.showdownjs.com/)

## 5. Reorder your Pages!

![Figure 15. Screenshot of tabs in portal builder](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-MYv3SaGZpPDhQ3EjeiD%2F-MYzUHlwL-yMPI3wa62x%2FPortal-Tutorial-Tabs.png?alt=media\&token=091d9868-2b45-4731-bd2e-2f3ee956f416)

Drag-n-drop pages in the portal builder to easily reorder their appearance in the portal! Select the dotted icon to drag a tab.

## 6. Delete Pages

To delete a page from your portal, select the arrow in the tab and choose "Delete" from the dropdown.

![Figure 16. Delete a freeform page](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-M7ysF7x077qv1VZJV55%2F-M7yxTcOLkADVcrluN4Y%2FTutorial_delete.png?alt=media\&token=21fa30a5-a463-428a-acf9-43137dd80f7e)

## 7. Save your Portal

Be sure to save your portal when you complete a page to ensure your progress is retained.&#x20;

![Figure 17. Location of save button](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-M62iZcsw90iNTKQ_HlM%2F-M6bcMe9PWrMQF8Xd2Wt%2Ftutorial_save.png?alt=media\&token=2fd00417-40e7-48df-8acc-5aa4559351ca)

Whenever a portal is saved, a dialogue box will pop up at the top of the page prompting you to view your private portal in view mode. You can choose to ignore this and continue editing.

![Figure 18. Quick-access to latest version of portal](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-M62iZcsw90iNTKQ_HlM%2F-M6bcZtVz8fz5zwEozYN%2Ftutorial_view.png?alt=media\&token=e14a049b-4520-4be7-a79e-b3e3f3d0037a)

## 8. Find & Edit My Portals

Portals can be updated at anytime. If your portal is already public, any changes you make will be public immediately once you hit the "Save" button. You can view and edit your portal from the portals page.&#x20;

First, navigate back to the [portal page](https://data.ess-dive.lbl.gov/portals). Select the ESS-DIVE logo in the top-left corner to return to the main ESS-DIVE Data Portal page. Then select "Access Data Portals" from the dropdown underneath "Portals". You'll find your portal under the "My Portal" section at the top of the page.&#x20;

![Figure 19. Navigate back to the main data portal](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-M62iZcsw90iNTKQ_HlM%2F-M6bi366yOICXTq2nx8i%2Ftutorial_return.png?alt=media\&token=7ca2dcc1-ffd6-4c02-92e9-74f864ea4714)

Click on the a portal title to view it or select the edit button to make changes.

![Figure 20. Access your new portal](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FgfDCkM5HSidMfUSzXfOT%2FMy%20Portals%20\(1\).png?alt=media\&token=1583aae1-a227-414f-bec6-2920f4d33a3c)

## 9. Identify Private Portals

There are two ways to check if your portals in the "My Portals" section are private.

* **Logout** and go back to the Portal page. If your portal is not listed under "All Portals", then it is private.
* **Logout** and try navigating to the portal URL. If an error message appears, then it is private.

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-Mj1dnqzGlOoV5bJJSgt%2F-Mj1fXUBitCBa7lSCYPv%2Fprivat-portal-error.png?alt=media&amp;token=de83a06f-84c4-4148-a370-ef63d9688f51" alt=""><figcaption><p>Figure 21. Error message received when opening a private portal while signed out</p></figcaption></figure>

{% hint style="warning" %}
At this time there is no way to easily identify private portals from the portal list
{% endhint %}

## 10. Share Portal Permissions

Instructions for sharing data portal permissions with your colleagues are available here:

{% content-ref url="/pages/a2K40XOKJS5ozvn9soPK" %}
[Share Portals](/manage-data/share-data-permissions/share-portals)
{% endcontent-ref %}

## 11. Publish Portal

![Figure 22: Portal publishing process](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-M2o3et8hhX8ybX-e6sK%2F-M387xhwKdcNJEW2ocNX%2FPortal-Tutorial_publish.png?alt=media\&token=fb08c2a7-0d5f-45b0-803b-4309259631a7)

Just as with datasets created on ESS-DIVE, new portals are automatically set to private and only visible to the portal creator. The portal will remain private until reviewed and approved by the ESS-DIVE Team.

To start the publishing process, please contact the [ESS-DIVE Support Team](https://ess-dive.lbl.gov/contact/). We will review your portal and help publicly publish your new portal.


# Share Data Permissions

Registered data contributors can share their datasets and data portals with other individuals or teams to collaborate directly on ESS-DIVE.

When you share permissions to a dataset or data portal, you can choose who can **view** private content, **edit**, or **manage** that dataset or data portal.

You can share permissions with **project teams** or any individual who has logged into ESS-DIVE using their ORCID. But only registered data contributors will be able to edit content or manage permissions. &#x20;

![Figure 1: The Sharing Options window is used to select which individuals or teams you'd like to share your dataset with](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FQeyEpdnChvSKzD584AdQ%2FShare%20Options%20-%20save.png?alt=media\&token=21d7dba2-6b67-4c53-8eb3-354ffab3edc8)

## **Why share datasets and data portals?**

There are many reasons you may consider sharing data on ESS-DIVE, some reasons include:

* You need someone to review your dataset before publication
* A project manager needs to have access to both private and public datasets generated by your project
* A team member submitted a dataset to ESS-DIVE using the Dataset API and you need to manage or edit it
* You created a data portal and you need a team member to help contribute to the pages
* A team member started a dataset or data portal but someone else needs to complete it

## Choose who to share permissions with

* Share with specific people
* Share with [Project Teams](/manage-data/manage-project-data/project-teams)

## **What sharing permissions cannot do**

* **Concurrent editing is not possible**, only the changes made in the latest submission will be retained and it will overwrite all previous edits.
* You cannot see who has access to a dataset or data portal unless you have "manage" permissions.
* There is currently no way to review permissions for more than one dataset or data portal at once. To review dataset permission, you will need to open the Share Options window (Figure 1) on each dataset or portal individually.
* The person who originally created the dataset cannot be removed from the dataset share permissions.

## **Permission Types**

There are three permission types to choose from when sharing your dataset with a registered team member\*:

| Type       | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **View**   | <p>This team member can read the dataset when it is private. <strong>They cannot edit or share the dataset.</strong> This team member needs to login to ESS-DIVE. </p><p></p><p>The dataset can be found on the search page when they login to ESS-DIVE.</p>                                                                                                                                                                                                                                                                                     |
| **Edit**   | <p>This <strong>registered\*</strong> team member can read the dataset when it is private, view <a href="/publish-data/dataset-and-doi-status-badges">status badges</a>, change the dataset metadata, add/remove files, and edit the dataset after it is published. This team member must be a <a href="/contributing-data/new-contributor-registration">data contributor</a>. </p><p></p><p>The dataset can be found in their "My Data" list.</p>                                                                                               |
| **Manage** | <p>This <strong>registered\*</strong> team member has all the same access abilities as an editor. They can also <a href="/publish-data/publish-your-dataset/reserve-doi-before-publication">reserve a DOI</a>, <a href="/publish-data/publish-your-dataset/request-publication">request publication</a>, and share or remove people from the dataset permissions. This team member must be a <a href="/contributing-data/new-contributor-registration">data contributor</a>. </p><p></p><p>The dataset can be found in their "My Data" list.</p> |

***Table 1**: There are three permission types that you can grant to your team members when you share your dataset*

\*this person completed the [New Data Contributor form](/contributing-data/new-contributor-registration) and was approved to edit and submit metadata to ESS-DIVE

{% hint style="info" %}
The person who originally created the dataset will always have access to the dataset
{% endhint %}

## Find Datasets Shared with Me

When someone grants you permission to **edit** or **manage** a dataset, you can find that dataset in your “My Data” settings. Only registered data contributor will be able to edit or manage datasets.&#x20;

Click on your profile in the top right corner and select “My Data Packages” from the dropdown menu (Figure 2). This will automatically filter the data search page by datasets that you (a) created yourself, (b) have permission to edit, and (c) have permission to manage. You can then use the search filters to locate the shared dataset you are looking for (Figure 3).

If you were granted permission to **view** a dataset, this dataset **will not** be on your "My Data Packages" page. You can find these datasets by using the [search filters](/searching-and-accessing-data/data-access#data-search-tools) on the data search page (<https://data.ess-dive.lbl.gov/data>). We recommend searching by project title, the dataset creator name, or the dataset title.

![Figure 2: Select “My Data Packages” from the dropdown menu](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FHtu21O7TspCNwMbtuTQC%2Fmy-data-packages-dropdown.png?alt=media\&token=a0175fc2-afb9-42c6-ac75-8702191aeef1)

![Figure 3: The "My Data Packages" dropdown will automatically apply a filter to the data search page](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FFMiID55RHxJZGwsYAlKT%2Fcreator-me-search.png?alt=media\&token=659634d3-e254-4668-8a53-9fbd4226e0f9)


# Share Datasets

To share a dataset on ESS-DIVE means that you have given or received permission to **view, edit, or manage** a dataset. Dataset permissions can be changed at any time.&#x20;

Sharing a dataset with other people registered to contribute data on ESS-DIVE will allow them to:&#x20;

* view private datasets,&#x20;
* view status badges,
* change metadata,&#x20;
* add/remove files,&#x20;
* reserve a DOI,
* publish the dataset, and&#x20;
* share the dataset with others

See the [Permission Types](/manage-data/share-data-permissions#permission-types) section for details on which access level grants each capability.

## How to Share a Dataset

{% hint style="warning" %}
You must be the dataset creator or have **manage** access to share datasets
{% endhint %}

1. On the dataset landing page, select the **Edit** button (Figure 1).
2. Select the **Share** button in the top right corner of the file table (Figure 2).
3. A new window will appear titled **Sharing Options** (Figure 3) which shows you who already has access to the dataset and what permission type they have. By default the dataset creator has manage permissions.
4. **Share the dataset** with new people by searching for their ORCID, first or last names in the search bar\* (Figure 4). Or look up the **project team** name to share with a group.
   * *\*Only people who have logged into ESS-DIVE using their ORCID can be found in the Sharing Options search bar*
5. **Modify the permission type** as needed using the access level dropdown (Figure 5).
6. Review your changes and hit **Save** (Figure 6)**.**
7. When you have finished sharing your dataset, you can exit the edit session. **It is not necessary to hit the Submit Dataset button**.&#x20;

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FqTU9bBIdifuvSdvhTTTb%2Fselect-edit-button.png?alt=media&amp;token=4a7e2a20-1a33-499e-a435-9bb7f056b25e" alt=""><figcaption><p><strong>Figure 1:</strong> To share your dataset, open your dataset landing page and click the Edit button</p></figcaption></figure>

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2F9SyyMxhoJDiMebvOG5nm%2Fselect-share-button.png?alt=media&amp;token=a8c83cd2-2f32-41b7-b655-e8d54cf1fc54" alt=""><figcaption><p><strong>Figure 2:</strong> Select the Share button located to the left of the Add Files button</p></figcaption></figure>

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FMoY5cwdT9Ejz35NcNLl8%2FShare%20Options%20-%20new%20share%20(2).png?alt=media&amp;token=dbc2408a-f98a-46a1-a7bd-e3b4988a4bf5" alt=""><figcaption><p><strong>Figure 3:</strong> Review who the dataset is currently shared with</p></figcaption></figure>

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FUue5qA5M0hOKyRuw1WzA%2FShare%20Options%20-%20add%20perm.png?alt=media&amp;token=186edcda-d308-4d1e-9047-dd6eb5339599" alt=""><figcaption><p><strong>Figure 4:</strong> Search for your team members using their first or last names. Selecting their names will automatically add them to the list</p></figcaption></figure>

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FCgtYH0zU7Mttqmili3xZ%2FSshare%20Options%20-%20change%20perm%20(3).png?alt=media&amp;token=d55e3dbf-96cb-4cff-b80e-c72ddfdc1e53" alt=""><figcaption><p><strong>Figure 5:</strong> Change existing access levels using the dropdown bars in the Access column, if needed.</p></figcaption></figure>

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2Fx1NhnqZ9HZ7dqvBrQoeP%2FShare%20Options%20-%20save%20(1).png?alt=media&amp;token=c54f6702-0aad-479e-8e80-d5b6c61543cb" alt=""><figcaption><p><strong>Figure 6:</strong> Make sure to save all changes before closing the window and submitting your dataset edits to ESS-DIVE</p></figcaption></figure>

{% hint style="info" %}
**You can also share datasets programmatically using ESS-DIVE's Dataset API.**&#x20;

Visit the [Dataset API Tutorial](https://docs.ess-dive.lbl.gov/contributing-data/data-submission-guidelines/package-service-tutorial#getting-started-with-ess-dive-package-service-1.3) page to learn more about this service. Details on how to  write HTTP requests to (1) change dataset permissions and (2) review access permissions are available via the [Data Package Sharing section](https://api.ess-dive.lbl.gov/#/Data%20Package%20Sharing) of ESS-DIVE's technical documentation page, hosted at <https://api.ess-dive.lbl.gov/>.
{% endhint %}


# Share Portals

To share a data portal on ESS-DIVE means that you have given or received **permission to view, edit, or manage** a data portal. Data portal permissions can be changed at any time.&#x20;

Sharing a data portal with other people registered to contribute data on ESS-DIVE will allow them to:&#x20;

* Edit page markdown,&#x20;
* Add/remove pages, and
* Add/remove data filters

{% hint style="warning" %}
Sharing a portal with someone gives them permission to edit your portal page; it **does not** give them permission to edit or manage **any datasets** or view private datasets in the data portal
{% endhint %}

## How to Share Portals

If you have access to **manage** a data portal, you can follow the instructions below to change portal permissions through your portal's settings page. The instructions for changing the sharing options are the same for datasets and data portals, visit the dataset sharing guide to read more about how this feature is used.

1. **Login** to [ESS-DIVE](https://data.ess-dive.lbl.gov/) and select the [Portals](https://data.ess-dive.lbl.gov/portals) button in the navigation bar
   * *Only* [*registered data contributors*](/contributing-data/new-contributor-registration) *can create or share portals, make sure you are logged in and registered before attempting to share a portal*
2. **Find your portal** in the "My Portals" section and click edit, or create a new portal if you do not have one already
3. Open the **portal settings** page on the right-hand side (Figure 1)
4. Scroll down the settings page until you see the **sharing options table** (Figure 2) which shows you who the portal is already shared with and what permission type they have. By default the portal creator has manage permissions.
5. **Share the portal** with new people by searching for their first or last names in the search bar in the sharing options window (Figure 3). Select the appropriate name to be added to the list of people with portal permissions.
   * *Only people who have logged into ESS-DIVE using their ORCID can be found in the Sharing Options search bar*
6. **Modify existing** permission types as needed using the dropdown bars under the Access column
7. Review your changes and hit **Save**

![Figure 1: Open the portals settings to find portal sharing options](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2F9TKWp8W7RTg7Ir0rphfS%2Fportal-settings.png?alt=media\&token=4c7581d6-3750-4274-a7b8-797382e32f8b)

![Figure 2: The sharing options window is towards the bottom of the settings page](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2F6cTB1nqLOj49ztVNaYyo%2Fsharing-options-settings.png?alt=media\&token=8532eb48-7cc8-43e3-b96a-baeeb03a3933)

![Figure 3: Use the search bar to find your colleague's name and add them to the data portal permissions](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FUue5qA5M0hOKyRuw1WzA%2FShare%20Options%20-%20add%20perm.png?alt=media\&token=186edcda-d308-4d1e-9047-dd6eb5339599)


# Manage Project Data

A suite of features used to manage your project's data and teams on ESS-DIVE

ESS-DIVE project management features allow principal investigators (PIs) to access and edit information about their project, assign other project data managers, create teams of project members, and share permissions to datasets and data portals with teams.

**Project management is not set up by default for any project.** Principal Investigators (PIs) must set up project management on ESS-DIVE, using the [recommended workflow](#summary-workflow-for-pis-to-set-up-project-management) below. &#x20;

## Discover Project Management Features

<table data-view="cards"><thead><tr><th></th><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th><th data-hidden data-card-cover data-type="files"></th></tr></thead><tbody><tr><td></td><td><strong>Project List</strong></td><td>Check that your project is on this list, your project name is correct, and the PI list is up-to-date</td><td><a href="/manage-data/manage-project-data#project-list">Manage Project Data</a></td><td><a href="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2Fo6157zsh1akDzxcaD3Se%2FProject_List.png?alt=media&amp;token=10d4bdac-cce1-4157-b0e3-0cde68dbd5c3">Project_List.png</a></td></tr><tr><td></td><td><strong>Project Data Managers</strong></td><td>Gain access to additional information about your project and access the Project Teams feature</td><td><a href="/manage-data/manage-project-data/project-data-managers">Project Data Managers</a></td><td><a href="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2Fob6gUMUsEWfLBYyNZCLQ%2FData_Manage_Page.png?alt=media&amp;token=28712d9e-b766-4901-87d7-0a1e9f34f6b0">Data_Manage_Page.png</a></td></tr><tr><td></td><td><strong>Project Teams</strong></td><td>Create a team of project members on ESS-DIVE and  share datasets or data portals with your team</td><td><a href="/manage-data/manage-project-data/project-teams">Project Teams</a></td><td><a href="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FpZg2IW9YjVPuPstDsU98%2Fteam_multiple_datasets.png?alt=media&amp;token=a0c6fd4e-b780-4016-be90-3d8be5da7c31">team_multiple_datasets.png</a></td></tr><tr><td></td><td><strong>Project Data Portals</strong></td><td>Create a collection of all project datasets on ESS-DIVE</td><td><a href="/manage-data/why-use-data-portals">Create Data Portals</a></td><td><a href="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-Mj1dnqzGlOoV5bJJSgt%2F-Mj1enr7nBO4hjuyKbkT%2Fpublic-portals-list.png?alt=media&amp;token=9417f874-9159-408a-af03-9182efd81b85">public-portals-list.png</a></td></tr><tr><td></td><td><strong>Share with Teams</strong></td><td>Grant your project teams permission to edit datasets and data portals</td><td><a href="/manage-data/manage-project-data/project-teams#share-project-data-with-teams">Project Teams</a></td><td><a href="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FUo9Mmxtx3ES2lq4cSoSw%2Fsearching_for_team_whole_window.png?alt=media&amp;token=108caf39-80b4-4ef5-8309-c1bddf69b23d">searching_for_team_whole_window.png</a></td></tr></tbody></table>

{% hint style="info" %}
**These features are still in development and will be continually updated until Fall 2023.** \
\
[Subscribe to ESS-DIVE's newsletter](https://lbl.us2.list-manage.com/subscribe?u=4a9433904c29fa74aa1127282\&id=b6d9f8dc07) to receive announcements about our latest updates.
{% endhint %}

## Summary workflow for PIs to set up project management

Here we provide a summary of how to get setup with project data management on ESS-DIVE, with links to additional information contained in this page.

1. Make sure your project is in ESS-DIVE's [**project list**](#project-list)**.**
2. Become (or check whether you already are) a [**project data manager**](/manage-data/manage-project-data/project-data-managers)**.**
   * If you are a PI for multiple projects, you'll need to become a data manager for each
3. [**Assign additional data managers**](/manage-data/manage-project-data/project-data-managers#how-to-add-project-data-managers) to your project (if applicable).
   * Visit [**your project page**](/manage-data/manage-project-data/project-information#project-information-page) to make sure the list of data managers is correct
4. Create a [**team**](/manage-data/manage-project-data/project-teams#how-to-create-teams) for your project data managers (if applicable).
   * Name it carefully using our [**recommended rules**](/manage-data/manage-project-data/project-teams#rules-for-naming-teams) (e.g. "\<project-name>-data-managers")
   * Add all project managers to this team&#x20;
   * Make all project data managers owners of this team&#x20;
5. Tell your data contributors to [**share**](/manage-data/manage-project-data/project-teams#share-project-data-with-teams) any new datasets that they create with one or more specific team(s) that you want to have access to project datasets now and in the future.
6. Consider creating a [**data portal**](/manage-data/why-use-data-portals) for your project data. While not a requirement, it will allow you to view all of your project datasets in one place.

## [Project List](https://data.ess-dive.lbl.gov/projects)

ESS-DIVE provides a searchable list of projects that are approved to store and publish data on ESS-DIVE, including all projects funded by the Department of Energy's (DOE) [Environmental System Sciences (ESS) program](https://ess.science.energy.gov/program/).&#x20;

Find your project by searching for the official project title, project short name, or principal investigator (PI) name (Figure 1). An explanation of the project information is listed in Table 1 below.&#x20;

Your project will only appear on this page if it is approved to create and publish datasets on ESS-DIVE. **If you do not see your project, please email ESS-DIVE Support at** [**ess-dive-support@lbl.gov**](mailto:ess-dive-support@lbl.gov)**.**

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FNfB9KJN8QdeJaRuhcxNt%2Fproject-list-logged-in-v2.png?alt=media&amp;token=dab3cbc3-5ab4-47dd-9ad9-a51f55ffbefa" alt=""><figcaption><p>Figure 1: A preview of ESS-DIVE's project information page after login with an ORCID.</p></figcaption></figure>

{% hint style="info" %}
PI information is not visible without sign-in. [Login to ESS-DIVE](https://docs.ess-dive.lbl.gov/manage-data/pages/-Mfnee11wBUSc4TI3WHe#1.-login-to-ess-dive-using-orcid) using your ORCID to view PI information.
{% endhint %}

<table data-header-hidden><thead><tr><th width="273"></th><th></th></tr></thead><tbody><tr><td><strong>Project Title</strong></td><td>Default title submitted to the ESS program at the time funding was awarded. This title is used in dataset citations.<br><br>PIs can update their project title by contacting ESS-DIVE Support at <a href="mailto:ess-dive-support@lbl.gov">ess-dive-support@lbl.gov</a>. Changes to the project title will be reflected in dataset citations. We do not recommend changing the project title, unless required, as this will impact the ability to search and filter by project name and dataset citations.</td></tr><tr><td><strong>ESS-DIVE Project Identifier</strong></td><td>Unique identifier assigned to all projects, which will never change. You will need this identifier when <a href="/programmatic-tools/ess-dive-dataset-api">submitting datasets using the API</a>.</td></tr><tr><td><strong>Principal Investigator (PI)</strong></td><td><p>Default lead principal investigator(s) associated with the project submitted to the ESS program at the time funding was awarded.</p><p></p><p><strong>PI information does not automatically update.</strong> If your PI has changed, update project PI information by contacting ESS-DIVE Support at <a href="mailto:ess-dive-support@lbl.gov">ess-dive-support@lbl.gov</a>. ESS-DIVE will not retroactively update PI info in published datasets.</p></td></tr></tbody></table>

*Table 1. List of all project information fields that are available via ESS-DIVE's project information page.*

## Frequently Asked Questions

<details>

<summary>Why can't I see any project management features?</summary>

For now, project management features are only available to *project data managers*. Only PIs, co-PIs, and project members who received approval from their PI can become project data managers.

To become a data manager, you must first register as an ESS-DIVE *data contributor*. If you are a PI, but can't see the project management features you either are not a data contributor yet or only recently became a data contributor and have yet to inform the ESS-DIVE Team that you are ready to become a project data manager.&#x20;

Make sure you complete all steps outlined in [How do I become a project data manager?](/manage-data/manage-project-data/project-data-managers#how-to-become-a-project-data-manager)

\
If you are a **member of an ESS-DIVE project** but you are **not** a project data manager, then the *My Projects* page and all it's features will not be accessible to you.&#x20;

</details>

<details>

<summary>What are the different kinds of roles available on ESS-DIVE?</summary>

* **ESS-DIVE user:** Someone who logged into ESS-DIVE using their ORCID.&#x20;
* **Data Contributor**: An ESS-DIVE user who was approved by ESS-DIVE to upload data; they can create, edit, and publish datasets.
* **Project Data Manager**: A data contributor approved by a project PI to create and manage teams for that project.
* **Teams**: A group of ESS-DIVE users who share access permissions to datasets and data portals.
* **Team Owner:** A "starred" team member who has permission to add or remove individuals from teams. They can assign new team owners or remove existing ones. By default the project data manager who created the team is a team owner.

</details>


# Project Data Managers

First register yourself as a project data manager to gain access to project management features

A *Project Data Manager* is a role in ESS-DIVE that enables someone to access project information and the [**teams**](/manage-data/manage-project-data/project-teams) feature in ESS-DIVE.&#x20;

Project PIs have automatic approval to become a project data manager, but they must first register as an ESS-DIVE data contributor and notify ESS-DIVE Support. All other members of a project must receive approval from their PI to become a project data manager.

## How to become a project data manager

#### For PIs and Co-PIs

1. Create an ORCID if you do not already have one
2. Login to ESS-DIVE with your ORCID to start your ESS-DIVE account
3. [Register as a data contributor](/contributing-data/new-contributor-registration)
4. Notify ESS-DIVE support via email (<ess-dive-support@lbl.gov>) that you are ready to become a data manager for your project

You will receive a confirmation message from ESS-DIVE when you have been added as a data manager.

#### For all other project members:

Only PIs or existing project data managers can add new project data managers.

You will need to ask your PI/project data manager to fill out the Add Project Data Manager form (see below) to gain approval to become a project data manager. Direct your PI to the instructions provided above.

In the meantime, you can [register as a data contributor](/contributing-data/new-contributor-registration).&#x20;

## How to add project data managers

Only PIs or existing project data managers can add new project data managers.

1. Tell your project data managers to [register as ESS-DIVE data contributors](/contributing-data/new-contributor-registration)&#x20;
2. Fill out the [Add Project Data Manager form](https://docs.google.com/forms/d/e/1FAIpQLSf2x54tnjCmoAtcP6c9-GVXg-rDvzutCdc5OCm0uWr7evVMnA/viewform)

## Check whether you are a data manager

You can check whether you are currently a data manager for a project by selecting "[Find Projects](https://data.ess-dive.lbl.gov/projects)" from the main data portal navigation menu: if you are a data manager, you will see a "My Projects" tab that will list any project(s) for which you are a data manager.&#x20;

<details>

<summary>How can I tell if I'm already a project data manager?</summary>

1. Login to ESS-DIVE at [https://data.ess-dive.lbl.gov/](https://data.ess-dive.lbl.gov/data)
2. Click the "Projects" tab in the navigation bar and select "Find Projects" or navigate to the page directly <https://data.ess-dive.lbl.gov/projects>
3. If you do not see the "My Projects" tab, then you are not a project data manager on any project
4. If you see the "My Projects" tab, click it to review which projects you can manage

Visit Project Information for more details on the "My Projects" page.

</details>


# Project Information

Project Data Managers can these pages to review and update project metadata as needed.

Approved project data managers will gain access to additional pages on ESS-DIVE that are used to view and update project information. Use these pages to see which projects you have approval to manage, review information ESS-DIVE stores about your project, and correct information that may be out of date.

{% hint style="info" %}
These pages are accessible only to project PIs, co-PIs, and other project data managers.&#x20;
{% endhint %}

## View My Projects

The *My Projects* page lists all the projects that you have been assigned to as a project data manager. Access this page by selecting "Projects" from the top-level navigation bar, and you will then see the "All Projects", and "My Projects" tabs (Figure 1).&#x20;

If you are a **PI or project data manager** and do not see one of your projects listed, please contact ESS-DIVE at <ess-dive-support@lbl.gov>. This means you have not been assigned as project data manager for one of your projects.

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2F8jxpgGoFCkcx54n0tPoa%2FMy_Projects.png?alt=media&amp;token=279a9d7e-4f24-46c8-b466-a86a510c984a" alt=""><figcaption><p>Figure 1: The "My Projects" tab allows you to see which projects you can manage.</p></figcaption></figure>

## Project Information page

The *Project Information* page is used for reviewing important metadata about your project that ESS-DIVE tracks (Figure 2). Use this page to review who the data managers are for your project.

Project PIs, co-PIs, or data managers can update the information on this page at any time by contacting ESS-DIVE at <ess-dive-support@lbl.gov>.

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2Fob6gUMUsEWfLBYyNZCLQ%2FData_Manage_Page.png?alt=media&amp;token=28712d9e-b766-4901-87d7-0a1e9f34f6b0" alt=""><figcaption><p>Figure 2: Review who has permission to manage your project data on ESS-DIVE on the "Project Information" page.</p></figcaption></figure>

## My Teams page

This is how you access the teams feature. See our [Project Teams](/manage-data/manage-project-data/project-teams) documentation to learn more about the *My Teams* page.

{% content-ref url="/pages/baehrEnBF0s03cVVj1j4" %}
[Project Teams](/manage-data/manage-project-data/project-teams)
{% endcontent-ref %}


# Project Teams

Create project teams to leverage ESS-DIVE's evolving suite of project management features

**Teams** are groups of ESS-DIVE users, which can only be created by the project data managers.

Data contributors can now share datasets with teams, granting the team access to their project data.

#### Table of Contents

* [My Teams](#my-teams)
* [Why Create Teams](#why-create-teams)
* [How to Create Teams](#how-to-create-teams)
  * [Rules for Naming Teams](#rules-for-naming-teams)
* [How to Manage Teams](#how-to-manage-teams)
* [Share Project Data with Teams](#share-project-data-with-teams)
  * [How to share datasets with teams](#how-to-share-datasets-with-teams)
  * [How to share Portals with Teams](#how-to-share-portals-with-teams)
* [View Teams Datasets](#view-team-datasets)
* [Frequently Asked Questions](#frequently-asked-questions)

{% hint style="info" %}
Only project data managers can access and manage teams
{% endhint %}

## My Teams

The *My Teams* page (Figure 1) is used to view the teams you are a member of, create new teams, or manage your existing teams.&#x20;

You can only view teams **you are a member of**. It is not possible to view all teams created within a project if you are not a member of every team.

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FHps9vXR5g4j0ngoViIjH%2FMy_Teams.png?alt=media&amp;token=336367d6-df4d-46ca-a902-fd19e3698713" alt=""><figcaption><p>Figure 1: The "My Teams" page lists all teams you are a part of within your project</p></figcaption></figure>

## Why create teams?

The primary motivation to create a team is to make it easier to share project datasets with groups of project members. A team will always have access to datasets (once they have been granted "manage" permission for the dataset(s)), even as people leave and join the project; you can simply update who is on the team, rather than updating each dataset permission individually.&#x20;

You may be interested in creating teams if you have ever wanted to have groups of:&#x20;

* **Data managers** to oversee project dataset management, permissions, and publication
* **Collaborators** who need to edit the same datasets together
* **Reviewers** who need to view private datasets

Additionally, **project data managers will not be added to a team by default**; make sure to add your fellow data managers to your team.

## How to Create Teams

**Only project data managers can create teams**. If you'd like to become a project data manager, reach out to your project's PI and request that they add you; see the [Project Data Manager](/manage-data/manage-project-data/project-data-managers) section for more information.&#x20;

The following instructions detail how to create and add people to a team.&#x20;

1. Head to the My Teams page (Figure 1, above)
2. Select "+Add Team" at the bottom of the page
3. Enter a team name carefully, following the [Rules for Naming Teams](#rules-for-naming-teams) section below
4. Add team members by searching for their first or last name or ORCID
   * If they have not logged in to ESS-DIVE, then you won't be able to find them
5. Select "Create Team"
6. To ensure that you successfully created your team, you can select the "View it now" button or return to the "My Teams" page
7. Add at least one co-owner to your team, see the [How to Manage Teams](#how-to-manage-teams) section below for details

### Rules for Naming Teams

It is important to carefully choose your team name because teams cannot be renamed or deleted later. When you share a dataset with this team, you will have to look it up and select it from a dropdown list in which *every team created by any ESS-DIVE project* will appear as options. By making your team name unique and easy to understand, it will be easy for you and your project members to find your team.&#x20;

* **Choose your team name wisely**: Team names must be 50 characters or less and unique across DataONE. Teams cannot be renamed or deleted. &#x20;
* **Use a common prefix for all project teams**: Consider starting every team name with a reasonably unique and brief name or common short hand name for your project. This will enable easier search and lookup when sharing a dataset or portal with a team.&#x20;
* **Use the access level or purpose as a suffix**: Consider ending every team name with a suffix that illustrates what the group is for. We recommend using one of the sharing access levels  (i.e. "project-nam&#x65;**-view**" or "project-nam&#x65;**-edit**") or other suitable terms such as "-admin", "data-managers", "-all", or "-reviewers". This will enable easier search and lookup when sharing a dataset or portal with a team.&#x20;
* **Have one team for project data managers only**: ESS-DIVE recommends having one team for your project's designated data managers only, and naming it accordingly (e.g. ess-dive-data-managers). If you add a new data manager to this team, make sure they are also on your list of project data managers.

## How to Manage Teams

Manage your team by assigning co-owners and adding, or removing members. Team *co-owners* are people who can add or remove members from the team. By default, the person who created the team is a team owner.

We recommend adding at least 1 other co-owner to every team. This is a precaution in case the person who created the team leaves the project.

* **Changes to a team may take a few minutes to load.** Please wait and reload the page if you do not see your team changes right away.
* Remove team members by clicking the red "x" icon (Figures 2a and 2b).
* Assign team owners by starring the names of people you have added to the team (Figures 2a and 2b).
* Only project data managers can add or remove people from a team as a co-owner.
* Un-assign team owners by clicking the gold star next to the team members name.
* You cannot remove yourself from any team. Ask another team owner to remove you.&#x20;
* The person who created the team cannot be removed from the team.
* **You can add anyone to a team who has logged into ESS-DIVE with their ORCID**; their name won't show up if they haven't done this.&#x20;

<div><figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2Fy1yk3tUmxDeEDFj9FjmK%2FMake_group_owner.png?alt=media&amp;token=f704403a-10ed-44f4-9d87-26f21923e41e" alt=""><figcaption><p>Figure 2a. This person is not a team owner. Designate team owners by selecting the empty star next to their name.</p></figcaption></figure> <figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FEc6Egg075nREbLb7CRxJ%2FGroup_owner.png?alt=media&amp;token=1203c0c8-b78e-4354-8292-c0f2b9b1a3dc" alt=""><figcaption><p>Figure 2b. This person is a team owner. Select the yellow star next to their name to un-assign them as a team owner.</p></figcaption></figure></div>

## Share project data with teams

Now that you have your teams, what can you do? Encourage your project members to start sharing datasets or data portals with teams!&#x20;

Data contributors can easily share their datasets with a group of people all at once, rather than share with one person at a time. As datasets get shared, teams will eventually have a list of datasets that team members can access (Figure 3). Project managers can then add or remove people from that team to manage who can access that list of datasets.&#x20;

One benefit of sharing with teams is that you can change the team over time as people enter or leave the project.

{% content-ref url="/pages/xweNLgvrvm8Np7Ee79hS" %}
[Share Data Permissions](/manage-data/share-data-permissions)
{% endcontent-ref %}

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FpZg2IW9YjVPuPstDsU98%2Fteam_multiple_datasets.png?alt=media&amp;token=a0c6fd4e-b780-4016-be90-3d8be5da7c31" alt=""><figcaption><p>Figure 3: The ess-dive-data-managers team profile shows a list of team members and a list of datasets those team members can access.</p></figcaption></figure>

Teams provide a whole new way of sharing datasets. We recommend informing data contributors on your project about the project team(s) you created and ask them to share their datasets with any team(s) that you would like to have access to project datasets now and in the future.&#x20;

{% content-ref url="/pages/a2K40XOKJS5ozvn9soPK" %}
[Share Portals](/manage-data/share-data-permissions/share-portals)
{% endcontent-ref %}

Consider sharing your project portal with your team and work together to **edit** the portal. Or give your project team permission to **view** your portal before it's public to give them early access to the project's data collection.

{% hint style="warning" %}
Project managers and teams **do not** automatically have manage access to project datasets
{% endhint %}

### **How to share datasets with teams**

When teams are created they do not have access to any ESS-DIVE datasets by default. To enable team access to datasets, data contributors will need to share their datasets with the team (Figure 4).

{% hint style="info" %}
To share datasets, it is necessary to have permission to **manage** the dataset being shared. Only the data contributor who created the dataset will have manage access by default. They can then give manage access to others by sharing their dataset with them.&#x20;
{% endhint %}

Data contributors who are sharing datasets will select which access level the team will receive; **view**, **edit**, or **manage**.

Sharing datasets with teams works the same as sharing with individuals! Simply lookup the team name in the sharing options window (Figure 4). See the [**Share Datasets**](/manage-data/share-data-permissions/share-datasets) documentation page for detailed instructions.

{% hint style="info" %}
If you don't see your team, contact your PI/data manager to get the team name
{% endhint %}

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FUo9Mmxtx3ES2lq4cSoSw%2Fsearching_for_team_whole_window.png?alt=media&amp;token=108caf39-80b4-4ef5-8309-c1bddf69b23d" alt=""><figcaption><p>Figure 4: Search for a team in the sharing options panel by typing the team name.</p></figcaption></figure>

### How to Share Portals with Teams

A project data portal will allow your team of project data managers to view all public and private project datasets in one place. See thee [Create Data Portals](/manage-data/why-use-data-portals) documentation page to learn how to create a portal.

* [ ] Tip: you can share this data portal with your team of project data managers as well so you can edit it together.

Do you want to give all project members the ability to view private datasets? You can share view permissions to your project portal with your team of project members to give everyone early access to see your project's data collection.

## View Team Datasets

**Project managers** can use the team profile page to see exactly what datasets that team has access to.&#x20;

**How to find your team profile:**

1. Go to your My Teams page&#x20;
2. Scroll down to the team you're interested in and click on the hyperlinked team name
3. You'll be directed to the team profile; it will list all datasets the team has access to along with some metrics about those datasets (Figure 5 and 6).

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FpZg2IW9YjVPuPstDsU98%2Fteam_multiple_datasets.png?alt=media&amp;token=a0c6fd4e-b780-4016-be90-3d8be5da7c31" alt=""><figcaption><p>Figure 5. Team profiles list all datasets that the team has access to</p></figcaption></figure>

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2F1ce7AZTUiBQql6miESUe%2FSummary_Stats_Team.png?alt=media&amp;token=c8c4dca8-fda8-4ce4-86cd-b93d9bfcf7d1" alt=""><figcaption><p>Figure 6. Team profiles contain summary metrics of all datasets the team has access to</p></figcaption></figure>

## Frequently Asked Questions

#### Manage Teams

<details>

<summary>Why can't I see any teams on My Teams page?</summary>

If you cannot find a team on your My Teams page, you are not a member of any teams yet. Contact the other project data managers on your project and ask the one who created the team to add you.&#x20;

</details>

<details>

<summary>Why can't I add someone to my team?</summary>

**Having trouble finding an ESS-DIVE user to add to your team?** Make sure that they have logged into ESS-DIVE with their ORCiD.

*It is not possible to add someone to a team if they have not logged in to ESS-DIVE before*

</details>

#### Sharing with Teams

<details>

<summary>I was told my team has edit access, but I can't edit any datasets; what's going on?</summary>

You must register to become a data contributor to edit datasets. It's not possible to edit a dataset without completing this step, even if a team has edit access.

Teams share permission levels to all team members, but it doesn't give you the ability to edit datasets.

**Creating a team of dataset editors?** Your team members will need to register as ESS-DIVE *data contributors* in order to be able to edit datasets.

*It is not possible to edit a dataset without becoming a data contributor, even if you are in a team that has edit permission to a dataset.*

</details>


# Search for Data

### ESS-DIVE Overview

The ESS-DIVE main data portal (Figure 1) is made up of three sections - a search bar on the left, a dataset listing in the middle, and a map on the right.

By default, you will see all public datasets on ESS-DIVE. If you wish to view your unpublished datasets or download data, click on the “Sign in with Orcid” button in the top right corner and login with your ORCID credentials.

![Figure 1. ESS-DIVE's main data portal](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-Mj1OEOK80Q6bTq1oE40%2F-Mj1PEs0NrU67_K5Us51%2Fess-dive-main-data-portal.png?alt=media\&token=0fbc03b9-182f-4223-81c8-2586ca29092d)

Previews of each dataset listed include citations in the format: **Authors (Publication Date): Dataset Title. Project. DOI or ESS-DIVE ID** (Figure 2).

![Figure 2. ESS-DIVE dataset previews with citations](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-MdclyptOp8JSxeblIOS%2F-Mdcs_V6Q7oJqFIKZu3p%2FScreen%20Shot%202021-07-02%20at%201.21.38%20PM.png?alt=media\&token=e36c3036-4a04-402f-a77e-69f418bed4c3)

There are various symbols that appear beneath the data citation (not all of them appear on all the datasets):

| Icon                                                                                                                                                                                                                                                                                                                                | Meaning                                                                                                                                                                                          |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| <p><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2Fess-dive-docs%2F-LgZxMCpHjS5gBW_i5LG%2F-LgZxP1uncN1B2yaGug8%2F2.png?generation=1559693207658899&amp;alt=media" alt=""></p><p>Orange lock</p>                                                                                          | The record is unpublished, and only visible to you when you are logged in with your ORCID account. If the lock is not visible, the dataset is public and can be viewed and downloaded by anyone. |
| <p><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-MdclyptOp8JSxeblIOS%2F-Mdct3jBZomOZOf1cdG-%2FScreen%20Shot%202021-07-02%20at%201.23.46%20PM.png?alt=media&amp;token=8e63522d-8b40-4247-ab19-5753650b3494" alt="" data-size="line"></p><p>Number with eye </p> | This refers to the number of times this dataset has been viewed.                                                                                                                                 |
| <p><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2Fess-dive-docs%2F-LgZxMCpHjS5gBW_i5LG%2F-LgZxP1xh8IXHEQSq4oF%2F5.png?generation=1559693207682907&amp;alt=media" alt=""></p><p>i in circle     (Info) </p>                                                                              | Displays a preview of the dataset abstract.                                                                                                                                                      |
| <p><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2Fess-dive-docs%2F-LgZxMCpHjS5gBW_i5LG%2F-LgZxP1y5YCHY0EH8Fs8%2F6.png?generation=1559693207660178&amp;alt=media" alt=""></p><p>Paper with table (Files) </p>                                                                            | Indicates whether there are any data files associated with the dataset. If this icon is not visible, it means only metadata are available for the dataset.                                       |
| <p><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2Fess-dive-docs%2F-LgZxMCpHjS5gBW_i5LG%2F-LgZxP1zXu_FE76Z2xPt%2F7.png?generation=1559693207661172&amp;alt=media" alt=""></p><p>Map marker</p>                                                                                           | Refers to how many other datasets are associated with that location. Hovering over this icon will highlight the corresponding location on the map.                                               |

You can choose to hide Map by clicking on the Hide Map link (Figure 3), which will allow you to view more listings on your web page (Figure 4).

<div align="center"><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2Fess-dive-docs%2F-LgZxMCpHjS5gBW_i5LG%2F-LgZxP2-HRAQlmNusXzX%2F8.png?generation=1559693207656121&amp;alt=media" alt="Figure 3. The link to hide the map."></div>

![Figure 4. View of the home page without the map](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-Mj1OEOK80Q6bTq1oE40%2F-Mj1PcBIK_1y5hUdfY76%2Fhome-page-without-map.png?alt=media\&token=d46e4c41-4d3c-4c27-b900-fbbc35ebb203)

### Data Search Tools

You can search for data using the options in the left sidebar (Figure 6). It will allow you to do a generic search for dataset attributes. (e.g. Title, keyword, PI name). You can also refine the search by using more specific fields - identifier (e.g. DOI), location, creator (authors), and year (publication year or date range spanned in the dataset).

![Figure 5. Expanded Search options](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2Fess-dive-docs%2F-LgZxMCpHjS5gBW_i5LG%2F-LgZxP224FLXxgYAivEh%2F11.png?generation=1559693207663397\&alt=media)


# Download Data

Public data files on ESS-DIVE can be accessed by anyone through unique dataset landing pages, where they are accompanied by relevant metadata. There is no login requirement to download data.

## View Dataset Metadata&#x20;

Once you have found a dataset using the [search capabilities](/searching-and-accessing-data/data-access) of the main data portal, click on the dataset title. This will bring up the dataset landing page, which is a page with the dataset citation at the top, a list of included data files, and accompanying metadata fields (Figure 1). Metadata provides important information about the purpose for and collection of the data included and helps to support data reuse. Scroll below the files section to view the metadata fields for the dataset.

![Figure 1. Dataset landing page with data file table and accompanying metadata.](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2F4kBY4UXyKJCd48XcKork%2Fdataset-metadata-landingpage-2025.png?alt=media\&token=e3106fed-e887-4656-b442-ae1fa5cdd05b)

Below are the metadata fields available on each ESS-DIVE landing page and their descriptions. Only fields that have been completed by the data contributor will appear on the landing page. For more information about the metadata fields required by ESS-DIVE, please visit our [Package Level Metadata Guide](/contributing-data/package-level-metadata).&#x20;

* **Identifier/Alternate Identifiers:** Identifiers in ESS-DIVE or other systems for this dataset (includes the DOI if applicable)
* **Abstract:** Brief description of the dataset
* **Keywords and variables:** Categorical Keywords that indicate the general themes of this dataset, and data variables associated with the package
* **Publication Date:** Date this package was published
* **Data Table, Image and Other Data Details:** The details about the files uploaded, including files sources and derivations
* **People and Associated Parties:** Creators, Contact, Contributors, and related Funding agencies.
* **Geographic Region:** Information about where the data were collected, along with a map view
* **Temporal Coverage:** Date period the data spans
* **Project Information:** Information about the DOE project the data is associated with
* **Methods:** Methods that were used to produce the data, including processing, QA/QC, site information etc.
* **Usage Rights:** The usage rights under which this dataset is released. Anyone using the data must comply with the usage rights specified on the package.

## Download Data Files

Data files are displayed in a table at the top of the dataset landing page, below the citation (Figure 2). Here you can view the name, file type, and size of each datafile and select which ones you would like to download. Alternatively, you can select the download icon at the top to download all files as a zip file.&#x20;

![Figure 2. View of the data file table with options for downloading.](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FBkRyQBWM0PKbnkyY2qcj%2Fdataset-metadata-landingpage-files-2025.png?alt=media\&token=de72d6d7-a023-4045-83df-fd14efab9f29)

## Download Large Data on Tier 2

Tier 2 data files are accessible for download via the external link table at the top of the dataset metadata landing page (Figure 3). In this table, there are two options to choose from when downloading Tier 2 data: HTTP or Globus.&#x20;

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2Fk51koh2vtMmid7XWFlId%2Fdataset-with-tier2-2025.png?alt=media&amp;token=34f0c191-2875-4c5e-ba8d-c366754f0763" alt=""><figcaption><p>Figure 3: Tier 1 dataset landing page where metadata and data can be discovered and downloaded. Access to files on Tier 2 are provided via external link.</p></figcaption></figure>

### **Tier 2 (HTTP):**&#x20;

The HTTP access link can be used to download individual files directly from your browser. For those familiar with command line tools, it can be used to download data in bulk.

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2F2FQu8ulUcsDyZepr0W56%2Ftier2-landing-page-2025.png?alt=media&amp;token=299f672f-e8e4-4e1f-a38b-41c985c919ca" alt=""><figcaption><p>Figure 4: Tier 2 landing page for large file exploration and download. Too see the dataset metadata, select the "Access via ESS-DIVE" link.</p></figcaption></figure>

1. Scroll down to the "External Links to Data and Metadata" table and select the URL next to the external link titled "ESS-DIVE Tier 2 (HTTP)"; this will redirect you to the Tier 2 data page (Figure 4).
2. All data files will be located in the `data/` folder.
3. You can browse and download files individually as needed by selecting the file.

If you are familiar with command line tools, you can initiate a bulk download with `wget` or `curl` by pulling all the files in the data folder. The `manifest-md5.txt` contains a list of all files with MD5 sums.

* *Example code coming soon*

### **Tier 2 (Globus):**&#x20;

The Globus access link can be used to download multiple files at once from your browser. Additionally, you can choose to download the files locally or to transfer them to an existing cloud storage service, if applicable. To download files locally, it is necessary to install Globus Connect Personal.

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FvIVuQSvbETbnu81Topni%2Fglobus-tier2-data.png?alt=media&amp;token=cb2a8a78-7487-44c3-bbbb-23197f85bb36" alt=""><figcaption><p>Figure 5: Tier 2 data files can be browsed and downloaded from the Globus file manager.</p></figcaption></figure>

1. Scroll down to the "External Links to Data and Metadata" table (Figure 3) and select the URL next to the external link titled "ESS-DIVE Tier 2 (Globus)"; this will direct you to Globus.
2. You will be prompted to login to access the data on Globus (Figure 5). If you are new to Globus, see ESS-DIVE's instructions to [Create a Globus Account](/programmatic-tools/large-data-support/how-to-setup-globus).&#x20;
   1. **Individual file download:** At this point, you have the option to select individual files for download. **Continue reading to download multiple files at once or entire folders**.
3. If you are new to Globus, see ESS-DIVE's instructions to [Install Globus Connect Personal](https://docs.ess-dive.lbl.gov/searching-and-accessing-data/pages/-MX-liL4i-S8I9i6XJmB#id-2.-install-globus-connect-personal) on your desktop. Return to the File Manager in your browser (Figure 5) when you have finished setup.
4. In the Globus File Manager, use the panel toggle in the top right to change your view to two panels (Figure 6)**.**
5. In the empty panel, select the “Collection” search bar and navigate to the “Your Collections” tab (Figure 7). Then select your desktop collection. Alternatively, you can select any other existing cloud endpoint you may have on Globus.&#x20;
6. Now you’re ready to download (i.e. transfer) the data. On your desktop panel, navigate to where you want to download the files. On the dataset panel, select some or all files to download then either use the “Sync transfer...” button or the "Start" button to start the transfer (Figure 8). You can also drag the selected files to your desktop.
7. This will start a transfer job and you will see a green popup window on your screen confirming that the transfer has started (Figure 9).&#x20;

In many cases the transfer is quick, but if your transfer is taking a while, use the link from this popup window to check in on the status of the transfer or look in the Activity tab. You will know that the transfer is complete when Globus sends you a confirmation email.&#x20;

<div><figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2F3lVfKN8DeqavKkGVz1Mp%2Fglobus-tier2-two-panels.png?alt=media&amp;token=d4858d71-ee55-4d63-bc0c-90993a1fccd2" alt=""><figcaption><p>Figure 6</p></figcaption></figure> <figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2F1DBpGjXrpiLrLrzQeRwc%2FGlobus-Your-Collections.png?alt=media&amp;token=ea8fe8c3-6d1f-4964-9cba-a54b30d85d92" alt=""><figcaption><p>Figure 7</p></figcaption></figure> <figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FoMjWKg1lj4QVDxjk76nv%2Fglobus-start-tier2-download.png?alt=media&amp;token=afe8ff73-c21b-4151-bb24-2041551e6090" alt=""><figcaption><p>Figure 8</p></figcaption></figure> <figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FX9tlNXyP4xZ1zkMcQjsK%2Fglobus-tier2-transfer-started.png?alt=media&amp;token=3dd062ff-d8c7-437a-b880-489e10b614ef" alt=""><figcaption><p>Figure 9</p></figcaption></figure></div>


# Access Data Portals

Visit the portal page to view a list of all portals published on ESS-DIVE.

## [The Portal Page](https://data.ess-dive.lbl.gov/portals)

The Portal page shows a list of all portals published on ESS-DIVE. This feature is easily accessible from [the ESS-DIVE Repository](https://data.ess-dive.lbl.gov/data) main menu. The Portal page has an "All Portals" section for listing public portals and a "My Portals" section that is available to **data contributors** for listing their private portals.

![Figure 1. Portals navigation menu item](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FcAmVEMwNC6vieEx4fnnH%2FAcess%20Data%20Portals.png?alt=media\&token=d0b972ce-9a82-4fcc-9364-f3b64366c460)

## All Portals

Public portals are accessible to anyone via the Portal page. Portals are listed under the header "All Portals" and are ordered by date most recently updated.&#x20;

![Figure 2. A list of all portals that are public on ESS-DIVE](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-Mj1dnqzGlOoV5bJJSgt%2F-Mj1enr7nBO4hjuyKbkT%2Fpublic-portals-list.png?alt=media\&token=9417f874-9159-408a-af03-9182efd81b85)

{% hint style="warning" %}
At this time, there is no way to search through public portals with keywords.&#x20;
{% endhint %}


# Search with Dataset API

Public datasets can be programmatically searched and downloaded by anyone using the Dataset API. ORCID login is required to use the API.

**Dataset API Resources**: [Setup](/programmatic-tools/ess-dive-dataset-api#get-authentication-token) | [Technical Documentation](https://api.ess-dive.lbl.gov/#/) | [Search Tutorials](https://github.com/ess-dive/essdive-tutorials/tree/main/search_data) | [Search Code Examples](/searching-and-accessing-data/search-with-dataset-api/code-examples)

## About Searches with the Dataset API&#x20;

The Dataset API is a great tool for searching through multiple datasets at the same time. Unlike the website (<https://data.ess-dive.lbl.gov/>), you can search for **exact keyword matches** and download multiple datasets at once.

**Query Parameters:**

* **providerName**: The ESS **project** that published the dataset. Exact project names are required, find yours using [ESS-DIVE's Project List](https://data.ess-dive.lbl.gov/projects)
* **keywords**: Search for datasets that have an exact match for all the given keywords
* **creator**: The creator/submitter of datasets
* **text**: Searches any metadata field that contains the passed text
* **datePublished**: Dataset publication date
* **beginDate** & **endDate**: Temporal range of the data
* **bbox**: Coordinate bounding boxes of the data
* **lat, long, radius**: Specified coordinate range for the data
* **sort**: Order in which the datasets are returned

To see examples of what a request query using these parameters looks like, go to our Code Examples page:&#x20;

{% content-ref url="/pages/6YrdSdPFBsVsGcV7iU8m" %}
[Code Examples](/searching-and-accessing-data/search-with-dataset-api/code-examples)
{% endcontent-ref %}

## Dataset API Search Quick Start&#x20;

Get started with ESS-DIVE's Dataset API right away with our Jupyter Notebook tutorials! No coding knowledge is necessary to run the notebooks. Follow these quick steps to check it out on our GitHub.

#### 1. Visit ESS-DIVE's API Tutorial GitHub

*Dataset API search tutorials are currently only offered in Python.*

{% embed url="<https://github.com/ess-dive/essdive-tutorials/tree/main>" %}
Click this link to go to the GitHub.
{% endembed %}

#### 2. Launch the notebooks with the click of a button

You can select the **Google Colab** or **Jupyter+R Binder** button to launch the tutorials. We recommend Google Colab for a smoother experience (*a Google account is required to use Colab*).

<div data-full-width="true"><figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FPdw0dyddPYRW5loumjke%2FScreenshot%202025-04-10%20at%205.54.15%E2%80%AFAM.png?alt=media&amp;token=bc01fe25-a98d-466b-8361-5340194523f4" alt="" width="375"><figcaption><p>At the top of the Github README, select preferred tool.</p></figcaption></figure> <figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FB2gSK73QcSlIRVW3fORC%2FColab-startup.png?alt=media&amp;token=b35b7cef-4dca-497c-9e02-4efd8781b563" alt="" width="375"><figcaption><p><strong>Google Colab Option:</strong> Select one of the tutorials.</p></figcaption></figure> <figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FU01uT7x5oMqvTlS3OSEr%2FScreenshot%202025-04-10%20at%205.57.20%E2%80%AFAM.png?alt=media&amp;token=f6165aff-48d3-427f-97b3-6aaf8ad75338" alt="" width="375"><figcaption><p><strong>Jupyter Binder Option:</strong> Select a folder and then a tutorial.</p></figcaption></figure></div>

{% hint style="info" %}
You'll need to **login** to [ESS-DIVE](https://data.ess-dive.lbl.gov/data) with your ORCID to search with the Dataset API. See steps 1 and 2 in our [registration instructions](https://docs.ess-dive.lbl.gov/contributing-data/new-contributor-registration#id-1.-create-an-orcid) for help logging in.
{% endhint %}

#### 3. Open the search\_data folder

We recommend starting with the tutorial titled `ESS PI Meeting 2025: Using Data - Python.ipynb`. Follow the instructions to enter your login information. Click the play icon to run the cells and see the Dataset API in action.&#x20;

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FZSoBgf8ndJY6I2soBuOW%2FColab-searchDataNB.png?alt=media&amp;token=354b1fef-1140-4a34-9194-2d2c081bf563" alt=""><figcaption><p>Preview of the Google Colab interface</p></figcaption></figure>

***

## **What can you do with the Dataset API Search Capabilities?**

Once you have multiple metadata and data files in hand, you can perform custom searches and visualizations directly in the notebook. Here's an idea of some of the things you can do that are demonstrated in ESS-DIVE's data search tutorials:

* Review and download relevant datasets and their files,&#x20;
* Preview file sizes,
* Create publication reports,
* Look up the view, download, and citation **metrics** for each dataset,
* Get data citations,
* Visualize data directly after finding datasets, and
* Leverage the [**DeepDive API**](/searching-and-accessing-data/search-with-deep-dive-api) along with the **Dataset API** for even more powerful file-level search!

## What's Next?

After you've identified some datasets you're interested in, try out the Deep Dive API to query and explore certain\* file contents!&#x20;

*\*Available for datasets following the* [*File Level Metadata (FLMD)*](https://ess-dive.gitbook.io/file-level-metadata-reporting-format) *and* [*CSV Guidelines*](https://ess-dive.gitbook.io/csv-file-structure-reporting-format) *Reporting Formats*

{% content-ref url="/pages/uKUP9sEXNWacKKuAp2Zn" %}
[Search with Deep Dive API](/searching-and-accessing-data/search-with-deep-dive-api)
{% endcontent-ref %}


# Code Examples

This page provides quick coding examples for searching for datasets on ESS-DIVE using Python, R, and Java.

## Setup

The following code examples require installation of certain packages and authentication from your ESS-DIVE account. Follow the instructions for setting up the Dataset API for your preferred coding language before trying out the search code examples:

{% content-ref url="/pages/GTgJ2cW6V8fNrNXc5cZC" %}
[Setup and Troubleshoot](/programmatic-tools/ess-dive-dataset-api/setup-and-troubleshoot)
{% endcontent-ref %}

***

## **Search for Datasets**

Anyone can search for public datasets on ESS-DIVE using the Dataset API. If you are registered to submit data, you can also search for your private datasets. Query your dataset searches by defining parameters.&#x20;

Limited dataset metadata are returned in the response of this call. Additionally, this call cannot be used to download data files. To look up all dataset metadata and download data files, use the API to [**Download Datasets**](/searching-and-accessing-data/search-with-dataset-api/code-examples#download-dataset)**.**

:point\_right: Review **parameters** and **responses** for this call in the [Dataset API technical documentation](https://api.ess-dive.lbl.gov/#/Dataset/listDatasets).

The following lines of code will return the most recent 25 records. If the results contain more than 25 packages, use the `rowStart` and `pageSize` query parameters to page through the results.

{% hint style="warning" %}
**For data users:** Must pass `isPublic=true` to search for <mark style="color:blue;">**public datasets**</mark>. This call searches for public datasets by default.&#x20;

**For data contributors:** Must pass `isPublic=false`  to search for private datasets.
{% endhint %}

{% tabs %}
{% tab title="Python" %}

## Python Search Example

:star:  **Make sure you have followed the Setup instructions before running this code**

Search for datasets using any of the available query parameters. The code example below demonstrates how to format the search query with all parameters. It also demonstrates how to search for one parameter, in this case `providerName` or the publishing project.

```python
# Enter parameters
creator= "creator/submitter of datasets"
providerName =  ""\"Next-Generation Ecosystem Experiments (NGEE) Arctic"\" # Use exact match formatting to return datasets from only one project
text= "any text"
datePublished = "[YYYY-MM-DD]"
keywords = "Partial Match Keywords" # OR ""\"Exact Match Keywords"\"
```

```python
# Contruct query URL
get_packages_url = "{}/{}?creator=\"{}\"&providerName=\"{}\"&text=\"{}\"&datePublished=\"{}\"&keywords=\"{}\"&isPublic=true".format(base,endpoint,creator,providerName,text,datePublished,keywords)

## Not interested in using all the parameters? Just remove them from the query like so
## This example only uses the "providerName" parameter:
# get_packages_url = "{}/{}?providerName=\"{}\"&isPublic=true".format(base,endpoint,providerName)
```

```python
# Call GET /packages
get_packages_response = requests.get(get_packages_url, 
    headers={"Authorization":header_authorization})

if get_packages_response.status_code == 200:
   #Success
   print(get_packages_response.json())
else:
   # There was an error
   print(get_packages_response.text)
```

{% endtab %}

{% tab title="R" %}

## R Search Example

:star:  **Make sure you have followed the Setup instructions before running this code**

Search for datasets using any of the available query parameters. At this time, this code example does **not** demonstrate how to format the query parameters. This example will return all public datasets on ESS-DIVE.

```r
# Construct query URL
call_get_packages <- paste(base,endpoint, sep="/")
```

```r
# Call GET /packages
get_packages = GET(call_get_packages,
       add_headers(Authorization=header_authorization))
```

```r
# Transform the result into a data frame. (Ignore the warning)
get_packages_text <- content(get_packages, "text")
get_packages_json <- fromJSON(get_packages_text)
get_packages_df <- as.data.frame(get_packages_json)
```

```r
# Check the errors and view the data frame on success

# Check for errors
if(!http_error(post_package) ){
  # Print the returned columns
  print(colnames(get_packages_df))

  # print the ESS-DIVE Ids
  print(get_packages_df['result.id'])

  # Iterator over the dataset column and print the data package name
  for ( d in get_packages_df['result.dataset']) { print(d['name']) }
}else {
  http_status(post_package)
}
```

{% endtab %}

{% tab title="Java" %}

## Java Search Example

:star:  **Make sure you have followed the Setup instructions before running this code**

Search for datasets using any of the available query parameters. At this time, this code example does **not** demonstrate how to format the query parameters. This example will return all public datasets on ESS-DIVE.

```java
  try{
    String url = base + endpoint;
    HttpGet request = new HttpGet(url);
    StringEntity params = new StringEntity(JSON_LD.toString()); //Setting the JSON-LD Object to the request params
    request.addHeader("content-type", "application/json");
    request.addHeader("Authorization", header_authorization);
    
    HttpResponse response;
    response = httpClient.execute(request);
    
    HttpEntity entity = response.getEntity();
    String responseString = EntityUtils.toString(entity, "UTF-8");
    
    if(response.getStatusLine().getStatusCode() == 200){
      System.out.println(response.toString());
      System.out.println(responseString);
    } else {
      System.out.println(response.getStatusLine().getReasonPhrase());
      System.out.println(response.toString());
      System.out.println(responseString);
    }
  } catch (Exception ex) {
    System.out.print(ex.getMessage().toString());
  }
```

{% endtab %}
{% endtabs %}

***

## Download Dataset

Anyone can search for individual public datasets on ESS-DIVE using the Dataset API. If you are registered to submit data, you can also download your private dataset metadata.

This call will look up one dataset and return complete dataset metadata and file details. Metadata and data files can then be downloaded using standard request packages.&#x20;

If you'd like to look up the dataset upload date, last modified date, or dataset access status, use the API to [**Search for Datasets**](/searching-and-accessing-data/search-with-dataset-api/code-examples#search-for-datasets).

:point\_right: Review **responses** for this call in the [Dataset API technical documentation](https://api.ess-dive.lbl.gov/#/Dataset/getDataset).

{% tabs %}
{% tab title="Python" %}

## Python Download Example

:star: **Make sure you have followed the Setup instructions before running this code**

```python
# ESS-DIVE Identifiers are in the format of: ess-dive-0f0348396e46261-20181022T131245032205
dataset_id = "<Enter an ESS-DIVE Identifier here>"
```

### **Download Metadata**

```python
# Send request
get_package_url = "{}{}/{}?&isPublic=true".format(base,endpoint, dataset_id)

get_package_response = requests.get(get_package_url, 
    headers={"Authorization":header_authorization})

if get_package_response.status_code == 200:
   #Success
   print(get_package_response.json())
else:
   # There was an error
   print(get_package_response.text)
```

### **Download Data Files**

```python
# Use the json message from get_package_response to grab dataset details 
dataset_detail = get_package_response.json()
dist = dataset_detail.get('distribution')
file_url = dist.get('contentUrl')
fn = dist.get('name')

# Define where you want to download the file locally
local_dir = "local_dir"
file_path = local_dir / fn

# Download the dataset locally
urlretrieve(file_url, file_path)

# Get the dataset citation
citation = dataset_detail.get('citation')
```

{% endtab %}

{% tab title="R" %}

## R Download Example

:star:  **Make sure you have followed the Setup instructions before running this code**

```r
# ESS-DIVE Identifiers are in the format of: ess-dive-0f0348396e46261-20181022T131245032205
id <- "<Place an ESS-DIVE identifier here>"
```

### **Download Metadata**

```r
# Send request
call_get_package <- paste(base,endpoint,id, sep="/")
get_package = GET(call_get_package,
    add_headers(Authorization=header_authorization))

# Transform the result into a data frame. (Ignore the warning message)
get_package_text <- content(get_package, "text")
get_package_json <- fromJSON(get_package_text)
```

```r
# Check for errors and view the data frame on success
# Check for errors
if(!http_error(post_package) ){
  print(get_package_json)
}else {
  http_status(post_package)
}
```

### Download Data Files

The Dataset API search result will provide direct URLs that point to the data files. To download data files, you will need to grab the `contentUrl` and download it locally using a standard requests package.

&#x20;and use a standard request or urlretrive package to download it locally. The data file URLs are stored in the JSON as such:

```r
{
"dataset": {
  "distribution": [
      {
          "contentSize": 8.958984375,
          "contentUrl": "https://data.ess-dive.lbl.gov/catalog/d1/mn/v2/object/ess-dive-b8f2258b6f49e86-20210428T053005824019",
          "encodingFormat": "eml://ecoinformatics.org/eml-2.1.1",
          "identifier": "ess-dive-b8f2258b6f49e86-20210428T053005824019",
          "name": "SPRUCE_S1_Bog_Environmental_Monitoring_Data_2010_2016.xml"
        }
      ]
    }
}
```

{% endtab %}

{% tab title="Java" %}

## Java Download Example

:star:  **Make sure you have followed the Setup instructions before running this code**

### **Download Metadata**

```java
  // ESS-DIVE Identifiers are in the format of: ess-dive-0f0348396e46261-20181022T131245032205
  
  try{
    // Replace `String id` with your dataset's ESS-DIVE Identifier
    String id = "<Enter an ESS-DIVE Identifier here>";
    
    String url = base + endpoint + File.separator + id;
    HttpGet request = new HttpGet(url);
    StringEntity params = new StringEntity(JSON_LD.toString()); //Setting the JSON-LD Object to the request params
    request.addHeader("content-type", "application/json");
    request.addHeader("Authorization", header_authorization);

    HttpResponse response;
    response = httpClient.execute(request);

    HttpEntity entity = response.getEntity();
    String responseString = EntityUtils.toString(entity, "UTF-8");

    if(response.getStatusLine().getStatusCode() == 200){
      System.out.println(response.toString());
      System.out.println(responseString);
    } else {
      System.out.println(response.getStatusLine().getReasonPhrase());
      System.out.println(response.toString());
      System.out.println(responseString);
    }
  } catch (Exception ex) {
    System.out.print(ex.getMessage().toString());
  }
```

### Download Data Files

The Dataset API search result will provide direct URLs that point to the data files. To download data files, you will need to grab the `contentUrl` and download it locally using a standard requests package.

&#x20;and use a standard request or urlretrive package to download it locally. The data file URLs are stored in the JSON as such:

```r
{
"dataset": {
  "distribution": [
      {
          "contentSize": 8.958984375,
          "contentUrl": "https://data.ess-dive.lbl.gov/catalog/d1/mn/v2/object/ess-dive-b8f2258b6f49e86-20210428T053005824019",
          "encodingFormat": "eml://ecoinformatics.org/eml-2.1.1",
          "identifier": "ess-dive-b8f2258b6f49e86-20210428T053005824019",
          "name": "SPRUCE_S1_Bog_Environmental_Monitoring_Data_2010_2016.xml"
        }
      ]
    }
}
```

At the end of the document, don’t forget to close the class and main function blocks by adding two curly braces if you hadn’t done that already.
{% endtab %}
{% endtabs %}


# Search with Deep Dive API

An advanced search tool within data files on ESS-DIVE that meet certain standard criteria.

This page contains resources to get started with data search and discovery through the Deep Dive API, including an example search using the interactive documentation at [fusion.ess-dive.lbl.gov](https://fusion.ess-dive.lbl.gov/).&#x20;

{% hint style="success" %}
**See it in action:** Check out our [Jupyter Notebook tutorial ](https://github.com/ess-dive/essdive-tutorials/blob/main/search_data/Using_Data_with_Dataset_DeepDiveAPI_Python.ipynb)with ready-to-run code examples using the Deep Dive API. *(Tip: launch the tutorial directly in your browser with the* [*Google Collab button*](https://github.com/ess-dive/essdive-tutorials/tree/main)*)*&#x20;
{% endhint %}

## Search through files to find data

ESS-DIVE's Deep Dive API provides advanced search capabilities for standardized public data files and is separate from the [main ESS-DIVE search](/searching-and-accessing-data/data-access) and [Dataset API](/searching-and-accessing-data/search-with-dataset-api). Rather than searching across dataset-level metadata, the Deep Dive API allows for search **within data files** to efficiently find data relevant to your scientific research.&#x20;

Public datasets are indexed by the Fusion Database (Fusion DB), which parses datasets following the [File Level Metadata (FLMD)](https://ess-dive.gitbook.io/file-level-metadata-reporting-format) and [CSV Guidelines](https://ess-dive.gitbook.io/csv-file-structure-reporting-format) Reporting Formats. Datasets that use reporting formats are essential to enabling advanced search, and planned enhancements of the Fusion DB will leverage additional reporting formats where possible.

## What public data is available in DeepDive?

Not all data files on ESS-DIVE are available for search with the Deep Dive API. To be surfaced in this search tool, the **dataset metadata** and **data files** must meet the following criteria:

1. Dataset metadata includes ESS-DIVE’s standardized reporting format **keyword** for the File Level Metadata reporting format (ESS-DIVE File Level Metadata Reporting Format)
2. The dataset follows the [File Level Metadata reporting format](https://ess-dive.gitbook.io/file-level-metadata-reporting-format) by including both file level metadata and data dictionary files
3. The file level metadata and data dictionary files follow the respective file naming conventions
4. CSV files correctly follow the [CSV reporting format guidelines](https://ess-dive.gitbook.io/csv-file-structure-reporting-format) to ensure successful file parsing

For additional information on formatting files properly for validation and parsing, please refer to the [Reporting Format Requirements](/publish-data/review-cycle-and-criteria/reporting-format-requirements) page. Documentation and instructions for the use of all data and metadata reporting formats can be found on the [ESS-DIVE Workspace GitHub](https://github.com/ess-dive-workspace).

{% hint style="info" %}
Find all datasets using reporting formats from [ESS-DIVE's Reporting Format data portal](https://data.ess-dive.lbl.gov/portals/reporting-formats).&#x20;
{% endhint %}

## Types of search endpoints

There are two search endpoints currently available through the Deep Dive API:

<table><thead><tr><th width="173">Name</th><th width="292">Endpoint</th><th>Usage</th></tr></thead><tbody><tr><td>Query-Data</td><td><code>deepdive</code></td><td>Searches within files on public datasets for data that matches your search criteria. </td></tr><tr><td>Get-Dataset-File</td><td><code>deepdive/{doi}:{file_path}</code></td><td>Retrieves a dataset file by its file path and summarizes all column/row headers in the file.</td></tr></tbody></table>

## Workflow for Data Discovery

In the next page, we'll walk through the intended use of the available search endpoints in the Deep Dive API for discovering data in ESS-DIVE.&#x20;

{% content-ref url="/pages/8zLToIvaVOECJw5YmF9w" %}
[How to Query Data](/searching-and-accessing-data/search-with-deep-dive-api/how-to-query-data)
{% endcontent-ref %}


# How to Query Data

A workflow for data discovery that walks through the intended use of the Deep Dive API search endpoints.

{% hint style="success" %}
**See it in action:** Check out our [Jupyter Notebook tutorial ](https://github.com/ess-dive/essdive-tutorials/blob/main/search_data/Using_Data_with_Dataset_DeepDiveAPI_Python.ipynb)with ready-to-run code examples using the Deep Dive API. *(Tip: launch the tutorial directly in your browser with the* [*Google Collab button*](https://github.com/ess-dive/essdive-tutorials/tree/main)*)*&#x20;
{% endhint %}

## Deep Dive API Interactive Documentation

The documentation at [fusion.ess-dive.lbl.gov](https://fusion.ess-dive.lbl.gov/) includes technical details, expected schema, error information, and interactive query parameters for the endpoints available through the Deep Dive API. **We recommend starting out with this interactive documentation to help familiarize yourself with query parameters and outputs.**

While programmatic experience can be helpful, it is not required.

<details>

<summary>Reference Material: What's under each endpoint dropdown?</summary>

When you expand an endpoint, each one has **four major sections**. Here we provide a brief explanation. The sections in <mark style="color:green;">green</mark> are most relevant to this demonstration.

* **Header:** This displays the expected format for the base URL (e.g. `/api/v1/deepdive`) and the first sentence in the dropdown describes what the endpoint does.\
  &#x20;
* <mark style="color:green;">**Parameters:**</mark> This section provides a list of the available search parameters that can be input into the endpoint. \
  If a parameter must be filled out, it will say "<mark style="color:red;">\* required</mark>". Each parameter has a **definition** that explains what the parameter does. For your convenience, the screenshots on this page can be used to reference all available parameters and their definitions. <br>
* <mark style="color:green;">**Server Responses (interactive):**</mark> By default, this section is not visible until you select "try it out" and execute an example search (step 1-2). \
  The interactive response section lists a copy of the search request you made (e.g., the Curl command and Request URL) and the **server response** (i.e., the search result). **Definitions of the results in the server response can be found in the Schemas** section of the interactive documentation (visible in Fig 1; see step 4 for a demonstration). <br>
* **Responses (general)**: By default, this section is always visible in the Responses.\
  This lists the *possible* response codes and response messages that could be returned. This is useful reference material for debugging searches when writing your own code. For this demonstration, we will only be looking at successful responses (status code 200).&#x20;

![](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2Fonb8SvnksKGOZAYVKE6C%2FEndpoint-Structure-Description-Parameters.png?alt=media\&token=d00fb3f7-7de5-4ee5-825a-b3c5d62b09e0)

*Header and Parameters*

![](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2F1X0uB2wsppuqX54DanRl%2FEndpoint-Structure-Responses-Interactive.png?alt=media\&token=e1c21f6a-372d-4f35-b0ad-b9805292b5d6)![](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2F3KjPVYIjH7CTZ9TXM5ki%2FEndpoint-Structure-Responses-General.png?alt=media\&token=625e51f8-0623-4320-a464-4feec909fc8c)

*Interactive Server Responses                                     General Details about Responses*

</details>

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FjirdSngl91rY5nHbpGwD%2FDeep-Dive-Docs.png?alt=media&amp;token=21be6c7d-0ffc-486e-bc8d-a78d68d3126c" alt=""><figcaption><p>Figure 1: Documentation at fusion.ess-dive.lbl.gov</p></figcaption></figure>

## 1. Expand Query-Data endpoint &#x20;

Let’s first expand the Query-Data endpoint details.

This section shows the available search parameters, parameter descriptions, expected format, and editable value boxes. Select the “Try it out” button highlighted in orange to edit the values. From here, we can easily enter parameters to test out search queries.&#x20;

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FvKSEVNQfCJxzcsc6yOlY%2FDeep-Dive-Expand-Endpoint.png?alt=media&amp;token=2206f879-162c-43e5-a645-2bdbe669fdf1" alt=""><figcaption><p>Figure 2: Select the Query-Data endpoint to expand interactive query parameters</p></figcaption></figure>

{% hint style="info" %}
`fieldName` refers to the column or row name (aka field name, or variable name) in the data file, which come from the dataset's [Data Dictionary](https://ess-dive.gitbook.io/file-level-metadata-reporting-format/csv_dd). This is a component of the File Level Metadata Reporting Format. Search terms entered in the `fieldName` parameter do not have to be an exact match.&#x20;

The `fieldDefinition` parameter comes  provides definitions of each field name included in the data files.&#x20;
{% endhint %}

## 2. Enter example Query-Data search

Let’s execute the following search query (Table 1) to look for data within this public dataset that is using reporting formats and is available in the Deep Dive API.&#x20;

> Roley S ; Hall Jr. R O H J; Garayburu-Caruso V A ; Perkins W A ; Stegen J C (2023): Data and scripts associated with "Coupled primary production and respiration in a large river contrasts with smaller rivers and streams.". River Corridor and Watershed Biogeochemistry SFA, ESS-DIVE repository. Dataset. doi:10.15485/1985922 accessed via [https://data.ess-dive.lbl.gov/datasets/doi:10.15485/1985922 on 2024-06-24](https://data.ess-dive.lbl.gov/datasets/doi:10.15485/1985922)

Copy the values in Table 1 and enter the query parameters as follows:

<table><thead><tr><th width="192">Parameter Name</th><th width="127">Value</th><th>Logic</th></tr></thead><tbody><tr><td><code>doi</code></td><td>doi:10.15485/1985922</td><td>This is the DOI of the specific dataset that may have relevant data. It must be entered in the format of <code>doi:{DOI number}</code>.</td></tr><tr><td><code>fieldName</code></td><td>temp</td><td>I want to look for temperature data, but I’m not sure what the exact field name in the file is.</td></tr><tr><td><code>recordCountMin</code></td><td>20</td><td>I need at least 20 data points for the data to be useful for my research.</td></tr></tbody></table>

*Table 1: Example Query-Data parameters*

Our example parameters will look like this once completed (Fig 3). Select the "Execute" button to query the Deep Dive API.

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2Fv5VSTK1Pw92QDCBwfZ04%2FDeep-Dive-Query-Data-Search.png?alt=media&amp;token=801a7170-92eb-4701-afe2-7e2dedea50b3" alt=""><figcaption><p>Fig 3: Search parameters with the example query</p></figcaption></figure>

{% hint style="info" %}

#### :bulb: Try out different searches on other Reporting Format datasets

Find all datasets using reporting formats in [ESS-DIVE's Reporting Format data portal](https://data.ess-dive.lbl.gov/portals/reporting-formats).
{% endhint %}

## 2. Interpret the Query-Data response

The responses section (Fig 4) will show you the search results, which is referred to here as the Server Response (see [reference material](#reference-material-whats-under-each-endpoint-dropdown)).&#x20;

In the response body, the `pageCount` value tells us that 4 fields (e.g. column/row headers) were found by this query. The results were found in at least one file.

`next` and `previous` show us which page of the results are currently displayed in the response body. A page is simply defined by the `pageSize` limit. In our example, one page will show up to 25 results. Given that we received 4 search results, the current response contains all results and we do not have to page through the results.&#x20;

Let’s take a look at the `results` starting with first field name: `temp_C`.

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FbhxC7Q4wZBw4q0dO6rkm%2Fdeepdive-query-data-response.png?alt=media&amp;token=d9749868-bfef-4ebf-a5fe-c0bdd6756bce" alt=""><figcaption><p>Figure 4: Successful response from the example query</p></figcaption></figure>

{% hint style="info" %}
`fieldName` refers to the column or row name (aka field name, or variable name) in the data file, which come from the dataset's [Data Dictionary](https://ess-dive.gitbook.io/file-level-metadata-reporting-format/csv_dd).
{% endhint %}

Detailed information about this column/row is provided, including the `unit`, `definition`, and `data_type`. There are almost 80,000 data points available under this one column/row (`total_record_count`) and the data values are between a minimum of 1.02℃ and 23.28℃ (`values_summary`). Additionally, there are no missing values in this column/row (`missing_values_count`).&#x20;

:bulb: This information can help you determine whether this data is relevant for your research.

:bulb: Then the remaining four values in the response can be used to locate the exact file and dataset version where this column/row `temp_C` can be found for download.

The `version` number (also called an ESS-DIVE Identifier) is a unique ID given to all datasets on ESS-DIVE. This number points to the exact version of the dataset associated with this data at the time the file was published, so you can always find the data should changes be made. The provided `doi` number is a persistent identifier to the *latest* version of the dataset. Additionally, we can see the CSV file name (`data_file`) and `file_path` where the column/row can be found.&#x20;

**To find this file in your browser**, you can enter the DOI in the search bar (doi.org/10.15485/1985922) or you can look up the version number in ESS-DIVE using the standard URL format: `data.ess-dive.lbl.gov/view/<ESS-DIVE Identifier>`.

**To find this file programmatically**, we can use the `data_file_url`. <mark style="color:green;">**Example code is available in our**</mark> [<mark style="color:green;">**Jupyter Notebook Tutorial**</mark>](https://github.com/ess-dive/essdive-tutorials/blob/main/search_data/Using_Data_with_Dataset_DeepDiveAPI_Python.ipynb)<mark style="color:green;">**.**</mark>

## 3. Enter example Get-Dataset-File request

Here we demonstrate how to use the Deep Dive API to locate relevant files that appeared in our search results.&#x20;

As we did in step 1, first expand the Get-Dataset-File endpoint details and select the "Try it out" button to edit the parameter values (Fig 5).

Let’s execute the following search query (Table 2) using the DOI and file path associated with the `temp_C` field from our Query-Data results. Enter the parameters as follows:

<table><thead><tr><th width="180">Parameter Name</th><th width="261">Value</th><th>Logic</th></tr></thead><tbody><tr><td><code>doi</code></td><td>doi:10.15485/1985922</td><td>This is the DOI of the file that I am interested in looking up. It must be entered in the format of <code>doi:{DOI number}</code>.</td></tr><tr><td><code>file_path</code></td><td>Roley_CR_Metabolism_Data_Package.zip/DO_temp_sensor_data.csv</td><td>I found this file in the results of my Query-Data search. I want to see a summary of all the data in this file (by header/row name), find the size of the file, and/or get the URL to directly download this file.</td></tr></tbody></table>

*Table 2: Example Get-Dataset-File parameters*

Our example parameters will look like this once completed (Fig 5). Select the "Execute" button to query the Deep Dive API.

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FLidsQxwMVIDKKN3IoTsY%2Fdeepdive-get-data-file-search.png?alt=media&amp;token=3c9e8cf0-33f1-4f2c-82f6-32f9d9ac4d6e" alt=""><figcaption><p>Figure 5: Search parameters with the example query</p></figcaption></figure>

## 4. Interpret the Get-Dataset-File response

The Get-Dataset-File responses section (Fig 4) will show you the search results, which is referred to here as the Server Response (see [reference material](#reference-material-whats-under-each-endpoint-dropdown)).&#x20;

You'll notice that the Get-Dataset-File response contains all of the information returned by the Query-Data endpoint, with the addition of a few new values (Fig 6). It contains the dataset DOI, file name and the ESS-DIVE identifier that corresponds to the latest version of the dataset.

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FLl5nL1MGYPCjIxWTQcE7%2Fdeepdive-get-data-file-response.png?alt=media&amp;token=420e0026-76f2-4036-a044-399c0ad41860" alt=""><figcaption><p>Figure 6: Successful response from the data file lookup</p></figcaption></figure>

Then, under `fields`, you'll find a list of *all the column/row headers in the specified file*. It also lists the same detailed information about the data within the column/row header that Query-Data provides.&#x20;

:bulb: The power of the Get-Dataset-File endpoint is that it allows you to find all column/row information for an entire file at once. This provides more context about the data, allowing you to determine whether the file could be relevant to your research.&#x20;

Another key difference of this endpoint is `data_download` return which provides the size of the file in bytes (`contentSize`), the file format (`encoding_format`), the file version (`identifier`), and a URL that points directly to the file (`contentURL`). &#x20;

:bulb:By providing file specifications, you can make an informed decision on whether the file size and format are within your expected constraints for data use.&#x20;

<details>

<summary>Where to find definitions for Get-Dataset-File responses</summary>

The [server response](#reference-material-whats-inside-the-endpoint-documentation) for this query follows the `DatasetFile` schema. All definitions for the fields in this response are listed in the "Schema" > "DatasetFile" section of the [Deep Dive API documentation](https://fusion.ess-dive.lbl.gov/).&#x20;

Click the response value you are interested in to expand it's definition and expected formatting.&#x20;

![](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FOpRBtwias2IN8enaGC11%2Fdeep-dive-api-schemas-dataset-fields.png?alt=media\&token=d0630a34-642a-4bd3-8d9b-61f05a7b9387)

</details>


# ESS-DIVE Dataset API

The ESS-DIVE Dataset API is a service that enables users to programmatically submit and manage datasets with ESS-DIVE. This guide provides an overview of the API, resources, and how to get started.

## Quickstart

First time using APIs? Click on an action below to see complete tutorials that walkthrough each Dataset API operation! No coding experience is required to try out a tutorial.

<table data-view="cards" data-full-width="false"><thead><tr><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td>Search for Datasets</td><td><a href="/searching-and-accessing-data/search-with-dataset-api">Search with Dataset API</a></td></tr><tr><td>Submit Datasets</td><td><a href="/contributing-data/submit-data-with-the-dataset-api">Submit Data with the Dataset API</a></td></tr><tr><td>Project Dataset Metrics</td><td><a href="https://github.com/ess-dive/essdive-tutorials/blob/main/manage_data/Project%20Dataset%20Metric%20Report%20-%20Python.ipynb">https://github.com/ess-dive/essdive-tutorials/blob/main/manage_data/Project%20Dataset%20Metric%20Report%20-%20Python.ipynb</a></td></tr><tr><td>Share Datasets</td><td><a href="https://github.com/ess-dive/essdive-tutorials/blob/main/manage_data/ShareDatasets_Python.ipynb">https://github.com/ess-dive/essdive-tutorials/blob/main/manage_data/ShareDatasets_Python.ipynb</a></td></tr></tbody></table>

## Getting started with ESS-DIVE Dataset API                                  (Package Service API version 1)

ESS-DIVE's Dataset API is a REST API with a variety of operations available that allow you to programmatically manage datasets using HTTP requests. This is an alternative to using the [ESS-DIVE Online](https://data.ess-dive.lbl.gov/) form for managing data.&#x20;

This service encodes metadata using the [JSON-LD](https://json-ld.org/) specification. JSON-LD is a schema to encode linked Data using JSON, and is [used by Google](https://developers.google.com/search/docs/data-types/dataset) to index metadata for searches. The use of the standardized JSON-LD schema dramatically increases the visibility of datasets, and also enable projects to create one-time code that can be reused for periodic uploads of datasets to ESS-DIVE.

ESS-DIVE has a variety of resources available to help you get started with the Dataset API. We have example code and usage documentation available for new and experienced coders. **In addition to the linked resources below, you'll also find user-friendly introductions to specific API operations within this documentation!** Find these under the relevant Submit, Manage, and Search sections. &#x20;

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Dataset API Tutorials</strong></td><td>Browse through all ESS-DIVE's Jupyter Notebook Tutorials.</td><td><a href="https://github.com/ess-dive/essdive-tutorials">https://github.com/ess-dive/essdive-tutorials</a></td></tr><tr><td><strong>Technical Example Scripts</strong></td><td>Interested in building scripts to automate your publication workflow? Our GitHub repo has all example code available in R, Python and Java.</td><td><a href="https://github.com/ess-dive/essdive-package-service-examples">https://github.com/ess-dive/essdive-package-service-examples</a></td></tr><tr><td><strong>Technical API Documentation</strong></td><td>Experienced coders can come here to see all available HTTP operations and find details on expected schema for building custom scripts.</td><td><a href="https://api.ess-dive.lbl.gov/#/">https://api.ess-dive.lbl.gov/#/</a></td></tr></tbody></table>

Provide feedback on this service to <ess-dive-support@lbl.gov>.

## Get Authentication Token

1. Visit ESS-DIVE
   1. **Search & Download:** [https://data.ess-dive.lbl.gov](https://api.ess-dive.lbl.gov/#/)
   2. **Submit & Manage Datasets:** [https://data-sandbox.ess-dive.lbl.gov](https://data-sandbox.ess-dive.lbl.gov/data)
2. Sign in with Orcid
3. Click your Name in the right hand corner and select *My Profile* (Figure 1)
4. Now Click the *Settings>Authentication Token* (Figure 2)
5. Scroll down and click *Copy* on the “Token” tab to get your authentication token (Figure 2)

![Figure 1. Hover over your name and click on My Profile ](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-Mj1_ufatnr_-vuR-edq%2F-Mj1ayVTCZCUrzWOa3jB%2Fmy-profile.png?alt=media\&token=30acc227-7530-425a-88f5-e80637d1f89c)

![Figure 2. Click on Authentication Token under Settings and copy the token](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-LkaM78iKT0zasc8fJdJ%2F-LkaMEbV9vrGoK_Qkjdc%2Fimage.png?alt=media\&token=7df47405-0ca2-4c7d-9e3b-a33cf681c7d8)


# Setup and Troubleshoot

All Dataset API code examples require installation of certain packages and authentication from your ESS-DIVE account. This page provides instructions for setting up the Dataset API for your preferred coding language.

For additional information about the API, review the technical documentation at [https://api-sandbox.ess-dive.lbl.gov](https://api-sandbox.ess-dive.lbl.gov/).

**ESS-DIVE Test API URL (Sandbox):** [**https://api-sandbox.ess-dive.lbl.gov**](https://api-sandbox.ess-dive.lbl.gov)\
**ESS-DIVE Production API URL:** [**https://api.ess-dive.lbl.gov/**](https://api.ess-dive.lbl.gov/)\
**Help Desk:** [**ess-dive-support@lbl.gov**](mailto:ess-dive-support@lbl.gov)

## Setup Code

**Step 1:** [**Get Authentication Token**](/programmatic-tools/ess-dive-dataset-api#get-authentication-token): Copy your authentication token from ESS-DIVE, then paste it into the code where specified in the example.

**Step 2:** Setup your code in your preferred coding language as follows.

{% tabs %}
{% tab title="Python" %}

## Setup in Python

Install the following python module into your python environment

`$ pip install requests`

In a python console or script add the following lines to setup your script.

```python
import requests
import os
import json

token = "<Enter your authorization token here>"
base = "https://api-sandbox.ess-dive.lbl.gov/"
header_authorization =  "bearer {}".format(token)
endpoint = "packages"
```

{% hint style="warning" %}
Check that your token is up-to-date; it expires after 24 hours
{% endhint %}
{% endtab %}

{% tab title="R" %}

## Setup in R

Install the following R packages

```r
install.packages("httr")
install.packages("jsonlite")
install.packages("readr")

#Require the package so you can use it
require("jsonlite")
require("httr")
library(readr)
```

Setup the Dataset API information

```r
token <- "<Enter your authentication token here>"
base <- "https://api-sandbox.ess-dive.lbl.gov"
header_authorization <- paste("bearer",token, sep=" ")
endpoint <- "packages"
```

{% hint style="warning" %}
Check that your token is up-to-date; it expires after a 24 hours
{% endhint %}
{% endtab %}

{% tab title="Java" %}

## Setup in Java

To be able to do the setup correctly, you need a few prerequisites installed on your machine which are unzip, java, javac and curl.

Inside your project directory create a java file named `essdive.java` and a directory `lib`. The following are the *Linux commands* to create the file and directory:

```java
touch essdive.java
mkdir lib
cd lib
```

Add http client, http core and commons logging from [Apache HTTPComponents libraries version 4.5.6](https://hc.apache.org/downloads.cgi) and json\_simple-1.1 into your lib directory by downloading its zip file, extracting it and moving it inside `lib`.

```java
curl -O http://ftp.wayne.edu/apache//httpcomponents/httpclient/binary/httpcomponents-client-4.5.6-bin.zip

curl -O https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/json-simple/json_simple-1.1.jar

curl -O http://mirror.cc.columbia.edu/pub/software/apache//commons/io/binaries/commons-io-2.6-bin.tar.gz

tar -xvzf commons-io-2.6-bin.tar.gz
mv commons-io-2.6/commons-io-2.6.jar .


unzip httpcomponents-client-4.5.6-bin.zip 
mv httpcomponents-client-4.5.6/lib/* . 
```

Open `essdive.java` document created earlier using any text editor and add the following code to start importing the libraries needed for your java code:

```java
import java.io.File;
import java.io.IOException;

//JSON imports
import org.apache.http.HttpEntity;
import org.json.simple.JSONArray;
import org.json.simple.JSONObject;

//Apache http imports
import org.apache.commons.io.FileUtils;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.methods.HttpPut;
import org.apache.http.HttpResponse;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.entity.StringEntity;
import org.apache.http.entity.ContentType;
import org.apache.http.entity.mime.FormBodyPart;
import org.apache.http.entity.mime.FormBodyPartBuilder;
import org.apache.http.entity.mime.HttpMultipartMode;
import org.apache.http.util.EntityUtils;
import org.apache.http.entity.mime.MultipartEntityBuilder;
import org.apache.http.entity.mime.content.StringBody;
import org.apache.http.client.methods.HttpGet;
```

#### **Definition of configuration variables**

Inside the main function you should define your configuration variables:

{% hint style="warning" %}
**Note:** You will have to close the class and main blocks by adding two closing curly braces at the end of the document.
{% endhint %}

```java
public class essdive {

public static void main(String[] args) {
 //Configuration variables
  String token = "<Enter your authorization token here>";
  String base = "https://api-sandbox.ess-dive.lbl.gov/";
  String header_authorization = "bearer " + token;
  String endpoint = "packages";
```

Then define your utilities objects:

```java
         //Utilities objects 
        CloseableHttpClient httpClient = HttpClientBuilder.create().build();
```

{% hint style="warning" %}
Check that your token is up-to-date; it expires after a 24 hours
{% endhint %}
{% endtab %}
{% endtabs %}

## Troubleshoot

{% tabs %}
{% tab title="Python" %}

## Python Troubleshooting Tips

```python
{"detail":"You do not have authorized access"}
```

This error message indicates your token is either incorrect or expired. Please follow the instructions on the [ESS-DIVE Dataset API](/programmatic-tools/ess-dive-dataset-api) page to retrieve a new token.

```python
{"detail":"One or more fields raised validation errors.","errors":["provider/member 'familyName' is a required property"]}
```

This error message indicates a required field is missing from your JSON. In this case it is the "familyName". Revise your JSON to include the mandatory fields.

```python
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-28-7ae5b841a5b0> in <module>
      5 
      6 files_tuples_array.append((("json-ld", json.dumps(json_ld))))
----> 7 files_tuples_array.append(("data", open(file_directory ,'rb')))
      8 
      9 post_packages_url = "{}{}".format(base,endpoint)

FileNotFoundError: [Errno 2] No such file or directory: 'trials.csv'
```

This error message indicates the file entered was not found. This could be because you are searching in the wrong directory or because you misrepresented the file name.&#x20;
{% endtab %}

{% tab title="R" %}

## R Troubleshooting Tips

```r
 results = content(post_package)
>  attributes(results)$names
[1] "detail"
>  results$detail
[1] "You do not have authorized access"
>  results$errors
NULL
>  results$viewUrl
NULL
```

This error message indicates your token is either incorrect or expired. Please follow the instructions on the [ESS-DIVE Dataset API](/programmatic-tools/ess-dive-dataset-api) page to retrieve a new token.

```r
Error: 'C:\Users\Person\Desktop\API\API_Tutorials.json' does not exist.
```

This error message indicates the file entered was not found. This could be because you are searching in the wrong directory or because you misrepresented the file name.&#x20;
{% endtab %}

{% tab title="Java" %}
*Currently no troubleshooting tips are available in Java.*
{% endtab %}
{% endtabs %}


# API Updates and Changes

This page provides information about all versions of the Dataset API. It includes a log of new features, enhancements, and bug fixes that have been made to the API.

## 1.9.0

*Released on 2023-11-28*

#### New Features 🎉

* Adds `@id` (i.e. DOI) to search results
* Enables dataset search by keyword with new `keyword` parameter
* Enables search for older versions of dataset metadata by identifier (i.e. `ess-dive-9727d2718c0d1d3-20230504T212419449020`)&#x20;
* Adds the identifier for the next version of metadata (i.e. `next`) and previous version (i.e. `previous`) to search results

#### **Enhancements** :sparkles:

* Enhances keyword search to lookup exact matches

## 1.8.0

*Released on 2022-11-28*

#### New Features 🎉

* Adds file size in KB to distribution items

#### **Enhancements** :sparkles:

* Updates technical Swagger documentation hosted by OpenAPI: <https://api.ess-dive.lbl.gov/#/>

## 1.7.0

*Released on 2022-08-08*

#### New Features 🎉

* Enable dataset submissions with project ID in `provider` field (new syntax example provided below)

```json
"provider": {
   "identifier": {
      "@type": "PropertyValue",
      "propertyID": "ess-dive",
      "value": "<PROJECT ID>"
   }
}
```

## 1.6.0

*Released on 2022-02-04*

#### New Features 🎉

* Enable linking out to datasets stored at external repositories
* Enable capability for anyone to search for public datasets

#### New Code Examples ✨

* Provided a python code example for streaming data file uploads
  * `update_stream_package.py` now available via [essdive-package-service-examples](https://github.com/ess-dive/essdive-package-service-examples/blob/master/code/python/update_stream_package.py) GitHub repository.

## 1.5.0

*Released on 2021-09-20*

#### New Features 🎉

Enables dataset sharing

* Enable the ability to share dataset&#x20;
* Expanded swagger Docs for dataset sharing

#### Bug Fix 🛠️

* Corrected dataset sharing typos and added sharing schema example to technical documentation

## **1.4.0**&#x20;

*Released on July 23rd, 2021*

#### **Enhancements** :sparkles:

You can now submit longer metadata descriptions

* Increase `measurementTechnique` description (i.e. Methods) character limit to <5000 characters
* Increase `description` (i.e. Abstract) character limit to <5000 characters&#x20;
* Increase `spatialCoverage` description (i.e. Geographic Description) character limit to <5000 characters

**Software Upgrades ⚙️**

* Enable support for Metacat >= 2.13.0 and Solr8

## 1.3.4

*Released on 2021-04-29*

#### Software Upgrades ⚙️&#x20;

* Main focus of this release is to support the EML 2.2.0 upgrade.

## 1.3.3

*Released on 2021-04-20*

#### Bug Fix 🛠️&#x20;

* Bug fix release for json-ld 500 Errors.

## 1.3.2

*Released on 2021-05-27*

This release includes the addition of a Berkeley Source Distribution (BSD) license to the package service.

#### Bug Fix 🛠️&#x20;

* Some bug fixes are included for accessing packages.

## 1.3.1

*Released on 2019-12-17*&#x20;

#### **Enhancements** :sparkles:

* This release enables up to 100GB data uploads. They are minor features and do not affect the API.

## 1.3.0

*Released on 2019-10-21*&#x20;

#### **Enhancements** :sparkles:

* This release enables up to 10GB data uploads.

## 1.2.0

*Released on 2019-06-26*&#x20;

#### New Features 🎉&#x20;

* Ability edit a data package&#x20;
* Data files in query results&#x20;
* Special characters should be removed from metadata filename&#x20;
* JSON-LD returned for single data package requests

This API update is backwards compatible with v1.1.0.


# ESS-DIVE AI Assistant

Connect ESS-DIVE datasets to AI-powered tools to retrieve, search, and interpret datasets.

### What is the AI Assistant?

The AI Assistant is a research partner that can help you search, download, and interpret ESS-DIVE data with the help of any AI model of your choosing. The assistant will help AI models properly fetch metadata, generate citations, look up reporting format data files on ESS-DIVE, [and more](#what-tasks-can-the-ai-assistant-help-you-with) through systems like Codex, Claude Code and VS Code.&#x20;

Figure 1 below demonstrates what it would be like to have the assistant find ESS-DIVE datasets by data type and summarize the search results.

This tool also includes [Skills](#skill) (i.e. reusable instructions documents) that can be installed independent of our assistant to help AI models perform specific tasks related to ESS-DIVE, such as generating dataset citations correctly and exploring published reporting format files and fields.&#x20;

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FZyC8fZvH28ggY2nPaihs%2Fimage.png?alt=media&amp;token=eda19805-aac2-4cc5-9d26-c6eb1bb0bd53" alt=""><figcaption><p>Figure 1. Example prompt in the AI-Assistant interface</p></figcaption></figure>

The program needs to be installed and connected to an AI Client, such as Goose, with an AI model, such as Gemini. The program allows you the flexibility to choose your preferred AI model to answer questions and perform tasks within your preferred AI client interface.&#x20;

:point\_right: To find the best starting point, check out our [Setup](#setup) section below. Otherwise, head over to [ESS-DIVE's GitHub Repository](https://github.com/ess-dive/essdive-mcp) to jump right in.

{% hint style="info" icon="timer" %}
The most basic set up through Goose takes around 20 minutes.&#x20;
{% endhint %}

{% hint style="success" icon="list-check" %}
**What will you need?**

:key: Access to an AI model API key (e.g. Gemini, GPT)

:computer: Client desktop app (e.g. Goose, VSCode)

:robot: Install AI Assistant&#x20;
{% endhint %}

<details>

<summary>AI Model vs. AI Client Terminology</summary>

**AI Model:** <mark style="color:$info;">(The Brain</mark> :brain:<mark style="color:$info;">)</mark> The AI that determines and executes goals and steps to take based on prompts.

**AI Client:** <mark style="color:$info;">(The interface to talk to the brain</mark> :computer:<mark style="color:$info;">)</mark> The interface where you interact with an AI model. E.g. Goose, Codex, Claude Code.

</details>

#### Why use ESS-DIVE's AI Assistant?

The type of tasks needed for interacting with data on ESS-DIVE are not natively supported in large language model (LLM) chat interfaces and would require a lot of assistance to complete a task. Once set up, the assistant can:

* Access data in consistent, predictable ways without guesswork
* Provide context from a variety of sources
* Work with variations
* Interpret results

#### What tasks can the AI Assistant help you with?

* Searching public ESS-DIVE datasets
* Fetching dataset metadata, version history, and sharing permissions
* Converting between ESS-DIVE dataset IDs and DOIs
* Generating consistent ESS-DIVE data citations with MCP/API access details, plus warning-backed Crossref fallback for non-ESS-DIVE DOIs
* Parsing File Level Metadata (FLMD) CSV content
* Searching with [Deep Dive API](/searching-and-accessing-data/search-with-deep-dive-api) to find information within [Reporting Format](/publish-data/review-cycle-and-criteria/reporting-format-requirements) files
* Looking up ESS-DIVE project acronyms, descriptions, and portal URLs
* Turning coordinates into map links

{% hint style="warning" %}
**Always make sure to cite your data!**
\
Datasets are valuable research contributions that should be cited to ensure that authors receive credit for producing and curating data. Always retain citations when using data from a published ESS-DIVE dataset.

The AI Assistant can help provide you with citations.
\\

\
Example prompt you can ask the AI: `Could you please dump the citations for these datasets to a
 TSV? Use ESS-DIVE format.`
{% endhint %}

### How does the AI Assistant work?

The AI Assistant is a combination between an **MCP** (Model Context Protocol), an **AI model** (e.g. Chat GPT, Gemini) and **AI client** of your choice (e.g. Claude, Codex, Goose). An **MCP-Server** is a software that provides instructions, via **tools** and **skills**, to AI models on how to accomplish tasks.&#x20;

The ESS-DIVE AI Assistant is an MCP-Server written by ESS-DIVE developers to instruct AI on the proper use of ESS-DIVE's APIs.
&#x20;When you install the AI Assistant, you are installing an MCP-Server.

The AI Assistant is not a web interface that you can interact with in your browser.

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FUXG35zagwNItzBUZ7fm3%2Fimage.png?alt=media&amp;token=c8f64a69-557f-48f3-b9a1-51a8f96b8fca" alt=""><figcaption><p>Figure 2: MCPs allow models to interact with APIs, files, software tools, databases, and virtually any other data source or interface that traditional software can work with.</p></figcaption></figure>

{% columns %}
{% column %}

#### Tool

The MCP Server operates by having a collection of tools, or functions written in python, which tell the AI model how to use ESS-DIVE's APIs and execute specific tasks. The tools are what enable the actions listed under "[What tasks can the AI Assistant help you with?](#what-tasks-can-the-ai-assistant-help-you-with)".&#x20;

AI models will know when to use ESS-DIVE's tools based on the context within your prompt. It won't be necessary to specify which tool you want it to use.

<p align="center"><a href="https://github.com/ess-dive/essdive-mcp/tree/main#available-tools" class="button secondary">GitHub Documentation</a></p>
{% endcolumn %}

{% column %}

#### Skill

Skills provide instructions to models for accomplishing common tasks. They are markdown[^1] files that follow an open, natural language standard understood by most AI clients.&#x20;

Skills can be installed as reusable instruction documents outside of ESS-DIVE's MCP Server.&#x20;

Additionally, you can choose to use the MCP Server with or without Skills. We recommend starting off with the AI Assistant without Skills, then add them later if you want more predictable model behavior for repeated tasks.

The MCP server provides the tools. A Skill helps the model decide how and when to use them.

<p align="center"><a href="https://github.com/ess-dive/essdive-mcp/tree/main#agent-skills" class="button secondary">GitHub Documentation</a></p>
{% endcolumn %}
{% endcolumns %}

## Setup

The [GitHub Documentation](https://github.com/ess-dive/essdive-mcp/tree/main) provides multiple options and detailed instructions to set up your AI Assistant + Skills based on your preferences. Select which part of the setup documentation you would like to jump to:

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th><th data-hidden data-card-cover data-type="image">Cover image</th></tr></thead><tbody><tr><td><h4>Setup the MCP-Server in Goose</h4></td><td>The best option for those just starting out with AI Models and Clients.</td><td><a href="https://github.com/ess-dive/essdive-mcp/blob/main/docs/GOOSE_SETUP.md">https://github.com/ess-dive/essdive-mcp/blob/main/docs/GOOSE_SETUP.md</a></td><td></td></tr><tr><td><h4>Connect the MCP-Server to a specific Client</h4></td><td>Select this setup option if you prefer to use a Client other than Goose. Common Clients include: VS Code with GitHub Copilot Chat, Claude Code, and Codex.</td><td><a href="https://github.com/ess-dive/essdive-mcp/blob/main/README.md#install-and-connect-to-a-client">https://github.com/ess-dive/essdive-mcp/blob/main/README.md#install-and-connect-to-a-client</a></td><td></td></tr><tr><td><h4>Install Skills</h4></td><td>See all Skills available for download and what they can do. <br><br>This is optional.</td><td><a href="https://github.com/ess-dive/essdive-mcp/blob/main/docs/SKILLS.md">https://github.com/ess-dive/essdive-mcp/blob/main/docs/SKILLS.md</a></td><td></td></tr><tr><td><h4>Example Queries</h4></td><td>Test these prompts to check that the AI Assistant is working.</td><td><a href="https://github.com/ess-dive/essdive-mcp/blob/main/README.md#first-queries-to-try">https://github.com/ess-dive/essdive-mcp/blob/main/README.md#first-queries-to-try</a></td><td></td></tr><tr><td><h4>Troubleshooting</h4></td><td>Find answers to common setup questions.</td><td><a href="https://github.com/ess-dive/essdive-mcp/blob/main/README.md#troubleshooting">https://github.com/ess-dive/essdive-mcp/blob/main/README.md#troubleshooting</a></td><td></td></tr><tr><td><h4>Tool-Level Examples</h4></td><td>If your client supports direct tool calling, these examples demonstrate how to reference our tools in prompts. </td><td><a href="https://github.com/ess-dive/essdive-mcp/blob/main/README.md#tool-level-examples">https://github.com/ess-dive/essdive-mcp/blob/main/README.md#tool-level-examples</a></td><td></td></tr></tbody></table>

[^1]: Markdown is a file containing plain text and natural language with very simple formatting.


# Globus Data Transfer Service

Globus is an offline service used by ESS-DIVE to upload large data files to the ESS-DIVE repository.

ESS-DIVE can upload a wide variety of data files to dataset publications via our [web interface](https://data.ess-dive.lbl.gov/data) or [Dataset API](/programmatic-tools/ess-dive-dataset-api). For large data files or for anyone experiencing issues uploading small data files, the Globus data transfer service can be used as an alternative upload method.

The Globus transfer service is setup offline with close assistance from the ESS-DIVE Team. Data contributors should get in touch with ESS-DIVE Support at <ess-dive-support@lbl.gov> to begin the process of publishing data with Globus.

{% hint style="info" %}
Globus should be used only when data cannot be uploaded with the web interface or the Dataset API. Contact the ESS-DIVE Support Team to learn if your data is suitable for Globus upload.
{% endhint %}

## What is Globus

[Globus](https://docs.nersc.gov/services/globus/) is a free, cloud-based data transfer service designed to move significant amounts of data. ESS-DIVE uses this service to move data from your local desktop or existing Globus endpoint to ESS-DIVE's storage services.

## Why use Globus

ESS-DIVE started using Globus to support data uploads to address two common use cases that our data contributors encounter: uploading data files greater than 500GB and resolving upload errors for smaller files. Both these use cases are not readily supported by ESS-DIVE's other upload methods.

#### Upload data greater than 500GB

Large data volumes that are greater than the maximum file upload limits of the web submission form (<10GB) and the Dataset API (<500GB) can be uploaded to ESS-DIVE using Globus. Certain data types can commonly create large volumes of data and require Globus support to archive, such as sensor data, model data and remote sensing data. Learn more about the file upload limits of ESS-DIVE's submission tools on the Get Started page:

{% content-ref url="/pages/-LuExpA\_2N2EXi53RjSk" %}
[Get Started](/contributing-data/get-started)
{% endcontent-ref %}

#### Resolve upload errors for small files

Additionally, Globus can be used to resolve common upload errors with relatively small data volumes (10-500GB). If your data is less than 500GB and you've encountered any of the following issues, you may be interested in using Globus:

* **Unstable internet connection** causes upload to timeout, error, or otherwise fail
* You have many files that are too large to be uploaded all at once but it’s too tedious to upload them in tens of batches
* Data is too large to download from storage location onto data manager’s local system for upload via web interface
* Available local memory for upload is smaller than data volume stored locally
* Lack of familiarity scripting with application programming interfaces (API)
* Your cloud storage service provider’s API and tools aren’t made accessible to help you move data

Of the above reasons, an unstable internet connection is the most likely cause for large data upload failures.&#x20;

## How to use Globus

In order to upload data using Globus, you will need to first:

1. Create a dataset on ESS-DIVE and provide required dataset metadata&#x20;
2. Create a Globus account using your ORCID or institutional login (if applicable)

Detailed information on these steps are provided in ESS-DIVE's Globus instructions. Head to our documentation page on setting up Globus to get started:

{% content-ref url="/pages/-MX-liL4i-S8I9i6XJmB" %}
[Setup Globus](/programmatic-tools/large-data-support/how-to-setup-globus)
{% endcontent-ref %}

## Where is data stored after Globus?

All data uploaded with Globus will be published on ESS-DIVE. Data can be stored for public access on either ESS-DIVE's Tier 1 or Tier 2 storage services. **Tier 1** is the primary, underlying storage capability at ESS-DIVE where data and metadata can be accessed via dataset landing pages on <https://data.ess-dive.lbl.gov/>. **Tier 2** is ESS-DIVE's extended storage service for large (>500GB) and/or hierarchical data files and is not used for storing dataset metadata.&#x20;

Before the Globus upload process begins, the ESS-DIVE Team will help determine whether your data should be published on ESS-DIVE's Tier 1 or Tier 2 data storage. To learn more about ESS-DIVE's tiers of storage, how they differ, and how ESS-DIVE stores large data, see our Large Data Support documentation:

{% content-ref url="/pages/FBB6pXi8rET74QMiuPNT" %}
[Large Data Support](/contributing-data/file-upload-guidance/large-data-support)
{% endcontent-ref %}


# Setup Globus

Learn how to log in to Globus, install the Globus desktop application, and create a collection.

{% hint style="info" %}
Globus should be used *only when data cannot* be uploaded with the web interface or the Dataset API
{% endhint %}

## 1. Create Globus Account

:point\_right: *If you have logged into Globus before, skip to* [*Install Globus Connect*](#id-2.-install-globus-connect-personal) *section*

1. Navigate to the Globus Web App [login](https://app.globus.org/) page
2. Select the “Sign in with ORCID iD” button&#x20;
   * You may also choose to sign in using your **existing organization login** if your organization is available in the drop down list. If your organization is not available, we recommend signing in with your ORCID iD.
3. Enter your ORCiD credentials and authorize access
4. Fill out the basic account information to complete your sign up

You are now signed in. Welcome to Globus!

![Figure 1: The Globus web app login page](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-MXt8B-oVPd_fAe1PHJt%2F-MXt9tc3B_sj9d2Ns7h2%2FGlobus%20Login.png?alt=media\&token=ceec2dda-30ba-452b-a9a7-12ea9dd06040)

## 2. Install Globus Connect Personal

Globus connect personal is a desktop application that is necessary to upload data from your desktop to Globus.&#x20;

:point\_right: *If you are transferring data from an existing Globus collection or already have Globus Connect Personal setup, skip to the* [*Upload Data with Globus*](/programmatic-tools/large-data-support/how-to-upload-large-data) *page.*

1. [Login](https://app.globus.org/) to the Globus Web App
2. Go to File Manager
3. Click the empty “Collection” search bar (Figure 2)
   * There may be two panels on your screen, select either Collection search bar to continue
4. You will be prompted to use Globus Connect Personal; click on the button “ + Install Globus Connect Personal” to download the app (or refer to the Globus Connect Personal instructions for [MacOS](https://docs.globus.org/how-to/globus-connect-personal-mac/), [Windows](https://docs.globus.org/how-to/globus-connect-personal-windows/), or [Linux](https://docs.globus.org/how-to/globus-connect-personal-linux/))
5. Once the installation is complete, launch the Globus Connect Personal application
6. Click login, you will be directed to the Globus login webpage in your browser. Login to your Globus account and allow Globus access to your personal computer
7. After logging in, a dialog box will automatically appear, prompting you to name and describe your collection. Proceed to step 3, Setup your Collection, where we will talk more about this.

![Figure 2: You can find the Globus Connect installation prompt on the File Manager collection search page](https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LgU7GrufgIOpY1Ufd5R%2F-MXt8B-oVPd_fAe1PHJt%2F-MXtA6k0bSKkbb3R4lhh%2FGlobus%20Connect%20Personal.png?alt=media\&token=66ef4070-abe8-4c4b-8794-ab7f253e4e22)

## 3. Name & Save your Desktop Collection

Globus “collections” are discoverable access points that allow data to be transferred to and from them; essentially these “collections” are shareable folders in Globus. In this step, we will create a collection for your desktop using Globus Connect Personal. This is how Globus will access the files on your computer and allow you to easily transfer your files.

1. Name and describe your collection. We recommend naming it after your computer with something easily recognizable, such as “Jane Doe Macbook Pro” or “Doe Desktop”.\
   Click Save.
2. Click the link “Access Data in this Collection”. This will take you to the File Manager where you can see the contents of your new collection; review these files and make sure you can see the contents of your desktop files.&#x20;
3. Globus is now connected to your desktop and is ready for upload! Proceed to the [Upload Data with Globus](/programmatic-tools/large-data-support/how-to-upload-large-data) page to learn how to use Globus Connect.

Access your desktop collection at any time by opening the File Manager, selecting the Collections search bar, and navigating to the "Your Collections" tab (Figure 3).

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FEs1uUAxfxpRZ1JDAfTCm%2FGlobus.png?alt=media&amp;token=dceac34e-3029-43e8-92f5-d6a289b3f37f" alt=""><figcaption><p>Figure 3: "Your Collections" tab can be accessed from the file manager after selecting the collection search bar</p></figcaption></figure>


# Upload Data with Globus

Learn how to transfer your files to Globus from Globus Connect Personal or an existing Globus endpoint.

{% hint style="warning" %}
Please note that once your large data is made accessible on NERSC via Globus, neither you nor ESS-DIVE will be able to edit the files. They can either be transferred elsewhere or deleted.
{% endhint %}

### 1. Create ESS-DIVE Dataset

Before we can transfer data to ESS-DIVE using Globus, it is necessary to create a dataset on ESS-DIVE and get the metadata ready for publication. It is recommended to complete as much metadata as possible before moving on to the next step.

1. Navigate to [ESS-DIVE](https://data.ess-dive.lbl.gov/), click the "Submit Data" button to start a new dataset. Fill out the minimum metadata and click the **“Submit Dataset”** button to create a private dataset.
   * :point\_right:  Read more about creating datasets on our [Web Form tutorial](https://docs.ess-dive.lbl.gov/contributing-data/data-submission-guidelines/complete-guide#accessing-the-data-submission-web-form).
   * :point\_right:  You can also use the [Dataset API](/programmatic-tools/ess-dive-dataset-api) to submit metadata to ESS-DIVE.&#x20;
2. Do not upload any data files to this dataset.
3. Initiate a publication request to begin your Globus upload. Select "**Manage Publication**" from your dataset landing page, then click "**Start Publication Process**" in the pop-up window.&#x20;
   * :point\_right:  See complete instructions on our [Request Publication](/publish-data/publish-your-dataset/request-publication) documentation page.
4. Shortly after, you will receive an email that confirms your request has been received. Reply to this email to inform the ESS-DIVE Team that you need to upload your data files using Globus. If you did not receive a confirmation email, please [contact our support team](https://ess-dive.lbl.gov/contact/).
5. The ESS-DIVE Team will then reply to your message to continue the Globus upload process.&#x20;
   * **Tier 2 datasets:** At this time, an inactive DOI will be minted for your dataset. The inactive DOI will be used at a later step in the Globus upload process and it will not become active until after your dataset is published.

### 2. Find your ESS-DIVE Globus Collection

The ESS-DIVE Team will create a Globus collection that will be managed by ESS-DIVE and it will be specific to the dataset you are publishing. This section will walk you through how to enable access to this collection, which will later allow you to transfer your data here.

<div><figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FepjRhtJWSsodaJOkHg8T%2FGlobus-Account.png?alt=media&amp;token=6ffff6e1-d309-4fd1-a70b-07b7eda2d0b0" alt=""><figcaption><p>Figure 1: Visit the settings page to look up your account identity information</p></figcaption></figure> <figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2Fvvpn1B5rUlnHOXlJvL2g%2FGlobus-SharedWithYou.png?alt=media&amp;token=b992e9fc-67bc-4241-a21f-09c4dd4ed0f5" alt=""><figcaption><p>Figure 2: Your ESS-DIVE Globus collection can be easily found in the file manager search by selecting the "Shared with You" tab</p></figcaption></figure></div>

1. The ESS-DIVE Team will need the email or ORCID associated with your Globus “identity” (i.e. your Globus account ID) to enable shared access to the Globus collection we create. Reply to the confirmation email you received from ESS-DIVE in the previous step (“Create ESS-DIVE Dataset") and send your email or ORCID to the ESS-DIVE Team, whichever you used to setup your account.&#x20;
   * Confirm your account information by checking your identity. To find your identity, go to Globus (<https://app.globus.org/>) and navigate to the “Settings” tab (Figure 1). There you will see your primary identity. Select the dropdown arrow and copy the information listed there.
2. **Wait for a reply from the ESS-DIVE Team to proceed.** We will send you the name of your personalized Globus collection where you will upload your data. The collection name will be named after the inactive DOI minted for your dataset in the format of `doi-10-15485-1813868`.
3. Once you receive your ESS-DIVE Globus collection,[ log in](https://app.globus.org) to the Globus Web App.
4. Verify that you now have access to your collection by navigating to the "File Manager".  Select the “Collection” search bar, then click the “Shared with you” tab (Figure 2) and look for the Globus collection name provided via email by ESS-DIVE.<br>

   If you do not see your ESS-DIVE collection here, then you are not connected to it. Please email the Support Team if this occurs.

### 3. Transfer Files to ESS-DIVE Globus Collection

If you are transferring data from your local desktop, make sure you have Globus Connect Personal installed (see [Setup Globus](/programmatic-tools/large-data-support/how-to-setup-globus) page for instructions). If you’re using an existing public endpoint in Globus, make sure you have access to that endpoint. We will now move your data into your ESS-DIVE collection.

<div><figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FuVVFTEFTVDEK2M0hnVvX%2FGlobus-FileManagerEmpty.png?alt=media&amp;token=7767d91d-58ec-4de3-a9cd-f44f4f7d16ab" alt=""><figcaption><p>Figure 3: The panel view can alternate between displaying the left, right, or both panels.</p></figcaption></figure> <figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FE6B0p7WnRLJ8xnR6pBcq%2FGlobus-TransferData.png?alt=media&amp;token=fa2042f5-e584-463d-98f9-632758e19811" alt=""><figcaption><p>Figure 4: Choose the collections to transfer data from and to, select your files, and start the transfer job using the intuitive buttons on-screen.</p></figcaption></figure></div>

1. To begin transferring files to your shared Globus collection, go to the File Manager. If you have only one panel on screen, use the panel toggle in the top right to change your view at any time (Figure 3).
2. Select the “Collection” search bar and navigate to the “Your Collections” tab (see example in [Setup your Collection](https://docs.ess-dive.lbl.gov/programmatic-tools/large-data-support/pages/-MX-liL4i-S8I9i6XJmB#3.-setup-your-collection)). Then select your desktop collection or select an existing source endpoint if you will be using an existing one.&#x20;
3. Now, in the empty panel, select the “Collection” search bar. Navigate to the “Shared with You” tab (Figure 2). Select the personalized ESS-DIVE folder that we shared with you (in the format of `doi-10-15485-1813868`).&#x20;
   * Note that this collection will be empty. At the top, a yellow banner will say "ESS-DIVE Ingest Endpoint" confirming that this is the correct collection.
4. Now you’re ready to transfer your data. Select the files or folders from your collection that you’d like to upload and publish with your ESS-DIVE dataset (Figure 4).
5. Once all your data is selected, hit the “Sync transfer...” button, the "Start" button, or drag your selected files to the shared ESS-DIVE collection (Figure 4).
6. This will start a transfer job and you will see a green popup window on your screen confirming that the transfer has started (Figure 5). In many cases the transfer is quick, but if your transfer is taking a while, use the link from this popup window to check in on the status of the transfer or look in the Activity tab (Figure 6).
7. You will know that the transfer job is complete when Globus sends you a confirmation email that the transfer finished.\
   \
   Congrats! You have completed the Globus portion of this process.

<figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2F7G6HzvxULjYMIbBDaQzh%2FGlobus-StartTransfer.png?alt=media&amp;token=066c2c74-9ead-4ce7-94f0-11b0281128a2" alt=""><figcaption><p>Figure 5: A confirmation message pops up on screen once you initiate a transfer </p></figcaption></figure>

<div align="center"><figure><img src="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FcCQdVTikUUzwZFkwJ0EZ%2FGlobus-ActivityMonitor.png?alt=media&amp;token=e6e1b2e1-4b4a-4f20-a23c-0b7676122223" alt="" width="375"><figcaption><p>Figure 6: Use the Activity tab to check the status of your transfer</p></figcaption></figure></div>

At this point, your files have not been uploaded or published to the ESS-DIVE repository. This is a temporary collection that the ESS-DIVE Team will use to stage your data for publication. In the next step, we will upload your data and publish it on ESS-DIVE.

### 4. Move Data from Globus to ESS-DIVE

This step is done by the ESS-DIVE Team. Your review will be necessary.

1. Go to your inbox and reply to the ESS-DIVE Team that your data files have been transferred to your shared Globus collection and are ready for publication.
2. **Wait for a reply from the ESS-DIVE Team to proceed.** Once we can access your data on the ESS-DIVE Globus collection, we will upload your data to Tier 1 or Tier 2. This process can take a few days depending on how large your files are. ESS-DIVE will email you after the upload is complete and provide a link to your data files either on Tier 1 or Tier 2.
3. Review the data files in your dataset to make sure everything looks correct to you. Let the ESS-DIVE Team know if there are any issues.&#x20;
4. Reply to the ESS-DIVE Team to let us know you approve the file upload.&#x20;
   * For data stored on Tier 2, ESS-DIVE will add external links in your dataset that direct end users to access your data via Tier 2.&#x20;

At this point, the Globus upload process is complete! The final section before publication is to complete your dataset metadata and initiate a publication review. The publication process has been reiterated below for ease of reference.

### 5. Review and Publish ESS-DIVE Dataset

After your files are successfully uploaded to your dataset on Tier 1 or Tier 2 and you approve the files, ESS-DIVE will wait to hear confirmation from you that your metadata is complete and ready for publication. &#x20;

1. Respond to the ESS-DIVE Team letting them know that your metadata is ready for review.
2. The ESS-DIVE Team will email you any revision requests, if necessary. Revise your metadata as requested.
3. If there are no further revisions requested, your dataset will be approved for publication. Your dataset metadata will be published and the data you transfered to Globus will become publicly accessible via the ESS-DIVE repository.


# FAQs

### How ESS-DIVE defines large data

A dataset that meets any of the following criteria is considered large data by ESS-DIVE:

* Contains one or more files greater than 500GB
* Contains more than 100 individual files outside of zipped hierarchy

### What is a Globus "collection"?

Globus “collections” are discoverable access points that allow data to be transferred to and from them; essentially these “collections” are shareable folders in Globus.&#x20;

### What is Globus Connect Personal?

Globus connect personal is a desktop application that is necessary to upload data from your desktop to Globus. When using Globus Connect Personal, it creates a collection (or folder) for the files on your desktop which allows Globus to access the files on your computer and enable high-performance transfers.