> For the complete documentation index, see [llms.txt](https://docs.ess-dive.lbl.gov/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.ess-dive.lbl.gov/contributing-data/file-upload-guidance.md).

# File Upload Guidance

This page provides all necessary information to determine how to transfer your data to ESS-DIVE.

## Dataset Submission Workflow

{% stepper %}
{% step %}

#### **Always submit your metadata first!**

Use the **Online Submission Form** or **Dataset API** to create and submit your metadata before providing any files. You can finalize your metadata later, it is not necessary to meet all publication requirements right away. 
{% endstep %}

{% step %}

#### **Pick the** [**Storage Option**](#storage-options) **and** [**Transfer Option**](#data-transfer-options) **that best suits your data**

Continue reading this page to find quick summaries of your options.
{% endstep %}

{% step %}

#### **Read through** [**ESS-DIVE's File Publication Requirements**](#file-requirements)

ESS-DIVE has certain file requirements that are checked during publication review. Review them before transferring your data to expedite the review process.
{% endstep %}

{% step %}

#### **Start uploading your files**

Each transfer option below provides links to the relevant instructions to get started.
{% endstep %}

{% step %}

#### Finalize your dataset metadata

Come back at anytime to finish your metadata! In general, always submit your metadata edits separately from your files.
{% endstep %}
{% endstepper %}

{% hint style="info" %}
**Do you have Large Data?**

On ESS-DIVE, we define "large data" as any individual file that is greater than 10GB or a dataset that contains more than 100 individual files. 

Large data should be uploaded following the [Large Data Support](/contributing-data/file-upload-guidance/large-data-support.md) instructions.
{% endhint %}

{% hint style="warning" icon="circle-question" %} <mark style="color:orange;">**Do you have Terabytes of data?**</mark>&#x20;

If your project needs to upload more than a Terabyte of data, ESS-DIVE will need to prepare to accommodate this. Contact <ess-dive-support@lbl.gov> to inform us of your needs.
{% endhint %}

<details>

<summary><strong>Why are the data storage and transfer volumes different?</strong></summary>

An **transfer limit** is the maximum volume of data you can upload *at one time* using one of ESS-DIVE's transfer tools (limit varies per tool). If your data exceeds this limit, you can batch your data into multiple uploads.

The **storage limit per dataset** is the maximum quantity of files and total volume of data that you can store on one dataset (limit varies per storage method). If your dataset volume exceeds the Tier 1 storage limit, you have to store your dataset on Tier 2.

</details>

***

## Storage Options

ESS-DIVE has two storage options available: Tier 1 and Tier 2. Your file size and structure will determine which storage option is right for your data.

<table data-card-size="large" data-view="cards" data-full-width="false"><thead><tr><th align="center"></th><th></th><th>Benefits</th><th>Action Item</th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td align="center"><strong>Tier 1</strong></td><td><ul><li>Folders must be zipped</li><li>Recommended for &#x3C;100 files</li><li>Recommended for &#x3C;100GB total volume</li><li>Upload data using any transfer option</li></ul></td><td><ul><li>Tracks data file views and downloads</li><li>DataONE backups</li><li>Supported by all ESS-DIVE features<br></li></ul></td><td><ul><li>Zip all folders before upload</li><li>Pick the <a href="#data-transfer-options">Data Transfer Option</a> that best suits your data</li></ul></td><td></td></tr><tr><td align="center"><strong>Tier 2</strong></td><td><ul><li>Unzipped, hierarchical folders</li><li>Recommended for >100 files</li><li>Recommended for >100GB total volume</li><li>Upload data using Globus</li></ul></td><td><ul><li>Browsable folder hierarchies</li><li>Public copy available on Globus to enable cloud-based downloads</li><li>Generates manifest with checksums for each file to verify integrity</li></ul></td><td><ul><li><a href="https://app.gitbook.com/o/-LgU7CB5CVrJQMt0aaNu/s/-LgU7GrufgIOpY1Ufd5R/~/edit/~/changes/211/contributing-data/file-upload-guidance/large-data-support/~/comments#how-to-contact-ess-dive-about-publishing-large-data">Contact ESS-DIVE to use Globus</a></li></ul></td><td><a href="/contributing-data/file-upload-guidance/large-data-support.md#tier-2-storage-for-large-data">Large Data Support</a></td></tr></tbody></table>

## **Data Transfer Options**

ESS-DIVE has three tools available for transferring data: Online Submission Form, Dataset API, and Globus Transfer Service. Your storage method, data size, and/or personal preference will determine which option is best for your data.

If you have large data or need to store your data on Tier 2, you will need to use Globus.

<table data-view="cards" data-full-width="false"><thead><tr><th align="center"></th><th>Volume Limit per Transfer</th><th>Folder Organization</th><th>What is this for?</th><th>What can you submit with this tool?</th><th data-hidden data-card-target data-type="content-ref"></th><th data-hidden data-card-cover data-type="image">Cover image</th></tr></thead><tbody><tr><td align="center"><strong>Online Submission Form</strong> </td><td><strong>&#x3C; 10 GB</strong></td><td>Folders must be zipped before upload</td><td>Quick, one-off datasets</td><td>Metadata + data</td><td><a href="/contributing-data/submit-data-with-online-form.md">Submit Data with Online Form</a></td><td><a href="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2Fx6DkzUjgu8FMXHTvZGwz%2FScreenshot%202026-08-17%20103357.png?alt=media&amp;token=d79342be-86e8-47d9-8500-ef4b4b237ddd">Screenshot 2026-08-17 103357.png</a></td></tr><tr><td align="center"><strong>Dataset API</strong></td><td><strong>&#x3C; 10 GB</strong></td><td>Folders must be zipped before upload</td><td>Programmatic, bulk dataset management </td><td>Metadata + data</td><td><a href="/contributing-data/submit-data-with-the-dataset-api.md">Submit Data with the Dataset API</a></td><td><a href="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FUY50LsfzhAVNBBsxGzlm%2FScreenshot%202026-08-17%20103519.png?alt=media&amp;token=0e99e074-cce2-4074-ba16-7e154734294a">Screenshot 2026-08-17 103519.png</a></td></tr><tr><td align="center"><strong>Globus Transfer Service</strong></td><td><strong>No Limit</strong></td><td>Supports unzipped folders</td><td>Default transfer method for files >10GB and Tier 2 data</td><td>Data only</td><td><a href="/contributing-data/file-upload-guidance/large-data-support.md">Large Data Support</a></td><td><a href="https://3166205607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgU7GrufgIOpY1Ufd5R%2Fuploads%2FvTco6WrXNJU2jhXOlULi%2FScreenshot%202026-08-17%20103529.png?alt=media&amp;token=e899b1e6-bb2b-4ae1-b469-b122c9b595a7">Screenshot 2026-08-17 103529.png</a></td></tr></tbody></table>

## File Publication Requirements

These requirements are checked by the ESS-DIVE Team during publication review. \
**Review these requirements before transferring your data to expedite the review process.**

<table><thead><tr><th width="112.30145263671875"></th><th>Requirements</th></tr></thead><tbody><tr><td><strong>File Name</strong></td><td><ul><li>File is named concisely and descriptively</li><li>Any acronyms used in file names are explained in the abstract</li></ul></td></tr><tr><td><strong>File Extension</strong></td><td><ul><li><p>Use non-proprietary file formats where possible <mark style="color:$info;">(Included in</mark> <a href="/publish-data/check-dataset-metadata-quality.md#automated-checks"><mark style="color:$info;">Automated Checks</mark></a><mark style="color:$info;">)</mark></p><ul><li>Examples include Comma Separated Values (CSV) files, text (TXT) files, PNG, JPEG or TIFF image files, R or Python scripts, GeoJSON, and NetCDF files, among many others.</li><li>Keeping both the proprietary and converted version of the file with the same name is acceptable</li></ul></li><li>Word and Excel documents are proprietary formats and should be converted into non-proprietary formats (e.g., PDFs and CSVs)</li><li>If specific software is needed to view any uncommon file formats, specify the software in your abstract</li></ul></td></tr><tr><td><strong>Zip File</strong></td><td><ul><li>Describe contents of any zip files in the abstract </li><li><p>Zip files should not be corrupted. Ensure zip files can be opened before upload.</p><ul><li><p><span data-gb-custom-inline data-tag="emoji" data-code="1f4a1">💡</span> Quickly check your zip compression method in your terminal with:</p><pre class="language-ruby"><code class="lang-ruby">unzip -t file_name
</code></pre></li></ul></li><li><p>Use non-proprietary compression methods to zip your files. Open source compression methods allow zip files to be opened on any operating system.</p><ul><li>Non-proprietary compression methods include deflate.</li><li>Proprietary compression methods include <mark style="color:$danger;">deflate64</mark>.</li><li><span data-gb-custom-inline data-tag="emoji" data-code="26a0">⚠️</span> Windows uses a proprietary compression method (deflate64) to zip folders greater than 2GB. </li></ul></li></ul></td></tr></tbody></table>

{% hint style="info" icon="readme" %}
**Metadata Publication Requirements**

The ESS-DIVE Team also checks your metadata during the publication review process<a href="/contributing-data/package-level-metadata.md" class="button primary medium" data-icon="arrow-right">Click here to see metadata requirements</a>
{% endhint %}

{% hint style="info" icon="table-pivot" %}
**Reporting Format File Requirements**

A series of checks are performed on reporting format files during the publication review process&#x20;

<a href="/publish-data/review-cycle-and-criteria.md#review-process-for-reporting-formats" class="button primary medium" data-icon="arrow-right">Click here to see reporting format file requirements</a>
{% endhint %}

## Troubleshooting

#### Online Form

**\*Coming Soon\***&#x20;

#### Dataset API

{% content-ref url="/pages/GTgJ2cW6V8fNrNXc5cZC" %}
[Setup and Troubleshoot](/programmatic-tools/ess-dive-dataset-api/setup-and-troubleshoot.md)
{% endcontent-ref %}

#### Globus

{% content-ref url="/pages/-MXss07PasswM3LQmsGA" %}
[FAQs](/programmatic-tools/large-data-support/faqs.md)
{% endcontent-ref %}