Ctrlk
Contact Us
For the complete documentation index, see llms.txt. This page is also available as Markdown.

File Upload Guidance

This page provides all necessary information to determine how to transfer your data to ESS-DIVE.

Dataset Submission Workflow

1

Always submit your metadata first!

Use the Online Submission Form or Dataset API to create and submit your metadata before providing any files. You can finalize your metadata later, it is not necessary to meet all publication requirements right away.

2

Pick the Storage Option and Transfer Option that best suits your data

Continue reading this page to find quick summaries of your options.

3

Read through ESS-DIVE's File Publication Requirements

ESS-DIVE has certain file requirements that are checked during publication review. Review them before transferring your data to expedite the review process.

4

Start uploading your files

Each transfer option below provides links to the relevant instructions to get started.

5

Finalize your dataset metadata

Come back at anytime to finish your metadata! In general, always submit your metadata edits separately from your files.

Do you have Large Data?

On ESS-DIVE, we define "large data" as any individual file that is greater than 10GB or a dataset that contains more than 100 individual files.

Large data should be uploaded following the Large Data Support instructions.

circle-question

Do you have Terabytes of data?

If your project needs to upload more than a Terabyte of data, ESS-DIVE will need to prepare to accommodate this. Contact ess-dive-support@lbl.gov to inform us of your needs.

Why are the data storage and transfer volumes different?

An transfer limit is the maximum volume of data you can upload at one time using one of ESS-DIVE's transfer tools (limit varies per tool). If your data exceeds this limit, you can batch your data into multiple uploads.

The storage limit per dataset is the maximum quantity of files and total volume of data that you can store on one dataset (limit varies per storage method). If your dataset volume exceeds the Tier 1 storage limit, you have to store your dataset on Tier 2.

Storage Options

ESS-DIVE has two storage options available: Tier 1 and Tier 2. Your file size and structure will determine which storage option is right for your data.

Tier 1

  • Folders must be zipped

  • Recommended for <100 files

  • Recommended for <100GB total volume

  • Upload data using any transfer option

Benefits

  • Tracks data file views and downloads

  • DataONE backups

  • Supported by all ESS-DIVE features

Action Item

Tier 2

  • Unzipped, hierarchical folders

  • Recommended for >100 files

  • Recommended for >100GB total volume

  • Upload data using Globus

Benefits

  • Browsable folder hierarchies

  • Public copy available on Globus to enable cloud-based downloads

  • Generates manifest with checksums for each file to verify integrity

Action Item

Data Transfer Options

ESS-DIVE has three tools available for transferring data: Online Submission Form, Dataset API, and Globus Transfer Service. Your storage method, data size, and/or personal preference will determine which option is best for your data.

If you have large data or need to store your data on Tier 2, you will need to use Globus.

Cover

Online Submission Form

Volume Limit per Transfer

< 10 GB

Folder Organization

Folders must be zipped before upload

What is this for?

Quick, one-off datasets

What can you submit with this tool?

Metadata + data

Cover

Dataset API

Volume Limit per Transfer

< 10 GB

Folder Organization

Folders must be zipped before upload

What is this for?

Programmatic, bulk dataset management

What can you submit with this tool?

Metadata + data

Cover

Globus Transfer Service

Volume Limit per Transfer

No Limit

Folder Organization

Supports unzipped folders

What is this for?

Default transfer method for files >10GB and Tier 2 data

What can you submit with this tool?

Data only

File Publication Requirements

These requirements are checked by the ESS-DIVE Team during publication review. Review these requirements before transferring your data to expedite the review process.

Requirements

File Name

  • File is named concisely and descriptively

  • Any acronyms used in file names are explained in the abstract

File Extension

  • Use non-proprietary file formats where possible (Included in Automated Checks)

    • Examples include Comma Separated Values (CSV) files, text (TXT) files, PNG, JPEG or TIFF image files, R or Python scripts, GeoJSON, and NetCDF files, among many others.

    • Keeping both the proprietary and converted version of the file with the same name is acceptable

  • Word and Excel documents are proprietary formats and should be converted into non-proprietary formats (e.g., PDFs and CSVs)

  • If specific software is needed to view any uncommon file formats, specify the software in your abstract

Zip File

  • Describe contents of any zip files in the abstract

  • Zip files should not be corrupted. Ensure zip files can be opened before upload.

    • 💡 Quickly check your zip compression method in your terminal with:

  • Use non-proprietary compression methods to zip your files. Open source compression methods allow zip files to be opened on any operating system.

    • Non-proprietary compression methods include deflate.

    • Proprietary compression methods include deflate64.

    • ⚠️ Windows uses a proprietary compression method (deflate64) to zip folders greater than 2GB.

readme

Metadata Publication Requirements

The ESS-DIVE Team also checks your metadata during the publication review processClick here to see metadata requirements

table-pivot

Reporting Format File Requirements

A series of checks are performed on reporting format files during the publication review process

Click here to see reporting format file requirements

Troubleshooting

Online Form

*Coming Soon*

Dataset API

Setup and Troubleshoot

Globus

FAQs

Last updated