Package Service Tutorial

Getting started with ESS-DIVE Package Service 1.3

The ESS-DIVE Package Service is a service that enables projects to programmatically store data packages with ESS-DIVE. This is an alternative to using the ESS-DIVE web portal form for data uploads. This service encodes metadata using the JSON-LD specification. JSON-LD is a schema to encode linked Data using JSON, and in the future will be used by Google to index metadata for searches. The use of the standardized JSON-LD schema will dramatically increase the visibility of data packages, and also enable projects to create one-time code that can be reused for periodic uploads of data packages to ESS-DIVE.

Current Maximum Upload Limit: 10 GB per file Please contact ess-dive-support@lbl.gov to upload packages above our limit, up to 100GB.

The ESS-DIVE Package service, allows you to test submissions of JSON-LD data package metadata to ESS-DIVE’s sandbox instance, to check whether metadata curated by projects are mapped correctly onto ESS-DIVE’s data package metadata schema. Data package metadata refers to the top level metadata that enables a data package to be “discoverable” in search results. Examples of top-level metadata include the title, abstract, authors, variables and keywords. Other file-level metadata, such as those that describe the data file structure or variables are not included in this service.

See the ESS-DIVE metadata guide for descriptions, expectations, & JSON-LD equivalents for each metadata field.

You can get access to functioning coding examples on ESS-DIVE package service repository.

Provide feedback on this service to ess-dive-support@lbl.gov.

Get Access

Follow the steps below to get access to the ESS-DIVE

  1. If you are not registered with ESS-DIVE, follow the new user instructions

  2. Review the presentation given at the ESS-DIVE Archive Partnership Board (APB) Meeting Monday November 5th.

  3. Get an Authentication Token

    1. Sign In with Orcid

    2. Click your Name in the right hand corner and select My Profile

    3. Now Click the Settings>Authentication Token

    4. Scroll down and click Copy on the “Token” tab to get your authentication token.

Figure 1. Hover over your name and click on My Profile
Figure 2. Click on Authentication Token under Settings and copy the token

Preparation

ESS-DIVE is a REST API which allows you to interact through the tools of your choice. You may start learning the expected schema for the metadata on https://api-sandbox.ess-dive.lbl.gov. This documentation allows you to get familiar with the interface, to understand the available operations, possible errors and the structure of the JSON produced and expected.

You can also review the expected JSON-LD structure from the example JSON-LDs on ESS-DIVE coding examples github repository.

To review the expectations for submitted metadata, refer to our Package-Level Metadata Guide

After you have familiarized yourself with the package service REST API, you can use the programming language of your choice to submit JSON-LD metadata to be validated. Two of the most common programming languages in the ESS space are Python and R. Either of these languages can be used to write scripts for uploading data package metadata to be validated.

  • Python: data package JSON-LD metadata can be submitted using the requests module

  • R: data package metadata can be submitted using the httr and jsonlite packages

  • Java: data package JSON-LD metadata can be submitted using Apache HTTPComponents

Coding Examples

The following pages provide coding examples for retrieving a list of submitted data packages and submitting data package metadata for validation.

  • Submitting a data package

  • Updating a data package

  • Retrieving a single data package's metadata

  • Retrieving a list of all data package's metadata

For additional information about the api, you can also review the documentation at https://api-sandbox.ess-dive.lbl.gov

Ready to Submit

Once you've familiarized yourself with ESS-DIVE's metadata and package service schema, use our production domain https://api.ess-dive.lbl.gov/ to submit datasets to ESS-DIVE for publishing and review.

Note that the https://data.ess-dive.lbl.gov/ domain is for data submission via web user interface only.