Search with Deep Dive API

An advanced search tool within data files on ESS-DIVE that meet certain standard criteria.

This page contains resources to get started with data search and discovery through the Deep Dive API, including an example search using the interactive documentation at fusion.ess-dive.lbl.gov.

See it in action: Check out our Jupyter Notebook tutorial with ready-to-run code examples using the Deep Dive API. (Tip: launch the tutorial directly in your browser with the Google Collab button)

Search through files to find data

ESS-DIVE's Deep Dive API provides advanced search capabilities for standardized public data files and is separate from the main ESS-DIVE search and Dataset API. Rather than searching across dataset-level metadata, the Deep Dive API allows for search within data files to efficiently find data relevant to your scientific research.

Public datasets are indexed by the Fusion Database (Fusion DB), which parses datasets following the File Level Metadata (FLMD) and CSV Guidelines Reporting Formats. Datasets that use reporting formats are essential to enabling advanced search, and planned enhancements of the Fusion DB will leverage additional reporting formats where possible.

What public data is available in DeepDive?

Not all data files on ESS-DIVE are available for search with the Deep Dive API. To be surfaced in this search tool, the dataset metadata and data files must meet the following criteria:

1. Dataset metadata includes ESS-DIVE’s standardized reporting format keyword for the File Level Metadata reporting format (ESS-DIVE File Level Metadata Reporting Format)

2. The dataset follows the File Level Metadata reporting format by including both file level metadata and data dictionary files

3. The file level metadata and data dictionary files follow the respective file naming conventions

4. CSV files correctly follow the CSV reporting format guidelines to ensure successful file parsing

For additional information on formatting files properly for validation and parsing, please refer to the Reporting Format Requirements page. Documentation and instructions for the use of all data and metadata reporting formats can be found on the ESS-DIVE Workspace GitHub.

Find all datasets using reporting formats from ESS-DIVE's Reporting Format data portal.

Types of search endpoints

There are two search endpoints currently available through the Deep Dive API:

Name	Endpoint	Usage
Query-Data	deepdive	Searches within files on public datasets for data that matches your search criteria.
Get-Dataset-File	deepdive/{doi}:{file_path}	Retrieves a dataset file by its file path and summarizes all column/row headers in the file.

Workflow for Data Discovery

In the next page, we'll walk through the intended use of the available search endpoints in the Deep Dive API for discovering data in ESS-DIVE.

How to Query Data