How to Query Data
A workflow for data discovery that walks through the intended use of the Deep Dive API search endpoints.
Last updated
A workflow for data discovery that walks through the intended use of the Deep Dive API search endpoints.
Last updated
The documentation at includes technical details, expected schema, error information, and interactive query parameters for the endpoints available through the Deep Dive API. We recommend starting out with this interactive documentation to help familiarize yourself with query parameters and outputs.
While programmatic experience can be helpful, it is not required.
When you expand an endpoint, each one has four major sections. Here we provide a brief explanation. The sections in green are most relevant to this demonstration.
Header: This displays the expected format for the base URL (e.g. /api/v1/deepdive) and the first sentence in the dropdown describes what the endpoint does.
Parameters: This section provides a list of the available search parameters that can be input into the endpoint. If a parameter must be filled out, it will say "* required". Each parameter has a definition that explains what the parameter does. For your convenience, the screenshots on this page can be used to reference all available parameters and their definitions.
Server Responses (interactive): By default, this section is not visible until you select "try it out" and execute an example search (step 1-2). The interactive response section lists a copy of the search request you made (e.g., the Curl command and Request URL) and the server response (i.e., the search result). Definitions of the results in the server response can be found in the Schemas section of the interactive documentation (visible in Fig 1; see step 4 for a demonstration).
Responses (general): By default, this section is always visible in the Responses. This lists the possible response codes and response messages that could be returned. This is useful reference material for debugging searches when writing your own code. For this demonstration, we will only be looking at successful responses (status code 200).
Header and Parameters
Interactive Server Responses General Details about Responses
Letās first expand the Query-Data endpoint details.
This section shows the available search parameters, parameter descriptions, expected format, and editable value boxes. Select the āTry it outā button highlighted in orange to edit the values. From here, we can easily enter parameters to test out search queries.
fieldName refers to the column or row name (aka field name, or variable name) in the data file, which come from the dataset's Data Dictionary. This is a component of the File Level Metadata Reporting Format. Search terms entered in the fieldName parameter do not have to be an exact match.
The fieldDefinition parameter comes provides definitions of each field name included in the data files.
Letās execute the following search query (Table 1) to look for data within this public dataset that is using reporting formats and is available in the Deep Dive API.
Roley S ; Hall Jr. R O H J; Garayburu-Caruso V A ; Perkins W A ; Stegen J C (2023): Data and scripts associated with "Coupled primary production and respiration in a large river contrasts with smaller rivers and streams.". River Corridor and Watershed Biogeochemistry SFA, ESS-DIVE repository. Dataset. doi:10.15485/1985922 accessed via https://data.ess-dive.lbl.gov/datasets/doi:10.15485/1985922 on 2024-06-24
Copy the values in Table 1 and enter the query parameters as follows:
Table 1: Example Query-Data parameters
Our example parameters will look like this once completed (Fig 3). Select the "Execute" button to query the Deep Dive API.
Find all datasets using reporting formats in ESS-DIVE's Reporting Format data portal.
The responses section (Fig 4) will show you the search results, which is referred to here as the Server Response (see reference material).
In the response body, the pageCount value tells us that 4 fields (e.g. column/row headers) were found by this query. The results were found in at least one file.
next and previous show us which page of the results are currently displayed in the response body. A page is simply defined by the pageSize limit. In our example, one page will show up to 25 results. Given that we received 4 search results, the current response contains all results and we do not have to page through the results.
Letās take a look at the results starting with first field name: temp_C.
fieldName refers to the column or row name (aka field name, or variable name) in the data file, which come from the dataset's Data Dictionary.
Detailed information about this column/row is provided, including the unit, definition, and data_type. There are almost 80,000 data points available under this one column/row (total_record_count) and the data values are between a minimum of 1.02ā and 23.28ā (values_summary). Additionally, there are no missing values in this column/row (missing_values_count).
The version number (also called an ESS-DIVE Identifier) is a unique ID given to all datasets on ESS-DIVE. This number points to the exact version of the dataset associated with this data at the time the file was published, so you can always find the data should changes be made. The provided doi number is a persistent identifier to the latest version of the dataset. Additionally, we can see the CSV file name (data_file) and file_path where the column/row can be found.
To find this file in your browser, you can enter the DOI in the search bar (doi.org/10.15485/1985922) or you can look up the version number in ESS-DIVE using the standard URL format: data.ess-dive.lbl.gov/view/<ESS-DIVE Identifier>.
To find this file programmatically, we can use the data_file_url. Code example coming soon.
Here we demonstrate how to use the Deep Dive API to locate relevant files that appeared in our search results.
As we did in step 1, first expand the Get-Dataset-File endpoint details and select the "Try it out" button to edit the parameter values (Fig 5).
Letās execute the following search query (Table 2) using the DOI and file path associated with the temp_C field from our Query-Data results. Enter the parameters as follows:
Table 2: Example Get-Dataset-File parameters
Our example parameters will look like this once completed (Fig 5). Select the "Execute" button to query the Deep Dive API.
The Get-Dataset-File responses section (Fig 4) will show you the search results, which is referred to here as the Server Response (see reference material).
You'll notice that the Get-Dataset-File response contains all of the information returned by the Query-Data endpoint, with the addition of a few new values (Fig 6). It contains the dataset DOI, file name and the ESS-DIVE identifier that corresponds to the latest version of the dataset.
Then, under fields, you'll find a list of all the column/row headers in the specified file. It also lists the same detailed information about the data within the column/row header that Query-Data provides.
Another key difference of this endpoint is data_download return which provides the size of the file in bytes (contentSize), the file format (encoding_format), the file version (identifier), and a URL that points directly to the file (contentURL).
The server response for this query follows the DatasetFile schema. All definitions for the fields in this response are listed in the "Schema" > "DatasetFile" section of the Deep Dive API documentation.
Click the response value you are interested in to expand it's definition and expected formatting.
| Parameter Name | Value | Logic |
|---|---|---|
This information can help you determine whether this data is relevant for your research.
Then the remaining four values in the response can be used to locate the exact file and dataset version where this column/row temp_C can be found for download.
| Parameter Name | Value | Logic |
|---|---|---|
The power of the Get-Dataset-File endpoint is that it allows you to find all column/row information for an entire file at once. This provides more context about the data, allowing you to determine whether the file could be relevant to your research.
By providing file specifications, you can make an informed decision on whether the file size and format are within your expected constraints for data use.
doi
doi:10.15485/1985922
This is the DOI of the specific dataset that may have relevant data. It must be entered in the format of doi:{DOI number}.
fieldName
temp
I want to look for temperature data, but Iām not sure what the exact field name in the file is.
recordCountMin
20
I need at least 20 data points for the data to be useful for my research.
doi
doi:10.15485/1985922
This is the DOI of the file that I am interested in looking up. It must be entered in the format of doi:{DOI number}.
file_path
Roley_CR_Metabolism_Data_Package.zip/DO_temp_sensor_data.csv
I found this file in the results of my Query-Data search. I want to see a summary of all the data in this file (by header/row name), find the size of the file, and/or get the URL to directly download this file.