R Example

This page provides coding examples for retrieving a list of submitted datasets and submitting dataset metadata for validation using R.

While learning the expected schema for the metadata use https://api-sandbox.ess-dive.lbl.gov, as this is the domain shown in the examples in this documentation. Once you've familiarized yourself with ESS-DIVE's metadata and dataset schema, use our production domain https://api.ess-dive.lbl.gov/ to submit datasets to ESS-DIVE for publishing and review.

For additional information about the API, review the documentation at https://api-sandbox.ess-dive.lbl.gov.

ESS-DIVE Test API URL: https://api-sandbox.ess-dive.lbl.gov ESS-DIVE Production API URL: https://api.ess-dive.lbl.gov/ Help Desk: [e-mail]

Contents

• Setup

• Submitting a Dataset

  • Create Metadata

  • Metadata Only

  • Metadata and Data

• Search for Datasets

• Download Dataset Metadata

• Update a dataset

  • Metadata Only

  • Metadata plus new data file

• Troubleshooting

Setup

Install the following R packages

install.packages("httr")
install.packages("jsonlite")
install.packages("readr")

#Require the package so you can use it
require("jsonlite")
require("httr")
library(readr)

Setup the Dataset API information

token <- "<Enter your authentication token here>"
base <- "https://api-sandbox.ess-dive.lbl.gov"
header_authorization <- paste("bearer",token, sep=" ")
endpoint <- "packages"

Check that your token is up-to-date; it expires after a few days

Submit a Dataset

After creating metadata, you have the option to submit a dataset with only metadata or to submit a dataset with both metadata and data files.

Create Metadata

The following lines of code validate JSON-LD metadata for a single dataset. The example provided is from the ESS-DIVE sandbox site. (See https://data-sandbox.ess-dive.lbl.gov/#view/doi:10.3334/CDIAC/spruce.001).

Due to R complex JSON-LD support limitations, you need to create a text file of your JSON-LD and add it’s directory in the following read_file function.

Here’s an example for a JSON-LD located on our ESS-DIVE package service examples github repository.

To make sure your file is properly saved in the JSON-LD format, consider using the Atom text editor (https://atom.io)

json_file <- read_file("~/directory/to/your/jsonld/file")

Metadata Only

Prepare and submit the metadata for validation

Make sure to enter your own file path.

E.g. C:\\User\\Files\API_Tutorial.json for Windows & /Users/Files/API_Tutorial.json for Mac

call_post_package <- paste(base,endpoint, sep="/")
post_package = POST(call_post_package,
                    body = json_file,
                       add_headers(Authorization=header_authorization,
                       "Content-Type"="application/json"))

Review the results

results = content(post_package)
attributes(results)
$names
[1] "id"      "viewUrl" "detail"  "errors"  "dataset"

results$detail
[1] "Data Package created successfully."
results$errors
NULL

results$viewUrl
[1] "https://data-dev.ess-dive.lbl.gov/view/ess-dive-XXXXXXXXXXXX-20190621T175431086775"

Metadata and Data

To submit the metadata and data files, create a folder and add your data files to it then execute the following code:

call_post_package <- paste(base,endpoint, sep="/")

post_package = POST(call_post_package, body=list("json-ld"=json_file,      
        data=upload_file("your-directory/your-file","text/csv")),
        add_headers(Authorization=header_authorization, 
                    "Content-Type"="multipart/form-data"))

Review the results

content(post_package)$viewUrl
[1] "https://data-dev.ess-dive.lbl.gov/view/ess-dive-XXXXXXXXXXXX-20190621T175431086775"

Search for Datasets

Anyone can search for public datasets on ESS-DIVE using the Dataset API. If you are registered to submit data, you can also search for your private datasets. Query your dataset searches by defining parameters.

Limited dataset metadata are returned in the response of this call. Additionally, this call cannot be used to download data files. To look up all dataset metadata and download data files, use the API to Download Dataset Metadata.

👉 Review parameters and responses for this call in the Dataset API technical documentation.

The following lines of code will get the list of dataset metadata that you have permissions to edit. This will return the most recent 25 records. If the results contain more than 25 packages, use the row_start and page_size query parameters to page through the results.

# Call GET /packages
call_get_packages <- paste(base,endpoint, sep="/")
get_packages = GET(call_get_packages,
       add_headers(Authorization=header_authorization))

Transform the result into a data frame. (Ignore the warning)

get_packages_text <- content(get_packages, "text")
get_packages_json <- fromJSON(get_packages_text)
get_packages_df <- as.data.frame(get_packages_json)

Check the errors and view the data frame on success

# Check for errors
if(!http_error(post_package) ){
  # Print the returned columns
  print(colnames(get_packages_df))

  # print the ESS-DIVE Ids
  print(get_packages_df['result.id'])

  # Iterator over the dataset column and print the data package name
  for ( d in get_packages_df['result.dataset']) { print(d['name']) }
}else {
  http_status(post_package)
}

Download Dataset Metadata

Anyone can search for individual public datasets on ESS-DIVE using the Dataset API. If you are registered to submit data, you can also download your private dataset metadata.

The response for this call will return all dataset metadata and attached data files. Metadata and data files can then be downloaded. If you'd like to look up the dataset upload date, last modified date, or dataset access status, use the API to Search for Datasets.

👉 Review responses for this call in the Dataset API technical documentation.

The following lines of code will get the metadata for a single dataset that you have permissions to edit.

id <- "<Place an ESS-DIVE identifier here>"
call_get_package <- paste(base,endpoint,id, sep="/")
get_package = GET(call_get_package,
    add_headers(Authorization=header_authorization))

Transform the result into a data frame. (Ignore the warning message)

get_package_text <- content(get_package, "text")
get_package_json <- fromJSON(get_package_text)

Check for errors and view the data frame on success

# Check for errors
if(!http_error(post_package) ){
  print(get_package_json)
}else {
  http_status(post_package)
}

Update a Dataset

It is possible to both update the metadata and data of an existing dataset. The following update scenarios are possible :

• update metadata only

• replace/add files only

• both update metadata and replace/add files.

These examples will demonstrate both updating metadata and adding new files to the dataset created in previous sections.

Metadata Only

Use the PUT function to update the metadata of a dataset. This example updates the name of a dataset.

call_put_package <- paste(base,endpoint,get_package_json$id, sep="/")

put_package = PUT(call_put_package,
                  body = "{ \"name\": \"My Tutorial Title\" }",
                  add_headers(Authorization=header_authorization,
                      "Content-Type"="application/json"))

NOTE: The body argument is a string not a file.

Transform the result into a data frame. (Ignore the warning message)

put_package_text <- content(put_package, "text")
put_package_json <- fromJSON(put_package_text)

Check the results for the changed metadata attribute

# Check for errors
if(!http_error(put_package) ){
  attributes(put_package_json)
  put_package_json$viewUrl
  put_package_json$dataset$name
}else {
  http_status(put_package)
  print(put_package_text)
}

[1] "Data Package updated successfully."
NULL
[1] "https://data-dev.ess-dive.lbl.gov/view/ess-dive-xxx-20190621T185438900719"
[1] "My Tutorial Title"

Metadata plus new data file

Use the PUT function to update a dataset. This example updates the date published to 2019 of a dataset and adds a new data file.

call_put_package <- paste(base,endpoint,put_package_json$id, sep="/")
put_package_data = PUT(call_put_package, 
           body=list("json-ld"="{ \"datePublished\": \"2019\" }",
           data=upload_file("your-directory/your-file","text/csv")),
           add_headers(Authorization=header_authorization,
                     "Content-Type"="multipart/form-data"))

NOTE: The body argument is a string not a file.

Transform the result into a data frame. (Ignore the warning message)

put_package_data_text <- content(put_package_data, "text")
put_package_data_json <- fromJSON(put_package_data_text)

Check the results for the changed metadata attribute and newly uploaded file

# Check for errors
if(!http_error(put_package_data) ){
  attributes(put_package_data_json)
  print(put_package_data_json$detail)
  print(put_package_data_json$errors)
  print(put_package_data_json$viewUrl)
  print(put_package_data_json$dataset$datePublished)
  print(put_package_data_json$dataset$distribution)
}else {
  http_status(put_package_data)
  print(put_package_data_text)
}

[1] "Data Package updated successfully."
NULL
[1] "https://data-dev.ess-dive.lbl.gov/view/ess-dive-XXXX-20190621T191953176893"
[1] "2019"
               name encodingFormat
1 <your first file> text/csv
2 <your second file> text/csv

Check for errors and view the data frame on success

# Check for errors
if(!http_error(post_package) ){
  print(get_package_json)
}else {
  http_status(post_package)
}

Troubleshooting

 results = content(post_package)
>  attributes(results)$names
[1] "detail"
>  results$detail
[1] "You do not have authorized access"
>  results$errors
NULL
>  results$viewUrl
NULL

This error message indicates your token is either incorrect or expired. Please follow the instructions on the ESS-DIVE Dataset API page to retrieve a new token.

Error: 'C:\Users\Person\Desktop\API\API_Tutorials.json' does not exist.

This error message indicates the file entered was not found. This could be because you are searching in the wrong directory or because you misrepresented the file name.