R Example
This page provides coding examples for retrieving a list of submitted datasets and submitting dataset metadata for validation using R.
While learning the expected schema for the metadata use https://api-sandbox.ess-dive.lbl.gov, as this is the domain shown in the examples in this documentation. Once you've familiarized yourself with ESS-DIVE's metadata and dataset schema, use our production domain https://api.ess-dive.lbl.gov/ to submit datasets to ESS-DIVE for publishing and review.
For additional information about the API, review the documentation at https://api-sandbox.ess-dive.lbl.gov.
ESS-DIVE Test API URL: https://api-sandbox.ess-dive.lbl.gov
ESS-DIVE Production API URL: https://api.ess-dive.lbl.gov/
Help Desk: [email protected]
Install the following R packages
install.packages("httr")
install.packages("jsonlite")
install.packages("readr")
#Require the package so you can use it
require("jsonlite")
require("httr")
library(readr)
Setup the Dataset API information
token <- "<Enter your authentication token here>"
base <- "https://api-sandbox.ess-dive.lbl.gov"
header_authorization <- paste("bearer",token, sep=" ")
endpoint <- "packages"
Check that your token is up-to-date; it expires after a few days
After creating metadata, you have the option to submit a dataset with only metadata or to submit a dataset with both metadata and data files.
The following lines of code validate JSON-LD metadata for a single dataset. The example provided is from the ESS-DIVE sandbox site. (See https://data-sandbox.ess-dive.lbl.gov/#view/doi:10.3334/CDIAC/spruce.001).
Due to R complex JSON-LD support limitations, you need to create a text file of your JSON-LD and add it’s directory in the following read_file function.
To make sure your file is properly saved in the JSON-LD format, consider using the Atom text editor (https://atom.io)
json_file <- read_file("~/directory/to/your/jsonld/file")
Prepare and submit the metadata for validation
Make sure to enter your own file path.
E.g.
C:\\User\\Files\API_Tutorial.json for Windows & /Users/Files/API_Tutorial.json for Maccall_post_package <- paste(base,endpoint, sep="/")
post_package = POST(call_post_package,
body = json_file,
add_headers(Authorization=header_authorization,
"Content-Type"="application/json"))
Review the results
results = content(post_package)
attributes(results)
$names
[1] "id" "viewUrl" "detail" "errors" "dataset"
results$detail
[1] "Data Package created successfully."
results$errors
NULL
results$viewUrl
[1] "https://data-dev.ess-dive.lbl.gov/view/ess-dive-XXXXXXXXXXXX-20190621T175431086775"
To submit the metadata and data files, create a folder and add your data files to it then execute the following code:
call_post_package <- paste(base,endpoint, sep="/")
post_package = POST(call_post_package, body=list("json-ld"=json_file,
data=upload_file("your-directory/your-file","text/csv")),
add_headers(Authorization=header_authorization,
"Content-Type"="multipart/form-data"))
Review the results
content(post_package)$viewUrl
[1] "https://data-dev.ess-dive.lbl.gov/view/ess-dive-XXXXXXXXXXXX-20190621T175431086775"
The following lines of code will get the list of dataset metadata that you have permissions to edit. This will return the most recent 25 records. If you have access to more than 25 packages, use the
row_start and page_size query parameters to page through the results.# Call GET /packages
call_get_packages <- paste(base,endpoint, sep="/")
get_packages = GET(call_get_packages,
add_headers(Authorization=header_authorization))
Transform the result into a data frame. (Ignore the warning)
get_packages_text <- content(get_packages, "text")
get_packages_json <- fromJSON(get_packages_text)
get_packages_df <- as.data.frame(get_packages_json)
Check the errors and view the data frame on success
# Check for errors
if(!http_error(post_package) ){
# Print the returned columns
print(colnames(get_packages_df))
# print the ESS-DIVE Ids
print(get_packages_df['result.id'])
# Iterator over the dataset column and print the data package name
for ( d in get_packages_df['result.dataset']) { print(d['name']) }
}else {
http_status(post_package)
}
The following lines of code will get the metadata for a single dataset that you have permissions to edit.
id <- "<Place an ESS-DIVE identifier here>"
call_get_package <- paste(base,endpoint,id, sep="/")
get_package = GET(call_get_package,
add_headers(Authorization=header_authorization))
Transform the result into a data frame. (Ignore the warning message)
get_package_text <- content(get_package, "text")
get_package_json <- fromJSON(get_package_text)
Check for errors and view the data frame on success
# Check for errors
if(!http_error(post_package) ){
print(get_package_json)
}else {
http_status(post_package)
}
It is possible to both update the metadata and data of an existing dataset. The following update scenarios are possible :
- update metadata only
- replace/add files only
- both update metadata and replace/add files.
These examples will demonstrate both updating metadata and adding new files to the dataset created in previous sections.
Use the PUT function to update the metadata of a dataset. This example updates the name of a dataset.
call_put_package <- paste(base,endpoint,get_package_json$id, sep="/")
put_package = PUT(call_put_package,
body = "{ \"name\": \"My Tutorial Title\" }",
add_headers(Authorization=header_authorization,
"Content-Type"="application/json"))
NOTE: The body argument is a string not a file.
Transform the result into a data frame. (Ignore the warning message)
put_package_text <- content(put_package, "text")
put_package_json <- fromJSON(put_package_text)
Check the results for the changed metadata attribute
# Check for errors
if(!http_error(put_package) ){
attributes(put_package_json)
put_package_json$viewUrl
put_package_json$dataset$name
}else {
http_status(put_package)
print(put_package_text)
}
[1] "Data Package updated successfully."
NULL
[1] "https://data-dev.ess-dive.lbl.gov/view/ess-dive-xxx-20190621T185438900719"
[1] "My Tutorial Title"
Use the PUT function to update a dataset. This example updates the date published to 2019 of a dataset and adds a new data file.
call_put_package <- paste(base,endpoint,put_package_json$id, sep="/")
put_package_data = PUT(call_put_package,
body=list("json-ld"="{ \"datePublished\": \"2019\" }",
data=upload_file("your-directory/your-file","text/csv")),
add_headers(Authorization=header_authorization,
"Content-Type"="multipart/form-data"))
NOTE: The body argument is a string not a file.
Transform the result into a data frame. (Ignore the warning message)
put_package_data_text <- content(put_package_data, "text")
put_package_data_json <- fromJSON(put_package_data_text)
Check the results for the changed metadata attribute and newly uploaded file
# Check for errors
if(!http_error(put_package_data) ){
attributes(put_package_data_json)
print(put_package_data_json$detail)
print(put_package_data_json$errors)
print(put_package_data_json$viewUrl)
print(put_package_data_json$dataset$datePublished)
print(put_package_data_json$dataset$distribution)
}else {
http_status(put_package_data)
print(put_package_data_text)
}
[1] "Data Package updated successfully."
NULL
[1] "https://data-dev.ess-dive.lbl.gov/view/ess-dive-XXXX-20190621T191953176893"
[1] "2019"
name encodingFormat
1 <your first file> text/csv
2 <your second file> text/csv
Check for errors and view the data frame on success
# Check for errors
if(!http_error(post_package) ){
print(get_package_json)
}else {
http_status(post_package)
}
results = content(post_package)
> attributes(results)$names
[1] "detail"
> results$detail
[1] "You do not have authorized access"
> results$errors
NULL
> results$viewUrl
NULL
This error message indicates your token is either incorrect or expired. Please follow the instructions on the ESS-DIVE Dataset API page to retrieve a new token.
Error: 'C:\Users\Person\Desktop\API\API_Tutorials.json' does not exist.
This error message indicates the file entered was not found. This could be because you are searching in the wrong directory or because you misrepresented the file name.
Last modified 6mo ago