Java Example

This page provides coding examples for retrieving a list of submitted dataset and submitting dataset metadata for validation using Java

While learning the expected schema for the metadata use https://api-sandbox.ess-dive.lbl.gov, as this is the domain shown in the examples in this documentation. Once you've familiarized yourself with ESS-DIVE's metadata and dataset schema, use our production domain https://api.ess-dive.lbl.gov/ to submit datasets to ESS-DIVE for publishing and review.

For additional information about the API, review the documentation at https://api-sandbox.ess-dive.lbl.gov.

ESS-DIVE Test API URL: https://api-sandbox.ess-dive.lbl.gov ESS-DIVE Production API URL: https://api.ess-dive.lbl.gov/ Help Desk: ess-dive-support@lbl.gov

Contents

Setup

To be able to do the setup correctly, you need a few prerequisites installed on your machine which are unzip, java, javac and curl.

Inside your project directory create a java file named essdive.java and a directory lib. The following are the Linux commands to create the file and directory:

touch essdive.java
mkdir lib
cd lib

Add http client, http core and commons logging from Apache HTTPComponents libraries version 4.5.6 and json_simple-1.1 into your lib directory by downloading its zip file, extracting it and moving it inside lib.

curl -O http://ftp.wayne.edu/apache//httpcomponents/httpclient/binary/httpcomponents-client-4.5.6-bin.zip

curl -O https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/json-simple/json_simple-1.1.jar

curl -O http://mirror.cc.columbia.edu/pub/software/apache//commons/io/binaries/commons-io-2.6-bin.tar.gz

tar -xvzf commons-io-2.6-bin.tar.gz
mv commons-io-2.6/commons-io-2.6.jar .


unzip httpcomponents-client-4.5.6-bin.zip 
mv httpcomponents-client-4.5.6/lib/* .

Open essdive.java document created earlier using any text editor and add the following code to start importing the libraries needed for your java code:

import java.io.File;
import java.io.IOException;

//JSON imports
import org.apache.http.HttpEntity;
import org.json.simple.JSONArray;
import org.json.simple.JSONObject;

//Apache http imports
import org.apache.commons.io.FileUtils;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.methods.HttpPut;
import org.apache.http.HttpResponse;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.entity.StringEntity;
import org.apache.http.entity.ContentType;
import org.apache.http.entity.mime.FormBodyPart;
import org.apache.http.entity.mime.FormBodyPartBuilder;
import org.apache.http.entity.mime.HttpMultipartMode;
import org.apache.http.util.EntityUtils;
import org.apache.http.entity.mime.MultipartEntityBuilder;
import org.apache.http.entity.mime.content.StringBody;
import org.apache.http.client.methods.HttpGet;

Definition of configuration variables

Inside the main function you should define your configuration variables:

Note: You will have to close the class and main blocks by adding two closing curly braces at the end of the document.

public class essdive {

public static void main(String[] args) {
 //Configuration variables
  String token = "<Enter your authorization token here>";
  String base = "https://api-sandbox.ess-dive.lbl.gov/";
  String header_authorization = "bearer " + token;
  String endpoint = "packages";

Then define your utilities objects:

         //Utilities objects 
        CloseableHttpClient httpClient = HttpClientBuilder.create().build();

Submit a Dataset

After creating metadata, you have the option to submit a dataset with only metadata or to submit a dataset with both metadata and data files.

Create Metadata

The following lines of code validate JSON-LD metadata for a single dataset. The example provided is from the ESS-DIVE sandbox site. (See https://data-sandbox.ess-dive.lbl.gov/#view/doi:10.3334/CDIAC/spruce.001).

Setup the JSON definitions to build your JSON_LD.

 //JSON objects variables
  JSONObject provider_spruce_json = new JSONObject();
  JSONObject member = new JSONObject();
  JSONObject funder = new JSONObject();
  JSONObject temporalCoverage = new JSONObject();
  JSONObject editor = new JSONObject();
  JSONObject spatial_coverage_json = new JSONObject();
  JSONObject primary_Creator = new JSONObject();
  JSONObject secondary_Creator = new JSONObject();
  JSONObject geo_northwest = new JSONObject();
  JSONObject geo_southeast = new JSONObject();
  JSONObject JSON_LD = new JSONObject();
  
  JSONArray creators_json = new JSONArray();
  JSONArray spatial_coverage_array = new JSONArray();
  JSONArray geo = new JSONArray();
  JSONArray measurementTechnique = new JSONArray();
  JSONArray JSON_LD_Description = new JSONArray();
  JSONArray keywords = new JSONArray();
  JSONArray variableMeasured = new JSONArray();

Now fill the details about the “provider”. This is the details about the project. The project will be listed as the publisher in the citation.

  // JSON_LD member assignment
  member.put("@id","http://orcid.org/0000-0001-7293-3561");
  member.put("givenName","Paul J");
  member.put("familyName","Hanson");
  member.put("email","hansonpj@ornl.gov");
  member.put("jobTitle","Principal Investigator");
 
  // JSON_LD provider spruce assignment
  provider_spruce_json.put("name","SPRUCE");
  provider_spruce_json.put("member",member);

Prepare the dataset authors in the order that you would like them to appear in the citation. Please add the ORCID for all authors, especially the first author, if possible. 

  // JSON_LD primary creator assignment
  primary_Creator.put("@id","http://orcid.org/0000-0001-7293-3561");
  primary_Creator.put("givenName","Paul J");
  primary_Creator.put("familyName","Hanson");
  primary_Creator.put("affiliation","Oak Ridge National Laboratory");
  primary_Creator.put("email","hansonpj@ornl.gov");
 
  // JSON_LD secondary creator assignment
  secondary_Creator.put("givenName","Jeffrey");
  secondary_Creator.put("familyName","Riggs");
  secondary_Creator.put("affiliation","Oak Ridge National Laboratory");
 
  // Define as many creators as you need into newer JSON Objects and add them to the creators_json_array
  creators_json.add(primary_Creator);
  creators_json.add(secondary_Creator);

Initialize JSON_LD strings

  // JSON_LD Strings arrays
  variableMeasured.add("EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC TEMPERATURE > SURFACE TEMPERATURE > AIR TEMPERATURE");
  variableMeasured.add("EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC WATER VAPOR > WATER VAPOR INDICATORS > HUMIDITY > RELATIVE HUMIDITY");
  variableMeasured.add("EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC PRESSURE > SEA LEVEL PRESSURE");
  variableMeasured.add("EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC TEMPERATURE > SURFACE TEMPERATURE > DEW POINT TEMPERATURE > DEWPOINT DEPRESSION");
  variableMeasured.add("EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC WINDS > SURFACE WINDS > WIND SPEED");
  variableMeasured.add("EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC WINDS > SURFACE WINDS > WIND DIRECTION");
  variableMeasured.add("EARTH SCIENCE > BIOSPHERE > VEGETATION > PHOTOSYNTHETICALLY ACTIVE RADIATION");
  variableMeasured.add("EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC RADIATION > NET RADIATION");
  variableMeasured.add("EARTH SCIENCE > LAND SURFACE > SURFACE RADIATIVE PROPERTIES > ALBEDO");
  variableMeasured.add("EARTH SCIENCE > LAND SURFACE > SOILS > SOIL TEMPERATURE");
  variableMeasured.add("Precipitation (Total)");
  variableMeasured.add("Irradiance");
  variableMeasured.add("Groundwater Temperature");
  variableMeasured.add("Groundwater Level");
  variableMeasured.add("Volumetric Water Content");
  variableMeasured.add("surface_albedo");
  measurementTechnique.add("The stations are equipped with standard sensors for measuring meteorological parameters, solar radiation, soil temperature and moisture, and groundwater temperature and elevation. Note that some sensor locations are relative to nearby vegetation and bog microtopographic features (i.e., hollows and hummocks). See Table 1 in the attached pdf (SPRUCE_EM_DATA_2010_2016_20170620) for a list of measurements and further details. Sensors and data loggers were initially installed and became operational in June, July, and August of 2010. Additional sensors were added in September 2011. Station 3 was removed from service on May 12, 2014.");
  measurementTechnique.add("These data are considered at Quality Level 1. Level 1 indicates an internally consistent data product that has been subjected to quality checks and data management procedures. Established calibration procedures were followed.");
  JSON_LD_Description.add("This data set reports selected ambient environmental monitoring data from the S1 bog in Minnesota for the period June 2010 through December 2016. Measurements of the environmental conditions at these stations will serve as a pre-treatment baseline for experimental treatments and provide driver data for future modeling activities.");
  JSON_LD_Description.add("The site is the S1 bog, a Picea mariana [black spruce] - Sphagnum spp. bog forest in northern Minnesota, 40 km north of Grand Rapids, in the USDA Forest Service Marcell Experimental Forest (MEF). There are/were three monitoring sites located in the bog: Stations 1 and 2 are co-located at the southern end of the bog and Station 3 is located north central and adjacent to an existing U.S. Forest Service monitoring well.");
  JSON_LD_Description.add("There are eight data files with selected results of ambient environmental monitoring in the S1 bog for the period June 2010 through December 2016. One file has the ");
  JSON_LD_Description.add("other seven have the available data for a given calendar year. Not all measurements started in June 2010 and EM3 measurements ended in May 2014.");
  JSON_LD_Description.add("Further details about the data package are in the attached pdf file (SPRUCE_EM_DATA_2010_2016_20170620).");
  keywords.add("EARTH SCIENCE > BIOSPHERE > VEGETATION");
  keywords.add("Climate Change");

Add nested information in JSON_objects.

  // JSON_LD spatial coverage assignment
  spatial_coverage_json.put("description","Site ID: S1 Bog Site name: S1 Bog, Marcell Experimental Forest Description: The site is the 8.1-ha S1 bog, a Picea mariana [black spruce] - Sphagnum spp. ombrotrophic bog forest in northern Minnesota, 40 km north of Grand Rapids, in the USDA Forest Service Marcell Experimental Forest (MEF). The S1 bog was harvested in successive strip cuts in 1969 and 1974 and the cut areas were allowed to naturally regenerate. Stations 1 and 2 are located in a 1974 strip that is characterized by a medium density of 3-5 meter black spruce and larch trees with an open canopy. The area was suitable for siting a monitoring station for representative meteorological conditions on the S1 bog. Station 3 is located in a 1969 harvest strip that is characterized by a higher density of 3-5 meter black spruce and larch trees with a generally closed canopy. Measurements at this station represent conditions in the surrounding stand. Site Photographs are in the attached document");
  spatial_coverage_json.put("geo", geo);

  spatial_coverage_array.add(spatial_coverage_json);
 
  // JSON_LD funder assignment
  funder.put("@id", "http://dx.doi.org/10.13039/100006206");
  funder.put("name", "U.S. DOE > Office of Science > Biological and Environmental Research (BER)");
 
  // JSON_LD temporalCoverage assignment
  temporalCoverage.put("startDate","2010-07-16");
  temporalCoverage.put("endDate","2016-12-31");
  
  // JSON_LD editor assignment
  editor.put("@id", "http://orcid.org/0000-0001-7293-3561");
  editor.put("givenName", "Paul J");
  editor.put("familyName", "Hanson");
  editor.put("email", "hansonpj@ornl.gov");
 
  // JSON_LD geo variables assignments
  geo_northwest.put("name","Northwest");
  geo_northwest.put("latitude",47.50285);
  geo_northwest.put("longitude",-93.48283);
 
  geo_southeast.put("name","Southeast");
  geo_southeast.put("latitude",47.50285);
  geo_southeast.put("longitude",-93.48283);
 
  geo.add(geo_northwest);
  geo.add(geo_southeast);

Create the rest of the JSON-LD object

  // Main JSON_LD
  JSON_LD.put("@context","http://schema.org/");
  JSON_LD.put("@type","Dataset");
  JSON_LD.put("@id","http://dx.doi.org/10.3334/CDIAC/spruce.001");
  JSON_LD.put("name","SPRUCE S1 Bog Environmental Monitoring Data: 2010-2016");
  JSON_LD.put("description",JSON_LD_Description);
  JSON_LD.put("creator",creators_json);
  JSON_LD.put("datePublished","2015");
  JSON_LD.put("keywords",keywords);
  JSON_LD.put("variableMeasured",variableMeasured);
  JSON_LD.put("license","http://creativecommons.org/licenses/by/4.0/");
  JSON_LD.put("spatialCoverage",spatial_coverage_array);
  JSON_LD.put("funder",funder);
  JSON_LD.put("temporalCoverage",temporalCoverage);
  JSON_LD.put("editor",editor);
  JSON_LD.put("provider", provider_spruce_json);
  JSON_LD.put("measurementTechnique",measurementTechnique);

Metadata Only

Submit the JSON-LD object with the Dataset API

  try{
    String url = base + endpoint;
    HttpPost request = new HttpPost(url);
    StringEntity params = new StringEntity(JSON_LD.toString()); //Setting the JSON-LD Object to the request params
    request.addHeader("content-type", "application/json");
    request.addHeader("Authorization", header_authorization);
    request.setEntity(params);
  
    HttpResponse response;
    response = httpClient.execute(request);
    HttpEntity entity = response.getEntity();
    String responseString = EntityUtils.toString(entity, "UTF-8");

    if(response.getStatusLine().getStatusCode() == 201){
      System.out.println(response.toString());
      System.out.println(responseString);
    } else {
      System.out.println(response.getStatusLine().getReasonPhrase());
      System.out.println(responseString);
    }
  } catch (Exception ex) {
    System.out.print(ex.getMessage().toString());
  }

Metadata and Data

Submit the JSON-LD object and upload files with the Dataset API:

  try{
    String url = base + endpoint;
    HttpPost uploadFile = new HttpPost(url);
    uploadFile.addHeader("Authorization", header_authorization);
    File file_to_upload = new File("/directory-to-your-file/file");
    
    String content = FileUtils.readFileToString(file_to_upload, "UTF-8");
    
    FormBodyPart bodyPart = FormBodyPartBuilder.create()
    .setName("files")
    .addField("Content-Disposition", "form-data; name=\"data\"; filename=\"<your-file-name>\"")
    .setBody(new StringBody(content, ContentType.TEXT_PLAIN))
    .build();
    
    MultipartEntityBuilder builder = MultipartEntityBuilder.create()
    .setMode(HttpMultipartMode.BROWSER_COMPATIBLE)
    .setContentType(ContentType.MULTIPART_FORM_DATA);
    
    builder.addPart("json-ld", new StringBody(JSON_LD.toString(), ContentType.TEXT_PLAIN));
    builder.addPart(bodyPart);
    
    uploadFile.setEntity(builder.build());
    
    CloseableHttpResponse response = httpClient.execute(uploadFile);
    HttpEntity responseEntity = response.getEntity();
    
    response = httpClient.execute(uploadFile);
    HttpEntity entity = response.getEntity();
    String responseString = EntityUtils.toString(responseEntity, "UTF-8");
    
    if(response.getStatusLine().getStatusCode() == 201){
      System.out.println(response.toString());
      System.out.println(responseString);
    } else {
      System.out.println(response.getStatusLine().getReasonPhrase());
      System.out.println(responseString);
    }
  } catch (Exception ex) {
    System.out.print(ex.getMessage().toString());
  }

Search for Datasets

Anyone can search for public datasets on ESS-DIVE using the Dataset API. If you are registered to submit data, you can also search for your private datasets. Query your dataset searches by defining parameters.

Limited dataset metadata are returned in the response of this call. Additionally, this call cannot be used to download data files. To look up all dataset metadata and download data files, use the API to Download Dataset Metadata.

The following lines of code will get the list of dataset metadata that you have permissions to edit. This will return the most recent 25 records. If the results contain more than 25 packages, use the row_start and page_size query parameters to page through the results.

  try{
    String url = base + endpoint;
    HttpGet request = new HttpGet(url);
    StringEntity params = new StringEntity(JSON_LD.toString()); //Setting the JSON-LD Object to the request params
    request.addHeader("content-type", "application/json");
    request.addHeader("Authorization", header_authorization);
    
    HttpResponse response;
    response = httpClient.execute(request);
    
    HttpEntity entity = response.getEntity();
    String responseString = EntityUtils.toString(entity, "UTF-8");
    
    if(response.getStatusLine().getStatusCode() == 200){
      System.out.println(response.toString());
      System.out.println(responseString);
    } else {
      System.out.println(response.getStatusLine().getReasonPhrase());
      System.out.println(response.toString());
      System.out.println(responseString);
    }
  } catch (Exception ex) {
    System.out.print(ex.getMessage().toString());
  }

Download Dataset Metadata

Anyone can search for individual public datasets on ESS-DIVE using the Dataset API. If you are registered to submit data, you can also download your private dataset metadata.

The response for this call will return all dataset metadata and attached data files. Metadata and data files can then be downloaded. If you'd like to look up the dataset upload date, last modified date, or dataset access status, use the API to Search for Datasets.

The following lines of code will get the metadata for a single dataset that you have permission to edit.

  try{
    String id = "<Enter an ESS-DIVE Identifier here>";
    String url = base + endpoint + File.separator + id;
    HttpGet request = new HttpGet(url);
    StringEntity params = new StringEntity(JSON_LD.toString()); //Setting the JSON-LD Object to the request params
    request.addHeader("content-type", "application/json");
    request.addHeader("Authorization", header_authorization);

    HttpResponse response;
    response = httpClient.execute(request);

    HttpEntity entity = response.getEntity();
    String responseString = EntityUtils.toString(entity, "UTF-8");

    if(response.getStatusLine().getStatusCode() == 200){
      System.out.println(response.toString());
      System.out.println(responseString);
    } else {
      System.out.println(response.getStatusLine().getReasonPhrase());
      System.out.println(response.toString());
      System.out.println(responseString);
    }
  } catch (Exception ex) {
    System.out.print(ex.getMessage().toString());
  }

At the end of the document, don’t forget to close the class and main function blocks by adding two curly braces if you hadn’t done that already:

Update a Dataset

It is possible to both update the metadata and data of an existing dataset. The following update scenarios are possible

  • update metadata only

  • replace/add files only

  • both update metadata and replace/add files.

These examples will demonstrate both updating metadata and adding new files to the dataset created in previous sections.

Metadata Only

Use the PUT function to update the metadata of a dataset. This example updates the name of a dataset. 

  String id = "<Enter-your-dataset-id>";
  JSONObject JSON_LD_update = new JSONObject();
  JSON_LD_update.put("name","Updated dataset title");        
     try{
          String url = base + endpoint + "/" + id;
          HttpPut request = new HttpPut(url);
          StringEntity params = new StringEntity(JSON_LD_update.toString()); //Setting the JSON-LD Object to the request params
          request.addHeader("content-type", "application/json");
          request.addHeader("Authorization", header_authorization);
          request.setEntity(params);
          
          HttpResponse response;
          response = httpClient.execute(request);
          HttpEntity entity = response.getEntity();
          String responseString = EntityUtils.toString(entity, "UTF-8");
          
          System.out.println(response.getStatusLine().getStatusCode());
          System.out.println(response.toString());
          System.out.println(response.getStatusLine());
          
          if(response.getStatusLine().getStatusCode() == 200){
               System.out.println(response.toString());
               System.out.println("Dataset updated");
               System.out.println(responseString);
          } else {
               System.out.println(response.getStatusLine().getReasonPhrase());
               System.out.println(responseString);
               }
     } catch (Exception ex) {
          System.out.print(ex.getMessage().toString());
     }

Metadata plus new data file

Use the PUT function to update a dataset. This example updates the date published to 2019 of a dataset and adds a new data file.

    String id = "<Enter-your-dataset-id>";
    JSONObject JSON_LD_update = new JSONObject();
    JSON_LD_update.put("name","Updated dataset");

    try{
        String url = base + endpoint + "/" + id;
        HttpPut uploadFile = new HttpPut(url);
        uploadFile.addHeader("Authorization", header_authorization);
        File file_to_upload = new File("/directory-to-your-file/file");
  
        String content = FileUtils.readFileToString(file_to_upload, "UTF-8");
        
        
        FormBodyPart bodyPart = FormBodyPartBuilder.create()                    
        .setName("files")
        .addField("Content-Disposition", "form-data; name=\"data\"; filename=\"<your-file-name>\"")
        .setBody(new StringBody(content, ContentType.TEXT_PLAIN))
        .build();

        MultipartEntityBuilder builder = MultipartEntityBuilder.create()
        .setMode(HttpMultipartMode.BROWSER_COMPATIBLE)
        .setContentType(ContentType.MULTIPART_FORM_DATA);
        
        builder.addPart("json-ld", new StringBody(JSON_LD_update.toString(), ContentType.TEXT_PLAIN));
        builder.addPart(bodyPart);


        uploadFile.setEntity(builder.build());
        
        CloseableHttpResponse response = httpClient.execute(uploadFile);
        HttpEntity responseEntity = response.getEntity();

        String responseString = EntityUtils.toString(responseEntity, "UTF-8");

Check the results for the changed metadata attribute and newly uploaded file

    String id = "<Enter-your-dataset-id>";
    JSONObject JSON_LD_update = new JSONObject();
    JSON_LD_update.put("name","Updated dataset");

    try{
        String url = base + endpoint + "/" + id;
        HttpPut uploadFile = new HttpPut(url);
        uploadFile.addHeader("Authorization", header_authorization);
        File file_to_upload = new File("/directory-to-your-file/file");
  
        String content = FileUtils.readFileToString(file_to_upload, "UTF-8");
        
        
        FormBodyPart bodyPart = FormBodyPartBuilder.create()                    
        .setName("files")
        .addField("Content-Disposition", "form-data; name=\"data\"; filename=\"<your-file-name>\"")
        .setBody(new StringBody(content, ContentType.TEXT_PLAIN))
        .build();

        MultipartEntityBuilder builder = MultipartEntityBuilder.create()
        .setMode(HttpMultipartMode.BROWSER_COMPATIBLE)
        .setContentType(ContentType.MULTIPART_FORM_DATA);
        
        builder.addPart("json-ld", new StringBody(JSON_LD_update.toString(), ContentType.TEXT_PLAIN));
        builder.addPart(bodyPart);


        uploadFile.setEntity(builder.build());
        
        CloseableHttpResponse response = httpClient.execute(uploadFile);
        HttpEntity responseEntity = response.getEntity();

        String responseString = EntityUtils.toString(responseEntity, "UTF-8");
        // Review the results
        System.out.println(responseString);
    } catch (Exception ex) {
        System.out.print(ex.getMessage().toString());
    }
    if(response.getStatusLine().getStatusCode() == 200){
        System.out.println(response.toString());
        System.out.println("package updated");
        System.out.println(responseString);
    } else {
        System.out.println(response.getStatusLine().getReasonPhrase());
        System.out.println(responseString);
    }
    } catch (Exception ex) {
        System.out.print(ex.getMessage().toString());
    }

At the end of the document, don’t forget to close the class and main function blocks by adding two curly braces (}}) if you hadn’t done that already!

To compile the code and run it, make sure you’re on the parent directory where the essdive.java file is not inside the lib folder.

Now assuming Java is already installed on your machine, we will start by compiling the java code you wrote using the following terminal command:

javac -cp .:"lib/*" essdive.java

This will create a new file that has the compiled code where it can run using the following command:

java -cp .:"lib/*" essdive

PreviousPython ExampleNextAPI Updates and Changes

Last updated