ESS-DIVE Documentation
  • ESS-DIVE Documentation
  • Frequently Asked Questions
  • Submit Data
    • Get Started
      • Large Data Support
    • Register to Submit Data
    • Data Reporting Formats
    • Submit Data with Online Form
    • Submit Data with Dataset API
    • Link to External Data Sources
  • Publish Data
    • Check Dataset Metadata Quality
    • Dataset and DOI Status Badges
    • Review Cycle and Criteria
      • Metadata Requirements
      • Reporting Format Requirements
    • Publish your Dataset
      • Request Publication
      • Reserve DOI Before Publication
      • Publish with Existing DOI
      • Troubleshooting
  • Manage Data
    • Register Dataset Citations
    • Create Data Portals
      • How to Create & Publish Portals
    • Share Data Permissions
      • Share Datasets
      • Share Portals
    • Manage Project Data
      • Project Data Managers
      • Project Information
      • Project Teams
  • Search & Download Data
    • Search for Data
    • Download Data
    • Access Data Portals
    • Search with Dataset API
      • Code Examples
    • Search with Deep Dive API
      • How to Query Data
  • Programmatic Tools
    • ESS-DIVE Dataset API
      • R Example
      • Python Example
      • Java Example
      • API Updates and Changes
    • Globus Data Transfer Service
      • Setup Globus
      • Upload Data with Globus
      • FAQs
Powered by GitBook
On this page
  • 1. Create ESS-DIVE Dataset
  • 2. Find your ESS-DIVE Globus Collection
  • 3. Transfer Files to ESS-DIVE Globus Collection
  • 4. Move Data from Globus to ESS-DIVE
  • 5. Review and Publish ESS-DIVE Dataset
  1. Programmatic Tools
  2. Globus Data Transfer Service

Upload Data with Globus

Learn how to transfer your files to Globus from Globus Connect Personal or an existing Globus endpoint.

PreviousSetup GlobusNextFAQs

Last updated 8 months ago

Please note that once your large data is made accessible on NERSC via Globus, neither you nor ESS-DIVE will be able to edit the files. They can either be transferred elsewhere or deleted.

1. Create ESS-DIVE Dataset

Before we can transfer data to ESS-DIVE using Globus, it is necessary to create a dataset on ESS-DIVE and get the metadata ready for publication. It is recommended to complete as much metadata as possible before moving on to the next step.

  1. Navigate to , click the "Submit Data" button to start a new dataset. Fill out the minimum metadata and click the “Submit Dataset” button to create a private dataset.

    • Read more about creating datasets on our .

    • You can also use the to submit metadata to ESS-DIVE.

  2. Do not upload any data files to this dataset.

  3. Initiate a publication request to begin your Globus upload. Select "Manage Publication" from your dataset landing page, then click "Start Publication Process" in the pop-up window.

    • See complete instructions on our documentation page.

  4. Shortly after, you will receive an email that confirms your request has been received. Reply to this email to inform the ESS-DIVE Team that you need to upload your data files using Globus. If you did not receive a confirmation email, please .

  5. The ESS-DIVE Team will then reply to your message to continue the Globus upload process.

    • Tier 2 datasets: At this time, an inactive DOI will be minted for your dataset. The inactive DOI will be used at a later step in the Globus upload process and it will not become active until after your dataset is published.

2. Find your ESS-DIVE Globus Collection

The ESS-DIVE Team will create a Globus collection that will be managed by ESS-DIVE and it will be specific to the dataset you are publishing. This section will walk you through how to enable access to this collection, which will later allow you to transfer your data here.

  1. The ESS-DIVE Team will need the email or ORCID associated with your Globus “identity” (i.e. your Globus account ID) to enable shared access to the Globus collection we create. Reply to the confirmation email you received from ESS-DIVE in the previous step (“Create ESS-DIVE Dataset") and send your email or ORCID to the ESS-DIVE Team, whichever you used to setup your account.

  2. Wait for a reply from the ESS-DIVE Team to proceed. We will send you the name of your personalized Globus collection where you will upload your data. The collection name will be named after the inactive DOI minted for your dataset in the format of doi-10-15485-1813868.

  3. Verify that you now have access to your collection by navigating to the "File Manager". Select the “Collection” search bar, then click the “Shared with you” tab (Figure 2) and look for the Globus collection name provided via email by ESS-DIVE.

    If you do not see your ESS-DIVE collection here, then you are not connected to it. Please email the Support Team if this occurs.

3. Transfer Files to ESS-DIVE Globus Collection

  1. To begin transferring files to your shared Globus collection, go to the File Manager. If you have only one panel on screen, use the panel toggle in the top right to change your view at any time (Figure 3).

  2. Now, in the empty panel, select the “Collection” search bar. Navigate to the “Shared with You” tab (Figure 2). Select the personalized ESS-DIVE folder that we shared with you (in the format of doi-10-15485-1813868).

    • Note that this collection will be empty. At the top, a yellow banner will say "ESS-DIVE Ingest Endpoint" confirming that this is the correct collection.

  3. Now you’re ready to transfer your data. Select the files or folders from your collection that you’d like to upload and publish with your ESS-DIVE dataset (Figure 4).

  4. Once all your data is selected, hit the “Sync transfer...” button, the "Start" button, or drag your selected files to the shared ESS-DIVE collection (Figure 4).

  5. This will start a transfer job and you will see a green popup window on your screen confirming that the transfer has started (Figure 5). In many cases the transfer is quick, but if your transfer is taking a while, use the link from this popup window to check in on the status of the transfer or look in the Activity tab (Figure 6).

  6. You will know that the transfer job is complete when Globus sends you a confirmation email that the transfer finished. Congrats! You have completed the Globus portion of this process.

At this point, your files have not been uploaded or published to the ESS-DIVE repository. This is a temporary collection that the ESS-DIVE Team will use to stage your data for publication. In the next step, we will upload your data and publish it on ESS-DIVE.

4. Move Data from Globus to ESS-DIVE

This step is done by the ESS-DIVE Team. Your review will be necessary.

  1. Go to your inbox and reply to the ESS-DIVE Team that your data files have been transferred to your shared Globus collection and are ready for publication.

  2. Wait for a reply from the ESS-DIVE Team to proceed. Once we can access your data on the ESS-DIVE Globus collection, we will upload your data to Tier 1 or Tier 2. This process can take a few days depending on how large your files are. ESS-DIVE will email you after the upload is complete and provide a link to your data files either on Tier 1 or Tier 2.

  3. Review the data files in your dataset to make sure everything looks correct to you. Let the ESS-DIVE Team know if there are any issues.

  4. Reply to the ESS-DIVE Team to let us know you approve the file upload.

    • For data stored on Tier 2, ESS-DIVE will add external links in your dataset that direct end users to access your data via Tier 2.

At this point, the Globus upload process is complete! The final section before publication is to complete your dataset metadata and initiate a publication review. The publication process has been reiterated below for ease of reference.

5. Review and Publish ESS-DIVE Dataset

After your files are successfully uploaded to your dataset on Tier 1 or Tier 2 and you approve the files, ESS-DIVE will wait to hear confirmation from you that your metadata is complete and ready for publication.

  1. Respond to the ESS-DIVE Team letting them know that your metadata is ready for review.

  2. The ESS-DIVE Team will email you any revision requests, if necessary. Revise your metadata as requested.

  3. If there are no further revisions requested, your dataset will be approved for publication. Your dataset metadata will be published and the data you transfered to Globus will become publicly accessible via the ESS-DIVE repository.

Confirm your account information by checking your identity. To find your identity, go to Globus () and navigate to the “Settings” tab (Figure 1). There you will see your primary identity. Select the dropdown arrow and copy the information listed there.

Once you receive your ESS-DIVE Globus collection, to the Globus Web App.

If you are transferring data from your local desktop, make sure you have Globus Connect Personal installed (see page for instructions). If you’re using an existing public endpoint in Globus, make sure you have access to that endpoint. We will now move your data into your ESS-DIVE collection.

Select the “Collection” search bar and navigate to the “Your Collections” tab (see example in ). Then select your desktop collection or select an existing source endpoint if you will be using an existing one.

https://app.globus.org/
log in
Setup Globus
👉
👉
👉
ESS-DIVE
Web Form tutorial
Dataset API
Request Publication
contact our support team
Setup your Collection
Figure 1: Visit the settings page to look up your account identity information
Figure 2: Your ESS-DIVE Globus collection can be easily found in the file manager search by selecting the "Shared with You" tab
Figure 3: The panel view can alternate between displaying the left, right, or both panels.
Figure 4: Choose the collections to transfer data from and to, select your files, and start the transfer job using the intuitive buttons on-screen.
Figure 5: A confirmation message pops up on screen once you initiate a transfer
Figure 6: Use the Activity tab to check the status of your transfer