Upload Data with Globus

Learn how to transfer your files to Globus from Globus Connect Personal or an existing Globus endpoint.

Please note that once your large data is made accessible on NERSC via Globus, neither you nor ESS-DIVE will be able to edit the files. They can either be transferred elsewhere or deleted.

1. Create ESS-DIVE Dataset

Before we can transfer data to ESS-DIVE using Globus, it is necessary to create a dataset on ESS-DIVE and get the metadata ready for publication. It is recommended to complete as much metadata as possible before moving on to the next step.

1. Navigate to ESS-DIVE, click the "Submit Data" button to start a new dataset. Fill out the minimum metadata and click the “Submit Dataset” button to create your private dataset (read more about creating datasets on our Web Form tutorial).

   • 👉 You may also use the Dataset API to submit metadata to ESS-DIVE. Refer to our API documentation to learn more and see our python tutorial for submitting metadata only using the API as an example.
2. Complete your dataset metadata on ESS-DIVE using our Metadata Guide to help you create high-quality metadata. Do not upload any data files to this dataset. Then hit "Publish".

3. You will receive an email in your inbox shortly after confirming your request has been received. Reply to this email to inform the ESS-DIVE Team that you need to upload your data files using Globus. If you did not receive a confirmation email, please contact our support team.

4. The ESS-DIVE Team will then reply to your message to continue the Globus upload process. At this time, an inactive DOI will be minted for your dataset if you do not have a DOI already. This inactive DOI will be used at a later step in the Globus upload process and it will not become active until after your dataset metadata and files are published.

2. Get your ESS-DIVE Globus Collection

The ESS-DIVE Team will create a Globus collection that will be managed by ESS-DIVE and it will be specific to the dataset you are publishing. This section will walk you through how to enable access to this collection, which will later allow you to transfer your data here.

Figure 1: Visit the settings page to look up your account identity information

Figure 2: Your ESS-DIVE Globus collection can be easily found in the file manager search by selecting the "Shared with You" tab

1. The ESS-DIVE Team will use your Globus “identity” (i.e. your Globus account ID) as well as your email and ORCID to enable shared access to the Globus collection we create. To find your identity, go to Globus (https://app.globus.org/) and navigate to the “Settings” tab (Figure 1). There you will see your primary identity. Select the dropdown arrow and copy the information listed there (Figure 1). If you don't have a Globus account, please see the Setup Globus page.

   • Once you have your information copied, go back to the confirmation email you received from ESS-DIVE in the previous step (“Create ESS-DIVE Dataset") and send your identity information to the ESS-DIVE Team. If you setup your account with your ORCID, please send the ORCID associated with your account.

2. Wait for a reply from the ESS-DIVE Team to proceed. We will send you the name of your personalized Globus collection where you will upload your data. The collection name will be named after the inactive DOI minted for your dataset in the format of doi-10-15485-1813868.

3. Once you receive your ESS-DIVE Globus collection, log in to the Globus Web App.

4. Verify that you now have access to your collection by navigating to the "File Manager". Select the “Collection” search bar, then click the “Shared with you” tab (Figure 2) and look for the Globus collection name provided via email by ESS-DIVE.

   If you do not see your ESS-DIVE collection here, then you are not connected to it. Please email the Support Team if this occurs.

3. Transfer Files to ESS-DIVE Globus Collection

If you are transferring data from your local desktop, make sure you have Globus Connect Personal installed (see Setup Globus page for instructions). If you’re using an existing public endpoint in Globus, make sure you have access to that endpoint. We will now move your data into your ESS-DIVE collection.

Figure 3: The panel view can alternate between displaying the left, right, or both panels.

Figure 4: Choose the collections to transfer data from and to, select your files, and start the transfer job using the intuitive buttons on-screen.

1. To begin transferring files to your shared Globus collection, go to the File Manager. If you have only one panel on screen, use the panel toggle in the top right to change your view at any time (Figure 3).

2. Select the “Collection” search bar and navigate to the “Your Collections” tab (see example in Setup your Collection). Then select your desktop collection or select an existing source endpoint if you will be using an existing one.

3. Now, in the empty panel, select the “Collection” search bar. Navigate to the “Shared with You” tab (Figure 2). Select the personalized ESS-DIVE folder that we shared with you (in the format of doi-10-15485-1813868).

   • Note that this collection will be empty. At the top, a yellow banner will say "ESS-DIVE Ingest Endpoint" confirming that this is the correct collection.

4. Now you’re ready to transfer your data. Select the files or folders from your collection that you’d like to upload and publish with your ESS-DIVE dataset (Figure 4).

5. Once all your data is selected, hit the “Sync transfer...” button, the "Start" button, or drag your selected files to the shared ESS-DIVE collection (Figure 4).

6. This will start a transfer job and you will see a green popup window on your screen confirming that the transfer has started (Figure 5). In many cases the transfer is quick, but if your transfer is taking a while, use the link from this popup window to check in on the status of the transfer or look in the Activity tab (Figure 6).

7. You will know that the transfer job is complete when Globus sends you a confirmation email that the transfer finished. Congrats! You have completed the Globus portion of this process.

Figure 5: A confirmation message pops up on screen once you initiate a transfer

Figure 6: Use the Activity tab to check the status of your transfer

At this point, your files have not been uploaded or published to the ESS-DIVE repository. This is a temporary collection that the ESS-DIVE Team will use to stage your data for publication. In the next step, we will upload your data and publish it on ESS-DIVE.

4. Move Data from Globus to ESS-DIVE

This step is done by the ESS-DIVE Team. Your review will be necessary.

1. Go to your inbox and reply to the ESS-DIVE Team that your data files have been transferred to your shared Globus collection and are ready for publication.

2. Wait for a reply from the ESS-DIVE Team to proceed. Once we can access your data on the ESS-DIVE Globus collection, we will upload your data to Tier 1 or Tier 2. This process can take a few days depending on how large your files are. ESS-DIVE will email you after the upload is complete and provide a link to your data files either on Tier 1 or Tier 2.

3. Review the data files in your dataset to make sure everything looks correct to you. Let the ESS-DIVE Team know if there are any issues.

4. Reply to the ESS-DIVE Team to let us know you approve the file upload.

   • For data stored on Tier 2, ESS-DIVE will add external links in your dataset that direct end users to access your data via Tier 2.

At this point, the Globus upload process is complete! The final section before publication is to complete your dataset metadata and initiate a publication review. The publication process has been reiterated below for ease of reference.

5. Review and Publish ESS-DIVE Dataset

After your files are successfully uploaded to your dataset on Tier 1 or Tier 2 and you approve the files, ESS-DIVE will wait to hear confirmation from you that your metadata is complete and ready for publication.

1. Respond to the ESS-DIVE Team letting them know that your metadata is ready for review

2. The ESS-DIVE Team will email you any revision requests, if necessary. Revise your metadata as requested.

3. If there are no further revisions requested, your dataset will be approved for publication. Your dataset metadata will be published and the data you transfered to Globus will become publicly accessible via the ESS-DIVE repository.