Upload Data with Globus
Learn how to transfer your files to Globus from Globus Connect Personal or an existing Globus endpoint.
Last updated
Learn how to transfer your files to Globus from Globus Connect Personal or an existing Globus endpoint.
Last updated
Please note that once your large data is made accessible on NERSC via Globus, neither you nor ESS-DIVE will be able to edit the files. They can either be transferred elsewhere or deleted.
Before we can transfer data to ESS-DIVE using Globus, it is necessary to create a dataset on ESS-DIVE and get the metadata ready for publication. It is recommended to complete as much metadata as possible before moving on to the next step.
Navigate to ESS-DIVE, click the "Submit Data" button to start a new dataset. Fill out the minimum metadata and click the “Submit Dataset” button to create a private dataset.
Read more about creating datasets on our Web Form tutorial.
You can also use the Dataset API to submit metadata to ESS-DIVE.
Do not upload any data files to this dataset.
Initiate a publication request to begin your Globus upload. Select "Manage Publication" from your dataset landing page, then click "Start Publication Process" in the pop-up window.
See complete instructions on our Request Publication documentation page.
Shortly after, you will receive an email that confirms your request has been received. Reply to this email to inform the ESS-DIVE Team that you need to upload your data files using Globus. If you did not receive a confirmation email, please contact our support team.
The ESS-DIVE Team will then reply to your message to continue the Globus upload process.
Tier 2 datasets: At this time, an inactive DOI will be minted for your dataset. The inactive DOI will be used at a later step in the Globus upload process and it will not become active until after your dataset is published.
The ESS-DIVE Team will create a Globus collection that will be managed by ESS-DIVE and it will be specific to the dataset you are publishing. This section will walk you through how to enable access to this collection, which will later allow you to transfer your data here.
The ESS-DIVE Team will need the email or ORCID associated with your Globus “identity” (i.e. your Globus account ID) to enable shared access to the Globus collection we create. Reply to the confirmation email you received from ESS-DIVE in the previous step (“Create ESS-DIVE Dataset") and send your email or ORCID to the ESS-DIVE Team, whichever you used to setup your account.
Confirm your account information by checking your identity. To find your identity, go to Globus (https://app.globus.org/) and navigate to the “Settings” tab (Figure 1). There you will see your primary identity. Select the dropdown arrow and copy the information listed there.
Wait for a reply from the ESS-DIVE Team to proceed. We will send you the name of your personalized Globus collection where you will upload your data. The collection name will be named after the inactive DOI minted for your dataset in the format of doi-10-15485-1813868.
Once you receive your ESS-DIVE Globus collection, log in to the Globus Web App.
Verify that you now have access to your collection by navigating to the "File Manager". Select the “Collection” search bar, then click the “Shared with you” tab (Figure 2) and look for the Globus collection name provided via email by ESS-DIVE.
If you do not see your ESS-DIVE collection here, then you are not connected to it. Please email the Support Team if this occurs.
If you are transferring data from your local desktop, make sure you have Globus Connect Personal installed (see Setup Globus page for instructions). If you’re using an existing public endpoint in Globus, make sure you have access to that endpoint. We will now move your data into your ESS-DIVE collection.
To begin transferring files to your shared Globus collection, go to the File Manager. If you have only one panel on screen, use the panel toggle in the top right to change your view at any time (Figure 3).
Select the “Collection” search bar and navigate to the “Your Collections” tab (see example in Setup your Collection). Then select your desktop collection or select an existing source endpoint if you will be using an existing one.
Now, in the empty panel, select the “Collection” search bar. Navigate to the “Shared with You” tab (Figure 2). Select the personalized ESS-DIVE folder that we shared with you (in the format of doi-10-15485-1813868).
Note that this collection will be empty. At the top, a yellow banner will say "ESS-DIVE Ingest Endpoint" confirming that this is the correct collection.
Now you’re ready to transfer your data. Select the files or folders from your collection that you’d like to upload and publish with your ESS-DIVE dataset (Figure 4).
Once all your data is selected, hit the “Sync transfer...” button, the "Start" button, or drag your selected files to the shared ESS-DIVE collection (Figure 4).
This will start a transfer job and you will see a green popup window on your screen confirming that the transfer has started (Figure 5). In many cases the transfer is quick, but if your transfer is taking a while, use the link from this popup window to check in on the status of the transfer or look in the Activity tab (Figure 6).
You will know that the transfer job is complete when Globus sends you a confirmation email that the transfer finished. Congrats! You have completed the Globus portion of this process.
At this point, your files have not been uploaded or published to the ESS-DIVE repository. This is a temporary collection that the ESS-DIVE Team will use to stage your data for publication. In the next step, we will upload your data and publish it on ESS-DIVE.
This step is done by the ESS-DIVE Team. Your review will be necessary.
Go to your inbox and reply to the ESS-DIVE Team that your data files have been transferred to your shared Globus collection and are ready for publication.
Wait for a reply from the ESS-DIVE Team to proceed. Once we can access your data on the ESS-DIVE Globus collection, we will upload your data to Tier 1 or Tier 2. This process can take a few days depending on how large your files are. ESS-DIVE will email you after the upload is complete and provide a link to your data files either on Tier 1 or Tier 2.
Review the data files in your dataset to make sure everything looks correct to you. Let the ESS-DIVE Team know if there are any issues.
Reply to the ESS-DIVE Team to let us know you approve the file upload.
For data stored on Tier 2, ESS-DIVE will add external links in your dataset that direct end users to access your data via Tier 2.
At this point, the Globus upload process is complete! The final section before publication is to complete your dataset metadata and initiate a publication review. The publication process has been reiterated below for ease of reference.
After your files are successfully uploaded to your dataset on Tier 1 or Tier 2 and you approve the files, ESS-DIVE will wait to hear confirmation from you that your metadata is complete and ready for publication.
Respond to the ESS-DIVE Team letting them know that your metadata is ready for review.
The ESS-DIVE Team will email you any revision requests, if necessary. Revise your metadata as requested.
If there are no further revisions requested, your dataset will be approved for publication. Your dataset metadata will be published and the data you transfered to Globus will become publicly accessible via the ESS-DIVE repository.
