Roadmap: Data Transfer
Transferring data and code between your workstation and a remote computer is a common part of scientific workflows. Sometimes this data can be quite large, and sometimes you wish to transmit your data securely. And recently, data transfers between cloud storage and computing facilities are becoming increasingly common. There are a number of utilities available to help you accomplish these essential tasks.
Your choice of data transfer utility will depend on how much data you are transferring, how you prefer to perform the transfer, and your priorities (including transfer speed, ease of use, security and validation). This topic presents several data transfer options and their pros and cons, as well as ways to make these transfers faster. While the file transfer techniques presented here are useful in many situations, the included examples will use TACC's Stampede2 and Frontera as the remote computers.
Objectives
After you complete this workshop, you should be able to transfer files...
- between your home environment and a computing environment.
- between a cloud storage provider and a computing environment.
- using a command line program, which can be invoked from scripts.
- using an application with a Graphical User Interface (GUI).
Your ability to decide which transfer utility works best in a given situation should also be improved.
Prerequisites
- You should have some familiarity with the file systems and access requirements for high performance computing environments. Example environments would be those of Stampede2 and Frontera, described in the Stampede2 Environment and the Getting Started on Frontera topics. In particular, see the sections on "Storing and Moving Data" for Stampede2 and Frontera, respectively.
Requirements
System requirements may include:
- Access to a computer system with some of the described utilities installed, or have the ability to install them on a computer.
- An account on Stampede2 or Frontera through ACCESS or TACC.
- A Globus account, which can be acquired through the Globus website.