This wiki provides documentation for Red Cloud, an on-demand research cloud computing service maintained and supported by the CAC. At present, Red Cloud is an Infrastructure as a Service (IaaS) based on OpenStack.
Instructions on these pages apply to users who have a Red Cloud subscription they are managing, though some instructions may also apply to users of subscriptions managed by someone else. Individuals who manage a Red Cloud subscription can create, administer, and delete virtual servers and storage in Red Cloud.
- 1 How To Read This Documentation
- 2 New Users
- 3 All Users
- 4 Software on Red Cloud
- 5 FAQ
How To Read This Documentation
- Exploratory Account Users
- Read through all the sections on this page targeting New Users.
- Look through the Important Pages listed below to help you get started managing Red Cloud resources.
- Pay particular attention to the Accounting section on this page, as your exploratory project ends when you have exhausted your subscription.
- An important point to remember is that you are not the PI on your account, so you can ignore any instructions targeting PIs on a project.
- New Users
- Read through all the sections on this page targeting New Users.
- Look through the Important Pages listed to help you get started managing Red Cloud resources.
- If you are not the PI on your account, you can ignore any instructions targeting PIs.
- Returning Users
- Exploratory Account Users
Note: All links on our wiki are colored red and underlined. Each of the external links will have dotted underlining and an icon next to them that looks like an arrow pointing out of a box, whereas internal links have a solid underline and do not have any icon. This can help you navigate by knowing that the external links are not part of our documentation or "how to" instructions.
Here is a suggested list of pages to look over to help with getting started managing resources.
- This page - includes information about:
- OpenStack - a highly recommended quick-start page including instructions for:
- Either instructions for Linux Instances OR Windows Instances
- If you are planning to use GPUs in Red Cloud:
New users would be best served by reading this complete page first, then reading through the pages listed in the Important Pages section. New users are also encouraged to explore the Getting Started page, which includes a lot more general information on using CAC resources beyond Red Cloud.
Note for new Linux users: As the root user, you will have complete control over access to the system, such as setting up users and their permissions, defining the firewall, and more. This means that the primary user of a Linux system must be familiar with Linux system administration. Aside from the basics of using the command line, this includes familiarity with: creating and modifying users, installing software, configuring software for remote logins, and managing/transferring data. For users that want to use Red Cloud, but do not have much system administration experience, we've written a Linux Tutorial that should work for RedHat/CentOS and Ubuntu Linux systems. Consulting is also available to answer general questions about systems administration, or for help on specific software and research problems.
First Time Login
When you are added to a CAC project, you will receive an e-mail confirming your Red Cloud access. You must change the automatically generated password immediately for security reasons and to access computing resources. Refer to the instructions for managing your password as needed.
If you are a PI or a PI's proxy for a new project, verify that you have added a subscription to your project; see the Manage Projects page. After waiting up to an hour for account information to propagate, you will then be ready to download the OpenStack credentials and start managing Red Cloud resources.
How to Create and Manage Red Cloud Resources
Red Cloud is a private research cloud with an OpenStack backend. Interacting with OpenStack is how resources can be managed. In this case, resources can refer to instances (or virtual machines), images, and volumes. There are two ways to interact with OpenStack:
- The OpenStack Web Interface (Horizon)
- The Command-Line Interface (CLI) called the OpenStack CLI
Note: Regardless which method you choose (Web Interface or Command Line Interface), you must follow the First Time Login instructions.
How to Access Instances
Depending on which operating system you are planning on running on your instances, you should also refer to one of the following pages:
Accounting: Don't Use Up Your Subscription by Accident!
To understand how billing works, it is necessary to understand a bit about how Red Cloud operates. Red Cloud enables the user to control the state of system virtual machines (VMs), such as start, pause, suspend, shelve, and delete (see Instance States for a full list). Since starting a VM allocates memory and CPU resources on a physical machine to that VM, subscriptions are billed based on the length of time a VM is running, even if it is idle and doing NO work for the user. This is fair because your running instance will prevent others from using the hardware, even if the hardware is idle.
Thus, the best way to avoid using up your subscription needlessly is to make sure you shelve your Red Cloud instance any time you are not using it. It is very simple to do this via the menu in the OpenStack Web Interface. You can always start the instance again later, and the disk contents will be unchanged. It is just like shutting down your laptop.
Whenever you have one or more instances that are up and running, the amount that is deducted from your Red Cloud subscription is: the length of time that your instances are running, multiplied by the number of cores that you are occupying with those instances. This implies that you should also take advantage of the various instance sizes available. For example, it is usually best to choose a small instance type to do your development work.
It is worth pointing out that Red Cloud allows the instance type to be changed if the VM is stopped (i.e. shut down). This allows you to "scale up" an instance at any time by stopping it, choosing a larger size for it, and starting it back up. You can shrink an instance in the same way. If you intend to use a large instance, we recommend that you start with the smallest instance size you can to install software and get used to your instance before resizing your instance to the full size you would like.
Here are a couple of motivating examples for you. Let's say you have an exploratory account, with just 165 core hours to start. If you leave a 1-core node running around the clock, you will use up the entire account in a little less than a week. Similarly, let's say you are on a CAC project with a Red Cloud subscription (8,585 core hours). If you start up an instance with 4 cores (sometimes called CPUs in OpenStack), and you leave the instance running for a week, or 168 hours, you will use up (168 hours)*(4 cores) or 672 core hours, or 8% of the subscription.
The current Red Cloud System Status can be checked anytime.
Here are some links to help you with particular aspects of using Red Cloud:
- Linux Tutorial - This may help you get up and running with some basic systems administration tasks. It is not intended to be comprehensive.
- Information on choosing instance type (the size of the virtual machine). Resizing volumes is a separate issue, and is somewhat more involved.
- EZ-backup - a CIT solution for backups. Data stored on Red Cloud is not backed up by default; users are responsible for their own backups.
- Data in CAC Archival Storage is intended to be an additional copy of user data; CAC Archival Storage is not backed up or snapshotted.
- All CAC resources are suitable for unregulated, non-confidential data (reference for details).
When you publish a paper, make presentations, or are interviewed by the Cornell Chronicle, national news media, etc., please acknowledge the Center by including:
"This research was conducted with support from the Cornell University Center for Advanced Computing."
Alternatively, the full acknowledgement is:
"This research was conducted with support from the Cornell University Center for Advanced Computing, which receives funding from Cornell University, the National Science Foundation, and members of its Partner Program."
Software on Red Cloud
Generally, new instances launched on Red Cloud will contain basic operating system software and utilities, but will not contain pre-installed scientific applications. It is your responsibility to install any relevant applications either using a built-in package manager or by transferring your application code to the instance (e.g., via scp or sftp). In some cases, however, there are resources available to support running particular applications, as described below:
- On Linux instances, information on using package managers to install software: ( Using apt on Ubuntu ) ( Using yum on Centos )
- Installing R, a commonly used programming language and statistical analysis environment
- Running MATLAB Parallel Server in Red Cloud
- Running OpenFOAM-7 in a Docker container
- Creating and using a GPU Instance with pre-installed software (CUDA, NVIDIA Driver, Anaconda, Docker, Jupyter, MATLAB, etc.)