Red Cloud

This wiki provides documentation for Red Cloud, an on-demand research cloud computing service maintained and supported by CAC. At present, Red Cloud is an Infrastructure as a Service (IaaS) based on OpenStack.

Instructions on these pages apply to users who have a Red Cloud subscription they are managing, though some instructions may also apply to users of subscriptions managed by someone else. Individuals who manage a Red Cloud subscription can create, administer, and delete virtual servers and storage in Red Cloud.

How To Read This Documentation

  • Exploratory Account Users (if you received a trial account)
    • Read through all the sections on this page targeting New Users.
    • Look through the Important Pages listed below to help you get started managing Red Cloud resources.
    • Pay particular attention to the Accounting section on this page, as your exploratory project ends when you have exhausted your subscription.
    • An important point to remember is that you are not the PI on your account, so you can ignore any instructions targeting PIs on a project.
  • New Users (if you are new to Red Cloud)
    • Read through all the sections on this page targeting New Users.
    • Look through the Important Pages listed to help you get started managing Red Cloud resources.
    • If you are not the PI on your account, you can ignore any instructions targeting PIs.
  • Returning Users (all other users)
    • Check out the section dedicated to all users.
    • You may also want to look through the Important Pages listed to help you manage Red Cloud resources.
    • If you are not the PI on your account, you can ignore any instructions targeting PIs.

Note: All links on our wiki are colored red and underlined. Each of the external links will have dotted underlining and an icon next to them that looks like an arrow pointing out of a box, whereas internal links have a solid underline and do not have any icon. This can help you navigate by knowing that the external links are not part of our documentation or "how to" instructions.

Important Pages

Here is a suggested list of pages to look over to help with getting started managing resources.

  1. This page - includes information about:
  2. OpenStack - a highly recommended quick-start page including instructions for:
  3. Either instructions for Linux Instances OR Windows Instances
    • There are special instructions if you intend to use MATLAB Parallel Server
    • There is also a Linux Tutorial for those new to Linux system administration, or if you want a refresher
  4. If you are planning to use GPUs in Red Cloud:

New Users

New users would be best served by reading this complete page first, then reading through the pages listed in the Important Pages section. New users are also encouraged to explore the Getting Started page, which includes a lot more general information on using CAC resources beyond Red Cloud.

Note for new Linux users: As the root user, you will have complete control over access to the system, such as setting up users and their permissions, defining the firewall, and more. This means that the primary user of a Linux system must be familiar with Linux system administration. Aside from the basics of using the command line, this includes familiarity with: creating and modifying users, installing software, configuring software for remote logins, and managing/transferring data. For users that want to use Red Cloud, but do not have much system administration experience, we've written a Linux Tutorial that should work for RedHat/CentOS, Rocky, and Ubuntu Linux systems. Consulting is also available to answer general questions about systems administration, or for help on specific software and research problems.

CAC Account First Time Login

When you are added to a CAC project, you will receive an e-mail confirming your Red Cloud access. You must change the automatically generated password immediately for security reasons and to access computing resources. Refer to the instructions for managing your password as needed.

If you are a PI or a PI's proxy for a new project (If you are not sure what your role is or what you can do, please review this page), verify that you have added a subscription to your project; see the Manage Projects page. After waiting up to an hour for account information to propagate, you will then be ready to download the OpenStack credentials and start managing Red Cloud resources.

How to Create and Manage Red Cloud Resources

Red Cloud is a private research cloud with an OpenStack backend. Interacting with OpenStack is how resources can be managed. In this case, resources can refer to instances (or virtual machines), images, and volumes.

An instance is a virtual machine (VM), which is the main computational resource. To create an instance, you will need an image. You can use default images to create your instance, or you can create and upload an image to OpenStack and use that to create your instance. A volume is a collection of data that is attached to the instance. You should start by creating an instance. For more information on each resource, click the corresponding links.

There are two ways to interact with OpenStack:

Note: Regardless which method you choose (Web Interface or Command Line Interface), you must first follow the First Time Login instructions.

How to Access Instances

Depending on which operating system you are planning on running on your instances, you should also refer to one of the following pages:

Using Up Your Subscription by Accident

Accounting: Don't Use Up Your Subscription by Accident!

To understand how billing works, it is necessary to understand a bit about how Red Cloud operates. Red Cloud enables the user to control the state of system virtual machines (VMs), such as start, pause, suspend, shelve, and delete (see Instance States for a full list). Since starting a VM allocates memory and CPU resources on a physical machine to that VM, subscriptions are billed based on the length of time a VM is running, even if it is idle and doing NO work for the user. This is fair because your running instance will prevent others from using the hardware, even if the hardware is idle.

Thus, the best way to avoid using up your subscription needlessly is to make sure you shelve your Red Cloud instance any time you are not using it. It is very simple to do this via the menu in the OpenStack Web Interface. You can always start the instance again later, and the disk contents will be unchanged. It is just like shutting down your laptop.

Whenever you have one or more instances that are up and running, the amount that is deducted from your Red Cloud subscription is: the length of time that your instances are running, multiplied by the number of cores that you are occupying with those instances. This implies that you should also take advantage of the various instance sizes available. For example, it is usually best to choose a small instance type to do your development work.

It is worth pointing out that Red Cloud allows the instance type to be changed if the VM is stopped (i.e. shut down). This allows you to "scale up" an instance at any time by stopping it, choosing a larger size for it, and starting it back up. You can shrink an instance in the same way. If you intend to use a large instance, we recommend that you start with the smallest instance size you can to install software and get used to your instance before resizing your instance to the full size you would like.

Here are a couple of motivating examples for you. Let's say you have an exploratory account, with just 165 core hours to start. If you leave a 1-core node running around the clock, you will use up the entire account in a little less than a week. Similarly, let's say you are on a CAC project with a Red Cloud subscription (8,585 core hours). If you start up an instance with 4 cores (sometimes called CPUs in OpenStack), and you leave the instance running for a week, or 168 hours, you will use up (168 hours)*(4 cores) or 672 core hours, or 8% of the subscription.

All of the above is true for Linux instances and Windows instances; note that Cornell users do not need to pay for a Windows license in Red Cloud.

We recommend you check your balance frequently using pages provided for Cornell or external users.

Accounting: What is using my Red Cloud storage space?

If a project is using more storage space than expected, the CAC Project PI or a CAC Project Proxy should view the volumes associated with the project at the project management page. The Red Cloud Storage entry in the Project Resource Limits section shows the total amount of storage used and identifies the volumes associated with the project. It should resemble the output shown below. This information is also available in a less compact form at the project's Red Cloud Openstack dashboard

272 GB used as of 11/23/2021 4:01:42 PM
* 36 GBs on 7e6de1a7-1e01-4ccc-99d7-f98b140f7525
* 50 GBs on b5f02c0f-1dba-45d4-a029-94b2714bedbe
* 100 GBs on 3483f718-6673-4080-9b89-97d3c578bf19
* 36 GBs on 739ee9dc-3fb3-46a1-98d8-3e994b31f0aa
* 50 GBs on cf4787b6-3afd-4606-ad11-caa0bed1b61f

Example of storage usage.

Common unintentional storage use patterns

The most common cause of unintentional storage use is orphaned boot volumes left behind from deleted instances. When a project member deletes an instance without noting and deleting the corresponding boot volume, the boot volume will remain in the project storage. While it is possible to configure an instance to delete the boot volume when the instance is deleted, this option is not the default setting and must be selected when the instance is created.

Your project might be storing orphaned boot images if you have more volumes than instances. The default size of boot volumes tends to be in the 30-50 GB range. Your project may also use volumes for data storage but these volumes tend to be larger than 50 GB.

You can avoid creating orphaned boot volumes by following a two step procedure when deleting a Red Cloud instance. If the instances were already deleted at some point in the past, skip to step 2 in deleting a Red Cloud instance.

All Users

Please refer to the OpenStack page for more in-depth guidance on how to use Red Cloud, and read either Linux instances or Windows instances based on what systems will be used.

The current Red Cloud System Status can be checked anytime.

Common Tasks

Here are some links to help you with particular aspects of using Red Cloud:

  • Linux Tutorial - This may help you get up and running with some basic systems administration tasks. It is not intended to be comprehensive.
  • Information on choosing instance flavor (the CPU and RAM configuration of the virtual machine).
  • Extending or shrinking a volume is a separate issue, and is somewhat more involved.
  • EZ-backup - a CIT solution for backups. Data stored on Red Cloud is not backed up by default; users are responsible for their own backups.
  • Data in CAC Archival Storage is intended to be an additional copy of user data; CAC Archival Storage is not backed up or snapshotted.
  • All CAC resources are suitable for unregulated, non-confidential data (reference for details).

Acknowledging CAC

When you publish a paper, make presentations, or are interviewed by the Cornell Chronicle, national news media, etc., please acknowledge the Center by including:

"This research was conducted with support from the Cornell University Center for Advanced Computing."

Alternatively, the full acknowledgement is:

"This research was conducted with support from the Cornell University Center for Advanced Computing, which receives funding from Cornell University, the National Science Foundation, and members of its Partner Program."

Software on Red Cloud

Software

FAQ

Red Cloud FAQ