ICSE Cluster Transition

Data Migration

The file systems will be reformatted during the move to CAC. All user data will be lost. Current ICSE users who will be transitioning to the CAC-hosted ICSE cluster should migrate their ICSE cluster data to a temporary file server at CAC, icse-data.cac.cornell.edu. After the file servers are moved, users can restore their data from icse-data.cac.cornell.edu.

Please note:

1. Users are responsible for copying their own data. If you don't actively copy your data over to CAC, the data will be lost.
2. Data transfer takes time (many days and even weeks), especially if your home directories contain terabytes of data. Please start migrating your data ASAP.

Data Migration Procedures

• Request Your CAC account:
  1. Point your web browser to the Request UserID Form for the ICSE Transition project.
  2. If you have a NetID, select the "Yes I have a CU NetID" option. After logging into CUWebLogin, your request will be submitted.
  3. After your request is approved, you will get an email confirmation at NetID@cornell.edu. Follow the instructions in the email to set your CAC account password. Please note: your CAC account password is NOT the same as the password for your NetID.
  4. Verify your CAC account by ssh to icse-data.cac.cornell.edu using your CAC user name and password. If you have problems logging in, please submit a ticket to help@cac.cornell.edu and someone will get back to you.

• Copy Data to CAC: from the current ICSE cluster login node icse.cornell.edu,

• If you want to stay logged in and monitor the data migration progress, use the following rsync command. Enter your CAC password while prompted.

 rsync -av <source directory> <user id>@icse-data.cac.cornell.edu:<destination directory>

For example, the following command will copy your entire ICSE home directory to CAC:

 rsync -av ~/ <user id>@icse-data.cac.cornell.edu:~

• If you want to initiate the data transfer, log off, and check back later:

• Create password-less login between ICSE cluster and `icse-data.cac.cornell.edu`:

 mkdir .ssh
 chmod 0700 .ssh
 cd .ssh
 ssh-keygen (and take all default options)
 cat id_rsa.pub >> authorized_keys
 scp -r ~/.ssh <user id>@icse.cac.cornell.edu:~

• Initiate data transfer on ICSE login node:

 rsync -av <source directory> <user id>@icse-data.cac.cornell.edu:<destination directory> &> /dev/null & disown

For example, the following command will copy the entire ICSE home directory to CAC:

 rsync -av ~/ shl1@icse-data.cac.cornell.edu:~ &> /dev/null & disown

• You can now log off ICSE and the data transfer will continue. To check if data transfer is still in progress, log back into ICSE login node and do:

ps -u <ICSE user id> | grep rsync

If data transfer is completed, you will get a empty output / response. If you get rsync in the output, wait for data transfer to complete.

• IMPORTANT Verify data transfer: Re-run the same rsync command as above (without the >& /dev/null & disown output redirection options) after data transfer is completed. The subsequent rsync command should only list and copy the differences between source and destination directories. It is imperative that you keep running the same rsync command until the rsync command no longer returns any files.

• Any future changes you make in the ICSE home directory from this point on will be lost unless you re-run the rsync command to copy the changes to CAC.