Difference between revisions of "ICSE Cluster Transition"

From CAC Documentation wiki
Jump to navigation Jump to search
Line 7: Line 7:
  
 
=== Data Migration Procedures ===
 
=== Data Migration Procedures ===
 +
Data transfer takes time (many days and even weeks), especially your home directories contain terabytes of data. Please start migrating your data ASAP using the follwoing procedures.
  
 
* Request Your CAC account:  
 
* Request Your CAC account:  
Line 33: Line 34:
 
:* You can now log off ICSE and the data transfer will continue. To check if data transfer is still in progress, log back into ICSE login node and do:
 
:* You can now log off ICSE and the data transfer will continue. To check if data transfer is still in progress, log back into ICSE login node and do:
 
  ps -u <ICSE user id> | grep rsync
 
  ps -u <ICSE user id> | grep rsync
::: If data transfer is completed, you will get a empty output / response.
+
::: If data transfer is completed, you will get a empty output / response. If you get <code>rsync</code> in the output, wait for data transfer to complete.
 +
 
 +
* '''IMPORTANT''' Verify data transfer: Re-run the same <code>rsync</code> command as above (without the <code>>& /dev/null & disown</code> output redirection options) after data transfer is completed. The subsequent <code>rsync</code> command should only list and copy the differences between source and destination directories. It is imperative that you keep running the same <code>rsync</code> command until the <code>rsync</code> command no longer returns any files.
 +
 
 +
* Any future changes you make in the ICSE home directory from this point on will be lost unless you re-run the <code>rsync</code> command to copy the changes to CAC.

Revision as of 15:30, 3 December 2018

Data Migration

The file systems will be reformatted during the move to CAC so all user data will be lost. Current ICSE users who will be transitioning to the CAC-hosted ICSE cluster should migrate their ICSE cluster data to a temporary file server at CAC, icse-data.cac.cornell.edu. After the file server moved are completed, users can restore their data from icse-data.cac.cornell.edu.

Please note: users are responsible for copying their own data. Any data that's not copied by their owner will be lost.

Data Migration Procedures

Data transfer takes time (many days and even weeks), especially your home directories contain terabytes of data. Please start migrating your data ASAP using the follwoing procedures.

  • Request Your CAC account:
    1. Point your web browser to the Request UserID Form for the ICSE Transition project.
    2. If you have a NetID, select the "Yes I have a CU NetID" option. After logging into CUWebLogin, your request will be submitted.
    3. After your request is approved, you will get an email confirmation at NetID@cornell.edu. Follow the instructions in the email to set your CAC account password. Please note: your CAC account password is NOT the same as the password for your NetID.
    4. Verify your CAC account by ssh to icse-data.cac.cornell.edu using your CAC user name and password. If you have problems logging in, please submit a ticket to help@cac.cornell.edu and someone will get back to you.
  • Copy Data to CAC: from the current ICSE cluster login node icse.cornell.edu,
  • If you want to stay logged in and monitor the data migration progress, use the following rsync command. Enter your CAC password while prompted.
 rsync -av <source directory> <user id>@icse-data.cac.cornell.edu:<destination directory>
For example, the following command will copy your entire ICSE home directory to CAC:
 rsync -av ~/ <user id>@icse-data.cac.cornell.edu:~
  • If you want to initiate the data transfer, log off, and check back later:
  • Create password-less login between ICSE cluster and `icse-data.cac.cornell.edu`:
 mkdir .ssh
 chmod 0700 .ssh
 cd .ssh
 ssh-keygen (and take all default options)
 cat id_rsa.pub >> authorized_keys
 scp -r ~/.ssh <user id>@icse.cac.cornell.edu:~
  • Initiate data transfer on ICSE login node:
 rsync -av <source directory> <user id>@icse-data.cac.cornell.edu:<destination directory> &> /dev/null & disown
For example, the following command will copy the entire ICSE home directory to CAC:
 rsync -av ~/ shl1@icse-data.cac.cornell.edu:~ &> /dev/null & disown
  • You can now log off ICSE and the data transfer will continue. To check if data transfer is still in progress, log back into ICSE login node and do:
ps -u <ICSE user id> | grep rsync
If data transfer is completed, you will get a empty output / response. If you get rsync in the output, wait for data transfer to complete.
  • IMPORTANT Verify data transfer: Re-run the same rsync command as above (without the >& /dev/null & disown output redirection options) after data transfer is completed. The subsequent rsync command should only list and copy the differences between source and destination directories. It is imperative that you keep running the same rsync command until the rsync command no longer returns any files.
  • Any future changes you make in the ICSE home directory from this point on will be lost unless you re-run the rsync command to copy the changes to CAC.