Difference between revisions of "ICSE Cluster Transition"
Jump to navigation
Jump to search
(10 intermediate revisions by the same user not shown) | |||
Line 2: | Line 2: | ||
== Data Migration == | == Data Migration == | ||
− | The file systems will be reformatted during the move to CAC | + | '''The file systems will be reformatted during the move to CAC. All user data will be lost'''. Current ICSE users who will be transitioning to the CAC-hosted ICSE cluster should migrate their ICSE cluster data to a temporary file server at CAC, <code>icse-data.cac.cornell.edu</code>. After the file servers are moved, users can restore their data from <code>icse-data.cac.cornell.edu</code>. |
− | |||
− | |||
+ | Please note: | ||
+ | # Users are responsible for copying their own data. If you don't actively copy your data over to CAC, the data will be lost. | ||
+ | # Data transfer takes time (many days and even weeks), especially if your home directories contain terabytes of data. Please start migrating your data ASAP. | ||
=== Data Migration Procedures === | === Data Migration Procedures === | ||
− | |||
* Request Your CAC account: | * Request Your CAC account: | ||
*# Point your web browser to the [https://www.cac.cornell.edu/services/external/RequestCACid.aspx?ProjectID=lma36_0001 Request UserID Form for the ICSE Transition project]. | *# Point your web browser to the [https://www.cac.cornell.edu/services/external/RequestCACid.aspx?ProjectID=lma36_0001 Request UserID Form for the ICSE Transition project]. | ||
Line 15: | Line 15: | ||
* Copy Data to CAC: from the current ICSE cluster login node <code>icse.cornell.edu</code>, | * Copy Data to CAC: from the current ICSE cluster login node <code>icse.cornell.edu</code>, | ||
− | * If you want to stay logged in and monitor the data migration progress, use the following <code>rsync</code> command. Enter your CAC password while prompted. | + | :* If you want to stay logged in and monitor the data migration progress, use the following <code>rsync</code> command. Enter your CAC password while prompted. |
rsync -av <source directory> <user id>@icse-data.cac.cornell.edu:<destination directory> | rsync -av <source directory> <user id>@icse-data.cac.cornell.edu:<destination directory> | ||
− | : For example, the following command will copy your entire ICSE home directory to CAC: | + | :: For example, the following command will copy your entire ICSE home directory to CAC: |
rsync -av ~/ <user id>@icse-data.cac.cornell.edu:~ | rsync -av ~/ <user id>@icse-data.cac.cornell.edu:~ | ||
− | * If you initiate the data transfer, log off, and check back later: | + | :* If you want to initiate the data transfer, log off, and check back later: |
− | :* Create password-less login between ICSE cluster and `icse-data.cac.cornell.edu`: | + | ::* Create password-less login between ICSE cluster and `icse-data.cac.cornell.edu`: |
mkdir .ssh | mkdir .ssh | ||
chmod 0700 .ssh | chmod 0700 .ssh | ||
Line 27: | Line 27: | ||
cat id_rsa.pub >> authorized_keys | cat id_rsa.pub >> authorized_keys | ||
scp -r ~/.ssh <user id>@icse.cac.cornell.edu:~ | scp -r ~/.ssh <user id>@icse.cac.cornell.edu:~ | ||
− | :* Initiate data transfer on ICSE login node: | + | ::* Initiate data transfer on ICSE login node: |
rsync -av <source directory> <user id>@icse-data.cac.cornell.edu:<destination directory> &> /dev/null & disown | rsync -av <source directory> <user id>@icse-data.cac.cornell.edu:<destination directory> &> /dev/null & disown | ||
− | :: For example, the following command will copy the entire ICSE home directory to CAC: | + | ::: For example, the following command will copy the entire ICSE home directory to CAC: |
rsync -av ~/ shl1@icse-data.cac.cornell.edu:~ &> /dev/null & disown | rsync -av ~/ shl1@icse-data.cac.cornell.edu:~ &> /dev/null & disown | ||
+ | ::* You can now log off ICSE and the data transfer will continue. To check if data transfer is still in progress, log back into ICSE login node and do: | ||
+ | ps -u <ICSE user id> | grep rsync | ||
+ | ::: If data transfer is completed, you will get a empty output / response. If you get <code>rsync</code> in the output, wait for data transfer to complete. | ||
+ | |||
+ | * '''IMPORTANT''' Verify data transfer: Re-run the same <code>rsync</code> command as above (without the <code>>& /dev/null & disown</code> output redirection options) after data transfer is completed. The subsequent <code>rsync</code> command should only list and copy the differences between source and destination directories. It is imperative that you keep running the same <code>rsync</code> command until the <code>rsync</code> command no longer returns any files. | ||
+ | |||
+ | * Any future changes you make in the ICSE home directory from this point on will be lost unless you re-run the <code>rsync</code> command to copy the changes to CAC. |
Latest revision as of 17:09, 3 December 2018
Data Migration
The file systems will be reformatted during the move to CAC. All user data will be lost. Current ICSE users who will be transitioning to the CAC-hosted ICSE cluster should migrate their ICSE cluster data to a temporary file server at CAC, icse-data.cac.cornell.edu
. After the file servers are moved, users can restore their data from icse-data.cac.cornell.edu
.
Please note:
- Users are responsible for copying their own data. If you don't actively copy your data over to CAC, the data will be lost.
- Data transfer takes time (many days and even weeks), especially if your home directories contain terabytes of data. Please start migrating your data ASAP.
Data Migration Procedures
- Request Your CAC account:
- Point your web browser to the Request UserID Form for the ICSE Transition project.
- If you have a NetID, select the "Yes I have a CU NetID" option. After logging into CUWebLogin, your request will be submitted.
- After your request is approved, you will get an email confirmation at NetID@cornell.edu. Follow the instructions in the email to set your CAC account password. Please note: your CAC account password is NOT the same as the password for your NetID.
- Verify your CAC account by ssh to
icse-data.cac.cornell.edu
using your CAC user name and password. If you have problems logging in, please submit a ticket to help@cac.cornell.edu and someone will get back to you.
- Copy Data to CAC: from the current ICSE cluster login node
icse.cornell.edu
,
- If you want to stay logged in and monitor the data migration progress, use the following
rsync
command. Enter your CAC password while prompted.
- If you want to stay logged in and monitor the data migration progress, use the following
rsync -av <source directory> <user id>@icse-data.cac.cornell.edu:<destination directory>
- For example, the following command will copy your entire ICSE home directory to CAC:
rsync -av ~/ <user id>@icse-data.cac.cornell.edu:~
- If you want to initiate the data transfer, log off, and check back later:
- Create password-less login between ICSE cluster and `icse-data.cac.cornell.edu`:
mkdir .ssh chmod 0700 .ssh cd .ssh ssh-keygen (and take all default options) cat id_rsa.pub >> authorized_keys scp -r ~/.ssh <user id>@icse.cac.cornell.edu:~
- Initiate data transfer on ICSE login node:
rsync -av <source directory> <user id>@icse-data.cac.cornell.edu:<destination directory> &> /dev/null & disown
- For example, the following command will copy the entire ICSE home directory to CAC:
rsync -av ~/ shl1@icse-data.cac.cornell.edu:~ &> /dev/null & disown
- You can now log off ICSE and the data transfer will continue. To check if data transfer is still in progress, log back into ICSE login node and do:
ps -u <ICSE user id> | grep rsync
- If data transfer is completed, you will get a empty output / response. If you get
rsync
in the output, wait for data transfer to complete.
- If data transfer is completed, you will get a empty output / response. If you get
- IMPORTANT Verify data transfer: Re-run the same
rsync
command as above (without the>& /dev/null & disown
output redirection options) after data transfer is completed. The subsequentrsync
command should only list and copy the differences between source and destination directories. It is imperative that you keep running the samersync
command until thersync
command no longer returns any files.
- Any future changes you make in the ICSE home directory from this point on will be lost unless you re-run the
rsync
command to copy the changes to CAC.