https://www.cac.cornell.edu/wiki/api.php?action=feedcontributions&user=Ad876&feedformat=atomCAC Documentation wiki - User contributions [en]2024-03-29T15:26:51ZUser contributionsMediaWiki 1.35.5https://www.cac.cornell.edu/wiki/index.php?title=Linux_Usage_Tips&diff=928Linux Usage Tips2015-09-30T20:27:33Z<p>Ad876: </p>
<hr />
<div>==== Linux shells====<br />
:* <tt>/bin/sh</tt> is the default login shell.<br />
:** Edit <tt>$HOME/.profile</tt> to change interactive variables.<br />
:** The <tt>$HOME/.bashrc</tt> file ''will not'' be run for non-interactive shells.<br />
:* <tt>/bin/bash</tt><br />
:** Edit <tt>$HOME/.profile</tt> to change interactive variables.<br />
:** The <tt>$HOME/.bashrc</tt> file ''will'' be run for non-interactive shells.<br />
:* <tt>/bin/csh</tt> and <tt>/bin/tcsh</tt><br />
:** Edit <tt>$HOME/.login</tt> to change interactive variables.<br />
:** The <tt>$HOME/.cshrc</tt> file ''will'' be run for non-interactive shells.<br />
<br />
The change shell command, <tt>chsh</tt>, will not permanently change your shell. You must send a request instead. {{ContactCAC}}<br />
<br />
The default login shell on v4 interactive and batch nodes is ''sh''. Be aware that in Red Hat Enterprise Linux, /bin/sh is a soft-link to /bin/bash, so you are really using a variant of ''bash''. Accordingly, you will find that "man sh" brings up the man page (the help document) for ''bash''. In a way, then, you can think of your login shell as being ''bash'', too.<br />
<br />
There are slight differences between ''sh'' and ''bash'', however. The "Invocation" section of the man page states: "If bash is invoked with the name sh, it tries to mimic the startup behavior of historical versions of ''sh'' as closely as possible." Therefore, you will find that ~/.profile is run at login, because this behavior is common to both ''sh'' and ''bash''; but any interactive ''sh'' shells you start thereafter will not run ~/.bashrc as you might expect from ''bash''. The way to get ''sh'' to do this is to "export ENV=~/.bashrc" beforehand (perhaps as part of your .profile).<br />
<br />
Let's say you simply prefer to have ''bash'' as your default shell and be done with it. There are two ways to accomplish this. First, you can "export SHELL=/bin/bash" in your .profile; then all subsequent interactive shells will truly be ''bash''. Second, you can enter "chsh -s /bin/bash", which forces all login and interactive shells to be ''bash'' (because you have changed your default shell). The problem with the second method is it may well wreck your batch environment, too, because the scheduler sets it up under the assumption that the login shell is ''sh''.<br />
<br />
The relationship between the ''csh'' and ''tcsh'' shells is similar to the one between ''sh'' and ''bash''. For instance, your ''csh'' shells are automatically endowed with the ''tcsh''-style ability to retrieve history through the up- and down-arrow keys. The best way to make ''tcsh'' into your everyday working shell is to run it on top of ''sh'' after you log in (again, you can do this as part of your .profile).<br />
<br />
References<br />
:* "man bash" from the command line.<br />
:* [//tldp.org/LDP/abs/html/ Advanced Bash Scripting Guide], one of the Linux Documentation Project [//tldp.org/guides.html guides]<br />
:* [//mywiki.wooledge.org/BashFAQ Bash FAQ]<br />
:*[//mywiki.wooledge.org/BashPitfalls Bash Pitfalls]<br />
<br />
====Compiling and linking code on Linux====<br />
{{:Compiling Code Linux}}<br />
<br />
====FAQ====<br />
=====How do I determine my program's dependencies on shared library (.so) files?=====<br />
:*ldd - see the man page.<br />
If your program cannot find all the .so files it needs, you may need to add paths to the LD_LIBRARY_PATH shell variable.<br />
<br />
=====How do I display an image file (such as jpeg or gif)?=====<br />
:*display mypic.jpg - uses one of the many ImageMagick tools - see "man ImageMagick" for help on this and various file format converters.<br />
:*firefox mypic.jpg - any decent Web browser can handle it.<br />
Note, the image will show up only if you have [[Getting_Started#Connect_to_Linux | X11 forwarding]] enabled.</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=File_transfer&diff=927File transfer2015-09-30T20:27:21Z<p>Ad876: </p>
<hr />
<div>A single, central file server, storage01.cac.cornell.edu, serves all CAC user home directories. You can connect to this server in a variety of ways from any operating system to access your files. <br />
<br />
Note: by default, your home directory and its contents will be readable and executable by all other users of CAC systems. If this is not what you want, you can change the permissions of the home directory and its files and subdirectories via the standard Linux or Windows mechanisms. However, be aware that this may lead to conflicts for cross-platform applications, as Windows and Linux permissions are not 100% compatible.<br />
<br />
====Linux users====<br />
=====Secure Copy=====<br />
Secure copy is a standard tool to copy files to and from remote hosts.<br />
localhost$ scp localfile.dat username@linuxlogin.cac.cornell.edu:remoteinput.dat<br />
localhost$ scp username@linuxlogin.cac.cornell.edu:results.dat localresults.dat<br />
<br />
=====Secure FTP=====<br />
FTP is disabled for security reasons, but sftp's interface is nearly identical.<br />
=====Samba Client=====<br />
Type<br />
smbclient //storage01.cac.cornell.edu/<user name> -U <user name><br />
Enter the password for your CAC account when prompted<br />
You will see the '''''smb:\>''''' prompt. You can now start transferring files between your local machine and CAC home directory similar to ftp client. Type '''''help''''' for more instructions.<br />
-sh-3.2$ smbclient //storage01.cac.cornell.edu/<user name> -U <user name><br />
Password: <br />
Domain=[CTC_ITH] OS=[Unix] Server=[Samba 3.0.28-1.el5_2.1]<br />
smb: \> help<br />
<br />
====MacOS X users====<br />
<br />
====Windows users====<br />
=====Secure Copy=====<br />
The people who make Putty provide a secure copy client called [//www.chiark.greenend.org.uk/~sgtatham/putty/download.html pscp]. From the command prompt, type:<br />
cmd> pscp localfile.dat username@linuxlogin.cac.cornell.edu:remoteinput.dat<br />
<enter your username's password when prompted><br />
cmd> pscp username@linuxlogin.cac.cornell.edu:results.dat localresults.dat<br />
<br />
=====Secure FTP=====<br />
FTP is disabled for security reasons, but [//www.chiark.greenend.org.uk/~sgtatham/putty/download.html psftp's] interface is nearly identical. From the command prompt, type:<br />
cmd> psftp username@linuxlogin.cac.cornell.edu<br />
<enter your username's password when prompted><br />
psftp> put localresults.dat results.dat<br />
psftp> quit</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Linux_Usage_Tips&diff=924Linux Usage Tips2015-09-30T20:24:17Z<p>Ad876: /* Linux shells */</p>
<hr />
<div>==== Linux shells====<br />
:* <tt>/bin/sh</tt> is the default login shell.<br />
:** Edit <tt>$HOME/.profile</tt> to change interactive variables.<br />
:** The <tt>$HOME/.bashrc</tt> file ''will not'' be run for non-interactive shells.<br />
:* <tt>/bin/bash</tt><br />
:** Edit <tt>$HOME/.profile</tt> to change interactive variables.<br />
:** The <tt>$HOME/.bashrc</tt> file ''will'' be run for non-interactive shells.<br />
:* <tt>/bin/csh</tt> and <tt>/bin/tcsh</tt><br />
:** Edit <tt>$HOME/.login</tt> to change interactive variables.<br />
:** The <tt>$HOME/.cshrc</tt> file ''will'' be run for non-interactive shells.<br />
<br />
The change shell command, <tt>chsh</tt>, will not permanently change your shell. You must send a request instead. {{ContactCAC}}<br />
<br />
The default login shell on v4 interactive and batch nodes is ''sh''. Be aware that in Red Hat Enterprise Linux, /bin/sh is a soft-link to /bin/bash, so you are really using a variant of ''bash''. Accordingly, you will find that "man sh" brings up the man page (the help document) for ''bash''. In a way, then, you can think of your login shell as being ''bash'', too.<br />
<br />
There are slight differences between ''sh'' and ''bash'', however. The "Invocation" section of the man page states: "If bash is invoked with the name sh, it tries to mimic the startup behavior of historical versions of ''sh'' as closely as possible." Therefore, you will find that ~/.profile is run at login, because this behavior is common to both ''sh'' and ''bash''; but any interactive ''sh'' shells you start thereafter will not run ~/.bashrc as you might expect from ''bash''. The way to get ''sh'' to do this is to "export ENV=~/.bashrc" beforehand (perhaps as part of your .profile).<br />
<br />
Let's say you simply prefer to have ''bash'' as your default shell and be done with it. There are two ways to accomplish this. First, you can "export SHELL=/bin/bash" in your .profile; then all subsequent interactive shells will truly be ''bash''. Second, you can enter "chsh -s /bin/bash", which forces all login and interactive shells to be ''bash'' (because you have changed your default shell). The problem with the second method is it may well wreck your batch environment, too, because the scheduler sets it up under the assumption that the login shell is ''sh''.<br />
<br />
The relationship between the ''csh'' and ''tcsh'' shells is similar to the one between ''sh'' and ''bash''. For instance, your ''csh'' shells are automatically endowed with the ''tcsh''-style ability to retrieve history through the up- and down-arrow keys. The best way to make ''tcsh'' into your everyday working shell is to run it on top of ''sh'' after you log in (again, you can do this as part of your .profile).<br />
<br />
References<br />
:* "man bash" from the command line.<br />
:* [http://tldp.org/LDP/abs/html/ Advanced Bash Scripting Guide], one of the Linux Documentation Project [http://tldp.org/guides.html guides]<br />
:* [http://mywiki.wooledge.org/BashFAQ Bash FAQ]<br />
:*[http://mywiki.wooledge.org/BashPitfalls Bash Pitfalls]<br />
<br />
====Compiling and linking code on Linux====<br />
{{:Compiling Code Linux}}<br />
<br />
====FAQ====<br />
=====How do I determine my program's dependencies on shared library (.so) files?=====<br />
:*ldd - see the man page.<br />
If your program cannot find all the .so files it needs, you may need to add paths to the LD_LIBRARY_PATH shell variable.<br />
<br />
=====How do I display an image file (such as jpeg or gif)?=====<br />
:*display mypic.jpg - uses one of the many ImageMagick tools - see "man ImageMagick" for help on this and various file format converters.<br />
:*firefox mypic.jpg - any decent Web browser can handle it.<br />
Note, the image will show up only if you have [[Getting_Started#Connect_to_Linux | X11 forwarding]] enabled.</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=File_transfer&diff=923File transfer2015-09-30T20:23:10Z<p>Ad876: /* Secure FTP */</p>
<hr />
<div>A single, central file server, storage01.cac.cornell.edu, serves all CAC user home directories. You can connect to this server in a variety of ways from any operating system to access your files. <br />
<br />
Note: by default, your home directory and its contents will be readable and executable by all other users of CAC systems. If this is not what you want, you can change the permissions of the home directory and its files and subdirectories via the standard Linux or Windows mechanisms. However, be aware that this may lead to conflicts for cross-platform applications, as Windows and Linux permissions are not 100% compatible.<br />
<br />
====Linux users====<br />
=====Secure Copy=====<br />
Secure copy is a standard tool to copy files to and from remote hosts.<br />
localhost$ scp localfile.dat username@linuxlogin.cac.cornell.edu:remoteinput.dat<br />
localhost$ scp username@linuxlogin.cac.cornell.edu:results.dat localresults.dat<br />
<br />
=====Secure FTP=====<br />
FTP is disabled for security reasons, but sftp's interface is nearly identical.<br />
=====Samba Client=====<br />
Type<br />
smbclient //storage01.cac.cornell.edu/<user name> -U <user name><br />
Enter the password for your CAC account when prompted<br />
You will see the '''''smb:\>''''' prompt. You can now start transferring files between your local machine and CAC home directory similar to ftp client. Type '''''help''''' for more instructions.<br />
-sh-3.2$ smbclient //storage01.cac.cornell.edu/<user name> -U <user name><br />
Password: <br />
Domain=[CTC_ITH] OS=[Unix] Server=[Samba 3.0.28-1.el5_2.1]<br />
smb: \> help<br />
<br />
====MacOS X users====<br />
<br />
====Windows users====<br />
=====Secure Copy=====<br />
The people who make Putty provide a secure copy client called [http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html pscp]. From the command prompt, type:<br />
cmd> pscp localfile.dat username@linuxlogin.cac.cornell.edu:remoteinput.dat<br />
<enter your username's password when prompted><br />
cmd> pscp username@linuxlogin.cac.cornell.edu:results.dat localresults.dat<br />
<br />
=====Secure FTP=====<br />
FTP is disabled for security reasons, but [http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html psftp's] interface is nearly identical. From the command prompt, type:<br />
cmd> psftp username@linuxlogin.cac.cornell.edu<br />
<enter your username's password when prompted><br />
psftp> put localresults.dat results.dat<br />
psftp> quit</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=File_transfer&diff=922File transfer2015-09-30T20:22:49Z<p>Ad876: /* Secure Copy */</p>
<hr />
<div>A single, central file server, storage01.cac.cornell.edu, serves all CAC user home directories. You can connect to this server in a variety of ways from any operating system to access your files. <br />
<br />
Note: by default, your home directory and its contents will be readable and executable by all other users of CAC systems. If this is not what you want, you can change the permissions of the home directory and its files and subdirectories via the standard Linux or Windows mechanisms. However, be aware that this may lead to conflicts for cross-platform applications, as Windows and Linux permissions are not 100% compatible.<br />
<br />
====Linux users====<br />
=====Secure Copy=====<br />
Secure copy is a standard tool to copy files to and from remote hosts.<br />
localhost$ scp localfile.dat username@linuxlogin.cac.cornell.edu:remoteinput.dat<br />
localhost$ scp username@linuxlogin.cac.cornell.edu:results.dat localresults.dat<br />
<br />
=====Secure FTP=====<br />
FTP is disabled for security reasons, but sftp's interface is nearly identical.<br />
=====Samba Client=====<br />
Type<br />
smbclient //storage01.cac.cornell.edu/<user name> -U <user name><br />
Enter the password for your CAC account when prompted<br />
You will see the '''''smb:\>''''' prompt. You can now start transferring files between your local machine and CAC home directory similar to ftp client. Type '''''help''''' for more instructions.<br />
-sh-3.2$ smbclient //storage01.cac.cornell.edu/<user name> -U <user name><br />
Password: <br />
Domain=[CTC_ITH] OS=[Unix] Server=[Samba 3.0.28-1.el5_2.1]<br />
smb: \> help<br />
<br />
====MacOS X users====<br />
<br />
====Windows users====<br />
=====Secure Copy=====<br />
The people who make Putty provide a secure copy client called [http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html pscp]. From the command prompt, type:<br />
cmd> pscp localfile.dat username@linuxlogin.cac.cornell.edu:remoteinput.dat<br />
<enter your username's password when prompted><br />
cmd> pscp username@linuxlogin.cac.cornell.edu:results.dat localresults.dat<br />
<br />
=====Secure FTP=====<br />
FTP is disabled for security reasons, but [//www.chiark.greenend.org.uk/~sgtatham/putty/download.html psftp's] interface is nearly identical. From the command prompt, type:<br />
cmd> psftp username@linuxlogin.cac.cornell.edu<br />
<enter your username's password when prompted><br />
psftp> put localresults.dat results.dat<br />
psftp> quit</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Connect_to_Windows&diff=921Connect to Windows2015-09-30T20:21:27Z<p>Ad876: </p>
<hr />
<div>'''''Using Remote Desktop Connection to connect to winlogin'''''<br />
<br />
This method of connecting to winlogin is preferred because it provides you with a fully functional Windows desktop. At the login screen, if the domain is specified, it should be set to '''''CTC_ITH''''', not the local name of the machine to which you are connecting.<br />
<br />
:'''Remote Desktop Connect Details''':<br />
<br />
Remote Desktop sessions do not expire, but they will end when machines are rebooted during down times.<br />
<br />
:* '''If you use a Windows machine''':<br />Use the Remote Desktop Connection (older name Terminal Services Client) to connect to a login machine. This software is pre-installed with Windows 7 and later. To run it, click <tt>Start</tt>, then <tt>All Programs > Accessories > Communications > Remote Desktop Connection</tt>. Otherwise you need to [//www.microsoft.com/windowsxp/downloads/tools/rdclientdl.mspx download the client] before you can use it.<br />
:*''' If you use Mac OS X 10.7 or later''':<br />Use the free download from [//itunes.apple.com/us/app/microsoft-remote-desktop/id715768417?mt=12 the Mac App Store]. Works just like the Remote Desktop Connection in Windows 7. You can also use rdesktop (see below). Tip: if authentication fails, make sure your software updates are current.<br />
:*''' If you use Unix or Linux or Mac''':<br />You can access the login machines by using the cross-platform rdesktop client. If you are running Linux, typically it is part of the distribution. If you prefer to build it yourself, it is available for download from [http://www.rdesktop.org/ rdesktop]. Executables for old versions are available from [//jacco2.dds.nl/rdesktop/index.html. here]</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Connect_to_Windows&diff=920Connect to Windows2015-09-30T20:20:07Z<p>Ad876: </p>
<hr />
<div>'''''Using Remote Desktop Connection to connect to winlogin'''''<br />
<br />
This method of connecting to winlogin is preferred because it provides you with a fully functional Windows desktop. At the login screen, if the domain is specified, it should be set to '''''CTC_ITH''''', not the local name of the machine to which you are connecting.<br />
<br />
:'''Remote Desktop Connect Details''':<br />
<br />
Remote Desktop sessions do not expire, but they will end when machines are rebooted during down times.<br />
<br />
:* '''If you use a Windows machine''':<br />Use the Remote Desktop Connection (older name Terminal Services Client) to connect to a login machine. This software is pre-installed with Windows 7 and later. To run it, click <tt>Start</tt>, then <tt>All Programs > Accessories > Communications > Remote Desktop Connection</tt>. Otherwise you need to [//www.microsoft.com/windowsxp/downloads/tools/rdclientdl.mspx download the client] before you can use it.<br />
:*''' If you use Mac OS X 10.7 or later''':<br />Use the free download from [//itunes.apple.com/us/app/microsoft-remote-desktop/id715768417?mt=12 the Mac App Store]. Works just like the Remote Desktop Connection in Windows 7. You can also use rdesktop (see below). Tip: if authentication fails, make sure your software updates are current.<br />
:*''' If you use Unix or Linux or Mac''':<br />You can access the login machines by using the cross-platform rdesktop client. If you are running Linux, typically it is part of the distribution. If you prefer to build it yourself, it is available for download from [//rdesktop.org/ rdesktop]. Executables for old versions are available from [//jacco2.dds.nl/rdesktop/index.html. here]</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Connect_to_Linux&diff=919Connect to Linux2015-09-30T20:18:17Z<p>Ad876: </p>
<hr />
<div>__TOC__<br />
There are three distinct ways to connect to a login node:<br />
# [[Getting_Started#Using_Secure_Shell | Use SSH]] to open a Linux shell on a login node, which provides a text-only interface.<br />
# [[Getting_Started#Using_Secure_Shell | Use SSH]] together with [[Getting_Started#Using_X-Windows | X-Windows]], which sends any interactive graphics back to your machine window-by-window through an SSH tunnel.<br />
# [[Getting_Started#Using_VNC | Use VNC]] to get a remote desktop with multiple text and graphics windows. This is not as straightforward as it sounds, due to the need to set up a secure tunnel for the remote desktop first.<br />
<br />
These instructions are intended mainly for users of personal computers and workstations. However, much of the material carries over to mobile computing platforms such as tablets and smartphones. You will have to locate and download an app to enable SSH or VNC connectivity; even a browser plug-in may suffice.<br />
<br />
Whichever method you choose, at your first login, you will be challenged for a new password. Find help at [[Getting_Started#Change_a_password_at_first_login | Changing a Password at First Login]]. You will also be asked for an ssh passphrase. You can just leave this blank; hit the <tt>Enter</tt> key in response.<br />
<br />
=====Using Secure Shell=====<br />
For basic command-line access, a Secure Shell (SSH) client will give you a remote command shell on one of the login nodes.<br />
:* Nearly all Unix/Linux varieties (including Mac) already have a built-in SSH2 implementation, required by our clusters.<br />
:* If you are coming from a Microsoft Windows machine, an SSH2 client must first be installed, as described below.<br />
:* The non-secure predecessor of SSH, telnet, is disabled for security reasons.<br />
<br />
'''''Linux users:'''''<br />
<br />
To connect to the second login node with ssh, you simply open a terminal window and type<br />
localhost$ ssh username@linuxlogin.cac.cornell.edu<br />
<br />
'''''Mac OS X users:'''''<br />
<br />
OS X on the Mac is built on a version of Unix, so ssh is available directly from the Terminal application.<br />
:*One option is to use the shortcut <tt>'''cmd-space'''</tt> to open Spotlight and then type <tt>"Terminal"</tt> to open a Terminal window.<br />
Otherwise:<br />
:*Navigate in the Finder to the Applications folder and Utilities sub-folder.<br />
:*:[[Image:MacApplicationsFolder.png|500px]]<br />
:*Then double-click on the Terminal application to see a Bash command-line.<br />
:*:[[Image:MacTerminalWindow.png]]<br />
:*As in Linux, simply type "ssh username@linuxlogin.cac.cornell.edu" into this window.<br />
<br />
'''''Windows users:'''''<br />
<br />
Secure Shell (ssh) clients work nicely as long as they support the SSH2 protocol. As mentioned, telnet is disabled for security reasons. A popular client for Windows is the free [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY client].<br />
:* The simplest installation is to download the [http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html Windows installer], called '''putty-0.65-installer.exe''', and run it. This installs PuTTY into your Start menu.<br />
:* To connect, start PuTTY, then type in a host name such as linuxlogin.cac.cornell.edu, and click "Open".<br />
:*:[[Image:Putty_address.jpg|Setting the host name in PuTTY]]<br />
<br />
=====Using X-Windows=====<br />
<br />
X-Windows or X11 is the longstanding Unix mechanism for displaying interactive graphics in a window. Your "X server" software runs locally, but it is capable of displaying windows that have been generated either locally or remotely. An "X client" on a remote machine can create X-Windows for local display, but it is necessary first to establish a shell on that machine using SSH.<br />
<br />
'''''Appropriate use'''''<br />
<br />
Among other things, X-Windows gives you the ability to display a GUI that originates on a login node. However, this ability does '''''NOT''''' imply that you are permitted to run compute-intensive, GUI-driven applications on these machines. Such usage is not only contrary to CAC policy, it is disrespectful toward other users, because the login node may become unresponsive through your actions.<br />
<br />
'''''Linux users:'''''<br />
<br />
The standard way to use X-Windows is to tunnel the X-Windows protocol through an ssh connection. If you open your ssh session with the '''-X''' option, it will automatically set up the necessary tunnel and environment variables.<br />
localhost$ ssh -X username@linuxlogin.cac.cornell.edu<br />
linuxlogin$ echo $DISPLAY<br />
localhost:11.0<br />
linuxlogin$ xclock&<br />
You can see that your DISPLAY environment variable is set and test it with xclock. There is another option to use a trusted version of X-windows forwarding<br />
linuxlogin$ ssh -Y compute-3-48.v4linux<br />
Th trusted version is necessary for forwarding X11 connections from a compute node to the login node, then back to your client machine.<br />
<br />
'''''Mac OS X users:'''''<br />
<br />
If you start ssh with the '''-X''' or '''-Y''' option, X-Windows should start up automatically. You can then try the "xclock" test, as described above for Linux. <br />
<br />
X11 is preinstalled on Macs starting with OS X 10.6 (Snow Leopard). For Mac OS X 10.5 (Leopard), you may need to install X11 in order for X-Windows applications to launch. If there is no X11 application in the Applications->Utilities folder, you'll have to find your OS X install disk. From the ''Mac OS X Server Introduction to Command-Line Administration,'' "The X11 server and an application to access X windows from the Finder are available as an optional installation in the Optional Installs folder of your installation disc (X11 is in the Applications package)."<br />
<br />
'''''Windows users:'''''<br />
<br />
Along with your ssh client (e.g., PuTTY), you will need to install an X-Windows server on your Windows machine.<br />
:* [http://www.straightrunning.com/XmingNotes/ Xming] - Open Source. A shareware contribution will get you a version with improved performance for graphics (GLX). There are two pieces to download<br />
:*:[[Image:Xming-download.jpg]]<br />
:** Xming-mesa (public domain release). There are two links together, one for Xming, one for Xming-mesa. Either will work, but Xming-mesa has some newer features that might come in handy some time.<br />
:** Xming-fonts (public domain release)<br />
If you purchase the website release of Xming, remember to install the Xming-fonts, as well.<br />
:* OpenText's [//cit.cornell.edu/services/software_licensing/available/exceed.cfm Exceed and Exceed 3D] - Cornell no longer has a site license. Installing Exceed 3D will improve performance of graphics applications. Exceed installs several icons under the Start menu. Choose the one that just says "Exceed" because it starts the program in multi-window mode, which is what we want.<br />
<br />
Here is how to start a session using PuTTY and Xming.<br />
<br />
# Start Xming from the Start menu. It will appear briefly and disappear except for an X in the application tray.<br />
# Start PuTTY.<br />
# In the window that appears, type a host name, <tt>linuxlogin.cac.cornell.edu</tt>.<br />
# Use the tree menu on the left to set X11 forwarding. It's in the <tt>Connection > SSH branch</tt>.<br />
#:[[Image:Putty_x11forwarding.jpg|Setting X11 forwarding in PuTTY]]<br />
# For PuTTY 0.61 only - In the "Auth" section of the SSH branch, go to GSSAPI and uncheck <tt>"Attempt GSSAPI authentication"</tt>. This will prevent an annoying <tt>"Access denied"</tt> message from appearing in your terminal window.<br />
# You can return to the Session category and Save this session's configuration for future use. Give it a logical name like linuxlogin.<br />
# Click Open, and it will connect to a login node.<br />
# Test your X-Windows setup by typing<br />
xclock<br />
You should see a clock appear in the corner of your screen. You can stop it by typing <tt>Ctrl-c</tt> in the terminal window.<br />
<br />
=====Using VNC=====<br />
<br />
[http://en.wikipedia.org/wiki/Vnc VNC] lets you see a whole Linux desktop from the login node on your computer.<br />
Using SSH and X-Windows is generally faster, and uses a lot less of the login node's resources,<br />
but VNC can be much faster if you are doing visualization on the login node from off campus.<br />
<br />
For security reasons, we are requiring all VNC connections to be tunneled inside ssh. You will therefore need to be able to connect to the login nodes [[Getting_Started#Using_Secure_Shell | using SSH]]. Because the firewall running on linuxlogin blocks all incoming ports except for ssh, VNC connections must be made over a ssh tunnel as described below.<br />
<br />
'''''Appropriate use'''''<br />
<br />
VNC gives you the ability to establish a remote desktop on the login nodes, but this ability does '''''NOT''''' imply that you are permitted to run compute-intensive, GUI-driven applications on these machines. Such usage is not only contrary to CAC policy, it is disrespectful toward other users, because the login node may become unresponsive through your actions.<br />
<br />
Here is a good example of how to use VNC appropriately. By following these steps you can run (say) Abaqus in GUI-driven mode on a compute node that has been allocated to you through an interactive batch job.<br />
# Open a VNC connection to linuxlogin through an ssh tunnel using the instructions below, in order to gain access to a Linux desktop. Make sure two terminal windows are available on this desktop.<br />
# In one of the terminal windows, submit an interactive job to the queue of your choice (add the #PBS -I directive to your job submission script).<br />
# Once the job starts, you will be given a command prompt on your assigned machine. Note the result of "hostname". There is no need to enter further commands at this prompt (except to exit the job).<br />
# Go to the other terminal window and open a second ssh connection to the compute node using "ssh -Y <userid>@<hostname>"<br />
# This new ssh session will tunnel X-Windows from the compute node back to the VNC desktop. Therefore (if Abaqus is on your path), you can now open the Abaqus GUI using "abaqus cae -mesa".<br />
<br />
'''''Initial setup'''''<br />
''(You only need to do this once)''<br />
<br />
:* Install a VNC client if one isn't installed. [http://www.tightvnc.com/ TightVNC] works well, but so do others.<br />
:* Login to linuxlogin, and set the password for your VNC server using the "vncpasswd" command.<br />
<br />
'''''Start your VNC server'''''<br />
<br />
:* '''On linuxlogin''', start the VNC server using the "vncserver" command like this:<br />
vncserver -geometry 1024x768 -localhost<br />
The geometry numbers, 1024x768, specify the size, in pixels, of the desktop.<br />
:* You will need to get the display number from the output of the vncserver command:<br />
<br />
New 'linuxlogin.cac.cornell.edu:1 (shl1)' desktop is linuxlogin.cac.cornell.edu:1<br />
Starting applications specified in /home/gfs01/shl1/.vnc/xstartup<br />
Log file is /home/gfs01/shl1/.vnc/linuxlogin.cac.cornell.edu:1.log<br />
<br />
:* vncserver is running on port 5900 + display number. In the above example, the display number is :1, therefore vncserver is running on port 5901.<br />
<br />
'''''Connect your VNC client'''''<br />
<br />
:* Set up ssh forwarding on your client computer. Let's say the port number on linuxlogin is 5901 (as above), and your CAC userid is uid12. From Linux, type into a terminal:<br />
<br />
ssh -L 10000:localhost:5901 uid12@linuxlogin.cac.cornell.edu<br />
'''From Windows''', ssh clients such as PuTTY can do X11 port forwarding. See [[VNC Tunnel Windows]]. <br />
<br />
'''For MacOS X users''', see [[CAC VPN Server for MacOS Users |here]]<br />
:* Leave this ssh session running on your local client computer. (It can run in the background.)<br />
:* Launch your VNC client program. Connect to localhost:10000. When prompted, type in your VNC server password.<br />
<br />
'''''To disconnect your client'''''<br />
<br />
:* Close the vnc client program.<br />
:* Disconnect the ssh forwarding session (i.e., kill it).<br />
<br />
'''''To reconnect your client'''''<br />
<br />
:* Restart port forwarding with ssh, using the same remote port number as before.<br />
:* Again connect the VNC client to localhost:10000.<br />
<br />
'''''When you are all done'''''<br />
<br />
:* On linuxlogin, type this command to shut down the VNC server<br />
vncserver -kill :<display number><br />
:* If you merely log out from linuxlogin, it will leave the VNC server running. You must shut down the VNC server explicitly when you are finished with it. (Actually this can be a nice feature.)<br />
<br />
=====Passwordless SSH=====<br />
<br />
''''' Create ssh key pair '''''<br />
<br />
Your ssh key pair will only need to be created once. You will not need to repeat this step. You can complete this step from either a Linux or Windows login node. If this is your first login to a CAC login node, it will ask you to [[Getting_Started#Change_a_password_at_first_login|change your password]]. This will become your password for connecting to the nodes. <br />
<br />
Create your ssh key pair by logging into the linux login node (linuxlogin.cac.cornell.edu), which will begin the process of creating the keys; you can use the defaults or empty responses for all prompts.<br />
<br />
Alternatively, you can create your ssh key pair on the linux login node by logging into the Windows login node (winx64login.cac.cornell.edu), opening a Command Prompt window, and running <tt>plink.exe</tt> to connect to the linux login node, as shown in this example:<br />
<br />
<pre>>"C:\Programs Files (x86)\Putty\plink.exe" %USERNAME%@linuxlogin.cac.cornell.edu<br />
Password: Enter Your Password<br />
Rocks 5.0 (V)<br />
Profile built 12:54 06-May-2008<br />
<br />
Kickstarted 09:22 06-May-2008<br />
-----------------------------------------------------------<br />
Welcome to the Center for Advanced Computing Cluster!<br />
-----------------------------------------------------------<br />
Please send your questions to help@cac.cornell.edu<br />
-----------------------------------------------------------<br />
<br />
<br />
It doesn't appear that you have set up your ssh key.<br />
This process will make the files:<br />
/home/gfs01/cacshl1/.ssh/id_rsa.pub<br />
/home/gfs01/cacshl1/.ssh/id_rsa<br />
/home/gfs01/cacshl1/.ssh/authorized_keys<br />
<br />
Generating public/private rsa key pair.<br />
Enter file in which to save the key (/home/gfs01/cacshl1/.ssh/id_rsa): Press Enter to accept default<br />
Created directory '/home/gfs01/cacshl1/.ssh'.<br />
Enter passphrase (empty for no passphrase): Press Enter to accept default<br />
Enter same passphrase again: Press Enter to accept default<br />
Your identification has been saved in /home/gfs01/cacshl1/.ssh/id_rsa.<br />
Your public key has been saved in /home/gfs01/cacshl1/.ssh/id_rsa.pub.<br />
</pre><br />
<br />
After this is done, type '''"exit"''' to log out of the linux login node.<br />
<br />
''''' Convert ssh Private Key for Putty / Plink '''''<br />
<br />
Next run PuTTYgen to generate public and private keys to be used with PuTTY and Plink:<br />
<br />
:* Log in to <tt>winx64login.tc.cornell.edu </tt>(if you are not already)<br />
:* Run <tt>C:\Program Files (x86)\Putty\puttygen.exe</tt>.<br />
:* Select <tt>Import Key</tt> from the <tt>Conversions</tt> menu and select <tt>H:\.ssh\id_rsa</tt> in your home directory. And click on the <tt>Open</tt> button.<br />
<center>[[image:LoadPrivateKey.jpg]]</center><br />
:* Click on the <tt>"Save Private Key"</tt> button. <br />
<center>[[image:SavePrivateKey.jpg]]</center><br />
:* Click on "Yes" when asked to save the private key without a passphrase.<br />
:* Save the private key as private.ppk in the .ssh directory inside your home directory.<br />
<center>[[image:SpecifyPrivateKey.jpg]]</center><br />
:* Close (choose File, then Exit)<br />
:* To confirm you have converted the ssh private key successfully, do:<br />
<pre>"C:\Program Files (x86)\Putty\plink.exe" -i %HOMEDRIVE%\.ssh\private.ppk %USERNAME%@linuxlogin.cac.cornell.edu</pre><br />
It may notify you that "The server's host key is not cached in the registry." Type "y" to "store the key in cache."<br />
:* You should now be logged into linuxlogin without being prompted for a password. Stay logged in for the next step.</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Connect_to_Linux&diff=918Connect to Linux2015-09-30T20:16:45Z<p>Ad876: </p>
<hr />
<div>__TOC__<br />
There are three distinct ways to connect to a login node:<br />
# [[Getting_Started#Using_Secure_Shell | Use SSH]] to open a Linux shell on a login node, which provides a text-only interface.<br />
# [[Getting_Started#Using_Secure_Shell | Use SSH]] together with [[Getting_Started#Using_X-Windows | X-Windows]], which sends any interactive graphics back to your machine window-by-window through an SSH tunnel.<br />
# [[Getting_Started#Using_VNC | Use VNC]] to get a remote desktop with multiple text and graphics windows. This is not as straightforward as it sounds, due to the need to set up a secure tunnel for the remote desktop first.<br />
<br />
These instructions are intended mainly for users of personal computers and workstations. However, much of the material carries over to mobile computing platforms such as tablets and smartphones. You will have to locate and download an app to enable SSH or VNC connectivity; even a browser plug-in may suffice.<br />
<br />
Whichever method you choose, at your first login, you will be challenged for a new password. Find help at [[Getting_Started#Change_a_password_at_first_login | Changing a Password at First Login]]. You will also be asked for an ssh passphrase. You can just leave this blank; hit the <tt>Enter</tt> key in response.<br />
<br />
=====Using Secure Shell=====<br />
For basic command-line access, a Secure Shell (SSH) client will give you a remote command shell on one of the login nodes.<br />
:* Nearly all Unix/Linux varieties (including Mac) already have a built-in SSH2 implementation, required by our clusters.<br />
:* If you are coming from a Microsoft Windows machine, an SSH2 client must first be installed, as described below.<br />
:* The non-secure predecessor of SSH, telnet, is disabled for security reasons.<br />
<br />
'''''Linux users:'''''<br />
<br />
To connect to the second login node with ssh, you simply open a terminal window and type<br />
localhost$ ssh username@linuxlogin.cac.cornell.edu<br />
<br />
'''''Mac OS X users:'''''<br />
<br />
OS X on the Mac is built on a version of Unix, so ssh is available directly from the Terminal application.<br />
:*One option is to use the shortcut <tt>'''cmd-space'''</tt> to open Spotlight and then type <tt>"Terminal"</tt> to open a Terminal window.<br />
Otherwise:<br />
:*Navigate in the Finder to the Applications folder and Utilities sub-folder.<br />
:*:[[Image:MacApplicationsFolder.png|500px]]<br />
:*Then double-click on the Terminal application to see a Bash command-line.<br />
:*:[[Image:MacTerminalWindow.png]]<br />
:*As in Linux, simply type "ssh username@linuxlogin.cac.cornell.edu" into this window.<br />
<br />
'''''Windows users:'''''<br />
<br />
Secure Shell (ssh) clients work nicely as long as they support the SSH2 protocol. As mentioned, telnet is disabled for security reasons. A popular client for Windows is the free [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY client].<br />
:* The simplest installation is to download the [http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html Windows installer], called '''putty-0.65-installer.exe''', and run it. This installs PuTTY into your Start menu.<br />
:* To connect, start PuTTY, then type in a host name such as linuxlogin.cac.cornell.edu, and click "Open".<br />
:*:[[Image:Putty_address.jpg|Setting the host name in PuTTY]]<br />
<br />
=====Using X-Windows=====<br />
<br />
X-Windows or X11 is the longstanding Unix mechanism for displaying interactive graphics in a window. Your "X server" software runs locally, but it is capable of displaying windows that have been generated either locally or remotely. An "X client" on a remote machine can create X-Windows for local display, but it is necessary first to establish a shell on that machine using SSH.<br />
<br />
'''''Appropriate use'''''<br />
<br />
Among other things, X-Windows gives you the ability to display a GUI that originates on a login node. However, this ability does '''''NOT''''' imply that you are permitted to run compute-intensive, GUI-driven applications on these machines. Such usage is not only contrary to CAC policy, it is disrespectful toward other users, because the login node may become unresponsive through your actions.<br />
<br />
'''''Linux users:'''''<br />
<br />
The standard way to use X-Windows is to tunnel the X-Windows protocol through an ssh connection. If you open your ssh session with the '''-X''' option, it will automatically set up the necessary tunnel and environment variables.<br />
localhost$ ssh -X username@linuxlogin.cac.cornell.edu<br />
linuxlogin$ echo $DISPLAY<br />
localhost:11.0<br />
linuxlogin$ xclock&<br />
You can see that your DISPLAY environment variable is set and test it with xclock. There is another option to use a trusted version of X-windows forwarding<br />
linuxlogin$ ssh -Y compute-3-48.v4linux<br />
Th trusted version is necessary for forwarding X11 connections from a compute node to the login node, then back to your client machine.<br />
<br />
'''''Mac OS X users:'''''<br />
<br />
If you start ssh with the '''-X''' or '''-Y''' option, X-Windows should start up automatically. You can then try the "xclock" test, as described above for Linux. <br />
<br />
X11 is preinstalled on Macs starting with OS X 10.6 (Snow Leopard). For Mac OS X 10.5 (Leopard), you may need to install X11 in order for X-Windows applications to launch. If there is no X11 application in the Applications->Utilities folder, you'll have to find your OS X install disk. From the ''Mac OS X Server Introduction to Command-Line Administration,'' "The X11 server and an application to access X windows from the Finder are available as an optional installation in the Optional Installs folder of your installation disc (X11 is in the Applications package)."<br />
<br />
'''''Windows users:'''''<br />
<br />
Along with your ssh client (e.g., PuTTY), you will need to install an X-Windows server on your Windows machine.<br />
:* [//www.straightrunning.com/XmingNotes/ Xming] - Open Source. A shareware contribution will get you a version with improved performance for graphics (GLX). There are two pieces to download<br />
:*:[[Image:Xming-download.jpg]]<br />
:** Xming-mesa (public domain release). There are two links together, one for Xming, one for Xming-mesa. Either will work, but Xming-mesa has some newer features that might come in handy some time.<br />
:** Xming-fonts (public domain release)<br />
If you purchase the website release of Xming, remember to install the Xming-fonts, as well.<br />
:* OpenText's [//cit.cornell.edu/services/software_licensing/available/exceed.cfm Exceed and Exceed 3D] - Cornell no longer has a site license. Installing Exceed 3D will improve performance of graphics applications. Exceed installs several icons under the Start menu. Choose the one that just says "Exceed" because it starts the program in multi-window mode, which is what we want.<br />
<br />
Here is how to start a session using PuTTY and Xming.<br />
<br />
# Start Xming from the Start menu. It will appear briefly and disappear except for an X in the application tray.<br />
# Start PuTTY.<br />
# In the window that appears, type a host name, <tt>linuxlogin.cac.cornell.edu</tt>.<br />
# Use the tree menu on the left to set X11 forwarding. It's in the <tt>Connection > SSH branch</tt>.<br />
#:[[Image:Putty_x11forwarding.jpg|Setting X11 forwarding in PuTTY]]<br />
# For PuTTY 0.61 only - In the "Auth" section of the SSH branch, go to GSSAPI and uncheck <tt>"Attempt GSSAPI authentication"</tt>. This will prevent an annoying <tt>"Access denied"</tt> message from appearing in your terminal window.<br />
# You can return to the Session category and Save this session's configuration for future use. Give it a logical name like linuxlogin.<br />
# Click Open, and it will connect to a login node.<br />
# Test your X-Windows setup by typing<br />
xclock<br />
You should see a clock appear in the corner of your screen. You can stop it by typing <tt>Ctrl-c</tt> in the terminal window.<br />
<br />
=====Using VNC=====<br />
<br />
[http://en.wikipedia.org/wiki/Vnc VNC] lets you see a whole Linux desktop from the login node on your computer.<br />
Using SSH and X-Windows is generally faster, and uses a lot less of the login node's resources,<br />
but VNC can be much faster if you are doing visualization on the login node from off campus.<br />
<br />
For security reasons, we are requiring all VNC connections to be tunneled inside ssh. You will therefore need to be able to connect to the login nodes [[Getting_Started#Using_Secure_Shell | using SSH]]. Because the firewall running on linuxlogin blocks all incoming ports except for ssh, VNC connections must be made over a ssh tunnel as described below.<br />
<br />
'''''Appropriate use'''''<br />
<br />
VNC gives you the ability to establish a remote desktop on the login nodes, but this ability does '''''NOT''''' imply that you are permitted to run compute-intensive, GUI-driven applications on these machines. Such usage is not only contrary to CAC policy, it is disrespectful toward other users, because the login node may become unresponsive through your actions.<br />
<br />
Here is a good example of how to use VNC appropriately. By following these steps you can run (say) Abaqus in GUI-driven mode on a compute node that has been allocated to you through an interactive batch job.<br />
# Open a VNC connection to linuxlogin through an ssh tunnel using the instructions below, in order to gain access to a Linux desktop. Make sure two terminal windows are available on this desktop.<br />
# In one of the terminal windows, submit an interactive job to the queue of your choice (add the #PBS -I directive to your job submission script).<br />
# Once the job starts, you will be given a command prompt on your assigned machine. Note the result of "hostname". There is no need to enter further commands at this prompt (except to exit the job).<br />
# Go to the other terminal window and open a second ssh connection to the compute node using "ssh -Y <userid>@<hostname>"<br />
# This new ssh session will tunnel X-Windows from the compute node back to the VNC desktop. Therefore (if Abaqus is on your path), you can now open the Abaqus GUI using "abaqus cae -mesa".<br />
<br />
'''''Initial setup'''''<br />
''(You only need to do this once)''<br />
<br />
:* Install a VNC client if one isn't installed. [http://www.tightvnc.com/ TightVNC] works well, but so do others.<br />
:* Login to linuxlogin, and set the password for your VNC server using the "vncpasswd" command.<br />
<br />
'''''Start your VNC server'''''<br />
<br />
:* '''On linuxlogin''', start the VNC server using the "vncserver" command like this:<br />
vncserver -geometry 1024x768 -localhost<br />
The geometry numbers, 1024x768, specify the size, in pixels, of the desktop.<br />
:* You will need to get the display number from the output of the vncserver command:<br />
<br />
New 'linuxlogin.cac.cornell.edu:1 (shl1)' desktop is linuxlogin.cac.cornell.edu:1<br />
Starting applications specified in /home/gfs01/shl1/.vnc/xstartup<br />
Log file is /home/gfs01/shl1/.vnc/linuxlogin.cac.cornell.edu:1.log<br />
<br />
:* vncserver is running on port 5900 + display number. In the above example, the display number is :1, therefore vncserver is running on port 5901.<br />
<br />
'''''Connect your VNC client'''''<br />
<br />
:* Set up ssh forwarding on your client computer. Let's say the port number on linuxlogin is 5901 (as above), and your CAC userid is uid12. From Linux, type into a terminal:<br />
<br />
ssh -L 10000:localhost:5901 uid12@linuxlogin.cac.cornell.edu<br />
'''From Windows''', ssh clients such as PuTTY can do X11 port forwarding. See [[VNC Tunnel Windows]]. <br />
<br />
'''For MacOS X users''', see [[CAC VPN Server for MacOS Users |here]]<br />
:* Leave this ssh session running on your local client computer. (It can run in the background.)<br />
:* Launch your VNC client program. Connect to localhost:10000. When prompted, type in your VNC server password.<br />
<br />
'''''To disconnect your client'''''<br />
<br />
:* Close the vnc client program.<br />
:* Disconnect the ssh forwarding session (i.e., kill it).<br />
<br />
'''''To reconnect your client'''''<br />
<br />
:* Restart port forwarding with ssh, using the same remote port number as before.<br />
:* Again connect the VNC client to localhost:10000.<br />
<br />
'''''When you are all done'''''<br />
<br />
:* On linuxlogin, type this command to shut down the VNC server<br />
vncserver -kill :<display number><br />
:* If you merely log out from linuxlogin, it will leave the VNC server running. You must shut down the VNC server explicitly when you are finished with it. (Actually this can be a nice feature.)<br />
<br />
=====Passwordless SSH=====<br />
<br />
''''' Create ssh key pair '''''<br />
<br />
Your ssh key pair will only need to be created once. You will not need to repeat this step. You can complete this step from either a Linux or Windows login node. If this is your first login to a CAC login node, it will ask you to [[Getting_Started#Change_a_password_at_first_login|change your password]]. This will become your password for connecting to the nodes. <br />
<br />
Create your ssh key pair by logging into the linux login node (linuxlogin.cac.cornell.edu), which will begin the process of creating the keys; you can use the defaults or empty responses for all prompts.<br />
<br />
Alternatively, you can create your ssh key pair on the linux login node by logging into the Windows login node (winx64login.cac.cornell.edu), opening a Command Prompt window, and running <tt>plink.exe</tt> to connect to the linux login node, as shown in this example:<br />
<br />
<pre>>"C:\Programs Files (x86)\Putty\plink.exe" %USERNAME%@linuxlogin.cac.cornell.edu<br />
Password: Enter Your Password<br />
Rocks 5.0 (V)<br />
Profile built 12:54 06-May-2008<br />
<br />
Kickstarted 09:22 06-May-2008<br />
-----------------------------------------------------------<br />
Welcome to the Center for Advanced Computing Cluster!<br />
-----------------------------------------------------------<br />
Please send your questions to help@cac.cornell.edu<br />
-----------------------------------------------------------<br />
<br />
<br />
It doesn't appear that you have set up your ssh key.<br />
This process will make the files:<br />
/home/gfs01/cacshl1/.ssh/id_rsa.pub<br />
/home/gfs01/cacshl1/.ssh/id_rsa<br />
/home/gfs01/cacshl1/.ssh/authorized_keys<br />
<br />
Generating public/private rsa key pair.<br />
Enter file in which to save the key (/home/gfs01/cacshl1/.ssh/id_rsa): Press Enter to accept default<br />
Created directory '/home/gfs01/cacshl1/.ssh'.<br />
Enter passphrase (empty for no passphrase): Press Enter to accept default<br />
Enter same passphrase again: Press Enter to accept default<br />
Your identification has been saved in /home/gfs01/cacshl1/.ssh/id_rsa.<br />
Your public key has been saved in /home/gfs01/cacshl1/.ssh/id_rsa.pub.<br />
</pre><br />
<br />
After this is done, type '''"exit"''' to log out of the linux login node.<br />
<br />
''''' Convert ssh Private Key for Putty / Plink '''''<br />
<br />
Next run PuTTYgen to generate public and private keys to be used with PuTTY and Plink:<br />
<br />
:* Log in to <tt>winx64login.tc.cornell.edu </tt>(if you are not already)<br />
:* Run <tt>C:\Program Files (x86)\Putty\puttygen.exe</tt>.<br />
:* Select <tt>Import Key</tt> from the <tt>Conversions</tt> menu and select <tt>H:\.ssh\id_rsa</tt> in your home directory. And click on the <tt>Open</tt> button.<br />
<center>[[image:LoadPrivateKey.jpg]]</center><br />
:* Click on the <tt>"Save Private Key"</tt> button. <br />
<center>[[image:SavePrivateKey.jpg]]</center><br />
:* Click on "Yes" when asked to save the private key without a passphrase.<br />
:* Save the private key as private.ppk in the .ssh directory inside your home directory.<br />
<center>[[image:SpecifyPrivateKey.jpg]]</center><br />
:* Close (choose File, then Exit)<br />
:* To confirm you have converted the ssh private key successfully, do:<br />
<pre>"C:\Program Files (x86)\Putty\plink.exe" -i %HOMEDRIVE%\.ssh\private.ppk %USERNAME%@linuxlogin.cac.cornell.edu</pre><br />
It may notify you that "The server's host key is not cached in the registry." Type "y" to "store the key in cache."<br />
:* You should now be logged into linuxlogin without being prompted for a password. Stay logged in for the next step.</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Getting_Started&diff=914Getting Started2015-09-30T20:09:03Z<p>Ad876: /* Managing your password */</p>
<hr />
<div>__TOC__<br />
==Managing your password==<br />
CAC has a [[Getting_Started#Rules_for_Creating_passwords|Password Policy]] in effect. The first time that you login to the <tt>cac.cornell.edu</tt> domain, you will be required to change your password. Each password must have at least eight characters and must contain at least three of the following four elements: (1) uppercase letters (2) lowercase letters (3) special characters (4) digits. Your password can be set or changed on any of the CAC login nodes, and the password will be updated on all CAC resources. Passwords expire every six months. Do not share your password. There are more detailed instructions below.<br />
<br />
===Rules for Creating passwords===<br />
{{:Rules for Creating Passwords}}<br />
<br />
===Change a password at first login===<br />
{{:Changing password at first login}}<br />
<br />
=== Change password at any time ===<br />
{{:Changing password any time}}<br />
<br />
=== Locked Accounts===<br />
<br />
There have been instances in which user accounts have been locked. Some common causes of locked accounts and the solutions are:<br />
<br />
:*Mistyping your password several times in a row. <br />
:::<tt>Solution</tt>: Wait about a 1/2 hour and then try again. Be sure that your caps lock key is not on!<br />
:*Trying to login to a Windows login node by using SSH when you have a new or expired password. <br />
:::<tt>Solution</tt>: Login to a Windows login node using Remote Desktop Connection or SSH to a linux login node.<br />
:*Failing to log off all other sessions connected to login nodes. <br />
:::<tt>Solution</tt>: Log off all remote connections. Disconnecting the sessions is not enough.<br />
:*Failing to disconnect locally mapped drives to the CAC file server before changing your password. <br />
:::<tt>Solution</tt>: Disconnect all locally mapped drives, wait a 1/2 hour until account is unlocked, and then re-map the drive with the new password.<br />
<br />
If you can't log on or can't wait you can submit a Password Reset ticket on our [https://rt.cac.cornell.edu/index.html issue tracking system]<br />
<br />
==Checking your CAC project ==<br />
Cornell University users can view their account limits at [https://{{SERVERNAME}}/services/cu/memberlimits.aspx CAC Account Limits].<br /><br />
Partner Program members should contact Paul Redfern at [mailto:red@cac.cornell.edu red@cac.cornell.edu] if they need information on their membership limits.<br />
<br />
==Using CAC resources==<br />
===Connecting to CAC===<br />
<br />
There are two types of login nodes:<br />
:* Linux login nodes: <tt>linuxlogin.cac.cornell.edu</tt> as well as the head nodes for the various Linux-based private clusters.<br />
:* Windows login node: <tt>winlogin.cac.cornell.edu</tt><br />
<br />
====Connect to Linux====<br />
<br />
{{:Connect to Linux}}<br />
<br />
====Connect to Windows====<br />
<br />
{{:Connect to Windows}}<br />
<br />
===Home Directory Access===<br />
{{:Home Directory Access}}<br />
<br />
===File transfer===<br />
{{:File transfer}}<br />
<br />
<br />
===Linux Usage Tips===<br />
{{:Linux Usage Tips}}<br />
==== More information on Linux nodes at CAC====<br />
For more detailed instructions on how to use the Linux node, see [[Tutorial for the Linux nodes at CAC| here]]<br />
<br />
===Windows Usage Tips===<br />
{{:Windows Usage Tips}}<br />
<br />
===More information===<br />
The CAC Web site is [https://{{SERVERNAME}}/ here] . There are many useful documents on the Support page at [[ Main Page| CAC documentation]].<br />
<br />
===Acknowledging CAC===<br />
{{:Acknowledging CAC}}<br />
<br />
===FAQ/Troubleshooting===<br />
#[[FAQ#Account| Account FAQ]]<br />
#[[FAQ#Login| Login FAQ]]<br />
# If you have more questions, see [[FAQ| here]]</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Archival_Storage&diff=911Archival Storage2015-09-30T18:50:23Z<p>Ad876: </p>
<hr />
<div>== What is CAC Archival Storage? ==<br />
:* CAC Archival Storage is a low-cost, high-performance option for storing research data '''available only to users within Cornell University'''. <br />
:* CAC Archival Storage is not mountable by running jobs, instead the user must transfer their data from the CAC Archival Storage to an accessible server using [//globusonline.org/ Globus Online]. <br />
:* Globus Online users have easy access to add, delete, and share their data using any Globus Online endpoints. <br />
:* Some of the Globus Online endpoints available include:<br />
:** storage01.cac.cornell.edu (cac#home) where all CAC user home directories are found<br />
:** XSEDE sites:<br />
:*** Stampede (xsede#stampede) <br />
:*** Lonestar (xsede#lonestar4) <br />
:*** TACC Archival Storage (xsede#ranch).<br />
<br />
== First step - Enable (or create) CAC project for Archival Storage and add users where appropriate ==<br />
:* To use the CAC Archival Storage service, '''you must be a user of a CAC project where Archival Storage is enabled'''.<br />
:* The project PI can add users and verify that Archival Storage is enabled at the [https://{{SERVERNAME}}/Services/Projects/manage.aspx Manage CAC project page].<br />
:* Don't have a project? [https://{{SERVERNAME}}/Services/projects.aspx How to start a CAC project?].<br />
<br />
== Second step - create your Globus Online account ==<br />
<br />
[//globus.org/SignUp Sign up] for a Globus account. CAC's Archival system is '''only accessible''' through '''[//globusonline.org/ Globus Online]'''.<br />
<br />
== CAC specifics ==<br />
=== Technical Information ===<br />
CAC's EndPoint is <b>cac#archive01</b>.<br />
<br />
:*When activating cac#archive01 endpoint in Globus Online web GUI, you will be prompted by a dialog box saying: <br />
<br />
<blockquote>The administrator of this endpoint, cac#archive01, requires that you authenticate using their MyProxy OAuth server to activate the endpoint. When you click 'Continue' you will be redirected to their website.</blockquote><br />
<br />
:*You will be redirected to the <nowiki>https://archive01.cac.cornell.edu/oath/authorize...</nowiki> page. <br />
:*Enter your CAC credentials.<br />
:*When login is successful, you will be redirected back to Globus Online web GUI with the endpoint activated.<br />
<br />
=== Administrative Information ===<br />
:* cac#archive01's default path is '''/export'''.<br />
:* Each project with access to CAC Archival Storage has a shared directory (named the project) in which '''all project members have full read/write access'''.<br />
:* Users can rename and move files and directories within their project directory on the endpoint. Globus Online added this feature recently.<br />
<br />
==Automated Archival==<br />
:* Install Globus Connect Personal on the Linux/MacOS/Windows host you wish to archive by clicking on the "Get Globus Connect Personal" link on the Transfer Files screen on Globus. <br />
::[[File:Install_Globus_Connect_Personal.jpg]]<br />
:* On the host you wish to archive, download and untar [[Media:archive_scripts.tar.gz]].<br />
:* To enable running Globus Connect Personal as root, add <br />
<br />
"-allow-root",<br />
<br />
::to globusconnectpersonal-2.0.3/gc.py (on line ~ 360):<br />
<source lang="c"><br />
args = [os.path.basename(PDEATH_LAUNCH),<br />
GRIDFTP_SERVER,<br />
"-allow-root",<br />
"-i", "-always-send-markers",<br />
"-hostname", "127.0.0.1",<br />
</source><br />
:* Copy root-bin directory from the archive_scripts.tar.gz to /root/bin. If you are archiving directories outside /home, modify the -restrict-path argument in /root/bin/gc_start.sh.<br />
:* Generate a ssh key pair using the "ssh-keygen" command, leave private key in ~/.ssh, and upload the private key to Globus<br />
::[[File:Upload_ssh_private_key.jpg]]<br />
:* Make sure you can access Globus CLI like this:<br />
ssh -i .ssh/<private key> <globus user name>@cli.globusonline.org<br />
:* Modify archive.sh to match your Globus user name, private key file name, CAC project and archive directory.<br />
:* On Globus, make sure your connection to cac#archive01 endpoint is activated. <br />
:* You should now be able to run archive.sh to upload your archive directory to CAC archive. You can automate this script using cron.<br />
<br />
==Globus Online links ==<br />
:*[//globus.org/how-it-works How Globus Online works?]<br />
:*[//globusonline.org/quickstart/ Globus Online Quickstart] guide for setting up an account.<br />
:*[//support.globus.org/entries/23583857-Sign-Up-and-Transfer-Files-with-Globus-Online Transfer Data using Globus Online]<br />
:*[//support.globus.org/entries/23602336-Sharing-Data-using-Globus Sharing Data using Globus Online]<br />
:*[//globus.org/group-management Group Management] NOTE: A Globus Plus or a Globus Provider plan are required; CAC's Globus endpoint '''does not''' '''support''' Group Management.<br />
==Syncing to Archival Storage==<br />
See [[Syncing_to_Archival_Storage| here]] for how to sun to Archival Storage</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Connect_to_Linux&diff=909Connect to Linux2015-09-30T18:16:31Z<p>Ad876: </p>
<hr />
<div>__TOC__<br />
There are three distinct ways to connect to a login node:<br />
# [[Getting_Started#Using_Secure_Shell | Use SSH]] to open a Linux shell on a login node, which provides a text-only interface.<br />
# [[Getting_Started#Using_Secure_Shell | Use SSH]] together with [[Getting_Started#Using_X-Windows | X-Windows]], which sends any interactive graphics back to your machine window-by-window through an SSH tunnel.<br />
# [[Getting_Started#Using_VNC | Use VNC]] to get a remote desktop with multiple text and graphics windows. This is not as straightforward as it sounds, due to the need to set up a secure tunnel for the remote desktop first.<br />
<br />
These instructions are intended mainly for users of personal computers and workstations. However, much of the material carries over to mobile computing platforms such as tablets and smartphones. You will have to locate and download an app to enable SSH or VNC connectivity; even a browser plug-in may suffice.<br />
<br />
Whichever method you choose, at your first login, you will be challenged for a new password. Find help at [[Getting_Started#Change_a_password_at_first_login | Changing a Password at First Login]]. You will also be asked for an ssh passphrase. You can just leave this blank; hit the <tt>Enter</tt> key in response.<br />
<br />
=====Using Secure Shell=====<br />
For basic command-line access, a Secure Shell (SSH) client will give you a remote command shell on one of the login nodes.<br />
:* Nearly all Unix/Linux varieties (including Mac) already have a built-in SSH2 implementation, required by our clusters.<br />
:* If you are coming from a Microsoft Windows machine, an SSH2 client must first be installed, as described below.<br />
:* The non-secure predecessor of SSH, telnet, is disabled for security reasons.<br />
<br />
'''''Linux users:'''''<br />
<br />
To connect to the second login node with ssh, you simply open a terminal window and type<br />
localhost$ ssh username@linuxlogin.cac.cornell.edu<br />
<br />
'''''Mac OS X users:'''''<br />
<br />
OS X on the Mac is built on a version of Unix, so ssh is available directly from the Terminal application.<br />
:*One option is to use the shortcut <tt>'''cmd-space'''</tt> to open Spotlight and then type <tt>"Terminal"</tt> to open a Terminal window.<br />
Otherwise:<br />
:*Navigate in the Finder to the Applications folder and Utilities sub-folder.<br />
:*:[[Image:MacApplicationsFolder.png|500px]]<br />
:*Then double-click on the Terminal application to see a Bash command-line.<br />
:*:[[Image:MacTerminalWindow.png]]<br />
:*As in Linux, simply type "ssh username@linuxlogin.cac.cornell.edu" into this window.<br />
<br />
'''''Windows users:'''''<br />
<br />
Secure Shell (ssh) clients work nicely as long as they support the SSH2 protocol. As mentioned, telnet is disabled for security reasons. A popular client for Windows is the free [//www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY client].<br />
:* The simplest installation is to download the [//chiark.greenend.org.uk/~sgtatham/putty/download.html Windows installer], called '''putty-0.65-installer.exe''', and run it. This installs PuTTY into your Start menu.<br />
:* To connect, start PuTTY, then type in a host name such as linuxlogin.cac.cornell.edu, and click "Open".<br />
:*:[[Image:Putty_address.jpg|Setting the host name in PuTTY]]<br />
<br />
=====Using X-Windows=====<br />
<br />
X-Windows or X11 is the longstanding Unix mechanism for displaying interactive graphics in a window. Your "X server" software runs locally, but it is capable of displaying windows that have been generated either locally or remotely. An "X client" on a remote machine can create X-Windows for local display, but it is necessary first to establish a shell on that machine using SSH.<br />
<br />
'''''Appropriate use'''''<br />
<br />
Among other things, X-Windows gives you the ability to display a GUI that originates on a login node. However, this ability does '''''NOT''''' imply that you are permitted to run compute-intensive, GUI-driven applications on these machines. Such usage is not only contrary to CAC policy, it is disrespectful toward other users, because the login node may become unresponsive through your actions.<br />
<br />
'''''Linux users:'''''<br />
<br />
The standard way to use X-Windows is to tunnel the X-Windows protocol through an ssh connection. If you open your ssh session with the '''-X''' option, it will automatically set up the necessary tunnel and environment variables.<br />
localhost$ ssh -X username@linuxlogin.cac.cornell.edu<br />
linuxlogin$ echo $DISPLAY<br />
localhost:11.0<br />
linuxlogin$ xclock&<br />
You can see that your DISPLAY environment variable is set and test it with xclock. There is another option to use a trusted version of X-windows forwarding<br />
linuxlogin$ ssh -Y compute-3-48.v4linux<br />
Th trusted version is necessary for forwarding X11 connections from a compute node to the login node, then back to your client machine.<br />
<br />
'''''Mac OS X users:'''''<br />
<br />
If you start ssh with the '''-X''' or '''-Y''' option, X-Windows should start up automatically. You can then try the "xclock" test, as described above for Linux. <br />
<br />
X11 is preinstalled on Macs starting with OS X 10.6 (Snow Leopard). For Mac OS X 10.5 (Leopard), you may need to install X11 in order for X-Windows applications to launch. If there is no X11 application in the Applications->Utilities folder, you'll have to find your OS X install disk. From the ''Mac OS X Server Introduction to Command-Line Administration,'' "The X11 server and an application to access X windows from the Finder are available as an optional installation in the Optional Installs folder of your installation disc (X11 is in the Applications package)."<br />
<br />
'''''Windows users:'''''<br />
<br />
Along with your ssh client (e.g., PuTTY), you will need to install an X-Windows server on your Windows machine.<br />
:* [//straightrunning.com/XmingNotes/ Xming] - Open Source. A shareware contribution will get you a version with improved performance for graphics (GLX). There are two pieces to download<br />
:*:[[Image:Xming-download.jpg]]<br />
:** Xming-mesa (public domain release). There are two links together, one for Xming, one for Xming-mesa. Either will work, but Xming-mesa has some newer features that might come in handy some time.<br />
:** Xming-fonts (public domain release)<br />
If you purchase the website release of Xming, remember to install the Xming-fonts, as well.<br />
:* OpenText's [//cit.cornell.edu/services/software_licensing/available/exceed.cfm Exceed and Exceed 3D] - Cornell no longer has a site license. Installing Exceed 3D will improve performance of graphics applications. Exceed installs several icons under the Start menu. Choose the one that just says "Exceed" because it starts the program in multi-window mode, which is what we want.<br />
<br />
Here is how to start a session using PuTTY and Xming.<br />
<br />
# Start Xming from the Start menu. It will appear briefly and disappear except for an X in the application tray.<br />
# Start PuTTY.<br />
# In the window that appears, type a host name, <tt>linuxlogin.cac.cornell.edu</tt>.<br />
# Use the tree menu on the left to set X11 forwarding. It's in the <tt>Connection > SSH branch</tt>.<br />
#:[[Image:Putty_x11forwarding.jpg|Setting X11 forwarding in PuTTY]]<br />
# For PuTTY 0.61 only - In the "Auth" section of the SSH branch, go to GSSAPI and uncheck <tt>"Attempt GSSAPI authentication"</tt>. This will prevent an annoying <tt>"Access denied"</tt> message from appearing in your terminal window.<br />
# You can return to the Session category and Save this session's configuration for future use. Give it a logical name like linuxlogin.<br />
# Click Open, and it will connect to a login node.<br />
# Test your X-Windows setup by typing<br />
xclock<br />
You should see a clock appear in the corner of your screen. You can stop it by typing <tt>Ctrl-c</tt> in the terminal window.<br />
<br />
=====Using VNC=====<br />
<br />
[http://en.wikipedia.org/wiki/Vnc VNC] lets you see a whole Linux desktop from the login node on your computer.<br />
Using SSH and X-Windows is generally faster, and uses a lot less of the login node's resources,<br />
but VNC can be much faster if you are doing visualization on the login node from off campus.<br />
<br />
For security reasons, we are requiring all VNC connections to be tunneled inside ssh. You will therefore need to be able to connect to the login nodes [[Getting_Started#Using_Secure_Shell | using SSH]]. Because the firewall running on linuxlogin blocks all incoming ports except for ssh, VNC connections must be made over a ssh tunnel as described below.<br />
<br />
'''''Appropriate use'''''<br />
<br />
VNC gives you the ability to establish a remote desktop on the login nodes, but this ability does '''''NOT''''' imply that you are permitted to run compute-intensive, GUI-driven applications on these machines. Such usage is not only contrary to CAC policy, it is disrespectful toward other users, because the login node may become unresponsive through your actions.<br />
<br />
Here is a good example of how to use VNC appropriately. By following these steps you can run (say) Abaqus in GUI-driven mode on a compute node that has been allocated to you through an interactive batch job.<br />
# Open a VNC connection to linuxlogin through an ssh tunnel using the instructions below, in order to gain access to a Linux desktop. Make sure two terminal windows are available on this desktop.<br />
# In one of the terminal windows, submit an interactive job to the queue of your choice (add the #PBS -I directive to your job submission script).<br />
# Once the job starts, you will be given a command prompt on your assigned machine. Note the result of "hostname". There is no need to enter further commands at this prompt (except to exit the job).<br />
# Go to the other terminal window and open a second ssh connection to the compute node using "ssh -Y <userid>@<hostname>"<br />
# This new ssh session will tunnel X-Windows from the compute node back to the VNC desktop. Therefore (if Abaqus is on your path), you can now open the Abaqus GUI using "abaqus cae -mesa".<br />
<br />
'''''Initial setup'''''<br />
''(You only need to do this once)''<br />
<br />
:* Install a VNC client if one isn't installed. [http://www.tightvnc.com/ TightVNC] works well, but so do others.<br />
:* Login to linuxlogin, and set the password for your VNC server using the "vncpasswd" command.<br />
<br />
'''''Start your VNC server'''''<br />
<br />
:* '''On linuxlogin''', start the VNC server using the "vncserver" command like this:<br />
vncserver -geometry 1024x768 -localhost<br />
The geometry numbers, 1024x768, specify the size, in pixels, of the desktop.<br />
:* You will need to get the display number from the output of the vncserver command:<br />
<br />
New 'linuxlogin.cac.cornell.edu:1 (shl1)' desktop is linuxlogin.cac.cornell.edu:1<br />
Starting applications specified in /home/gfs01/shl1/.vnc/xstartup<br />
Log file is /home/gfs01/shl1/.vnc/linuxlogin.cac.cornell.edu:1.log<br />
<br />
:* vncserver is running on port 5900 + display number. In the above example, the display number is :1, therefore vncserver is running on port 5901.<br />
<br />
'''''Connect your VNC client'''''<br />
<br />
:* Set up ssh forwarding on your client computer. Let's say the port number on linuxlogin is 5901 (as above), and your CAC userid is uid12. From Linux, type into a terminal:<br />
<br />
ssh -L 10000:localhost:5901 uid12@linuxlogin.cac.cornell.edu<br />
'''From Windows''', ssh clients such as PuTTY can do X11 port forwarding. See [[VNC Tunnel Windows]]. <br />
<br />
'''For MacOS X users''', see [[CAC VPN Server for MacOS Users |here]]<br />
:* Leave this ssh session running on your local client computer. (It can run in the background.)<br />
:* Launch your VNC client program. Connect to localhost:10000. When prompted, type in your VNC server password.<br />
<br />
'''''To disconnect your client'''''<br />
<br />
:* Close the vnc client program.<br />
:* Disconnect the ssh forwarding session (i.e., kill it).<br />
<br />
'''''To reconnect your client'''''<br />
<br />
:* Restart port forwarding with ssh, using the same remote port number as before.<br />
:* Again connect the VNC client to localhost:10000.<br />
<br />
'''''When you are all done'''''<br />
<br />
:* On linuxlogin, type this command to shut down the VNC server<br />
vncserver -kill :<display number><br />
:* If you merely log out from linuxlogin, it will leave the VNC server running. You must shut down the VNC server explicitly when you are finished with it. (Actually this can be a nice feature.)<br />
<br />
=====Passwordless SSH=====<br />
<br />
''''' Create ssh key pair '''''<br />
<br />
Your ssh key pair will only need to be created once. You will not need to repeat this step. You can complete this step from either a Linux or Windows login node. If this is your first login to a CAC login node, it will ask you to [[Getting_Started#Change_a_password_at_first_login|change your password]]. This will become your password for connecting to the nodes. <br />
<br />
Create your ssh key pair by logging into the linux login node (linuxlogin.cac.cornell.edu), which will begin the process of creating the keys; you can use the defaults or empty responses for all prompts.<br />
<br />
Alternatively, you can create your ssh key pair on the linux login node by logging into the Windows login node (winx64login.cac.cornell.edu), opening a Command Prompt window, and running <tt>plink.exe</tt> to connect to the linux login node, as shown in this example:<br />
<br />
<pre>>"C:\Programs Files (x86)\Putty\plink.exe" %USERNAME%@linuxlogin.cac.cornell.edu<br />
Password: Enter Your Password<br />
Rocks 5.0 (V)<br />
Profile built 12:54 06-May-2008<br />
<br />
Kickstarted 09:22 06-May-2008<br />
-----------------------------------------------------------<br />
Welcome to the Center for Advanced Computing Cluster!<br />
-----------------------------------------------------------<br />
Please send your questions to help@cac.cornell.edu<br />
-----------------------------------------------------------<br />
<br />
<br />
It doesn't appear that you have set up your ssh key.<br />
This process will make the files:<br />
/home/gfs01/cacshl1/.ssh/id_rsa.pub<br />
/home/gfs01/cacshl1/.ssh/id_rsa<br />
/home/gfs01/cacshl1/.ssh/authorized_keys<br />
<br />
Generating public/private rsa key pair.<br />
Enter file in which to save the key (/home/gfs01/cacshl1/.ssh/id_rsa): Press Enter to accept default<br />
Created directory '/home/gfs01/cacshl1/.ssh'.<br />
Enter passphrase (empty for no passphrase): Press Enter to accept default<br />
Enter same passphrase again: Press Enter to accept default<br />
Your identification has been saved in /home/gfs01/cacshl1/.ssh/id_rsa.<br />
Your public key has been saved in /home/gfs01/cacshl1/.ssh/id_rsa.pub.<br />
</pre><br />
<br />
After this is done, type '''"exit"''' to log out of the linux login node.<br />
<br />
''''' Convert ssh Private Key for Putty / Plink '''''<br />
<br />
Next run PuTTYgen to generate public and private keys to be used with PuTTY and Plink:<br />
<br />
:* Log in to <tt>winx64login.tc.cornell.edu </tt>(if you are not already)<br />
:* Run <tt>C:\Program Files (x86)\Putty\puttygen.exe</tt>.<br />
:* Select <tt>Import Key</tt> from the <tt>Conversions</tt> menu and select <tt>H:\.ssh\id_rsa</tt> in your home directory. And click on the <tt>Open</tt> button.<br />
<center>[[image:LoadPrivateKey.jpg]]</center><br />
:* Click on the <tt>"Save Private Key"</tt> button. <br />
<center>[[image:SavePrivateKey.jpg]]</center><br />
:* Click on "Yes" when asked to save the private key without a passphrase.<br />
:* Save the private key as private.ppk in the .ssh directory inside your home directory.<br />
<center>[[image:SpecifyPrivateKey.jpg]]</center><br />
:* Close (choose File, then Exit)<br />
:* To confirm you have converted the ssh private key successfully, do:<br />
<pre>"C:\Program Files (x86)\Putty\plink.exe" -i %HOMEDRIVE%\.ssh\private.ppk %USERNAME%@linuxlogin.cac.cornell.edu</pre><br />
It may notify you that "The server's host key is not cached in the registry." Type "y" to "store the key in cache."<br />
:* You should now be logged into linuxlogin without being prompted for a password. Stay logged in for the next step.</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Linux_Usage_Tips&diff=907Linux Usage Tips2015-09-30T18:14:13Z<p>Ad876: </p>
<hr />
<div>==== Linux shells====<br />
:* <tt>/bin/sh</tt> is the default login shell.<br />
:** Edit <tt>$HOME/.profile</tt> to change interactive variables.<br />
:** The <tt>$HOME/.bashrc</tt> file ''will not'' be run for non-interactive shells.<br />
:* <tt>/bin/bash</tt><br />
:** Edit <tt>$HOME/.profile</tt> to change interactive variables.<br />
:** The <tt>$HOME/.bashrc</tt> file ''will'' be run for non-interactive shells.<br />
:* <tt>/bin/csh</tt> and <tt>/bin/tcsh</tt><br />
:** Edit <tt>$HOME/.login</tt> to change interactive variables.<br />
:** The <tt>$HOME/.cshrc</tt> file ''will'' be run for non-interactive shells.<br />
<br />
The change shell command, <tt>chsh</tt>, will not permanently change your shell. You must send a request instead. {{ContactCAC}}<br />
<br />
The default login shell on v4 interactive and batch nodes is ''sh''. Be aware that in Red Hat Enterprise Linux, /bin/sh is a soft-link to /bin/bash, so you are really using a variant of ''bash''. Accordingly, you will find that "man sh" brings up the man page (the help document) for ''bash''. In a way, then, you can think of your login shell as being ''bash'', too.<br />
<br />
There are slight differences between ''sh'' and ''bash'', however. The "Invocation" section of the man page states: "If bash is invoked with the name sh, it tries to mimic the startup behavior of historical versions of ''sh'' as closely as possible." Therefore, you will find that ~/.profile is run at login, because this behavior is common to both ''sh'' and ''bash''; but any interactive ''sh'' shells you start thereafter will not run ~/.bashrc as you might expect from ''bash''. The way to get ''sh'' to do this is to "export ENV=~/.bashrc" beforehand (perhaps as part of your .profile).<br />
<br />
Let's say you simply prefer to have ''bash'' as your default shell and be done with it. There are two ways to accomplish this. First, you can "export SHELL=/bin/bash" in your .profile; then all subsequent interactive shells will truly be ''bash''. Second, you can enter "chsh -s /bin/bash", which forces all login and interactive shells to be ''bash'' (because you have changed your default shell). The problem with the second method is it may well wreck your batch environment, too, because the scheduler sets it up under the assumption that the login shell is ''sh''.<br />
<br />
The relationship between the ''csh'' and ''tcsh'' shells is similar to the one between ''sh'' and ''bash''. For instance, your ''csh'' shells are automatically endowed with the ''tcsh''-style ability to retrieve history through the up- and down-arrow keys. The best way to make ''tcsh'' into your everyday working shell is to run it on top of ''sh'' after you log in (again, you can do this as part of your .profile).<br />
<br />
References<br />
:* "man bash" from the command line.<br />
:* [//tldp.org/LDP/abs/html/ Advanced Bash Scripting Guide], one of the Linux Documentation Project [//tldp.org/guides.html guides]<br />
:* [//mywiki.wooledge.org/BashFAQ Bash FAQ]<br />
:*[//mywiki.wooledge.org/BashPitfalls Bash Pitfalls]<br />
<br />
====Compiling and linking code on Linux====<br />
{{:Compiling Code Linux}}<br />
<br />
====FAQ====<br />
=====How do I determine my program's dependencies on shared library (.so) files?=====<br />
:*ldd - see the man page.<br />
If your program cannot find all the .so files it needs, you may need to add paths to the LD_LIBRARY_PATH shell variable.<br />
<br />
=====How do I display an image file (such as jpeg or gif)?=====<br />
:*display mypic.jpg - uses one of the many ImageMagick tools - see "man ImageMagick" for help on this and various file format converters.<br />
:*firefox mypic.jpg - any decent Web browser can handle it.<br />
Note, the image will show up only if you have [[Getting_Started#Connect_to_Linux | X11 forwarding]] enabled.</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Connect_to_Linux&diff=905Connect to Linux2015-09-30T18:12:32Z<p>Ad876: </p>
<hr />
<div>__TOC__<br />
There are three distinct ways to connect to a login node:<br />
# [[Getting_Started#Using_Secure_Shell | Use SSH]] to open a Linux shell on a login node, which provides a text-only interface.<br />
# [[Getting_Started#Using_Secure_Shell | Use SSH]] together with [[Getting_started#Using_X-Windows | X-Windows]], which sends any interactive graphics back to your machine window-by-window through an SSH tunnel.<br />
# [[Getting_Started#Using_VNC | Use VNC]] to get a remote desktop with multiple text and graphics windows. This is not as straightforward as it sounds, due to the need to set up a secure tunnel for the remote desktop first.<br />
<br />
These instructions are intended mainly for users of personal computers and workstations. However, much of the material carries over to mobile computing platforms such as tablets and smartphones. You will have to locate and download an app to enable SSH or VNC connectivity; even a browser plug-in may suffice.<br />
<br />
Whichever method you choose, at your first login, you will be challenged for a new password. Find help at [[Getting_Started#Change_a_password_at_first_login | Changing a Password at First Login]]. You will also be asked for an ssh passphrase. You can just leave this blank; hit the <tt>Enter</tt> key in response.<br />
<br />
=====Using Secure Shell=====<br />
For basic command-line access, a Secure Shell (SSH) client will give you a remote command shell on one of the login nodes.<br />
:* Nearly all Unix/Linux varieties (including Mac) already have a built-in SSH2 implementation, required by our clusters.<br />
:* If you are coming from a Microsoft Windows machine, an SSH2 client must first be installed, as described below.<br />
:* The non-secure predecessor of SSH, telnet, is disabled for security reasons.<br />
<br />
'''''Linux users:'''''<br />
<br />
To connect to the second login node with ssh, you simply open a terminal window and type<br />
localhost$ ssh username@linuxlogin.cac.cornell.edu<br />
<br />
'''''Mac OS X users:'''''<br />
<br />
OS X on the Mac is built on a version of Unix, so ssh is available directly from the Terminal application.<br />
:*One option is to use the shortcut <tt>'''cmd-space'''</tt> to open Spotlight and then type <tt>"Terminal"</tt> to open a Terminal window.<br />
Otherwise:<br />
:*Navigate in the Finder to the Applications folder and Utilities sub-folder.<br />
:*:[[Image:MacApplicationsFolder.png|500px]]<br />
:*Then double-click on the Terminal application to see a Bash command-line.<br />
:*:[[Image:MacTerminalWindow.png]]<br />
:*As in Linux, simply type "ssh username@linuxlogin.cac.cornell.edu" into this window.<br />
<br />
'''''Windows users:'''''<br />
<br />
Secure Shell (ssh) clients work nicely as long as they support the SSH2 protocol. As mentioned, telnet is disabled for security reasons. A popular client for Windows is the free [//www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY client].<br />
:* The simplest installation is to download the [//chiark.greenend.org.uk/~sgtatham/putty/download.html Windows installer], called '''putty-0.65-installer.exe''', and run it. This installs PuTTY into your Start menu.<br />
:* To connect, start PuTTY, then type in a host name such as linuxlogin.cac.cornell.edu, and click "Open".<br />
:*:[[Image:Putty_address.jpg|Setting the host name in PuTTY]]<br />
<br />
=====Using X-Windows=====<br />
<br />
X-Windows or X11 is the longstanding Unix mechanism for displaying interactive graphics in a window. Your "X server" software runs locally, but it is capable of displaying windows that have been generated either locally or remotely. An "X client" on a remote machine can create X-Windows for local display, but it is necessary first to establish a shell on that machine using SSH.<br />
<br />
'''''Appropriate use'''''<br />
<br />
Among other things, X-Windows gives you the ability to display a GUI that originates on a login node. However, this ability does '''''NOT''''' imply that you are permitted to run compute-intensive, GUI-driven applications on these machines. Such usage is not only contrary to CAC policy, it is disrespectful toward other users, because the login node may become unresponsive through your actions.<br />
<br />
'''''Linux users:'''''<br />
<br />
The standard way to use X-Windows is to tunnel the X-Windows protocol through an ssh connection. If you open your ssh session with the '''-X''' option, it will automatically set up the necessary tunnel and environment variables.<br />
localhost$ ssh -X username@linuxlogin.cac.cornell.edu<br />
linuxlogin$ echo $DISPLAY<br />
localhost:11.0<br />
linuxlogin$ xclock&<br />
You can see that your DISPLAY environment variable is set and test it with xclock. There is another option to use a trusted version of X-windows forwarding<br />
linuxlogin$ ssh -Y compute-3-48.v4linux<br />
Th trusted version is necessary for forwarding X11 connections from a compute node to the login node, then back to your client machine.<br />
<br />
'''''Mac OS X users:'''''<br />
<br />
If you start ssh with the '''-X''' or '''-Y''' option, X-Windows should start up automatically. You can then try the "xclock" test, as described above for Linux. <br />
<br />
X11 is preinstalled on Macs starting with OS X 10.6 (Snow Leopard). For Mac OS X 10.5 (Leopard), you may need to install X11 in order for X-Windows applications to launch. If there is no X11 application in the Applications->Utilities folder, you'll have to find your OS X install disk. From the ''Mac OS X Server Introduction to Command-Line Administration,'' "The X11 server and an application to access X windows from the Finder are available as an optional installation in the Optional Installs folder of your installation disc (X11 is in the Applications package)."<br />
<br />
'''''Windows users:'''''<br />
<br />
Along with your ssh client (e.g., PuTTY), you will need to install an X-Windows server on your Windows machine.<br />
:* [//straightrunning.com/XmingNotes/ Xming] - Open Source. A shareware contribution will get you a version with improved performance for graphics (GLX). There are two pieces to download<br />
:*:[[Image:Xming-download.jpg]]<br />
:** Xming-mesa (public domain release). There are two links together, one for Xming, one for Xming-mesa. Either will work, but Xming-mesa has some newer features that might come in handy some time.<br />
:** Xming-fonts (public domain release)<br />
If you purchase the website release of Xming, remember to install the Xming-fonts, as well.<br />
:* OpenText's [//cit.cornell.edu/services/software_licensing/available/exceed.cfm Exceed and Exceed 3D] - Cornell no longer has a site license. Installing Exceed 3D will improve performance of graphics applications. Exceed installs several icons under the Start menu. Choose the one that just says "Exceed" because it starts the program in multi-window mode, which is what we want.<br />
<br />
Here is how to start a session using PuTTY and Xming.<br />
<br />
# Start Xming from the Start menu. It will appear briefly and disappear except for an X in the application tray.<br />
# Start PuTTY.<br />
# In the window that appears, type a host name, <tt>linuxlogin.cac.cornell.edu</tt>.<br />
# Use the tree menu on the left to set X11 forwarding. It's in the <tt>Connection > SSH branch</tt>.<br />
#:[[Image:Putty_x11forwarding.jpg|Setting X11 forwarding in PuTTY]]<br />
# For PuTTY 0.61 only - In the "Auth" section of the SSH branch, go to GSSAPI and uncheck <tt>"Attempt GSSAPI authentication"</tt>. This will prevent an annoying <tt>"Access denied"</tt> message from appearing in your terminal window.<br />
# You can return to the Session category and Save this session's configuration for future use. Give it a logical name like linuxlogin.<br />
# Click Open, and it will connect to a login node.<br />
# Test your X-Windows setup by typing<br />
xclock<br />
You should see a clock appear in the corner of your screen. You can stop it by typing <tt>Ctrl-c</tt> in the terminal window.<br />
<br />
=====Using VNC=====<br />
<br />
[http://en.wikipedia.org/wiki/Vnc VNC] lets you see a whole Linux desktop from the login node on your computer.<br />
Using SSH and X-Windows is generally faster, and uses a lot less of the login node's resources,<br />
but VNC can be much faster if you are doing visualization on the login node from off campus.<br />
<br />
For security reasons, we are requiring all VNC connections to be tunneled inside ssh. You will therefore need to be able to connect to the login nodes [[Getting_Started#Using_Secure_Shell | using SSH]]. Because the firewall running on linuxlogin blocks all incoming ports except for ssh, VNC connections must be made over a ssh tunnel as described below.<br />
<br />
'''''Appropriate use'''''<br />
<br />
VNC gives you the ability to establish a remote desktop on the login nodes, but this ability does '''''NOT''''' imply that you are permitted to run compute-intensive, GUI-driven applications on these machines. Such usage is not only contrary to CAC policy, it is disrespectful toward other users, because the login node may become unresponsive through your actions.<br />
<br />
Here is a good example of how to use VNC appropriately. By following these steps you can run (say) Abaqus in GUI-driven mode on a compute node that has been allocated to you through an interactive batch job.<br />
# Open a VNC connection to linuxlogin through an ssh tunnel using the instructions below, in order to gain access to a Linux desktop. Make sure two terminal windows are available on this desktop.<br />
# In one of the terminal windows, submit an interactive job to the queue of your choice (add the #PBS -I directive to your job submission script).<br />
# Once the job starts, you will be given a command prompt on your assigned machine. Note the result of "hostname". There is no need to enter further commands at this prompt (except to exit the job).<br />
# Go to the other terminal window and open a second ssh connection to the compute node using "ssh -Y <userid>@<hostname>"<br />
# This new ssh session will tunnel X-Windows from the compute node back to the VNC desktop. Therefore (if Abaqus is on your path), you can now open the Abaqus GUI using "abaqus cae -mesa".<br />
<br />
'''''Initial setup'''''<br />
''(You only need to do this once)''<br />
<br />
:* Install a VNC client if one isn't installed. [http://www.tightvnc.com/ TightVNC] works well, but so do others.<br />
:* Login to linuxlogin, and set the password for your VNC server using the "vncpasswd" command.<br />
<br />
'''''Start your VNC server'''''<br />
<br />
:* '''On linuxlogin''', start the VNC server using the "vncserver" command like this:<br />
vncserver -geometry 1024x768 -localhost<br />
The geometry numbers, 1024x768, specify the size, in pixels, of the desktop.<br />
:* You will need to get the display number from the output of the vncserver command:<br />
<br />
New 'linuxlogin.cac.cornell.edu:1 (shl1)' desktop is linuxlogin.cac.cornell.edu:1<br />
Starting applications specified in /home/gfs01/shl1/.vnc/xstartup<br />
Log file is /home/gfs01/shl1/.vnc/linuxlogin.cac.cornell.edu:1.log<br />
<br />
:* vncserver is running on port 5900 + display number. In the above example, the display number is :1, therefore vncserver is running on port 5901.<br />
<br />
'''''Connect your VNC client'''''<br />
<br />
:* Set up ssh forwarding on your client computer. Let's say the port number on linuxlogin is 5901 (as above), and your CAC userid is uid12. From Linux, type into a terminal:<br />
<br />
ssh -L 10000:localhost:5901 uid12@linuxlogin.cac.cornell.edu<br />
'''From Windows''', ssh clients such as PuTTY can do X11 port forwarding. See [[VNC Tunnel Windows]]. <br />
<br />
'''For MacOS X users''', see [[CAC VPN Server for MacOS Users |here]]<br />
:* Leave this ssh session running on your local client computer. (It can run in the background.)<br />
:* Launch your VNC client program. Connect to localhost:10000. When prompted, type in your VNC server password.<br />
<br />
'''''To disconnect your client'''''<br />
<br />
:* Close the vnc client program.<br />
:* Disconnect the ssh forwarding session (i.e., kill it).<br />
<br />
'''''To reconnect your client'''''<br />
<br />
:* Restart port forwarding with ssh, using the same remote port number as before.<br />
:* Again connect the VNC client to localhost:10000.<br />
<br />
'''''When you are all done'''''<br />
<br />
:* On linuxlogin, type this command to shut down the VNC server<br />
vncserver -kill :<display number><br />
:* If you merely log out from linuxlogin, it will leave the VNC server running. You must shut down the VNC server explicitly when you are finished with it. (Actually this can be a nice feature.)<br />
<br />
=====Passwordless SSH=====<br />
<br />
''''' Create ssh key pair '''''<br />
<br />
Your ssh key pair will only need to be created once. You will not need to repeat this step. You can complete this step from either a Linux or Windows login node. If this is your first login to a CAC login node, it will ask you to [[Getting_Started#Change_a_password_at_first_login|change your password]]. This will become your password for connecting to the nodes. <br />
<br />
Create your ssh key pair by logging into the linux login node (linuxlogin.cac.cornell.edu), which will begin the process of creating the keys; you can use the defaults or empty responses for all prompts.<br />
<br />
Alternatively, you can create your ssh key pair on the linux login node by logging into the Windows login node (winx64login.cac.cornell.edu), opening a Command Prompt window, and running <tt>plink.exe</tt> to connect to the linux login node, as shown in this example:<br />
<br />
<pre>>"C:\Programs Files (x86)\Putty\plink.exe" %USERNAME%@linuxlogin.cac.cornell.edu<br />
Password: Enter Your Password<br />
Rocks 5.0 (V)<br />
Profile built 12:54 06-May-2008<br />
<br />
Kickstarted 09:22 06-May-2008<br />
-----------------------------------------------------------<br />
Welcome to the Center for Advanced Computing Cluster!<br />
-----------------------------------------------------------<br />
Please send your questions to help@cac.cornell.edu<br />
-----------------------------------------------------------<br />
<br />
<br />
It doesn't appear that you have set up your ssh key.<br />
This process will make the files:<br />
/home/gfs01/cacshl1/.ssh/id_rsa.pub<br />
/home/gfs01/cacshl1/.ssh/id_rsa<br />
/home/gfs01/cacshl1/.ssh/authorized_keys<br />
<br />
Generating public/private rsa key pair.<br />
Enter file in which to save the key (/home/gfs01/cacshl1/.ssh/id_rsa): Press Enter to accept default<br />
Created directory '/home/gfs01/cacshl1/.ssh'.<br />
Enter passphrase (empty for no passphrase): Press Enter to accept default<br />
Enter same passphrase again: Press Enter to accept default<br />
Your identification has been saved in /home/gfs01/cacshl1/.ssh/id_rsa.<br />
Your public key has been saved in /home/gfs01/cacshl1/.ssh/id_rsa.pub.<br />
</pre><br />
<br />
After this is done, type '''"exit"''' to log out of the linux login node.<br />
<br />
''''' Convert ssh Private Key for Putty / Plink '''''<br />
<br />
Next run PuTTYgen to generate public and private keys to be used with PuTTY and Plink:<br />
<br />
:* Log in to <tt>winx64login.tc.cornell.edu </tt>(if you are not already)<br />
:* Run <tt>C:\Program Files (x86)\Putty\puttygen.exe</tt>.<br />
:* Select <tt>Import Key</tt> from the <tt>Conversions</tt> menu and select <tt>H:\.ssh\id_rsa</tt> in your home directory. And click on the <tt>Open</tt> button.<br />
<center>[[image:LoadPrivateKey.jpg]]</center><br />
:* Click on the <tt>"Save Private Key"</tt> button. <br />
<center>[[image:SavePrivateKey.jpg]]</center><br />
:* Click on "Yes" when asked to save the private key without a passphrase.<br />
:* Save the private key as private.ppk in the .ssh directory inside your home directory.<br />
<center>[[image:SpecifyPrivateKey.jpg]]</center><br />
:* Close (choose File, then Exit)<br />
:* To confirm you have converted the ssh private key successfully, do:<br />
<pre>"C:\Program Files (x86)\Putty\plink.exe" -i %HOMEDRIVE%\.ssh\private.ppk %USERNAME%@linuxlogin.cac.cornell.edu</pre><br />
It may notify you that "The server's host key is not cached in the registry." Type "y" to "store the key in cache."<br />
:* You should now be logged into linuxlogin without being prompted for a password. Stay logged in for the next step.</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Connect_to_Linux&diff=904Connect to Linux2015-09-30T17:23:02Z<p>Ad876: </p>
<hr />
<div>__TOC__<br />
There are three distinct ways to connect to a login node:<br />
# [[Getting_Started#Using_Secure_Shell | Use SSH]] to open a Linux shell on a login node, which provides a text-only interface.<br />
# [[Getting_Started#Using_Secure_Shell | Use SSH]] together with [[Getting_started#Using_X-Windows | X-Windows]], which sends any interactive graphics back to your machine window-by-window through an SSH tunnel.<br />
# [[Getting_Started#Using_VNC | Use VNC]] to get a remote desktop with multiple text and graphics windows. This is not as straightforward as it sounds, due to the need to set up a secure tunnel for the remote desktop first.<br />
<br />
These instructions are intended mainly for users of personal computers and workstations. However, much of the material carries over to mobile computing platforms such as tablets and smartphones. You will have to locate and download an app to enable SSH or VNC connectivity; even a browser plug-in may suffice.<br />
<br />
Whichever method you choose, at your first login, you will be challenged for a new password. Find help at [[Getting_Started#Change_a_password_at_first_login | Changing a Password at First Login]]. You will also be asked for an ssh passphrase. You can just leave this blank; hit the <tt>Enter</tt> key in response.<br />
<br />
=====Using Secure Shell=====<br />
For basic command-line access, a Secure Shell (SSH) client will give you a remote command shell on one of the login nodes.<br />
:* Nearly all Unix/Linux varieties (including Mac) already have a built-in SSH2 implementation, required by our clusters.<br />
:* If you are coming from a Microsoft Windows machine, an SSH2 client must first be installed, as described below.<br />
:* The non-secure predecessor of SSH, telnet, is disabled for security reasons.<br />
<br />
'''''Linux users:'''''<br />
<br />
To connect to the second login node with ssh, you simply open a terminal window and type<br />
localhost$ ssh username@linuxlogin.cac.cornell.edu<br />
<br />
'''''Mac OS X users:'''''<br />
<br />
OS X on the Mac is built on a version of Unix, so ssh is available directly from the Terminal application.<br />
:*One option is to use the shortcut <tt>'''cmd-space'''</tt> to open Spotlight and then type <tt>"Terminal"</tt> to open a Terminal window.<br />
Otherwise:<br />
:*Navigate in the Finder to the Applications folder and Utilities sub-folder.<br />
:*:[[Image:MacApplicationsFolder.png|500px]]<br />
:*Then double-click on the Terminal application to see a Bash command-line.<br />
:*:[[Image:MacTerminalWindow.png]]<br />
:*As in Linux, simply type "ssh username@linuxlogin.cac.cornell.edu" into this window.<br />
<br />
'''''Windows users:'''''<br />
<br />
Secure Shell (ssh) clients work nicely as long as they support the SSH2 protocol. As mentioned, telnet is disabled for security reasons. A popular client for Windows is the free [//www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY client].<br />
:* The simplest installation is to download the [//chiark.greenend.org.uk/~sgtatham/putty/download.html Windows installer], called '''putty-0.65-installer.exe''', and run it. This installs PuTTY into your Start menu.<br />
:* To connect, start PuTTY, then type in a host name such as linuxlogin.cac.cornell.edu, and click "Open".<br />
:*:[[Image:Putty_address.jpg|Setting the host name in PuTTY]]<br />
<br />
=====Using X-Windows=====<br />
<br />
X-Windows or X11 is the longstanding Unix mechanism for displaying interactive graphics in a window. Your "X server" software runs locally, but it is capable of displaying windows that have been generated either locally or remotely. An "X client" on a remote machine can create X-Windows for local display, but it is necessary first to establish a shell on that machine using SSH.<br />
<br />
'''''Appropriate use'''''<br />
<br />
Among other things, X-Windows gives you the ability to display a GUI that originates on a login node. However, this ability does '''''NOT''''' imply that you are permitted to run compute-intensive, GUI-driven applications on these machines. Such usage is not only contrary to CAC policy, it is disrespectful toward other users, because the login node may become unresponsive through your actions.<br />
<br />
'''''Linux users:'''''<br />
<br />
The standard way to use X-Windows is to tunnel the X-Windows protocol through an ssh connection. If you open your ssh session with the '''-X''' option, it will automatically set up the necessary tunnel and environment variables.<br />
localhost$ ssh -X username@linuxlogin.cac.cornell.edu<br />
linuxlogin$ echo $DISPLAY<br />
localhost:11.0<br />
linuxlogin$ xclock&<br />
You can see that your DISPLAY environment variable is set and test it with xclock. There is another option to use a trusted version of X-windows forwarding<br />
linuxlogin$ ssh -Y compute-3-48.v4linux<br />
Th trusted version is necessary for forwarding X11 connections from a compute node to the login node, then back to your client machine.<br />
<br />
'''''Mac OS X users:'''''<br />
<br />
If you start ssh with the '''-X''' or '''-Y''' option, X-Windows should start up automatically. You can then try the "xclock" test, as described above for Linux. <br />
<br />
X11 is preinstalled on Macs starting with OS X 10.6 (Snow Leopard). For Mac OS X 10.5 (Leopard), you may need to install X11 in order for X-Windows applications to launch. If there is no X11 application in the Applications->Utilities folder, you'll have to find your OS X install disk. From the ''Mac OS X Server Introduction to Command-Line Administration,'' "The X11 server and an application to access X windows from the Finder are available as an optional installation in the Optional Installs folder of your installation disc (X11 is in the Applications package)."<br />
<br />
'''''Windows users:'''''<br />
<br />
Along with your ssh client (e.g., PuTTY), you will need to install an X-Windows server on your Windows machine.<br />
:* [//straightrunning.com/XmingNotes/ Xming] - Open Source. A shareware contribution will get you a version with improved performance for graphics (GLX). There are two pieces to download<br />
:*:[[Image:Xming-download.jpg]]<br />
:** Xming-mesa (public domain release). There are two links together, one for Xming, one for Xming-mesa. Either will work, but Xming-mesa has some newer features that might come in handy some time.<br />
:** Xming-fonts (public domain release)<br />
If you purchase the website release of Xming, remember to install the Xming-fonts, as well.<br />
:* OpenText's [//cit.cornell.edu/services/software_licensing/available/exceed.cfm Exceed and Exceed 3D] - Cornell no longer has a site license. Installing Exceed 3D will improve performance of graphics applications. Exceed installs several icons under the Start menu. Choose the one that just says "Exceed" because it starts the program in multi-window mode, which is what we want.<br />
<br />
Here is how to start a session using PuTTY and Xming.<br />
<br />
# Start Xming from the Start menu. It will appear briefly and disappear except for an X in the application tray.<br />
# Start PuTTY.<br />
# In the window that appears, type a host name, <tt>linuxlogin.cac.cornell.edu</tt>.<br />
# Use the tree menu on the left to set X11 forwarding. It's in the <tt>Connection > SSH branch</tt>.<br />
#:[[Image:Putty_x11forwarding.jpg|Setting X11 forwarding in PuTTY]]<br />
# For PuTTY 0.61 only - In the "Auth" section of the SSH branch, go to GSSAPI and uncheck <tt>"Attempt GSSAPI authentication"</tt>. This will prevent an annoying <tt>"Access denied"</tt> message from appearing in your terminal window.<br />
# You can return to the Session category and Save this session's configuration for future use. Give it a logical name like linuxlogin.<br />
# Click Open, and it will connect to a login node.<br />
# Test your X-Windows setup by typing<br />
xclock<br />
You should see a clock appear in the corner of your screen. You can stop it by typing <tt>Ctrl-c</tt> in the terminal window.<br />
<br />
=====Using VNC=====<br />
<br />
[http://en.wikipedia.org/wiki/Vnc VNC] lets you see a whole Linux desktop from the login node on your computer.<br />
Using SSH and X-Windows is generally faster, and uses a lot less of the login node's resources,<br />
but VNC can be much faster if you are doing visualization on the login node from off campus.<br />
<br />
For security reasons, we are requiring all VNC connections to be tunneled inside ssh. You will therefore need to be able to connect to the login nodes [[Getting_started#Using_Secure_Shell | using SSH]]. Because the firewall running on linuxlogin blocks all incoming ports except for ssh, VNC connections must be made over a ssh tunnel as described below.<br />
<br />
'''''Appropriate use'''''<br />
<br />
VNC gives you the ability to establish a remote desktop on the login nodes, but this ability does '''''NOT''''' imply that you are permitted to run compute-intensive, GUI-driven applications on these machines. Such usage is not only contrary to CAC policy, it is disrespectful toward other users, because the login node may become unresponsive through your actions.<br />
<br />
Here is a good example of how to use VNC appropriately. By following these steps you can run (say) Abaqus in GUI-driven mode on a compute node that has been allocated to you through an interactive batch job.<br />
# Open a VNC connection to linuxlogin through an ssh tunnel using the instructions below, in order to gain access to a Linux desktop. Make sure two terminal windows are available on this desktop.<br />
# In one of the terminal windows, submit an interactive job to the queue of your choice (add the #PBS -I directive to your job submission script).<br />
# Once the job starts, you will be given a command prompt on your assigned machine. Note the result of "hostname". There is no need to enter further commands at this prompt (except to exit the job).<br />
# Go to the other terminal window and open a second ssh connection to the compute node using "ssh -Y <userid>@<hostname>"<br />
# This new ssh session will tunnel X-Windows from the compute node back to the VNC desktop. Therefore (if Abaqus is on your path), you can now open the Abaqus GUI using "abaqus cae -mesa".<br />
<br />
'''''Initial setup'''''<br />
''(You only need to do this once)''<br />
<br />
:* Install a VNC client if one isn't installed. [http://www.tightvnc.com/ TightVNC] works well, but so do others.<br />
:* Login to linuxlogin, and set the password for your VNC server using the "vncpasswd" command.<br />
<br />
'''''Start your VNC server'''''<br />
<br />
:* '''On linuxlogin''', start the VNC server using the "vncserver" command like this:<br />
vncserver -geometry 1024x768 -localhost<br />
The geometry numbers, 1024x768, specify the size, in pixels, of the desktop.<br />
:* You will need to get the display number from the output of the vncserver command:<br />
<br />
New 'linuxlogin.cac.cornell.edu:1 (shl1)' desktop is linuxlogin.cac.cornell.edu:1<br />
Starting applications specified in /home/gfs01/shl1/.vnc/xstartup<br />
Log file is /home/gfs01/shl1/.vnc/linuxlogin.cac.cornell.edu:1.log<br />
<br />
:* vncserver is running on port 5900 + display number. In the above example, the display number is :1, therefore vncserver is running on port 5901.<br />
<br />
'''''Connect your VNC client'''''<br />
<br />
:* Set up ssh forwarding on your client computer. Let's say the port number on linuxlogin is 5901 (as above), and your CAC userid is uid12. From Linux, type into a terminal:<br />
<br />
ssh -L 10000:localhost:5901 uid12@linuxlogin.cac.cornell.edu<br />
'''From Windows''', ssh clients such as PuTTY can do X11 port forwarding. See [[VNC Tunnel Windows]]. <br />
<br />
'''For MacOS X users''', see [[CAC VPN Server for MacOS Users |here]]<br />
:* Leave this ssh session running on your local client computer. (It can run in the background.)<br />
:* Launch your VNC client program. Connect to localhost:10000. When prompted, type in your VNC server password.<br />
<br />
'''''To disconnect your client'''''<br />
<br />
:* Close the vnc client program.<br />
:* Disconnect the ssh forwarding session (i.e., kill it).<br />
<br />
'''''To reconnect your client'''''<br />
<br />
:* Restart port forwarding with ssh, using the same remote port number as before.<br />
:* Again connect the VNC client to localhost:10000.<br />
<br />
'''''When you are all done'''''<br />
<br />
:* On linuxlogin, type this command to shut down the VNC server<br />
vncserver -kill :<display number><br />
:* If you merely log out from linuxlogin, it will leave the VNC server running. You must shut down the VNC server explicitly when you are finished with it. (Actually this can be a nice feature.)<br />
<br />
=====Passwordless SSH=====<br />
<br />
''''' Create ssh key pair '''''<br />
<br />
Your ssh key pair will only need to be created once. You will not need to repeat this step. You can complete this step from either a Linux or Windows login node. If this is your first login to a CAC login node, it will ask you to [[Getting_Started#Change_a_password_at_first_login|change your password]]. This will become your password for connecting to the nodes. <br />
<br />
Create your ssh key pair by logging into the linux login node (linuxlogin.cac.cornell.edu), which will begin the process of creating the keys; you can use the defaults or empty responses for all prompts.<br />
<br />
Alternatively, you can create your ssh key pair on the linux login node by logging into the Windows login node (winx64login.cac.cornell.edu), opening a Command Prompt window, and running <tt>plink.exe</tt> to connect to the linux login node, as shown in this example:<br />
<br />
<pre>>"C:\Programs Files (x86)\Putty\plink.exe" %USERNAME%@linuxlogin.cac.cornell.edu<br />
Password: Enter Your Password<br />
Rocks 5.0 (V)<br />
Profile built 12:54 06-May-2008<br />
<br />
Kickstarted 09:22 06-May-2008<br />
-----------------------------------------------------------<br />
Welcome to the Center for Advanced Computing Cluster!<br />
-----------------------------------------------------------<br />
Please send your questions to help@cac.cornell.edu<br />
-----------------------------------------------------------<br />
<br />
<br />
It doesn't appear that you have set up your ssh key.<br />
This process will make the files:<br />
/home/gfs01/cacshl1/.ssh/id_rsa.pub<br />
/home/gfs01/cacshl1/.ssh/id_rsa<br />
/home/gfs01/cacshl1/.ssh/authorized_keys<br />
<br />
Generating public/private rsa key pair.<br />
Enter file in which to save the key (/home/gfs01/cacshl1/.ssh/id_rsa): Press Enter to accept default<br />
Created directory '/home/gfs01/cacshl1/.ssh'.<br />
Enter passphrase (empty for no passphrase): Press Enter to accept default<br />
Enter same passphrase again: Press Enter to accept default<br />
Your identification has been saved in /home/gfs01/cacshl1/.ssh/id_rsa.<br />
Your public key has been saved in /home/gfs01/cacshl1/.ssh/id_rsa.pub.<br />
</pre><br />
<br />
After this is done, type '''"exit"''' to log out of the linux login node.<br />
<br />
''''' Convert ssh Private Key for Putty / Plink '''''<br />
<br />
Next run PuTTYgen to generate public and private keys to be used with PuTTY and Plink:<br />
<br />
:* Log in to <tt>winx64login.tc.cornell.edu </tt>(if you are not already)<br />
:* Run <tt>C:\Program Files (x86)\Putty\puttygen.exe</tt>.<br />
:* Select <tt>Import Key</tt> from the <tt>Conversions</tt> menu and select <tt>H:\.ssh\id_rsa</tt> in your home directory. And click on the <tt>Open</tt> button.<br />
<center>[[image:LoadPrivateKey.jpg]]</center><br />
:* Click on the <tt>"Save Private Key"</tt> button. <br />
<center>[[image:SavePrivateKey.jpg]]</center><br />
:* Click on "Yes" when asked to save the private key without a passphrase.<br />
:* Save the private key as private.ppk in the .ssh directory inside your home directory.<br />
<center>[[image:SpecifyPrivateKey.jpg]]</center><br />
:* Close (choose File, then Exit)<br />
:* To confirm you have converted the ssh private key successfully, do:<br />
<pre>"C:\Program Files (x86)\Putty\plink.exe" -i %HOMEDRIVE%\.ssh\private.ppk %USERNAME%@linuxlogin.cac.cornell.edu</pre><br />
It may notify you that "The server's host key is not cached in the registry." Type "y" to "store the key in cache."<br />
:* You should now be logged into linuxlogin without being prompted for a password. Stay logged in for the next step.</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Getting_Started&diff=901Getting Started2015-09-30T17:16:11Z<p>Ad876: </p>
<hr />
<div>__TOC__<br />
==Managing your password==<br />
CAC has a [[Getting_Started#Rules_for_Creating_Passwords|Password Policy]] in effect. The first time that you login to the <tt>cac.cornell.edu</tt> domain, you will be required to change your password. Each password must have at least eight characters and must contain at least three of the following four elements: (1) uppercase letters (2) lowercase letters (3) special characters (4) digits. Your password can be set or changed on any of the CAC login nodes, and the password will be updated on all CAC resources. Passwords expire every six months. Do not share your password. There are more detailed instructions below.<br />
<br />
===Rules for Creating passwords===<br />
{{:Rules for Creating Passwords}}<br />
<br />
===Change a password at first login===<br />
{{:Changing password at first login}}<br />
<br />
=== Change password at any time ===<br />
{{:Changing password any time}}<br />
<br />
=== Locked Accounts===<br />
<br />
There have been instances in which user accounts have been locked. Some common causes of locked accounts and the solutions are:<br />
<br />
:*Mistyping your password several times in a row. <br />
:::<tt>Solution</tt>: Wait about a 1/2 hour and then try again. Be sure that your caps lock key is not on!<br />
:*Trying to login to a Windows login node by using SSH when you have a new or expired password. <br />
:::<tt>Solution</tt>: Login to a Windows login node using Remote Desktop Connection or SSH to a linux login node.<br />
:*Failing to log off all other sessions connected to login nodes. <br />
:::<tt>Solution</tt>: Log off all remote connections. Disconnecting the sessions is not enough.<br />
:*Failing to disconnect locally mapped drives to the CAC file server before changing your password. <br />
:::<tt>Solution</tt>: Disconnect all locally mapped drives, wait a 1/2 hour until account is unlocked, and then re-map the drive with the new password.<br />
<br />
If you can't log on or can't wait you can submit a Password Reset ticket on our [https://rt.cac.cornell.edu/index.html issue tracking system]<br />
<br />
==Checking your CAC project ==<br />
Cornell University users can view their account limits at [https://{{SERVERNAME}}/services/cu/memberlimits.aspx CAC Account Limits].<br /><br />
Partner Program members should contact Paul Redfern at [mailto:red@cac.cornell.edu red@cac.cornell.edu] if they need information on their membership limits.<br />
<br />
==Using CAC resources==<br />
===Connecting to CAC===<br />
<br />
There are two types of login nodes:<br />
:* Linux login nodes: <tt>linuxlogin.cac.cornell.edu</tt> as well as the head nodes for the various Linux-based private clusters.<br />
:* Windows login node: <tt>winlogin.cac.cornell.edu</tt><br />
<br />
====Connect to Linux====<br />
<br />
{{:Connect to Linux}}<br />
<br />
====Connect to Windows====<br />
<br />
{{:Connect to Windows}}<br />
<br />
===Home Directory Access===<br />
{{:Home Directory Access}}<br />
<br />
===File transfer===<br />
{{:File transfer}}<br />
<br />
<br />
===Linux Usage Tips===<br />
{{:Linux Usage Tips}}<br />
==== More information on Linux nodes at CAC====<br />
For more detailed instructions on how to use the Linux node, see [[Tutorial for the Linux nodes at CAC| here]]<br />
<br />
===Windows Usage Tips===<br />
{{:Windows Usage Tips}}<br />
<br />
===More information===<br />
The CAC Web site is [https://{{SERVERNAME}}/ here] . There are many useful documents on the Support page at [[ Main Page| CAC documentation]].<br />
<br />
===Acknowledging CAC===<br />
{{:Acknowledging CAC}}<br />
<br />
===FAQ/Troubleshooting===<br />
#[[FAQ#Account| Account FAQ]]<br />
#[[FAQ#Login| Login FAQ]]<br />
# If you have more questions, see [[FAQ| here]]</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Connect_to_Linux&diff=900Connect to Linux2015-09-30T17:10:14Z<p>Ad876: /* Passwordless SSH */</p>
<hr />
<div>__TOC__<br />
There are three distinct ways to connect to a login node:<br />
# [[Getting_started#Using_Secure_Shell | Use SSH]] to open a Linux shell on a login node, which provides a text-only interface.<br />
# [[Getting_started#Using_Secure_Shell | Use SSH]] together with [[Getting_started#Using_X-Windows | X-Windows]], which sends any interactive graphics back to your machine window-by-window through an SSH tunnel.<br />
# [[Getting_started#Using_VNC | Use VNC]] to get a remote desktop with multiple text and graphics windows. This is not as straightforward as it sounds, due to the need to set up a secure tunnel for the remote desktop first.<br />
<br />
These instructions are intended mainly for users of personal computers and workstations. However, much of the material carries over to mobile computing platforms such as tablets and smartphones. You will have to locate and download an app to enable SSH or VNC connectivity; even a browser plug-in may suffice.<br />
<br />
Whichever method you choose, at your first login, you will be challenged for a new password. Find help at [[Getting_started#Change_a_password_at_first_login | Changing a Password at First Login]]. You will also be asked for an ssh passphrase. You can just leave this blank; hit the <tt>Enter</tt> key in response.<br />
<br />
=====Using Secure Shell=====<br />
For basic command-line access, a Secure Shell (SSH) client will give you a remote command shell on one of the login nodes.<br />
:* Nearly all Unix/Linux varieties (including Mac) already have a built-in SSH2 implementation, required by our clusters.<br />
:* If you are coming from a Microsoft Windows machine, an SSH2 client must first be installed, as described below.<br />
:* The non-secure predecessor of SSH, telnet, is disabled for security reasons.<br />
<br />
'''''Linux users:'''''<br />
<br />
To connect to the second login node with ssh, you simply open a terminal window and type<br />
localhost$ ssh username@linuxlogin.cac.cornell.edu<br />
<br />
'''''Mac OS X users:'''''<br />
<br />
OS X on the Mac is built on a version of Unix, so ssh is available directly from the Terminal application.<br />
:*One option is to use the shortcut <tt>'''cmd-space'''</tt> to open Spotlight and then type <tt>"Terminal"</tt> to open a Terminal window.<br />
Otherwise:<br />
:*Navigate in the Finder to the Applications folder and Utilities sub-folder.<br />
:*:[[Image:MacApplicationsFolder.png|500px]]<br />
:*Then double-click on the Terminal application to see a Bash command-line.<br />
:*:[[Image:MacTerminalWindow.png]]<br />
:*As in Linux, simply type "ssh username@linuxlogin.cac.cornell.edu" into this window.<br />
<br />
'''''Windows users:'''''<br />
<br />
Secure Shell (ssh) clients work nicely as long as they support the SSH2 protocol. As mentioned, telnet is disabled for security reasons. A popular client for Windows is the free [//www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY client].<br />
:* The simplest installation is to download the [//chiark.greenend.org.uk/~sgtatham/putty/download.html Windows installer], called '''putty-0.65-installer.exe''', and run it. This installs PuTTY into your Start menu.<br />
:* To connect, start PuTTY, then type in a host name such as linuxlogin.cac.cornell.edu, and click "Open".<br />
:*:[[Image:Putty_address.jpg|Setting the host name in PuTTY]]<br />
<br />
=====Using X-Windows=====<br />
<br />
X-Windows or X11 is the longstanding Unix mechanism for displaying interactive graphics in a window. Your "X server" software runs locally, but it is capable of displaying windows that have been generated either locally or remotely. An "X client" on a remote machine can create X-Windows for local display, but it is necessary first to establish a shell on that machine using SSH.<br />
<br />
'''''Appropriate use'''''<br />
<br />
Among other things, X-Windows gives you the ability to display a GUI that originates on a login node. However, this ability does '''''NOT''''' imply that you are permitted to run compute-intensive, GUI-driven applications on these machines. Such usage is not only contrary to CAC policy, it is disrespectful toward other users, because the login node may become unresponsive through your actions.<br />
<br />
'''''Linux users:'''''<br />
<br />
The standard way to use X-Windows is to tunnel the X-Windows protocol through an ssh connection. If you open your ssh session with the '''-X''' option, it will automatically set up the necessary tunnel and environment variables.<br />
localhost$ ssh -X username@linuxlogin.cac.cornell.edu<br />
linuxlogin$ echo $DISPLAY<br />
localhost:11.0<br />
linuxlogin$ xclock&<br />
You can see that your DISPLAY environment variable is set and test it with xclock. There is another option to use a trusted version of X-windows forwarding<br />
linuxlogin$ ssh -Y compute-3-48.v4linux<br />
Th trusted version is necessary for forwarding X11 connections from a compute node to the login node, then back to your client machine.<br />
<br />
'''''Mac OS X users:'''''<br />
<br />
If you start ssh with the '''-X''' or '''-Y''' option, X-Windows should start up automatically. You can then try the "xclock" test, as described above for Linux. <br />
<br />
X11 is preinstalled on Macs starting with OS X 10.6 (Snow Leopard). For Mac OS X 10.5 (Leopard), you may need to install X11 in order for X-Windows applications to launch. If there is no X11 application in the Applications->Utilities folder, you'll have to find your OS X install disk. From the ''Mac OS X Server Introduction to Command-Line Administration,'' "The X11 server and an application to access X windows from the Finder are available as an optional installation in the Optional Installs folder of your installation disc (X11 is in the Applications package)."<br />
<br />
'''''Windows users:'''''<br />
<br />
Along with your ssh client (e.g., PuTTY), you will need to install an X-Windows server on your Windows machine.<br />
:* [//straightrunning.com/XmingNotes/ Xming] - Open Source. A shareware contribution will get you a version with improved performance for graphics (GLX). There are two pieces to download<br />
:*:[[Image:Xming-download.jpg]]<br />
:** Xming-mesa (public domain release). There are two links together, one for Xming, one for Xming-mesa. Either will work, but Xming-mesa has some newer features that might come in handy some time.<br />
:** Xming-fonts (public domain release)<br />
If you purchase the website release of Xming, remember to install the Xming-fonts, as well.<br />
:* OpenText's [//cit.cornell.edu/services/software_licensing/available/exceed.cfm Exceed and Exceed 3D] - Cornell no longer has a site license. Installing Exceed 3D will improve performance of graphics applications. Exceed installs several icons under the Start menu. Choose the one that just says "Exceed" because it starts the program in multi-window mode, which is what we want.<br />
<br />
Here is how to start a session using PuTTY and Xming.<br />
<br />
# Start Xming from the Start menu. It will appear briefly and disappear except for an X in the application tray.<br />
# Start PuTTY.<br />
# In the window that appears, type a host name, <tt>linuxlogin.cac.cornell.edu</tt>.<br />
# Use the tree menu on the left to set X11 forwarding. It's in the <tt>Connection > SSH branch</tt>.<br />
#:[[Image:Putty_x11forwarding.jpg|Setting X11 forwarding in PuTTY]]<br />
# For PuTTY 0.61 only - In the "Auth" section of the SSH branch, go to GSSAPI and uncheck <tt>"Attempt GSSAPI authentication"</tt>. This will prevent an annoying <tt>"Access denied"</tt> message from appearing in your terminal window.<br />
# You can return to the Session category and Save this session's configuration for future use. Give it a logical name like linuxlogin.<br />
# Click Open, and it will connect to a login node.<br />
# Test your X-Windows setup by typing<br />
xclock<br />
You should see a clock appear in the corner of your screen. You can stop it by typing <tt>Ctrl-c</tt> in the terminal window.<br />
<br />
=====Using VNC=====<br />
<br />
[http://en.wikipedia.org/wiki/Vnc VNC] lets you see a whole Linux desktop from the login node on your computer.<br />
Using SSH and X-Windows is generally faster, and uses a lot less of the login node's resources,<br />
but VNC can be much faster if you are doing visualization on the login node from off campus.<br />
<br />
For security reasons, we are requiring all VNC connections to be tunneled inside ssh. You will therefore need to be able to connect to the login nodes [[Getting_started#Using_Secure_Shell | using SSH]]. Because the firewall running on linuxlogin blocks all incoming ports except for ssh, VNC connections must be made over a ssh tunnel as described below.<br />
<br />
'''''Appropriate use'''''<br />
<br />
VNC gives you the ability to establish a remote desktop on the login nodes, but this ability does '''''NOT''''' imply that you are permitted to run compute-intensive, GUI-driven applications on these machines. Such usage is not only contrary to CAC policy, it is disrespectful toward other users, because the login node may become unresponsive through your actions.<br />
<br />
Here is a good example of how to use VNC appropriately. By following these steps you can run (say) Abaqus in GUI-driven mode on a compute node that has been allocated to you through an interactive batch job.<br />
# Open a VNC connection to linuxlogin through an ssh tunnel using the instructions below, in order to gain access to a Linux desktop. Make sure two terminal windows are available on this desktop.<br />
# In one of the terminal windows, submit an interactive job to the queue of your choice (add the #PBS -I directive to your job submission script).<br />
# Once the job starts, you will be given a command prompt on your assigned machine. Note the result of "hostname". There is no need to enter further commands at this prompt (except to exit the job).<br />
# Go to the other terminal window and open a second ssh connection to the compute node using "ssh -Y <userid>@<hostname>"<br />
# This new ssh session will tunnel X-Windows from the compute node back to the VNC desktop. Therefore (if Abaqus is on your path), you can now open the Abaqus GUI using "abaqus cae -mesa".<br />
<br />
'''''Initial setup'''''<br />
''(You only need to do this once)''<br />
<br />
:* Install a VNC client if one isn't installed. [http://www.tightvnc.com/ TightVNC] works well, but so do others.<br />
:* Login to linuxlogin, and set the password for your VNC server using the "vncpasswd" command.<br />
<br />
'''''Start your VNC server'''''<br />
<br />
:* '''On linuxlogin''', start the VNC server using the "vncserver" command like this:<br />
vncserver -geometry 1024x768 -localhost<br />
The geometry numbers, 1024x768, specify the size, in pixels, of the desktop.<br />
:* You will need to get the display number from the output of the vncserver command:<br />
<br />
New 'linuxlogin.cac.cornell.edu:1 (shl1)' desktop is linuxlogin.cac.cornell.edu:1<br />
Starting applications specified in /home/gfs01/shl1/.vnc/xstartup<br />
Log file is /home/gfs01/shl1/.vnc/linuxlogin.cac.cornell.edu:1.log<br />
<br />
:* vncserver is running on port 5900 + display number. In the above example, the display number is :1, therefore vncserver is running on port 5901.<br />
<br />
'''''Connect your VNC client'''''<br />
<br />
:* Set up ssh forwarding on your client computer. Let's say the port number on linuxlogin is 5901 (as above), and your CAC userid is uid12. From Linux, type into a terminal:<br />
<br />
ssh -L 10000:localhost:5901 uid12@linuxlogin.cac.cornell.edu<br />
'''From Windows''', ssh clients such as PuTTY can do X11 port forwarding. See [[VNC Tunnel Windows]]. <br />
<br />
'''For MacOS X users''', see [[CAC VPN Server for MacOS Users |here]]<br />
:* Leave this ssh session running on your local client computer. (It can run in the background.)<br />
:* Launch your VNC client program. Connect to localhost:10000. When prompted, type in your VNC server password.<br />
<br />
'''''To disconnect your client'''''<br />
<br />
:* Close the vnc client program.<br />
:* Disconnect the ssh forwarding session (i.e., kill it).<br />
<br />
'''''To reconnect your client'''''<br />
<br />
:* Restart port forwarding with ssh, using the same remote port number as before.<br />
:* Again connect the VNC client to localhost:10000.<br />
<br />
'''''When you are all done'''''<br />
<br />
:* On linuxlogin, type this command to shut down the VNC server<br />
vncserver -kill :<display number><br />
:* If you merely log out from linuxlogin, it will leave the VNC server running. You must shut down the VNC server explicitly when you are finished with it. (Actually this can be a nice feature.)<br />
<br />
=====Passwordless SSH=====<br />
<br />
''''' Create ssh key pair '''''<br />
<br />
Your ssh key pair will only need to be created once. You will not need to repeat this step. You can complete this step from either a Linux or Windows login node. If this is your first login to a CAC login node, it will ask you to [[Getting_Started#Change_a_password_at_first_login|change your password]]. This will become your password for connecting to the nodes. <br />
<br />
Create your ssh key pair by logging into the linux login node (linuxlogin.cac.cornell.edu), which will begin the process of creating the keys; you can use the defaults or empty responses for all prompts.<br />
<br />
Alternatively, you can create your ssh key pair on the linux login node by logging into the Windows login node (winx64login.cac.cornell.edu), opening a Command Prompt window, and running <tt>plink.exe</tt> to connect to the linux login node, as shown in this example:<br />
<br />
<pre>>"C:\Programs Files (x86)\Putty\plink.exe" %USERNAME%@linuxlogin.cac.cornell.edu<br />
Password: Enter Your Password<br />
Rocks 5.0 (V)<br />
Profile built 12:54 06-May-2008<br />
<br />
Kickstarted 09:22 06-May-2008<br />
-----------------------------------------------------------<br />
Welcome to the Center for Advanced Computing Cluster!<br />
-----------------------------------------------------------<br />
Please send your questions to help@cac.cornell.edu<br />
-----------------------------------------------------------<br />
<br />
<br />
It doesn't appear that you have set up your ssh key.<br />
This process will make the files:<br />
/home/gfs01/cacshl1/.ssh/id_rsa.pub<br />
/home/gfs01/cacshl1/.ssh/id_rsa<br />
/home/gfs01/cacshl1/.ssh/authorized_keys<br />
<br />
Generating public/private rsa key pair.<br />
Enter file in which to save the key (/home/gfs01/cacshl1/.ssh/id_rsa): Press Enter to accept default<br />
Created directory '/home/gfs01/cacshl1/.ssh'.<br />
Enter passphrase (empty for no passphrase): Press Enter to accept default<br />
Enter same passphrase again: Press Enter to accept default<br />
Your identification has been saved in /home/gfs01/cacshl1/.ssh/id_rsa.<br />
Your public key has been saved in /home/gfs01/cacshl1/.ssh/id_rsa.pub.<br />
</pre><br />
<br />
After this is done, type '''"exit"''' to log out of the linux login node.<br />
<br />
''''' Convert ssh Private Key for Putty / Plink '''''<br />
<br />
Next run PuTTYgen to generate public and private keys to be used with PuTTY and Plink:<br />
<br />
:* Log in to <tt>winx64login.tc.cornell.edu </tt>(if you are not already)<br />
:* Run <tt>C:\Program Files (x86)\Putty\puttygen.exe</tt>.<br />
:* Select <tt>Import Key</tt> from the <tt>Conversions</tt> menu and select <tt>H:\.ssh\id_rsa</tt> in your home directory. And click on the <tt>Open</tt> button.<br />
<center>[[image:LoadPrivateKey.jpg]]</center><br />
:* Click on the <tt>"Save Private Key"</tt> button. <br />
<center>[[image:SavePrivateKey.jpg]]</center><br />
:* Click on "Yes" when asked to save the private key without a passphrase.<br />
:* Save the private key as private.ppk in the .ssh directory inside your home directory.<br />
<center>[[image:SpecifyPrivateKey.jpg]]</center><br />
:* Close (choose File, then Exit)<br />
:* To confirm you have converted the ssh private key successfully, do:<br />
<pre>"C:\Program Files (x86)\Putty\plink.exe" -i %HOMEDRIVE%\.ssh\private.ppk %USERNAME%@linuxlogin.cac.cornell.edu</pre><br />
It may notify you that "The server's host key is not cached in the registry." Type "y" to "store the key in cache."<br />
:* You should now be logged into linuxlogin without being prompted for a password. Stay logged in for the next step.</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Red_Cloud_with_MATLAB/Install&diff=888Red Cloud with MATLAB/Install2015-09-30T16:03:22Z<p>Ad876: </p>
<hr />
<div><br />
== Automated Installer Quick Start ==<br />
Note: if you plan to use Red Cloud with MATLAB from muliple client machines, the installation procedure must be completed on every client machine you plan to use.<br />
<br />
=== Prerequisites ===<br />
* MATLAB version R2010b, R2011a, R2011b, R2012a, or R2013a, with [http://www.mathworks.com/products/parallel-computing/index.html Parallel Computing Toolbox (PCT)]<br />
* An active CAC project (you must be a project member on a CAC project, and you must be enabled to use the project's subscription)<br />
<br />
=== Client Install ===<br />
Recommended preliminary step: [[Password_Policies_Linux | change your CAC password]] if it was just issued to you, or if you have not changed your password recently. For established users, [https://{{SERVERNAME}}/services/myacct.aspx an alternative method] is available. (You may want to review CAC's [[Rules_for_Creating_Passwords | password complexity requirements]] first.)<br />
# Start MATLAB. The install process must be run within the default startup folder (i.e. the one returned by the MATLAB command "userpath"). In the Linux commandline, you must first cd to "userpath" before starting matlab.<br />
# Have you ever previously installed the client code for Red Cloud with MATLAB, or the earlier MATLAB on the TeraGrid? If so, you will need to delete the cacscheduler configuration before proceeding, regardless of whether you deleted the client code: <br />
#* Select ''Parallel'', then ''Manage Configurations'' (in R2012a and R2013a: ''Manage Cluster Profiles'')<br />
#* Highlight ''cacscheduler''<br />
#* Select ''Edit'', then ''Delete'' (in R2012a and R2013a: click the ''Delete'' button)<br />
# Download and run the installation script<br />
## Download the Installation Script: <pre>>> urlwrite('https://{{SERVERNAME}}/redcloud/downloads/cac_install.m','cac_install.m')</pre> This will save a file called cac_install.m to your MATLAB user directory.<br />
## Run the installation script <pre>>> cac_install()</pre><br />
## cac_install.m can be removed if you wish<br />
# Restart MATLAB <pre>>> exit</pre><br />
<br />
=== Initial Configuration ===<br />
# Register your CAC (or XSEDE) Certificate. <pre> >> cacRegisterCertificate()</pre> You may skip this step if both of the following are true:<br />
#* The Red Cloud client code (this software installation) was previously installed on your machine, and<br />
#* You have not changed your password since the last time you ran <tt>cacRegisterCertificate()</tt><br />
# Create a MATLAB config file <pre>>> cac_make_config('YOUR_USER_NAME'); </pre> where <tt>YOUR_USER_NAME</tt> is your CAC-issued username. You may be prompted for a username and password.<br />
# Make this configuration the default:<br />
#* Drop down the ''Parallel'' menu<br />
#* Select the ''cacscheduler'' configuration.<br />
<br />
=== Verification ===<br />
# Set up the scheduler object; you may be prompted for your certificate username/password <pre>>> sched = findResource('scheduler', 'configuration', 'cacscheduler')</pre><br />
# Set Job Parameters<br />
## Set your wall time limit to 10 minutes <pre>>> ClusterInfo.setWallTime(10);</pre><br />
## Specify the quick turnaround queue <pre>>> ClusterInfo.setQueueName('Quick');</pre><br />
## Specify the project you are on <pre>>> ClusterInfo.setProjectName('YOUR_PROJECT_NAME')</pre>where YOUR_PROJECT_NAME is your CAC project name. It typically resembles xyz3_0001.<br />
# Run the tests. Normally this takes a few minutes; it will be longer if there are not enough free cores available <pre>>> cac_runtests(sched,0);</pre><br />
<br />
== Subsequent Sessions ==<br />
<br />
With the Red Cloud with MATLAB client installed, keep the following in mind each time you start MATLAB<br />
* MATLAB should remember your choice of scheduler between sessions. However, you should always verify that the selected scheduler is the one you want.<br />
** Under the ''Parallel'' menu, make sure ''cacscheduler'' is chosen whenever you wish to run jobs on Red Cloud with MATLAB<br />
** Make sure ''local'' (or some other choice) is selected when you do '''not''' want jobs to run on Red Cloud with MATLAB (e.g. testing locally)<br />
* Wall time limits and choice of queue persist between sessions. Always check these values to make sure they match your current assumptions. see <tt>help ClusterInfo</tt> for how to verify the various job settings.<br />
* If you are using a function or script that requires access to the scheduler data itself, you will need to create a MATLAB object as shown:<pre>sched = findResource('scheduler', 'configuration', 'cacscheduler');</pre><br />
* Run this periodically to download the newest contrib folder files: <pre>updateContrib();</pre><br />
<br />
== '''Manual Install''' ==<br />
These are the manual installation instructions for those who prefer to avoid using an installation script. For example, if you have a customized MATLAB environment or you are an administrator installing the software in a computer lab/shared environment, the detailed instructions give additional options on how to install the program.<br />
<br />
=== Terms ===<br />
<br />
* ''MATLABROOT'' refers to the MATLAB installation directory. <br />
** View in MATLAB using <tt>>> matlabroot</tt><br />
* ''MATLABSTARTUP'' refers to the default directory that MATLAB starts in. <br />
** View in MATLAB using <tt>>> userpath</tt><br />
* ''CACSCHEDULERHOME'' refers to the location where you install these files. <br />
** Be sure to substitute your folder path for ''CACSCHEDULERHOME'' in all installation steps.<br />
<br />
=== Installation ===<br />
Recommended preliminary step: [[Password_Policies_Linux | change your CAC password]] if it was just issued to you, or if you have not changed your password recently. For established users, [https://{{SERVERNAME}}/services/myacct.aspx an alternative method] is available. (You may want to review CAC's [[Rules_for_Creating_Passwords | password complexity requirements]] first.) Then:<br />
# Download the zip file, https://{{SERVERNAME}}/redcloud/downloads/cacscheduler-distro.zip<br />
# Unzip in a location accessible by MATLAB; we suggest ''MATLABSTARTUP''. This location will be known as ''CACSCHEDULERHOME''. <br />
#* Note: if this installation is a shared machine or lab environment, we suggest <tt>MATLABROOT\toolbox\local</tt><br />
#Start MATLAB<br />
#Add the installation path <pre>>> addpath('CACSCHEDULERHOME');</pre><br />
# Load additional paths and attempt to create startup.m <pre>>> cac_addpath();</pre><br />
#Load additional files <pre>>> updateContrib();</pre><br />
# Create classpath.txt <pre>>> cacMakeClassPath();</pre><br />
#If this installation is a shared machine or lab environment<br />
#* Copy startup.m to location accessible by all users, e.g. MATLABROOT ‘toolbox\local\startup.m’<br />
#* Save a copy of the original classpath.txt file, then copy classpath.txt to location accessible by all users , e.g. ''MATLABROOT'' ‘toolbox\local\classpath.txt’<br />
#The user(s) should then complete the [[Red_Cloud_with_MATLAB/Install#Initial_Configuration|Initial Configuration]] and [[Red_Cloud_with_MATLAB/Install#Verification|Verification]] steps above.<br />
<br />
== '''Installation Notes''' ==<br />
For releases prior to R2013a:<br />
* The cac_install script will attempt to create files called startup.m and classpath.txt in the default or <tt>userpath</tt> startup folder. If these files already exist, the user can choose either to replace them (by using the supplied "override" options) or modify them by hand. A copy of the existing classpath.txt file (if any) will be preserved.<br />
* In order to use the cacscheduler, MATLAB must always start up in the same directory where these files were created. Otherwise, the Java class path will not be set correctly.<br />
* If a user wants the Java path to be set correctly regardless of the startup folder, he or she should copy the new classpath.txt file into matlabroot/toolbox/local, after renaming the one already in that location.<br />
For R2013a:<br />
* A file called javaclasspath.txt (rather than classpath.txt) will always be created in the <tt>prefdir</tt> directory. A copy of any existing javaclasspath.txt file will be preserved.<br />
For a list of past and present client versions and associated release notes, please see CHANGELOG.txt in the ''CACSCHEDULERHOME'' directory.</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=FAQ&diff=877FAQ2015-09-30T15:33:32Z<p>Ad876: </p>
<hr />
<div>=Account=<br />
<br />
====How can I determine number of hours left on my allocation?====<br />
* Check the account management page at [https://{{SERVERNAME}}/services/projects.aspx].<br />
* When logged into one of the v4 linuxlogin nodes, you can run 'showbalance' to view remaining compute time. (If you have jobs currently running, the showbalance result has deducted the time requested for the current running job(s) and adjusts to time used once the current running job(s) complete.)<br />
<br />
====How can I obtain a CAC account?====<br />
See [https://{{SERVERNAME}}/services/projects.aspx Project Requests].<br />
<br />
====My account is locked.====<br />
If it was locked after repeated password failures, it should automatically unlock after 30 minutes. Otherwise: {{ContactCAC}}<br />
<br />
====I changed my password. Now I'm locked out.====<br />
{{ContactCAC}}<br />
<br />
<br />
====Forgot password.====<br />
{{ContactCAC}}<br />
<br />
====Problems with new password.====<br />
{{ContactCAC}}<br />
<br />
====Need password reset.====<br />
{{ContactCAC}}<br />
<br />
====Having trouble logging in but know your username and password are correct====<br />
To resolve, clear your browser cache and then login again.<br />
<br />
====Are my login id and password the same for all machines?====<br />
Yes. For an ssh connection give your login id at the prompt. With a Windows GUI, specify the User Name as <login_id>@tc.cornell.edu or CTC_ITH\<login_id>.<br />
<br />
====When I use a Remote Desktop Client to connect to winx64login, it says that my username/password are incorrect.====<br />
Make sure that you are logging using the CTC_ITH domain. If you just put your username in the "username" box, it will try to log you into winx64login as a local user, which won't work. Put CTC_ITH\<username> in the "username" box.<br />
<br />
<br />
=Login=<br />
<br />
====Which machines are the login nodes?====<br />
ctclogina, ctcloginb, ctcloginc, ctclogind. <br />
<br />
====Can't use login machine because of compute-bound processes on the machine.====<br />
{{ContactCAC}}<br />
<br />
====Can't get to login node with RDC. Times out.====<br />
{{ContactCAC}}<br />
<br />
====rdesktop gives an error message: $ rdesktop ctclogina.tc.cornell.edu ERROR: connect: Connection timed out====<br />
A firewall may be blocking outgoing connections. <br />
<br />
====Can't connect to login node.====<br />
{{ContactCAC}}<br />
<br />
====Can't login using ssh.====<br />
Try different type of connection and see if need to change password. Otherwise send email to useracct@tc.cornell.edu and ask to have password reset. <br />
<br />
====Could not get to scicenter2(sp?) machine, could yesterday====<br />
Terminal serve to login node to change password. <br />
<br />
====Can't get to ctclogina. Gets error msg "The specified remote computer could not be found."====<br />
Use complete name ctclogina.tc.cornell.edu.<br />
<br />
====mpirun command not found on login node.====<br />
This is as expected. Don't run jobs on login node. <br />
<br />
====Connect from login node to batch node. Disconnect from login node. At reconnect, session hung. Can't close window or logoff.====<br />
{{ContactCAC}}<br />
<br />
====I have a disconnected session on a login node. When I reconnect, the login screen is blank. What should I do?====<br />
Issue ctrl-shift-esc to bring up Task Manager. Select the Applications tab, then New Task. Enter "explorer" and click OK. A normal desktop should reappear. If it doesn't, send e-mail to consult@tc.cornell.edu and ask to be logged off. <br />
<br />
====I have a login process on ctcloginb that I can not log off.====<br />
{{ContactCAC}}<br />
<br />
====Wants to debug on login nodes in visual studio.====<br />
Told user why debugging is not permitted on login nodes. Suggested collaboratory. <br />
<br />
====Can't use rdc to login node.====<br />
{{ContactCAC}}<br />
<br />
====Can't close command windows on login node.====<br />
{{ContactCAC}}<br />
<br />
====Why does RDC to ctclogina fail?====<br />
It could be that you need to use the completely qualified name ctclogina.tc.cornell.edu.<br />
====How Do I Connect to CAC Machines to Run Programs?====<br />
There are two options:<br />
# '''Command-line access''' - [[SecureShell]] (Windows or Linux clusters from any computer)<br />
#* More efficient over slower network connections.<br />
#* [[WindowsXWindows]] X-Windows provides pointing-and-clicking, if you want.<br />
# '''Work with a desktop of the CAC computer''' - [[RemoteDesktop]] (to Windows), [[VNCAccess]] (to Linux)<br />
#* Maybe more familiar for moving files and editing.<br />
#* You will still have to use the command line to submit jobs.<br />
<br />
=Batch=<br />
<br />
====My batch job includes vii0047, but I can't login and MPI/Pro says: MPI/Pro error: Failed to login the user on server: vii0047.tc.cornell.edu System Error: Logon failure: the user has not been granted the requested logon type at this com.====<br />
{{ContactCAC}} <br />
<br />
====Output from batch is not copied to H:.====<br />
{{ContactCAC}} <br />
<br />
====Allocated 2 nodes, only allowed to use remote desktop connection to master node.====<br />
{{ContactCAC}} <br />
<br />
====MPI/Pro Error:SocketException System Error:No connection could be made because the target machine actively refused it.====<br />
{{ContactCAC}} <br />
<br />
====Copied files to T:\%USERNAME%, but job doesn't give output.====<br />
Must cd to T:\%USERNAME% before running job. <br />
<br />
====Can't move or delete some files in T: on some batch nodes.====<br />
{{ContactCAC}} <br />
<br />
====Batch jobs that just disappear from queue, having done nothing.====<br />
User had set some parameters with a space before and after the = sign, putting a trailing space on the parameter. Remove the spaces. <br />
<br />
====What can users do about the long time it takes for jobs to clear?====<br />
See the "MPI Cleanup" tip at http://www.tc.cornell.edu/services/support/batch/faster_cleanup.asp<br />
<br />
====Is there a way to make the copyback.bat (which copies the output files periodocally ) file to copy output from all the nodes to the H: drive====<br />
Yes. Start /b mpirun -np N parallel_copyback.bat <br />
<br />
====Need to have different files for each process. How to do this? Problem doing this by a system call in C++ program.====<br />
As part of setup file, use commands <br />
cd /D T: <br />
del /Q T:\%USERNAME%<br />
mkdir T:\%USERNAME%\%MSTI_RANK%<br />
copy files.* T:\%USERNAME%\%MSTI_RANK% <br />
<br />
====Jobs are stuck clearing.====<br />
{{ContactCAC}} <br />
<br />
====How to direct jobs from remote machines to CAC for batch? Need software on CAC batch nodes.====<br />
Explained that we can do this and how.<br />
<br />
====What do I need to do to use v3?====<br />
See http://www.tc.cornell.edu/Services/Policies/Pages/usage.htm<br />
<br />
====Copy of executable and input files failed on vi0004.====<br />
System problem. {{ContactCAC}} <br />
<br />
====I have an error about the path when connecting to a batch machine.====<br />
<br />
Check your userlogin.bat file. There may be a reference to Visual Studio(VS) in userlogin.bat, but VS is not on batch nodes. Change syntax to "call setup_visualc.bat" or call a different setup file as appropriate.<br />
<br />
====Can I telnet to batch machines?====<br />
No. You need to use a remote desktop connection from a login node to login interactively to a machine on which you have a job running.<br />
<br />
=Files=<br />
<br />
====How can I copy files to my desktop from H:?====<br />
Use SSH client to sftp files. See [[File_Transfer_To_Clusters]].<br />
<br />
====Can't use scp to transfer files to the CAC.====<br />
Use sftp.<br />
<br />
====Problems using WinSCP.====<br />
Use sftp. <br />
<br />
====Needed to share a file with a colleague outside the university. This is typically available on to CAC personnel.====<br />
Showed how to use outgoing ftp folder and sent detailed instructions by email.<br />
<br />
====Can't access files.====<br />
System problem. Send email to consult@tc.cornell.edu.<br />
<br />
====Can see files in explorer, but sees files only in home directory with dir at command prompt.====<br />
User had navigated Start | Run, then typed the command command. Needs to use the command cmd.<br />
<br />
====How Do I Transfer Files To and From CAC Machines?====<br />
# '''Use a program to send them''' - [[SecureShell]]<br />
#* Faster over slower connections.<br />
#* Less hassle.<br />
# '''Make your CAC home directory look like a local drive''' - [[FileAccess]]<br />
#* Works fine on campus.<br />
#* Convenient for editing.<br />
<br />
If you have any questions, please [mailto:help@cac.cornell.edu?subject=CAC Web site contact Send email] or call 607.254.8686.<br />
<br />
====Why use a temporary directory====<br />
'''''It is faster to perform local file I/O and copy complete data files to/from $HOME at the beginning and the end of the job, rather than perform I/O over the network ($HOME is network mounted on the compute nodes).''''' <br />
'''<br />
<br />
* Torque creates a uniquely named directory (/tmp/$PBS_JOBID) when a job starts and stores the path of this directory in the $TMPDIR environment variable. This directory is cleaned up when the job exits.<br />
** To use this feature, reference $TMPDIR<br />
<br />
* You may create directories for file read/writes outside your /tmp/$PBS_JOBID in /tmp. You do risk leaving any data there; it may be deleted at any time we see /tmp getting full.<br />
<br />
=Red Cloud=<br />
==How secure is Red Cloud==<br />
<br />
=== Red Cloud Security ===<br />
[https://{{SERVERNAME}}/redcloud Red Cloud], CAC's infrastructure as a service cloud, runs [http://eucalyptus.com Eucalyptus] cloud management software. Because Eucalyptus implements an Amazon Web Service (AWS) compatible private cloud, Red Cloud's security model follows closely after AWS.<br />
<br />
=== User Interface and API ===<br />
==== User Authentication ====<br />
Red Cloud accepts two types of user authentication: password and AWS-style keys consisting of 2 randomly generated strings. Users log into the [http://euca3.cac.cornell.edu web management console] using passwords. The user name and password is authenticated against CAC's Active Directory via Kerberos. For making AWS compatible API calls, users can obtain their keys from the web console. All API calls are SSL encrypted, as are web console sessions.<br />
==== User Access Management ====<br />
Eucalyptus fully implements AWS's Identity and Access Management (IAM) features. Group and user polices can be used for controlling access on per resource and API call basis. See AWS's [https://aws.amazon.com/documentation/iam IAM documentation] for details.<br />
<br />
===Instance Access Control ===<br />
Red Cloud runs Eucalyptus in "Managed" mode to implement security group and elastic IP address features described below. In Managed mode, all user data passed within the cloud infrastructure are VLAN tagged according to the security groups. The network switch connecting the cloud controller and physical nodes running the instances performs layer 2 switching guaranteeing network isolation between security groups. Instances have no access to network packets belonging to other instances outside their own security groups.<br />
<br />
To provide elastic IP addresses, Eucalyptus configures iptables running on the cloud controller host to perform the required source and destination network address translation (SNAT and DNAT).<br />
<br />
These features are implemented in Red Cloud infrastructure, independent of the configurations by the users on their instances.<br />
<br />
==== Security Group====<br />
Each instance (virtual machine) is assigned a security group at launch time. A security group is a private network in the cloud where network access between instances in the same security group is unrestricted.<br />
<br />
Access to an instance from outside its security group is subject to the group's access rules. Users can define the access rules by protocol, source IP address and destination port.<br />
<br />
Instances have unrestricted outbound access to the Internet.<br />
==== Elastic IP Address ====<br />
Each instance is assigned a private IP address belonging to its security group at launch time. An ephemeral routable public IP address is also assigned so the instance can be accessed from the Internet. Users can optionally reserve fixed public IP addresses that they can assign to their instances. Assigning a reserved public address to a running instance takes just a few seconds and does not require rebooting the instance.<br />
<br />
=== Cloud Infrastructure ===<br />
Cloud infrastructure hosts (cloud controller, storage controller, and the physical nodes running the instances) run CentOS 6 Linux distribution on a private network isolated from cloud user traffic.<br />
<br />
== How do I give my virtual server a domain name? ==<br />
A virtual server in Red Cloud is randomly assigned an IP address from 128.84.8.101 to 128.84.8.196 every time it is booted. If you want to create a domain name for your virtual server (e.g. mycloudserver.cac.cornell.edu) that stays consistent, follow the instructions on [[Using Dynamic DNS with Red Cloud]] page.<br />
<br />
== Why won't ssh let me log in to my virtual server? ==<br />
* You may not have given your instance a keypair for root access when you started it up. You should always use the -k option to assign one of the keypairs named in <tt>euca-describe-keypairs</tt> to your instance:<br />
<br />
euca-run-instances -n 1 -k mykey [...etc...]<br />
<br />
* If you get a response that looks like this:<br />
<br />
-sh-3.2$ ssh -X -i mykey.private root@128.84.8.105<br />
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@<br />
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @<br />
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@<br />
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!<br />
Someone could be eavesdropping on you right now (man-in-the-middle attack)!<br />
It is also possible that the RSA host key has just been changed.<br />
The fingerprint for the RSA key sent by the remote host is...<br />
<br />
...most likely this means that the numeric IP address for your instance (128.84.8.105 in the above example) has been assigned to you previously for a different instance. For a typical Linux ssh client, the way to fix this is to edit ~/.ssh/known_hosts on your machine, deleting the line that contains the re-used numeric IP address together with its old RSA fingerprint. For an ssh client on Windows or Mac, you might need to consult the documentation for that particular client.<br />
<br />
== Can I create clones of my office workstation in Red Cloud? ==<br />
Yes, if you have a Linux workstation. Your goal will be to take an image of your hard disk drive and combine it with a kernel and a ramdisk (from an official Eucalyptus image) that gives it virtio and iscsi support. Here is the outline of what you do:<br />
# Check <tt>euca-describe-images</tt>. If your preferred Linux kernel doesn't appear in our pre-registered list of kernels, please [mailto:help@cac.cornell.edu contact us].<br />
# Make sure the appropriate kernel modules for your chosen kernel and ramdisk are preloaded in your /lib/modules directory. <br />
# Use <tt>dd</tt> to capture your hard disk into an image. Here are some helpful links: [http://www.backuphowto.info/linux-backup-hard-disk-clone-dd Linux Backup: Hard Disk Clone with "dd"], and [http://www.linuxweblog.com/dd-image Image Your Hard Drive using dd].<br />
# Transfer your workstation's root disk image to cloud-login. This could take some time, assuming your disk image has a typical size in the multi-GB range.<br />
# Follow the procedure for [[Red_Cloud#Uploading_a_Root_Disk_Image | Uploading a Root Disk Image]] in the [[Red_Cloud | Red Cloud user guide]].<br />
<br />
==How do I migrate my data and image from Eucalyptus 3==<br />
<br />
===When to migrate===<br />
<br />
If there is little customization in your instance or image, it may be easier to copy over files using rsync or sftp, which have widely available documentation online and <br />
in your system's man pages. However, if your instance has been highly customized and you think it may take more than a few hours to recreate your system, then you may wish<br />
to follow the migration instructions below. If you wish to migrate but do not wish to perform the migration yourself, [https://{{SERVERNAME}}/services/rates/ consulting is available].<br />
<br />
===Migrate Image from Euca 3===<br />
<br />
====EBS Image====<br />
===== In Euca 3 Cloud =====<br />
* Find the snapshot ID of the EBS image you are migrating:<br />
[shl1@localhost shl1]$ euca-describe-images emi-5A38465A<br />
IMAGE emi-5A38465A 448419271023/centos-6-ebs 448419271023 available public x86_64 machine ebs hvm <br />
BLOCKDEVICEMAPPING /dev/sdb <br />
BLOCKDEVICEMAPPING /dev/sda '''snap-F3FA421D''' 10 true<br />
* If you want to migrate an existing EBS instance, <br />
*# Stop your EBS instance, and <br />
*# Take a snapshot of EBS volume that hosts your instance's root disk.<br />
* Create a volume from the snapshot:<br />
[shl1@localhost shl1]$ euca-create-volume --snapshot snap-F3FA421D -z redcloud<br />
VOLUME vol-42E23E2E 10 snap-F3FA421D redcloud creating 2014-10-02T18:52:51.868Z<br />
[shl1@localhost shl1]$ euca-describe-volumes vol-42E23E2E<br />
VOLUME vol-42E23E2E 10 snap-F3FA421D redcloud available 2014-10-02T18:52:51.868Z standard<br />
* Attach the volume to an Euca 3instance:<br />
[shl1@localhost shl1]$ euca-attach-volume -i i-0DBC41A2 -d /dev/vdb vol-42E23E2E<br />
ATTACHMENT vol-42E23E2E i-0DBC41A2 /dev/vdb attaching 2<br />
* ssh into the Euca 3 instance to which the volume is attached. <br />
[shl1@localhost shl1]$ ssh -i ~/euca3/mykey.private root@128.84.9.138<br />
Last login: Thu Apr 3 14:22:37 2014 from stevenleemac.cac.cornell.edu<br />
[root@euca-172-16-173-49 ~]# lsblk<br />
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT<br />
vda 253:0 0 100G 0 disk <br />
├─vda1 253:1 0 10G 0 part /<br />
├─vda2 253:2 0 89.5G 0 part <br />
└─vda3 253:3 0 512M 0 part [SWAP]<br />
vdb 253:16 0 10G 0 disk <br />
└─vdb1 253:17 0 10G 0 part <br />
[root@euca-172-16-173-49 ~]# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vda1 9.9G 1.2G 8.3G 13% /<br />
tmpfs 2.0G 0 2.0G 0% /dev/shm<br />
/dev/vda2 89G 184M 84G 1% /mnt/ephemeral0<br />
* If your volume is from a snapshot of a live instance's root disk running CentOS 6, 7, and Ubuntu, mount the root file system on the EBS volume somewhere in the instance and remove the etc/udev/rules.d/70-persistent-net.rules file or your migrated image will not be able to boot the instance in Euca 4. Unmount the root file system on the EBS volume before proceeding to the next step.<br />
* Create an img file of the attached EBS volume:<br />
[root@euca-172-16-173-49 ~]# cd /mnt/ephemeral0/<br />
[root@euca-172-16-173-49 ephemeral0]# dd if=/dev/vdb of=./centos-6.img <br />
20971520+0 records in<br />
20971520+0 records out<br />
10737418240 bytes (11 GB) copied, 332.985 s, 32.2 MB/s<br />
* Copy the .img file to an Euca 4 instance using the "rsync --sparse" command:<br />
[root@euca-172-16-173-49 ephemeral0]# rsync --sparse centos-6.img shl1@128.84.8.100:/localdisk/shl1/<br />
* Note: to login as root with ssh, you will want to continue using the same key that was used in Euca3, as this is not updated in the above process.<br />
<br />
===== In Euca 4 =====<br />
* Create an EBS volume matching the size of the EBS image:<br />
[shl1@ip-128-84-11-214 shl1]$ euca-create-volume -s 10 -z redcloud-ith<br />
VOLUME vol-317a4980 10 redcloud-ith creating 2014-10-02T19:49:45.283Z<br />
[shl1@ip-128-84-11-214 shl1]$ euca-describe-volumes vol-317a4980<br />
VOLUME vol-317a4980 10 redcloud-ith available 2014-10-02T19:49:45.283Z standard<br />
* Attach the EBS volume to an Euca4 instance:<br />
[shl1@ip-128-84-11-214 shl1]$ euca-attach-volume -i i-dfbb062f -d /dev/vdc vol-317a4980<br />
ATTACHMENT vol-317a4980 i-dfbb062f /dev/vdc attaching 2014-10-03T14:05:01.694Z<br />
[shl1@ip-128-84-11-214 shl1]$ ls /dev/vd*<br />
/dev/vda /dev/vda1 /dev/vda2 /dev/vdb /dev/vdb1 /dev/vdb2 /dev/vdc<br />
* As root in the Euca 4 instance, use dd to restore the image from Euca 3 the attached volume. Detach the volume from the instance when done.<br />
-bash-4.1# dd if=./centos-6.img of=/dev/vdc<br />
and then:<br />
[shl1@ip-128-84-11-214 shl1]$ euca-detach-volume vol-317a4980<br />
ATTACHMENT vol-317a4980 i-dfbb062f /dev/vdc detaching 2014-10-03T14:05:01.697Z<br />
[shl1@ip-128-84-11-214 shl1]$ euca-describe-volumes vol-317a4980<br />
VOLUME vol-317a4980 10 redcloud-ith available 2014-10-02T19:49:45.283Z standard<br />
* Take a snapshot of the volume:<br />
[shl1@ip-128-84-11-214 shl1]$ euca-create-snapshot -d "CentOS 6 Image from Euca 3" vol-317a4980<br />
SNAPSHOT snap-cdcc5768 vol-317a4980 pending 2014-10-03T14:38:11.727904951369483 10 CentOS 6 Image from Euca 3<br />
[shl1@ip-128-84-11-214 shl1]$ euca-describe-snapshots snap-cdcc5768<br />
SNAPSHOT snap-cdcc5768 vol-317a4980 completed 2014-10-03T14:38:11.727Z 100% 904951369483 10 CentOS 6 Image from Euca 3<br />
* Register the image:<br />
[shl1@ip-128-84-11-214 shl1]$ euca-register -n centos-6-ebs-from-euca-3 -a x86_64 -d "CentOS 6 EBS image from Euca 3" -b /dev/sdb=ephemeral0 -s snap-cdcc5768<br />
IMAGE emi-655661f2<br />
[shl1@ip-128-84-11-214 shl1]$ euca-describe-images<br />
IMAGE emi-655661f2 904951369483/centos-6-ebs-from-euca-3 904951369483 available private x86_64 machine ebs hvm <br />
BLOCKDEVICEMAPPING EPHEMERAL /dev/sdb ephemeral0<br />
BLOCKDEVICEMAPPING EBS /dev/sda snap-cdcc5768 10 true<br />
<br />
====Instance Store Image====<br />
<br />
===== In Euca 3 Cloud =====<br />
* On a host with euca2ools 3.0.x installed, or a m1.small instance running emi-E2A53625 (CentOS 6.6 with euca2ools)., download and unbundle the instance-store image you want to migrate to Euca 4. You should get a .img file after running the eaca-unbundle command.<br />
source <path to your Euca 3 credentials>/eucarc<br />
euca-download-bundle -b <bucket name> -m <manifest> -d <working directory><br />
euca-unbundle -m <manifest> -s <working directory> -d <working directory><br />
* Copy the .img file to an instance running the same image (or launch a new one with the same image). <br />
* Log into that instance as root.<br />
* Mount the .img file you just copied over somewhere.<br />
mkdir /mnt/target<br />
mount -o loop <image file> /mnt/target<br />
* Install the kernel version for your distribution listed in the following table:<br />
yum --installroot=/mnt/target install 2.6.18-400.1.1.el5.x86_64<br />
{| class="wikitable"<br />
|Distribution<br />
|Kernel Version<br />
|-<br />
|CentOS 5<br />
|2.6.18-400.1.1.el5.x86_64<br />
|-<br />
|CentOS 6<br />
|2.6.32-504.8.1.el6.x86_64<br />
|-<br />
|CentOS 7<br />
|3.10.0-123.20.1.el7.x86_64<br />
|-<br />
|Ubuntu 14.04<br />
|3.13.0-44-generic<br />
|}<br />
* Umount the image.<br />
umount /mnt/target<br />
<br />
===== In Euca 4 Cloud =====<br />
* Copy the image to a host running euca2ools 3.1.2 to upload to Euca 4 cloud. This host could be an instance in Euca 4 running emi-4404c688 image.<br />
* Bundle and upload the image to Euca 4. Use the eki and eri ID listed in the following table<br />
euca-bundle-image -i <image file> --kernel <eki> --ramdisk <eri> -r x86_64 -d <working directory path><br />
euca-upload-bundle -b <bucket name> -m <manifest><br />
eaca-register -n <image name> -a x86_64 <manifest path from the euca-upload-bundle command><br />
{| class="wikitable"<br />
|Distribution<br />
|Kernel Image<br />
|RAM Disk Image<br />
|-<br />
|CentOS 5<br />
|eki-076589e4<br />
|eri-819f562a<br />
|-<br />
|CentOS 6<br />
|eki-a04f8296<br />
|eri-683a4412<br />
|-<br />
|CentOS 7<br />
|eki-08e13f6f<br />
|eri-f0e9b392<br />
|-<br />
|Ubuntu 14.04<br />
|eki-8609afc5<br />
|eri-01ed05dd<br />
|}<br />
* Launch your Euca 4 instance with the migrated image. The image will need to be converted when it is run the first time so it will take a few minutes. eaca-describe-instances <instance ID> will show the progress of the image conversion in the instance's tags:<br />
TAG instance i-daba2262 euca:image-conversion-state active<br />
TAG instance i-daba2262 euca:image-conversion-status Converting images<br />
<br />
==Red Cloud et Amazon Web Services (AWS)==<br />
===How do I migrate an Amazon Web Services (AWS) EC2 image to Red Cloud===<br />
# Download the bundle from AWS and decrypt it. You will end up with an image file:<br />
#:<blockquote>ec2-download-bundle -b <S3 bucket name> -d .</blockquote><br />
#:<blockquote>ec2-unbundle -s . -d . <manifest></blockquote><br />
# Mount this image somewhere using "mount -o loop" option. <br />
## Edit <image mount point>/etc/fstab and <image mount point>/etc/grub.conf such that the root disk is /dev/vda instead of /dev/xvde used by AWS.<br />
## Download the tarball corresponding to your Linux distribution [https://euca3.cac.cornell.edu/lib-modules here]. Unpack the tarball in /lib/modules<br />
##:<blockquote>cd <image mount point>/lib/modules; tar jxvf <path to the tarball></blockquote><br />
## Unmount image<br />
# Bundle image for Red Cloud:<br />
#:<blockquote>euca-bundle-image -i <path to image file> -d <working directory> --kernel <eki> --ramdisk <eri></blockquote><br />
#:Use the following <eki> and <eri> according to your Linux distribution:<br />
#* CentOS 5.10: eki-CE97382C and eri-91003AD3<br />
#* CentOS 6.5: eki-921637A4 and eri-52B4381E<br />
# Upload the bundle to Red Cloud:<br />
#:<blockquote>euca-upload-bundle -b <bucket name> -m <manifest from the previous euca-bundle-image command></blockquote><br />
# Register the image in Red Cloud:<br />
#:<blockquote>euca-register -a x86_64 <bucket name>/<manifest></blockquote><br />
<br />
===How do I migrate a Red Cloud (instance store) image to Amazon Web Services (AWS) ===<br />
# Download the bundle from Red Cloud and decrypt it. You will end up with an img file.<br />
#:<blockquote>euca-download-bundle -b <bucket name> -d .</blockquote><br />
#:<blockquote>euca-unbundle -s . -d . <manifest> </blockquote><br />
# Mount this image somewhere using "mount -o loop" option. <br />
## Edit /etc/fstab and /etc/grub.conf such that the root disk is /dev/xvde instead of /dev/vda like on Red Cloud.<br />
## Make sure your instance store image has the latest and greatest CentOS 6 kernel installed. If not, do <br />
##:<blockquote>yum --installroot <mount point of the image> install kernel</blockquote> <br />
##Check <mount point of the image>/etc/grub.conf to make sure it looks right to you. Add "console=hvc0" to the end of the kernel line (reference: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/UserProvidedKernels.html)<br />
##unmount image<br />
##Create an AWS bundle and upload it using "ec2-bundle-image" and "ec2-upload-bundle" commands.<br />
# According to this [http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/UserProvidedKernels.html article], You will want to register the image with kernel aki-919dcaf8 in your ec2-register -k command, assuming you want to run it in us-east-1 region. Or select the appropriate aki ID for the region you want to run.<br />
<br />
==Euca-describe-instances or the web console says my instance is running, but why is it not responding to ping or ssh connections?==<br />
External access from the Internet to Red Cloud instances is blocked by default. Follow the instructions in the '''[[Red_Cloud_(Eucalyptus_3)#Managing_Network_Access | Manage Network Access]]''' section to enable network access to the instance.<br />
<br />
=Red Cloud with Matlab=<br />
== Why does the cacscheduler configuration seem to fail MATLAB's built-in validation test? ==<br />
<br />
MATLAB allows you to "validate" a configuration via the Parallel > Manage Configurations menu. If you do this for cacscheduler, the first few steps will work fine. But upon reaching the Parallel section of the procedure the validation will appear to fail with a message like, "Please set the maximum number of workers to a finite value prior to submission of a parallel job." This is expected behavior. The CAC parallel configuration purposely specifies a ClusterSize of Inf (infinity). This allows for total flexibility in adding or subtracting hardware from the resource and/or its various queues over time.<br />
<br />
In spite of this alarming-looking message from MATLAB's built-in test, it is not at all a sign that your setup is somehow defective. To make the full validation procedure work, simply do the following: in the Parallel menu, go to Manage Configuations, then double-click "cacscheduler". This will allow you to edit the CAC configuration directly. Change the ClusterSize parameter from Inf to (say) 4 and click "Save". Re-run the validation; you should find that cacscheduler now passes. When you're done with the test, change the ClusterSize back to Inf and save again.<br />
<br />
From the above, it should be clear that in any parallel job you submit, you'll want to set job.MaximumNumberOfWorkers appropriately for the queue to which you are submitting your job.<br />
<br />
Generally, if you are concerned about whether you have a working configuration, it's best to try running cac_runtests. This will test more aspects of Red Cloud's functionality.<br />
<br />
== How many MATLAB workers can you use at a time? ==<br />
<br />
The answer depends on both the job type and the queue to which you submit.<br />
<br />
For a parallel job, the workers must all be able to communicate with each other; therefore, the max size is limited to the number of cores that are present in your chosen queue. In the Default queue, there are 52 cores, so you could have up to 52 parallel workers in there. In the Quick queue, the max is 4; in the GPU queue, it's 8.<br />
<br />
For a pool job, the max is again limited by the number of cores. But bear in mind that one worker must take the place of your local MATLAB session, and its only role is to run the main job function for you. This means that the matlabpool size will be 1 less than the number of workers. Therefore, in the Default queue the max is 51 labs (52 workers), in Quick it's 3 labs, and in GPU it's 7 labs.<br />
<br />
For a distributed job, there is no limit, essentially. All tasks are independent, and they will make their way through any of the queues singly--task by task by task--until the list is exhausted. However, at any given instant, a maximum of 52 of these tasks could be running simultaneously in the Default queue, etc.<br />
<br />
== How do I save extremely large arrays in my pool job? ==<br />
<br />
In a pool job, if a distributed array is too large to <tt>gather()</tt> into the memory of the master process, you can <tt>save()</tt> it piecemeal from within a spmd or parfor loop, using the technique described [http://www.mathworks.in/support/solutions/en/data/1-D8103H/index.html?product=ML&solution=1-D8103H here]. Subsequently, you may reassemble the array on your local workstation, after transferring its various pieces via gridFTP.<br />
<br />
<br />
<br />
=Linux Batch=<br />
==Scheduler Frequently Asked Questions==<br />
{{ContactCAC}}<br />
====Why are you using Maui and Torque now?====<br />
We have switched to using a nationally recognized resource manager and scheduler in order to make the usage of our systems align more closely with the national community. This also allows us to leverage the considerable capabilities of the Maui software to ensure optimal and flexible use of our systems.<br />
====When's my job going to run?====<br />
If you have already submitted your job and you'd like to know that, use the '''showstart''' command to find estimated start times. If you are trying to decide where to run your job so that it runs the soonest, you'll want to examine the '''showbf''' command. This allows you to search for when a job with particular resource requirements will run.<br />
====Why is my job stuck in the queue?====<br />
Sometimes your job doesn't run, even though it looks like it should. Maybe there are few jobs running in the cluster, and your job still won't run.<br />
# Find your jobids with "showq -u username"<br />
# Use "checkjob -v jobid" to examine one of the jobs. [[Examining Checkjob -v]] discusses how to read this output.<br />
Jobs in the "Batch Hold" state initiate emails to the system administrators. For other problems, contact CAC help.<br />
====Why is my job deferred?====<br />
There can be several reasons for a job to defer. Sometimes when the Maui scheduler's queue is full, two jobs attempt to start on a node at the same time, and one will switch to being deferred. On this occasion, if you type "checkjob -v <jobid>", you will see, at the bottom, the message:<br />
Message[0] job rejected by RM 'scheduler' - job started on hostlist<br />
compute-3-40.v4linux,compute-3-37.v4linux,compute-3-35.v4linux,compute-3-34.v4linux<br />
at time 13:11:22_07/20, job reported idle at time 13:11:53_07/20 (see RM logs for details)<br />
In this case, the only way to make this job run is to notify help at CAC.<br />
====What are the queues/affiliations?====<br />
Affiliations was the term used by the vsched scheduler to indicate the name of the queue that jobs were submitted to. Most schedulers use the term queue (The scheduler also uses the term "class" to represent the same entity), so you can substitute the word you prefer. V4 queues are listed on the [[v4 Linux Cluster]] page.<br />
<br />
====When I try to run mpdboot I get an error regarding bad python version====<br />
This type of message goes on to say, "You can't run mpdboot on ['compute-3-44.v4linux'] version of python must be >= 2.4, current..." Mpdboot uses python and ssh to start MPI daemons on all nodes of your job. It begins by using ssh to ask what version of python is running on each node.<br />
<br />
Usually, this error means that ssh is having a problem establishing communication for the<br />
mpds. First, make sure you added "-r ssh" to your mpdboot line. If that<br />
looks OK, then try to rename (mv) the .ssh directory in your home directory<br />
to something like .ssh_bak. Log out, and log back in. A new .ssh<br />
directory should be recreated for you automatically (you can verify with<br />
"ls -la") which should have valid keys in it.<br />
<br />
You may also get this error if you are using a version of Python which does not work with mpdboot. In general, mpdboot needs python 2.3 or newer, but it gets very picky about versions newer than 2.4, as well. If you are trying to run Python 2.5 or 2.6 from your own directory, sometimes mpdboot will find only older versions when it does ssh to the other nodes in your job (because a non-interactive ssh can have a different path). One way to ensure mpdboot runs properly in this case is to ensure it uses the system copy of python. In bash, you can set the path for a command before you invoke it, here so that the system Python is used.<br />
PATH=/usr/bin:/bin:/opt/intel/impi/3.1/bin64/ mpdboot ...<br />
<br />
====What variables does PBS define in the job script?====<br />
Some of the variables are listed in [http://www.adaptivecomputing.com/resources/docs/torque/2-5-9/commands/qsub.php qsub documentation] but a good way to see the working environment is to submit a batch job which just does "env>variables.txt" and look for the ones starting in "PBS_".<br />
<br />
====No Job Control Warning for CSH and TCSH====<br />
The output file from the script starts with the error:<br />
Warning: no access to tty (Bad file descriptor).<br />
Thus no job control in this shell.<br />
This warning means that the <tt>fg</tt>, <tt>bg</tt>, and ampersand will not work in your script files. If your default user shell is csh or tcsh, the job will try to execute your script using csh or tcsh, and you'll get this warning. Bash doesn't have this problem.<br />
<br />
You can force your script to start with the Bash shell using a PBS directive:<br />
#PBS -S /bin/sh<br />
When Torque starts your job, it will now use Bash, but it won't actually call your .bashrc. If you have any startup files to modify the path or set other variables, you can add to the start of your script, after the PBS directives:<br />
source ~/.bashrc<br />
<br />
Another nice way to ensure your favorite variables are defined is to submit the script with the -V option:<br />
nsub -V batch.sh<br />
This option copied whatever environment variables you have defined on the command line to the script when it runs. In short, if you could run something interactively, it should run when the scheduler executes the job.<br />
<br />
====Mpiexec Won't Accept -ppn Argument====<br />
The default MPI, Intel MPI, requires that you put the -ppn argument before the -np argument.<br />
The nodes have at least three versions of mpiexec installed. The default is Intel MPI under /opt/intel. If you modify your shell's path, in .bashrc or .cshrc, to put /usr/local/bin before the default path, then you may be getting the [http://www.osc.edu/~pw/mpiexec/ OSC mpiexec]. This version does not depend on mpdboot. It talks directly with Torque to start jobs. A drawback is that the OSC mpiexec, on our system, cannot start more than one job per node. That's why it's not the default one to use.<br />
<br />
====I cannot find my output file====<br />
If you do not specify an output file when submitting a batch script, then it will automatically produce a file with a name like 110432.scheduler.v4linux.OU in the directory which was the working directory when you submitted your job. If you specify an output file with a command like "#PBS -o out.txt", then that file will be in your $HOME directory. This behavior has changed in recent versions of the scheduler.<br />
<br />
<br />
{{Template:ContactCAC}}<br />
<br />
=Compilers=<br />
====Where is nmake?====<br />
C:\Program Files\Microsoft Visual Studio\VC98\bin\nmake. Call setup_visualc.bat <br />
<br />
====How can you find the cl compiler?====<br />
Call setup_visualc.bat <br />
<br />
====forrtl: severe (157): Program Exception - access violation====<br />
Segmentation fault. Look for a place where writing more than declared. <br />
<br />
====Trouble with stack overflow in a Compaq Visual Fortran program.====<br />
Increase the stack reserve quota, through a flag to nmake or using editbin.<br />
<br />
====Intel 8.1 compiler gives stack overflow. Intel 7.1 fine. What to do? 0: forrtl: severe (170): Program Exception - stack overflow====<br />
Increase the space available on the stack with the flag /F, where is the size of the stack in bytes. The default is 1000000. Try /F10000000. Increase as necessary.<br />
<br />
====Can't find uuid.lib.====<br />
It's in C:\Program Files\Microsoft SDK\lib on the login nodes.<br />
<br />
====LINK fatal error LNK1201: error writing to program database H:\users\...\some.pdb; check for insufficient disk space, invalid path, or insufficient privilege.====<br />
Suspicion is that there is an older version of the file some.pdb. Delete that file and rebuild.<br />
<br />
====How do I use Intel Fortran at the command line?====<br />
First, call setup_intelf32.bat. The compilation command is ifort.<br />
<br />
====What is the command line syntax to compile with OpenMP?====<br />
See the info provided by "ifort -h". There are 4 options beginning with /Qopenmp.<br />
<br />
====Does the CAC have a tutorial on OpenMP with Fortran?====<br />
No, we don't. The focus is on MPI.<br />
<br />
====Getting convergence errors with Intel 8.1 Fortran with /O1, /O2, /O3. Answer comes out OK. Performance not obviously degraded. How can I fix this so that I don't get the errors?====<br />
Add /Op flag to enable better floating point precision. The convergence errors disappear.<br />
<br />
====I would like to debug an optimized Intel Fortran code, compiled with a flag such as /O2 , created either as a Release version in Visual Studio (VS) or at a command prompt. A Debug version in VS sets the correct debugging flags, but disables optimization. How do I set the appropriate debugging environment for a Release version in VS or at a command prompt?====<br />
Add the command-line flags /Zi /debug:full /traceback. Specify the linker option /pdbfile:filename.pdb to create the program database file. This file and the executable must be copied to the same directory on T: when you run the program.<br />
<br />
==== Can the Intel C compiler handle makefile dependencies without having to use cygwin's makedepend?====<br />
<br />
Yes. You can use the /QMM compiler option, which is OFF by default.<br />
* /QM - Generates makefile dependency lines for each source file, based on the #include lines found in the source file.<br />
* /QMD - Preprocess and compile. Generate output file (.d extension) containing dependency information.<br />
* /QMF file - Generate makefile dependency information in file. Must specify /QM or /QMM.<br />
* /QMG - Similar to /QM, but treats missing header files as generated files.<br />
* /QMM - Similar to /QM, but does not include system header files.<br />
* /QMMD - Similar to /QMD, but does not include system header files.<br />
<br />
=Network Drive=<br />
===H: Network Drive===<br />
<br />
====Mapping H:, can't see files.====<br />
Make sure that the DNS settings are correct. Look under [[Home_Directory_Access]] for DNS instructions.<br />
<br />
====Can't map H: any more. Nothing changed.====<br />
Could be that the password had expired. Connect to login node with RDC to change password, then map drive. <br />
<br />
====Can't find H:.====<br />
Send e-mail to consult@tc.cornell.edu.<br />
<br />
====Problems mapping H:. Can see files in CAC Tools but not home directory.====<br />
Disconnect H: and remap.<br />
<br />
====At home, can see his home directory, but no files.====<br />
Only certain domains can map H: (need vpn) <br />
<br />
====Can't see the files in one of his directories.====<br />
Permissions problem. Send email to useracct@tc.cornell.edu <br />
<br />
====Mapping H: with correct DNS settings, but can's see files.====<br />
Send email to consult@tc.cornell.edu. <br />
<br />
====Cannot see files on H:====<br />
Send email to consult@tc.cornell.edu. <br />
<br />
====User can now map drive but cannot enter directory. Files are located on ctcfsrv8\tc_k.====<br />
User needs correct DNS settings. User resolved by pointing to 128.84.5.28 (ctcfsrv8) in his host file. <br />
<br />
====Can't map H: with DFS in MAC OS X.====<br />
MAC user's need to obtain Thursby to map H:. <br />
<br />
====Using rover, pointing to the ctc winsserver does not allow him to see files when mapping H:.====<br />
Try mapping ctcfsrv8, which is where the files are. This worked. Can't use the DNS settings with rover unless using vpn. It isn't trusted the way Cornell ip addresses are.<br />
<br />
====Can't access files.====<br />
System problem. Send email to consult@tc.cornell.edu. <br />
<br />
====Can see files in explorer, but sees files only in home directory with dir at command prompt.====<br />
User had navigated Start | Run, then typed the command command. Needs to use the command cmd.<br />
<br />
=Web services=<br />
====User wants access to CAC web space for a personal web page.====<br />
This is available only for CAC personnel.<br />
<br />
====Old links break on new CAC web site.====<br />
Navigate from the [https://{{SERVERNAME}}/ CAC home page].</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Mainpage&diff=871Mainpage2015-09-30T15:10:01Z<p>Ad876: </p>
<hr />
<div><html><br />
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.4.0/css/font-awesome.min.css"><br />
<table role="presentation" style="border:0; margin: 0;" width="100%" cellspacing="10"><br />
<tr> <br />
<td valign="top" id="mainpage_opportunitiescell"><br />
<div class="mainpage_boxtitle">WELCOME TO CAC SUPPORT</div><br />
<br />
<div class="mainpage_boxcontents_small"><br />
<p>This wiki provides Cornell University Center for Advanced Computing <a href="/">(CAC)</a> users with user documentation and other kinds of support information. If you're not a current user and would like to become one, visit <a href="/services/projects.aspx">how to start a project</a>. If you are a PI, visit <a href="/services/projects/manage.aspx">how to manage your current project</a>. Please remember to <a href="/docwiki/index.php?title=Acknowledging_CAC">acknowledge CAC support</a> in your publications.</p><br />
</div><br />
</td><br />
</tr><br />
</table><br />
<br />
<br />
<table role="presentation" style="border:0; margin: 0;" width="100%" cellspacing="10"><br />
<tr><br />
<td valign="top" class="mainpage_hubbox"><br />
<br />
<div class="col-sm-4 panel-item"><br />
<a class="panel panel-circle-contrast" href="/docwiki/index.php?title=Special:Search"><br />
<div class="panel-icon"><br />
<i class="fa fa-info-circle fa-5x"></i><br />
</div><br />
<div class="panel-body text-center"><br />
<h4 class="panel-title">Search support</h4><br />
<p>Search CAC support site.</p><br />
</div><br />
</a><br />
</div><br />
</td><br />
<td valign="top" class="mainpage_hubbox"><br />
<br />
<div class="col-sm-4 panel-item"><br />
<a class="panel panel-circle-contrast" href="//rt.cac.cornell.edu/index.html"><br />
<div class="panel-icon"><br />
<i class="fa fa-question-circle fa-5x"></i><br />
</div><br />
<div class="panel-body text-center"><br />
<h4 class="panel-title">Contact support</h4><br />
<p>Submit a ticket or call 607-254-8691.</p><br />
</div><br />
</a><br />
</div><br />
</td> <br />
<td valign="top" class="mainpage_hubbox"><br />
<br />
<br />
<div class="col-sm-4 panel-item"><br />
<a class="panel panel-circle-contrast" href="/datafeed/status.aspx"><br />
<div class="panel-icon"><br />
<i class="fa fa-check-circle fa-5x"></i><br />
</div><br />
<div class="panel-body text-center"><br />
<h4 class="panel-title">Check operating status</h4><br />
<p>Plan ahead for CAC infrastructure downtimes.</p><br />
</div><br />
</a><br />
</div><br />
</td> <br />
</tr><br />
</table><br />
<br />
<br />
<table role="presentation" style="border:0; margin: 0;" width="100%" cellspacing="10"><br />
<tr><br />
<br />
<!-- POUR LA PREMIERE COLONNE: USER DOCUMENTATION --><br />
<td valign="top" id="mainpage_opportunitiescell"><br />
<div class="mainpage_boxtitle">USER DOCUMENTATION</div><br />
<div class="mainpage_boxcontents_small"><br />
<ul><br />
<li><a href="/docwiki/index.php?title=Getting_Started">Getting Started</a><span> - password rules, home directories, and more</span></li><br />
<li><a href="/docwiki/index.php?title=Red_Cloud" >Red Cloud</a><span> - on-demand cloud services</span></li><br />
<li><a href="/docwiki/index.php?title=Private_Clusters"> Private Clusters</a> <span> - maintained by CAC </span></li><br />
<li><a href="/docwiki/index.php?title=Archival_Storage" >Archival Storage</a><span> - how to use and</span><a href="/docwiki/index.php?title=Syncing_to_Archival_Storage"> sync directories</a> <span> to CAC Archival Storage</span> </li><br />
<li><a href="/docwiki/index.php?title=File_Transfer_using_Globus">File Transfer using Globus</a><span> - high speed file transfers to/from CAC</span></li><br />
</ul><br />
</div><br />
</td><br />
<br />
<!-- TRAINING & EDUCATION --><br />
<td valign="top" id="mainpage_opportunitiescell"><br />
<div class="mainpage_boxtitle">TRAINING & EDUCATION</div><br />
<div class="mainpage_boxcontents_small"><br />
<ul><br />
<li><a href="/education/Default.aspx"> CAC Education and Outreach</a> <span> - overview </span></li><br />
<li><a href="https://www.xsede.org/training1" >XSEDE Training</a><span> - CAC is training lead (NSF program)</span></li><br />
<li><a href="https://cvw.cac.cornell.edu/topics">Cornell Virtual Workshop</a><span> - online training </span></li><br />
<li><a href="http://www.math.cornell.edu/~scan/" >SCAN</a><span> - Scientific Computing and Numerics seminar</span></li><br />
<li><a href="http://cornell-cs5220-f15.github.io/">CS 5220</a><span> - Applications of Parallel Computers course</span></li><br />
</ul><br />
</div><br />
</td><br />
<br />
<br />
</tr><br />
</table><br />
<br />
</html></div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=THECUBE_Cluster&diff=869THECUBE Cluster2015-09-30T15:01:59Z<p>Ad876: </p>
<hr />
<div>This is a private cluster.<br />
<br />
==Hardware==<br />
:* Head node: '''thecube.cac.cornell.edu'''.<br />
:* access modes: ssh<br />
:* Rocks 6.1 with CentOS 6.3<br />
:* 32 compute nodes with Dual 8-core E5-2680 CPUs @ 2.7 GHz, 128 GB of RAM<br />
:* THECUBE Cluster Status: [http://thecube.cac.cornell.edu/ganglia/ Ganglia].<br />
:* Submit HELP requests: [https://{{SERVERNAME}}/help help] OR by sending an email to [mailto:help@cac.cornell.edu CAC support] please include THECUBE in the subject area.<br />
<br />
==File Systems==<br />
===Home Directories===<br />
:* Path: ~<br />
<br />
User home directories is located on a NFS export from the head node. Use your home directory (~) for archiving the data you wish to keep. Do NOT use this file system for computation as bandwidth to the compute nodes is very limited and will quickly be overwhelmed by file I/Os from large jobs.<br />
<br />
'''Unless special arrangements are made, data in user's home directories are NOT backed up.'''<br />
<br />
===Scratch File System===<br />
LUSTRE file system provided by Terascala and Dell<br />
:* Path: /scratch/<user name><br />
<br />
The scratch file system is a fast parallel file system. Use this file system for scratch space for your jobs. Copy the results you want to keep back to your home directory for safe keeping.<br />
<br />
==Scheduler/Queues==<br />
:* Maui/Torque scheduler <br />
:* Queues:<br />
::{| border="1" cellspacing="0" cellpadding="10"<br />
! Name<br />
! Description<br />
! Time Limit<br />
|-<br />
| default<br />
| all nodes<br />
| no limit<br />
|}<br />
<br />
=Software=<br />
Set up the working environment for each package using the module command. <br />
The module command will activate dependent modules if there are any. <br />
<br />
To show currently loaded modules:<br />
<pre><br />
-sh-4.1$ module list<br />
Currently Loaded Modulefiles:<br />
1) openmpi-1.6.5-intel-x86_64<br />
</pre><br />
To show all available modules (as of Sept 30, 2013):<br />
<pre><br />
-sh-4.1$ module avail<br />
<br />
----------------------------------- /usr/share/Modules/modulefiles -----------------------------------<br />
dot module-info null rocks-openmpi_ib<br />
module-cvs modules rocks-openmpi use.own<br />
<br />
------------------------------------------ /etc/modulefiles ------------------------------------------<br />
boost-1.54.0 mathematica-9.0 sas-9.3<br />
cmake-2.8.11.2 matlab-r2013a valgrind-3.8.1<br />
eclipse-4.3 netcdf-4.3.0 visit-2.5.2<br />
hdf5-1.8.11 openmpi-1.6.5-intel-x86_64 zlib-1.2.8<br />
</pre><br />
To load a module and verify:<br />
<pre><br />
-sh-4.1$ module load mathematica-9.0<br />
-sh-4.1$ module list<br />
Currently Loaded Modulefiles:<br />
1) openmpi-1.6.5-intel-x86_64 2) mathematica-9.0<br />
</pre><br />
To unload a module and verify:<br />
<pre><br />
-sh-4.1$ module unload mathematica-9.0<br />
-sh-4.1$ module list<br />
Currently Loaded Modulefiles:<br />
1) openmpi-1.6.5-intel-x86_64 <br />
</pre><br />
<br />
SOFTWARE LIST<br />
::{| border="1" cellspacing="0" cellpadding="10"<br />
! Software<br />
! Path<br />
! Notes<br />
|-<br />
| Intel Compilers (including MKL) || /opt/intel <br />
|<br />
* Included in user's default path. <br />
|-<br />
| Openmpi 1.6.5 || /opt/openmpi<br />
|<br />
* Included in user's default path. <br />
|-<br />
| Mathematica || /opt/Mathematica<br />
|<br />
* module load mathematica-9.0 <br />
|-<br />
| Matlab || /opt/MATLAB<br />
|<br />
* module load matlab-r2013a<br />
|-<br />
| SAS || /opt/SAS<br />
|<br />
* module load sas-9.3<br />
|-<br />
| Boost || /opt/boost<br />
|<br />
* module load boost-1.54.0<br />
|-<br />
| cmake || /opt/cmake<br />
|<br />
* module load cmake-2.8.11.2<br />
|-<br />
| eclipse|| /opt/eclipse<br />
|<br />
* module load eclipse-4.3<br />
|-<br />
| hdf5|| /opt/hdf5<br />
|<br />
* module load hdf5-1.8.11<br />
|-<br />
| netcdf|| /opt/netcdf<br />
|<br />
* module load netcdf-4.3.0<br />
|-<br />
| valgrind|| /opt/valgrind<br />
|<br />
* module load valgrind-3.8.1<br />
|-<br />
| visit|| /opt/visit<br />
|<br />
* module load visit-2.5.2<br />
|-<br />
| zlib|| /opt/zlib<br />
|<br />
* module load zlib-1.2.8<br />
|-<br />
| acml|| /opt/acml<br />
|<br />
* AMD Core Math Library<br />
* no module file<br />
* not in default path<br />
|-<br />
| R, ffmpeg || /usr/bin<br />
|<br />
* in default path<br />
|-<br />
| BLAS, LAPACK || libraries<br />
|<br />
* in default path<br />
|-<br />
| Thrust || <br />
|<br />
* Coming soon<br />
|}<br />
<br />
==Quick Tutorial==<br />
The batch system treats each core of a node as a "virtual processor." That means the nodes keyword in batch scripts refers to the number of cores that are scheduled.<br />
<br />
===Running an MPI Job on the Whole Cluster===<br />
:*We are assuming /opt/openmpi/ is the default, which it is on thecube cluster. The mpiexec options may change depending on your selected MPI.<br />
:*First use showq to see how many cores are available. It may be less than 512 if a node is down.<br />
<pre><br />
-sh-4.1$ showq<br />
ACTIVE JOBS--------------------<br />
JOBNAME USERNAME STATE PROC REMAINING STARTTIME<br />
<br />
<br />
0 Active Jobs 0 of 512 Processors Active (0.00%)<br />
0 of 32 Nodes Active (0.00%)<br />
<br />
IDLE JOBS----------------------<br />
JOBNAME USERNAME STATE PROC WCLIMIT QUEUETIME<br />
<br />
<br />
0 Idle Jobs<br />
<br />
BLOCKED JOBS----------------<br />
JOBNAME USERNAME STATE PROC WCLIMIT QUEUETIME<br />
<br />
<br />
Total Jobs: 0 Active Jobs: 0 Idle Jobs: 0 Blocked Jobs: 0<br />
</pre><br />
<br />
:*Next create a script (using your favorite editor ex. vim) named runmyfile.sh that contains the following lines of code:<br />
<br />
<source lang="bash"><br />
#!/bin/sh<br />
#PBS -l nodes=32:ppn=16 (note, this is PBS -l (small case L))<br />
#PBS -N test<br />
#PBS -j oe<br />
#PBS -S /bin/bash<br />
<br />
set -x<br />
cd "$PBS_O_WORKDIR"<br />
<br />
mpiexec --hostfile $PBS_NODEFILE <executable> (substitute executable for the program you wish to run)<br />
</source><br />
:*Submit the job to the cluster <br />
<pre><br />
-sh-4.1$qsub runmyfile.sh<br />
</pre><br />
:*Look for the output file in a file named test.<br />
<br />
===Running an MPI Job using 16 Tasks Per Node===<br />
Because the nodes have 16 physical cores, you may want to limit jobs to 16 tasks per node.<br />
The node file lists each node 1 time, so make a copy with each node listed 16 times, and<br />
hand that version to MPI.<br />
<br />
<source lang="bash"><br />
#!/bin/sh<br />
#PBS -l nodes=64<br />
#PBS -N test<br />
#PBS -j oe<br />
#PBS -S /bin/bash<br />
<br />
set -x<br />
cd "$PBS_O_WORKDIR"<br />
<br />
# Construct a copy of the hostfile with only 16 entries per node.<br />
# MPI can use this to run 16 tasks on each node.<br />
uniq "$PBS_NODEFILE"|awk '{for(i=0;i<16;i+=1) print}'>nodefile.16way<br />
<br />
# to Run 16-way on 4 nodes, we request 64 core to obtain 4 nodes<br />
mpiexec --hostfile nodefile.16way ring -v<br />
</source><br />
<br />
===Running Many Copies of a Serial Job===<br />
In order to run 30 separate instances of the same program, use the scheduler's task array feature, through the "-t" option. The "nodes" parameter here refers to a core.<br />
<br />
<source lang="bash"><br />
#!/bin/sh<br />
#PBS -l nodes=1 (note, this is PBS -l (small case L))<br />
#PBS -t 30<br />
#PBS -N test<br />
#PBS -j oe<br />
#PBS -S /bin/bash<br />
<br />
set -x<br />
cd "$PBS_O_WORKDIR"<br />
echo Run my job.<br />
</source><br />
<br />
When you start jobs this way, separate jobs will pile one-per-core onto nodes like a box of hamsters.<br />
<br />
===Running on a specific node===<br />
To run on a specific node use the host= option <br />
<br />
<source lang="bash"><br />
#!/bin/sh<br />
#PBS -l host=compute-1-16 (note, this is PBS -l (small case L))<br />
#PBS -N test<br />
#PBS -j oe<br />
#PBS -S /bin/bash<br />
<br />
<br />
set -x<br />
cd "$PBS_O_WORKDIR"<br />
echo Run my job.<br />
</source><br />
===Running an interactive job===<br />
from the command line:<br />
qsub -l nodes=1 -I<br />
<br />
==HELP==<br />
:* THECUBE Cluster Status: [http://thecube.cac.cornell.edu/ganglia/ Ganglia].<br />
:* Submit HELP requests: [https://{{SERVERNAME}}/help help] OR by sending email to: help@cac.cornell.edu, please include THECUBE in the subject area.</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=MARVIN_Cluster&diff=864MARVIN Cluster2015-09-30T14:57:11Z<p>Ad876: </p>
<hr />
<div>__TOC__<br />
This is a private cluster.<br />
<br />
==Hardware==<br />
:* Head node: marvin.cac.cornell.edu.<br />
:* access modes: ssh<br />
:* Rocks 5.4.3 with CentOS 5.6<br />
:* 92 compute nodes with Dual 6-core X5670 CPUs @ 3 GHz, Hyperthreaded, 48 GB of RAM; 4 high memory nodes with 96 GB of RAM<br />
:* Cluster Status: [http://marvin.cac.cornell.edu/ganglia/ Ganglia].<br />
:* Submit HELP requests: [FAQ#Why_use_a_temporary_directory/help help] OR by sending email to: help@cac.cornell.edu<br />
<br />
==File Systems==<br />
===Home Directories===<br />
:* Path: ~<br />
<br />
User home directories is located on a NFS export from the head node. Use your home directory (~) for archiving the data you wish to keep. Do NOT use this file system for computation as bandwidth to the compute nodes is very limited and will quickly be overwhelmed by file I/Os from large jobs.<br />
<br />
Unless special arrangements are made, data in user home directories are NOT backed up.<br />
<br />
===Scratch File System===<br />
LUSTRE file system provided by Terascala and Dell<br />
Path: /scratch/<user name><br />
<br />
The scratch file system is a fast parallel file system. Use this file system for scratch space for your jobs. Copy the results you want to keep back to your home directory for safe keeping.<br />
<br />
==Scheduler/Queues==<br />
:* Maui/Torque scheduler; <br />
:* Queues:<br />
::{| border="1" cellspacing="0" cellpadding="10"<br />
! Name<br />
! Description<br />
! Time Limit<br />
|-<br />
| viz<br />
| 4 visualization Ensight Servers, each has 96GB RAM<br />
| 24 hours<br />
|-<br />
| default<br />
| all nodes except for those in viz queue<br />
| 24 hours<br />
|-<br />
| long<br />
| all nodes except for those in viz queue<br />
| 72 hours<br />
|-<br />
| all<br />
| all nodes<br />
| none<br />
|}<br />
<br />
==Software==<br />
::{| border="1" cellspacing="0" cellpadding="10"<br />
! Software<br />
! Path<br />
! Notes<br />
|-<br />
| Intel Cluster Studio || /opt/intel <br />
|<br />
:* 30-day trial license. <br />
:* Intel compilers in user default path. <br />
:* Use mpi-selector to select Intel MPI<br />
|-<br />
| gcc 4.6.2 || /opt/gcc/4.6.2 || Prepend /opt/gcc/4.6.2/bin to $PATH to use this gcc version<br />
|-<br />
| openmpi 1.6.3 (gnu) || /opt/openmpi/gnu/1.6.3<br />
|<br />
:* Compiled by gcc 4.6.2<br />
:* To select this MPI implementation, use "mpi-selector --set openmpi-1.6.3-gcc-4.6.2" command. Log out and log back in. This will set gcc 4.6.2 as the default compiler as well.<br />
|-<br />
| openmpi 1.6.3 (Intel) || /opt/openmpi/intel/1.6.3<br />
|<br />
:* Compiled by Intel 12.1<br />
:* To select this MPI implementation, use "mpi-selector --set openmpi-1.6.3-intel" command. Log out and log back in.<br />
|-<br />
| openmpi 1.4.4 (gnu) || /opt/openmpi/gnu/1.4.4 <br />
|<br />
:* Compiled by gcc 4.6.2<br />
:* To select this MPI implementation, use "mpi-selector --set openmpi-1.4.4-gcc-4.6.2" command. Log out and log back in. This will set gcc 4.6.2 as the default compiler as well.<br />
|-<br />
| openmpi 1.4.4 (Intel) || /opt/openmpi/intel/1.4.4 <br />
|<br />
:* Compiled by Intel 12.1<br />
:* To select this MPI implementation, use "mpi-selector --set openmpi-1.4.4-intel" command. Log out and log back in.<br />
|-<br />
| mvapich 1.2 (gnu) || /opt/mvapich/gnu/1.2 <br />
|<br />
:* Compiled by gcc 4.6.2<br />
:* To select this MPI implementation, use "mpi-selector --set mvapich-1.2-gcc-4.6.2" command. Log out and log back in. This will set gcc 4.6.2 as the default compiler as well.<br />
|-<br />
| mvapich 1.2 (Intel) || /opt/mvapich/intel/1.2<br />
|<br />
:* DO NOT USE -- So far, the Intel 12.1 compiler has failed to produce a working build of mvapich.<br />
:* (in the future) To select this MPI implementation, use "mpi-selector --set mvapich-1.2-intel" command. Log out and log back in. <br />
|- <br />
| Intel MPI || /opt/intel/impi/3.1<br />
| <br />
:* To select this MPI implementation, use "mpi-selector --set intel-4.0.3" command. Log out and log back in.<br />
|-<br />
| fftw 3.3 (gnu) || /opt/fftw/gnu/3.3 <br />
|<br />
:* Compiled by gcc 4.6.2<br />
:* With Intel compilers, Use MKL (Intel Math Kernel Library) in /opt/intel/mkl.<br />
|-<br />
| lapack 3.4.0 (gnu) || /opt/lapack/gnu/3.4.0<br />
|<br />
:* Compiled by gcc 4.6.2<br />
:* With Intel compilers, Use MKL (Intel Math Kernel Library) in /opt/intel/mkl.<br />
|-<br />
| hypre 2.0.0 (gnu) || /opt/hypre/gnu/2.0.0 || Compiled by gcc 4.1.2<br />
|-<br />
| hypre 2.6.0b (gnu) || /opt/hypre/gnu/2.6.0b || Compiled by gcc 4.6.2<br />
|-<br />
| hypre 2.6.0b (Intel) || /opt/hypre/intel/2.6.0b || Compiled by Intel Compilers 12.1<br />
|-<br />
| ensight 9.2 || /usr/local/CEI <br />
|<br />
* Installed only on the head node and "viz" queue (compute-3-13 to compute-3-16)<br />
|-<br />
| ensight 10.0 || /usr/local/CEI <br />
|<br />
:* Installed only on the head node and "viz" queue (compute-3-13 to compute-3-16)<br />
|-<br />
| VisIt 2.9.2 || /opt/visit<br />
|<br />
:* Must default to use the following OpenMPI version for parallel visualization like this:<br />
<pre><br />
-bash-3.2$ mpi-selector --set openmpi-1.6.3-gcc-4.6.2<br />
Defaults already exist; overwrite them? (y/N) y<br />
-bash-3.2$ mpi-selector --query<br />
default:openmpi-1.6.3-gcc-4.6.2<br />
level:user <br />
</pre><br />
:* Log out and log back in for the change to take effect.<br />
|-<br />
| Anaconda Python || /opt/anaconda-python<br />
|<br />
:* Add the following line to your ~/.bashrc to use anaconda python:<br />
<pre><br />
export PATH="/opt/anaconda-python/bin:$PATH"<br />
</pre><br />
|}<br />
<br />
==Quick Tutorial==<br />
The batch system treats each core of a node as a "virtual processor." That means the nodes keyword in batch scripts refers to the number of cores that are scheduled.<br />
<br />
===Select your default MPI===<br />
There are several versions of MPI on the Marvin cluster. Use the following commands to modify your default mpi.<br />
<br />
:* mpi-selector --query -> shows your default mpi<br />
:* mpi-selector --list -> shows all available mpi installations<br />
:* mpi-selector --set <mpi installation> -> sets your default mpi, note, you will have to exit and log back in for this to take effect.<br />
<br />
===Running an MPI Job on the Whole Cluster===<br />
:*assuming /opt/openmpi/ is the default, the mpiexec options may change depending on your selected MPI.<br />
:*First use showq to see how many cores are available. It may be less than 1152 if a node is down.<br />
<br />
<source lang="bash"><br />
#!/bin/sh<br />
#PBS -l nodes=96:ppn=12 (note, this is PBS -l (small case L))<br />
#PBS -N test<br />
#PBS -j oe<br />
#PBS -S /bin/bash<br />
<br />
set -x<br />
cd "$PBS_O_WORKDIR"<br />
<br />
mpiexec --hostfile $PBS_NODEFILE <executable> (where executable is the program you wish to run)<br />
</source><br />
<br />
===Running an MPI Job using 12 Tasks Per Node===<br />
Because the nodes have 12 physical cores, you may want to limit jobs to 12 tasks per node.<br />
The node file lists each node 1 time, so make a copy with each node listed 12 times, and<br />
hand that version to MPI.<br />
<br />
<source lang="bash"><br />
#!/bin/sh<br />
#PBS -l nodes=48<br />
#PBS -N test<br />
#PBS -j oe<br />
#PBS -S /bin/bash<br />
<br />
set -x<br />
cd "$PBS_O_WORKDIR"<br />
<br />
# Construct a copy of the hostfile with only 12 entries per node.<br />
# MPI can use this to run 12 tasks on each node.<br />
uniq "$PBS_NODEFILE"|awk '{for(i=0;i<12;i+=1) print}'>nodefile.12way<br />
<br />
# to Run 12-way on 4 nodes, we request 48 core to obtain 4 nodes<br />
mpiexec --hostfile nodefile.12way ring -v<br />
</source><br />
<br />
<br />
===Running Many Copies of a Serial Job===<br />
In order to run 30 separate instances of the same program, use the scheduler's task array feature, through the "-t" option. The "nodes" parameter here refers to a core.<br />
<br />
<source lang="bash"><br />
#!/bin/sh<br />
#PBS -l nodes=1 (note, this is PBS -l (small case L))<br />
#PBS -t 30<br />
#PBS -N test<br />
#PBS -j oe<br />
#PBS -S /bin/bash<br />
<br />
set -x<br />
cd "$PBS_O_WORKDIR"<br />
echo Run my job.<br />
</source><br />
<br />
When you start jobs this way, separate jobs will pile one-per-core onto nodes like a box of hamsters.<br />
<br />
===Running on a specific node===<br />
To run on a specific node use the host= option <br />
<br />
<source lang="bash"><br />
#!/bin/sh<br />
#PBS -l host=compute-3-16 (note, this is PBS -l (small case L))<br />
#PBS -N test<br />
#PBS -j oe<br />
#PBS -S /bin/bash<br />
<br />
<br />
set -x<br />
cd "$PBS_O_WORKDIR"<br />
echo Run my job.<br />
</source><br />
===Running in the viz queue ===<br />
To run in the viz queue use the -q option<br />
<br />
<source lang="bash"><br />
#!/bin/sh<br />
#PBS -l nodes=1 (note, this is PBS -l (small case L))<br />
#PBS -N test<br />
#PBS -j oe<br />
#PBS -S /bin/bash<br />
#PBS -q viz<br />
<br />
<br />
set -x<br />
cd "$PBS_O_WORKDIR"<br />
echo Run my job.<br />
</source><br />
===Running an interactive job===<br />
from the command line:<br />
qsub -l nodes=1 -I</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Private_Clusters&diff=858Private Clusters2015-09-30T14:45:36Z<p>Ad876: </p>
<hr />
<div>===Restricted Use - Privately owned computer resources (in alphabetical order)===<br />
<br />
<br />
* [[ASTRA Cluster]]<br />
* [[ATLAS Cluster]]<br />
* [//cac.cornell.edu/~slantz/CATS/tipstricks.html CATS] - Combustion And Turbulence Simulator, running Microsoft Windows Server 2008 HPC Edition<br />
* [[CLAL cluster]] is a restricted-use cluster of Apple Xserve machines running MacOS X 10.5. The P.I. is Professor Lust.<br />
* [[DSS1 Cluster]]<br />
* [[ECCO Cluster]]<br />
* [[HD Human Neuroscience Institute (HD-HNI) Computing]]<br />
* [[KINGLAB Cluster]]<br />
* [[LIPID Cluster]]<br />
* [[MARVIN Cluster]]<br />
* [[OSG Cluster]]<br />
* [[TARDIS Cluster]]<br />
* [[THECUBE Cluster]]<br />
* [[WALLER Cluster]]<br />
<br />
===General Documentation===<br />
*[https://{{SERVERNAME}}/services/hpcsystemslist.aspx List of HPC Systems at CAC] (Partial)<br />
*[[Connect to Linux]]<br />
*[[Rules for Creating Passwords]]<br />
*[[Linux Tips and Tricks]]<br />
*[[FAQ|Troubleshooting]]</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Archival_Storage&diff=848Archival Storage2015-09-30T14:30:42Z<p>Ad876: </p>
<hr />
<div>== What is CAC Archival Storage? ==<br />
:* CAC Archival Storage is a low-cost, high-performance option for storing research data '''available only to users within Cornell University'''. <br />
:* CAC Archival Storage is not mountable by running jobs, instead the user must transfer their data from the CAC Archival Storage to an accessible server using [//globusonline.org/ Globus Online]. <br />
:* Globus Online users have easy access to add, delete, and share their data using any Globus Online endpoints. <br />
:* Some of the Globus Online endpoints available include:<br />
:** storage01.cac.cornell.edu (cac#home) where all CAC user home directories are found<br />
:** XSEDE sites:<br />
:*** Stampede (xsede#stampede) <br />
:*** Lonestar (xsede#lonestar4) <br />
:*** TACC Archival Storage (xsede#ranch).<br />
<br />
== First step - Enable (or create) CAC project for Archival Storage and add users where appropriate ==<br />
:* To use the CAC Archival Storage service, '''you must be a user of a CAC project where Archival Storage is enabled'''.<br />
:* The project PI can add users and verify that Archival Storage is enabled at the [https://{{SERVERNAME}}/Services/Projects/manage.aspx Manage CAC project page].<br />
:* Don't have a project? [https://{{SERVERNAME}}/Services/projects.aspx How to start a CAC project?].<br />
<br />
== Second step - create your Globus Online account ==<br />
<br />
[//globus.org/SignUp Sign up] for a Globus account. CAC's Archival system is '''only accessible''' through '''[//globusonline.org/ Globus Online]'''.<br />
<br />
== CAC specifics ==<br />
=== Technical Information ===<br />
CAC's EndPoint is <b>cac#archive01</b>.<br />
<br />
:*When activating cac#archive01 endpoint in Globus Online web GUI, you will be prompted by a dialog box saying: <br />
<br />
<blockquote>The administrator of this endpoint, cac#archive01, requires that you authenticate using their MyProxy OAuth server to activate the endpoint. When you click 'Continue' you will be redirected to their website.</blockquote><br />
<br />
:*You will be redirected to the <nowiki>https://archive01.cac.cornell.edu/oath/authorize...</nowiki> page. <br />
:*Enter your CAC credentials.<br />
:*When login is successful, you will be redirected back to Globus Online web GUI with the endpoint activated.<br />
<br />
=== Administrative Information ===<br />
:* cac#archive01's default path is '''/export'''.<br />
:* Each project with access to CAC Archival Storage has a shared directory (named the project) in which '''all project members have full read/write access'''.<br />
:* Users can rename and move files and directories within their project directory on the endpoint. Globus Online added this feature recently.<br />
<br />
==Automated Archival==<br />
:* Install Globus Connect Personal on the Linux/MacOS/Windows host you wish to archive by clicking on the "Get Globus Connect Personal" link on the Transfer Files screen on Globus. <br />
::[[File:Install_Globus_Connect_Personal.jpg]]<br />
:* On the host you wish to archive, download and untar [[Media:archive_scripts.tar.gz]].<br />
:* To enable running Globus Connect Personal as root, add <br />
<br />
"-allow-root",<br />
<br />
::to globusconnectpersonal-2.0.3/gc.py (on line ~ 360):<br />
<source lang="c"><br />
args = [os.path.basename(PDEATH_LAUNCH),<br />
GRIDFTP_SERVER,<br />
"-allow-root",<br />
"-i", "-always-send-markers",<br />
"-hostname", "127.0.0.1",<br />
</source><br />
:* Copy root-bin directory from the archive_scripts.tar.gz to /root/bin. If you are archiving directories outside /home, modify the -restrict-path argument in /root/bin/gc_start.sh.<br />
:* Generate a ssh key pair using the "ssh-keygen" command, leave private key in ~/.ssh, and upload the private key to Globus<br />
::[[File:Upload_ssh_private_key.jpg]]<br />
:* Make sure you can access Globus CLI like this:<br />
ssh -i .ssh/<private key> <globus user name>@cli.globusonline.org<br />
:* Modify archive.sh to match your Globus user name, private key file name, CAC project and archive directory.<br />
:* On Globus, make sure your connection to cac#archive01 endpoint is activated. <br />
:* You should now be able to run archive.sh to upload your archive directory to CAC archive. You can automate this script using cron.<br />
<br />
==Globus Online links ==<br />
:*[//globus.org/how-it-works How Globus Online works?]<br />
:*[//globusonline.org/quickstart/ Globus Online Quickstart] guide for setting up an account.<br />
:*[//support.globus.org/entries/23583857-Sign-Up-and-Transfer-Files-with-Globus-Online Transfer Data using Globus Online]<br />
:*[//support.globus.org/entries/23602336-Sharing-Data-using-Globus Sharing Data using Globus Online]<br />
:*[//globus.org/group-management Group Management] NOTE: A Globus Plus or a Globus Provider plan are required; CAC's Globus endpoint '''does not''' '''support''' Group Management.</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Template:ContactCAC&diff=846Template:ContactCAC2015-09-30T14:13:58Z<p>Ad876: </p>
<hr />
<div>[https://{{SERVERNAME}}/contact/default.aspx Contact Support]</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Getting_Started&diff=843Getting Started2015-09-30T14:05:00Z<p>Ad876: </p>
<hr />
<div>__TOC__<br />
==Managing your password==<br />
CAC has a [[Getting_started#Rules_for_Creating_Passwords|Password Policy]] in effect. The first time that you login to the <tt>cac.cornell.edu</tt> domain, you will be required to change your password. Each password must have at least eight characters and must contain at least three of the following four elements: (1) uppercase letters (2) lowercase letters (3) special characters (4) digits. Your password can be set or changed on any of the CAC login nodes, and the password will be updated on all CAC resources. Passwords expire every six months. Do not share your password. There are more detailed instructions below.<br />
<br />
===Rules for Creating passwords===<br />
{{:Rules for Creating Passwords}}<br />
<br />
===Change a password at first login===<br />
{{:Changing password at first login}}<br />
<br />
=== Change password at any time ===<br />
{{:Changing password any time}}<br />
<br />
=== Locked Accounts===<br />
<br />
There have been instances in which user accounts have been locked. Some common causes of locked accounts and the solutions are:<br />
<br />
:*Mistyping your password several times in a row. <br />
:::<tt>Solution</tt>: Wait about a 1/2 hour and then try again. Be sure that your caps lock key is not on!<br />
:*Trying to login to a Windows login node by using SSH when you have a new or expired password. <br />
:::<tt>Solution</tt>: Login to a Windows login node using Remote Desktop Connection or SSH to a linux login node.<br />
:*Failing to log off all other sessions connected to login nodes. <br />
:::<tt>Solution</tt>: Log off all remote connections. Disconnecting the sessions is not enough.<br />
:*Failing to disconnect locally mapped drives to the CAC file server before changing your password. <br />
:::<tt>Solution</tt>: Disconnect all locally mapped drives, wait a 1/2 hour until account is unlocked, and then re-map the drive with the new password.<br />
<br />
If you can't log on or can't wait you can submit a Password Reset ticket on our [https://rt.cac.cornell.edu/index.html issue tracking system]<br />
<br />
==Checking your CAC project ==<br />
Cornell University users can view their account limits at [https://{{SERVERNAME}}/services/cu/memberlimits.aspx CAC Account Limits].<br /><br />
Partner Program members should contact Paul Redfern at [mailto:red@cac.cornell.edu red@cac.cornell.edu] if they need information on their membership limits.<br />
<br />
==Using CAC resources==<br />
===Connecting to CAC===<br />
<br />
There are two types of login nodes:<br />
:* Linux login nodes: <tt>linuxlogin.cac.cornell.edu</tt> as well as the head nodes for the various Linux-based private clusters.<br />
:* Windows login node: <tt>winlogin.cac.cornell.edu</tt><br />
<br />
====Connect to Linux====<br />
<br />
{{:Connect to Linux}}<br />
<br />
====Connect to Windows====<br />
<br />
{{:Connect to Windows}}<br />
<br />
===Home Directory Access===<br />
{{:Home Directory Access}}<br />
<br />
===File transfer===<br />
{{:File transfer}}<br />
<br />
<br />
===Linux Usage Tips===<br />
{{:Linux Usage Tips}}<br />
==== More information on Linux nodes at CAC====<br />
For more detailed instructions on how to use the Linux node, see [[Tutorial for the Linux nodes at CAC| here]]<br />
<br />
===Windows Usage Tips===<br />
{{:Windows Usage Tips}}<br />
<br />
===More information===<br />
The CAC Web site is [https://{{SERVERNAME}}/ here] . There are many useful documents on the Support page at [[ Main Page| CAC documentation]].<br />
<br />
===Acknowledging CAC===<br />
{{:Acknowledging CAC}}<br />
<br />
===FAQ/Troubleshooting===<br />
#[[FAQ#Account| Account FAQ]]<br />
#[[FAQ#Login| Login FAQ]]<br />
# If you have more questions, see [[FAQ| here]]</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=File_transfer&diff=842File transfer2015-09-30T13:26:56Z<p>Ad876: /* Secure Copy */</p>
<hr />
<div>A single, central file server, storage01.cac.cornell.edu, serves all CAC user home directories. You can connect to this server in a variety of ways from any operating system to access your files. <br />
<br />
Note: by default, your home directory and its contents will be readable and executable by all other users of CAC systems. If this is not what you want, you can change the permissions of the home directory and its files and subdirectories via the standard Linux or Windows mechanisms. However, be aware that this may lead to conflicts for cross-platform applications, as Windows and Linux permissions are not 100% compatible.<br />
<br />
====Linux users====<br />
=====Secure Copy=====<br />
Secure copy is a standard tool to copy files to and from remote hosts.<br />
localhost$ scp localfile.dat username@linuxlogin.cac.cornell.edu:remoteinput.dat<br />
localhost$ scp username@linuxlogin.cac.cornell.edu:results.dat localresults.dat<br />
<br />
=====Secure FTP=====<br />
FTP is disabled for security reasons, but sftp's interface is nearly identical.<br />
=====Samba Client=====<br />
Type<br />
smbclient //storage01.cac.cornell.edu/<user name> -U <user name><br />
Enter the password for your CAC account when prompted<br />
You will see the '''''smb:\>''''' prompt. You can now start transferring files between your local machine and CAC home directory similar to ftp client. Type '''''help''''' for more instructions.<br />
-sh-3.2$ smbclient //storage01.cac.cornell.edu/<user name> -U <user name><br />
Password: <br />
Domain=[CTC_ITH] OS=[Unix] Server=[Samba 3.0.28-1.el5_2.1]<br />
smb: \> help<br />
<br />
====MacOS X users====<br />
<br />
====Windows users====<br />
=====Secure Copy=====<br />
The people who make Putty provide a secure copy client called [//www.chiark.greenend.org.uk/~sgtatham/putty/download.html pscp]. From the command prompt, type:<br />
cmd> pscp localfile.dat username@linuxlogin.cac.cornell.edu:remoteinput.dat<br />
<enter your username's password when prompted><br />
cmd> pscp username@linuxlogin.cac.cornell.edu:results.dat localresults.dat<br />
<br />
=====Secure FTP=====<br />
FTP is disabled for security reasons, but [//www.chiark.greenend.org.uk/~sgtatham/putty/download.html psftp's] interface is nearly identical. From the command prompt, type:<br />
cmd> psftp username@linuxlogin.cac.cornell.edu<br />
<enter your username's password when prompted><br />
psftp> put localresults.dat results.dat<br />
psftp> quit</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=File_transfer&diff=841File transfer2015-09-30T13:26:41Z<p>Ad876: /* Secure Copy */</p>
<hr />
<div>A single, central file server, storage01.cac.cornell.edu, serves all CAC user home directories. You can connect to this server in a variety of ways from any operating system to access your files. <br />
<br />
Note: by default, your home directory and its contents will be readable and executable by all other users of CAC systems. If this is not what you want, you can change the permissions of the home directory and its files and subdirectories via the standard Linux or Windows mechanisms. However, be aware that this may lead to conflicts for cross-platform applications, as Windows and Linux permissions are not 100% compatible.<br />
<br />
====Linux users====<br />
=====Secure Copy=====<br />
Secure copy is a standard tool to copy files to and from remote hosts.<br />
localhost$ scp localfile.dat username@linuxlogin3.cac.cornell.edu:remoteinput.dat<br />
localhost$ scp username@linuxlogin3.cac.cornell.edu:results.dat localresults.dat <br />
=====Secure FTP=====<br />
FTP is disabled for security reasons, but sftp's interface is nearly identical.<br />
=====Samba Client=====<br />
Type<br />
smbclient //storage01.cac.cornell.edu/<user name> -U <user name><br />
Enter the password for your CAC account when prompted<br />
You will see the '''''smb:\>''''' prompt. You can now start transferring files between your local machine and CAC home directory similar to ftp client. Type '''''help''''' for more instructions.<br />
-sh-3.2$ smbclient //storage01.cac.cornell.edu/<user name> -U <user name><br />
Password: <br />
Domain=[CTC_ITH] OS=[Unix] Server=[Samba 3.0.28-1.el5_2.1]<br />
smb: \> help<br />
<br />
====MacOS X users====<br />
<br />
====Windows users====<br />
=====Secure Copy=====<br />
The people who make Putty provide a secure copy client called [//www.chiark.greenend.org.uk/~sgtatham/putty/download.html pscp]. From the command prompt, type:<br />
cmd> pscp localfile.dat username@linuxlogin.cac.cornell.edu:remoteinput.dat<br />
<enter your username's password when prompted><br />
cmd> pscp username@linuxlogin.cac.cornell.edu:results.dat localresults.dat<br />
<br />
=====Secure FTP=====<br />
FTP is disabled for security reasons, but [//www.chiark.greenend.org.uk/~sgtatham/putty/download.html psftp's] interface is nearly identical. From the command prompt, type:<br />
cmd> psftp username@linuxlogin.cac.cornell.edu<br />
<enter your username's password when prompted><br />
psftp> put localresults.dat results.dat<br />
psftp> quit</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Rules_for_Creating_Passwords&diff=835Rules for Creating Passwords2015-09-30T11:43:52Z<p>Ad876: </p>
<hr />
<div>Do not share your password. Each user should be the only one to know the password for his or her account. Well-chosen passwords are essential to preserve the integrity of the system and individual user accounts. Never leave your password in plain text (unencrypted) in any of your files. Passwords stored in this way are easily stolen.<br />
<br />
When you change your password, the new password must comply with our password complexity policy:<br />
<br />
:*Each password must have at least eight characters.<br />
:*Each password must contain at least three of the following four elements among its first eight characters:<br />
:::- uppercase letters (English, A through Z)<br />
:::- lowercase letters (English, a through z)<br />
:::- special characters (for example, !, $, #, %)<br />
:::- digits (0 through 9)<br />
:*Do not use a space in a password. Though technically allowed, it may be a source of confusion.<br />
:*Do not form a password by appending a digit to a word--this type of password is easily guessed.<br />
:*Each password must differ from the user's login name and any permutation of that login name. For comparison purposes, an upper case letter and its corresponding lower case letter are equivalent.<br />
:*New passwords should differ from the old by at least three characters.<br />
<br />
If you need additional ideas for creating a new password, [http://windows.microsoft.com/en-us/windows-vista/tips-for-creating-a-strong-password Microsoft has a few tips.]</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Rules_for_Creating_Passwords&diff=834Rules for Creating Passwords2015-09-30T11:43:31Z<p>Ad876: </p>
<hr />
<div>Do not share your password. Each user should be the only one to know the password for his or her account. Well-chosen passwords are essential to preserve the integrity of the system and individual user accounts. Never leave your password in plain text (unencrypted) in any of your files. Passwords stored in this way are easily stolen.<br />
<br />
When you change your password, the new password must comply with our password complexity policy:<br />
<br />
:*Each password must have at least eight characters.<br />
:*Each password must contain at least three of the following four elements among its first eight characters:<br />
:::- uppercase letters (English, A through Z)<br />
:::- lowercase letters (English, a through z)<br />
:::- special characters (for example, !, $, #, %)<br />
:::- digits (0 through 9)<br />
:*Do not use a space in a password. Though technically allowed, it may be a source of confusion.<br />
:*Do not form a password by appending a digit to a word--this type of password is easily guessed.<br />
:*Each password must differ from the user's login name and any permutation of that login name. For comparison purposes, an upper case letter and its corresponding lower case letter are equivalent.<br />
:*New passwords should differ from the old by at least three characters.<br />
<br />
If you need additional ideas for creating a new password, [//www.windows.microsoft.com/en-us/windows-vista/tips-for-creating-a-strong-password Microsoft has a few tips.]</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Rules_for_Creating_Passwords&diff=833Rules for Creating Passwords2015-09-30T11:43:05Z<p>Ad876: </p>
<hr />
<div>Do not share your password. Each user should be the only one to know the password for his or her account. Well-chosen passwords are essential to preserve the integrity of the system and individual user accounts. Never leave your password in plain text (unencrypted) in any of your files. Passwords stored in this way are easily stolen.<br />
<br />
When you change your password, the new password must comply with our password complexity policy:<br />
<br />
:*Each password must have at least eight characters.<br />
:*Each password must contain at least three of the following four elements among its first eight characters:<br />
:::- uppercase letters (English, A through Z)<br />
:::- lowercase letters (English, a through z)<br />
:::- special characters (for example, !, $, #, %)<br />
:::- digits (0 through 9)<br />
:*Do not use a space in a password. Though technically allowed, it may be a source of confusion.<br />
:*Do not form a password by appending a digit to a word--this type of password is easily guessed.<br />
:*Each password must differ from the user's login name and any permutation of that login name. For comparison purposes, an upper case letter and its corresponding lower case letter are equivalent.<br />
:*New passwords should differ from the old by at least three characters.<br />
<br />
If you need additional ideas for creating a new password, [//windows.microsoft.com/en-us/windows-vista/tips-for-creating-a-strong-password Microsoft has a few tips.]</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Linux_Usage_Tips&diff=832Linux Usage Tips2015-09-30T11:42:38Z<p>Ad876: </p>
<hr />
<div>==== Linux shells====<br />
:* <tt>/bin/sh</tt> is the default login shell.<br />
:** Edit <tt>$HOME/.profile</tt> to change interactive variables.<br />
:** The <tt>$HOME/.bashrc</tt> file ''will not'' be run for non-interactive shells.<br />
:* <tt>/bin/bash</tt><br />
:** Edit <tt>$HOME/.profile</tt> to change interactive variables.<br />
:** The <tt>$HOME/.bashrc</tt> file ''will'' be run for non-interactive shells.<br />
:* <tt>/bin/csh</tt> and <tt>/bin/tcsh</tt><br />
:** Edit <tt>$HOME/.login</tt> to change interactive variables.<br />
:** The <tt>$HOME/.cshrc</tt> file ''will'' be run for non-interactive shells.<br />
<br />
The change shell command, <tt>chsh</tt>, will not permanently change your shell. You must send a request instead. {{ContactCAC}}<br />
<br />
The default login shell on v4 interactive and batch nodes is ''sh''. Be aware that in Red Hat Enterprise Linux, /bin/sh is a soft-link to /bin/bash, so you are really using a variant of ''bash''. Accordingly, you will find that "man sh" brings up the man page (the help document) for ''bash''. In a way, then, you can think of your login shell as being ''bash'', too.<br />
<br />
There are slight differences between ''sh'' and ''bash'', however. The "Invocation" section of the man page states: "If bash is invoked with the name sh, it tries to mimic the startup behavior of historical versions of ''sh'' as closely as possible." Therefore, you will find that ~/.profile is run at login, because this behavior is common to both ''sh'' and ''bash''; but any interactive ''sh'' shells you start thereafter will not run ~/.bashrc as you might expect from ''bash''. The way to get ''sh'' to do this is to "export ENV=~/.bashrc" beforehand (perhaps as part of your .profile).<br />
<br />
Let's say you simply prefer to have ''bash'' as your default shell and be done with it. There are two ways to accomplish this. First, you can "export SHELL=/bin/bash" in your .profile; then all subsequent interactive shells will truly be ''bash''. Second, you can enter "chsh -s /bin/bash", which forces all login and interactive shells to be ''bash'' (because you have changed your default shell). The problem with the second method is it may well wreck your batch environment, too, because the scheduler sets it up under the assumption that the login shell is ''sh''.<br />
<br />
The relationship between the ''csh'' and ''tcsh'' shells is similar to the one between ''sh'' and ''bash''. For instance, your ''csh'' shells are automatically endowed with the ''tcsh''-style ability to retrieve history through the up- and down-arrow keys. The best way to make ''tcsh'' into your everyday working shell is to run it on top of ''sh'' after you log in (again, you can do this as part of your .profile).<br />
<br />
References<br />
:* "man bash" from the command line.<br />
:* [//tldp.org/LDP/abs/html/ Advanced Bash Scripting Guide], one of the Linux Documentation Project [//tldp.org/guides.html guides]<br />
:* [//mywiki.wooledge.org/BashFAQ Bash FAQ]<br />
:*[//mywiki.wooledge.org/BashPitfalls Bash Pitfalls]<br />
<br />
====Compiling and linking code on Linux====<br />
{{:Compiling Code Linux}}<br />
<br />
====FAQ====<br />
=====How do I determine my program's dependencies on shared library (.so) files?=====<br />
:*ldd - see the man page.<br />
If your program cannot find all the .so files it needs, you may need to add paths to the LD_LIBRARY_PATH shell variable.<br />
<br />
=====How do I display an image file (such as jpeg or gif)?=====<br />
:*display mypic.jpg - uses one of the many ImageMagick tools - see "man ImageMagick" for help on this and various file format converters.<br />
:*firefox mypic.jpg - any decent Web browser can handle it.<br />
Note, the image will show up only if you have [[Getting_started#Connect_to_Linux | X11 forwarding]] enabled.</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Linux_Usage_Tips&diff=831Linux Usage Tips2015-09-30T11:41:59Z<p>Ad876: </p>
<hr />
<div>==== Linux shells====<br />
:* <tt>/bin/sh</tt> is the default login shell.<br />
:** Edit <tt>$HOME/.profile</tt> to change interactive variables.<br />
:** The <tt>$HOME/.bashrc</tt> file ''will not'' be run for non-interactive shells.<br />
:* <tt>/bin/bash</tt><br />
:** Edit <tt>$HOME/.profile</tt> to change interactive variables.<br />
:** The <tt>$HOME/.bashrc</tt> file ''will'' be run for non-interactive shells.<br />
:* <tt>/bin/csh</tt> and <tt>/bin/tcsh</tt><br />
:** Edit <tt>$HOME/.login</tt> to change interactive variables.<br />
:** The <tt>$HOME/.cshrc</tt> file ''will'' be run for non-interactive shells.<br />
<br />
The change shell command, <tt>chsh</tt>, will not permanently change your shell. You must send a request instead. {{ContactCAC}}<br />
<br />
The default login shell on v4 interactive and batch nodes is ''sh''. Be aware that in Red Hat Enterprise Linux, /bin/sh is a soft-link to /bin/bash, so you are really using a variant of ''bash''. Accordingly, you will find that "man sh" brings up the man page (the help document) for ''bash''. In a way, then, you can think of your login shell as being ''bash'', too.<br />
<br />
There are slight differences between ''sh'' and ''bash'', however. The "Invocation" section of the man page states: "If bash is invoked with the name sh, it tries to mimic the startup behavior of historical versions of ''sh'' as closely as possible." Therefore, you will find that ~/.profile is run at login, because this behavior is common to both ''sh'' and ''bash''; but any interactive ''sh'' shells you start thereafter will not run ~/.bashrc as you might expect from ''bash''. The way to get ''sh'' to do this is to "export ENV=~/.bashrc" beforehand (perhaps as part of your .profile).<br />
<br />
Let's say you simply prefer to have ''bash'' as your default shell and be done with it. There are two ways to accomplish this. First, you can "export SHELL=/bin/bash" in your .profile; then all subsequent interactive shells will truly be ''bash''. Second, you can enter "chsh -s /bin/bash", which forces all login and interactive shells to be ''bash'' (because you have changed your default shell). The problem with the second method is it may well wreck your batch environment, too, because the scheduler sets it up under the assumption that the login shell is ''sh''.<br />
<br />
The relationship between the ''csh'' and ''tcsh'' shells is similar to the one between ''sh'' and ''bash''. For instance, your ''csh'' shells are automatically endowed with the ''tcsh''-style ability to retrieve history through the up- and down-arrow keys. The best way to make ''tcsh'' into your everyday working shell is to run it on top of ''sh'' after you log in (again, you can do this as part of your .profile).<br />
<br />
References<br />
:* "man bash" from the command line.<br />
:* [//www.tldp.org/LDP/abs/html/ Advanced Bash Scripting Guide], one of the Linux Documentation Project [//www.tldp.org/guides.html guides]<br />
:* [//www.mywiki.wooledge.org/BashFAQ Bash FAQ]<br />
:*[//www.mywiki.wooledge.org/BashPitfalls Bash Pitfalls]<br />
<br />
====Compiling and linking code on Linux====<br />
{{:Compiling Code Linux}}<br />
<br />
====FAQ====<br />
=====How do I determine my program's dependencies on shared library (.so) files?=====<br />
:*ldd - see the man page.<br />
If your program cannot find all the .so files it needs, you may need to add paths to the LD_LIBRARY_PATH shell variable.<br />
<br />
=====How do I display an image file (such as jpeg or gif)?=====<br />
:*display mypic.jpg - uses one of the many ImageMagick tools - see "man ImageMagick" for help on this and various file format converters.<br />
:*firefox mypic.jpg - any decent Web browser can handle it.<br />
Note, the image will show up only if you have [[Getting_started#Connect_to_Linux | X11 forwarding]] enabled.</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Linux_Usage_Tips&diff=830Linux Usage Tips2015-09-30T11:40:46Z<p>Ad876: </p>
<hr />
<div>==== Linux shells====<br />
:* <tt>/bin/sh</tt> is the default login shell.<br />
:** Edit <tt>$HOME/.profile</tt> to change interactive variables.<br />
:** The <tt>$HOME/.bashrc</tt> file ''will not'' be run for non-interactive shells.<br />
:* <tt>/bin/bash</tt><br />
:** Edit <tt>$HOME/.profile</tt> to change interactive variables.<br />
:** The <tt>$HOME/.bashrc</tt> file ''will'' be run for non-interactive shells.<br />
:* <tt>/bin/csh</tt> and <tt>/bin/tcsh</tt><br />
:** Edit <tt>$HOME/.login</tt> to change interactive variables.<br />
:** The <tt>$HOME/.cshrc</tt> file ''will'' be run for non-interactive shells.<br />
<br />
The change shell command, <tt>chsh</tt>, will not permanently change your shell. You must send a request instead. {{ContactCAC}}<br />
<br />
The default login shell on v4 interactive and batch nodes is ''sh''. Be aware that in Red Hat Enterprise Linux, /bin/sh is a soft-link to /bin/bash, so you are really using a variant of ''bash''. Accordingly, you will find that "man sh" brings up the man page (the help document) for ''bash''. In a way, then, you can think of your login shell as being ''bash'', too.<br />
<br />
There are slight differences between ''sh'' and ''bash'', however. The "Invocation" section of the man page states: "If bash is invoked with the name sh, it tries to mimic the startup behavior of historical versions of ''sh'' as closely as possible." Therefore, you will find that ~/.profile is run at login, because this behavior is common to both ''sh'' and ''bash''; but any interactive ''sh'' shells you start thereafter will not run ~/.bashrc as you might expect from ''bash''. The way to get ''sh'' to do this is to "export ENV=~/.bashrc" beforehand (perhaps as part of your .profile).<br />
<br />
Let's say you simply prefer to have ''bash'' as your default shell and be done with it. There are two ways to accomplish this. First, you can "export SHELL=/bin/bash" in your .profile; then all subsequent interactive shells will truly be ''bash''. Second, you can enter "chsh -s /bin/bash", which forces all login and interactive shells to be ''bash'' (because you have changed your default shell). The problem with the second method is it may well wreck your batch environment, too, because the scheduler sets it up under the assumption that the login shell is ''sh''.<br />
<br />
The relationship between the ''csh'' and ''tcsh'' shells is similar to the one between ''sh'' and ''bash''. For instance, your ''csh'' shells are automatically endowed with the ''tcsh''-style ability to retrieve history through the up- and down-arrow keys. The best way to make ''tcsh'' into your everyday working shell is to run it on top of ''sh'' after you log in (again, you can do this as part of your .profile).<br />
<br />
References<br />
:* "man bash" from the command line.<br />
:* [//tldp.org/LDP/abs/html/ Advanced Bash Scripting Guide], one of the Linux Documentation Project [//tldp.org/guides.html guides]<br />
:* [//mywiki.wooledge.org/BashFAQ Bash FAQ]<br />
:*[//mywiki.wooledge.org/BashPitfalls Bash Pitfalls]<br />
<br />
====Compiling and linking code on Linux====<br />
{{:Compiling Code Linux}}<br />
<br />
====FAQ====<br />
=====How do I determine my program's dependencies on shared library (.so) files?=====<br />
:*ldd - see the man page.<br />
If your program cannot find all the .so files it needs, you may need to add paths to the LD_LIBRARY_PATH shell variable.<br />
<br />
=====How do I display an image file (such as jpeg or gif)?=====<br />
:*display mypic.jpg - uses one of the many ImageMagick tools - see "man ImageMagick" for help on this and various file format converters.<br />
:*firefox mypic.jpg - any decent Web browser can handle it.<br />
Note, the image will show up only if you have [[Getting_started#Connect_to_Linux | X11 forwarding]] enabled.</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=File_transfer&diff=829File transfer2015-09-30T11:39:02Z<p>Ad876: </p>
<hr />
<div>A single, central file server, storage01.cac.cornell.edu, serves all CAC user home directories. You can connect to this server in a variety of ways from any operating system to access your files. <br />
<br />
Note: by default, your home directory and its contents will be readable and executable by all other users of CAC systems. If this is not what you want, you can change the permissions of the home directory and its files and subdirectories via the standard Linux or Windows mechanisms. However, be aware that this may lead to conflicts for cross-platform applications, as Windows and Linux permissions are not 100% compatible.<br />
<br />
====Linux users====<br />
=====Secure Copy=====<br />
Secure copy is a standard tool to copy files to and from remote hosts.<br />
localhost$ scp localfile.dat username@linuxlogin3.cac.cornell.edu:remoteinput.dat<br />
localhost$ scp username@linuxlogin3.cac.cornell.edu:results.dat localresults.dat <br />
=====Secure FTP=====<br />
FTP is disabled for security reasons, but sftp's interface is nearly identical.<br />
=====Samba Client=====<br />
Type<br />
smbclient //storage01.cac.cornell.edu/<user name> -U <user name><br />
Enter the password for your CAC account when prompted<br />
You will see the '''''smb:\>''''' prompt. You can now start transferring files between your local machine and CAC home directory similar to ftp client. Type '''''help''''' for more instructions.<br />
-sh-3.2$ smbclient //storage01.cac.cornell.edu/<user name> -U <user name><br />
Password: <br />
Domain=[CTC_ITH] OS=[Unix] Server=[Samba 3.0.28-1.el5_2.1]<br />
smb: \> help<br />
<br />
====MacOS X users====<br />
<br />
====Windows users====<br />
=====Secure Copy=====<br />
The people who make Putty provide a secure copy client called [//www.chiark.greenend.org.uk/~sgtatham/putty/download.html pscp]. From the command prompt, type:<br />
cmd> pscp localfile.dat username@linuxlogin3.cac.cornell.edu:remoteinput.dat<br />
<enter your username's password when prompted><br />
cmd> pscp username@linuxlogin.cac.cornell.edu:results.dat localresults.dat<br />
<br />
=====Secure FTP=====<br />
FTP is disabled for security reasons, but [//www.chiark.greenend.org.uk/~sgtatham/putty/download.html psftp's] interface is nearly identical. From the command prompt, type:<br />
cmd> psftp username@linuxlogin.cac.cornell.edu<br />
<enter your username's password when prompted><br />
psftp> put localresults.dat results.dat<br />
psftp> quit</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Connect_to_Windows&diff=828Connect to Windows2015-09-30T11:38:03Z<p>Ad876: </p>
<hr />
<div>'''''Using Remote Desktop Connection to connect to winlogin'''''<br />
<br />
This method of connecting to winlogin is preferred because it provides you with a fully functional Windows desktop. At the login screen, if the domain is specified, it should be set to '''''CTC_ITH''''', not the local name of the machine to which you are connecting.<br />
<br />
:'''Remote Desktop Connect Details''':<br />
<br />
Remote Desktop sessions do not expire, but they will end when machines are rebooted during down times.<br />
<br />
:* '''If you use a Windows machine''':<br />Use the Remote Desktop Connection (older name Terminal Services Client) to connect to a login machine. This software is pre-installed with Windows 7 and later. To run it, click <tt>Start</tt>, then <tt>All Programs > Accessories > Communications > Remote Desktop Connection</tt>. Otherwise you need to [//microsoft.com/windowsxp/downloads/tools/rdclientdl.mspx download the client] before you can use it.<br />
:*''' If you use Mac OS X 10.7 or later''':<br />Use the free download from [//itunes.apple.com/us/app/microsoft-remote-desktop/id715768417?mt=12 the Mac App Store]. Works just like the Remote Desktop Connection in Windows 7. You can also use rdesktop (see below). Tip: if authentication fails, make sure your software updates are current.<br />
:*''' If you use Unix or Linux or Mac''':<br />You can access the login machines by using the cross-platform rdesktop client. If you are running Linux, typically it is part of the distribution. If you prefer to build it yourself, it is available for download from [//rdesktop.org/ rdesktop]. Executables for old versions are available from [//jacco2.dds.nl/rdesktop/index.html. here]</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Connect_to_Linux&diff=827Connect to Linux2015-09-30T11:36:17Z<p>Ad876: </p>
<hr />
<div>__TOC__<br />
There are three distinct ways to connect to a login node:<br />
# [[Getting_started#Using_Secure_Shell | Use SSH]] to open a Linux shell on a login node, which provides a text-only interface.<br />
# [[Getting_started#Using_Secure_Shell | Use SSH]] together with [[Getting_started#Using_X-Windows | X-Windows]], which sends any interactive graphics back to your machine window-by-window through an SSH tunnel.<br />
# [[Getting_started#Using_VNC | Use VNC]] to get a remote desktop with multiple text and graphics windows. This is not as straightforward as it sounds, due to the need to set up a secure tunnel for the remote desktop first.<br />
<br />
These instructions are intended mainly for users of personal computers and workstations. However, much of the material carries over to mobile computing platforms such as tablets and smartphones. You will have to locate and download an app to enable SSH or VNC connectivity; even a browser plug-in may suffice.<br />
<br />
Whichever method you choose, at your first login, you will be challenged for a new password. Find help at [[Getting_started#Change_a_password_at_first_login | Changing a Password at First Login]]. You will also be asked for an ssh passphrase. You can just leave this blank; hit the <tt>Enter</tt> key in response.<br />
<br />
=====Using Secure Shell=====<br />
For basic command-line access, a Secure Shell (SSH) client will give you a remote command shell on one of the login nodes.<br />
:* Nearly all Unix/Linux varieties (including Mac) already have a built-in SSH2 implementation, required by our clusters.<br />
:* If you are coming from a Microsoft Windows machine, an SSH2 client must first be installed, as described below.<br />
:* The non-secure predecessor of SSH, telnet, is disabled for security reasons.<br />
<br />
'''''Linux users:'''''<br />
<br />
To connect to the second login node with ssh, you simply open a terminal window and type<br />
localhost$ ssh username@linuxlogin.cac.cornell.edu<br />
<br />
'''''Mac OS X users:'''''<br />
<br />
OS X on the Mac is built on a version of Unix, so ssh is available directly from the Terminal application.<br />
:*One option is to use the shortcut <tt>'''cmd-space'''</tt> to open Spotlight and then type <tt>"Terminal"</tt> to open a Terminal window.<br />
Otherwise:<br />
:*Navigate in the Finder to the Applications folder and Utilities sub-folder.<br />
:*:[[Image:MacApplicationsFolder.png|500px]]<br />
:*Then double-click on the Terminal application to see a Bash command-line.<br />
:*:[[Image:MacTerminalWindow.png]]<br />
:*As in Linux, simply type "ssh username@linuxlogin.cac.cornell.edu" into this window.<br />
<br />
'''''Windows users:'''''<br />
<br />
Secure Shell (ssh) clients work nicely as long as they support the SSH2 protocol. As mentioned, telnet is disabled for security reasons. A popular client for Windows is the free [//www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY client].<br />
:* The simplest installation is to download the [//chiark.greenend.org.uk/~sgtatham/putty/download.html Windows installer], called '''putty-0.65-installer.exe''', and run it. This installs PuTTY into your Start menu.<br />
:* To connect, start PuTTY, then type in a host name such as linuxlogin.cac.cornell.edu, and click "Open".<br />
:*:[[Image:Putty_address.jpg|Setting the host name in PuTTY]]<br />
<br />
=====Using X-Windows=====<br />
<br />
X-Windows or X11 is the longstanding Unix mechanism for displaying interactive graphics in a window. Your "X server" software runs locally, but it is capable of displaying windows that have been generated either locally or remotely. An "X client" on a remote machine can create X-Windows for local display, but it is necessary first to establish a shell on that machine using SSH.<br />
<br />
'''''Appropriate use'''''<br />
<br />
Among other things, X-Windows gives you the ability to display a GUI that originates on a login node. However, this ability does '''''NOT''''' imply that you are permitted to run compute-intensive, GUI-driven applications on these machines. Such usage is not only contrary to CAC policy, it is disrespectful toward other users, because the login node may become unresponsive through your actions.<br />
<br />
'''''Linux users:'''''<br />
<br />
The standard way to use X-Windows is to tunnel the X-Windows protocol through an ssh connection. If you open your ssh session with the '''-X''' option, it will automatically set up the necessary tunnel and environment variables.<br />
localhost$ ssh -X username@linuxlogin.cac.cornell.edu<br />
linuxlogin$ echo $DISPLAY<br />
localhost:11.0<br />
linuxlogin$ xclock&<br />
You can see that your DISPLAY environment variable is set and test it with xclock. There is another option to use a trusted version of X-windows forwarding<br />
linuxlogin$ ssh -Y compute-3-48.v4linux<br />
Th trusted version is necessary for forwarding X11 connections from a compute node to the login node, then back to your client machine.<br />
<br />
'''''Mac OS X users:'''''<br />
<br />
If you start ssh with the '''-X''' or '''-Y''' option, X-Windows should start up automatically. You can then try the "xclock" test, as described above for Linux. <br />
<br />
X11 is preinstalled on Macs starting with OS X 10.6 (Snow Leopard). For Mac OS X 10.5 (Leopard), you may need to install X11 in order for X-Windows applications to launch. If there is no X11 application in the Applications->Utilities folder, you'll have to find your OS X install disk. From the ''Mac OS X Server Introduction to Command-Line Administration,'' "The X11 server and an application to access X windows from the Finder are available as an optional installation in the Optional Installs folder of your installation disc (X11 is in the Applications package)."<br />
<br />
'''''Windows users:'''''<br />
<br />
Along with your ssh client (e.g., PuTTY), you will need to install an X-Windows server on your Windows machine.<br />
:* [//straightrunning.com/XmingNotes/ Xming] - Open Source. A shareware contribution will get you a version with improved performance for graphics (GLX). There are two pieces to download<br />
:*:[[Image:Xming-download.jpg]]<br />
:** Xming-mesa (public domain release). There are two links together, one for Xming, one for Xming-mesa. Either will work, but Xming-mesa has some newer features that might come in handy some time.<br />
:** Xming-fonts (public domain release)<br />
If you purchase the website release of Xming, remember to install the Xming-fonts, as well.<br />
:* OpenText's [//cit.cornell.edu/services/software_licensing/available/exceed.cfm Exceed and Exceed 3D] - Cornell no longer has a site license. Installing Exceed 3D will improve performance of graphics applications. Exceed installs several icons under the Start menu. Choose the one that just says "Exceed" because it starts the program in multi-window mode, which is what we want.<br />
<br />
Here is how to start a session using PuTTY and Xming.<br />
<br />
# Start Xming from the Start menu. It will appear briefly and disappear except for an X in the application tray.<br />
# Start PuTTY.<br />
# In the window that appears, type a host name, <tt>linuxlogin.cac.cornell.edu</tt>.<br />
# Use the tree menu on the left to set X11 forwarding. It's in the <tt>Connection > SSH branch</tt>.<br />
#:[[Image:Putty_x11forwarding.jpg|Setting X11 forwarding in PuTTY]]<br />
# For PuTTY 0.61 only - In the "Auth" section of the SSH branch, go to GSSAPI and uncheck <tt>"Attempt GSSAPI authentication"</tt>. This will prevent an annoying <tt>"Access denied"</tt> message from appearing in your terminal window.<br />
# You can return to the Session category and Save this session's configuration for future use. Give it a logical name like linuxlogin.<br />
# Click Open, and it will connect to a login node.<br />
# Test your X-Windows setup by typing<br />
xclock<br />
You should see a clock appear in the corner of your screen. You can stop it by typing <tt>Ctrl-c</tt> in the terminal window.<br />
<br />
=====Using VNC=====<br />
<br />
[http://en.wikipedia.org/wiki/Vnc VNC] lets you see a whole Linux desktop from the login node on your computer.<br />
Using SSH and X-Windows is generally faster, and uses a lot less of the login node's resources,<br />
but VNC can be much faster if you are doing visualization on the login node from off campus.<br />
<br />
For security reasons, we are requiring all VNC connections to be tunneled inside ssh. You will therefore need to be able to connect to the login nodes [[Getting_started#Using_Secure_Shell | using SSH]]. Because the firewall running on linuxlogin blocks all incoming ports except for ssh, VNC connections must be made over a ssh tunnel as described below.<br />
<br />
'''''Appropriate use'''''<br />
<br />
VNC gives you the ability to establish a remote desktop on the login nodes, but this ability does '''''NOT''''' imply that you are permitted to run compute-intensive, GUI-driven applications on these machines. Such usage is not only contrary to CAC policy, it is disrespectful toward other users, because the login node may become unresponsive through your actions.<br />
<br />
Here is a good example of how to use VNC appropriately. By following these steps you can run (say) Abaqus in GUI-driven mode on a compute node that has been allocated to you through an interactive batch job.<br />
# Open a VNC connection to linuxlogin through an ssh tunnel using the instructions below, in order to gain access to a Linux desktop. Make sure two terminal windows are available on this desktop.<br />
# In one of the terminal windows, submit an interactive job to the queue of your choice (add the #PBS -I directive to your job submission script).<br />
# Once the job starts, you will be given a command prompt on your assigned machine. Note the result of "hostname". There is no need to enter further commands at this prompt (except to exit the job).<br />
# Go to the other terminal window and open a second ssh connection to the compute node using "ssh -Y <userid>@<hostname>"<br />
# This new ssh session will tunnel X-Windows from the compute node back to the VNC desktop. Therefore (if Abaqus is on your path), you can now open the Abaqus GUI using "abaqus cae -mesa".<br />
<br />
'''''Initial setup'''''<br />
''(You only need to do this once)''<br />
<br />
:* Install a VNC client if one isn't installed. [http://www.tightvnc.com/ TightVNC] works well, but so do others.<br />
:* Login to linuxlogin, and set the password for your VNC server using the "vncpasswd" command.<br />
<br />
'''''Start your VNC server'''''<br />
<br />
:* '''On linuxlogin''', start the VNC server using the "vncserver" command like this:<br />
vncserver -geometry 1024x768 -localhost<br />
The geometry numbers, 1024x768, specify the size, in pixels, of the desktop.<br />
:* You will need to get the display number from the output of the vncserver command:<br />
<br />
New 'linuxlogin.cac.cornell.edu:1 (shl1)' desktop is linuxlogin.cac.cornell.edu:1<br />
Starting applications specified in /home/gfs01/shl1/.vnc/xstartup<br />
Log file is /home/gfs01/shl1/.vnc/linuxlogin.cac.cornell.edu:1.log<br />
<br />
:* vncserver is running on port 5900 + display number. In the above example, the display number is :1, therefore vncserver is running on port 5901.<br />
<br />
'''''Connect your VNC client'''''<br />
<br />
:* Set up ssh forwarding on your client computer. Let's say the port number on linuxlogin is 5901 (as above), and your CAC userid is uid12. From Linux, type into a terminal:<br />
<br />
ssh -L 10000:localhost:5901 uid12@linuxlogin.cac.cornell.edu<br />
'''From Windows''', ssh clients such as PuTTY can do X11 port forwarding. See [[VNC Tunnel Windows]]. <br />
<br />
'''For MacOS X users''', see [[CAC VPN Server for MacOS Users |here]]<br />
:* Leave this ssh session running on your local client computer. (It can run in the background.)<br />
:* Launch your VNC client program. Connect to localhost:10000. When prompted, type in your VNC server password.<br />
<br />
'''''To disconnect your client'''''<br />
<br />
:* Close the vnc client program.<br />
:* Disconnect the ssh forwarding session (i.e., kill it).<br />
<br />
'''''To reconnect your client'''''<br />
<br />
:* Restart port forwarding with ssh, using the same remote port number as before.<br />
:* Again connect the VNC client to localhost:10000.<br />
<br />
'''''When you are all done'''''<br />
<br />
:* On linuxlogin, type this command to shut down the VNC server<br />
vncserver -kill :<display number><br />
:* If you merely log out from linuxlogin, it will leave the VNC server running. You must shut down the VNC server explicitly when you are finished with it. (Actually this can be a nice feature.)<br />
<br />
=====Passwordless SSH=====<br />
<br />
''''' Create ssh key pair '''''<br />
<br />
Your ssh key pair will only need to be created once. You will not need to repeat this step. You can complete this step from either a Linux or Windows login node. If this is your first login to a CAC login node, it will ask you to [[Getting_started#Change_a_password_at_first_login|change your password]]. This will become your password for connecting to the nodes. <br />
<br />
Create your ssh key pair by logging into the linux login node (linuxlogin.cac.cornell.edu), which will begin the process of creating the keys; you can use the defaults or empty responses for all prompts.<br />
<br />
Alternatively, you can create your ssh key pair on the linux login node by logging into the Windows login node (winx64login.cac.cornell.edu), opening a Command Prompt window, and running <tt>plink.exe</tt> to connect to the linux login node, as shown in this example:<br />
<br />
<pre>>"C:\Programs Files (x86)\Putty\plink.exe" %USERNAME%@linuxlogin.cac.cornell.edu<br />
Password: Enter Your Password<br />
Rocks 5.0 (V)<br />
Profile built 12:54 06-May-2008<br />
<br />
Kickstarted 09:22 06-May-2008<br />
-----------------------------------------------------------<br />
Welcome to the Center for Advanced Computing Cluster!<br />
-----------------------------------------------------------<br />
Please send your questions to help@cac.cornell.edu<br />
-----------------------------------------------------------<br />
<br />
<br />
It doesn't appear that you have set up your ssh key.<br />
This process will make the files:<br />
/home/gfs01/cacshl1/.ssh/id_rsa.pub<br />
/home/gfs01/cacshl1/.ssh/id_rsa<br />
/home/gfs01/cacshl1/.ssh/authorized_keys<br />
<br />
Generating public/private rsa key pair.<br />
Enter file in which to save the key (/home/gfs01/cacshl1/.ssh/id_rsa): Press Enter to accept default<br />
Created directory '/home/gfs01/cacshl1/.ssh'.<br />
Enter passphrase (empty for no passphrase): Press Enter to accept default<br />
Enter same passphrase again: Press Enter to accept default<br />
Your identification has been saved in /home/gfs01/cacshl1/.ssh/id_rsa.<br />
Your public key has been saved in /home/gfs01/cacshl1/.ssh/id_rsa.pub.<br />
</pre><br />
<br />
After this is done, type '''"exit"''' to log out of the linux login node.<br />
<br />
''''' Convert ssh Private Key for Putty / Plink '''''<br />
<br />
Next run PuTTYgen to generate public and private keys to be used with PuTTY and Plink:<br />
<br />
:* Log in to <tt>winx64login.tc.cornell.edu </tt>(if you are not already)<br />
:* Run <tt>C:\Program Files (x86)\Putty\puttygen.exe</tt>.<br />
:* Select <tt>Import Key</tt> from the <tt>Conversions</tt> menu and select <tt>H:\.ssh\id_rsa</tt> in your home directory. And click on the <tt>Open</tt> button.<br />
<center>[[image:LoadPrivateKey.jpg]]</center><br />
:* Click on the <tt>"Save Private Key"</tt> button. <br />
<center>[[image:SavePrivateKey.jpg]]</center><br />
:* Click on "Yes" when asked to save the private key without a passphrase.<br />
:* Save the private key as private.ppk in the .ssh directory inside your home directory.<br />
<center>[[image:SpecifyPrivateKey.jpg]]</center><br />
:* Close (choose File, then Exit)<br />
:* To confirm you have converted the ssh private key successfully, do:<br />
<pre>"C:\Program Files (x86)\Putty\plink.exe" -i %HOMEDRIVE%\.ssh\private.ppk %USERNAME%@linuxlogin.cac.cornell.edu</pre><br />
It may notify you that "The server's host key is not cached in the registry." Type "y" to "store the key in cache."<br />
:* You should now be logged into linuxlogin without being prompted for a password. Stay logged in for the next step.</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Private_Clusters&diff=825Private Clusters2015-09-29T20:28:51Z<p>Ad876: </p>
<hr />
<div>===Restricted Use - Privately owned computer resources (in alphabetical order)===<br />
<br />
<br />
* [[ASTRA Cluster]]<br />
* [[ATLAS Cluster]]<br />
* [//cac.cornell.edu/~slantz/CATS/tipstricks.html CATS] - Combustion And Turbulence Simulator, running Microsoft Windows Server 2008 HPC Edition<br />
* [[CLAL cluster]] is a restricted-use cluster of Apple Xserve machines running MacOS X 10.5. The P.I. is Professor Lust.<br />
* [[DSS1 Cluster]]<br />
* [[ECCO Cluster]]<br />
* [[HD Human Neuroscience Institute (HD-HNI) Computing]]<br />
* [[KINGLAB Cluster]]<br />
* [[LIPID Cluster]]<br />
* [[MARVIN Cluster]]<br />
* [[OSG Cluster]]<br />
* [[TARDIS Cluster]]<br />
* [[THECUBE Cluster]]<br />
* [[WALLER Cluster]]<br />
<br />
===General Documentation===<br />
*[//cac.cornell.edu/services/hpcsystemslist.aspx List of HPC Systems at CAC] (Partial)<br />
*[[Connect to Linux]]<br />
*[[Rules for Creating Passwords]]<br />
*[[Linux Tips and Tricks]]<br />
*[[FAQ|Troubleshooting]]</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=File_Transfer_using_Globus&diff=824File Transfer using Globus2015-09-29T20:28:18Z<p>Ad876: </p>
<hr />
<div>__TOC__<br />
<br />
==Introduction to Globus==<br />
Many CAC resources are on [//globus.org Globus] for high speed file transfers. Globus uses multiple parallel TCP streams to achieve high throughput. Globus also monitors file transfers and automatically retries when errors are encountered. It provides a fast and reliable method for transferring research data to and from CAC.<br />
<br />
==First-Time Use==<br />
# Sign up for an account on [//globus.org Globus].<br />
# If you need to transfer files to and from your local laptop or desktop, download and install [//globus.org/globus-connect-personal Globus Connect Personal] on your host.<br />
# To use [//globus.org Globus web interface], log into your Globus account and [//globus.org/xfer/StartTransfer start transfer files].<br />
# To use Globus command line interface (CLI) or script your file transfers, read the [//dev.globus.org/cli/using-the-cli Using Command Line Interface documentation] for more information.<br />
<br />
==CAC Globus Endpoints==<br />
The following CAC resources are on Globus via these endpoints:<br />
<br />
{| class="wikitable"<br />
! style="text-align:left;" | Endpoint<br />
! style="text-align:left;" | CAC Resource<br />
|-<br />
| '''cac#home''' <br />
| CAC home directories<br />
|-<br />
| '''cac#archive01'''<br />
| CAC [[Archival Storage]]<br />
|-<br />
| '''cac#hd-hni'''<br />
| [[HD Human Neuroscience Institute (HD-HNI) Computing]]<br />
|-<br />
| '''cac#marvin'''<br />
| [[MARVIN Cluster]]<br />
|-<br />
| '''cac#thecube'''<br />
| [[THECUBE Cluster]]<br />
|}</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Syncing_to_Archival_Storage&diff=823Syncing to Archival Storage2015-09-29T20:25:59Z<p>Ad876: </p>
<hr />
<div>==General Information==<br />
===Purpose of this how-to===<br />
This is a user-level guide for syncing a Linux or a Windows machine with Globus, particularly to the CAC Archival Storage.<br />
<br />
===Assumptions and definitions===<br />
<br />
:* You have a Globus account, which we'll indicate by <globusAccountName><br />
:* You have a local endpoint you want to sync, indicated by <localEndpointName><br />
:* There is a destination endpoint <destinationEndpointName>; for syncing to the CAC archive resource it is cac#archive01/export/archive01/<CACProjectName>/<path><br />
:** The CAC archive endpoint is active<br />
:* Any subsidiary paths with be written as <path>.<br />
<br />
===Limitations===<br />
This requires that both endpoints be active. Although it can be run from any machine (not necessarily the one with an endpoint on it) Globus Connect Personal runs under a user identity and dies when that person logs out; this is not an issue on Linux, but limits its usefulness if syncing as a scheduled task on Windows, where it may be possible to run Globus Connect Personal as a service but that has not yet been tested.<br />
<br />
==Linux==<br />
<br />
===Setup===<br />
<br />
To backup a directory from the a linux file server to CAC's archive you must first start GlobusConnect on the file server. Designate or create an account to run the syncing process, which we will call <sync-user>. Create these scripts so that the <sync-user> account can execute them:<br />
<br />
gc_start.sh:<br />
<br />
<pre>#!/bin/bash<br />
sh /opt/globusconnectpersonal-2.0.3/globusconnect -start -restrict-paths rw/<path to back up>&</pre><br />
<br />
gc_status.sh:<br />
<br />
<pre>#!/bin/bash<br />
sh /opt/globusconnectpersonal-2.0.3/globusconnect -status</pre><br />
<br />
gc_stop.sh<br />
<br />
<pre>#!/bin/bash<br />
sh /opt/globusconnectpersonal-2.0.3/globusconnect -stop<br />
</pre><br />
<br />
Execute gc_start.sh:<br />
<br />
<pre><pathToScript>/gc_start.sh</pre><br />
<br />
You also need ssh keys set up with the Globus system. You store your private key locally (typically in the .ssh subdirectory of your home directory and give it a name, which we'll represent as <mykey>) and:<br />
<br />
:* Go to the globus website and click on your account name at top-right, and select "manage identities"<br />
:** Select "Add linked identity" and pick "Add SSH public key"<br />
:** Paste the public key into the box for it and give the key a name<br />
:** Click "Submit"<br />
<br />
=== How to perform a backup ===<br />
<br />
Once GlobusConnect is started you next issue a command to the CLI via ssh. For example:<br />
<pre>ssh -t <globusAccountName>@cli.globusonline.org transfer -s 2 --preserve-mtime --verify-checksum <br />
-- <globusAccountName>#<localEndpointName> cac#archive01/export/archive01/<CACProjectName>/<path> -r</pre><br />
This command will backup the /home/fs01 directory to the CAC archive preserving the last modified timestamp, performing a checksum, and only backing up files with timestamps newer than those existing in the archive or new files. Nothing will be deleted.<br />
<br />
To monitor the status of your backup go to [https://www.globusonline.org/xfer/ViewTransfers the cacsystems GlobusOnline transfer activity page]. If you don't have the password, talk to other CAC staff to obtain it. Once your backup is completed an automated summary will be mailed to cac-systems. Next you need to stop the GlobusConnect client on the file server by running:<br />
<pre><pathToScript>/gc_stop.sh</pre><br />
<br />
You can check the status of the GlobusConnect by running:<br />
<pre><pathToScript>/gc_status.sh</pre><br />
<br />
=== Scheduled Backups ===<br />
You can use cron jobs to perform scheduled backups. You need a user in which context these services will run; we will call this user <sync-user>. Our example services are<br />
<br />
# Daily backup of /home/fs01/ running at 11:00PM<br />
# Weekly backup of /home/shared running on Sunday at 11:30PM<br />
<br />
Say you want to run these daily and weekly sync cronjobs in the context of the CTC_ITH\<sync-user> user. In this user's home folder create a daily-sync.sh and a weekly-sync.sh file. Each file should be scheduled accordingly via crontab.<br />
<br />
<pre>5 0 * * * /home/<sync-user>/daily-sync.sh<br />
5 0 * * 6 /home/<sync-user>/weekly-sync.sh</pre><br />
<br />
Because of limited program control available in the batch files each file does the following at present:<br />
<br />
# attempts to start the GlobusConnect client.<br />
# auto-activates the GlobusConnect endpoint on hd-hni-fs.cac.cornell.edu<br />
# initiates a transfer command<br />
<br />
Example content of weekly-sync.sh<br />
<source lang="bash"><br />
#!/bin/bash<br />
<pathtTo>/gc_start.sh<br />
ssh -i .ssh/<mykey> -t <globusAccount>@cli.globusonline.org endpoint-activate <globusAccount>#<my-endpoint><br />
ssh -i .ssh/<mykey> -t <globusAccount>@cli.globusonline.org transfer -s 2 --verify-checksum -- <globusAccount>#<my-endpoint>/home/shared/ cac#archive01/export/archive01/<CACProjectName>/<path>$<br />
</source><br />
<br />
Ideally each script would terminate the GLobusConnect client when the transfer completed but, this is not yet implemented and may never be depending on the time and effort required to make it work.<br />
<br />
==Windows==<br />
<br />
===Assumption===<br />
<br />
The client endpoint -- the one containing the resources to be transferred to the CAC endpoint -- is active.<br />
<br />
<br />
===Explanation===<br />
<br />
We're going to use the command-line interface (CLI) to Globus, which basically means logging into their dedicated server over SSH and sending commands. The CLI is detailed [//support.globus.org/forums/22861518-Command-Line-Interface here].<br />
<br />
===Setup===<br />
<br />
:* Download the latest version of PuTTY (some older versions won't work), including PuTTYGen and plink (the Windows installer contains them all: http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html )<br />
:* Launch PuTTYGen<br />
:** Make sure the "SSH2-RSA" radio button is selected, and a key length of at least 2048 in the box below that<br />
:** Click the "Generate" button. You'll need to keep moving the cursor over the blank grey area to generate randomness<br />
:** You don't want to use a passphrase!<br />
:** Save the keys: <br />
:*** the private key should be called something like <privateKeyName>_id_rsa.ppk and stored somewhere safe but accessible to the scheduled task<br />
:*** The public key can be saved or you can just copy the key, which should be in clear test in the box to the clipboard<br />
:* With the public key in the clipboard, go to the globus website and click on your account name at top-right, and select "manage identities"<br />
:** Select "Add linked identity" and pick "Add SSH public key"<br />
:** Paste the public key into the box for it and give the key a name<br />
:** Click "Submit"<br />
<br />
:*Create a connection:<br />
:** Start PuTTY<br />
:** In the "Session" tab: <br />
:*** put <accountName>@cli.globusonline.org in the Host Name box<br />
:*** in the "Saved Sessions" textbox give the session a name; I use "globusSync" and we'll refer to this as <sessionName><br />
:** On the Connection > SSH > Auth tab, for "Private Key for authentication" click "browse" and select the file in which you wrote your private key<br />
:** Back on the "Session" tab, click "save"; your session name should now appear on the list of saved sessions<br />
:** You can test that it works, now; double click on the saved session name; after accepting the server key, you should find yourself in an ssh session<br />
<br />
You'll use plink to actually send the sync command (or any other Globus CLI commands you want to use); depending on whether it's in the right paths, you may wish to use the full path to the plink executable (for example, C:\Program Files (x86)\PuTTY\plink.exe) when you set this up as a scheduled task. The basic command, to run on the command line, is this:<br />
<br />
"C:\Program Files (x86)\PuTTY\plink.exe" <sessionName> transfer -s 2 --preserve-mtime --verify-checksum -- <accountName>#<localEndpointName>/<path> cac#archive01/export/archive01/<CACProjectName>/<path> -r<br />
<br />
You should test that it works by calling up cmd.exe and executing it.<br />
<br />
===Creating the scheduled task===<br />
<br />
:* Start up the task scheduler<br />
:* Select "Create task" from the "Actions" tab<br />
:* Give the task a name, select a user identity as which this should run (ensuring it has access to the ssh session and key information you set up) and select the "run whether user is logged on or not" radio button (note that this doesn't fix the issue of the endpoint going down if the owning user isn't logged on). If only local resources will be required you can select to not store password details<br />
:** On the "Triggers" tab<br />
:*** Select "New"<br />
:*** Select "run on a schedule" from the drop-down, and select when you want it to run and on what cadence<br />
:*** Select the checkbox for "Enabled" (important!)<br />
:** On the "Actions" tab<br />
:*** Select "New"<br />
:*** Select "Start a program" from the drop-down<br />
:*** For "Program name" put the full path to plink.exe, enclosed in double quotes if it contains a space, eg: <br />
"C:\Program Files (x86)\PuTTY\plink.exe"<br />
:* For "arguments" enter: <br />
<sessionName> transfer -s 2 --preserve-mtime --verify-checksum -- <globusAccountName>#<localEndpointName>/<path> cac#archive01/export/archive01/<CACProjectName>/<path> -r<br />
:* Accept the other defaults, and click "OK". You'll have to enter the Windows credentials for the account under which the process will run if you didn't select the option not to store the password.</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Archival_Storage&diff=822Archival Storage2015-09-29T20:23:06Z<p>Ad876: </p>
<hr />
<div>== What is CAC Archival Storage? ==<br />
:* CAC Archival Storage is a low-cost, high-performance option for storing research data '''available only to users within Cornell University'''. <br />
:* CAC Archival Storage is not mountable by running jobs, instead the user must transfer their data from the CAC Archival Storage to an accessible server using [//globusonline.org/ Globus Online]. <br />
:* Globus Online users have easy access to add, delete, and share their data using any Globus Online endpoints. <br />
:* Some of the Globus Online endpoints available include:<br />
:** storage01.cac.cornell.edu (cac#home) where all CAC user home directories are found<br />
:** XSEDE sites:<br />
:*** Stampede (xsede#stampede) <br />
:*** Lonestar (xsede#lonestar4) <br />
:*** TACC Archival Storage (xsede#ranch).<br />
<br />
== First step - Enable (or create) CAC project for Archival Storage and add users where appropriate ==<br />
:* To use the CAC Archival Storage service, '''you must be a user of a CAC project where Archival Storage is enabled'''.<br />
:* The project PI can add users and verify that Archival Storage is enabled at the [//cac.cornell.edu/Services/Projects/manage.aspx Manage CAC project page].<br />
:* Don't have a project? [//cac.cornell.edu/Services/projects.aspx How to start a CAC project?].<br />
<br />
== Second step - create your Globus Online account ==<br />
<br />
[//globus.org/SignUp Sign up] for a Globus account. CAC's Archival system is '''only accessible''' through '''[//globusonline.org/ Globus Online]'''.<br />
<br />
== CAC specifics ==<br />
=== Technical Information ===<br />
CAC's EndPoint is <b>cac#archive01</b>.<br />
<br />
:*When activating cac#archive01 endpoint in Globus Online web GUI, you will be prompted by a dialog box saying: <br />
<br />
<blockquote>The administrator of this endpoint, cac#archive01, requires that you authenticate using their MyProxy OAuth server to activate the endpoint. When you click 'Continue' you will be redirected to their website.</blockquote><br />
<br />
:*You will be redirected to the <nowiki>https://archive01.cac.cornell.edu/oath/authorize...</nowiki> page. <br />
:*Enter your CAC credentials.<br />
:*When login is successful, you will be redirected back to Globus Online web GUI with the endpoint activated.<br />
<br />
=== Administrative Information ===<br />
:* cac#archive01's default path is '''/export'''.<br />
:* Each project with access to CAC Archival Storage has a shared directory (named the project) in which '''all project members have full read/write access'''.<br />
:* Users can rename and move files and directories within their project directory on the endpoint. Globus Online added this feature recently.<br />
<br />
==Automated Archival==<br />
:* Install Globus Connect Personal on the Linux/MacOS/Windows host you wish to archive by clicking on the "Get Globus Connect Personal" link on the Transfer Files screen on Globus. <br />
::[[File:Install_Globus_Connect_Personal.jpg]]<br />
:* On the host you wish to archive, download and untar [[Media:archive_scripts.tar.gz]].<br />
:* To enable running Globus Connect Personal as root, add <br />
<br />
"-allow-root",<br />
<br />
::to globusconnectpersonal-2.0.3/gc.py (on line ~ 360):<br />
<source lang="c"><br />
args = [os.path.basename(PDEATH_LAUNCH),<br />
GRIDFTP_SERVER,<br />
"-allow-root",<br />
"-i", "-always-send-markers",<br />
"-hostname", "127.0.0.1",<br />
</source><br />
:* Copy root-bin directory from the archive_scripts.tar.gz to /root/bin. If you are archiving directories outside /home, modify the -restrict-path argument in /root/bin/gc_start.sh.<br />
:* Generate a ssh key pair using the "ssh-keygen" command, leave private key in ~/.ssh, and upload the private key to Globus<br />
::[[File:Upload_ssh_private_key.jpg]]<br />
:* Make sure you can access Globus CLI like this:<br />
ssh -i .ssh/<private key> <globus user name>@cli.globusonline.org<br />
:* Modify archive.sh to match your Globus user name, private key file name, CAC project and archive directory.<br />
:* On Globus, make sure your connection to cac#archive01 endpoint is activated. <br />
:* You should now be able to run archive.sh to upload your archive directory to CAC archive. You can automate this script using cron.<br />
<br />
==Globus Online links ==<br />
:*[//globus.org/how-it-works How Globus Online works?]<br />
:*[//globusonline.org/quickstart/ Globus Online Quickstart] guide for setting up an account.<br />
:*[//support.globus.org/entries/23583857-Sign-Up-and-Transfer-Files-with-Globus-Online Transfer Data using Globus Online]<br />
:*[//support.globus.org/entries/23602336-Sharing-Data-using-Globus Sharing Data using Globus Online]<br />
:*[//globus.org/group-management Group Management] NOTE: A Globus Plus or a Globus Provider plan are required; CAC's Globus endpoint '''does not''' '''support''' Group Management.</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Linux_Usage_Tips&diff=815Linux Usage Tips2015-09-29T19:45:18Z<p>Ad876: /* Linux shells */</p>
<hr />
<div>==== Linux shells====<br />
:* <tt>/bin/sh</tt> is the default login shell.<br />
:** Edit <tt>$HOME/.profile</tt> to change interactive variables.<br />
:** The <tt>$HOME/.bashrc</tt> file ''will not'' be run for non-interactive shells.<br />
:* <tt>/bin/bash</tt><br />
:** Edit <tt>$HOME/.profile</tt> to change interactive variables.<br />
:** The <tt>$HOME/.bashrc</tt> file ''will'' be run for non-interactive shells.<br />
:* <tt>/bin/csh</tt> and <tt>/bin/tcsh</tt><br />
:** Edit <tt>$HOME/.login</tt> to change interactive variables.<br />
:** The <tt>$HOME/.cshrc</tt> file ''will'' be run for non-interactive shells.<br />
<br />
The change shell command, <tt>chsh</tt>, will not permanently change your shell. You must send a request instead. {{ContactCAC}}<br />
<br />
The default login shell on v4 interactive and batch nodes is ''sh''. Be aware that in Red Hat Enterprise Linux, /bin/sh is a soft-link to /bin/bash, so you are really using a variant of ''bash''. Accordingly, you will find that "man sh" brings up the man page (the help document) for ''bash''. In a way, then, you can think of your login shell as being ''bash'', too.<br />
<br />
There are slight differences between ''sh'' and ''bash'', however. The "Invocation" section of the man page states: "If bash is invoked with the name sh, it tries to mimic the startup behavior of historical versions of ''sh'' as closely as possible." Therefore, you will find that ~/.profile is run at login, because this behavior is common to both ''sh'' and ''bash''; but any interactive ''sh'' shells you start thereafter will not run ~/.bashrc as you might expect from ''bash''. The way to get ''sh'' to do this is to "export ENV=~/.bashrc" beforehand (perhaps as part of your .profile).<br />
<br />
Let's say you simply prefer to have ''bash'' as your default shell and be done with it. There are two ways to accomplish this. First, you can "export SHELL=/bin/bash" in your .profile; then all subsequent interactive shells will truly be ''bash''. Second, you can enter "chsh -s /bin/bash", which forces all login and interactive shells to be ''bash'' (because you have changed your default shell). The problem with the second method is it may well wreck your batch environment, too, because the scheduler sets it up under the assumption that the login shell is ''sh''.<br />
<br />
The relationship between the ''csh'' and ''tcsh'' shells is similar to the one between ''sh'' and ''bash''. For instance, your ''csh'' shells are automatically endowed with the ''tcsh''-style ability to retrieve history through the up- and down-arrow keys. The best way to make ''tcsh'' into your everyday working shell is to run it on top of ''sh'' after you log in (again, you can do this as part of your .profile).<br />
<br />
References<br />
:* "man bash" from the command line.<br />
:* [http://tldp.org/LDP/abs/html/ Advanced Bash Scripting Guide], one of the Linux Documentation Project [http://www.tldp.org/guides.html guides]<br />
:* [http://mywiki.wooledge.org/BashFAQ Bash FAQ]<br />
:*[http://mywiki.wooledge.org/BashPitfalls Bash Pitfalls]<br />
<br />
====Compiling and linking code on Linux====<br />
{{:Compiling Code Linux}}<br />
<br />
====FAQ====<br />
=====How do I determine my program's dependencies on shared library (.so) files?=====<br />
:*ldd - see the man page.<br />
If your program cannot find all the .so files it needs, you may need to add paths to the LD_LIBRARY_PATH shell variable.<br />
<br />
=====How do I display an image file (such as jpeg or gif)?=====<br />
:*display mypic.jpg - uses one of the many ImageMagick tools - see "man ImageMagick" for help on this and various file format converters.<br />
:*firefox mypic.jpg - any decent Web browser can handle it.<br />
Note, the image will show up only if you have [[Getting_started#Connect_to_Linux | X11 forwarding]] enabled.</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Linux_Usage_Tips&diff=814Linux Usage Tips2015-09-29T19:43:20Z<p>Ad876: </p>
<hr />
<div>==== Linux shells====<br />
:* <tt>/bin/sh</tt> is the default login shell.<br />
:** Edit <tt>$HOME/.profile</tt> to change interactive variables.<br />
:** The <tt>$HOME/.bashrc</tt> file ''will not'' be run for non-interactive shells.<br />
:* <tt>/bin/bash</tt><br />
:** Edit <tt>$HOME/.profile</tt> to change interactive variables.<br />
:** The <tt>$HOME/.bashrc</tt> file ''will'' be run for non-interactive shells.<br />
:* <tt>/bin/csh</tt> and <tt>/bin/tcsh</tt><br />
:** Edit <tt>$HOME/.login</tt> to change interactive variables.<br />
:** The <tt>$HOME/.cshrc</tt> file ''will'' be run for non-interactive shells.<br />
<br />
The change shell command, <tt>chsh</tt>, will not permanently change your shell. You must send a request instead. {{ContactCAC}}<br />
<br />
The default login shell on v4 interactive and batch nodes is ''sh''. Be aware that in Red Hat Enterprise Linux, /bin/sh is a soft-link to /bin/bash, so you are really using a variant of ''bash''. Accordingly, you will find that "man sh" brings up the man page (the help document) for ''bash''. In a way, then, you can think of your login shell as being ''bash'', too.<br />
<br />
There are slight differences between ''sh'' and ''bash'', however. The "Invocation" section of the man page states: "If bash is invoked with the name sh, it tries to mimic the startup behavior of historical versions of ''sh'' as closely as possible." Therefore, you will find that ~/.profile is run at login, because this behavior is common to both ''sh'' and ''bash''; but any interactive ''sh'' shells you start thereafter will not run ~/.bashrc as you might expect from ''bash''. The way to get ''sh'' to do this is to "export ENV=~/.bashrc" beforehand (perhaps as part of your .profile).<br />
<br />
Let's say you simply prefer to have ''bash'' as your default shell and be done with it. There are two ways to accomplish this. First, you can "export SHELL=/bin/bash" in your .profile; then all subsequent interactive shells will truly be ''bash''. Second, you can enter "chsh -s /bin/bash", which forces all login and interactive shells to be ''bash'' (because you have changed your default shell). The problem with the second method is it may well wreck your batch environment, too, because the scheduler sets it up under the assumption that the login shell is ''sh''.<br />
<br />
The relationship between the ''csh'' and ''tcsh'' shells is similar to the one between ''sh'' and ''bash''. For instance, your ''csh'' shells are automatically endowed with the ''tcsh''-style ability to retrieve history through the up- and down-arrow keys. The best way to make ''tcsh'' into your everyday working shell is to run it on top of ''sh'' after you log in (again, you can do this as part of your .profile).<br />
<br />
References<br />
:* "man bash" from the command line.<br />
:* [//tldp.org/LDP/abs/html/ Advanced Bash Scripting Guide], one of the Linux Documentation Project [http://www.tldp.org/guides.html guides]<br />
:* [//mywiki.wooledge.org/BashFAQ Bash FAQ]<br />
:*[//mywiki.wooledge.org/BashPitfalls Bash Pitfalls]<br />
<br />
====Compiling and linking code on Linux====<br />
{{:Compiling Code Linux}}<br />
<br />
====FAQ====<br />
=====How do I determine my program's dependencies on shared library (.so) files?=====<br />
:*ldd - see the man page.<br />
If your program cannot find all the .so files it needs, you may need to add paths to the LD_LIBRARY_PATH shell variable.<br />
<br />
=====How do I display an image file (such as jpeg or gif)?=====<br />
:*display mypic.jpg - uses one of the many ImageMagick tools - see "man ImageMagick" for help on this and various file format converters.<br />
:*firefox mypic.jpg - any decent Web browser can handle it.<br />
Note, the image will show up only if you have [[Getting_started#Connect_to_Linux | X11 forwarding]] enabled.</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Home_Directory_Access&diff=813Home Directory Access2015-09-29T19:42:45Z<p>Ad876: </p>
<hr />
<div>There is one filesystem which is shared by both linux and windows systems:<br />
:*Your home directory on linux is: <tt>/home/fs01/userid</tt><br />
:*Your home directory on windows is: <tt>\\storage01.cac.cornell.edu\userid</tt><br />
In Linux, it is generally safe to refer to your home directory as either ~, ~username, or $HOME, so you never need to specify the exact mount point. On CAC's Windows systems, similarly, the letter drive H: is mapped to your home directory; however, it is often preferable to use the full UNC path to the network share, as given above.<br />
<br />
You can mount your home directory on your local machine, as long as your machine is connected to Cornell campus network or [//it.cornell.edu/services/managed_servers/howto/remote/vpn.cfm Cornell VPN]<br />
<br />
====Linux Users====<br />
<br />
You mount your CAC home directory via SMB/CIFS like this:<br />
sudo mount -t cifs //storage01.cac.cornell.edu/<myid> /mnt/pt -o user=<myid>,domain=CTC_ITH,uid=<localid> <br />
where <tt><myid></tt>is your CAC user name, <localid> is your local user name, and <tt>/mnt/pt</tt> is the name of a directory you have created ahead of time to be the mount point on your local filesystem. Enter the password for CAC account when prompted. See <tt>'''man mount.cifs'''</tt> for available options for the mount command. <br />
<br />
If you see errors, such as "missing codepage or helper program," then you have not installed the mount and umount packages for CIFS on your local machine. If problems persist, send your initial command and the results of <tt>'''dmesg | tail'''</tt>.<br />
<br />
====MacOS X users====<br />
<br />
:*In the Finder, either select '''''Connect to Server...''''' from the '''''Go''''' menu or use the shortcut <tt>'''cmd-K'''</tt>. <br /><br />[[Image:FileAccess1.jpg]]<br />
:* Enter '''''smb://storage01.cac.cornell.edu/<user name>''''' in the Server Address field as shown below. You may need to use '''smb://<username>@storage01.cac.cornell.edu/<username>'''. <br /><br />[[Image:FileAccess2.jpg]]<br />
:* Enter your CAC user name and password to log in. You may need to use '''<username>@tc.cornell.edu''' in place of your username.<br />
<br />
====Windows users====<br />
<br />
:* Open My Computer<br />
:*Click on Tools -> Map Network Drive<br />
:* Drive H: (if you are already using this drive letter, use another letter)<br />
:* Folder: \\storage01.cac.cornell.edu\<userid><br />
:* Then:<br />
::: -Select "Connect using a different user name:". This will allow you to enter the domain associated with CAC and your userid at CAC, rather than those associated with your own machine.<br />
::: -User name: '''''CTC_ITH\your_userid'''''<br />
::: -Password: your CAC password<br />
:* Troubleshooting: If you have already mapped the drive and subsequently have problems, disconnect the drive and remap it.</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=File_transfer&diff=812File transfer2015-09-29T19:41:55Z<p>Ad876: /* Secure FTP */</p>
<hr />
<div>A single, central file server, storage01.cac.cornell.edu, serves all CAC user home directories. You can connect to this server in a variety of ways from any operating system to access your files. <br />
<br />
Note: by default, your home directory and its contents will be readable and executable by all other users of CAC systems. If this is not what you want, you can change the permissions of the home directory and its files and subdirectories via the standard Linux or Windows mechanisms. However, be aware that this may lead to conflicts for cross-platform applications, as Windows and Linux permissions are not 100% compatible.<br />
<br />
====Linux users====<br />
=====Secure Copy=====<br />
Secure copy is a standard tool to copy files to and from remote hosts.<br />
localhost$ scp localfile.dat username@linuxlogin3.cac.cornell.edu:remoteinput.dat<br />
localhost$ scp username@linuxlogin3.cac.cornell.edu:results.dat localresults.dat <br />
=====Secure FTP=====<br />
FTP is disabled for security reasons, but sftp's interface is nearly identical.<br />
=====Samba Client=====<br />
Type<br />
smbclient //storage01.cac.cornell.edu/<user name> -U <user name><br />
Enter the password for your CAC account when prompted<br />
You will see the '''''smb:\>''''' prompt. You can now start transferring files between your local machine and CAC home directory similar to ftp client. Type '''''help''''' for more instructions.<br />
-sh-3.2$ smbclient //storage01.cac.cornell.edu/<user name> -U <user name><br />
Password: <br />
Domain=[CTC_ITH] OS=[Unix] Server=[Samba 3.0.28-1.el5_2.1]<br />
smb: \> help<br />
<br />
====MacOS X users====<br />
<br />
====Windows users====<br />
=====Secure Copy=====<br />
The people who make Putty provide a secure copy client called [http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html pscp]. From the command prompt, type:<br />
cmd> pscp localfile.dat username@linuxlogin3.cac.cornell.edu:remoteinput.dat<br />
<enter your username's password when prompted><br />
cmd> pscp username@linuxlogin.cac.cornell.edu:results.dat localresults.dat<br />
<br />
=====Secure FTP=====<br />
FTP is disabled for security reasons, but [http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html psftp's] interface is nearly identical. From the command prompt, type:<br />
cmd> psftp username@linuxlogin.cac.cornell.edu<br />
<enter your username's password when prompted><br />
psftp> put localresults.dat results.dat<br />
psftp> quit</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=File_transfer&diff=811File transfer2015-09-29T19:41:31Z<p>Ad876: /* Secure Copy */</p>
<hr />
<div>A single, central file server, storage01.cac.cornell.edu, serves all CAC user home directories. You can connect to this server in a variety of ways from any operating system to access your files. <br />
<br />
Note: by default, your home directory and its contents will be readable and executable by all other users of CAC systems. If this is not what you want, you can change the permissions of the home directory and its files and subdirectories via the standard Linux or Windows mechanisms. However, be aware that this may lead to conflicts for cross-platform applications, as Windows and Linux permissions are not 100% compatible.<br />
<br />
====Linux users====<br />
=====Secure Copy=====<br />
Secure copy is a standard tool to copy files to and from remote hosts.<br />
localhost$ scp localfile.dat username@linuxlogin3.cac.cornell.edu:remoteinput.dat<br />
localhost$ scp username@linuxlogin3.cac.cornell.edu:results.dat localresults.dat <br />
=====Secure FTP=====<br />
FTP is disabled for security reasons, but sftp's interface is nearly identical.<br />
=====Samba Client=====<br />
Type<br />
smbclient //storage01.cac.cornell.edu/<user name> -U <user name><br />
Enter the password for your CAC account when prompted<br />
You will see the '''''smb:\>''''' prompt. You can now start transferring files between your local machine and CAC home directory similar to ftp client. Type '''''help''''' for more instructions.<br />
-sh-3.2$ smbclient //storage01.cac.cornell.edu/<user name> -U <user name><br />
Password: <br />
Domain=[CTC_ITH] OS=[Unix] Server=[Samba 3.0.28-1.el5_2.1]<br />
smb: \> help<br />
<br />
====MacOS X users====<br />
<br />
====Windows users====<br />
=====Secure Copy=====<br />
The people who make Putty provide a secure copy client called [http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html pscp]. From the command prompt, type:<br />
cmd> pscp localfile.dat username@linuxlogin3.cac.cornell.edu:remoteinput.dat<br />
<enter your username's password when prompted><br />
cmd> pscp username@linuxlogin.cac.cornell.edu:results.dat localresults.dat<br />
<br />
=====Secure FTP=====<br />
FTP is disabled for security reasons, but [//chiark.greenend.org.uk/~sgtatham/putty/download.html psftp's] interface is nearly identical. From the command prompt, type:<br />
cmd> psftp username@linuxlogin.cac.cornell.edu<br />
<enter your username's password when prompted><br />
psftp> put localresults.dat results.dat<br />
psftp> quit</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=File_transfer&diff=810File transfer2015-09-29T19:39:00Z<p>Ad876: </p>
<hr />
<div>A single, central file server, storage01.cac.cornell.edu, serves all CAC user home directories. You can connect to this server in a variety of ways from any operating system to access your files. <br />
<br />
Note: by default, your home directory and its contents will be readable and executable by all other users of CAC systems. If this is not what you want, you can change the permissions of the home directory and its files and subdirectories via the standard Linux or Windows mechanisms. However, be aware that this may lead to conflicts for cross-platform applications, as Windows and Linux permissions are not 100% compatible.<br />
<br />
====Linux users====<br />
=====Secure Copy=====<br />
Secure copy is a standard tool to copy files to and from remote hosts.<br />
localhost$ scp localfile.dat username@linuxlogin3.cac.cornell.edu:remoteinput.dat<br />
localhost$ scp username@linuxlogin3.cac.cornell.edu:results.dat localresults.dat <br />
=====Secure FTP=====<br />
FTP is disabled for security reasons, but sftp's interface is nearly identical.<br />
=====Samba Client=====<br />
Type<br />
smbclient //storage01.cac.cornell.edu/<user name> -U <user name><br />
Enter the password for your CAC account when prompted<br />
You will see the '''''smb:\>''''' prompt. You can now start transferring files between your local machine and CAC home directory similar to ftp client. Type '''''help''''' for more instructions.<br />
-sh-3.2$ smbclient //storage01.cac.cornell.edu/<user name> -U <user name><br />
Password: <br />
Domain=[CTC_ITH] OS=[Unix] Server=[Samba 3.0.28-1.el5_2.1]<br />
smb: \> help<br />
<br />
====MacOS X users====<br />
<br />
====Windows users====<br />
=====Secure Copy=====<br />
The people who make Putty provide a secure copy client called [http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html pscp]. From the command prompt, type:<br />
cmd> pscp localfile.dat username@linuxlogin3.cac.cornell.edu:remoteinput.dat<br />
<enter your username's password when prompted><br />
cmd> pscp username@linuxlogin.cac.cornell.edu:results.dat localresults.dat <br />
=====Secure FTP=====<br />
FTP is disabled for security reasons, but [//chiark.greenend.org.uk/~sgtatham/putty/download.html psftp's] interface is nearly identical. From the command prompt, type:<br />
cmd> psftp username@linuxlogin.cac.cornell.edu<br />
<enter your username's password when prompted><br />
psftp> put localresults.dat results.dat<br />
psftp> quit</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Connect_to_Windows&diff=809Connect to Windows2015-09-29T19:38:11Z<p>Ad876: </p>
<hr />
<div>'''''Using Remote Desktop Connection to connect to winlogin'''''<br />
<br />
This method of connecting to winlogin is preferred because it provides you with a fully functional Windows desktop. At the login screen, if the domain is specified, it should be set to '''''CTC_ITH''''', not the local name of the machine to which you are connecting.<br />
<br />
:'''Remote Desktop Connect Details''':<br />
<br />
Remote Desktop sessions do not expire, but they will end when machines are rebooted during down times.<br />
<br />
:* '''If you use a Windows machine''':<br />Use the Remote Desktop Connection (older name Terminal Services Client) to connect to a login machine. This software is pre-installed with Windows 7 and later. To run it, click <tt>Start</tt>, then <tt>All Programs > Accessories > Communications > Remote Desktop Connection</tt>. Otherwise you need to [http://www.microsoft.com/windowsxp/downloads/tools/rdclientdl.mspx download the client] before you can use it.<br />
:*''' If you use Mac OS X 10.7 or later''':<br />Use the free download from [//itunes.apple.com/us/app/microsoft-remote-desktop/id715768417?mt=12 the Mac App Store]. Works just like the Remote Desktop Connection in Windows 7. You can also use rdesktop (see below). Tip: if authentication fails, make sure your software updates are current.<br />
:*''' If you use Unix or Linux or Mac''':<br />You can access the login machines by using the cross-platform rdesktop client. If you are running Linux, typically it is part of the distribution. If you prefer to build it yourself, it is available for download from [//rdesktop.org/ rdesktop]. Executables for old versions are available from [//jacco2.dds.nl/rdesktop/index.html. here]</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Connect_to_Linux&diff=808Connect to Linux2015-09-29T19:35:18Z<p>Ad876: </p>
<hr />
<div>__TOC__<br />
There are three distinct ways to connect to a login node:<br />
# [[Getting_started#Using_Secure_Shell | Use SSH]] to open a Linux shell on a login node, which provides a text-only interface.<br />
# [[Getting_started#Using_Secure_Shell | Use SSH]] together with [[Getting_started#Using_X-Windows | X-Windows]], which sends any interactive graphics back to your machine window-by-window through an SSH tunnel.<br />
# [[Getting_started#Using_VNC | Use VNC]] to get a remote desktop with multiple text and graphics windows. This is not as straightforward as it sounds, due to the need to set up a secure tunnel for the remote desktop first.<br />
<br />
These instructions are intended mainly for users of personal computers and workstations. However, much of the material carries over to mobile computing platforms such as tablets and smartphones. You will have to locate and download an app to enable SSH or VNC connectivity; even a browser plug-in may suffice.<br />
<br />
Whichever method you choose, at your first login, you will be challenged for a new password. Find help at [[Getting_started#Change_a_password_at_first_login | Changing a Password at First Login]]. You will also be asked for an ssh passphrase. You can just leave this blank; hit the <tt>Enter</tt> key in response.<br />
<br />
=====Using Secure Shell=====<br />
For basic command-line access, a Secure Shell (SSH) client will give you a remote command shell on one of the login nodes.<br />
:* Nearly all Unix/Linux varieties (including Mac) already have a built-in SSH2 implementation, required by our clusters.<br />
:* If you are coming from a Microsoft Windows machine, an SSH2 client must first be installed, as described below.<br />
:* The non-secure predecessor of SSH, telnet, is disabled for security reasons.<br />
<br />
'''''Linux users:'''''<br />
<br />
To connect to the second login node with ssh, you simply open a terminal window and type<br />
localhost$ ssh username@linuxlogin.cac.cornell.edu<br />
<br />
'''''Mac OS X users:'''''<br />
<br />
OS X on the Mac is built on a version of Unix, so ssh is available directly from the Terminal application.<br />
:*One option is to use the shortcut <tt>'''cmd-space'''</tt> to open Spotlight and then type <tt>"Terminal"</tt> to open a Terminal window.<br />
Otherwise:<br />
:*Navigate in the Finder to the Applications folder and Utilities sub-folder.<br />
:*:[[Image:MacApplicationsFolder.png|500px]]<br />
:*Then double-click on the Terminal application to see a Bash command-line.<br />
:*:[[Image:MacTerminalWindow.png]]<br />
:*As in Linux, simply type "ssh username@linuxlogin.cac.cornell.edu" into this window.<br />
<br />
'''''Windows users:'''''<br />
<br />
Secure Shell (ssh) clients work nicely as long as they support the SSH2 protocol. As mentioned, telnet is disabled for security reasons. A popular client for Windows is the free [//chiark.greenend.org.uk/~sgtatham/putty/ PuTTY client].<br />
:* The simplest installation is to download the [//chiark.greenend.org.uk/~sgtatham/putty/download.html Windows installer], called '''putty-0.65-installer.exe''', and run it. This installs PuTTY into your Start menu.<br />
:* To connect, start PuTTY, then type in a host name such as linuxlogin.cac.cornell.edu, and click "Open".<br />
:*:[[Image:Putty_address.jpg|Setting the host name in PuTTY]]<br />
<br />
=====Using X-Windows=====<br />
<br />
X-Windows or X11 is the longstanding Unix mechanism for displaying interactive graphics in a window. Your "X server" software runs locally, but it is capable of displaying windows that have been generated either locally or remotely. An "X client" on a remote machine can create X-Windows for local display, but it is necessary first to establish a shell on that machine using SSH.<br />
<br />
'''''Appropriate use'''''<br />
<br />
Among other things, X-Windows gives you the ability to display a GUI that originates on a login node. However, this ability does '''''NOT''''' imply that you are permitted to run compute-intensive, GUI-driven applications on these machines. Such usage is not only contrary to CAC policy, it is disrespectful toward other users, because the login node may become unresponsive through your actions.<br />
<br />
'''''Linux users:'''''<br />
<br />
The standard way to use X-Windows is to tunnel the X-Windows protocol through an ssh connection. If you open your ssh session with the '''-X''' option, it will automatically set up the necessary tunnel and environment variables.<br />
localhost$ ssh -X username@linuxlogin.cac.cornell.edu<br />
linuxlogin$ echo $DISPLAY<br />
localhost:11.0<br />
linuxlogin$ xclock&<br />
You can see that your DISPLAY environment variable is set and test it with xclock. There is another option to use a trusted version of X-windows forwarding<br />
linuxlogin$ ssh -Y compute-3-48.v4linux<br />
Th trusted version is necessary for forwarding X11 connections from a compute node to the login node, then back to your client machine.<br />
<br />
'''''Mac OS X users:'''''<br />
<br />
If you start ssh with the '''-X''' or '''-Y''' option, X-Windows should start up automatically. You can then try the "xclock" test, as described above for Linux. <br />
<br />
X11 is preinstalled on Macs starting with OS X 10.6 (Snow Leopard). For Mac OS X 10.5 (Leopard), you may need to install X11 in order for X-Windows applications to launch. If there is no X11 application in the Applications->Utilities folder, you'll have to find your OS X install disk. From the ''Mac OS X Server Introduction to Command-Line Administration,'' "The X11 server and an application to access X windows from the Finder are available as an optional installation in the Optional Installs folder of your installation disc (X11 is in the Applications package)."<br />
<br />
'''''Windows users:'''''<br />
<br />
Along with your ssh client (e.g., PuTTY), you will need to install an X-Windows server on your Windows machine.<br />
:* [http://www.straightrunning.com/XmingNotes/ Xming] - Open Source. A shareware contribution will get you a version with improved performance for graphics (GLX). There are two pieces to download<br />
:*:[[Image:Xming-download.jpg]]<br />
:** Xming-mesa (public domain release). There are two links together, one for Xming, one for Xming-mesa. Either will work, but Xming-mesa has some newer features that might come in handy some time.<br />
:** Xming-fonts (public domain release)<br />
If you purchase the website release of Xming, remember to install the Xming-fonts, as well.<br />
:* OpenText's [//cit.cornell.edu/services/software_licensing/available/exceed.cfm Exceed and Exceed 3D] - Cornell no longer has a site license. Installing Exceed 3D will improve performance of graphics applications. Exceed installs several icons under the Start menu. Choose the one that just says "Exceed" because it starts the program in multi-window mode, which is what we want.<br />
<br />
Here is how to start a session using PuTTY and Xming.<br />
<br />
# Start Xming from the Start menu. It will appear briefly and disappear except for an X in the application tray.<br />
# Start PuTTY.<br />
# In the window that appears, type a host name, <tt>linuxlogin.cac.cornell.edu</tt>.<br />
# Use the tree menu on the left to set X11 forwarding. It's in the <tt>Connection > SSH branch</tt>.<br />
#:[[Image:Putty_x11forwarding.jpg|Setting X11 forwarding in PuTTY]]<br />
# For PuTTY 0.61 only - In the "Auth" section of the SSH branch, go to GSSAPI and uncheck <tt>"Attempt GSSAPI authentication"</tt>. This will prevent an annoying <tt>"Access denied"</tt> message from appearing in your terminal window.<br />
# You can return to the Session category and Save this session's configuration for future use. Give it a logical name like linuxlogin.<br />
# Click Open, and it will connect to a login node.<br />
# Test your X-Windows setup by typing<br />
xclock<br />
You should see a clock appear in the corner of your screen. You can stop it by typing <tt>Ctrl-c</tt> in the terminal window.<br />
<br />
=====Using VNC=====<br />
<br />
[http://en.wikipedia.org/wiki/Vnc VNC] lets you see a whole Linux desktop from the login node on your computer.<br />
Using SSH and X-Windows is generally faster, and uses a lot less of the login node's resources,<br />
but VNC can be much faster if you are doing visualization on the login node from off campus.<br />
<br />
For security reasons, we are requiring all VNC connections to be tunneled inside ssh. You will therefore need to be able to connect to the login nodes [[Getting_started#Using_Secure_Shell | using SSH]]. Because the firewall running on linuxlogin blocks all incoming ports except for ssh, VNC connections must be made over a ssh tunnel as described below.<br />
<br />
'''''Appropriate use'''''<br />
<br />
VNC gives you the ability to establish a remote desktop on the login nodes, but this ability does '''''NOT''''' imply that you are permitted to run compute-intensive, GUI-driven applications on these machines. Such usage is not only contrary to CAC policy, it is disrespectful toward other users, because the login node may become unresponsive through your actions.<br />
<br />
Here is a good example of how to use VNC appropriately. By following these steps you can run (say) Abaqus in GUI-driven mode on a compute node that has been allocated to you through an interactive batch job.<br />
# Open a VNC connection to linuxlogin through an ssh tunnel using the instructions below, in order to gain access to a Linux desktop. Make sure two terminal windows are available on this desktop.<br />
# In one of the terminal windows, submit an interactive job to the queue of your choice (add the #PBS -I directive to your job submission script).<br />
# Once the job starts, you will be given a command prompt on your assigned machine. Note the result of "hostname". There is no need to enter further commands at this prompt (except to exit the job).<br />
# Go to the other terminal window and open a second ssh connection to the compute node using "ssh -Y <userid>@<hostname>"<br />
# This new ssh session will tunnel X-Windows from the compute node back to the VNC desktop. Therefore (if Abaqus is on your path), you can now open the Abaqus GUI using "abaqus cae -mesa".<br />
<br />
'''''Initial setup'''''<br />
''(You only need to do this once)''<br />
<br />
:* Install a VNC client if one isn't installed. [http://www.tightvnc.com/ TightVNC] works well, but so do others.<br />
:* Login to linuxlogin, and set the password for your VNC server using the "vncpasswd" command.<br />
<br />
'''''Start your VNC server'''''<br />
<br />
:* '''On linuxlogin''', start the VNC server using the "vncserver" command like this:<br />
vncserver -geometry 1024x768 -localhost<br />
The geometry numbers, 1024x768, specify the size, in pixels, of the desktop.<br />
:* You will need to get the display number from the output of the vncserver command:<br />
<br />
New 'linuxlogin.cac.cornell.edu:1 (shl1)' desktop is linuxlogin.cac.cornell.edu:1<br />
Starting applications specified in /home/gfs01/shl1/.vnc/xstartup<br />
Log file is /home/gfs01/shl1/.vnc/linuxlogin.cac.cornell.edu:1.log<br />
<br />
:* vncserver is running on port 5900 + display number. In the above example, the display number is :1, therefore vncserver is running on port 5901.<br />
<br />
'''''Connect your VNC client'''''<br />
<br />
:* Set up ssh forwarding on your client computer. Let's say the port number on linuxlogin is 5901 (as above), and your CAC userid is uid12. From Linux, type into a terminal:<br />
<br />
ssh -L 10000:localhost:5901 uid12@linuxlogin.cac.cornell.edu<br />
'''From Windows''', ssh clients such as PuTTY can do X11 port forwarding. See [[VNC Tunnel Windows]]. <br />
<br />
'''For MacOS X users''', see [[CAC VPN Server for MacOS Users |here]]<br />
:* Leave this ssh session running on your local client computer. (It can run in the background.)<br />
:* Launch your VNC client program. Connect to localhost:10000. When prompted, type in your VNC server password.<br />
<br />
'''''To disconnect your client'''''<br />
<br />
:* Close the vnc client program.<br />
:* Disconnect the ssh forwarding session (i.e., kill it).<br />
<br />
'''''To reconnect your client'''''<br />
<br />
:* Restart port forwarding with ssh, using the same remote port number as before.<br />
:* Again connect the VNC client to localhost:10000.<br />
<br />
'''''When you are all done'''''<br />
<br />
:* On linuxlogin, type this command to shut down the VNC server<br />
vncserver -kill :<display number><br />
:* If you merely log out from linuxlogin, it will leave the VNC server running. You must shut down the VNC server explicitly when you are finished with it. (Actually this can be a nice feature.)<br />
<br />
=====Passwordless SSH=====<br />
<br />
''''' Create ssh key pair '''''<br />
<br />
Your ssh key pair will only need to be created once. You will not need to repeat this step. You can complete this step from either a Linux or Windows login node. If this is your first login to a CAC login node, it will ask you to [[Getting_started#Change_a_password_at_first_login|change your password]]. This will become your password for connecting to the nodes. <br />
<br />
Create your ssh key pair by logging into the linux login node (linuxlogin.cac.cornell.edu), which will begin the process of creating the keys; you can use the defaults or empty responses for all prompts.<br />
<br />
Alternatively, you can create your ssh key pair on the linux login node by logging into the Windows login node (winx64login.cac.cornell.edu), opening a Command Prompt window, and running <tt>plink.exe</tt> to connect to the linux login node, as shown in this example:<br />
<br />
<pre>>"C:\Programs Files (x86)\Putty\plink.exe" %USERNAME%@linuxlogin.cac.cornell.edu<br />
Password: Enter Your Password<br />
Rocks 5.0 (V)<br />
Profile built 12:54 06-May-2008<br />
<br />
Kickstarted 09:22 06-May-2008<br />
-----------------------------------------------------------<br />
Welcome to the Center for Advanced Computing Cluster!<br />
-----------------------------------------------------------<br />
Please send your questions to help@cac.cornell.edu<br />
-----------------------------------------------------------<br />
<br />
<br />
It doesn't appear that you have set up your ssh key.<br />
This process will make the files:<br />
/home/gfs01/cacshl1/.ssh/id_rsa.pub<br />
/home/gfs01/cacshl1/.ssh/id_rsa<br />
/home/gfs01/cacshl1/.ssh/authorized_keys<br />
<br />
Generating public/private rsa key pair.<br />
Enter file in which to save the key (/home/gfs01/cacshl1/.ssh/id_rsa): Press Enter to accept default<br />
Created directory '/home/gfs01/cacshl1/.ssh'.<br />
Enter passphrase (empty for no passphrase): Press Enter to accept default<br />
Enter same passphrase again: Press Enter to accept default<br />
Your identification has been saved in /home/gfs01/cacshl1/.ssh/id_rsa.<br />
Your public key has been saved in /home/gfs01/cacshl1/.ssh/id_rsa.pub.<br />
</pre><br />
<br />
After this is done, type '''"exit"''' to log out of the linux login node.<br />
<br />
''''' Convert ssh Private Key for Putty / Plink '''''<br />
<br />
Next run PuTTYgen to generate public and private keys to be used with PuTTY and Plink:<br />
<br />
:* Log in to <tt>winx64login.tc.cornell.edu </tt>(if you are not already)<br />
:* Run <tt>C:\Program Files (x86)\Putty\puttygen.exe</tt>.<br />
:* Select <tt>Import Key</tt> from the <tt>Conversions</tt> menu and select <tt>H:\.ssh\id_rsa</tt> in your home directory. And click on the <tt>Open</tt> button.<br />
<center>[[image:LoadPrivateKey.jpg]]</center><br />
:* Click on the <tt>"Save Private Key"</tt> button. <br />
<center>[[image:SavePrivateKey.jpg]]</center><br />
:* Click on "Yes" when asked to save the private key without a passphrase.<br />
:* Save the private key as private.ppk in the .ssh directory inside your home directory.<br />
<center>[[image:SpecifyPrivateKey.jpg]]</center><br />
:* Close (choose File, then Exit)<br />
:* To confirm you have converted the ssh private key successfully, do:<br />
<pre>"C:\Program Files (x86)\Putty\plink.exe" -i %HOMEDRIVE%\.ssh\private.ppk %USERNAME%@linuxlogin.cac.cornell.edu</pre><br />
It may notify you that "The server's host key is not cached in the registry." Type "y" to "store the key in cache."<br />
:* You should now be logged into linuxlogin without being prompted for a password. Stay logged in for the next step.</div>Ad876https://www.cac.cornell.edu/wiki/index.php?title=Getting_Started&diff=806Getting Started2015-09-29T19:30:42Z<p>Ad876: </p>
<hr />
<div>__TOC__<br />
==Managing your password==<br />
CAC has a [[Getting_started#Rules_for_Creating_Passwords|Password Policy]] in effect. The first time that you login to the <tt>cac.cornell.edu</tt> domain, you will be required to change your password. Each password must have at least eight characters and must contain at least three of the following four elements: (1) uppercase letters (2) lowercase letters (3) special characters (4) digits. Your password can be set or changed on any of the CAC login nodes, and the password will be updated on all CAC resources. Passwords expire every six months. Do not share your password. There are more detailed instructions below.<br />
<br />
===Rules for Creating passwords===<br />
{{:Rules for Creating Passwords}}<br />
<br />
===Change a password at first login===<br />
{{:Changing password at first login}}<br />
<br />
=== Change password at any time ===<br />
{{:Changing password any time}}<br />
<br />
=== Locked Accounts===<br />
<br />
There have been instances in which user accounts have been locked. Some common causes of locked accounts and the solutions are:<br />
<br />
:*Mistyping your password several times in a row. <br />
:::<tt>Solution</tt>: Wait about a 1/2 hour and then try again. Be sure that your caps lock key is not on!<br />
:*Trying to login to a Windows login node by using SSH when you have a new or expired password. <br />
:::<tt>Solution</tt>: Login to a Windows login node using Remote Desktop Connection or SSH to a linux login node.<br />
:*Failing to log off all other sessions connected to login nodes. <br />
:::<tt>Solution</tt>: Log off all remote connections. Disconnecting the sessions is not enough.<br />
:*Failing to disconnect locally mapped drives to the CAC file server before changing your password. <br />
:::<tt>Solution</tt>: Disconnect all locally mapped drives, wait a 1/2 hour until account is unlocked, and then re-map the drive with the new password.<br />
<br />
If you can't log on or can't wait you can submit a Password Reset ticket on our [//rt.cac.cornell.edu/index.html issue tracking system]<br />
<br />
==Checking your CAC project ==<br />
Cornell University users can view their account limits at [//cac.cornell.edu/services/cu/memberlimits.aspx CAC Account Limits].<br /><br />
Partner Program members should contact Paul Redfern at [mailto:red@cac.cornell.edu red@cac.cornell.edu] if they need information on their membership limits.<br />
<br />
==Using CAC resources==<br />
===Connecting to CAC===<br />
<br />
There are two types of login nodes:<br />
:* Linux login nodes: <tt>linuxlogin.cac.cornell.edu</tt> as well as the head nodes for the various Linux-based private clusters.<br />
:* Windows login node: <tt>winlogin.cac.cornell.edu</tt><br />
<br />
====Connect to Linux====<br />
<br />
{{:Connect to Linux}}<br />
<br />
====Connect to Windows====<br />
<br />
{{:Connect to Windows}}<br />
<br />
===Home Directory Access===<br />
{{:Home Directory Access}}<br />
<br />
===File transfer===<br />
{{:File transfer}}<br />
<br />
<br />
===Linux Usage Tips===<br />
{{:Linux Usage Tips}}<br />
==== More information on Linux nodes at CAC====<br />
For more detailed instructions on how to use the Linux node, see [[Tutorial for the Linux nodes at CAC| here]]<br />
<br />
===Windows Usage Tips===<br />
{{:Windows Usage Tips}}<br />
<br />
===More information===<br />
The CAC Web site is [//cac.cornell.edu/ here] . There are many useful documents on the Support page at [//cac.cornell.edu/docwiki CAC documentation].<br />
<br />
===Acknowledging CAC===<br />
{{:Acknowledging CAC}}<br />
<br />
===FAQ/Troubleshooting===<br />
#[[FAQ#Account| Account FAQ]]<br />
#[[FAQ#Login| Login FAQ]]<br />
# If you have more questions, see [[FAQ| here]]</div>Ad876