Difference between revisions of "MATLAB Parallel Server in Red Cloud"

From CAC Documentation wiki
Jump to navigation Jump to search
(created a table to display the Security Group entries; clarified the text in some places)
(wrote new section - Connect to Your MATLAB Cluster)
Line 7: Line 7:
 
In the Eucalyptus Management Console (for [https://euca4.cac.cornell.edu recdcloud-ith]), you need to create a Security Group for your MATLAB instances. Its purpose is to open up certain TCP ports so your client has proper access to your MDCS server(s).
 
In the Eucalyptus Management Console (for [https://euca4.cac.cornell.edu recdcloud-ith]), you need to create a Security Group for your MATLAB instances. Its purpose is to open up certain TCP ports so your client has proper access to your MDCS server(s).
  
Choose "Security Groups", then "Create a new Security Group", and set the following Rules. Note, to auto-fill <your client IP address>/32, you can select the "Use my IP address" link after entering either the Custom TCP port range or the standard SSH port.  
+
Choose "Security Groups", then "Create a new Security Group", and set the following Rules. Note, to auto-fill <your client IP address>/32, you can select the "Use my IP address" link after entering either the Custom TCP port range or the standard SSH port. You must click "Add Rule" at the bottom of the page after making each new entry.
  
 
{| border="1" cellspacing="0" cellpadding="5" align="center" style="text-align:left;"
 
{| border="1" cellspacing="0" cellpadding="5" align="center" style="text-align:left;"
Line 26: Line 26:
 
In general, if other users require access to your Red Cloud instance for their MDCS jobs--or if you will be accessing MDCS from multiple IP addresses--then it is probably best to add the extra IP addresses to the security group individually, in the same way as the first.
 
In general, if other users require access to your Red Cloud instance for their MDCS jobs--or if you will be accessing MDCS from multiple IP addresses--then it is probably best to add the extra IP addresses to the security group individually, in the same way as the first.
  
== Start Matlab Cluster ==
+
== Start the MATLAB Cluster ==
 
Currently only one-node clusters are supported. However, a single node can support many workers, up to the total number of cores that you assigned to your instance. Multi-node cluster support is forthcoming.
 
Currently only one-node clusters are supported. However, a single node can support many workers, up to the total number of cores that you assigned to your instance. Multi-node cluster support is forthcoming.
  
Line 33: Line 33:
 
* From the dashboard, click on the "Launch Instance" button.
 
* From the dashboard, click on the "Launch Instance" button.
 
* Select '''emi-b10e13b4''' on the "Image" screen. Click on the Next button.
 
* Select '''emi-b10e13b4''' on the "Image" screen. Click on the Next button.
* On the "Details" screen, select the desired instance type. '''One Matlab worker will be started per CPU in the instance.''' Click on the Next button.
+
* On the "Details" screen, select the desired instance type. '''One MATLAB worker will be started per CPU in the instance.''' Click on the Next button.
* On the "Security" screen, select the ssh keypair you'd like to use to access the instance via ssh (optional), and select your Matlab security group.
+
* On the "Security" screen, select the ssh keypair you'd like to use to access the instance via ssh (optional), and select your MATLAB security group.
 
* Click on "Launch Instance"
 
* Click on "Launch Instance"
* After the instance is running, the Matlab cluster should be reachable at the public IP address of the instance within 2 minutes.
+
* After the instance is running, the MATLAB cluster should be reachable at the public IP address of the instance within 2 minutes.
  
Note: after you finish using your Matlab cluster, remember to stop or terminate the instance on the Euca Console to stop charges against your Red Cloud subscription.
+
Note: after you finish using your MATLAB cluster, remember to stop or terminate the instance on the Euca Console to stop charges against your Red Cloud subscription.
 +
 
 +
== Connect to Your MATLAB Cluster ==
 +
Perform the following steps in your local MATLAB client:
 +
 
 +
* Open Parallel > Manage Cluster Profiles.
 +
* Choose Add > Custom > MATLAB Job Scheduler (MJS).
 +
* Click OK, then Edit. Only the top three entries are required.
 +
*# Description, '''Red Cloud''' (or other name that you choose)
 +
*# Host, '''128.84.8.XXX''' (where XXX matches the public address of your Red Cloud instance)
 +
*# Name, '''EC2_job_manager'''
 +
* Click Done, then Validate. As each stage completes, it should display a green circle with a check in it.
 +
 
 +
The most common reason for the first validation stage to fail is that the Euca Security Group is not allowing access from the IP address of your client. Log into the Euca Console ([https://euca4.cac.cornell.edu redcloud-ith]) using a web browser on the same machine as your MATLAB client. Choose the Security Group you created and try to add one of the above rules, ending with "Use my IP address". The answer should be "Rule already exists".  If not, then go ahead and add all three of the above rules using your current IP address. Wait a few minutes so the new Rules can take effect in your running instance (you may also want to restart the MATLAB client). Now your MATLAB cluster should be able to pass all the validation stages from your client.

Revision as of 12:04, 13 September 2016

Assumptions

  • The MATLAB R2016a client, including the Parallel Computing Toolbox (PCT), is installed on your local workstation.
  • First time login has been completed.
  • Create key pair has been completed.

Create a Security Group

In the Eucalyptus Management Console (for recdcloud-ith), you need to create a Security Group for your MATLAB instances. Its purpose is to open up certain TCP ports so your client has proper access to your MDCS server(s).

Choose "Security Groups", then "Create a new Security Group", and set the following Rules. Note, to auto-fill <your client IP address>/32, you can select the "Use my IP address" link after entering either the Custom TCP port range or the standard SSH port. You must click "Add Rule" at the bottom of the page after making each new entry.

Protocol Port range Allow traffic from IP address
Custom TCP 14000 - 15000 <your client IP address>/32
Custom TCP 27000 - 28000 <your client IP address>/32
SSH (for terminal access) 22 - 22 <your client IP address>/32


If you wish, you can set the security group to allow access from a broader range of IP addresses. For example, if the allowed IP range is set to 128.84.0.0/16, access is permitted from any Cornell subnet. However, you should be aware that in this case, any Cornell user who has knowledge of the IP address and the cluster name will be able to submit MDCS jobs to your Red Cloud instance(s).

In general, if other users require access to your Red Cloud instance for their MDCS jobs--or if you will be accessing MDCS from multiple IP addresses--then it is probably best to add the extra IP addresses to the security group individually, in the same way as the first.

Start the MATLAB Cluster

Currently only one-node clusters are supported. However, a single node can support many workers, up to the total number of cores that you assigned to your instance. Multi-node cluster support is forthcoming.

In the Euca Console (redcloud-ith):

  • From the dashboard, click on the "Launch Instance" button.
  • Select emi-b10e13b4 on the "Image" screen. Click on the Next button.
  • On the "Details" screen, select the desired instance type. One MATLAB worker will be started per CPU in the instance. Click on the Next button.
  • On the "Security" screen, select the ssh keypair you'd like to use to access the instance via ssh (optional), and select your MATLAB security group.
  • Click on "Launch Instance"
  • After the instance is running, the MATLAB cluster should be reachable at the public IP address of the instance within 2 minutes.

Note: after you finish using your MATLAB cluster, remember to stop or terminate the instance on the Euca Console to stop charges against your Red Cloud subscription.

Connect to Your MATLAB Cluster

Perform the following steps in your local MATLAB client:

  • Open Parallel > Manage Cluster Profiles.
  • Choose Add > Custom > MATLAB Job Scheduler (MJS).
  • Click OK, then Edit. Only the top three entries are required.
    1. Description, Red Cloud (or other name that you choose)
    2. Host, 128.84.8.XXX (where XXX matches the public address of your Red Cloud instance)
    3. Name, EC2_job_manager
  • Click Done, then Validate. As each stage completes, it should display a green circle with a check in it.

The most common reason for the first validation stage to fail is that the Euca Security Group is not allowing access from the IP address of your client. Log into the Euca Console (redcloud-ith) using a web browser on the same machine as your MATLAB client. Choose the Security Group you created and try to add one of the above rules, ending with "Use my IP address". The answer should be "Rule already exists". If not, then go ahead and add all three of the above rules using your current IP address. Wait a few minutes so the new Rules can take effect in your running instance (you may also want to restart the MATLAB client). Now your MATLAB cluster should be able to pass all the validation stages from your client.