ALERT: /v4scratch is down.

/v4scratch file system on the V4 cluster has crashed. Estimated time for availability is unknown. CAC staff will report more when possible.

May 2013 downtime tasks have been completed.

5/14/2013: Power outage affecting many CAC managed clusters and servers

A campus power outage at 7:20am has brought down many CAC managed clusters and servers. Staff are working to restore all as quickly as possible.

DOWNTIME: Wednesday, May 15th, 2013, 8am-5pm

REASONS: To apply May 2013 security patches (Windows and Linux).

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE?

• Windows Servers: ARECIBOWEB, ARECIBOFS, CATO, CATSLOGINxx, CBSUFTP, CBSUSRVxx, MOLMED, and WINLOGIN
• Windows Clusters: CATS and Red Cloud with MATLAB
• Linux Cluster's head nodes will be patched and rebooted: ASTRA, ATLAS, CLASSE, DSS, KINGLAB, LIPID, MARVIN, OSG (NYS1), SDS, SSG, TARDIS, V4, and WALLER
• Linux Servers: GRIDFTP1, GRIDFTP2, HLB, LINUXLOGIN2, MONITOR, MYPROXY, PETIE, PWSLINUX, RT, TALE-NT
• Database servers: CTCSQL, ARECIBOSQL, and CBSUSQLxx
• Website: http://www.cac.cornell.edu/

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, June 12th, 2013

DOWNTIME: Wednesday, April 10th, 2013, 8am-5pm

REASONS: To apply April 2013 security patches (Windows and Linux).

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE?

• Windows Servers: ARECIBOWEB, ARECIBOFS, CATO, CATSLOGINxx, CBSUFTP, CBSUSRVxx, MOLMED, and WINLOGIN
• Windows Clusters: CATS and Red Cloud with MATLAB
• Linux Cluster head nodes will be patched and rebooted: ASTRA, ATLAS, CLASSE, DSS, KINGLAB, LIPID, MARVIN, OSG (NYS1), SDS, SSG, TARDIS, V4, and WALLER
• Linux Servers: GRIDFTP1, GRIDFTP2, HLB, LINUXLOGIN2, MONITOR, MYPROXY, PETIE, PWSLINUX, RT, ZELLIG
• Database servers: CTCSQL, ARECIBOSQL, and CBSUSQLxx
• Website: http://www.cac.cornell.edu/

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, May 15th, 2013

CIT network maintenance on CAC subnets 4/2/2013 at 5am

WHEN? Tuesday, April 2nd, 2013, 5am - 7am

WHAT? CIT will be performing network maintenance on the CAC subnetslisted below. Remote connections (SSH and RDP) to servers with connections to these subnets will be lost. Users will need to reconnect after the maintenance is complete.

SUBNETS AFFECTED:

128.84.1.x
128.84.5.x
128.84.20.x
128.84.29.x
128.84.40.x
128.84.70.x
128.84.162.x

USERS AFFECTED: CBSU, CFG, KINGLAB, LIPID, PETIE, SDS, SSG, TARDIS

WHAT IS NOT AFFECTED: CAC subnet 128.84.3.x; connections to servers on this subnet will not be affected.

STATUS: Maintenance status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, April 10th, 2013

EMERGENCY NETWORK MAINTENANCE: Monday, March 11th, 2013, 3pm

REASONS: To correct a network routing issue.

WHO WILL BE AFFECTED? Users connected to CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE? Existing connections to CAC managed resources will be temporarily dropped while the switch reboots.

STATUS: Status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, April 10th, 2013

Storage01 is back online

Storage01 is online and all users home directories and CAC leased storage are available.

Please report any problems to help@cac.cornell.edu

Unplanned outage: Storage01 down - home directories affected

CAC Server storage01 is currently down. This server provides access to the majoority of user's home directores and CAC leased storage.

CAC Staff is addressing. Updates will be posted.

DOWNTIME: Wednesday, March 13th, 2013, 8am-5pm

REASONS: To apply March 2013 security patches (Windows and Linux).

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE?

• Windows Servers: WINLOGIN, CATO, CATSLOGINxx, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, MOLMED
• Windows Clusters: CATS, Red Cloud with MATLAB
• Linux Cluster head nodes running Centos 6 will be patched and rebooted on: ASTRA, DSS, TARDIS
• Linux Servers running Centos 6: TALE-NT
• Windows Database servers: CTCSQL, ARECIBOSQL, CBSUSQLX
• Website: http://www.cac.cornell.edu/

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, April 10th, 2013

Emergency Down Time on 2/14 Completed

All file systems should be back online. Report any problems to help@cac.cornell.edu. Thank you!

EMERGENCY DOWNTIME: Thursday, February 14, 2013, 1:00-1:20 pm

REASONS: To reboot a shared file server (storage02) to correct a problem encountered after yesterday's scheduled maintenance.

WHO WILL BE AFFECTED? Users belonging to the following groups/projects:

• Home directories for Tardis cluster users (cjf76_0001)
• Home directories for Lipid cluster users (gwf3_0001)
• Home directories of Alon Keinan's group (ak735_0001)
• Storage for frmaxfie_0001, har2001_0001, taryan_0001 that have access via Globus Connect/GridFTP.

QUESTIONS: http://www.cac.cornell.edu/help/

DOWNTIME: Wednesday, February 13th, 2013, 8am-5pm

REASONS: To apply February 2013 security patches (Windows and Linux).

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, March 13th, 2013

Several CAC systems are down due to a power outage late Wednesday night

CAC staff is working to bring up all affected servers as soon as possible.

We will update the status page periodically until all systems are online.

January 2013 security patching complete.

DOWNTIME: Wednesday, January 9th, 2013, 8am-5pm

REASONS: To apply January 2013 security patches (Windows and Linux).

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE?

• Windows Servers: WINLOGIN, CATO, CATSLOGINxx, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, MOLMED
• Windows Clusters: CATS, Red Cloud with MATLAB
• Linux Cluster head nodes running Centos 6 will be patched and rebooted on: ASTRA, DSS, TARDIS
• Linux Servers running Centos 6: TALE-NT, ZELLIG, HLB, CLOUD-LOGIN, RT
• Windows Database servers: CTCSQL, ARECIBOSQL, CBSUSQLX
• http://www.cac.cornell.edu/

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, February 13th, 2013

/v4scratch has been rebuilt and is available on v4.

The hardware housing /v4scratch suffered a catastrophic failure on Saturday, Dec 15th and all data on this scratch space was lost. The volume has been rebuilt and is available on V4. Please remember that this is a true scratch space so please do not store valuable data there permanently.

CAC Winter Break Schedule

CAC will observe the Cornell University winter break starting December 24, 2012, returning on Wednesday, January 2, 2013.

The operating status of all CAC resources will be monitored as usual during this period.

CAC staff will periodically monitor consulting questions throughout the holiday period. Critical user questions will be answered; non-critical user questions will be addressed beginning Wednesday, January 2, 2013.

Please submit any questions you may have via the help system: http://www.cac.cornell.edu/help/ or help@cac.cornell.edu.

Happy Holidays
Cornell University Center for Advanced Computing

/v4scratch back online

/v4scratch on v4linux cluster is now back online. Unfortunately, all data in this scratch space were lost due to hardware failure.

/v4scratch is unavailable

CAC staff is investigating.

December 2012 patching is complete.

All work scheduled for today has been completed. Please report any problems to help@cac.cornell.edu

DOWNTIME: Wednesday, December 12th, 2012, 8am-5pm

REASONS: To apply December 2012 security patches (Windows and Linux).

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE?

• Windows Servers: WINLOGIN, CATO, CATSLOGINxx, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, MOLMED
• Windows Clusters: CATS, Red Cloud with MATLAB
• Linux Cluster head nodes will be patched and rebooted on: ASTRA, ATLAS, CLASSE, DSS, KINGLAB, LIPID, MARVIN, OSG (NYS1), SDS, SSG, TARDIS, V4, WALLER
• Linux Servers: TALE-NT, PETIE, ZELLIG, HLB, MONITOR, PWSLINUX, CLOUD-LOGIN, RT, LINUXLOGIN2, GRIDFTP1, GRIDFTP2, MYPROXY
• Database servers: CTCSQL, ARECIBOSQL, CBSUSQLX
• http://www.cac.cornell.edu/

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, January 9th, 2013

Red Cloud with MATLAB disk maintenance planned

Downtime Nov 28, 2012 9-10AM Eastern Time.

Emergency Maintenance on Red Cloud with MATLAB is complete

Emergency Maintenance

Affects Red Cloud with MATLAB only. 9:30AM - Noon on Monday, Nov 26, 2012

November downtime and Unexpected Power Outage Recovery complete

Security and Critical patches have been applied to CAC servers per the announcement. All clusters and servers should now be available after unexpected campus wide power outage at approximately 10:45AM.

Please, as always, report problems and send inquiries to help@cac.cornell.edu.

Red Cloud Restored; All Running Instances Lost

Because of the unplanned power outage this morning, all Red Cloud nodes were rebooted. Consequently all running instances were lost. The service has been restored. Users should be able to start new instances and access EBS volumes again.

Power outage affecting many CAC managed clusters and servers

CAC staff are reacting to a power blip that brought down many clusters and servers. Stay tuned for more information.

Unplanned Power Outage

We recently had a ten second power outage, which was apparently campus wide.

The scheduled down time is still in progress. But now ALL servers and clusters have been power cycled since the power outage also hit the machine room.

DOWNTIME: Wednesday, November 14th, 2012, 8am-5pm

REASONS: To apply November 2012 security patches (Windows and Linux).

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE?

• Windows Servers: WINLOGIN, CATO, CATSLOGINxx, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, MOLMED
• Windows Clusters: CATS and Red Cloud with MATLAB
• Linux Servers: TALE-NT, PETIE, ZELLIG, HLB, MONITOR, PWSLINUX, SYSRH5, CONSULTRH5, LM-R900, LINUXSVC01, LINUXSVC02, PWSLINUX1, CLOUD-LOGIN, RT, LINUXLOGIN2, MYPROXY
• Database servers: CTCSQL, ARECIBOSQL, CBSUSQLX, etc
• http://www.cac.cornell.edu/

NOTE: Linux clusters ASTRA, ATLAS, CLASSE, DSS, KINGLAB, LIPID, MARVIN, OSG (NYS1), SDS, SSG, TARDIS, V4, and WALLER will NOT be affected.

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, December 12th, 2012

WORKSHOP ANNOUNCEMENT: CAC will present “Parallel Computing at TACC: Ranger to Stampede Transition” at Cornell University in Ithaca on December 11, 2012. For more informationand to register, visit https://www.xsede.org/web/xup/course-calendar/-/training/class/73. Register by Dec 3.

Storage02.cac.cornell.edu - emergency reboot Thursday, Nov 1st, 8am

Storage02 will be rebooted at 8am on Thursday, Nov 1st to improve performance.

The following users will be affected: Tardis cluster users Lipid cluster users Alon Keinan (group) leased storage users James VanEe's Gluster storage users The reboot should take less than 1 hour.

Emergency Maintenance complete. Red Cloud with MATLAB is back in projection.

Emergency Maintenance on Red Cloud with MATLAB file storage - Wednesday, October 24th 9am -noon

October downtime work complete.

Security and Critical patches have been applied to CAC servers per the downtime announcement. Please, as always, report problems and send inquiries to help@cac.cornell.edu.

DOWNTIME: Wednesday, October 10th, 2012, 8am-5pm

REASONS: To apply October 2012 security patches (Windows and Linux).

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE?

• Windows Servers: WINLOGIN, CATO, CATSLOGINxx, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, and MOLMED
• Windows Clusters: CATS and Red Cloud with MATLAB
• Linux Cluster head nodes will be patched and rebooted on: ASTRA, ATLAS, CLASSE, DSS, DSS1, KINGLAB, LIPID, MARVIN, OSG (NYS1), SDS, SSG, TARDIS, V4, and WALLER
• Linux Servers: PETIE, ZELLIG, HLB, MONITOR, PWSLINUX, CLOUD-LOGIN, RT, LINUXLOGIN2, DC, GRIDFTP1, GRIDFTP2, and MYPROXY
• Database servers: CTCSQL, ARECIBOSQL, and CBSUSQLX
• http://www.cac.cornell.edu/

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, November 14th, 2012

All work scheduled for the Sept. 21 downtime has been completed.

EMERGENCY DOWNTIME

Friday, 8-9AM ET. During the downtime we will replace a failed PERC battery in order to avoid the possibility of data loss. During this downtime you will not be able to submit batch jobs, and some web pages will fail.

September 2012 security patching is complete.

All work scheduled for the September 2012 downtime has been completed.

DOWNTIME: Wednesday, September 12th, 2012, 8am-5pm

REASONS: To apply September 2012 security patches (Windows and Linux).

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE?

• Windows Servers: (WINLOGIN, CATO, CATSLOGINxx, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, MOLMED)
• Windows Clusters: CATS, Red Cloud with MATLAB
• Linux Cluster: V4: All jobs will be cancelled and all compute nodes will be rebooted
• Linux Servers: (PETIE, ZELLIG, HLB, MONITOR, PWSLINUX, CLOUD-LOGIN, RT, LINUXLOGIN2, DC)
• Database servers (CTCSQL, ARECIBOSQL, CBSUSQLX, etc)
• http://www.cac.cornell.edu/

NOTE: Linux clusters ASTRA, ATLAS, CLASSE, DB, DSS, KINGLAB, LIPID, MARVIN, OSG (NYS1), SDS, SSG, TARDIS, and WALLER will NOT be affected.

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, October 12th, 2012

August 2012 security patching is complete.

All work scheduled for August 2012 downtime has been completed.

DOWNTIME: Wednesday, August 15th, 2012, 8am-noon

REASONS: To apply August 2012 security patches (Windows only this month).

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE?

• Windows Servers: (WINLOGIN, CATO, CATSLOGINxx, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, MOLMED)
• Windows Clusters: Red Cloud with MATLAB
• Database servers (CTCSQL, ARECIBOSQL, CBSUSQLX, etc).
• http://www.cac.cornell.edu/

NOTE: Linux stand-alone servers and Linux clusters ATLAS, CLASSE, DB, DSS, KINGLAB, LIPID, MARVIN, OSG (NYS1), SDS, SSG, TARDIS, V4, and WALLER will NOT be affected.

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, September 12th, 2012

V4 Scheduling Problems Resolved

Please send your questions and report any troubles to help@cac.cornell.edu

V4 Problems Under Investigation

Reported problems where V4 jobs are being blocked or deferred have been confirmed. CAC staff is investigating. Stay tuned for further updates.

Marvin lustre file system back online

The lustre file system /scratch2 is back online on the marvin cluster.

Marvin Cluster Lustre File System Down

the lustre file system on marvin cluster, /scratch2, is experiencing problems. CAC systems staff is working with Terascala Support to resolve the problem.

July 2012 security patching is complete.

All work scheduled for July 2012 downtime has been completed.

DOWNTIME: Wednesday, July 11th, 2012, 8am-5pm

REASONS: To apply July 2012 security patches (Windows and Linux).

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE?

• Windows Servers: (WINLOGIN, CATO, CATSLOGINxx, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, MOLMED)
• Windows Clusters: Red Cloud with MATLAB
• Linux Servers: (PETIE, ZELLIG, HLB, MONITOR, PWSLINUX, SYSRH5, CONSULTRH5, LM-R900, SCHEDULER, LINUXSVC01, LINUXSVC02, PWSLINUX1, CLOUD-LOGIN, RT, LINUXLOGIN2, MYPROXY, DC)
• Database servers (CTCSQL, ARECIBOSQL, CBSUSQLX, etc).
• http://www.cac.cornell.edu/

NOTE: Linux clusters ATLAS, CLASSE, DB, DSS, KINGLAB, LIPID, MARVIN, OSG (NYS1), SDS, SSG, TARDIS, V4, and WALLER will NOT be affected

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, August 15th, 2012

v4linuxlogin1.cac Emergency Power Cycle

v4linuxlogin1 was forced to have an emergency power cycle due to unresponsiveness. Please be reminded to NOT perform heavy I/O with heavy writes to your home directories NOR should one be running any jobs from the head node. Please submit jobs via the scheduler. If any further questions, please send message to: help@cac.cornell.edu.

June 2012 security patching is complete.

All work scheduled for June 2012 downtime has been completed.

DOWNTIME: Wednesday, June 13th, 2012, 8am-5pm

REASONS: To apply June 2012 security patches (Windows and Linux) and to migrate a few file systems (including home directories) to new disks.

WHAT WILL BE UNAVAILABLE?

• Windows Servers: (WINLOGIN, CATO, CATS cluster, CATSLOGINxx, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS)
• Windows Clusters: Red Cloud with MATLAB
• Linux Servers: (PETIE, ZELLIG, HLB, MONITOR, PWSLINUX, SYSRH5, CONSULTRH5, LM-R900, SCHEDULER, LINUXSVC01, LINUXSVC02, PWSLINUX1, CLOUD-LOGIN, RT, LINUXLOGIN2, MYPROXY, DC)
• Linux Clusters: V4, CLASSE, and DSS – all home directories on these clusters will be unavailable, all jobs will be canceled, head nodes will be patched and rebooted
• A few file systems served from storage01.cac.cornell.edu:
  • Sam Cartinhour leased storage (/home/fs02/sc167_0001 on v4linux)
  • CBSU home directories (/home/gfs08 on v4linux)
  • CBSU leased storage (mounted via CIFS)
  • Adam Siepel leased storage (/home/gfs05 on v4linux)
• Database servers (CTCSQL, ARECIBOSQL, CBSUSQLX, etc).
• http://www.cac.cornell.edu/

NOTE: Linux clusters OSG, ATLAS, DB, KINGLAB, LIPID, MARVIN, SDS, SSG, TARDIS, and WALLER will NOT be affected

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, July 11th, 2012

CAC SERVICES: Check out Red Cloud, a new Cornell cloud computing service with no charges for data movement in and out of the cloud. See http://www.cac.cornell.edu/redcloud.

May 2012 security patching is complete.

All work scheduled for May 2012 downtime has been completed.

DOWNTIME: Wednesday, May 9th, 2012 – between 8am-5pm each server will be unavailable for a brief time while it is patched and rebooted

REASONS: To apply May 2012 security patches (Windows only)

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE?

• Windows Servers: Windows Servers: (WINLOGIN, CATS cluster, CATSLOGINxx, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS)
• Database servers (CTCSQL, ARECIBOSQL, CBSUSQLX, etc).
• Red Cloud with MATLAB
• http://www.cac.cornell.edu/

NOTE: NOTE: The following will NOT be affected:

• Linux clusters (ATLAS, CLASSE, DB, DSS, KINGLAB, LIPID, MARVIN, NYS1, SDS, SSG, TARDIS, V4, WALLER)
• Home directories housed on CAC storage
• Red Cloud instances (these are normally maintained by the owner)

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, June 13th, 2012

CAC SERVICES: Check out Red Cloud, a new Cornell cloud computing service with no charges for data movement in and out of the cloud. See http://www.cac.cornell.edu/redcloud.

April 2012 Downtime Work Complete

All work scheduled for April 2012 downtime has been completed.

All work scheduled for April 2012 downtime has been completed. Please report any problems to help@cac.cornell.edu.

Progress continues to be made re. the DDN storage upgrade.

Drives continue to rebuild. We are anticipating users can log back in on Sat, 4/14/12. Check back for further status.

Progress continues to be made re. the DDN storage upgrade.

After several drive failures, drives are currently rebuilding. The ETA is still unknown. We will post another status approx. 3:00pm today.

Storage Upgrade Continues to be in Progress

We will post another status noon today.

Storage upgrade is still in progress

Storage upgrade is on-going

We are continuing work on the DDN upgrade and anticipate another 24 hours of unavailable services. Please see prior anouncement for list of CAC services that are unavailable.

Storage upgrade is in progress

Please see prior announcement for list of unavailable CAC services.

Storage upgrade is in progress. Please see prior announcement for list of unavailable CAC services.

CAC April Downtime: April 11th, 2012 (and possibly April 12, 2012

In addition to the DDN upgrade work listed below, April’s monthly security patches will be installed on CAC managed Windows servers including: • Windows Servers: (WINLOGIN, CATO, CATS cluster, CATSLOGINxx, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS) • Database servers (CTCSQL, ARECIBOSQL, CBSUSQLX, etc). • http://www.cac.cornell.edu/ These servers will be unavailable briefly around 10am while they reboot after the patches are applied. STATUS: Downtime status will be posted.

DDN upgrade scheduled for April 11 and April 12

REASONS: To increase data storage capacity

This work is scheduled for Wednesday April 11 and Thursday April 12!

To increase data storage capacity, CAC will double the number of disk slots available on the CAC Storage Farm. A two day outage is necessary to do the upgrade.

There is a risk that data will be lost as a result of the upgrade. If your data on CAC storage is your only copy, you may want to back it up prior to the upgrade. CAC has contracted with the vendor to perform this upgrade in a way that minimizes the risk of data loss, however there is still a risk.

BACKUP REMINDER: We want to remind you that your CAC storage is NOT backed up by default. Project owners can request regularly scheduled backups, which are administered through the CIT EZBackup service.

WHEN: April 11th, 8am to April 12th, 5pm

THE FOLLOWING STORAGE WILL BE UNAVAILABLE:

• All home directories for users on the following clusters: V4, CLASSE, DSS, LIPID, TARDIS
• Storage Leases: All, including the GlusterFS file system managed by James VanEe, and the xrootd file system served by OSG Storage Element
• EBS Volumes in Red Cloud: Red Cloud instances must detach all EBS volumes to avoid data corruption. Red Cloud instances can continue to run without EBS volumes attached and can re-attach EBS volumes after the upgrade
• Red Cloud with MATLAB

THE FOLLOWING COMPUTING WILL BE UNAVAILABLE:

• Red Cloud with MATLAB: computing will be unavailable
• V4, CLASSE, DSS, LIPID, TARDIS: During the upgrade, access to computing on these clusters will be limited. Your home directories will NOT be available
• OSG Storage Element: Storage Element will not be accessible. Jobs on the Compute Element not requiring access to the Storage Element should continue to run uninterrupted. The Compute Element should continue to accept and start new jobs

NOTE: The following will NOT be affected re storage or computing:

• Linux clusters (ATLAS, DB, KINGLAB, MARVIN, SDS, SSG, WALLER, OSG Compute Element)
• Local disks in Red Cloud instances

QUESTIONS: http://www.cac.cornell.edu/help/

March 2012 downtime work completed

All work scheduled for March 2012 downtime has been completed.

DOWNTIME: Wednesday, March 14th, 2012 8am-5pm

REASONS: To apply March 2012 security patches (Windows and Linux).

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE?

• Windows Servers: Windows Servers: (WINLOGIN, CATO, CATS cluster, CATSLOGINxx, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS)
• Linux Servers: (PETIE, ZELLIG, HLB, MONITOR, PWSLINUX, SYSRH5, CONSULTRH5, LM-R900, SCHEDULER, LINUXSVC01, LINUXSVC02, PWSLINUX1, CLOUD-LOGIN, RT, LINUXLOGIN2, MYPROXY, DC)
• Database servers (CTCSQL, ARECIBOSQL, CBSUSQLX, etc).
• Red Cloud with MATLAB
• http://www.cac.cornell.edu/

NOTE: Linux clusters OSG, ATLAS, CLASSE, DB, DSS, KINGLAB, LIPID, MARVIN, SDS, SSG, TARDIS, WALLER, and home directories housed on CAC storage will NOT be affected

NEW SOFTWARE: Intel 12.1 Compilers and Libraries
On March 14th, the default version of the Intel compilers will change from 11.1 to 12.1 on the login nodes linuxlogin1.cac.cornell.edu and linuxlogin2.cac.cornell.edu. Please refer to the V4 User Guide for instructions.

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, April 11th, 2012

CAC SERVICES: Check out Red Cloud, a new Cornell cloud computing service with no charges for data movement in and out of the cloud. See http://www.cac.cornell.edu/redcloud.

Red Cloud Lunch ‘n Learn seminar – March 28: CAC will be presenting a technical seminar on how to use Cornell’s Red Cloud, an Infrastructure as a Service that runs Eucalyptus, the open source, Amazon-compatible cloud computing platform. For more information, visit: http://www.cac.cornell.edu/about/news/120228.aspx.

Announcement:

CAC ANNOUNCEMENT: To increase data storage capacity, CAC will double the number of disk slots available on the CAC Storage Farm. A two day outage is necessary to do the upgrade.

There is a risk that data will be lost as a result of the upgrade. If your data on CAC storage is your only copy, you may want to back it up prior to the upgrade. CAC has contracted with the vendor to perform this upgrade in a way that minimizes the risk of data loss, however there is still a risk.

BACKUP REMINDER: We want to remind you that your CAC storage is NOT backed up by default. Project owners can request regularly scheduled backups, which are administered through the CIT EZBackup service.

WHEN: TBD. Our estimate is for two days in early April 2012. We will send another announcement when the actual dates are set.

THE FOLLOWING STORAGE WILL BE UNAVAILABLE:

• All home directories for users on the following clusters: V4, CLASSE, DSS, LIPID, TARDIS
• Storage Leases: All, including the GlusterFS file system managed by James VanEe, and the xrootd file system served by OSG Storage Element
• EBS Volumes in Red Cloud: Red Cloud instances must detach all EBS volumes to avoid data corruption. Red Cloud instances can continue to run without EBS volumes attached and can re-attach EBS volumes after the upgrade.
• Red Cloud with MATLAB

THE FOLLOWING COMPUTING WILL BE UNAVAILABLE:

• Red Cloud with MATLAB: computing will be unavailable.
• V4, CLASSE, DSS, LIPID, TARDIS: During the upgrade, access to computing on these clusters will be limited. Your home directories will NOT be available.
• OSG Storage Element: Storage Element will not be accessible. Jobs on the Compute Element not requiring access to the Storage Element should continue to run uninterrupted. The Compute Element should continue to accept and start new jobs.

NOTE: The following will NOT be affected re storage or computing:

• Linux clusters (ATLAS, DB, KINGLAB, MARVIN, SDS, SSG, WALLER, OSG Compute Element)
• Local disks in Red Cloud instances

QUESTIONS: http://www.cac.cornell.edu/help/

February 2012 downtime work completed

All work scheduled for February 2012 downtime has been completed.

DOWNTIME: Wednesday, February 15th, 2012 8am-5pm

REASONS: To apply February 2012 security patches (Windows and Linux).

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE?

• Windows and Linux Servers: (WINLOGIN, SCHEDULER (V4), CATO, CATS cluster, CATSLOGINxx, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS)
• Database servers (CTCSQL, ARECIBOSQL, CBSUSQLX, etc).
• Red Cloud with MATLAB
• http://www.cac.cornell.edu/

NOTE: The following will NOT be affected:

• Linux clusters (OSG, ATLAS, CLASSE, DB, DSS, KINGLAB, LIPID, MARVIN, SDS, SSG, TARDIS, WALLER)
• Home directories housed on CAC storage
• Red Cloud instances (these are normally maintained by the owner)

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, March 14th, 2012

NEW SOFTWARE: Intel 12.1 Compilers and Libraries
On March 14th, the default version of the Intel compilers will change from 11.1 to 12.1 on the login nodes linuxlogin1.cac.cornell.edu and linuxlogin2.cac.cornell.edu. We highly recommend recompiling and testing your codes prior to this change. Please refer to the V4 User Guide for instructions.

CAC SERVICES: Check out Red Cloud, a new Cornell cloud computing service with no charges for data movement in and out of the cloud. See http://www.cac.cornell.edu/redcloud.

STORAGE01 is back online and stable at this time.

Logins to v4linuxlogin1 and response times to commands like showq, ls, and showqlease should behave normally at this time.

UNPLANNED OUTAGE: CAC HOME DIRECTORIES SERVED FROM STORAGE01.CAC

STORAGE01.CAC.CORNELL.EDU is currently experiencing trouble. This is causing delayed responses when logging in to linuxlogin1 and to commands like df, ls, showq, showqlease, etc. CAC staff is addressing the issue at this time.

STORAGE01 is back online.

We apologize for the unplanned outage. Please report any problems to help@cac.cornell.edu.

UNPLANNED OUTAGE: CAC HOME DIRECTORIES SERVED FROM STORAGE01.CAC

CAC file server storage01.cac.cornell.edu crashed unexpectedly. CAC staff are working to bring it back online. Many CAC home directories are affected.

Jan 2012 downtime work completed

All work scheduled for Jan 2012 downtime has been completed.

DOWNTIME: Wednesday, January 11th, 2011 8am-5pm

REASONS: To apply January 2012 security patches (Windows and Linux).

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE?

• Windows and Linux Servers: (WINLOGIN, CATO, CATS cluster, CATSLOGINxx, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, RT, PETIE, ZELLIG, HLB, LINUXLOGIN2, DC)
• v4linuxlogin1 (alias linuxlogin1) - maintenance will include adding memory to improve performance
• V4 cluster - due to v4linuxlogin1's maintenance
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc).
• Red Cloud with MATLAB
• http://www.cac.cornell.edu/

NOTE: The following with NOT be affected:

• Linux clusters (OSG, ATLAS, CLASSE, DB, DSS, KINGLAB, LIPID, MARVIN, SDS, SSG, TARDIS, WALLER)
• Home directories housed on CAC storage, and Red Cloud instances.

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, February 15th, 2012

NEW SOFTWARE: Intel 12.1 Compilers and Libraries
The latest Intel Composer XE software, which includes C/C++ and Fortran compilers (icc, ifort), as well as MKL 10.3.6 and various other performance libraries, has recently been installed on v4.The new versions will become the defaults during the Februrary downtime. Therefore, you are encouraged to try compiling and linking with them now. To do this, you must first "source /opt/intel/intel-12.sh" (or .csh). This should be done not only on the login nodes at compile/link time, but also in your batch jobs at run time so that the correct dynamic libraries are found. More information is available.

DATA ANALYSIS ON RANGER AND LONGHORN TRAINING WORKSHOP: On Thursday and Friday, January 19-20, 2012, Cornell Center for Advanced Computing staff will present Data Analysis on Ranger and Longhorn, on the Ithaca, New York campus of Cornell University. After a review of getting started using Ranger's environment and batch system, the first day of the workshop will emphasize data movement and analysis tools. The second day will focus on scientific visualization and code improvement. You do not need to have an account on Ranger or Cornell Center for Advanced Computing HPC clusters to attend the workshop. There is no fee to attend.
Register
Questions
Agenda

CAC SERVICES: Check out Red Cloud, a new Cornell cloud computing service with no charges for data movement in and out of the cloud. See http://www.cac.cornell.edu/redcloud.

Today’s scheduled maintenance has been completed.

Please report problems to help@cac.cornell.edu or http://www.cac.cornell.edu/help

DOWNTIME: Wednesday, December 14th, 2011 8am-5pm

REASONS: To apply December 2011 security patches (Windows and Linux).

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE?

• Windows and Linux Servers: (WINLOGIN, CATSLOGINxx, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, RT, PETIE, ZELLIG, HLB, DSRG, LINUXLOGIN2, DC)
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc)
• Red Cloud with MATLAB
• http://www.cac.cornell.edu/

NOTE: The following with NOT be affected: Linux clusters (OSG, CLASSE, DSS, LIPID, KINGLAB, SDS, SSG, TARDIS, V4, WEBLAB, WALLER), home directories housed on CAC storage, and Red Cloud instances.

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, January 11th, 2012

NEW SERVICES: Check out Red Cloud, a new Cornell cloud computing service with no charges for data movement in and out of the cloud. See http://www.cac.cornell.edu/redcloud. Another service, Red Cloud with MATLAB, is also available for MATLAB users and educators.

/v4scratch's maintenance is complete.

The /v4scratch file system is available at this time.

/v4scratch file system: Emergency Maintenance today at 1pm (Monday, Nov 28)

CAC staff will be performing emergency maintenance on the /v4scratch file system today at 1pm. We anticipate the work to be completed in 5 hours. We will re-announce when it's available.

DOWNTIME: Wednesday, November 9th, 2011 8am-5pm

REASONS: To apply November 2011 security patches (Windows and Linux).

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE?

• Windows and Linux Servers: (WINLOGIN, CATSLOGINxx, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, RT, PETIE, ZELLIG, HLB, DSRG, LINUXLOGIN2, DC)
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc)
• http://www.cac.cornell.edu/

NOTE: Linux clusters will NOT be affected (OSG, CLASSE, DSS, LIPID, KINGLAB, SDS, SSG, TARDIS, V4, WEBLAB, WALLER). Home directories housed on CAC storage will not be affected.

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, December 14th, 2011

RT (CAC Help desk) is currently experiencing problems.

http://rt.cac.cornell.edu and http://www.cac.cornell.edu/help are currently unavailable. CAC staff is working on this.

Today’s scheduled maintenance has been completed.

Please report problems to help@cac.cornell.edu or http://www.cac.cornell.edu/help

DOWNTIME: Wednesday, October 12th, 2011 8am-5pm

REASONS: (1) Apply October 2011 security patches (Windows and Linux) (2) Install Intel 12 compilers on v4 (Intel 10 compilers will be removed)

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE? Servers listed will be unavailable for a brief period during the downtime window while they are patched and rebooted.

• Windows and Linux Servers: (WINLOGIN, CATSLOGINxx, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, RT, PETIE, ZELLIG, HLB, DSRG, LINUXLOGIN2, DC, CITRUS-GREENING)
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc)
• http://www.cac.cornell.edu/

NOTE: Linux clusters will not be affected (OSG, CLASSE, DSS, LIPID, KINGLAB, SDS, SSG, TARDIS, V4, WEBLAB, WALLER). Home directories housed on CAC storage will not be affected.

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, November 9th, 2011

48 Nodes added to the CAC's V4 cluster

48 additional nodes have been added to the V4 standard queue. These nodes are identical to all other v4 nodes (8core/16gb RAM).

Today’s scheduled maintenance has been completed.

Please report problems to help@cac.cornell.edu or http://www.cac.cornell.edu/help

DOWNTIME: Wednesday, September 14th, 2011 8am-5pm

REASON: To install September 2011 security patches (Windows and Linux).

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE?

• LOGIN servers (all sessions will be lost).
• Linux clusters (OSG, CLASSE, KINGLAB, DSS, WEBLAB, WALLER): Running jobs will be lost.
• Windows, Linux, and all virtual servers (CATSLOGINxx, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, PETIE, PWS, PWSLINUX, LINUXSVC0x, SEQPIPE, ZELLIG, etc).
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc).
• http://www.cac.cornell.edu/

NOTE: The SSG, LIPID, and TARDIS clusters and the SDS resource will not be affected.

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, October 12th, 2011

EMERGENCY MAINTENANCE: Monday, August 29th, 2011 8am-9am EDT

REASON: Reassign storage to a new file server.

WHO MAY BE AFFECTED? CAC users with home directories on CAC managed storage (storage01).

WHAT SHOULD USERS EXPECT? We don’t anticipate any service interruptions but there is a slight chance that the storage moves will require a reboot of storage01. Users should save files frequently.

STATUS: System status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

The scheduled maintenance for August 10th, 2011 has been completed (except for Kinglab cluster

Please report problems to www.cac.cornell.edu/help or help@cac.cornell.edu

Today's maintenance work has been completed except the Kinglab cluster rebuild. CAC staff is working to bring Kinglab online as soon as possible.

DOWNTIME: Wednesday, August 10th, 2011 8am-5pm

REASON: To install August 2011 security patches (Windows and Linux).

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE?

• LOGIN servers (all sessions will be lost).
• Linux clusters (CLASSE, TARDIS, KINGLAB, LIPID, DSS, WEBLAB, WALLER): A kernel patch will be applied to head nodes and compute nodes. Running jobs will be lost.
• Windows, Linux, and all virtual servers (CATSLOGINxx, MASSTOR, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, PETIE, PWS, PWSLINUX, LINUXSVC0x, etc).
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc).
• http://www.cac.cornell.edu/

NOTE: The SSG cluster and SDS resource will not be affected by this downtime.

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, September 14th, 2011

The scheduled maintenance for July 13th, 2011 has been completed

Please report problems to www.cac.cornell.edu/help or help@cac.cornell.edu

DOWNTIME: Wednesday, July 13th, 2011 8am-5pm

REASON: To install June 2011 security patches (Windows and Linux).

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE?

• LOGIN servers (all sessions will be lost).
• Linux clusters (CLASSE, TARDIS, KINGLAB, LIPID, DSS, WEBLAB, WALLER, OSG): head nodes will be patched, no reboot will be required. Running jobs should not be lost.
• Windows, Linux, and all virtual servers (CATSLOGINx, MASSTOR, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, PETIE, PWS, PWSLINUX, LINUXSVC0x, etc).
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc).
• http://www.cac.cornell.edu/

NOTE: The SSG cluster and SDS resource will not be affected by this downtime.

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, August 10th, 2011

NSUB problems resolved - July 6th, 1:15pm

The previously reported problems with NSUB on V4 have been resolved. We are sorry for any inconvenience this may have caused.

Unplanned Outage

Problems with nsub on V4

CAC staff is investigating reported problems with NSUB failures and jobs being "deferred". We will post updates as we know more.

Network outage June 25th between 4am and 7am

CIT will be performing network maintenance this Saturday between 4am and 7am. This work will affect CAC networks, specifically the network (128.84.3.x/24) which houses the v4 cluster, the CAC home directories, AMMA2, Arecibo servers and DDN, CATSLOGINn, Weblab, and more. Connectivity to resources on this network will be disrupted for approx. 30 minutes during the maintenance window.

The scheduled maintenance for June 15th, 2011 has been completed

Please report problems to www.cac.cornell.edu/help or help@cac.cornell.edu

Announcements:

/tmp cleanup policy revised: Beginning June 15, 2011, all files in /tmp on the compute nodes will be deleted prior to each job start.

/v4scratch: All users now have a subdirectory under /v4scratch, which is accessible from all v4 linux nodes. Data will not be erased from these directories, but there is no backup for this file system. You should use /v4scratch when you have large amounts of data that are needed for multiple jobs. Instead of copying your data from your home directory to /tmp on the v4 linux nodes each time you run a job, you should copy the data once to /v4scratch and then copy the data from /v4scratch to /tmp when you run your jobs. /v4scratch can also be used when your job fills /tmp and you need a place to offload temporary files to free up disk space on /tmp while your job is running. Data transfer will be faster between /v4scratch and /tmp than between /home and /tmp and will have significantly less impact on other users. We strongly encourage you to run from /tmp, not from /v4scratch and definitely not from /home.

DOWNTIME: Wednesday, June 15th, 2011 8am-5pm

REASON: To install June 2011 security patches (Windows and Linux).

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE?

• LOGIN servers (all sessions will be lost).
• Linux clusters (CLASSE, TARDIS, KINGLAB, LIPID, DSS, WEBLAB, WALLER, OSG): all compute nodes on all clusters will be rebooted to apply the security patches; all running jobs will be canceled.
• Windows, Linux, and all virtual servers (CATSLOGINx, MASSTOR, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, PETIE, PWS, PWSLINUX, LINUXSVC0x, etc).
• Database servers (CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, SCIDATA1, etc).
• http://www.cac.cornell.edu/

NOTE: The SSG, SDS, and OSG clusters are not affected by this downtime.

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, July 13th, 2011

The scheduled maintenance for May 11th, 2011 has been completed

Please report problems to www.cac.cornell.edu/help or help@cac.cornell.edu

DOWNTIME: Wednesday, May 11th, 2011 8am-5pm

REASON: To install May 2011 security patches (Windows and Linux).

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE?

• CAC Storage Leases, OSG Storage Elements, and all user home directories and project directories will be unavailable during a portion of the downtime while we update the file servers (storage01, storage02, storage03, mccoy, xrootd01).
• LOGIN servers (all sessions will be lost).
• Linux clusters (CLASSE, TARDIS, KINGLAB, LIPID, DSS, WEBLAB, WALLER, OSG): all compute nodes on all clusters will be rebooted to remount the home directories; all running jobs will be canceled.
• Windows, Linux, and all virtual servers (CATSLOGINx, MASSTOR, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, PETIE, PWS, PWSLINUX, LINUXSVC0x, etc).
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc).
• http://www.cac.cornell.edu/

NOTE: The SSG cluster and SDS resource will not be affected as these servers does not connect to the CAC storage.

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, June 15th, 2011

The scheduled maintenance for April 13th, 2011 has been completed

Please report problems to www.cac.cornell.edu/help or help@cac.cornell.edu

DOWNTIME: Wednesday, April 13th, 2011 8am-5pm

REASON: (1) To install April 2011 security patches (Windows and Linux), and (2) Network maintenance on the 128.84.3.0/24 subnet.

• Connections to the V4, CLASSE, WALLER, and WEBLAB clusters and all other servers on the 128.84.3.0/24 subnet will be interrupted between 8am and 8:30am for the network maintenance.
• Linux servers (PETIE, AMMA2, LM-R900, SYSRH5, CONSULTRH5, MONITOR, CUPS, RT, GRIDFTP1)
• Virtual Servers (DAS, DSRG, PWSLINUX, PWSLINUX1, LINUXLOGIN2, LITTLEJOHN-PH, DC, CITRUS-GREENING, MYPROXY)
• Windows servers (WINLOGIN, CATS cluster nodes and CATSLOGINn, MASSTOR, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, PWS, etc).
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc).
• http://www.cac.cornell.edu/

NOTE: The DSS, KINGLAB, LIPID, SSG, SDS, and TARDIS clusters will not be affected.

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, May 11th, 2011

The March 14th Emergency Maintenance is complete.

- Emergency Maintenance CAC DDN Storage Array - Monday, March 14th 10am -noon

Emergency outage to repair hardware on the CAC’s DDN storage unit.

WHO WILL BE AFFECTED? User home directories, project directories, CAC storage leases, OSG Storage Elements.

WHAT WILL BE UNAVAILABLE? CAC Storage Leases, OSG Storage Elements, and all user home directories and project directories will be unavailable during most of the downtime while we repair the DDN unit. All users should log off and close files before the downtime begins to prevent losing any data.

Note: The SSG and SDS resources will not be affected. The Arecibo DDN unit will not be affected.

Send questions/comments to help@cac.cornell.edu

The scheduled maintenance for March 9th, 2011 has been completed

Please report problems to www.cac.cornell.edu/help or help@cac.cornell.edu

DOWNTIME: Wednesday, March 9th, 2011 8am-5pm

REASON: (1) To install March 2011 security patches (Windows and Linux), and (2) To update the CAC accounting software on V4.

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE?

• LOGIN servers (all sessions will be lost).
• CAC Home Directory Servers (storage01, storage02, and storage03) – the servers housing user’s home directories will be rebooted; users will not be able to access their home directories while the servers reboot.
• Linux clusters (V4): unavailable while we install the new accounting software.
• Windows servers (CATS cluster nodes and CATSLOGINn, MASSTOR, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, PWS, etc).
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc).
• http://www.cac.cornell.edu/

NOTE: The CLASSE, DSS, KINGLAB, LIPID, PETIE, SSG, SDS, TARDIS, WALLER, and WEBLAB clusters should not be affected.

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, April 13th, 2011

WORKSHOP ANNOUNCEMENT: CAC will present 2-hour “Introduction to Using a CAC Linux Cluster” workshops on February 16 and March 31, 2011. Faculty, staff, students, and Corporate Program members are welcome. There is no fee to attend. Registration is required. Details, including registration, can be found at http://www.cac.cornell.edu/about/news/101202.aspx

All scheduled maintenance for Feb 9th, 2011 has been completed

Please report problems to www.cac.cornell.edu/help or help@cac.cornell.edu

DOWNTIME: Wednesday, February 9th, 2011 8am-5pm

REASON: To install February 2011 security patches (Windows and Linux).

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE?

• LOGIN servers (all sessions will be lost).
• Linux clusters (V4, CLASSE, TARDIS, KINGLAB, LIPID, DSS, WALLER): head nodes AND cluster nodes will be patched and rebooted; all running jobs will be cancelled.
• Windows and Linux servers (MCCOY, CATS cluster nodes and CATSLOGINn, MASSTOR, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, PETIE, PWS, PWSLINUX, LINUXSVC01, etc).
• WEBLAB(Hadoop) –all nodes will be patched and rebooted, all running jobs will be lost.
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc).
• http://www.cac.cornell.edu/
• NOTE: The SSG and SDS resources will not be affected.

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, March 9th, 2011

WORKSHOP ANNOUNCEMENT: CAC will present 2-hour “Introduction to Using a CAC Linux Cluster” workshops on February 16 and March 31, 2011. Faculty, staff, students, and Corporate Program members are welcome. There is no fee to attend. Registration is required. Details, including registration, can be found at http://www.cac.cornell.edu/about/news/101202.aspx

All scheduled maintenance for Jan 12th, 2011 has been completed.

Please report problems to www.cac.cornell.edu/help or help@cac.cornell.edu

DOWNTIME: Wednesday, January 12th, 2011 8am-5pm

REASON: To install January 2011 security patches (Windows and Linux).

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE?

• LOGIN servers (all sessions will be lost).
• Linux clusters (V4, CLASSE, TARDIS, KINGLAB, LIPID, DSS, WALLER): head nodes on these clusters will be patched and rebooted; all running jobs should be unaffected.
• Windows and Linux servers (MCCOY, CATSLOGIN, MASSTOR, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, PETIE, PWS, PWSLINUX, LINUXSVC01, etc).
• WEBLAB(Hadoop) –all nodes will be patched and rebooted, all running jobs will be lost.
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc).
• http://www.cac.cornell.edu/
• NOTE: The SSG and SDS resources will not be affected.

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, February 9th, 2011

WORKSHOP ANNOUNCEMENT: CAC will present 2-hour “Introduction to Using a CAC Linux Cluster” workshops on February 16 and March 31, 2011. Faculty, staff, students, and Corporate Program members are welcome. There is no fee to attend. Registration is required. Details, including registration, can be found at http://www.cac.cornell.edu/about/news/101202.aspx

Jan 8th and 9th Maintenance Completed.

The next scheduled maintenance is this Wednesday, Jan 12th, 2011 to apply security patches. Please submit questions to help@cac.cornell.edu or www.cac.cornell.edu/help.

Maintenance in progress.

Maintenance scheduled for Jan 8 and 9 is in progress. Access to many CAC home directories is not available at this time.

EMERGENCY OUTAGE: Saturday, Jan 8th, 2011 8pm – Sunday, Jan 9th, 2011 8am

REASONS: please see the CIT bulletin

WHO WILL BE AFFECTED? Users of CAC managed resources

WHAT WILL BE UNAVAILABLE? Users will be unable to connect to any CAC managed resources during the outage

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, January 12th, 2011

CAC Winter Break Schedule & Updates

Winter Break Schedule: CAC will observe Cornell’s winter break from 5 p.m. on Thursday, December 23, 2010, through 8 a.m. on Monday, January 3, 2011. The status of all resources will be monitored during the break. Please submit any questions you may have via the helpdesk. CAC staff will periodically monitor the consulting system throughout the holiday period and answer critical user questions. Non-critical questions will be addressed beginning Monday, 3 January 2011.

Power Outage January 8-9, 2011: Rhodes UPS Maintenance will occur on January 8-9, 2011. This activity will affect all CAC managed hardware in the Rhodes server farm. We will send more detailed information and timing early that week.

Interested in applying for a TeraGrid allocation? Please see complete information, including a proposal guideline telecon (January 3) and deadlines (January 15). In particular, those of you interested in the resources at TACC may access consulting services directly from CAC. We can also help guide you through the allocation process. If you’re interested in exploring other possibilities, please see Bridging to National Cyberinfrastructure.

All of the work scheduled for Dec 15th has been completed.

Please report any problems to www.cac.cornell.edu/help.

DOWNTIME: Wednesday, December 15th, 2010 8am-5pm

REASONS: (1) To install December 2010 security patches (Windows and Linux), and (2) To replace a failed controller on the DDN storage.

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall

WHAT WILL BE UNAVAILABLE? Updates will be applied affecting the following:

• CAC Storage Leases, OSG Storage Elements, and all user home directories and project directories will be unavailable during most of the downtime while we replace the DDN controller. All users should log off and close files before the downtime begins to prevent losing any data.
• LOGIN servers (all sessions will be lost).
• Linux clusters (CLASSE, TARDIS, KINGLAB, LIPID, DSS, WEBLAB(Hadoop)): all compute nodes on all clusters will be rebooted to remount the home directories; all running jobs will be canceled.
• Windows and Linux servers (MCCOY, CATSLOGIN, MASSTOR, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, PETIE, PWS, PWSLINUX, LINUXSVC01, etc).
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc).
• http://www.cac.cornell.edu/

NOTE: The SSG cluster and SDS resource will not be affected. These servers does not connect to the CAC storage.

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, January 12th, 2011

The problem has been resolved. All systems have been returned to service.

We are experiencing intermittent network outages to CAC managed clusters and servers. The problem is under investigation.

Status update: CAC managed storage, clusters, and servers are mostly back online.

The CAC managed storage, including all storage leases and users' home directories are back on-line, along with most of the CAC managed clusters, compute nodes, and servers.

The CAC managed storage, including all storage leases and users' home directories are back on-line, along with most of the CAC managed clusters, compute nodes, and servers.

CAC staff continues to work on the problems caused by the power outage. Several cluster nodes remain down. These will be brought back one by one as they are diagnosed and repaired.

The SSG cluster remains down at this time.

Please report any issues to www.cac.cornell.edu/help

UNPLANNED OUTAGE: CAC Storage (including all user's home directories)

The CAC DDN storage suffered a catastrophic failure as a result of last night's power outage. All CAC leased storage and CAC user's home directories are unavailable at this time. We have notified DDN for maintenance. We will post more information when we have any.

12.1.2010 10:30pm - Unplanned Power Outage in Rhodes Hall

Power is out to part of campus, including Rhodes Hall. This is affecting all CAC managed clusters and servers. We will post more information as soon as we can.

November 10th, 2010 scheduled work has been completed.

Please report any problems to www.cac.cornell.edu/help.

Monthly Downtime Announcement

DOWNTIME: Wednesday, November 10th, 2010 8am-5pm To install security patches and install Infiniband switch for accessing CAC storage.

DOWNTIME: Wednesday, November 10th, 2010 8am-5pm

REASONS: (1) To install November 2010 security patches (Windows), and (2) To install an Infiniband switch between the file servers and the DDN storage controllers.

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall

WHAT WILL BE UNAVAILABLE? Updates will be applied affecting the following:

NOTE: The SSG cluster and SDS server will not be affected. These servers do not connect to the CAC storage.

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, December 15th, 2010

October 2010 downtime work is complete.

Today's planned patching has been completed.

*** TODAY'S DOWNTIME, 10/13/10, IS COMPLETE, Please submit any problems or questions to http://www.cac.cornell.edu/help ***

DOWNTIME: Wednesday, October 13th, 2010 8am-5pm

REASONS: (1) To install October 2010 security patches (Windows and Linux), and (2) To install an Infiniband switch between the file servers and the DDN storage controllers. Update: item(2) has been moved to next month.

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall

WHAT WILL BE UNAVAILABLE? Updates will be applied affecting the following:

• Update: This item has been moved to next month.
  CAC Storage Leases, OSG Storage Elements,and all user home directories and project directories will be unavailable during most of the downtime while we install and connect to the Infiniband switch. The addition of the switch will allow CAC to add file servers as required. All users should log off and close files before the downtime begins to prevent losing any data.
• LOGIN servers (all sessions will be lost).
• Linux clusters (CLASSE, TARDIS, KINGLAB, LIPID, DSS, WEBLAB(Hadoop)): all compute nodes on all clusters will be rebooted to deploy the security kernel patches; all running jobs will be canceled.
• Windows and Linux servers (MCCOY, CATSLOGIN, MASSTOR, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, PETIE, PWS, PWSLINUX, LINUXSVC01, etc).
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc).
• http://www.cac.cornell.edu/

NOTE: The SSG cluster will not be affected. This cluster was patched last week and does not connect to the CAC storage.

STATUS: Downtime status will be posted.

QUESTIONS: http://www.cac.cornell.edu/help/

NEXT SCHEDULED DOWNTIME: Wednesday, November 10th, 2010

September 2010 server patching and maintenance is complete.

The work for this month's downtime has been completed. Please report problems to help@cac.cornell.edu or www.cac.cornell.edu/help

DOWNTIME: Wednesday, Sept 15th, 2010 8am – Friday, Sept 17th, 2010 5pm (we anticipate services will return within 24 hours but we are reserving the total time as a precaution)

REASONS: (1) To perform maintenance on the CAC DDN storage device and (2) To install September 2010 security patches (Windows and Linux).

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall

WHAT WILL BE UNAVAILABLE? Updates will be applied affecting the following:

• CAC Storage Leases, OSG Storage Elements, and ALL USER HOME DIRECTORIES and project directories will be unavailable while we perform maintenance on the DDN file systems. **All users should log off and close files before the downtime begins to prevent losing any data**.
• LOGIN servers (all sessions will be lost).
• Linux clusters (CLASSE, TARDIS, KINGLAB, DSS, WEBLAB(Hadoop)); all jobs writing to the home directories housed on CAC fileservers will be affected.
• Windows and Linux servers (MCCOY, CATSLOGIN, MASSTOR, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, PETIE, PWS, PWSLINUX, LINUXSVC01, etc).
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc).
• Web pages

STATUS: Downtime status will be posted

QUESTIONS

NEXT Scheduled DOWNTIME: Wednesday, October 13th, 2010

Monthly Downtime in progress today, Wed. Sept 15th

CAC staff is applying monthly security patches to CAC managed clusters and servers. This includes the storage that houses users home directories. We will post another status message when the maintenance is complete.

RESOLVED: slow response on CAC managed clusters

Response time has returned to normal on CAC managed clusters. Please remember that these clusters use a common home directory system. High I/O traffic to the home directories will hinder response time. Please use /tmp for your job I/O and write your data back to your home directory periodically during your job runs. This will help to maintain decent response for everyone.

slow response on CAC managed clusters

We are currently investigating the cause of the current sluggish response on CAC managed clusters (V4, TUC, Tardis, Kinglab, PETIE, etc).

DOWNTIME: DOWNTIME: Wednesday, August 11th , 2010 8am-5pm

REASONS: (1) To install August 2010 security patches (Windows and Linux), and (2) To increase the /opt partition on the v4 compute nodes to create room for additional third-party software package installations.

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE? Updates will be applied affecting the following:

• LOGIN servers (Winlogin.cac.cornell.edu and Linuxlogin.cac.cornell.edu) all sessions will be lost
• V4 cluster: ALL running jobs will be canceled to rebuild the compute nodes with a larger /opt partition
• Windows and Linux servers (CATSLOGINn, MASSTOR, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, PETIE, etc)
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc)
• Web pages

RESOLVED: Interactive Jobs are working correctly again.

We have resolved the problem with interactive jobs. Thank you for reporting the errors and thanks again for your patience.

Outage: Interactive jobs are not working

We are working on this and will update the status as soon as it is resolved. We apologize for the inconvenience.

Donwtime complete.

Downtime is now complete, with all systems updated and back in production. Configuring and testing the new scheduler took longer than anticipated; our apologies for the delay.

DOWNTIME: Wednesday, July 14th, 2010 8am-5pm

REASONS: (1) To install July 2010 security patches (Windows and Linux), and (2) To upgrade the controllers on the CAC DDN storage device.

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall

WHAT WILL BE UNAVAILABLE?Updates will be applied affecting the following:

• CAC Storage Leases, OSG Storage Elements, and all user home directories and project directories will be unavailable during most of the downtime while we update the controllers on the storage device. The upgrade will DOUBLE the network performance on the device.
  All users should log off and close files before the downtime begins to prevent losing any data.
• LOGIN servers (all sessions will be lost).
• Linux clusters (CLASSE, TARDIS, KINGLAB, DSS, WEBLAB(Hadoop)); all jobs writing to the home directories housed on CAC fileservers will be affected.
• Windows and Linux servers (MCCOY, CATSLOGIN, MASSTOR, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, PETIE, PWS, PWSLINUX, LINUXSVC01, etc).
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc).
• Web pages: http://www.cac.cornell.edu/

Scheduler and software will be updated July 14, 2010

The scheduler will be upgraded to a newer, different scheduler called Maui that has almost exactly the same features. The nsub, mjobctl, showq, and other commands should all work the same. CAC has used this scheduler on clusters other than V4 for some time and have tested the configuration we run on V4. Consultants will be on-hand after downtime to address any difficulties with the upgrade. Online documentation will reflect the upgraded scheduler before July 14.

The latest MATLAB, R2010a, has been installed as the default "matlab" on the v4 login nodes, and its runtimes are available on the compute nodes. You can still access the previous MATLAB and related tools by typing "r2007b" after their names, e.g., "matlabr2007b".

The Intel Compiler Suite Professional Edition 11.1 has been installed on the v4 login nodes, and its runtimes are available on the compute nodes. On July 14, this will become the default version for all Intel software, including ifort, icc, MKL, etc. Dynamically-linked codes may need to be recompiled unless a setup file is run first. For details on testing the 11.1 compilers and libraries prior to the switchover, please refer to Compiling and Linking Codes documentation.

DOWNTIME: Wednesday, May 12, 2010 8am-5pm

REASONS: To install May 2010 security patches (Windows and Linux)

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall

WHAT WILL BE UNAVAILABLE? Updates will be applied affecting the following:

• LOGIN servers (Winlogin.cac.cornell.edu and Linuxlogin.cac.cornell.edu) all sessions will be lost, running jobs should not be affected.
• Windows and Linux servers (MCCOY, CATSLOGINn, MASSTOR, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, PETIE, etc).
• SSG cluster: bios updates and security patches
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc).
• Web pages: http://www.cac.cornell.edu/ .

DOWNTIME: Wednesday, March 10, 2010 8am-5pm

REASONS: (1) To install March 2010 security patches (Windows and Linux), and (2) To upgrade the OS on storage01 (CAC home directory server).

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall

WHAT WILL BE UNAVAILABLE? Updates will be applied affecting the following:

• File Storage, including user home directories, will be unavailable during most of the downtime because we are updating the home directory server.
  **All users should log off and close files before the downtime begins.
• LOGIN servers (all sessions will be lost).
• Linux clusters (CLASSE, TARDIS, KINGLAB, DSS, WEBLAB(Hadoop)); jobs writing to the home directories housed on CAC fileservers will be affected.
• Windows cluster nodes (WINHPCnn).
• Windows and Linux servers (MCCOY, CATSLOGIN, MASSTOR, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, PETIE, PWS, PWSLINUX, LINUXSVC01, etc).
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc).
• http://www.cac.cornell.edu/

DOWNTIME: Wednesday, February 10th, 2010 8am-5pm

REASON: To install February 2010 security patches (Windows and Linux)

Please also see news announcements below

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall

WHAT WILL BE UNAVAILABLE?

• LOGIN servers (winlogin and linuxlogin, all sessions will be lost).
• Windows cluster nodes (WINHPCnn, TUCnnn), running jobs will be canceled.
• ALL CBSU cluster nodes.
• Windows and Linux servers (MASSTOR, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, PETIE, PWS, PWSLINUX, SCHEDULER, WEBLAB, etc).
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc).
• Web pages: http://www.cac.cornell.edu/ .

Seminar 2/26/2010: As part of the Cornell Center for Advanced Computing (CAC) Seminar Series, the CAC will be hosting Using Subversion RCS for Collaborative Code Development presented by Dr. Steven Lantz, February 26th, at noon in 655 Rhodes. RSVP required. For details see http://www.cac.cornell.edu/about/news/012710.aspx.

Out-of-band Microsoft Patch applied to Winlogin.cac.cornell.edu - 1/22/2010

MS10-002, the emergency security patch for Windows Internet Explorer, was applied to winlogin.cac.cornell.edu on Jan 22, 2010. As a result, all existing sessions were logged off and the machine was rebooted.

DOWNTIME: Wednesday, January 13, 2010 8am-5pm

REASON: To install January 2010 security patches (Windows and Linux)

** Please also see news announcements below **

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall

WHAT WILL BE UNAVAILABLE?

• LOGIN servers (winlogin and linuxlogin, all sessions will be lost).
• Windows cluster nodes (WINHPCnn, TUCnnn), running jobs will be canceled.
• Windows and Linux servers (MASSTOR, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, PETIE, PWS, PWSLINUX, SCHEDULER, etc).
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc).
• Web pages: http://www.cac.cornell.edu/ .

Other News from the CAC: Two new commands have been installed on linuxlogin. They were added in response to your requests. We welcome feedback on improving system usability.
showqlease summary information on the availability of queues and leases
showbalance remaining compute time balance

Seminar 1/28/2010: As part of the Cornell Center for Advanced Computing (CAC) Seminar Series, CAC will be hosting a Dell/VMware Virtual Desktop and Disaster Recovery Seminar with live demonstrations for New York State regional information technology professionals from academe and industry on Thursday, January 28 in room G10 of the Cornell University Biotechnology Building, Ithaca, NY. For more information see http://www.cac.cornell.edu/about/news/122209.aspx

Re: CAC Winter Break Schedule

CAC will observe the Cornell University winter break starting 24 December 2009, returning on Monday, 4 January 2010.

The operating status of all CAC resources will be monitored as usual during this period.

CAC staff will periodically monitor consulting questions throughout the holiday period. Critical user questions will be answered; non-critical user questions will be addressed beginning Monday, 4 January 2010.

Please submit any questions you may have via the help system: http://www.cac.cornell.edu/help/ .

Happy Holidays
Cornell Center for Advanced Computing

DOWNTIME: Wednesday December 16 8 AM through Thursday December 17 5 PM

REASON: New services for job submission and control will be installed and tested.

WHAT WILL BE UNAVAILABLE? MATLAB on the TeraGrid

DETAILS: The web services that enable job submission will be installed. We will be replace the authentication mechanism, requiring a total service shutdown, and which will break existing clients. On Thursday, after testing has been completed and the new service and client has been internally validated, we will send a new client (Littlejohn-0.2.0) to all current users. At that point please upgrade your system with the 0.2.0 client to resume work on the MATLAB on the TeraGrid resource.

We apologize for the inconvenience. Please let us know of any problems you have during or after upgrading your client.

DOWNTIME: Wednesday, December 9, 2009 8am-5pm

REASON: To install December 2009 security patches (Windows and Linux)

WHAT WILL BE UNAVAILABLE? Updates will be applied affecting the following:

• LOGIN servers (winlogin and linuxlogin, all sessions will be lost).
• Linux clusters - head nodes and cluster nodes (V4, TARDIS, KINGLAB, DSS, WALLER); running jobs will be canceled.
• Windows cluster nodes (WINHPCnn, TUCnnn), running jobs will be canceled.
• Windows and Linux servers (MASSTOR, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, PETIE, PWS, PWSLINUX, etc).
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc).
• Web pages: http://www.cac.cornell.edu/ .

DOWNTIME: Wednesday, Nov. 11th, 2009 8am-5pm

REASON: To install November 2009 security patches (Windows and Linux)

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall

WHAT WILL BE UNAVAILABLE? Updates will be applied affecting the following:

• LOGIN servers (all sessions will be lost).
• Windows cluster nodes (WINHPCnn, TUCnnn).
• Windows and Linux servers (CATSLOGIN, CATSFILES, MASSTOR, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, PETIE, PWS, PWSLINUX, etc).
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc).
• Web pages: http://www.cac.cornell.edu/ .

DOWNTIME: Wednesday, Oct 14, 2009 8am-5pm

REASONS:
1. To install October 2009 security patches (Windows and Linux)
2. To upgrade the scheduling software on V4
3. To move all users home directories to the CAC storage farm

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE? Updates will be applied affecting the following:

• File Storage, including user home directories, will be unavailable during most of the downtime because we are moving users home directories.
  **All users should log off and close files before the downtime begins.
• LOGIN servers (all sessions will be lost).
• Linux clusters (TARDIS, KINGLAB, DSS, WEBLAB(Hadoop)); jobs writing to the home directories housed on CAC fileservers will be affected.
• Windows cluster nodes (WINHPCnn, TUCnnn).
• Windows and Linux servers (CATSLOGIN, CATSFILES, MASSTOR, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, PETIE, PWS, PWSLINUX, etc).
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc).
• Web pages: http://www.cac.cornell.edu/

WHAT WILL YOU NEED TO DO AFTER THE DOWNTIME?

• Linux users: Continue to use ~ for your home directory.
• Windows users:
  • H: will be mapped to your new home directory during the downtime.
  • If you have any scripts or code that use pathnames, update them as follows:
  • Old UNC: \\cacfs01.cac.cornell.edu\<username>
  • New UNC: \\storage01.cac.cornell.edu\<username>

DOWNTIME: Wednesday, September 9, 2009 8am-5pm

REASON: To install September 2009 security patches (Windows and Linux)

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE? Updates will be applied affecting the following:

• File Storage, including user home directories, will be unavailable during the downtime.
  ** All users should log off and close files before the downtime begins.
• LOGIN servers (all sessions will be lost).
• Linux clusters (TARDIS, KINGLAB, DSS, WEBLAB(Hadoop)); jobs writing to the home directories housed on CAC fileservers will be affected.
• Windows cluster nodes (CATSnn, WINHPCnn, TUCnnn).
• Windows and Linux servers (MASSTOR, CBSUSRVxx, CBSUFTP, ARECIBOWEB, ARECIBOFS, PETIE, PWS, PWSLINUX, etc).
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc).
• Web pages: http://www.cac.cornell.edu/ .

DOWNTIME: Wednesday, August 12, 2009 8am-5pm

REASON: To install August 2009 security patches (Windows and Linux)

WHO WILL BE AFFECTED? Users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE? Updates will be applied affecting the following:

• File Storage, including user home directories, will be unavailable during the downtime.
• All users should log off and close files before the downtime begins.
• LOGIN servers (all sessions will be lost).
• Managed Linux clusters (TARDIS, KINGLAB, DSS, WEBLAB(Hadoop)); any jobs writing to the home directories housed on CAC fileservers will be affected.
• Managed Windows cluster nodes (CATSnn, WINHPCnn, TUCnnn).
• Managed Windows and Linux servers (MASSTOR, CBSUSRVx, CBSUFTP, ARECIBOWEB, ARECIBOFS, PETIE, PWSLINUX, etc).
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc).
• Web pages: http://www.cac.cornell.edu/ .

DOWNTIME: Wednesday, July 15, 2009 8am-5pm

REASON: To install July 2009 security patches (Windows and Linux)

WHO WILL BE AFFECTED?
Users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE?
Updates will be applied affecting the following:

• User’s home directories on CACFS01 and CACFS02 will be unavailable between 8am and 9am.
• User’s home directories on the DDN storage will be unavailable between 9am and 4pm.
• DDN Storage Users should unmount their DDN file systems prior to the downtime.
• LOGIN servers (all sessions will be lost).
• Managed Linux clusters (TARDIS, KINGLAB, DSS, WEBLAB(Hadoop)); any jobs writing to the home directories housed on CAC fileservers will be affected.
• Managed Windows cluster nodes (CATSnn, WINHPCnn, TUCnnn).
• Managed Windows and Linux servers (MASSTOR, CBSUSRVx, CBSUFTP, ARECIBOWEB, ARECIBOFS, PETIE, PWSLINUX, etc).
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc).
• Web pages: http://www.cac.cornell.edu/ .

Campus-wide power outage Saturday, June 6, 9:25pm

There was an unplanned campus-wide power outage at approximately 9:25pm Saturday, June 6, 2009. This outage caused many CAC managed Linux and Windows servers to experience brief interruptions while they rebooted. All services are back online. Please, as always, report any problems to help@cac.cornell.edu.

DOWNTIME: Wednesday, June 10th, 2009 8am-5pm

REASON: To install June 2009 security patches (Windows and Linux)

WHO WILL BE AFFECTED?

• All users of CAC managed resources housed in Rhodes Hall

WHAT WILL BE UNAVAILABLE?

Updates will be applied affecting the following (8am-5pm):

• ALL running jobs on the V4 clusters will be lost
• CAC file servers (home directories will be unavailable)
• LOGIN servers (all sessions will be lost)
• Managed Linux clusters (TARDIS, KINGLAB, DSS, WEBLAB(Hadoop)); any jobs writing to the home directories will be affected
• Managed Windows cluster nodes (CATSnn, CBSUMn, WINHPCnn)
• Managed Windows and Linux servers (MASSTOR, CBSUSRVx, CBSUFTP, ARECIBOWEB, ARECIBOFS, PETIE, PWSLINUX, etc)
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc)
• Web pages: http://www.cac.cornell.edu/

CAC Resource status can be found at: http://www.cac.cornell.edu/datafeed/status.aspx

DOWNTIME: Sunday, May 17th, 2009 6-7am

REASON: Cornell Information Technologies will be upgrading Ethernet switches in the server room.

WHO WILL BE AFFECTED? All users of CAC managed resources housed in Rhodes Hall.

WHAT WILL BE UNAVAILABLE? All remote connections to servers in the machine room will be disconnected. This includes SSH and Remote Desktop connections to the login nodes and any other servers that may allow remote connections. Running jobs will not be affected.

CAC Resource status can be found at: http://www.cac.cornell.edu/datafeed/status.aspx

CAC Scheduler back in service.

The patches have been applied successfully. Batch Jobs are being accepted at this time.

Brief scheduler downtime today

The scheduler will be paused on May 14 from 2:30-4:00pm to install patches. Running jobs will not be affected.

DOWNTIME: Wednesday, May 13th, 2009 8am-5pm

REASON: To install May 2009 security patches (Windows and Linux)

WHO WILL BE AFFECTED?

• All users of CAC managed resources housed in Rhodes Hall

WHAT WILL BE UNAVAILABLE?

Updates will be applied affecting the following (8am-5pm):

• ALL running jobs on the V4 clusters will be lost
• CAC file servers (home directories will be unavailable)
• LOGIN servers (all sessions will be lost)
• Managed Linux clusters (TARDIS, KINGLAB, DSS, WEBLAB(Hadoop)); any jobs writing to the home directories will be affected
• Managed Windows cluster nodes (CATSnn, CBSUMn, WINHPCnn)
• Managed Windows and Linux servers (MASSTOR, CBSUSRVx, CBSUFTP, ARECIBOWEB, ARECIBOFS, PETIE, PWSLINUX, etc)
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc)
• Web pages: http://www.cac.cornell.edu/

CAC Resource status can be found at: http://www.cac.cornell.edu/datafeed/status.aspx

CAC System Status

All systems have been returned to full service, including the linux login node (linuxlogin3.cac.cornell.edu) and batch job submission. Please let us know if you have any questions or problems.

Linux Login node is unavailable

linuxlogin3, the linux login node, is unavailable. We are working to resolve the problem as soon as possible.

Downtime Update

CAC Power and file system problems resolved

Details:

During Project Design and Construction's (PDC) monthly generator test in Rhodes Hall this morning, the wrong switch was accidentally turned off in the switchgear, affecting normal power in part of Rhodes Hall. As a result, some of the machines rebooted. This also resulted in the file system experiencing switch problems. These problems have now been resolved.

We will work with PDC to ensure that this does not happen again. We are sorry for any trouble or inconvenience this has caused.

Cornell Center for Advanced Computing
help@cac.cornell.edu

Unexpected downtime

REASON: Generator test

An uncontrolled shutdown of most systems occurred due to a generator test that did not progress as expected. All systems are expected to be back in normal operation by 11am.

DOWNTIME: Wednesday, April 15th, 2009 8am-5pm

REASON: To install April 2009 security patches (Windows and Linux)

WHO WILL BE AFFECTED?

• All users of CAC managed resources housed in Rhodes Hall

WHAT WILL BE UNAVAILABLE?

Updates will be applied to the CAC file servers (cacfs01, cacfs02) affecting the following (8am-5pm)

• ALL running jobs on all clusters (V4, DSS, WALLER, KINGLAB, WEBLAB(hadoop) and TARDIS) will be canceled.
• User’s home directories will be unavailable

After the file servers are completed, there will be brief interruptions while these servers are patched and rebooted (10am-5pm):

• Windows LOGIN servers (all sessions will be lost).
• Windows batch cluster nodes (v3, development). ALL running jobs on v3 and development will be cancelled.
• Managed Windows cluster nodes (CATSnn, CBSUMn, WINHPCnn).
• Managed Windows and Linux servers (MASSTOR, CBSUSRVx, CBSUFTP, ARECIBOWEB, ARECIBOFS, PETIE, PWSLINUX, etc).
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc).
• Web pages: http://www.cac.cornell.edu/ .

CAC Resource status can be found at: http://www.cac.cornell.edu/datafeed/status.aspx

OUTAGE: Thursday, March 19th, 2009 10-11am

REASON: To upgrade Moab from 5.2.2 to 5.3.

WHO WILL BE AFFECTED?

• All users who submit jobs using the Moab scheduler
  • Job submissions to V4, V4DEV, and V4-64G will be unavailable.
  • Anyone who has a running job during this time should check that it checkpointed properly.

DOWNTIME: Wednesday, March 11, 2009 8am-5pm

REASON: To install March 2009 security patches (Windows and Linux)

WHO WILL BE AFFECTED?

• All users who use CAC managed resources housed in Rhodes Hall, except for the following Linux clusters: USDA, Waller (Arecibo), Tardis, WebLab Hadoop, DSS, and Kinglab.

WHAT WILL BE UNAVAILABLE?

• Extended outages between 8am-5pm while configurations are modified:
  • Moab Scheduler for V4 and V4DEV jobs will be paused.
  • V4 Compute Nodes. All running jobs on V4 and V4DEV will be cancelled.


• Brief interruptions between 8am-noon while servers are patched and rebooted:
  • Windows LOGIN servers (all sessions will be lost).
  • Windows batch cluster nodes (v3, development). ALL running jobs on v3 and development will be cancelled.
  • Managed Windows cluster nodes (CATSnn, CBSUMn).
  • Managed Windows and Linux servers (CBSUSRVx, CBSUFTP, ARECIBOWEB, ARECIBOFS, PETIE, etc).
  • Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc).
  • Web pages: http://www.cac.cornell.edu/

CAC Resource status can be found at http://www.cac.cornell.edu/datafeed/status.aspx

DOWNTIME: Sunday, February 15th, 2009 6-8am

REASON: Cornell Information Technologies will upgrade network architecture to include redundant routers and more bandwidth capacity.

WHO WILL BE AFFECTED?

• All CAC managed servers in the 757 Rhodes Machine room.
• All CAC managed machines will experience network outages.
• All active connections to login nodes will be disconnected.

Note: This list pertains only to CAC resources. For a complete list, please see http://www.cit.cornell.edu/computer/news/articles.html#feb15down

DOWNTIME: Wednesday, February 11th,2009 8am-5pm

REASON: To install February 2009 security patches (Windows and Linux)

WHO WILL BE AFFECTED? All users who use CAC managed resources housed in Rhodes Hall, except for the following Linux clusters: USDA, Waller (Arecibo), Tardis, and Kinglab.

WHAT WILL BE UNAVAILABLE?

Extended outages between 8am-5pm while configurations are modified:

• Moab Scheduler for V4 and V4DEV jobs will be paused.
• V4 Compute Nodes. All running jobs on V4 and V4DEV will be cancelled.
• DSS cluster (networking maintenance and new nodes brought online).
• WEBLAB (HADOOP) cluster (configuration modifications).

Brief interruptions between 8am-noon while servers are patched and rebooted:

• Windows LOGIN servers (all sessions will be lost).
• Windows batch cluster nodes (v3, development). ALL running jobs on v3 and development will be cancelled.
• Managed Windows cluster nodes (CATSnn, CBSUMn).
• Managed Windows and Linux servers (CBSUSRVx, CBSUFTP, ARECIBOWEB, ARECIBOFS, PETIE, etc).
• Database servers (SCIDATA1, CTCSQL, SQLSRV02, ARECIBOSQL, CBSUSQLX, etc).
• Web pages: http://www.cac.cornell.edu/

CAC Resource status can be found at http://www.cac.cornell.edu/datafeed/status.aspx

DOWNTIME: Wednesday, January 14th, 9am-5pm

REASON: To install January 2009 security patches (Windows and Linux)

WHO WILL BE AFFECTED? All users who use CAC managed resources housed in Rhodes Hall, except for the following Linux clusters: v4, USDA, Waller, Weblab (hadoop), Tardis, and Kinglab.

WHAT WILL BE UNAVAILABLE?
NOTE: Most will see brief interruptions while servers are rebooted.

• ALL Windows LOGIN servers (all sessions will be lost)
• ALL Windows batch cluster nodes (v3, development). ALL running jobs on these nodes will be cancelled.
• ALL managed servers and Windows clusters (CBSUMn, CBSUFSRV1, CBSUSRVnn, CATSnn, ARECIBO, PETIE, etc)
• ALL Web pages will be unavailable: http://www.cac.cornell.edu/
• ALL Database servers (ctcsql, sqlsrv02, etc)
• Moab Scheduler for v4 and v4dev jobs (current running jobs on v4 and v4dev will not be affected)

CAC Resource status can be found at http://www.cac.cornell.edu/datafeed/status.aspx

DOWNTIME: Wednesday, November 12th, 8am-5pm

REASON: To install November 2008 security patches (Windows and Linux)

WHO WILL BE AFFECTED? All users who use CAC managed resources housed in Rhodes Hall, except for the following clusters: v4, USDA, Waller, Weblab (hadoop), and Kinglab.

WHAT WILL BE UNAVAILABLE?

• ALL Windows LOGIN servers (all sessions will be lost)
• ALL Windows batch cluster nodes (v3, development, cbsu1, cbsu2). ALL running batch jobs on these nodes will be cancelled
• ALL managed servers (CBSUFSRV1, CBSUSRVnn, SCIDATAn, CATSnn, PETIE, ARECIBO, DSSLINUXn, etc)
• ALL Web pages will be unavailable
• ALL Database servers (ctcsql, etc)
• Moab Scheduler for v4 and v4dev jobs (running jobs on v4 and v4dev will not be affected)

CAC Resource status can be found at http://www.cac.cornell.edu/datafeed/status.aspx

DOWNTIME: Wednesday, October 15th, 8am-5pm

REASON: To install October 2008 security patches (Windows and Linux)

WHO WILL BE AFFECTED? All users who use CAC managed resources housed in Rhodes Hall, except for the following clusters: v4, USDA, waller, and kinglab.

WHAT WILL BE UNAVAILABLE?

• ALL Windows LOGIN servers (all sessions will be lost)
• ALL Windows batch cluster nodes (v3, development, cbsu1, cbsu2). ALL running batch jobs on these nodes will be cancelled.
• ALL managed servers (CBSUFSRV1, CBSUSRVnn, SCIDATAn, CATSnn, WEBLAB (hadoop), ARECIBO, DSSLINUXn, etc)
• ALL Web pages will be unavailable: http://www.cac.cornell.edu/
• ALL Database servers (ctcsql, etc)

CAC Fileservers and batch queue will be removed from service

To Current CAC Researchers:

v3linux will be removed from service on September 17. If you are using this resource, please move your jobs to v4 now so your production computing can continue without disruption.

To Former CTC Researchers:

All old (formerly CTC) accounts will be deleted as of October 1st. In order to continue to use CAC resources you will need to be added to a faculty sponsored project. Project information is listed here: http://www.cac.cornell.edu/services/projects.aspx

To Both Groups:

After October 1, 2008, we will no longer be able to recover files from the old CAC fileservers (ctcfsrv6 through ctcfsrv13). The fileservers will be reformatted. Therefore, after 10/1, there will be no way to get any files from these machines or from backup. Please contact help if you have any questions about recovering or moving your files.

CAC Computing Update

Resource and fileserver update.

New computing resource - v4

A new computing resource will be available on September 10, 2008. v4 has 1024 cores (128 servers, each with 2 quad-cores/server), 2.5 GHz Intel Xeon, and 16GB RAM/server. It is running Red Hat Enterprise Linux Server release 5.1. There will be a charge for using the v4 compute nodes. v4 nodes are accessed using the Moab scheduler.

Login node name: linuxlogin3
Batch queue names:
v4 (up to 128 servers)
v4dev (up to 2 servers, 1 hour time limit)
v4-64g (4 servers with 64GB RAM/server)


Batch documentation
Software
CAC Resources

Training on v4 September 11 & 12

From noon until 1pm on Thursday September 11 and Friday September 12, we will hold training sessions on using Moab to compute on the v4 nodes. The sessions will include a demo, a review of the available documentation, and time to try the scheduler with help from CAC consultants. Join us in room 514 Rhodes Hall. Bring your laptop if you have one. Registration is not necessary.

v3linux will be removed from service on September 17

This will allow current users one week to migrate to the new v4 resource. Please make use of the transition time to move to v4 as soon as possible so your production computing can continue without disruption. The v3 Windows queue will continue to be available without charge for the short term; soon it too will be moved to the Moab scheduler, and the operating system will be updated to Windows server 2008 x64.

Fileserver update

After October 1, 2008, we will no longer be able to recover files from the old CAC fileservers (ctcfsrv6 through ctcfsrv13). Please contact help@cac.cornell.edu if you have any questions.

DOWNTIME: Wednesday, September 10th, 8am-5pm

REASON: To add the NEW v4 computing resource to production and to install September 2008 security patches (Linux and Windows)

WHO WILL BE AFFECTED? All users who use CAC, CBSU, WEBLAB, ARECIBO, DSS, ADMM and USDA resources housed in Rhodes Hall

WHAT WILL BE UNAVAILABLE?

• ALL LOGIN servers (all sessions will be lost)
• ALL batch cluster nodes (v3, development, v3Linux, v3linuxdev, admm, cbsu1, cbsu2, x64test, all running jobs will be cancelled).
• ALL managed servers
• ALL Web pages will be unavailable: http://www.cac.cornell.edu/
• ALL Fileservers containing home directories
• ALL Database servers

DOWNTIME: Wednesday, August 13th, 8am-5pm

REASON: To install August 2008 security patches (Linux and Windows)

WHO WILL BE AFFECTED? All users who use CAC, CBSU, WEBLAB, ARECIBO, DSS, ADMM and USDA resources housed in Rhodes Hall

WHAT WILL BE UNAVAILABLE?

• ALL LOGIN servers (all sessions will be lost)
• ALL batch cluster nodes (v3, development, v3Linux, v3linuxdev, admm, cbsu1, cbsu2, x64test, all running jobs will be cancelled).
• ALL managed servers
• ALL Web pages will be unavailable: http://www.cac.cornell.edu/
• ALL Fileservers containing home directories
• ALL Database servers

IMPORTANT DOWNTIME ANNOUNCEMENT

Downtime July 15-16 for power outage and to install security patches

WHEN? Tuesday, July 15th, 3pm – Wednesday, July 16th, 3pm (24 hours)

REASONS: Power outage for electrical repairs/upgrades in Rhodes Hall and to install July security patches (Linux and Windows)

WHO WILL BE AFFECTED? All users who use CAC, CBSU, WEBLAB, ARECIBO, DSS, ADMM and USDA resources housed in Rhodes Hall

WHAT WILL BE UNAVAILABLE?

• ALL LOGIN servers (all sessions will be lost)
• ALL batch cluster nodes (v3, development, v3Linux, v3linuxdev, admm, cbsu1, cbsu2, x64test, all running jobs will be cancelled).
• ALL Web pages will be unavailable: http://www.cac.cornell.edu/ .
• ALL Fileservers containing home directories
• ALL Database servers

UPDATE: Project information available

MANAGEMENT CONSOLE FOR PROJECT PIs

If you are the Principal Investigator for one or more projects, you can now view your project settings and view the list of userids you have enabled. By July 18th you will be able to update project settings from this page as well; that functionality will be added incrementally over the next ten days.

http://www.cac.cornell.edu/services/projects/manage.aspx

UPDATE: Cost-recovery model transition

Update on new project transition and deadlines for those not moving to new system

Notice to Researchers:

We are now in the first week of moving to a cost-recovery model. As you probably have noticed, we still have some work to do on systems and interfaces. We are still waiting for power installation in order to install the V4 linux cluster and four large shared memory nodes (16 core, 64 GB memory). Until the new systems are available, anyone who has created a project and added users can continue to use the V3 windows and linux clusters at no charge. We will begin charging for consulting, cluster maintenance and storage as of today, July 1. We are waiting final approval from the Cornell Division of Financial Affairs for the 2009 rates. As soon as the rates are approved we will update our web site. Any charges incurred starting July 1 will be billed at the new rates.

We will continue to improve the Web forms. In addition we plan to provide a web interface to allow PIs to view and manage their projects; we hope to have the first iteration next week, allowing the project PI to view project numbers, people added to the project(s), and limits set.

We welcome your feedback and will continue to make improvements. Regular updates will be provided as the changeover progresses.

If you are not moving to a new project:

• July 1: You will not be able to submit any jobs to the Velocity queues (v3,development, v3linux, or v3linuxdev). Your login will remain active for a short period of time (1-2 months) to allow you time to move your data to local storage.
• Sept 1: Your account will be disabled.
• Oct 1: We will no longer be able to recover files.

As always, please contact us if you have any questions or concerns.

DOWNTIME: Wednesday, June 11, 8am-5pm

REASON: Patches will be applied to both Linux and Windows servers.

WHO WILL BE AFFECTED?

All users:

• Fileservers containing home directories on H: and /home/nfs will be inaccessible.
• ctclogina, ctcloginb, ctclogind, linuxlogin1, linuxlogin2 - all login sessions will be lost.
• CAC batch machines (v3, development, v3linux, v3linuxdev, admm, cbsu1, cbsu2, x64test). All running jobs will be cancelled.
• CAC Web pages will be unavailable: http://www.cac.cornell.edu/ .

To minimize interruptions in accessing files, the fileservers containing home directories on H: and /home/nfs will be patched first.

Database: The following machines will be inaccessible:

• sqlsrv01, sqlsrv02, ctcsql, scidata1, scidata2, sonofstager

Specific groups: If you are using any of the following machines, you would make direct connections to them. These connections will be interrupted.

• Cbsusrv01-03
• Scicentr1-3
• Weblab
• Arecibo

Fileserver ctcfsrv13 experiencing hardware problems, only linux users affected.

ctcfsrv13 is experiencing hardware problems. This problem only affects linux users who have home folders on ctcfsrv13. Those users will be moved to other filesystems. If you think you are in this group, please contact us.

CAC downtime changes and updates

REASON: Patches and scheduler changes

Network maintenance 5/18/2008: Network connectivity to CAC managed servers will be lost from 7:00 - 7:15 AM. Running jobs will not be affected.

Velocity Scheduler (vsched) changes 5/14/2008: The changes will provide better access to the temporary smaller number of nodes and high demand.

• The v2linux and v2linuxdev machines will be removed to allow space and power for new hardware
• Cluster affiliation v3 will have the Max_Queue_Limit* raised from 244800 to 259200
• Cluster affiliation v3linux will have the Max_Queue_Limit* changed from unlimited to 230400

* Max_Queue_Limit is the maximum number of node-wallclock minutes for all jobs in a given queue belonging to a single user, both running and waiting.

Downtime changes 5/14/2008, 8am-5pm:

All users:

• Fileservers containing home directories on H: and /home/nfs will be inaccessible.
• ctclogina, ctcloginb, ctclogind, linuxlogin1, linuxlogin2 - all login sessions will be lost.
• CAC batch machines (v3, development, v3Linux, v3linuxdev, admm, cbsu1, cbsu2, x64test). All running jobs will be cancelled.
• CAC Web pages will be unavailable: http://www.cac.cornell.edu/ .

To minimize interruptions in accessing files, the fileservers containing home directories on H: and /home/nfs will be patched first.

Database: The following machines will be inaccessible:

• sqlsrv01, sqlsrv02, ctcsql, scidata1, scidata2, sonofstager

Specific groups: If you are using any of the following machines, you would make direct connections to them. These connections will be interrupted.

• Cbsusrv01-05
• Scicentr2&3
• Weblab
• Arecibo

Network maintenance 5/18/2008

Network connectivity to CAC managed servers will be lost on Sunday, May 18, 2008, 7:00 and 7:15 AM

CAC cluster changes

REASON: v2linux will be removed to be replaced with new cluster

The CAC is in the process of acquiring a new linux cluster, called v4linux. v4linux will consist of 128 M600 Dell Blades, each with 2 x Quad-core 2.5GHz Intel Processors (8 cores), 16GB RAM, and 1 x 120GB hard drive, running RHEL 5 linux. We plan to use the Moab scheduler on v4linux.

To make room for v4, the v2linux and v2linuxdev clusters must be removed first. In order to continue to offer both Windows and Linux clusters in the interim, we are temporarily moving half of the v3 nodes to Linux, configured as closely as possible to the current v2linux nodes. The v3 nodes will be returned to Windows as soon as v4linux comes online. Our goal is to have as many systems ready by July 1st as possible.

TIMEFRAME:

• Monday, April 28th - Friday, May 1: Half of the v3 nodes will move from Windows to Linux
• Monday, April 28th: v3linux nodes begin to be available for test and migration
• Wednesday, May 14th: v2linux and v2linuxdev will be removed from service

TEMPORARY CHANGES FOR V3 (WINDOWS) USAGE:

• Fewer nodes will be available in this pool. The nodes will be returned after v4linux is installed

TRANSITION CHANGES FOR V2 (LINUX) USAGE:

• Change the affiliation in your xml batch script from v2linux to v3linux
• Change the affiliation in your xml batch script from v2linuxdev to v3linuxdev
• The v3linux and v3linuxdev nodes have been configured as closely as possible to the v2linux nodes. The nodes have more memory and faster cpus than v2

DOWNTIME: Thursday, April 9, 8am-5pm

REASON: Patches will be applied to both Linux and Windows servers. Velocity Scheduler policy changes will be applied.

WHO WILL BE AFFECTED?

All users:

• Fileservers containing home directories on H: and /home/nfs will be inaccessible.
• ctclogina, ctcloginb, ctclogind, linuxlogin1, linuxlogin2 - all login sessions will be lost.
• CAC batch machines (v3, development, serial, v2Linux, v2linuxdev, admm, cbsu1, cbsu2, x64test). All running jobs will be cancelled.
• CAC Web pages will be unavailable: http://www.cac.cornell.edu/ .

To minimize interruptions in accessing files, the fileservers containing home directories on H: and /home/nfs will be patched first.

Velocity Scheduler (vsched) users:

• Cluster affiliations development and v2linuxdev will have the max_time_limit raised from 20 to 60 minutes
• Cluster affiliation v3 will have the max_time_limit raised from 24 to 48 hours
• Cluster affiliation v3 will have min_nodes reduced from 2 to 1
• The serial affiliation will be removed. All serial nodes will be merged into v3. Jobs submitted to serial will fail with the message Affiliation "serial" does not exist. If you have been using the serial affiliation, simply change it to v3.

Database: The following machines will be inaccessible:

• sqlsrv01, sqlsrv02, ctcsql, scidata1, scidata2, sonofstager

Specific groups: If you are using any of the following machines, you would make direct connections to them. These connections will be interrupted.

• Cbsusrv01-03
• Scicentr1-3
• Weblab
• Arecibo

DOWNTIME: Wednesday, March 12, 8am-5pm

REASON: Patches will be applied to both Linux and Windows servers.

WHO WILL BE AFFECTED?

All users:

• Fileservers containing home directories on H: and /home/nfs will be inaccessible.
• ctclogina, ctcloginb, ctclogind, linuxlogin1, linuxlogin2 - all login sessions will be lost.
• CAC batch machines (v3, development, serial, v2Linux, v2linuxdev, admm, cbsu1, cbsu2, x64test). All running jobs will be cancelled.
• CAC Web pages will be unavailable: http://www.cac.cornell.edu/ .

To minimize interruptions in accessing files, the fileservers containing home directories on H: and /home/nfs will be patched first.

Database: The following machines will be inaccessible:

• sqlsrv01, sqlsrv02, ctcsql, scidata1, scidata2, sonofstager

Specific groups: If you are using any of the following machines, you would make direct connections to them. These connections will be interrupted.

• Cbsusrv01-03
• Scicentr1-3
• Weblab
• Arecibo

DOWNTIME: Wednesday, February 13, 8am-5pm

REASON: Patches will be applied to both Linux and Windows servers.

WHO WILL BE AFFECTED?

All users:

• Fileservers containing home directories on H: and /home/nfs will be inaccessible.
• ctclogina, ctcloginb, ctclogind, linuxlogin1, linuxlogin2 - all login sessions will be lost.
• CAC batch machines (v3, development, serial, v2Linux, v2linuxdev, admm, cbsu1, cbsu2, x64test). All running jobs will be cancelled.
• CAC Web pages will be unavailable: http://www.cac.cornell.edu/ .

To minimize interruptions in accessing files, the fileservers containing home directories on H: and /home/nfs will be patched first.

Database: The following machines will be inaccessible:

• sqlsrv01, sqlsrv02, ctcsql, scidata1, scidata2, sonofstager

Specific groups: If you are using any of the following machines, you would make direct connections to them. These connections will be interrupted.

• Cbsusrv01-03
• Scicentr1-3
• Weblab
• Arecibo

DOWNTIME: Wednesday, January 9, 8am-5pm

REASON: Patches will be applied to both Linux and Windows servers.

WHO WILL BE AFFECTED?

All users:

• Fileservers containing home directories on H: and /home/nfs will be inaccessible.
• ctclogina, ctcloginb, ctclogind, linuxlogin1, linuxlogin2 - all login sessions will be lost.
• CAC batch machines (v3, development, serial, v2Linux, v2linuxdev, admm, cbsu1, cbsu2, x64test). All running jobs will be cancelled.
• CAC Web pages will be unavailable: http://www.cac.cornell.edu/ .

To minimize interruptions in accessing files, the fileservers containing home directories on H: and /home/nfs will be patched first.

Database: The following machines will be inaccessible:

• sqlsrv01, sqlsrv02, ctcsql, scidata1, scidata2, sonofstager

Specific groups: If you are using any of the following machines, you would make direct connections to them. These connections will be interrupted.

• Cbsusrv01-03
• Scicentr1-3
• Weblab
• Arecibo

ctcloginb is available again.

This machine is back online at this time.

Please report any problems with this machine.

ctcloginb is unavailable.

ctcloginb is experiencing hardware trouble. CAC systems staff is working on it now.

DOWNTIME: Wednesday, Dec 12th, 8am-5pm

REASON: Patches will be applied to both Linux and Windows servers.

• Fileservers containing home directories on H: and /home/nfs will be inaccessible.
• ctclogina, ctcloginb, ctclogind, linuxlogin1, linuxlogin2 - all login sessions will be lost.
• CAC batch machines (v3, development, serial, v2Linux, v2linuxdev, admm, cbsu1, cbsu2, x64test). All running jobs will be cancelled.
• CAC Web pages will be unavailable: http://www.cac.cornell.edu/.
• To minimize interruptions in accessing files, the fileservers containing home directories on H: and /home/nfs will be patched first.
• The following database machines will be inaccessible: sqlsrv01, sqlsrv02, ctcsql, scidata1, scidata2, sonofstager
• Specific groups: If you are using any of the following machines, you would make direct connections to them. These connections will be interrupted.
  • Cbsusrv01-03
  • Scicentr1-3
  • Weblab
  • Arecibo

DOWNTIME: Wednesday, Wed, Nov 14th., 8am-5pm

REASON: Patches will be applied to both Linux and Windows servers.

• Fileservers containing home directories on H: and /home/nfs will be inaccessible.
• ctclogina, ctcloginb, ctclogind, linuxlogin1, linuxlogin2 - all login sessions will be lost.
• CAC batch machines (v3, development, serial, v2Linux, v2linuxdev, admm, cbsu1, cbsu2, x64test). All running jobs will be cancelled.
• CAC Web pages will be unavailable: http://www.cac.cornell.edu/.
• To minimize interruptions in accessing files, the fileservers containing home directories on H: and /home/nfs will be patched first.
• The following database machines will be inaccessible: sqlsrv01, sqlsrv02, ctcsql, scidata1, scidata2, sonofstager
• Specific groups: If you are using any of the following machines, you would make direct connections to them. These connections will be interrupted.
  • Cbsusrv01-03
  • Scicentr1-3
  • Weblab
  • Arecibo

Sessions disconnected for more than 24 hours will be terminated

All sessions to login nodes that are disconnected for more than 24 hours will be terminated in order to improve system response time.