Puget Sound Technology Services

The next production maintenance window is on Sunday, March 28th, from 8:00 AM to noon.

From 8:00 AM to 9:30 AM all services may be off-line intermittently due to network upgrades, including:

• E-mail
• Personal and departmental file shares (on Alexandria and Merlin2)
• Web services (including www.pugetsound.edu and cs.pugetsound.edu)
• Database Applications: Cascade, FAMIS, Millennium, and Banner

Please note that Database Applications may continue to be unavailable until noon.

CascadeDev (ridge) data was refreshed yesterday. Here are the details:

Age of data copied: 3/14/2010 10:00 pm
Refresh started: 3/15/2010 1 pm
Refresh completed: 3/16/2010 1:30 am

Cascade Web has been intermittently unavailable this morning. Technology Services is researching the problem and will resolve just as soon as possible.

Implemented Data Guard database on Famis successfully, but failed on Basis due to 10.2.0.3 bug. Need to patch Basis home and retry.

A few wireless connection issues have been observed.  We observed an issue that has been resolved.  If you experienced some occational problems connecting to the wireless, that problem should now be resolved.

Cascade Release 10.4 is planned for April 14, 2010.  This release will include the following features and fixes:

• Ability for Cascade Web users to link from the Cascade Web Course Offerings page to textbook information as required by regulation.
• Ability to charge course fees by grade option as required to support university fee structure.
• Ability for students to charge textbooks to their student accounts as required by university policy.
• Transfer of current and disable future use of Residential Deposits to align with university fee structure.
• Fix Online Degree Progress Report error for associated course.
• Convert username to a consistent case to support application needs.

[Updated 10:46 AM] Telephone service was restored at 10:45 AM.

[Updated 9:08 AM] Our telephone provider, Integra Telecom, has informed us that this outage is part of a Tacoma-wide outage on parts of Click!’s network. Click! does not yet have a timeline for restoration of service.

Telephone calls into and out of campus have not been working since about 5:00 AM this morning. This appears to be a telephone provider problem. Our telephone service provider (Integra Telecom) has been contacted, and they are working to restore service as quickly as possible.

Update at 11:30 AM on 3/16/10:   Most issues have subsided, however daily digest e-mails still have some problems.    Some users do not see all received spam e-mail in their daily quarantine message. If you experience this issue, please visit the quarantine to see all your messages. If you have other questions or concerns please contact the Service Desk at X8585.

Update at 3:30PM on 3/5/10:  We continue to see scattered issues with PureMessage, however we believe the primary issue with the system is corrected.  While we expect new messages to be handled correctly in the system, messages received prior to today may need special treatment.  If you need access to a message unavailable in your quarantine, please contact the Service Desk at x8585 or servicedesk@pugetsound.edu.

Users are experiencing the following issue with our Sophos PureMessage system (spam protection):

• Some users are unable to release or delete some or all messages in their quarantine
• Some users cannot see all of their quarantined messages
• Some users receive a “Daily Quarantined” email that contains messages they cannot see or manipulate in their quarantine

Technology services is aware of these issues, and is working hard with the vendor to resolve them.

If you experience any of these symptoms and need access to quarantined messages, please contact the Service Desk at x8585 or servicedesk@pugetsound.edu for assistance.

Many campus e-mail accounts experienced login problems and messages being returned as undeliverable between 10:50 and 11:25 AM this morning. The problem has been resolved and root cause is being investigated.

Created Data Guard databases on Cascade, Banner, Millennium

Intermittent and widespread authentication problems began around 12:00 PM. Many systems are impacted including wireless and wired access, Cascade applications, Moodle, and computer logins. We are working to resolved the problem.

Due to a problem with Sophos PureMessage, some email messages from off-campus sources could have been delayed this weekend. The problem was corrected Sunday morning, and these messages have been delivered.

CascadeDev (ridge) data was refreshed this week. Here are the details:

Age of data copied: 2/17/2010 10:00 pm
Refresh started: 2/18/2010 2 pm
Refresh completed: 2/19/2010

Maintenance completed at 11:30am with no unexpected outages.  Cascade was back online by 10:30am.

Between 8:00 AM and noon on Sunday, February 21st outages of Cascade are expected as we perform important infrastructure maintenance.

Maintenance completed at 11:30am with no unexpected outages.  Cascade was back online by 10:30am.

Between 8:00 AM and noon on Sunday, February 21st intermittent outages of Cascade are expected as we perform important maintenance on our Fiber Channel Switches.  Other outages are at increased risk at this time, but no other outages are expected due to redundant hardware.

A failed disk on WEBSERVER1 has caused the pages of some faculty web pages to become unavailable. We are working on this problem right now, and the pages should be available this evening by approximately 6 PM.

Update at 2/18/2010 at 9:45am:  All services are back online.  Pilchuck and Lummi took a little longer than expected, but all other servers were up by 9am.

On Thursday February 18th from 7am-9am all pre-production servers are to be patched.  This means that all test systems may be intermittently or entirely unavailable during this time.  Most users will not be impacted, as most users do not have access to preproduction servers.  Specifically, the following servers will be impacted:

• vmhost2
• lummi
• pilchuck
• moodle2
• kickstart
• hope
• decatur
• shaw
• squaxin
• mysqldev
• EXDEV
• FIDALGO
• GALAXY
• KETRON
• PORTAL
• VS0

Update at 5:20PM:  All services were back online by 4pm.

Update at 1:30PM:  Most services are online as of 12:30PM, however all database services, Cascade, and the ASUPS web site continue to be unavailable.

The next production maintenance window is Sunday, February 28th, from 8:00 AM to 4:00 PM.

From 8:00 AM to Noon, all services may be intermittently or entirely unavailable, including:

• E-mail
• Personal and departmental file shares (on Alexandria and Merlin2)
• Database Applications: Cascade, FAMIS, Millennium, and Banner

Please note that Database Applications will continue to be unavailable until 4:00 PM.

At 5:30 PM today Technology Services staff will restart the university’s core network infrastructure. There will be a brief (< 1 min.) outage on all services, including email, file services, and web service. Cascade web will be unavailable for 20 minutes.

Thank you for your patience.

Mike Carroll at Perpetual Technologies made some changes to our production report server this morning. It was done during our regular 5:30 a.m. maintenance window:

1) Cleaned up logs and cache directory
2) Created a new reporting server with security enabled by default: UPS_CASCADE_PROD
3) Enabled tracing on existing reporting server: rep_SanJuan_UpsForms

The changes were complete by 6 a.m. and the following apps were tested to verify the report server was working:

1) Direct report server call
2) Report server call from a Cascade form
3) Report server call from Famis
4) Report server call from BJP

Cascade Web and Cascade Web Forms went down at around 6:00 pm.

Services were restored at around 6:40 pm after bouncing Application Server & Forms Server.

The middle tier on San Juan will be restarted after a configuration update on 2/5/2010 between 5:00 am and 5:30 am PST during the regular maintenance window. The configuration  update will help in troubleshooting.

Over the weekend of the 6th and 7th of February 2010, work in development will be conducted on allowing SSO to interface with Cascade Web Forms. This will cause intermittent report outages for all applications.

A new feature of the Account Management application allows supervisors to “unlock” an employee’s work session from a supervisor’s management page. Lockable systems are Cascade Web and Cascade database.

The following components were installed 1/27/2010 5:30-6am:
COM.INFORMATION_SYSTEM.TAB
COM.INFORMATION_SYSTEM.CON
COM.INFORMATION_SYSTEM.DML
COM.CAMPUS_ROLE.TAB
COM.CAMPUS_ROLE.CON
COM.PERSON_CAMPUS_ROLE.DRP
COM.PERSON_CAMPUS_ROLE.TAB
COM.PERSON_CAMPUS_ROLE.CON
COM.ROLE_LOCK_VIEW.VW
COM.ROLE_LOCK_VIEW.SYN
COM.ROLE_LOCK_VIEW.GRT
COM.ROLE_LOCK_API.PKS
COM.ROLE_LOCK_API.PKB
COM.ROLE_LOCK_API.SYN
COM.ROLE_LOCK_API.GRT
COM.PRIVCMD.PKS
COM.PRIVCMD.PKB
COM.DAY_TEXT.FNC
COM.DAY_TEXT.SYN
COM.DAY_TEXT.GRT
COM.DATE_TEXT.FNC
COM.DATE_TEXT.SYN
COM.DATE_TEXT.GRT
WEB.PRIVWEB.PKS
WEB.PRIVWEB.PKB
WEB.WEBSEC.PKB
WEB.CASCADE.PKB
COM.CASCADE_IS_LOCKED.FNC
COM.CASCADE_IS_LOCKED.SYN
COM.CASCADE_IS_LOCKED.GRT
COM.LOGON_DENY_LOCKED_USER.TRG
WEB.SUPERVISE_ACCESS.PKS
WEB.SUPERVISE_ACCESS.PKB
WEB.SUPERVISE_ACCESS.SYN
WEB.SUPERVISE_ACCESS.GRT
WEB.WEB_APP_MENU.DML

CascadeTest (summit) data was refreshed this weekend. Here are the details:

Age of data copied: 1/17/2010 10:00 pm
Refresh started: 1/18/2010 5 pm
Refresh completed: 1/18/2010 8:15 pm

On Monday, 1/18/2010, emergency maintenance will be performed on the wireless and Internet connections for all academic buildings (K20 network). Between 6:00 PM and 10:00 PM, expect a 40-70 minute period of down time.

Our residential Internet connections are experiencing intermittent outages due to a vendor outage in the Seattle area.  They are working to replace the equipment but we can expect intermittent outages for a short time.

[Update 1/14/10 4:00 PM] Service has been fully restored.

Updated 01/09/2010 at 12:26PM

Blackboard is now back online and functioning properly since approximately noon.

Blackboard will be unavailable from 8:00AM to 4:00PM on Saturday January 9th, 2010.  This outage is required to improve the reliability of the Blackboard system.  Please contact the ServiceDesk at x8585 with any questions or concerns on this outage.

CRM will be down for the next 15 minutes for a switch reboot.

The BJP proccessor in the development environment was stopped inadvertently. In restarting it, the Production database was mistakenly specified, so production batch jobs were started on a server that could not run them.

The servers have been restarted correctly, and batch jobs are being re-queued to run properly.

Due to conflicts with the start of the semester, there will be no scheduled maintenance during our normal window (last Sunday of every month).  We will resume scheduled maintenance in February with our quarterly window from 8:00am - 4:00pm on Sunday February 28th.

This morning at approxmiately 8:20 PST, the old Solaris listhost server was shutdown and powered off.  All mailing lists were migrated yesterday to a new listhost.ups.edu running Linux on our VMware cluster.

All sites were moved successfully by noon.

On Monday December 21st all Math and CS websites, including faculty sites, will be moved to an upgraded server (from orion to userweb).  The move will take place between 8 AM and 12 noon.  All effected were contacted last week.

Update at 4:30 PM — All testing was successful, migration completed.

Updated at 3:30 PM — All changes have been made and lists should be fully available once again, but we are continuing to test and verify all is working properly.

On Tuesday, December 22nd 2009 between noon and 5:00 PM, email lists hosted on listhost.ups.edu will be moved to a new server.  During this time new email sent to email lists will be held until the service migration is complete.  If possible, please do not post messages to email lists during this time.

Refer back to this post on Tuesday for updated information.

Mike Carroll installed new certificates for cascade.ups.edu and chinook.ups.edu (sso and dip).
The new cascade certificate will expire 10/13/2011.

Cascade Web and Cascade Web Forms went down approximately 2:30 p.m. this afternoon. We are investigating the problem and will have it back up as soon as we can.

Services were restored at about 4 p.m. Causes are unknown at this time but we will continue to investigate what happened.

Just added a new policy to the Internet Authentication Service - Remote Access Policies to not allow users of the “novpn” AD group to use VPN.

All maintenance work was completed by 8:30 am this morning.  Please contact TechHelp (x8585) if you experience any problems as a result of this change.

On Thursday December 10th from 7am-9am all preproduction servers are to be patched.  This means that all test systems may be intermittently or entirely unavailable during this time.  Most users will not be impacted, as most users do not have access to preproduction servers.  Specifically, the following servers will be impacted:

• lummi
• pilchuck
• moodle2
• kickstart
• sage
• EXDEV
• FIDALGO
• GALAXY
• KETRON
• PORTAL
• VS0

12/21 - Odyssey database purge

12/22 - Micros version Upgrade 0530

1/4 - Odyssey version Upgrade TBD

Today at approximately 4pm the server orion.ups.edu (also known as tock.ups.edu) crashed and rebooted.  Everything seems to have came back properly, but unexpected behavior may be experienced.  Please contact TechHelp at x8585 if you experience problems with Orion.

Updated 12/20/09 @ 11:15 AM

As of 11:00 AM all services are back online, earlier than expected.  Please contact TechHelp at x8585 if you see any problems with university services.

The next production maintenance window is Sunday, December 20, from 8:00 AM to 5:00 PM.

From 8:00 AM to Noon, all services may be intermittently or entirely unavailable, including:

• E-mail
• Personal and departmental file shares (on Alexandria and Merlin2)
• Database Applications: Cascade, FAMIS, Millennium, and Banner

Additionally file servers Alexandria and Merlin2 may continue to be unavailable until 5:00 PM.

BJP was modified to error appropriately when a runrep command fails. BJP was stopped and restarted at 12:30 pm for the new code to take effect.

As of noon today all mail list services are back online.  Please contact TechHelp at x8585 if you continue to experience problems with lists.

Early this morning our mail list server had a problem and is not working properly.  We are working on resolving the issue, but it is not likely to be resolved before 5pm today.  We will update this post as we have more information.

The ResNet internet connection was down from 12:00 midnight to 3:00 AM this morning. Service has been restored.

Today between 5pm and 6pm wiki.ups.edu was moved over to userweb.pugetsound.edu.  No problems were encountered, but please let us know if you encounter issues.

Nick Webb x3185

Both Crm11ip instance and Crm App had to be restarted due to the exceeded processes on the databasee. The problem was caused by the returning mails sent last week.

Installed Oracle Critical Patch, CpuOct2009, on Cascade, Banner, Famis, Millennium, and Rman databases.

Orcas needs a reboot to properly configure VMWare tools, the misconfiguration is causing performance issues on Orcas.  All services will be available by 6am.

BJP was migrated off of Camano, onto Orcas

We discovered that the CRM email bounce-backs were not being processed, resulting in a very full mailbox. Nick renamed the inbox, essentially cleaning out the mailbox, thinking that maybe the box was too full to process the messages properly. This appears to have fixed the problem. We will look into whether these are being cleaned out properly when the bounceback is processed (or, being marked for deletion but then never deleted).

We also discovered that messages were being routed to the Junk email mailbox. CRM only processes bounce-backs found in the Inbox. We suspect that our use of Outlook to access the bounces email account has caused this to occur. We will look into using a tool other than Outlook for accessing this IMAP account.