Puget Sound Technology Services

Orcas needs a reboot to properly configure VMWare tools, the misconfiguration is causing performance issues on Orcas.  All services will be available by 6am.

We discovered that the CRM email bounce-backs were not being processed, resulting in a very full mailbox. Nick renamed the inbox, essentially cleaning out the mailbox, thinking that maybe the box was too full to process the messages properly. This appears to have fixed the problem. We will look into whether these are being cleaned out properly when the bounceback is processed (or, being marked for deletion but then never deleted).

We also discovered that messages were being routed to the Junk email mailbox. CRM only processes bounce-backs found in the Inbox. We suspect that our use of Outlook to access the bounces email account has caused this to occur. We will look into using a tool other than Outlook for accessing this IMAP account.

The next production maintenance window is Sunday, November 29th, from 8:00 AM to noon.  From 8:00 AM to noon, all database services may be intermittently or entirely unavailable, including:  Cascade, FAMIS, Millennium, and Banner.

Updated 10/25/2009 @ noon — All production servers were patched successfully.  If you experience any issues with Puget Sound systems, please contact TechHelp at (253) 879-8585.

The next production maintenance window is Sunday, October 25th, from 8:00 AM to noon.

From 8:00 AM to Noon, all services may be intermittently or entirely unavailable, including:

• E-mail
• Personal and departmental file shares (on Alexandria and Merlin2)
• Database Applications:  Cascade, FAMIS, Millennium, and Banner

The following systems were patched today between 8am and noon:

• sthelens
• baker
• Tahoma
• userweb
• purgatory
• styx
• hades
• gehenna
• moodle
• quixote
• mysql
• oraweb
• orcas
• vmhost1
• Alexandria
• BE1
• BE2
• BE3
• CBORDFSS (VENDOR DC)
• CLTACC1
• CLTACC2
• CMS
• DHCP-1
• DM-1 to DM-8
• EPO
• FE1
• FE2
• ILLIAD
• INTCHK (VM)
• KEEPER
• KRONOS (VENDOR DC)
• LENEL (VENDOR Security)
• MAURY2
• MBS1
• MBS2
• MBS3
• MEDIA
• MERLIN2
• MICROS (VENDOR DC)
• MSPROJECT
• ODYSSEY (VENDOR DC)
• PROJECTS
• SANJUAN
• UPSCA03
• VASHON
• VERONICA
• VIAWARP (VM)
• VMMON (VM for now)
• WEBSERVER1
• WEBSERVER2
• CISCOWORKS (VM)

This month, on 10/15/2009 from 7am to 9pm we performed our monthly patch maintenance for PreProduction servers.  We patched the following systems:

• VMhost2
• Pilchuck
• Lummi
• Web
• moodle2
• kickstart
• Exdev
• Fidalgo
• Galaxy
• Ketron
• Portal
• VS0

As of 9:00pm 10/13/09 all users have complete access to their email once again.  Some messages into and out of these users accounts may be delayed as the system catches up.

Update as of 4:45PM 10/13/09

The outage of email for users EJ-LB continues as we run a database recovery process on the affected volume/users.  The process could take up to 18 hours, completing by mid-morning Wednesday (10am).

This post will be updated as more information is available.

Exchange Email is unavailable for users with usernames starting with EJ-LB.  Currently there is not a time estimate of when email for these users will be available again, but this post will be updated when the issue is corrected.

10/10/2009 The Aruba wirless controller was upgraded and additional new licensing applied starting at 4:30am.  The controller was rebooted at 5:20am.  The process is complete at 5:30am.  Monitoring until 6:00am shows all access points are functioning properly.

Update at 11:06 AM

All users now have access to their email and Calendars.  Please contact TechHelp if you continue to see problems.

Updated at 9:29 AM

Email is still unavailable for Students and Technology Services users starting with the letters L through S and Technology Services continues to work the problem.  We now expect the problem to be resolved by 12 noon, while doing everything we can to resolve the issue more quickly.  Watch this post for more information as it becomes available.

At this time some users are unable to access their email and calendars via Outlook, Webmail, and by other means.  Technology Services has identified the problem and expects resolution by 9:30am.

Update at 4:25PM

All users should have complete access to their email once again.  Some messages into and out of these users accounts may be delayed as the system catches up.

At this time Students and Technology Services staff members with usernames beginning in the letters L through S may not be able to access their email and calendars.  Technology Services is aware of the problem and is working quickly to resolve the issue.  We expect email to be accessible to all users by 6pm this evening and will update this post as conditions change.

On 9/18 from 5:30 to 6:30 AM we will migrate BJP and other applications on the Camano server to a new LUN.  This work is needed to decomission our older and soon to be unsupported disk array.

BJP and CRM applications will be offline for the duration of this change.

Updated 9/27/2009 at 2:30pm

All maintenance is completed as planned without any unexpected problems.  Most services were available by noon and all services are available as of 2:30pm.

The next production maintenance window is Sunday, September 27th, from 8:00 AM to 4:00 PM.

All services may be intermittently or entirely unavailable from 8:00 AM to 4:00 PM, including:

• E-mail
• Personal and departmental file shares (on Alexandria and Merlin2)
• Database Applications:  Cascade, FAMIS, Millennium, and Banner

On Sunday morning, 9/13/09, beginning at 7:00 PM, Telephone Services will be performing maintenance on off-campus telephone connections. The outage may interrupt all off-campus telephone calls for up to 30 minutes. This work will not affect telephone service within the college, itself, or cellular service.

Questions, contact phones@pugetsound.edu.

The training site for the new web site on BigTree, the new CMS in use by the university, has been set up.

AThe main web server was unavailable for approximately 20 minutes at 2:55 PM this afternoon, due to corrupted content caused by a faulty publishing process.  Content was restored at approximately 3:15 PM.

Do to the unscheduled extended outage this morning the following servers were affected during the outage (Camano, Grace, Crystal, Rainier, and Whidbey). This temporarily caused Cascade and other services to be unaccessible. The outage was extended from 06:30 am to 08:00 am, all services to these servers are back online and accessible now.

It will be necessary for us to reboot the core router between 5:30-6:00am on 08/27/2009.  This will cause the total network to be down during the time it takes the Core Router to reboot.  I estimate about 15 minutes.  This outage may also cause some servers to require a reboot to come back online.  Consequently various services may not be available due to this reboot of the core router.  We expect everything to be back to normal by 6:30am the very latest.  Thank you for your patience and understanding.

Changes to the provisioning system housed in the Cascade database will be installed on 8/20 starting at 5:30 a.m., and it shouldn’t take more than half an hour to finish.
These changes are to support Moodle integration to AD.
Problems with provisioning should be reported to Carol White immediately.

The network and wireless in Jones Hall will be down for upgrading of the network switches on Tuesday August 18th from 6:00pm until 11:00pm.  During this time there will be no email, network or internet availability of any kind until the job is complete.  Thank you for your patience and understanding.

PLANNED EVENT WINDOWS: 08/22/09 02:00 - 08/22/09 06:00 PDT

                       08/23/09 02:00 - 08/23/09 06:00 PDT

SUMMARY:

Our K20 internet circuit provider advised of planned maintenance involving the replacement of cards in some of their switches, on their network. We can expect multiple 50 second service interruptions during these 4 hour maintenance windows.

Updated at 8:30am, 8/14/2009:

Migration completed by 7:30am and post migration testing completed successfully.

On the morning of Friday August 14th from 6:30am to 7:30am some small, intermittent, outages might occur on the web server www2.ups.edu.  www2.ups.edu also houses sites such as library.ups.edu and blogs.ups.edu.  All these sites might be unavailable for 5-10 minutes during this hour of migration.

Update 8/20/09 @ 9:30am — The change encountered an issue and services were down until 9am.

On the morning of Thursday, August 20th from 7 - 8:30am users will not be able to access their quarantined spam messages on sophos.ups.edu.  During this time all email messages into and out of the University will continue to flow normally and any spam messages captured during this time may be retrieved after 9am.

This outage is necessary to make performance improvements to this very busy system.

The Millennium database was down this morning for a few hours because the archiver was stuck because the archive logs have not been getting purged like they should.
Space was cleared on the drive so archiving could resume and the database would be accessible again, and we will follow up on the log purging issue so this doesn’t happen again.

CRM, the system for sending campus-wide bulk messages, has experience an unexpected outage.
Technology Services is currently working to resolve the issue.

Service was restored to CRM at 3:59 pm.

Updated 8/7/09 at 10:45am:

Sophos systems are back up and operational.  Unfortunately the maintenance planned had to be backed out, so we will need to re-apply the changes at a future date/time.

Updated 8/7/09 at 10:00am:

We are still experiencing problems with the update.  Please stand by for more information.

Updated 8/7/09 at 8:15am:

We experienced some difficulty during maintenance and we expect the window to extend until 9:30am at this time.  All services should be up by 10am; sorry for the inconvenience.

On the morning of Friday, August 7th from 6:30 - 9:30am users will not be able to access their quarantined spam messages on sophos.ups.edu.  During this time all email messages into and out of the University will continue to flow normally and any spam messages captured during this time may be retrieved after 9am.

This outage is necessary to make performance improvements to this very busy system.

To prepare for our future e-mail server upgrade, it is necessary to update server addresses on Mobile Devices, such as the iPhone and Windows Mobile 6.0.

If your mobile device has issues syncing with the Puget Sound e-mail, change the server name to webmail.pugetsound.edu. If prompted, re-enter your domain password or accept any security certificate warnings.

For any other issues with the configuration or setup of your device, contact Tech Help at x8585 or techhelp@pugetsound.edu for further assistance.

The Puget Sound Webmail Interface was successfully upgraded as of 7:00am today.

Stay tuned for further email updates and improved features.

Renewed Cascade Web Certificate 7/23/2009

During routine service Sunday night, August 2nd, the voicemail server experienced an unexpected problem which resulted in an unscheduled service disruption from approximately 6:30-7:30pm. All functionality was restored around 7:30pm.

We have decommissioned all of the Cisco APs that were associated with the Tippingpoint servers.  Today the Tippingpoint server (Roving Planets) were shutdown and turned off.

Beginning at 6:00 p.m on Monday, July 27, 2009, Technology Services will deploy Microsoft Office 2007 to opt-in faculty and all staff computers. Office 2007 will be installed on all campus labs and e-classroom computers for the 2009-2010 academic year.

To facilitate a successful installation, staff and faculty opting in to the deployment should “Restart” their computers when leaving work on July 27. PCs should be left on and at the “CTRL+ALT+DEL” login screen. For more information on the deployment, including training and what to expect after deployment, visit our Office 2007 Web pages.

The following production servers were patched today, OS patches only:

Linux:

• orcas
• baker
• Tahoma
• purgatory
• styx
• hades
• gehenna
• moodle
• quixote
• mysql
• oraweb

And the following Windows servers:

• Alexandria
• BE1
• BE2
• BE3
• CLTACC1
• CLTACC2
• CMS
• DHCP-1
• EPO
• FE1
• FE2
• ILLIAD
• INTCHK
• KEEPER
• MAURY2
• MBS1
• MBS2
• MBS3
• MEDIA
• MERLIN2
• MSPROJECT
• PROJECTS
• SANJUAN
• UPSCA03
• VASHON
• VERONICA
• VIAWARP
• VMMON
• WEBSERVER1
• WEBSERVER2

Updated 7/26/09 @ 3:10PM

All changes were backed out and all mail delivery is back to normal by 3:00 PM.

Due to unforeseen issues with the Exchange 2007 interface upgrade today, the change is postponed.

Currently we are reverting back to the previous configuration, which should be complete by 3:00 PM today.  During the change and roll-back periods (8:00 AM to 3:00 PM Sunday, August, 26th) some mail sent from outside of the University may not have been accepted.  Any user sending email that failed will get an error message and will be able to re-send the message after 3:00 PM.

This post will be updated when the roll-back is complete.

We have resolved the issues related to the following non-production servers ( Pilchuck, Lummi, Kickstart and Moodle2). The servers are back on-line and operational once again.

We are currently experience patch related issues with the following non-production servers ( Pilchuck, Lummi, Kickstart and Moodle2). We are working to resolve these issues as quickly as possible. A notification will be sent out as soon as these issues have been resolved.

During this month’s pre-production patch window the following servers will be patched from 7am-9am on Thursday 7/16/09:

• lummi
• pilchuck
• kickstart
• moodle2
• exdev
• fidalgo
• galaxy
• ketron
• portal
• vso

\\merlin2\oiscommon\nssg\documentation\patchprocess\July_2009\July2009.docx

Some WebMail users are experiencing email outages.

[Update 06/30/2009 4:33 PM] Service has been fully restored.

See the post in the “Maintenance Windows” catagory for the public communication.

This month the Sophos PMX servers were patched with the latest Linux patches, and upgraded to the latest PureMessage version.

We also did the following non-recuring tasks:

• Upgraded Alexandria’s MD3000 Firmware
• Upgraded AntiVirus to EPO on Webserver1 and Webserver2
• Cleared stale VSS copies on SanJuan

Finally, all production Windows servers were patched:

• Alexandria
• BE1
• BE2
• BE3
• CBORDFSS
• CLTACC1
• CLTACC2
• CMS
• DHCP-1
• DM-1 to DM-8
• EPO
• FE1
• FE2
• ILLIAD
• INTCHK
• KEEPER
• KRONOS
• LENEL
• MAURY2
• MBS1
• MBS2
• MBS3
• MEDIA
• MERLIN2
• MICROS
• MSPROJECT
• ODYSSEY
• PROJECTS
• SANJUAN
• UPSCA03
• VASHON
• VERONICA
• VIAWARP
• VMMON
• WEBSERVER1
• WEBSERVER2

The next production maintenance window is Sunday, July 26th, from 8:00 AM to noon.

All services may be intermittently or entirely unavailable, including:

• E-mail
• Personal and departmental file shares (on Alexandria and Merlin2)
• Database Applications:  Cascade, FAMIS, Millennium, and Banner

This month’s outage was completed 1:30PM.  All services other than Cascade Web and Banner were available by noon as expected.

The next production maintenance window is Sunday, June 28th, from 8:00 AM to 4:00 PM.

From 8:00 AM to Noon, all services may be intermittently or entirely unavailable, including:

• E-mail
• Personal and departmental file shares (on Alexandria and Merlin2)
• Database Applications:  Cascade, FAMIS, Millennium, and Banner

Please note that Cascade Web and Banner will continue to be unavailable until 4:00 PM.

Pugetsound Webmail is unavailable. E-mail is available via desktop applications, such as Outlook. The issue is presently being investigated.

[Update 06/25/2009 6:06 PM] Service has been restored.

The Groups container was accidentally deleted in the OID test database today and had to be re-created based on priv data in the Summit database.
Here are the steps we had to do to recover everything:
1. Recreate the Groups container.
Export the Groups container definition (as an LDIF command) from the OID production instance, and import it into OID test. You’ll need an LDAP browser tool to do this, like JXplorer.
2. Recreate all the AD groups.
Set the status of all the pugetsound domain groups in the privilege table to PA and run
privcmd.resolve_pending_privdef on each one.
3. Recreate the members in the AD groups.
Set the status of all AD person_privilege records to PA and run privcmd.resolve_pending_privs.
4. Recreate the portal group container.
Export the portal.070109.134036.113589000 container definition (as an LDIF command) from the OID production instance, and import it into OID test. You’ll need an LDAP browser tool to do this, like JXplorer.
5. Recreate all the portal groups.
Set the status of all the portal groups in the privilege table to PA and run
privcmd.resolve_pending_privdef on each one.
6. Recreate the members in the portal groups.
Set the status of all portal person_privilege records to PA and run privcmd.resolve_pending_privs.
7. Address any unusual configuration issues.
ViewsFlash groups have a special setup, the administrator group is a member of the creator group, so that has to be done manually.

In preparation for the Office 2007 deployment in late July, Microsoft Office Compatibility files will be installed on campus computers this evening. In addition, we will be performing system scans over the next few days to ensure PCs meet the minimum requirements for Office 2007.

No action is needed from the campus community and the files will be automatically applied upon rebooting after June 23, 2009. The reboot may take a minute or two as files are applied.

To install the Microsoft Office Compatibility Pack on your home computer, follow the links on our download page.

Visit our Web site for information on today’s preparation phase or the coming Office 2007 project.

Only the following servers were patched due to limited staff availability:

• EXDEV
• FIDALGO
• GALAXY
• KETRON
• PORTAL
• VS0

During this month’s pre-production patch window the following servers will be patched from 7am-9am on Thursday 6/18/09:

• lummi
• pilchuck
• kickstart
• moodle2
• EXDEV
• FIDALGO
• GALAXY
• KETRON
• PORTAL
• VS0

\\merlin2\oiscommon\nssg\documentation\patchprocess\june2009\june2009.docx

A spammer gained control of a user account today when the account owner responded to a phishing message. The spammer sent a large volume of spam. This was noted at 9:30 PM and the user account was locked. TS staff will be checking with several recipient sites such as HotMail and Yahoo to make sure that the college has not been placed on blacklists, but you may experience E-mail delivery problems because of this incident.

Web forms was down this morning which included Cascade, Famis and Banner (a separate announcement has been posted to the Help Desk site for public viewing).
Patches were applied on the sanjuan server yesterday, and it wasn’t working after that. Paul fixed something in the configuration and bounced the server and now it’s working again.

[Update 6/1/2009 7:53 AM] Service has been restored.

Cascade, Banner and Famis web forms are experiencing errors. TS is working on this problem and will update with a resolution time when we know more.

See the post in the “Maintenance Windows” catagory for the public communication.  This post is to expand on what servers were patched, and basic changes.  All of the following patches were applied by noon.

The following windows servers were patched (Windows updates):

ALEXANDRIA

BE1

BE2

BE3

CLTACC1

CLTACC2

CMS

DHCP-1

DM-1 to DM-8

EPO

FE1

FE2

ILLIAD

INTCHK

KEEPER

MAURY2

MBS1

MBS2

MBS3

MEDIA

MERLIN2

PROJECTS

SANJUAN

VASHON

VERONICA

VIAWARP

VMMON

WEBSERVER1 (SP2 applied)

WEBSERVER2 (SP2 applied)

VMWare:

vmhost1

Linux (all available OS and Dell RAID FW/Drives as applicable):

orcas

tahoma

purgatory

styx

hades

gehenna

Additional changes on Sophos PMX servers:

Purgatory:

Turned Perc write cache on (write back), and changed linux readahead:

sudo blockdev –setra 8388608 /dev/sda

Rest:

policy on

adaptive read ahead

[Update 5/31/2009 1:00 PM] All service has been restored, and this maintenance window is complete.

[Update 5/31/2009 12:15 PM] Many services have been restored, with the exception of Database Applications:  Cascade, FAMIS, Millennium, and Banner. They are expected to be available by 4:00 PM as planned.

The next production maintenance window is Sunday, May 31, from 8:00 AM to 4:00 PM.

From 8:00 AM to Noon, all services may be intermittently or entirely unavailable, including:

• E-mail
• Personal and departmental file shares (on Alexandria and Merlin2)
• Database Applications:  Cascade, FAMIS, Millennium, and Banner

Please note that Database Applications will continue to be unavailable until 4:00 PM.

[Update 5/29/2009 7:34 AM] Power, network and wireless service have all been restored.

The power was unexpectedly cut to the fieldhouse this morning at 6:00 AM.  The campus network and wireless server are unavailable off-line until power is restored.