Puget Sound Technology Services

A few users were experiencing email outages.

[Update 06/30/2009] Service has been fully restored.

See the post in the “Maintenance Windows” catagory for the public communication.

This month the Sophos PMX servers were patched with the latest Linux patches, and upgraded to the latest PureMessage version.

We also did the following non-recuring tasks:

• Upgraded Alexandria’s MD3000 Firmware
• Upgraded AntiVirus to EPO on Webserver1 and Webserver2
• Cleared stale VSS copies on SanJuan

Finally, all production Windows servers were patched:

• Alexandria
• BE1
• BE2
• BE3
• CBORDFSS
• CLTACC1
• CLTACC2
• CMS
• DHCP-1
• DM-1 to DM-8
• EPO
• FE1
• FE2
• ILLIAD
• INTCHK
• KEEPER
• KRONOS
• LENEL
• MAURY2
• MBS1
• MBS2
• MBS3
• MEDIA
• MERLIN2
• MICROS
• MSPROJECT
• ODYSSEY
• PROJECTS
• SANJUAN
• UPSCA03
• VASHON
• VERONICA
• VIAWARP
• VMMON
• WEBSERVER1
• WEBSERVER2

The next production maintenance window is Sunday, July 26th, from 8:00 AM to noon.

All services may be intermittently or entirely unavailable, including:

• E-mail
• Personal and departmental file shares (on Alexandria and Merlin2)
• Database Applications:  Cascade, FAMIS, Millennium, and Banner

This month’s outage was completed 1:30PM.  All services other than Cascade Web and Banner were available by noon as expected.

The next production maintenance window is Sunday, June 28th, from 8:00 AM to 4:00 PM.

From 8:00 AM to Noon, all services may be intermittently or entirely unavailable, including:

• E-mail
• Personal and departmental file shares (on Alexandria and Merlin2)
• Database Applications:  Cascade, FAMIS, Millennium, and Banner

Please note that Cascade Web and Banner will continue to be unavailable until 4:00 PM.

[Resolved] Pugetsound Webmail is unavailable. This is an issue with webmail only, all other e-mail access is available. The issue is presently being resolved.

[Update 06/25/2009] Service has been restored.

The Groups container was accidentally deleted in the OID test database today and had to be re-created based on priv data in the Summit database.
Here are the steps we had to do to recover everything:
1. Recreate the Groups container.
Export the Groups container definition (as an LDIF command) from the OID production instance, and import it into OID test. You’ll need an LDAP browser tool to do this, like JXplorer.
2. Recreate all the AD groups.
Set the status of all the pugetsound domain groups in the privilege table to PA and run
privcmd.resolve_pending_privdef on each one.
3. Recreate the members in the AD groups.
Set the status of all AD person_privilege records to PA and run privcmd.resolve_pending_privs.
4. Recreate the portal group container.
Export the portal.070109.134036.113589000 container definition (as an LDIF command) from the OID production instance, and import it into OID test. You’ll need an LDAP browser tool to do this, like JXplorer.
5. Recreate all the portal groups.
Set the status of all the portal groups in the privilege table to PA and run
privcmd.resolve_pending_privdef on each one.
6. Recreate the members in the portal groups.
Set the status of all portal person_privilege records to PA and run privcmd.resolve_pending_privs.
7. Address any unusual configuration issues.
ViewsFlash groups have a special setup, the administrator group is a member of the creator group, so that has to be done manually.

In preparation for the Office 2007 deployment in late July, Microsoft Office Compatibility files will be installed on campus computers this evening. In addition, we will be performing system scans over the next few days to ensure PCs meet the minimum requirements for Office 2007.

No action is needed from the campus community and the files will be automatically applied upon rebooting after June 23, 2009. The reboot may take a minute or two as files are applied.

To install the Microsoft Office Compatibility Pack on your home computer, follow the links on our download page.

Visit our Web site for information on today’s preparation phase or the coming Office 2007 project.

Only the following servers were patched due to limited staff availability:

• EXDEV
• FIDALGO
• GALAXY
• KETRON
• PORTAL
• VS0

During this month’s pre-production patch window the following servers will be patched from 7am-9am on Thursday 6/18/09:

• lummi
• pilchuck
• kickstart
• moodle2
• EXDEV
• FIDALGO
• GALAXY
• KETRON
• PORTAL
• VS0

\\merlin2\oiscommon\nssg\documentation\patchprocess\june2009\june2009.docx

A spammer gained control of a user account today when the account owner responded to a phishing message. The spammer sent a large volume of spam. This was noted at 9:30 PM and the user account was locked. TS staff will be checking with several recipient sites such as HotMail and Yahoo to make sure that the college has not been placed on blacklists, but you may experience E-mail delivery problems because of this incident.

Web forms was down this morning which included Cascade, Famis and Banner (a separate announcement has been posted to the Help Desk site for public viewing).
Patches were applied on the sanjuan server yesterday, and it wasn’t working after that. Paul fixed something in the configuration and bounced the server and now it’s working again.

[Update 6/1/2009 7:53 AM] Service has been restored.

Cascade, Banner and Famis web forms are experiencing errors. TS is working on this problem and will update with a resolution time when we know more.

See the post in the “Maintenance Windows” catagory for the public communication.  This post is to expand on what servers were patched, and basic changes.  All of the following patches were applied by noon.

The following windows servers were patched (Windows updates):

ALEXANDRIA

BE1

BE2

BE3

CLTACC1

CLTACC2

CMS

DHCP-1

DM-1 to DM-8

EPO

FE1

FE2

ILLIAD

INTCHK

KEEPER

MAURY2

MBS1

MBS2

MBS3

MEDIA

MERLIN2

PROJECTS

SANJUAN

VASHON

VERONICA

VIAWARP

VMMON

WEBSERVER1 (SP2 applied)

WEBSERVER2 (SP2 applied)

VMWare:

vmhost1

Linux (all available OS and Dell RAID FW/Drives as applicable):

orcas

tahoma

purgatory

styx

hades

gehenna

Additional changes on Sophos PMX servers:

Purgatory:

Turned Perc write cache on (write back), and changed linux readahead:

sudo blockdev –setra 8388608 /dev/sda

Rest:

policy on

adaptive read ahead

[Update 5/31/2009 1:00 PM] All service has been restored, and this maintenance window is complete.

[Update 5/31/2009 12:15 PM] Many services have been restored, with the exception of Database Applications:  Cascade, FAMIS, Millennium, and Banner. They are expected to be available by 4:00 PM as planned.

The next production maintenance window is Sunday, May 31, from 8:00 AM to 4:00 PM.

From 8:00 AM to Noon, all services may be intermittently or entirely unavailable, including:

• E-mail
• Personal and departmental file shares (on Alexandria and Merlin2)
• Database Applications:  Cascade, FAMIS, Millennium, and Banner

Please note that Database Applications will continue to be unavailable until 4:00 PM.

[Update 5/29/2009 7:34 AM] Power, network and wireless service have all been restored.

The power was unexpectedly cut to the fieldhouse this morning at 6:00 AM.  The campus network and wireless server are unavailable off-line until power is restored.

Upgraded “Norton Symantec Virus” on server Viawarp to “McAfee VirusScan Enterprise.”

We set the debug level back to 63 to troubleshoot the password sync errors due to AD password policy problem. Here are the commands, executed as oracle on whidbey:

oidctl connect=AS1012P server=odisrv instance=1 configset=1 flags=”port=3636 sslauth=2″ stop
oidctl connect=AS1012P server=odisrv instance=1 configset=1 flags=”port=3636 sslauth=2 debug=63″ start

The Active Directory password policy was inadvertently set to reject passwords that did not contain any special (non-alphanumeric) character, such as *#$% etc.

The problem began about 3/21/2009 and was corrected at 3:15pm on 3/26/2009. During this period, anyone changing a password using Windows was instructed to include a special character.

Passwords changed using Cascade Web during this period were not synchronized to Active Directory, so the new password did not work for Webmail, Windows, etc. This can now be corrected by changing either the AD or OID password.

The problem was corrected by deselecting the special character requirement in the AD password policy.

Here is an example of the error in the ActiveExportUsers_Groups.trc log:

Error in executing mapping DIP_LDAPWRITER_ERROR_MODIFY
javax.naming.OperationNotSupportedException: [LDAP: error code 53 - 0000052D: SvcErr: DSID-031A0FC0, problem 5003 (WILL_NOT_PERFORM), data 0
]

The following non production systems were patched today:

pilchuck (OS, RAID firmware, powerpath)

fidalgo

lummi

vmhost2

kickstart

moodle2

sage

webdev

All patches progressed as expected and no lingering issues are expected.  Please see the following share for more information:

\\merlin2\oiscommon\NSSG\Documentation\PatchProcess

Webmail was unavailable for about 15 minutes at 8:30 PM tonight. Technology Services staff was notified and has resolved the problem.

The new Impulse Point NAC server had been implemented for the residential network.  It will check machines for virus software, windows updates and shared music and movie files.   It will ask to have a profile key installed on the machine for identification, but will be transparent unless a machine is non-compliant.

The login page for Banner Forms was changed early on 5/15/2009 (see Cascade middle tier http server restarted), so Banner Forms users are prompted for Puget Sound username and password instead of Banner Database username, password, and database name.

Initially, this did not work correctly for Macintosh users, who were required to enter their username and password twice.

This problem was resolved Saturday morning, so Macintosh users should now be able to log in to Banner Forms (and create purchase orders), entering their Puget Sound credentials only one time per session.

To fix the problem, Ed changed parameter baseHTML to point to our modified basejpisso_webutil.htm file.

Technology Services are researching the cause and will update as additional information has become available. This affects CASCADE and BANNER.

Form Server is working again.

As part of implementing Banner Password Synchronization, the Cascade middle tier http server was restarted so that the following change in the httpd.conf file would take effect:

#Redirect /banner https://psforms.ups.edu/forms/frmservlet?config=banner
Redirect /banner https://psforms.ups.edu/banner_sso/gokssso.p_login

This directs cascade.ups.edu/banner to the new Banner login page that authenticates against OID.

See also Banner Password Synchronization works for Macintosh users

Our service provider performed some maintenance to our off-campus telephone connection. We lost connectivity Thursday evening, May 13th from approximately 8:00 to 8:30 pm.

The recently released Mac OS X 10.5.7 and Safari 3.2.3 fix the problem where users received “Too many redirects” error when logging in to Cascade Web.

The Technical Help page has been updated to show which versions of Mac OS X and Safari are compatible with Cascade. Browser/OS combinations that are not compatible still receive a message that includes a link to this Technical Help page.

Password synchronization and network/domain password resetting are currently disabled. Technology Services is currently working on restoring this service and we expect it to be restored shortly. We will update you as soon as possible.

UPDATE:  Password Synchronization has been re-enabled as of 4:10pm on 5/12.

A person using a Web Browser that is not compatible with Cascade Web now sees a warning message before entering their username and password and encountering an error. At present, a person attempting to log into Cascade Web from Mac OSX 10.5.6 using Safari 3.1.2, 3.2.1, or 4.0 beta will see the following log-in page:

The OID schema on summit was inadvertently overwritten with Cascade’s OID schema during a refresh of Summit. This required re-registering the partner application for “/summit”. This did not work until we dropped oid.wwsec_enabler_config_info$ then executed @loadsdk from albert’s oracle account.

The listener_token value is: cascadetest.ups.edu:4445

Spam messages quarantined prior to 9:00 AM on April 27, 2009, are no longer available on the PureMessage Web site due to the domain name change which took effect at that time. Messages arriving after 9:00 AM will be available as usual.

If you need messages quarantined before this time, please contact the HelpDesk at helpdesk@ups.edu or x8585.

The reports server experienced the same problem as last Monday. Bounced the server to see if it resolves the issue. Paul will continue to research the cause of this problem.

As part of the domain change project, e-mail addresses and URLs stored in the Cascade database have been updated from “ups.edu” to “pugetsound.edu”. New account provisioning has also been updated accordingly.

The directory and most web applications display the new addresses and URLs. A few remaining Web applications will be changed in the coming week.

This morning between 8 and 9 AM, primary e-mail addresses were changed. E-mail messages from University of Puget Sound community members will now come from @pugetsound.edu. Messages sent to @ups.edu AND  @pugetsound.edu will be delivered normally.

I have the valid certificate ready but not installed for the following reasons.

1. There should be no known application using the Certificate.
2. In the past, the Certificate has been renewed, not knowing it’s used or not, just for the precaution.
3. Before not renewing anymore, I need to make sure the Certificate is not really used by any of the applications.

[Update 04/21/2009 03:17 PM] E-mail service has been restored. All mail will be delivered shortly.

Mail inbound to pugetsound.edu has been disrupted. We are working to restore service. No mail has been lost, and all mail will be delivered. We will update you as soon as possible.

Cascade and FAMIS users reported issues receiving their reports that are run through the reports server. Users received an error message saying that job ###### wouldn’t run. When we looked in the reports server (past jobs), that particular job_id did not show up (it was a future job_id!).

Resolution: bounced reports server

Kristen Spiese requested that the Shasta server be shutdown.

It will be available to be brought back online if needed.

We will wait to repurpose the machine temporarily.

A database error occurred in CRM about 11:30 am on 4/15/2009. DST was alerted to the problem, a table that was unable to extend, and fixed it about noon. A couple of email campaigns were in-progress and were adversely affected:
1. A message from the President’s office going to faculty, staff and students was sent out twice, but the records indicated it only went out once.
2. A message that was being created by Admission was in the middle of generating the target group and got stuck there. Every attempt to resolve it failed so the solution was to copy the schedule without the target group, re-create the target group and then the email was sent out successfully.

The server INTCHK will be moved from a physical to a virtaul machine on 4/17/2009 starting at 7am.  During this window INTCHK will be completely unavailable.

Performed the script test this morning, 7:00-8:30 am.

• Verified the script is working, so Nick installed it onto Tahoma as well.
• Verified Enterprise Manger’s Alert system is working successfully.
• Found one oracle bug on EM Agent which I’ll apply the patch soon.

   

This evening, a problem with publishing content to the Athletics site was resolved by republishing the entire Web site. During the republishing process, images may not have displayed correctly; however, content was available throught that period.

We replaced the yearly SSL certificates for the Webmail (webmail.ups.edu), IMAP (imap.ups.edu), and POP3 (pop3.ups.edu) servers this morning. All these services are up an running and accepting connections.

Performed Mil9ip tuning on 4/7/2008, based the test on mildemo instance with Sean Vincent.  According to Sean, this tuning improved the performance of Millennium dramatically.  The tuning includes the follwing changes in the parameter file.

·         optimizer_mode=FIRST_ROWS from CHOOSE

·         increased shared_pool_size, pga_aggregate_target size, and sga_max_size

AD must have sole control of its “mail” attribute to properly provision e-mail accounts. OID –> AD synchronization had included this, and the provisioning system (privcmd) had been filling this with any known e-mail address. This prevented new pugetsound domain accounts from being enabled when the e-mail address was not in the ups.edu domain, as is the case for most new students.

The mapping file was changed to no longer update the “mail” attribute when synchronizing from OID to AD.

Password Synchronization is here! You can now synchronize your password across many Puget Sound applications by changing your password in Cascade Web, Webmail/e-mail or Windows. Learn more at http://www.ups.edu/pssync.xml.

The Oracle listener on Albert stopped functioning 3/26/2009 around 9:15 or 9:30am, which caused all Oracle databases on the machine to stop responding.

This affected development and test application servers. Specifically Registrar staff testing was interrupted. No production services were affected.

The listener was bounced and service restored after about 20-30 minutes.

The following minor changes were installed 3/25/2009 10:30am, with no disruption:

• Cascade Home page now includes “Technical Problems” link
• Password Change page now includes buttons when using tab key to navigate
• Typo in error message was fixed
• Format of initial password was changed so it includes mixed case instead of symbol

To support Banner Single Sign On, minor software updates will be installed during our normal maintenance window on Wednesday, 3/25/09, between 5:30 and 6:00 AM. The service disruption to Cascade Web will last only a few minutes.

The authorized license count for our new Cisco ASA Anyconnect VPN client has been updated from 2 to 25 simultaneous users.

Due to configuration errors at the Internet Service Provider (Integra Telecom), VPN connections were not working from 6:00 to 7:20 AM. The VPN has been restored.

The emergency power repairs have been completed, and all systems have been restored to normal.