coe-staff: Update on this week's IT outages

[Laura Lee McIntyre]

Dear COE community,

I share the note (below) from VP and Chief Information Officer, Abhijit Pandit, about this week's technology outage that affected many of us. For those of us who were affected, it was pretty debilitating (and happened at a rough time of the academic year!). Please see below for a description of the problem and key action items.

Best,
Laura Lee

Laura Lee McIntyre | Dean & Castle-McIntosh-Knight Professor (she/her)
University of Oregon | College of Education
230 Hedco Education Building | Eugene, OR 97403-1215
llmcinty at uoregon.edu<mailto:email at uoregon.edu> |  https://education.uoregon.edu<https://education.uoregon.edu/>
[cid:image001.png at 01D99A2A.DDC8F960]<https://www.uoregon.edu/>




From: aallist-bounces at lists.uoregon.edu <aallist-bounces at lists.uoregon.edu> On Behalf Of Chief Information Officer
Sent: Thursday, June 8, 2023 4:52 PM
To: aallist at lists.uoregon.edu
Subject: [AALList] Update on this week's IT outages

Dear colleagues,

This week, the University of Oregon has experienced an unusual technology outage affecting thousands of people for multiple days. I understand how disruptive it has been to have some of our most fundamental services unavailable or degraded during such a critical time in the academic year. Please accept my sincere apologies.

I want to provide information to help support you and your teams while Information Services wraps up our cleanup work.

Key Action Items

  *   Please share this information with colleagues who have been affected by this outage.
  *   Visit the IT Service Status<https://status.uoregon.edu/> website for the latest updates, and encourage your teams to do the same - for this outage and any others.
  *   Consider forwarding other UO announcements, such as Tuesday's Around the O Workplace newsletter<https://urldefense.com/v3/__https:/t.e2ma.net/webview/0lccuh/7a81b2f1cd0fe195b4f6d68ee20e3f76__;!!C5qS4YX3!DsvO1yBTTUhjSrnavUD7qLFDrR3WD3ppl7vyRnrc4qr6584T4kOEMopyWJer8NP9B2SpeN4PPO2N_-0D$>, to affected colleagues.

What Happened
The problem began on Monday, June 5, around 8:00 p.m. Pacific Time, when our new Duck ID system vendor, Bravura, performed planned maintenance. That work had been requested by our staff to address performance issues. Despite testing by both UO and the vendor, an unexpected bug ultimately revoked service access and broke inbound email flow for about 4,600 UO accounts.

Those affected had problems accessing or using UOmail, Teams, and other Microsoft services, and couldn't log in to the UO Service Portal to submit help tickets. About 1,600 people lost Teams calling service, and some people lost access to Dropbox.

(Note: This was unrelated to Monday's widespread Microsoft outages that intermittently affected services such as UOmail and Teams at the UO and across the United States. Dropbox is also now reporting an unrelated partial outage.)

Remaining Impacts
Most of the impacts of this outage had been resolved by yesterday morning (Wednesday, June 7). Others have just been resolved today, and a few still remain. Please check the IT Service Status<https://status.uoregon.edu/> website for the most recent updates.


  *   Incoming email flow. For the vast majority of affected accounts, mail flow was fully restored early yesterday morning (Wednesday, June 7). For a small number of accounts that remained affected, inbound mail flow was restored early this morning (Thursday, June 8).
  *   Emails sent to affected accounts. During the outage (mainly on Tuesday), some emails sent to affected accounts bounced. However, other messages ended up in a limbo state within our systems. Most messages in the latter category were delivered into people's mailboxes in a process completed around 12:30 p.m. today. Our team is now doing a round of extended processing to ensure all affected users receive such emails. Depending on your email program, the messages may appear with either today's date or their original send dates.
  *   Teams calling. Service has been restored for those affected. If anyone with Teams calling still can't see their dial pad, they should log out of Teams and log back in again.
  *   Teams private channel access. People who've lost access to private Teams channels can have their access restored by the owners or members of those channels. If that's not possible, please submit a ticket at Microsoft Teams support<https://service.uoregon.edu/TDClient/2030/Portal/Requests/ServiceDet?ID=32268>.

Preventing a Recurrence
To prevent a recurrence at this critical time of year, we have paused any further work by Bravura until we better understand what went wrong on Monday. Once we resume work on the Duck ID system, Information Services will use our existing internal review and planning processes to limit disruptions to the university's academic and business functions.

I want to thank the project team that has worked hard for many months to launch our new Duck ID system<https://around.uoregon.edu/content/new-duck-id-platform-will-overhaul-account-management> in February and make dozens of post-launch improvements. It would be hard to overstate the complexity of switching identity management systems at a university. Our Duck ID system manages a constant flux in affiliations and access for more than 70,000 members of the UO community. After some early bumps in the road, the team has worked diligently with the vendor to refine and stabilize the system. The UO is benefiting from the greater security and self-service options of this new system, and more improvements to the user experience are on the docket for this summer.

If you or your teams need help relating to this outage, please contact the IT staff who support your unit<https://service.uoregon.edu/TDClient/2030/Portal/KB/ArticleDet?ID=101085> or the Technology Service Desk<https://service.uoregon.edu/TDClient/2030/Portal/KB/ArticleDet?ID=31704>.

Sincerely,

Abhijit Pandit
Vice President and Chief Information Officer

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.uoregon.edu/pipermail/coe-staff/attachments/20230609/ec00846b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 6813 bytes
Desc: image001.png
URL: <http://lists.uoregon.edu/pipermail/coe-staff/attachments/20230609/ec00846b/attachment.png>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: ATT00001.txt
URL: <http://lists.uoregon.edu/pipermail/coe-staff/attachments/20230609/ec00846b/attachment.txt>