UPDATE: Systems coming back online

UPDATE (8:06 a.m. May 29):

Below are updates on the three systems that were affected at the time of our last update:

• Grid Edit is now up and fully functional.
• Email continues to be available for about 70% of the organization. We expect the remainder of email access will be restored within 24 hours.
• Polaris databases are still being restored. This means that Polaris reports are available, but they only reflect data from before Thursday, May 24, when the outage occurred.

We will continue to update you as more systems are restored.

-------------------------------------------------------

UPDATE: 7 p.m. May 28

As you know, DSST experienced a major IT disruption last Thursday that continued into the weekend. We deeply apologize for the inconvenience this disruption has caused over the past 12 days. Here is a short overview of the problem we experienced to give you context:

Ten days ago we experienced an email outage (Thursday, May 18). We took corrective action which fixed email for many, but not all users. This issue continued to linger until last Thursday when the whole system crashed. After 24 hours of consulting with field experts, we discovered a bug impacted the software that runs our server cluster. Most of our applications and data sits in this cluster which was designed to be “a never go down platform.” Unfortunately, the bug in the software did just that and took down the whole server cluster -- something we have never experienced in the past decade. Since then, the DSST Home Office IT Team has been working non-stop In conjunction with experts in the U.S., Ireland, and Bangalore to resolve the problems. During this time we have moved many of the applications and a good portion of the data to new servers and a clean environment.

Here is where we stand as of 7 p.m. Monday night:

• All critical services out of the damaged cluster have been moved into an into a new working environment, with three exceptions:
  • Mail for about 30% of users is still unavailable
  • Polaris databases are down, preventing refreshed Polaris reports; and
  • The Engrade Grid Edit app is still down.
• We expect each of the above services to be restored in the next 36 hours, with a high probability that Grid Edit will come up far sooner than that.
• Outside of those three systems, we expect that all other services are fully available with possible hiccups.
• We expect to be able to administer all Tuesday IAs online, but each school has full paper versions and support just in case.

We’ve tested these services in the new environment and have seen positive results, but given the enormous complexity of the move, we expect there could be glitches. We ask that staff keep a close eye out for any issues, and alert helpdesk@scienceandtech.org (or your Helpdesk Technician) if you encounter any unexpected issues. You may e-mail the help desk both from a DSST e-mail and your own personal e-mail if that is not working

We know that the disruption to email, Engrade, Polaris and all of our other systems has been difficult as our schools work to close out the year for our students. Much of our work -- IAs/Finals, grades and year end wrap up -- depends upon these systems. We are sorry for the inconvenience this disruption all of our team members. We appreciate your patience and support. It has been a very trying couple of days and we feel fortunate to be part of such a strong team that has done whatever it takes to get our systems back online.