Agenda item

Consideration was given to a report of the Chief Officer, Northern Care Alliance which provided Members with an update on the NCA IT Outage Critical Incident debrief report.

The Committee was informed that on the 17^th May 2022 the Information Technology (IT) on call team received reports of the Symphony (A&E) system becoming unresponsive. This resulted in calls being logged with EMIS (Symphony support) and Dell and VMware (infrastructure support) and triggered an investigative process. By mid-morning on 18th of May it had become clear that this was an issue related to the Trust’s virtual infrastructure which was affecting most clinical systems in the North East Sector (NES) affecting Bury, Rochdale, Oldham and North Manchester (managed by MFT) acute provider teams. An Incident Response Meeting was scheduled and

chaired by the NCA Chief Delivery Officer. At this meeting the risks and impacted areas were identified with the decision made to establish twice daily meetings and for services across the NES and North Manchester to invoke their Business Continuity plans. Throughout the weekend of the 21st and 22nd of May the suppliers (Dell & VMware) continued to work on the issues with support from the IT & Digital Teams, with Care Organisations maintaining Business Continuity. On the 23rd of May the decision was made to build an emergency environment to transfer critical clinical services to. Discussions took place regarding escalating to a Critical Incident; and the final decision was made by the NCA Executive in a meeting chaired by the NCA Chief Executive.

Members were informed that staff had to shift to handwritten documents and the use of Dictaphones during the failure. Not all of the Dictaphone records had yet been added into patients’ records and the communication disruptions may require patients to have further hospital attendances and diagnostic checks. 67 per cent of staff who responded to a debrief form across the health organisations affected, felt that patient safety had been ‘compromised’ during the IT failure.

During the period of the outage there were 327 incidents of low harm reported across the hospitals, most of which related to medications and missed drug dosage, documentation and IT security. It was noted that there were two incidents of moderate harm which involved a medication error and a surgery related incident. Additionally, one serious incident was reported which related to end-of-life care, in which a referral was made to the coroner containing incorrect patient demographics.

Members were advised that the outages were caused by a previously unknown inherent software defect. This had been triggered by a very specific set of conditions which included a server being disconnected for routine maintenance purposes for less than five minutes, combined with the outcomes of nightly backups running while the server was disconnected.

The disconnection would not normally cause any adverse impact and the nightly backups were also part of the core design of the system.

It was noted that moving forward, the incident would be used as a case study with ongoing audits taking place which would ensure the services were performing as it should be. From 2023 all patients’ records would move to electronic and a patient portal be set up for patients to view results.

RESOLVED that the update be noted.