Task Reference

EN-25670

Service/s Affected

DC5 - UPS A Stream Array

Impact Start Time

03/07/2023 15:00

Outage Description/Impact

Dear Customer,

We are writing to inform you that we are aware of a transient fault on our UPS A Stream within DC5. Whilst this is not currently service affecting, resiliency has been reduced.

We are working with our Maintenance teams to have this resolved and the next update will occur in 4 hours or before should information become available.

Update: Tuesday 4th July 17:30pm

We are currently awaiting Engineering resources, further updates will be provided in due course.

Update: Tuesday 4th July 20:45pm

We are currently awaiting further engineer resources and will provide further updates as they become available.

Update: Monday 4th July 13:30PM

Further to our service notice, we now have our maintenance engineers on site and also third-party engineers who have carried out an initial assessment.

To allow further diagnostics, our teams have identified the next action is to isolate the affected UPS’s on our A Stream and migrate this onto a temporary UPS Supply until we can identify and fix the fault with the vendor.

At this time we are still trying to ascertain the cause and exact mode of failure of the UPS, which points to an inverter issue (or possibly controller card issue) and subsequent loss of synchronisation across UPS’s which have shifted into bypass mode.

Our next steps are to ensure the DC5 A stream is restored from bypass mode to a temporary UPS solution to mitigate the risk to our power supply whilst we work on the affected units.

Update: Wednesday 5th July 15:00pm

We have now isolated the affected A stream UPS’s and diverted the A stream onto a temporary UPS Supply.

We are now moving forward to ascertain the exact fault on the UPS array. Until we can determine what the fault is we are now suspending all power and rack commissioning activities within DC5 until we have confirmed a resolution of our ongoing UPS Issue.  If this affects you, please contact your account manager with the exact details of what works you need to do, which racks and when and we will endeavour whenever possible to accommodate within the bounds of safety and the security of all customers.   We will also be limiting the customer parking in DC5 to ensure that it does not impinge with the possible need for the delivery of replacement parts.  All of our plant and machinery are under support contracts and there is a need for third party contractors to gain access unimpeded.  Thanks for bearing with us during this period of intense activity.

We will continue to keep all DC5 Customer’s up to date as new information is available.

Update: Wednesday 12th July 2023

Following on from our UPS A feed Issue, we are still currently operating at a reduced level of resiliency. Our teams are currently liaising with our vendor working under our maintenance contract.

As of now, we do not have a timescale to resolution given we are awaiting further investigation/replacement of parts. We will update should we have more information or by 11 am on the 14th of July