Between 4pm and 5pm call centres reported seeing abnormal dialling patterns that caused a high level of abandon calls. We are currently urgently investigating the cause.
Today at 4pm a database node synchronisation process caused inconsistent data reads. Diallers using this cluster believed that numbers were not being dialled and increased the dial rates to compensate.
The issue continued for one hour and led to abandonment rates beyond Ofcom regulations.
The issue was not proactively detected leading to protracted diagnosis as the monitoring systems use the same databases to validate behaviour, and so saw the same dialling data as the diallers.
To prevent re-occurrence several measures are being taken including having a proactive write/read test to validate data consistency on a continuous basis and shut down diallers if any inconsistency is found.
We are also adding an independent monitoring process to read from the telecom reporting systems rather than the core dialler systems as a "belt and braces" approach.
The records that were dialled during this period have not been lost, they will be called back as normal after the 72hr moratorium that Ofcom requires for recontacting abandoned calls.
We apologies for the problems that this has caused.