Follow

Service Disruption - 12th March 08:00

Service has been restored at 12:15pm GMT

At around 8:00 GMT on Thursday, we received reports that EasyContactNow customers were having issues logging in to the system.

Initial investigations have shown that the EasyContactNow databases had lost synchronisation. We're currently re-syncronising the database to fully restore the system. Meanwhile, we'll be posting timely updates below every half an hour.

Please accept our apologies while we work to fix this issue.

------

8:20am GMT - Investigations have discovered that the EasyContactNow databases have lost synchronisation. We are in the process of synchronising the databases in order to restore service

9:00am GMT - Database synchronisation is continuing. Thank you for your patience. Next update in 30 minutes.

9:30am GMT - Database synchronisation is continuing. Thank you for your patience. Next update in 30 minutes.

10:00am GMT - Database synchronisation is continuing. Thank you for your patience. Next update in 30 minutes.

10:30am GMT - Database synchronisation is continuing. Thank you for your patience. Next update in 30 minutes.

11:00am GMT - Database synchronisation is continuing. Thank you for your patience. Next update in 30 minutes.

11:30am GMT - Database synchronisation is continuing. Thank you for your patience. Next update in 30 minutes.

12:00pm GMT - Database synchronisation is continuing. Thank you for your patience. Next update in 30 minutes.

12:15pm GMT - Database synchronisation has now been completed, and the service has been restored. Thank you for your patience. We will update a post-mortem investigation after we have completed the investigation.

-------

POST-MORTEM

While updating a table for a new feature release, we experienced a failure in a third party's database software, which caused the database cluster to lose consistency.

To re-establish the cluster, nodes had to be re-synchronised manually one at a time from a single common source, which took several hours. A minimum number of redundant servers was re-established at 12:05pm, and the service was immediately available to users. The remaining nodes were synchronised during the rest of the day and all nodes were fully synchronised by 3pm.

The software vendor has this failure mode logged as an open issue, and has provided a workaround to avoid this situation in future.

Was this article helpful?
0 out of 1 found this helpful

Have more questions? Submit a request

Have more questions? Submit a request

Comments

  • Avatar
    Gillian Cropper

    Can you provide a timescale as to when you feel this issue will be resolved please?

  • Avatar
    Tim Davies

    Hi Gillian, the issue has been solved. Thank you for your patience and once again sorry for the inconvenience

  • Avatar
    Gillian Cropper

    Thanks for the update Tim. From an operational perspective I would have preferred someone to contact me with a timescale as to how long you were anticipating this would take to resolve, so that I could have made alternative arrangements for my workforce, rather than receiving an email five hours after leaving my comments. However, I hope my feedback is worth considering!

Powered by Zendesk