On 18th November 2015, some customers complained that their outbound dialling statistics showed high call failure. This was occurring on calls offered to one of several outbound carrier gateways. The specific gateway had been moved as part of a process of pro-active maintenance, following the agreed process. However, in this instance there was bespoke configuration which meant the server would appear to handle calls, but they would not actually connect. The server was taken out of service, which resolved the symptoms. The configuration was corrected, and the servers brought back into service.
01:00 - Maintenance completed.
09:28 - Customer complaints received.
10:13 - Gateway removed from service.
12:07 - Gateway re-configured and brought back into service.
- The failing gateway was removed from service, and the other agent gateways took the load.
The specific gateway had bespoke configuration which was not identified by the current maintenance process. The process was updated to cover this eventuality, and increased testing and review approval was provided for future maintenance.
We apologise unreservedly for the inconvenience caused.