Intermittent Service Disruption
Incident Report for Glia
Postmortem

On September the 19th at 7:24AM UTC, Glia released changes to the Operator Console Web Application which introduced a bug that prevented Operators from accepting incoming transfer requests for Operators in multi-engagement.

Glia ensures software quality by extensive end-to-end testing, however there was a gap in our test suite and this particular flow was not covered. The problem was compounded by an unrelated breakage in our telemetry which prevented us from noticing the problem until September the 20th at 15.30 UTC when our Engineering team started the investigation. By 15.34 UTC the issue was identified and by 15.58 UTC a fix was implemented. Since the Operator Console is a mission critical tool for our clients Glia Engineering team was cautious about releasing the fix but by 16.29 UTC the fix was rolled out to all our clients.

We have already expanded our test suites and are taking steps to ensure that our tooling for telemetry does not fail again. Glia is sorry for the inconvenience this caused to our clients.

Posted Sep 23, 2019 - 12:21 UTC

Resolved
This incident has been resolved.
Posted Sep 20, 2019 - 16:29 UTC
Identified
We are experiencing intermittent failures to a percentage of our Engagements. Incoming transfer requests are not working for Operators who are already engaged. We are working to resolve this issue.
Posted Sep 20, 2019 - 15:54 UTC