Notice #505564
Recovering

Some inbound calls to a small subset of users are not terminating at our switches (UPSTREAM ISSUE)

Began:

The carrier (upstream) has provided further details:

"We would like to advise that our engineering team and our carrier partner have made progress and are advising that the issue is currently intermittent. At this time, the root cause is still under investigation, and we do not have an estimated time for resolution.

The Issue is calls inbound from west of Saskatchewan. The issue is on the Telus side, and the Engineers are working on the solution."

This affects ~3.8% of all active phone numbers today. We apologize for the upstream carrier issue that caused the disruption. We mitigated all outbound calling by routing through different non-Telus and related carrier partners.

We've fixed the core issue, and are waiting for things to recover. Issue appears to have impacted calls to 403 Area Code for a small subset of clients. We'll update with a complete post-mortem ASAP.

We are investigating a report that a small subset of calls into our network are not terminating on our switches. Upstream carriers are engaged, and we'll report back with more information shortly.

  • Voice Servers

    Call Processing, Recording, Portal

    • Ottawa (YOW)
    • Calgary (YYC)
  • Applications (AWS YUL)

    Conduit, Custom Applications and Databases

  • Upstream Carriers
Upcoming Maintenance

[Scheduled] Regular Maintenance (Daily from Midnight to 2:00 AM MST)

  • Voice Servers
  • Ottawa (YOW)
  • Calgary (YYC)
  • Applications (AWS YUL)
  • Upstream Carriers

We regularly upgrade and backup systems nightly and most often with no known impact to the network, servers or portals. If a Geo Datacenter is impacted (Vancouver, Calgary, Ottawa or Montreal), we will automatically fail over prior to the maintenance to ensure calls are not impacted.

Previous Incidents

[Resolved] Call quality issues

Began: Ended: Duration:
  • Voice Servers
  • Ottawa (YOW)
  • Calgary (YYC)

We are experiencing issues with call quality, and our Engineering Team is investigating. We will update as soon as we get more information.

We've confirmed the root cause, and we are currently monitoring it.

We've now resolved the incident. Thanks for your patience.

We are seeing additional sporadic lower MoS scores in the past 45 minutes. A dozen switch engineers are investigating live, but it looks to have stabilized. A full post-mortem will be issued later today, once the identified cause has been confirmed.

We've confirmed there is a problem and are working to resolve it for improved call quality during the recovery. All calls are progressing and always were during the period of lower call quality yesterday and this morning.

This incident is resolved. Thanks for your patience.

Investigating lower MoS scores on some calls is under investigation.

This incident is resolved. Thanks for your patience.

No further notices from the past 30 days.