Incident Reports
4/3/2024 Title: Service Outage
User Impact: Service may be unavailable.
Scope of Impact: Users trying to access the portal may see issues loading the page. Also call routing and recording may be affected.
We are continuing to investigate the issue.
4/3/2024 9:29 PM EST: Azure SQL service is experiencing an outage that is affecting the service.
4/3/2024 9:45 PM EST: Azure SQL service outage appears to have resolved. We are continuing to monitor.
RCA: Microsoft has stated that they suspect that a potential deployment on there frontend gateways has caused SQL database availability issues impacting customer connectivity.
Start Time: 4/3/2024 at 7:50PM EST
End Time: 4/3/2024 at 9:45PM EST
3/5/2024 Title: Teams user presence may appear incorrect in Landis contact center.
User Impact: Teams user presence may appear incorrect in Landis contact center. If Teams presence based routing is enabled on queues, call may not route to available agents.
It appears that the Microsoft presence API is delayed in sending presence updates to Landis Contact Center.
Microsoft has acknowledged that multiple customers have reported this issue and they are investigating.
Microsoft is working on a fix/mitigation for this issue. No ETA provided.
Microsoft reported this issue has been resolved.
Start Time: 3/5/2024 5:20AM EST
End Time: 3/7/2024 1:11PM EST
3/13/2024 Title: Call Recording
User Impact: Users may not be able to retrieve some call recordings.
Scope of Impact: Some calls may not be recorded.
RCA: Some of the recording nodes fell below acceptable health levels and were not able to function properly. Additional monitoring is being implemented to prevent similar impact from reoccurring.
Start Time: 3/13/2024 at 12:46PM EST
End Time 3/15/2024 at 8:45AM EST
12/27/2023 Title: Call Recording
User Impact: Users may not be able to retrieve some call recordings.
Scope of Impact: Some calls may not be recorded.
Update 1/4/24: Our engineers have pushed out a fix for the issue and we are monitoring.
Update 1/9/24: After further investigation, another issue was found and mitigated.
RCA: Invalid cache values were causing some calls to not be recorded. More redundancy was added to prevent this issue from reoccurring.
Start Time: 12/27/2023 at 09:30AM EST
End Time 1/9/2024 at 2:48PM EST
11/8/2023 Title: Intermittent Portal Service Outage
User Impact: Portal may be intermittently unavailable.
Scope of Impact: Some users trying to access the portal may see intermittent issues loading the page. Call routing and call recording are not affected.
Our engineers have remediated the issue and are further investigating cause.
RCA: One of the redundant front-end nodes fell bellow acceptable health levels and was recycled to restore optimal performance.
Start Time: 11/8/2023 at 1:17PM EST
End Time: 11/8/2023 at 1:43PM EST
11/1/2023 Title: Call Recording Outage
User Impact: Users may not be able to retrieve call recordings.
Scope of Impact: Call recording service is currently experiencing an outage.
RCA: An expired certificate caused the recording service to become degraded. Measures have been put in place to prevent this from reoccurring.
Start Time: 11/1/2023 at 12:00AM EST
End Time 11/1/2023 at 9:48AM EST
9/19/2023 Title: Intermittent Portal Service Outage
User Impact: Portal may be intermittently unavailable.
Scope of Impact: Some users trying to access the portal may see intermittent issues loading the page. Call routing and call recording are not affected.
Update: Intermittent Portal Service has been resolved. If a user has been affected, please refresh the Teams app or portal site.
RCA: A rare bug in the site utilized a high volume of resources and caused intermittent portal outages. A code fix was deployed to resolve this issue.
Start Time: 9/19/2023 at 11:48AM EST End Time: 9/19/2023 at 12:55PM EST
9/16/2023 Title: Teams user presence may appear incorrect in Landis contact center.
User Impact: Teams user presence may appear incorrect in Landis contact center. If Teams presence based routing is enabled on queues, call may not route to available agents.
RCA: An Azure infrastructure issue caused Teams presence to not update in a timely manner.
Start Time: 9/16/2023 4:48AM EST
End Time: 9/16/2023 8:05AM EST
7/20/2023 Title: A small percentage of callers connecting to queues and IVRs may not hear the configured Greetings, Prompts, or Hold Music.
User Impact: A small percentage of callers connecting to queues and IVRs may not hear the configured initial greeting, menu prompts, or hold music.
From our initial investigation into system logs, the errors for failed audio playback of prompts, greetings, or hold music for reported call scenarios are originating from the Microsoft Graph API which our system communicates with. We've opened a support ticket with the Microsoft Team and are awaiting a response from them on further root cause analysis.
Customers who are strongly affected by this issue should consider failing over to Microsoft Teams native call queues as a temporary measure.
Update - 7/21/2023 8:52 AM EST: Our team has escalated the severity of the Microsoft Support ticket and is awaiting a response from the Microsoft team.
Update - 7/21/2023 10:50 AM EST: A support rep assigned to the Microsoft Support ticket has confirmed that the ticket has been escalated for review by the Microsoft Engineering team.
Update - 7/24/2023 11:06 AM EST: Our Team is continuing to assist Microsoft engineers on this issue to find a root cause.
Update - 7/24/2023 1:20 PM EST: Microsoft has issued a fix for the issue. Our telemetry is showing the issue is resolving.
Microsoft's RCA: We are not sure about the root cause since the URL is not showing on our server-side logs, which has been redacted for privacy reasons. But our best guess is that the URL was encoded during transportation which caused the format validation failure. We can consider adding more logs to double confirm that and then add decoding logic before URL validation.
Start Time: 7/20/2023 9:24AM EST End Time: 7/24/2023 1:30PM EST
6/27/2023 Title: Queue Call Routing
6/27/2023 Title: Queue Call Routing
User Impact: Some calls may be routed to agents already on a call.
Scope of impact: Some calls may not be routed, or calls may be routed to an agent already on a call.
RCA: A recent code change was temporarily preventing existing queue calls from being routed as well as routing new queue calls to agents already on an existing call. The issue was resolved and steps are being taken to improve our testing and validation cycles to prevent similar impact from reoccurring.
Start Time: 6/27/2023 1:32PM EST End Time: 6/27/2023 1:47PM EST
6/19/2023 Title: Some customers are experiencing an issue where call controls are missing in the Teams client.
User Impact: Some calls when answered in Teams client are not able to be transferred or placed on hold. The call appears as a meeting in the Teams active call window.
This is a Microsoft Teams issue. We are currently working with Microsoft to help them resolve the issue.
6/20 6:27PM EST Microsoft Update: We have determined that a code path utilized by Microsoft Teams used to identify one-on-one calls was blocked with a recent code change. This causes the service to believe all calls are in a group meeting, preventing the advanced calling options from being available. We've restored this code path.
2/21 4:54PM EST Microsoft Update: Root cause: A recent code change was preventing incoming calls from being categorized as one-on-one calls, which was intermittently preventing advanced calling options from appearing.
Workaround: Use Landis Attendant Console to handle and transfer calls. Free 30 day trial. Getting Started guide: https://ac.docs.landis.cloud/getting-started
Start Time: 6/19/2023 2:59PM EST
End Time: 6/20/2023 4:54PM EST
6/5/2023 Title: Call Routing
6/5/2023 Title: Call Routing
User Impact: Calls may not be routed to agents as quickly as normal.
Scope of impact: Some customers may be experiencing call routing issues.
Start Time: 6/5/2023 2:49PM EST Update: 6/5/2023 2:57PM EST: Azure Service seems to have returned to normal. We are monitoring the Azure Service. End Time: 6/5/2023 2:57PM EST Root Cause Analysis: Degradation of an Azure service caused calls not to route to agents.
5/17/2023 Title: Some customers are experiencing an issue where call controls are missing in the Teams client.
User Impact: Some calls when answered in Teams client are not able to be transferred or placed on hold. The call appears as a meeting in the Teams active call window.
This is a Microsoft Teams issue. We are currently working with Microsoft to help them resolve the issue. https://admin.microsoft.com/#/servicehealth/:/alerts/TM557153
Latest Microsoft Update: We've reverted the offending code change as a remediation action, and we've verified with affected users that this action has successfully resolved the problem.
Workaround: Use Landis Attendant Console to handle and transfer calls. Free 30 day trial. Getting Started guide: https://ac.docs.landis.cloud/getting-started
Start Time: 5/17/2023 08:30AM EST
End Time: 5/18/2023 10:04 PM EST
5/4/2023 Title: Calls Intermittently are not able to be answered by a Teams endpoint.
User Impact: Some calls when answered by a Teams endpoint are not able to properly connect. From the agent perspective, the audio is not connected. The call will continue to route to agents after this has occurred.
From our telemetry this appears to be a Microsoft Graph/Teams issue. We are currently working with Microsoft to troubleshoot this issue.
Start Time: 5/4/2023 10:30AM EST
End Time: 5/4/2023 11:17PM EST
5/1/2023 Title: Call Routing
5/1/2023 Title: Call Routing
User Impact: Calls may not be routed to agents.
Scope of impact: Some customers may be experiencing call routing issues.
Start Time: 5/1/2023 1:18PM EST
End Time: 5/1/2023 1:40PM EST
Start Time 5/1/2023 2:49PM EST
End Time: 5/1/2023 3:12PM EST
Root Cause Analysis: Degradation of an Azure service caused calls not to route to agents.
1/6/2023 Title: Customer Service Recording Degradation
User Impact: Some recordings may not be able to be played or downloaded. These recordings will become available over the next 24-48 hours.
All Recordings are now available.
Start Time: 1/6/23 at 2:30PM EST
End Time: 1/7/23 at 1:00AM EST
11/10/2022 Title: Intermittent Alert Issue When Using Toast Alerts
Impact: When using toast alerts, agents intermittently may not be able to answer queue calls. Scope of impact: Inbound queue call toast alerts appear in the agent panel, but after selecting the accept button, the call may not alert the agent in Teams.
RCA: A subset of the portal infrastructure was experiencing an issue which was resolved. Start Time: 11/10/2022 3:00AM EST
End Time: 11/10/2022 9:30AM EST
10/29/2022 Title: Portal Service Outage
User Impact: Portal unavailable.
Scope of Impact: At least some users trying to access the portal.
Start Time: 10/29/2022 at 6:36AM EST
End Time 10/29/2022 6:49AM EST
9/26/2022 Title: Microsoft Teams App Sign in Failure
9/26/2022 Title: Teams App Sign in Failure
User Impact: Some users signing into the Landis Contact Center application in the Teams client may see a blank page.
Scope of Impact: Some users may not be able to sign into the Landis Contact Center application in the Teams client.
A temporary alternative solution is to use the browser to access Landis Contact Center.
Portal: https://cc.landis.cloud
Agent Panel: https://cc.landis.cloud/client/agentpanel
Start Time: 9/26/2022
End Time 10/4/2022
Root Cause Analysis: A Microsoft Teams client change caused an authentication issue in the Landis Contact Center Teams application. An update was released to mitigate the issue.
8/11/2022 Title: Portal Service and Call Routing Outage
8/11/2022 Title: Portal service and call routing outage
User Impact: Portal intermittently unavailable; Calls may not be routed to agents
Scope of Impact: Some users trying to access the portal, some calls not routed, and/or some calls may not connect to queue or IVR.
Start Time: 8/11/2022 at 11:00AM EST
End Time 8/12/2022 at 2:45AM EST
Root Cause Analysis: After the scheduled update was successfully deployed, an error appeared in the telemetry that did not occur in our QA environments. The development team identified the issue and released a hotfix to resolve the underlying problem. This resulted in a prolonged service window.
6/22/2022 Title: Portal Service Outage
6/22/2022 Title: Portal service outage.
User Impact: Portal unavailable.
Scope of Impact: At least some users trying to access the portal.
Start Time: 6/22/2022 at 5:41AM EST
End Time: 6/22/2022 at 5:49AM EST
Start Time: 6/22/2022 at 7:13AM EST
End Time: 6/22/2022 at 7:27AM EST
Root Cause Analysis: An infrastructure setting change caused the portal to become unavailable. This change was reverted.
6/9/2022 Title: Occasional Call Control Failures
Impact: Occasional call control failures could have affected routing, playing prompts, music on hold & other call control actions.
Scope of impact: Some customers may have experienced failures.
Start Time: 6/9/2022 6:17PM EST End Time: 6/9/2022 8:51PM EST
Root Cause Analysis: Transient Microsoft service disruption. We have requested a report from Microsoft and are still awaiting a response.
6/8/2022 Title: Intermittent Microsoft Graph Failures
Impact: Intermittent routing, playing prompts & music on hold & other call control failures. Scope of impact: Some customers may experience intermittent failures. Start Time: 6/8/2022 ~5:00PM EST Update: 6/8/2022 6:35PM EST (The issue appeared to be largely resolved, but had started again.) End Time: 6/8/2022 9:57PM EST Root Cause Analysis: Microsoft service update caused communication errors. We are requesting a report from Microsoft on why this Microsoft update cause these errors.
6/8/2022 Title: Microsoft Azure service causing portal access issues.
Impact: Some customers may experience portal access issues. Scope of impact: Some customers may experience portal access issues. Start Time: 6/8/2022 4:58PM EST End Time: 6/8/2022 5:11PM EST.
Root Cause Analysis: Microsoft Azure service issue during a Microsoft Azure upgrade. A few minutes prior to this incident a separate Microsoft Azure issue caused an interruption to call control component of Contact Center service.
6/8/2022 Title: Portal Service Outage
User Impact: Portal unavailable. Scope of Impact: At least some users trying to access the portal. Start Time: 6/8/2022 at 8:53AM EST End Time 6/8/2022 9:16AM EST
Root Cause Analysis: Microsoft Azure update interrupted operation of portal resource.
5/31/2022 Title: Call Routing
5/31/2022 Title: Call Routing
User Impact: Calls may not be routed to agents.
Scope of impact: Some customers are experiencing call routing issues.
Start Time: 5/31/2022 11:18PM EST
End Time 6/1/2022 12:45AM EST
Root Cause Analysis: Incident was caused by transient SQL timeouts from a maintenance task.
5/18/2022 Title: Portal Service Outage
5/18/2022 Title: Portal service outage.
User Impact: Portal unavailable.
Scope of Impact: At least some users trying to access the portal.
Start Time: 5/18/2022 at 3:21AM EST
End Time: 5/18/2022 at 3:43AM EST
Root Cause Analysis: The update on 5/17/2022 included additional resources which caused portal errors after the update maintenance window. We disabled the additional resources and the service was fully restored & operational. All new features mentioned in the 5/17/2022 update are available.
4/8/2022 Title: Portal Service Outage
4/8/2022 Title: Portal service outage.
User Impact: Portal unavailable.
Scope of Impact: At least some users trying to access the portal.
Start Time: 4/8/2022 at 8:42AM EST
End Time: 4/8/2022 at 9:15AM EST
Root Cause Analysis: The app service was unexpectedly restarted by the Azure platform. Because of the load on the application, the restart process took longer than normal. Configuration changes were made to the application to enable it to restart more quickly under load in the future.
4/8/2022 Title: Portal Service Outage
4/8/2022 Title: Portal service outage.
User Impact: Portal unavailable.
Scope of Impact: At least some users trying to access the portal.
Start Time: 4/8/2022 at 7:02AM EST
End Time: 4/8/2022 at 7:17AM EST
Root Cause Analysis: The app service was unexpectedly restarted by the Azure platform.
3/31/2022 Title: Portal Service Outage
3/31/2022 Title: Portal service outage.
User Impact: Portal unavailable.
Scope of Impact: At least some users trying to access the portal.
Start Time: 3/31/2022 at 6:25AM EST
End Time: 3/31/2022 at 6:33AM EST
Root Cause Analysis: The app service was unexpectedly restarted by the Azure platform.
2/24/2022 Title: Portal Service Outage
2/24/2022 Title: Portal service outage.
User Impact: Portal unavailable.
Scope of Impact: At least some users trying to access the portal.
Start Time: 2/24/2022 at 7:04PM EST
End Time: 2/24/2022 at 7:12PM EST
Root Cause Analysis: The app service was unexpectedly restarted by the Azure platform.
12/27/2021 Notice: Some wallboard columns are temporarily disabled.
Some wallboard columns are disabled temporarily. Our engineers are working to enable these wallboard columns again as soon as possible.
Start Time 12/27/2021
End Time 2/10/2022 Note that "Queue Status" can be used to monitor queue status as well.
2/4/2022 Title: Portal Service Outage
2/4/2022 Title: Portal service outage.
User Impact: Portal unavailable.
Scope of Impact: At least some users trying to access the portal.
Start Time: 2/4/2022 at 12:08PM EST
End Time: 2/4/2022 at 12:35PM EST
Root Cause Analysis: A maintenance process triggered an unexpected exception in portal. Portal was unavailable but active calls were unaffected.
1/26/2022 Title: Call Routing
Title: Call Routing
Scope of impact: Calls not routing to queues and alerting agents.
Time: 12:53-12:55 am EST; 1:41-1:47am EST Resolution: The problem was resolved in minutes. Some tenants experienced stuck calls in some queues, which has also been resolved.
Root Cause analysis: Incident was caused by transient Azure SQL timeouts. We have also increased the scale of our database.
12/17/2021 Title: We are investigating reports of intermittent performance issues.
User Impact: At least some users are affected.
Scope of Impact: At least some customers are affected.
12/17/2021 3:50AM EST: Engineers are working to resolve the issue. 12/17/2021 6:00AM EST: Engineers are working to resolved the issue. 12/17/2021 7:05AM EST: Additional compute resources have been added. 12/17/2021 8:07AM EST: The service is largely restored, but users may experience some intermittent performance degradation until 10pm EST Friday.
11/30/2021 Title: Some tenants routing slightly slower than normal.
User Impact: Routing to agents.
Scope of Impact: A small amount of tenants experienced slower routing than normal.
11/30/2021 at 2:41PM EST: Start time.
11/30/2021 at 3:00PM EST: Service fully restored.
Root cause analysis: A bug that can in unusual instances cause this scenario has been found and a fix will be deployed. Meanwhile mitigation measures are in place.
11/26/2021 Title: Portal Service Outage
11/26/2021 Title: Portal service outage.
User Impact: Portal unavailable.
Scope of Impact: At least some users trying to access the portal.
Start Time: 11/26/2021 at 11:24AM EST to 11:32AM
11/26/2021 11:32PM EST Service fully restored.
Root cause analysis: Azure logging enabled which caused portal to become unavailable. The Azure logging was disabled.
11/22/2021 Title: Portal Service Outage
11/22/2021 Title: Portal service outage.
User Impact: Portal unavailable.
Scope of Impact: Any users trying to access the portal.
Start Time: 11/22/2021 at 10:39AM EST.
11/22/2021 12:26PM EST Update: The Microsoft Azure resources servicing the portal appears to be failing. Our engineers are in communication with Microsoft and are moving portal to alternative Microsoft Azure resources.
11/22/2021 2:43PM EST Update: Service largely restored. (We are continuing resolve some intermittent failures)
11/22/2021 2:45PM EST Service fully restored.
Root cause analysis: The 11/19/2021 update introduced a bug that was mitigated by rolling back the update. We are adding an additional test to a our test regimen to catch this scenario for future updates. This issue was the responsibility of Landis Technologies and we apologize for this interruption of your service.
11/20/2021 Title: Call Routing
User Impact: Intermittent calls not routing to Agents.
Scope of impact: Calls status, queue status, live calls are working as expected. Call routing is affected.
Resolved 2:20pm EST: Our engineers deployed some fixes to remediate the call routing issue.
Start time: 11/20/2021 at 11:00am EST.
11/8/2021 Title: Teams Recording Audio Notifications
User Impact: Recording Audio Notifications may become enabled if previously disabled.
Scope of impact: Recording audio notifications may be heard on calls
Root Cause: Microsoft Voice API changes.
9/22/2021 Title: Slower than normal call routing
User Impact: Slower than normal call routing
Update 9/22 11:40am EST: Issue appears to be resolved. We are working with Microsoft to address the underlying issue.
Update 9/22 11:00am EST: Azure is experiencing significant errors and are working with Microsoft to resolve.
Scope of impact: Calls status, queue status, live calls are working as expected. Call routing is affected.
Start time: 9/22/2021 at 10:30am EST.
Root Cause: Azure infrastructure
9/16/2021 Title: Slower than normal call routing
User Impact: Slower than normal call routing
Scope of impact: Calls status, queue status, live calls are working as expected. Only call routing is affected. Our team is exploring the cause of this. This is a degradation notice, not an outage.
Update 9/16 5:45pm EST: In working with Microsoft, it appears that there were some Microsoft Azure infrastructure updates, which corresponds to a significant increase in timeouts between our routing components. We implemented some changes to work around the effect of these timeouts until they are resolved by Microsoft Azure.
Update 9/16 6:15pm EST: It appears Microsoft Azure has now addressed the underlying infrastructure issue causing the timeouts.
Start time: 9/16/2021 at 11:45am EST.
Root Cause: Azure infrastructure
9/13/2021 Title: Portal Service Outage
User Impact: Portal unavailable
9/13/2021 11:28 am EST After routine Microsoft Azure SQL maintenance, Azure SQL has become unavailable. We are working with Microsoft to restore connectivity.
9/13/2021 3:45 pm EST, service is up for call routing. Note: There may be some items for some customers that will resolve over the next hours as the system settles.
Scope of Impact: Any users trying to access the portal.
Start Time: 9/13/2021 at 11:28 am EST.
Root Cause: Microsoft Azure SQL
6/23/2021 Title: Portal Service Outage
6/23/2021 Title: Portal service outage.
User Impact: Portal unavailable for several minutes beginning on 6/23/2021 at 4:47PM EST.
Current Status: Service was completely restored several minutes later at 5:02PM EST.
Scope of Impact: Any users trying to access the portal.
Start Time: 6/23/2021 at 4:47PM EST.
Root Cause: There was a resource exhaustion on web servers which caused them to become non-responsive.
Mitigation: We will improve the monitoring and scaling of our systems to prevent resource exhaustion.
6/16/2021 Title: Portal Service Outage
Title: Portal service outage.
User Impact: Portal Unavailable for about 2 minutes
Current Status: Service has been restored.
Scope of impact: Any user accessing the portal.
Start Time: 6/15/2021 at 10:20am EST
Root Cause: Incident was caused by a spike in database traffic that went above our usage and scaling calculations. To resolve this, we have increased the scale of our database and increased the buffer in our scaling calculations.
3/3/2021 Title: Portal Service Outage
Title: Portal service outage.
User Impact: Portal unavailable for a few minutes on 3/3/2021. More info: Portal unavailable and some intermittent cases of interrupted queue alerts. Current Status: Service has been restored. A fix to avoid this condition in the future is scheduled to be included in the next maintenance window.
Scope of Impact: Any user trying to access the portal.
Start Time: 3/3/2021 at 5:01pm UTC (12:01pm EST)
Root Cause: We found an increased amount of out of memory exceptions which we believe led to greatly decreased responses times resulting in a period of several minutes where the contact center portal was not responsive.
Last updated