Incident Reports
Last updated
Was this helpful?
Last updated
Was this helpful?
User Impact: Teams user presence may intermittently appear incorrect in Landis contact center. If Teams presence based routing is enabled on queues, call may not route to available agents. If you are affected, we recommend temporarily disabling Teams presence based routing your queues.
6/5/2024 9:10AM EDT In some cases a backend process is not able to correctly update the Teams presence for some organizations. Our engineers are working to resolve the issue.
6/5/2024 9:33AM EDT A fix has been pushed and we are monitoring. Organization that are affected should see the issue resolved over the next few hours. Start Time: 6/4/2024 11:10AM EDT
End Time: 6/5/2024 11:50AM EDT
User Impact: Teams user presence may appear incorrect in Landis contact center. If Teams presence based routing is enabled on queues, call may not route to available agents.
It appears that the Microsoft presence API is delayed or is not sending presence updates to Landis Contact Center.
4/17/2024 1:11AM EDT We have confirmed the Microsoft Presence API is not sending valid presence data to Landis Contact Center. We have reported this to Microsoft and are working with them on a resolution.
Microsoft has resolved the issue.
Microsoft's RCA: A recent scheduled service update contained a change intended to optimize presence-related data flow, which was unexpectedly causing impact.
Start Time: 4/16/2024 6:11PM EDT
End Time: 4/17/2024 5:50 AM EDT
User Impact: Service may be unavailable.
Scope of Impact: Users trying to access the portal may see issues loading the page. Also call routing and recording may be affected.
We are continuing to investigate the issue.
4/3/2024 9:29 PM EST: Azure SQL service is experiencing an outage that is affecting the service.
4/3/2024 9:45 PM EST: Azure SQL service outage appears to have resolved. We are continuing to monitor.
RCA: Microsoft has stated that they suspect that a potential deployment on there frontend gateways has caused SQL database availability issues impacting customer connectivity.
Start Time: 4/3/2024 at 7:50PM EST
End Time: 4/3/2024 at 9:45PM EST
User Impact: Teams user presence may appear incorrect in Landis contact center. If Teams presence based routing is enabled on queues, call may not route to available agents.
It appears that the Microsoft presence API is delayed in sending presence updates to Landis Contact Center.
Microsoft has acknowledged that multiple customers have reported this issue and they are investigating.
Microsoft is working on a fix/mitigation for this issue. No ETA provided.
Microsoft reported this issue has been resolved.
Start Time: 3/5/2024 5:20AM EST
End Time: 3/7/2024 1:11PM EST
User Impact: Users may not be able to retrieve some call recordings.
Scope of Impact: Some calls may not be recorded.
RCA: Some of the recording nodes fell below acceptable health levels and were not able to function properly. Additional monitoring is being implemented to prevent similar impact from reoccurring.
Start Time: 3/13/2024 at 12:46PM EST
End Time 3/15/2024 at 8:45AM EST
User Impact: Users may not be able to retrieve some call recordings.
Scope of Impact: Some calls may not be recorded.
Update 1/4/24: Our engineers have pushed out a fix for the issue and we are monitoring.
Update 1/9/24: After further investigation, another issue was found and mitigated.
RCA: Invalid cache values were causing some calls to not be recorded. More redundancy was added to prevent this issue from reoccurring.
Start Time: 12/27/2023 at 09:30AM EST
End Time 1/9/2024 at 2:48PM EST
User Impact: Portal may be intermittently unavailable.
Scope of Impact: Some users trying to access the portal may see intermittent issues loading the page. Call routing and call recording are not affected.
Our engineers have remediated the issue and are further investigating cause.
RCA: One of the redundant front-end nodes fell bellow acceptable health levels and was recycled to restore optimal performance.
Start Time: 11/8/2023 at 1:17PM EST
End Time: 11/8/2023 at 1:43PM EST
User Impact: Users may not be able to retrieve call recordings.
Scope of Impact: Call recording service is currently experiencing an outage.
RCA: An expired certificate caused the recording service to become degraded. Measures have been put in place to prevent this from reoccurring.
Start Time: 11/1/2023 at 12:00AM EST
End Time 11/1/2023 at 9:48AM EST
User Impact: Portal may be intermittently unavailable.
Scope of Impact: Some users trying to access the portal may see intermittent issues loading the page. Call routing and call recording are not affected.
Update: Intermittent Portal Service has been resolved. If a user has been affected, please refresh the Teams app or portal site.
RCA: A rare bug in the site utilized a high volume of resources and caused intermittent portal outages. A code fix was deployed to resolve this issue.
Start Time: 9/19/2023 at 11:48AM EST End Time: 9/19/2023 at 12:55PM EST
User Impact: Teams user presence may appear incorrect in Landis contact center. If Teams presence based routing is enabled on queues, call may not route to available agents.
RCA: An Azure infrastructure issue caused Teams presence to not update in a timely manner.
Start Time: 9/16/2023 4:48AM EST
End Time: 9/16/2023 8:05AM EST
User Impact: A small percentage of callers connecting to queues and IVRs may not hear the configured initial greeting, menu prompts, or hold music.
From our initial investigation into system logs, the errors for failed audio playback of prompts, greetings, or hold music for reported call scenarios are originating from the Microsoft Graph API which our system communicates with. We've opened a support ticket with the Microsoft Team and are awaiting a response from them on further root cause analysis.
Customers who are strongly affected by this issue should consider failing over to Microsoft Teams native call queues as a temporary measure.
Update - 7/21/2023 8:52 AM EST: Our team has escalated the severity of the Microsoft Support ticket and is awaiting a response from the Microsoft team.
Update - 7/21/2023 10:50 AM EST: A support rep assigned to the Microsoft Support ticket has confirmed that the ticket has been escalated for review by the Microsoft Engineering team.
Update - 7/24/2023 11:06 AM EST: Our Team is continuing to assist Microsoft engineers on this issue to find a root cause.
Update - 7/24/2023 1:20 PM EST: Microsoft has issued a fix for the issue. Our telemetry is showing the issue is resolving.
Microsoft's RCA: We are not sure about the root cause since the URL is not showing on our server-side logs, which has been redacted for privacy reasons. But our best guess is that the URL was encoded during transportation which caused the format validation failure. We can consider adding more logs to double confirm that and then add decoding logic before URL validation.
Start Time: 7/20/2023 9:24AM EST End Time: 7/24/2023 1:30PM EST
6/27/2023 Title: Queue Call Routing
User Impact: Some calls may be routed to agents already on a call.
Scope of impact: Some calls may not be routed, or calls may be routed to an agent already on a call.
RCA: A recent code change was temporarily preventing existing queue calls from being routed as well as routing new queue calls to agents already on an existing call. The issue was resolved and steps are being taken to improve our testing and validation cycles to prevent similar impact from reoccurring.
Start Time: 6/27/2023 1:32PM EST End Time: 6/27/2023 1:47PM EST
User Impact: Some calls when answered in Teams client are not able to be transferred or placed on hold. The call appears as a meeting in the Teams active call window.
This is a Microsoft Teams issue. We are currently working with Microsoft to help them resolve the issue.
6/20 6:27PM EST Microsoft Update: We have determined that a code path utilized by Microsoft Teams used to identify one-on-one calls was blocked with a recent code change. This causes the service to believe all calls are in a group meeting, preventing the advanced calling options from being available. We've restored this code path.
2/21 4:54PM EST Microsoft Update: Root cause: A recent code change was preventing incoming calls from being categorized as one-on-one calls, which was intermittently preventing advanced calling options from appearing.
Start Time: 6/19/2023 2:59PM EST
End Time: 6/20/2023 4:54PM EST
6/5/2023 Title: Call Routing
User Impact: Calls may not be routed to agents as quickly as normal.
Scope of impact: Some customers may be experiencing call routing issues.
Start Time: 6/5/2023 2:49PM EST Update: 6/5/2023 2:57PM EST: Azure Service seems to have returned to normal. We are monitoring the Azure Service. End Time: 6/5/2023 2:57PM EST Root Cause Analysis: Degradation of an Azure service caused calls not to route to agents.
User Impact: Some calls when answered in Teams client are not able to be transferred or placed on hold. The call appears as a meeting in the Teams active call window.
Latest Microsoft Update: We've reverted the offending code change as a remediation action, and we've verified with affected users that this action has successfully resolved the problem.
Start Time: 5/17/2023 08:30AM EST
End Time: 5/18/2023 10:04 PM EST
User Impact: Some calls when answered by a Teams endpoint are not able to properly connect. From the agent perspective, the audio is not connected. The call will continue to route to agents after this has occurred.
From our telemetry this appears to be a Microsoft Graph/Teams issue. We are currently working with Microsoft to troubleshoot this issue.
Start Time: 5/4/2023 10:30AM EST
End Time: 5/4/2023 11:17PM EST
5/1/2023 Title: Call Routing
User Impact: Calls may not be routed to agents.
Scope of impact: Some customers may be experiencing call routing issues.
Start Time: 5/1/2023 1:18PM EST
End Time: 5/1/2023 1:40PM EST
Start Time 5/1/2023 2:49PM EST
End Time: 5/1/2023 3:12PM EST
Root Cause Analysis: Degradation of an Azure service caused calls not to route to agents.
User Impact: Some recordings may not be able to be played or downloaded. These recordings will become available over the next 24-48 hours.
All Recordings are now available.
Start Time: 1/6/23 at 2:30PM EST
End Time: 1/7/23 at 1:00AM EST
Impact: When using toast alerts, agents intermittently may not be able to answer queue calls. Scope of impact: Inbound queue call toast alerts appear in the agent panel, but after selecting the accept button, the call may not alert the agent in Teams.
RCA: A subset of the portal infrastructure was experiencing an issue which was resolved. Start Time: 11/10/2022 3:00AM EST
End Time: 11/10/2022 9:30AM EST
User Impact: Portal unavailable.
Scope of Impact: At least some users trying to access the portal.
Start Time: 10/29/2022 at 6:36AM EST
End Time 10/29/2022 6:49AM EST
9/26/2022 Title: Teams App Sign in Failure
User Impact: Some users signing into the Landis Contact Center application in the Teams client may see a blank page.
Scope of Impact: Some users may not be able to sign into the Landis Contact Center application in the Teams client.
A temporary alternative solution is to use the browser to access Landis Contact Center.
Start Time: 9/26/2022
End Time 10/4/2022
Root Cause Analysis: A Microsoft Teams client change caused an authentication issue in the Landis Contact Center Teams application. An update was released to mitigate the issue.
8/11/2022 Title: Portal service and call routing outage
User Impact: Portal intermittently unavailable; Calls may not be routed to agents
Scope of Impact: Some users trying to access the portal, some calls not routed, and/or some calls may not connect to queue or IVR.
Start Time: 8/11/2022 at 11:00AM EST
End Time 8/12/2022 at 2:45AM EST
Root Cause Analysis: After the scheduled update was successfully deployed, an error appeared in the telemetry that did not occur in our QA environments. The development team identified the issue and released a hotfix to resolve the underlying problem. This resulted in a prolonged service window.
6/22/2022 Title: Portal service outage.
User Impact: Portal unavailable.
Scope of Impact: At least some users trying to access the portal.
Start Time: 6/22/2022 at 5:41AM EST
End Time: 6/22/2022 at 5:49AM EST
Start Time: 6/22/2022 at 7:13AM EST
End Time: 6/22/2022 at 7:27AM EST
Root Cause Analysis: An infrastructure setting change caused the portal to become unavailable. This change was reverted.
Impact: Occasional call control failures could have affected routing, playing prompts, music on hold & other call control actions.
Scope of impact: Some customers may have experienced failures.
Start Time: 6/9/2022 6:17PM EST End Time: 6/9/2022 8:51PM EST
Root Cause Analysis: Transient Microsoft service disruption. We have requested a report from Microsoft and are still awaiting a response.
Impact: Intermittent routing, playing prompts & music on hold & other call control failures. Scope of impact: Some customers may experience intermittent failures. Start Time: 6/8/2022 ~5:00PM EST Update: 6/8/2022 6:35PM EST (The issue appeared to be largely resolved, but had started again.) End Time: 6/8/2022 9:57PM EST Root Cause Analysis: Microsoft service update caused communication errors. We are requesting a report from Microsoft on why this Microsoft update cause these errors.
Impact: Some customers may experience portal access issues. Scope of impact: Some customers may experience portal access issues. Start Time: 6/8/2022 4:58PM EST End Time: 6/8/2022 5:11PM EST.
Root Cause Analysis: Microsoft Azure service issue during a Microsoft Azure upgrade. A few minutes prior to this incident a separate Microsoft Azure issue caused an interruption to call control component of Contact Center service.
User Impact: Portal unavailable. Scope of Impact: At least some users trying to access the portal. Start Time: 6/8/2022 at 8:53AM EST End Time 6/8/2022 9:16AM EST
Root Cause Analysis: Microsoft Azure update interrupted operation of portal resource.
5/31/2022 Title: Call Routing
User Impact: Calls may not be routed to agents.
Scope of impact: Some customers are experiencing call routing issues.
Start Time: 5/31/2022 11:18PM EST
End Time 6/1/2022 12:45AM EST
Root Cause Analysis: Incident was caused by transient SQL timeouts from a maintenance task.
5/18/2022 Title: Portal service outage.
User Impact: Portal unavailable.
Scope of Impact: At least some users trying to access the portal.
Start Time: 5/18/2022 at 3:21AM EST
End Time: 5/18/2022 at 3:43AM EST
Root Cause Analysis: The update on 5/17/2022 included additional resources which caused portal errors after the update maintenance window. We disabled the additional resources and the service was fully restored & operational. All new features mentioned in the 5/17/2022 update are available.
4/8/2022 Title: Portal service outage.
User Impact: Portal unavailable.
Scope of Impact: At least some users trying to access the portal.
Start Time: 4/8/2022 at 8:42AM EST
End Time: 4/8/2022 at 9:15AM EST
Root Cause Analysis: The app service was unexpectedly restarted by the Azure platform. Because of the load on the application, the restart process took longer than normal. Configuration changes were made to the application to enable it to restart more quickly under load in the future.
4/8/2022 Title: Portal service outage.
User Impact: Portal unavailable.
Scope of Impact: At least some users trying to access the portal.
Start Time: 4/8/2022 at 7:02AM EST
End Time: 4/8/2022 at 7:17AM EST
Root Cause Analysis: The app service was unexpectedly restarted by the Azure platform.
3/31/2022 Title: Portal service outage.
User Impact: Portal unavailable.
Scope of Impact: At least some users trying to access the portal.
Start Time: 3/31/2022 at 6:25AM EST
End Time: 3/31/2022 at 6:33AM EST
Root Cause Analysis: The app service was unexpectedly restarted by the Azure platform.
2/24/2022 Title: Portal service outage.
User Impact: Portal unavailable.
Scope of Impact: At least some users trying to access the portal.
Start Time: 2/24/2022 at 7:04PM EST
End Time: 2/24/2022 at 7:12PM EST
Root Cause Analysis: The app service was unexpectedly restarted by the Azure platform.
Some wallboard columns are disabled temporarily. Our engineers are working to enable these wallboard columns again as soon as possible.
Start Time 12/27/2021
End Time 2/10/2022 Note that "Queue Status" can be used to monitor queue status as well.
2/4/2022 Title: Portal service outage.
User Impact: Portal unavailable.
Scope of Impact: At least some users trying to access the portal.
Start Time: 2/4/2022 at 12:08PM EST
End Time: 2/4/2022 at 12:35PM EST
Root Cause Analysis: A maintenance process triggered an unexpected exception in portal. Portal was unavailable but active calls were unaffected.
Title: Call Routing
Scope of impact: Calls not routing to queues and alerting agents.
Time: 12:53-12:55 am EST; 1:41-1:47am EST Resolution: The problem was resolved in minutes. Some tenants experienced stuck calls in some queues, which has also been resolved.
Root Cause analysis: Incident was caused by transient Azure SQL timeouts. We have also increased the scale of our database.
User Impact: At least some users are affected.
Scope of Impact: At least some customers are affected.
12/17/2021 3:50AM EST: Engineers are working to resolve the issue. 12/17/2021 6:00AM EST: Engineers are working to resolved the issue. 12/17/2021 7:05AM EST: Additional compute resources have been added. 12/17/2021 8:07AM EST: The service is largely restored, but users may experience some intermittent performance degradation until 10pm EST Friday.
User Impact: Routing to agents.
Scope of Impact: A small amount of tenants experienced slower routing than normal.
11/30/2021 at 2:41PM EST: Start time.
11/30/2021 at 3:00PM EST: Service fully restored.
Root cause analysis: A bug that can in unusual instances cause this scenario has been found and a fix will be deployed. Meanwhile mitigation measures are in place.
11/26/2021 Title: Portal service outage.
User Impact: Portal unavailable.
Scope of Impact: At least some users trying to access the portal.
Start Time: 11/26/2021 at 11:24AM EST to 11:32AM
11/26/2021 11:32PM EST Service fully restored.
Root cause analysis: Azure logging enabled which caused portal to become unavailable. The Azure logging was disabled.
11/22/2021 Title: Portal service outage.
User Impact: Portal unavailable.
Scope of Impact: Any users trying to access the portal.
Start Time: 11/22/2021 at 10:39AM EST.
11/22/2021 12:26PM EST Update: The Microsoft Azure resources servicing the portal appears to be failing. Our engineers are in communication with Microsoft and are moving portal to alternative Microsoft Azure resources.
11/22/2021 2:43PM EST Update: Service largely restored. (We are continuing resolve some intermittent failures)
11/22/2021 2:45PM EST Service fully restored.
Root cause analysis: The 11/19/2021 update introduced a bug that was mitigated by rolling back the update. We are adding an additional test to a our test regimen to catch this scenario for future updates. This issue was the responsibility of Landis Technologies and we apologize for this interruption of your service.
User Impact: Intermittent calls not routing to Agents.
Scope of impact: Calls status, queue status, live calls are working as expected. Call routing is affected.
Resolved 2:20pm EST: Our engineers deployed some fixes to remediate the call routing issue.
Start time: 11/20/2021 at 11:00am EST.
User Impact: Recording Audio Notifications may become enabled if previously disabled.
Scope of impact: Recording audio notifications may be heard on calls
Root Cause: Microsoft Voice API changes.
User Impact: Slower than normal call routing
Update 9/22 11:40am EST: Issue appears to be resolved. We are working with Microsoft to address the underlying issue.
Update 9/22 11:00am EST: Azure is experiencing significant errors and are working with Microsoft to resolve.
Scope of impact: Calls status, queue status, live calls are working as expected. Call routing is affected.
Start time: 9/22/2021 at 10:30am EST.
Root Cause: Azure infrastructure
User Impact: Slower than normal call routing
Scope of impact: Calls status, queue status, live calls are working as expected. Only call routing is affected. Our team is exploring the cause of this. This is a degradation notice, not an outage.
Update 9/16 5:45pm EST: In working with Microsoft, it appears that there were some Microsoft Azure infrastructure updates, which corresponds to a significant increase in timeouts between our routing components. We implemented some changes to work around the effect of these timeouts until they are resolved by Microsoft Azure.
Update 9/16 6:15pm EST: It appears Microsoft Azure has now addressed the underlying infrastructure issue causing the timeouts.
Start time: 9/16/2021 at 11:45am EST.
Root Cause: Azure infrastructure
User Impact: Portal unavailable
9/13/2021 11:28 am EST After routine Microsoft Azure SQL maintenance, Azure SQL has become unavailable. We are working with Microsoft to restore connectivity.
9/13/2021 3:45 pm EST, service is up for call routing. Note: There may be some items for some customers that will resolve over the next hours as the system settles.
Scope of Impact: Any users trying to access the portal.
Start Time: 9/13/2021 at 11:28 am EST.
Root Cause: Microsoft Azure SQL
6/23/2021 Title: Portal service outage.
User Impact: Portal unavailable for several minutes beginning on 6/23/2021 at 4:47PM EST.
Current Status: Service was completely restored several minutes later at 5:02PM EST.
Scope of Impact: Any users trying to access the portal.
Start Time: 6/23/2021 at 4:47PM EST.
Root Cause: There was a resource exhaustion on web servers which caused them to become non-responsive.
Mitigation: We will improve the monitoring and scaling of our systems to prevent resource exhaustion.
Title: Portal service outage.
User Impact: Portal Unavailable for about 2 minutes
Current Status: Service has been restored.
Scope of impact: Any user accessing the portal.
Start Time: 6/15/2021 at 10:20am EST
Root Cause: Incident was caused by a spike in database traffic that went above our usage and scaling calculations. To resolve this, we have increased the scale of our database and increased the buffer in our scaling calculations.
Title: Portal service outage.
User Impact: Portal unavailable for a few minutes on 3/3/2021. More info: Portal unavailable and some intermittent cases of interrupted queue alerts. Current Status: Service has been restored. A fix to avoid this condition in the future is scheduled to be included in the next maintenance window.
Scope of Impact: Any user trying to access the portal.
Start Time: 3/3/2021 at 5:01pm UTC (12:01pm EST)
Root Cause: We found an increased amount of out of memory exceptions which we believe led to greatly decreased responses times resulting in a period of several minutes where the contact center portal was not responsive.
Workaround: Use Landis Attendant Console to handle and transfer calls. Free 30 day trial. Getting Started guide:
This is a Microsoft Teams issue. We are currently working with Microsoft to help them resolve the issue.
Workaround: Use Landis Attendant Console to handle and transfer calls. Free 30 day trial. Getting Started guide:
Portal:
Agent Panel:
Microsoft has added the ability for the Microsoft Teams administrator to enable/disable the recording audio notifications on the Teams compliance recording policy. The ability for Landis Technologies to control the recording audio notification has been depreciated.