Horizon Cloud on Azure : Multiple alerts received (All Regions )
Incident Report for VMware Workspace ONE
Resolved
This incident has been resolved.

Microsoft Update:
Preliminary root cause: We identified that calls made during service management operations were failing as a required artifact version data could not be queried. Our investigation focused on the backend compute resource provider (CRP) to determine why the calls were failing, and identified that a required VMGuestAgent could not be queried from the repository.

The VM Guest Agent Extension publishing architecture was being migrated (as part of RDFE migration) to a new platform which leverages the latest Azure Resource Manager (ARM) capabilities

Mitigation: We mitigated impact by marking the appropriate extensions to the correct expected level (in this case, public). Engineers proactively verified the return to full success rate for operations after the updates were completed.

Next Steps: We will continue to investigate to establish the full root cause and prevent future occurrences. Stay informed about Azure service issues by creating custom service health

Thank you for your patience and apologies for any inconvenience caused.

Service reliability is a top priority at Horizon Cloud and we are making continuous improvements to better our systems.

Additional Information and Support:
Please visit Service Support Website at http://my.vmware.com or call Service Support Line at 1-877-4VMWARE if you need additional assistance.

Thank You,
VMware Horizon Communications Team
Posted Oct 13, 2021 - 08:19 EDT
Identified
Microsoft update: Microsoft have identified that calls made during service management operations are failing due to a required artifact version not returning as expected during query. The failure shows that that a required extension cannot be located. We are currently implementing a mitigation option to force a refresh of the extension and are seeing signs of recovery where mitigation has been deployed. We expect recovery to be observed as the mitigation progresses across regions.

Please check the Microsoft updated status using the below link.
https://status.azure.com/en-us/status.
Posted Oct 13, 2021 - 07:50 EDT
Investigating
Incident Description
The Horizon Cloud Monitoring has identified multiple alerts received in All regions. Currently, Microsoft is investigating the issue with high priority.

Impact details:
Starting at 07:00 UTC on 13 Oct 2021, a subset of customers using Windows Virtual Machines may experience failure notifications when performing service management operations - such as create, update, delete. Deployments of new VMS and any updates to extensions may fail. Non-Windows Virtual Machines and existing running Windows Virtual Machines should not be impacted by this issue.

Current Status: We are aware of this issue and are actively investigating the issue. The next update will be provided within 60 minutes, or as events warrant.
This message was last updated at 08:35 UTC on 13 October 2021

You may check the updated status using the link. https://status.azure.com/en-us/status.

Service reliability is a top priority at Horizon Cloud and we are making continuous improvements to better our systems.

Please visit Service Support Website at http://my.vmware.com or call Service Support Line at 1-877-4VMWARE if you need additional assistance.

Additional Information:
• VMware will provide additional updates on our Unified Status page (http://status.workspaceone.com) until this issue has been resolved.

Thank You,
VMware Horizon Communications Team
Posted Oct 13, 2021 - 05:19 EDT
This incident affected: Console Access (Horizon Cloud Admin Console) (Console Access (Horizon Cloud Admin Console) – AP Australia East (New South Wales), Console Access (Horizon Cloud Admin Console) – AP Australia East (New South Wales) 2, Console Access (Horizon Cloud Admin Console) – AP Japan East, Console Access (Horizon Cloud Admin Console) – EU UK South (London), Console Access (Horizon Cloud Admin Console) – EU North (Ireland) 2, Console Access (Horizon Cloud Admin Console) – EU North (Ireland), Console Access (Horizon Cloud Admin Console) – US West 2, Console Access (Horizon Cloud Admin Console) – US North Central (Illinois), Console Access (Horizon Cloud Admin Console) – Germany West Central (Frankfurt)).