Keeneyville School
Restoring Azure AD Sync After a Critical Outage
Recovered Azure Active Directory synchronization and global user access after a soft deletion incident locked out every user, including administrators, with no valid Global Admin account available.
While extending on-premises Active Directory attributes to Azure Active Directory for application authentication, a series of mistakes triggered a soft deletion of Azure AD. Every user, including administrators, was locked out of Microsoft services and cloud apps, the Azure AD Connector lost synchronization, and there was no valid Global Admin account to access the Azure portal.
The Challenge
The attempt to extend user attributes disrupted Azure Active Directory, causing a global lockout from Microsoft services. The Azure AD Connector lost sync between on-premises AD and Azure AD, and without a valid Global Admin account the portal itself was inaccessible, ruling out the normal recovery paths.
The Solution
The only viable path was to restore the synchronization server to a previous working state. Online protection restore was blocked by expired vault credentials that could not be regenerated while locked out of the portal. As an alternative, the disk option in Microsoft Azure Backup Server (MABS) restored files to a known-good date. We then manually restored the Azure AD Connector folders, including the SQL Server components, onto the sync server.
Resolution
After restoring the folders, the sync service would not run because its database had become read-only for the user. Recognizing that the Azure AD Sync service runs under the NT AUTHORITY\SYSTEM account, we granted that account access to the relevant folders. The synchronization service started successfully, the connector came back online, and synchronization between on-premises AD and Azure AD was reinstated, restoring user access to Microsoft services and cloud apps.
Key Takeaways
Maintaining valid credentials and break-glass Global Admin access is essential for emergencies. Regular backups and a defined restoration process are critical for recovering from accidental disruptions. A systematic approach grounded in an understanding of the system architecture solved a complex, high-pressure incident.
Tech Stack
Have a Project Like This?
Tell us what you are trying to build or fix. We design the integration, write the cloud, and stay on as your engineering team.