Accidents Happen
On May 17, 2019 Salesforce deployed a database script that inadvertently applied the “Modify All Data” permissions to Orgs that have a Pardot license. In other words, some of the users in these Orgs were accidentally transformed into system administrators. As a result of this, access was shut off to all NA and EU instances until the problem could be corrected.
By May 18, 2019, access for users with a system administrator profile had been restored to the affected Orgs. But now these system administrators needed to restore proper access for their other users. Salesforce suggested that Profiles and Permission Sets could be restored from a Sandbox if possible. You can read the security incident report here.
However, some of the Sandboxes were also affected by the problem. If all the Standard Object Permissions on a Profile are unchecked, then the Sandbox was not a valid source for recovery. We should also note that Sandboxes are used to develop new software and may not contain an accurate backup of Org permissions anyway. The Sandbox configuration might be different from the production Org.
If the Sandbox did not contain a valid copy of the Object Permissions, then administrators were advised to correct the problem manually. To put this in perspective, a large Salesforce Org might have 500 Custom Objects and 500 User Profiles. Restoring a quarter million Object Permissions by hand with the Setup Menu might prove difficult for some administrators.
If you are lucky enough to have a Sandbox with a valid copy of your Object Permissions, then Salesforce recommends using the Ant Migration Tool, Force.com IDE, or Change Sets to restore them. However, Change Sets will not work for Standard Object Permissions, so administrators would need to be familiar with the recommended developer tools to restore the permissions.
Disaster Recovery
Looking at this situation in retrospect, we can draw a few conclusions that every Salesforce administrator should think about:
First, depending on a Sandbox for disaster recovery is risky. The Sandbox configuration might be outdated. In this case some Sandboxes were also affected by the problem.
Second, this incident could have been more serious. Profiles and Permission Sets were affected, but there are 200 other Metadata types to worry about.
Third, Salesforce cannot always save you from Metadata loss and the need for disaster recovery. Administrators need Metadata backup and restore capabilities.
Administrators that use our Snapshot product would be in a much better position to evaluate the impact of the recent service outage and correct the problems than other companies without disaster recovery software. Snapshot provides a very effective way to backup time-series Org Metadata on a regular basis, compare the backup against any Org, and then restore all or part of the Metadata as needed. Even customers that were not affected by the outage could verify that their permissions and all other settings were not changed.
Administrators use Snapshot for Org cleanup and optimization, as well as security and compliance. Snapshot also provides other developer tools for working with Salesforce DX and performing Continuous Integration. The Metadata backup and restore capabilities are simply a core competence for Snapshot. However, the recent service outage shows that this is a very important part of our toolkit for system administrators.
Congratulations! 