With many Salesforce Orgs hitting the 10-year mark, the crippling effects of technical debt are becoming more common. Technical debt can result in slow performance, reduced agility, sluggish adoption, and the failure to meet business requirements. This complex problem is leaving Salesforce admins and IT executives alike wondering what to do.
To help combat the situation, Salesforce Premium Support now offers services geared towards tackling technical debt. As part of this, Salesforce recommends that customers devote a percentage of each release cycle to clean-up work. But while everyone agrees that technical debt is bad, there hasn’t been much discussion regarding exactly how technical debt reduces Org performance and agility. Without understanding this, it is difficult to know how to fix the problem or avoid it in the first place.
In software development, technical debt is often defined as the cost of choosing an easy solution now instead of a better approach that might take longer. Salesforce analysts like to associate technical debt with the complexity that results from extensive customization. The argument is basically that customizations cause complexity, and that results in technical debt which reduces Org agility.
At Metazoa, we don’t think that either of these definitions explain what is happening. First of all, Salesforce Orgs need to be highly customized in order to address important business requirements. Complex Salesforce Orgs can work just fine if they are properly managed. And secondly, we have found that the loss of agility and performance in older Salesforce Orgs can often be traced to specific malfunctions that can be fixed. We do not think the problem is too many customizations, but rather vital systems in the Org that have become unhealthy.
When customers come to us with technical debt issues, they often think that the only option is to clone the Org and start over. In many cases, common developer tools will no longer work on their Org, and various Salesforce APIs fail to respond in a timely fashion. These customers cannot transition to Lightning Experience or take advantage of other new Salesforce products. However, instead of starting over, the first step in removing technical debt is to identify which systems need repair.
Org Visualization
To get a good sense of where the problems are, you must first get an asset count for each metadata type. The List Metadata API can do this rather easily. However, this API will only return the first 100,000 assets. If you exceed that limit, you will need to get smarter and split metadata types across API calls to get the desired results.
Once you have an asset count for each metadata type, you then need to take a comprehensive metadata snapshot of the Org. This can be difficult because the Retrieve Metadata API has some governor limits. By conducting multiple retrieves in parallel, and stitching the results together, millions of assets can be assembled. This process can take some time because of Org size and API response times.
Once an overview of all the Org metadata is available, many problems will become apparent. Why are there 4000 inactive Assignment Rules? How did we end up with 7000 Groups? During this process, there will be business value judgments involved, not just technical issues. For example, you might need to consider how the Org should be structured to achieve corporate requirements.
To answer these questions, reports can be assembled to help administrators visualize complexity and understand where the bottlenecks are. Metazoa provides some great free reports on the AppExchange that can help, including Profiles and Permissions, Field and Picklist Usage, and the Data Dictionary Report. You can also create your own reports with the Metadata API and the Tooling API. Compliance reports can be used on the front end to identify the problems and on the back end to document the changes to the Org.
Profiles
One of the most serious problems is gigantic amounts of Profile data. Let’s say you have an Org with 500 Profiles and 1,000 Objects and 500 fields per Object. This Org will have 250 million field level security (FLS) permissions. Each one of these takes up about 250 bytes of metadata. If you download the profiles, the file size will be approximately 60 GB! That’s a ton of data, and it shuts down most developer tools like Salesforce DX, Visual Studio Code, and Workbench.
The data size is not the only problem. The Profile page in the Setup menu will take several minutes to load, and when the page finally does load there will be an overwhelming number of permissions displayed which introduces security compliance delays in addition to implementation issues. The best practices for resolving these issues include:
- Evaluate aggregate org security permissions
- Identify and remove underutilized Fields
- Identify and merge similar Profiles
- Transition to Permission Sets
Permission Sets
Each assignment between a Permission Set and a User is stored as an internal record in Salesforce. These assignments control access to Objects, Fields, Apex Classes, etc. If there are many Users and many Permission Sets, then the total number of assignments can become very large. There can be great difficulty documenting the combined effect of all these permissions for an individual User. Editing all these assignments through the Setup Menu can be tedious and error prone.
When you assign Permission Sets to a Permission Set Group, there is a background process that calculates the new aggregate permissions for the group. This recalculation process can cause performance problems and delays in Orgs with lots of active assignments. The best practices for resolving these issues include:
- Evaluate combined security permissions by User
- Identify and merge similar Permission Sets
- Reduce the number of assignments with Permission Set Groups
Unused Assets
There are several different metadata asset types that can end up being left behind because of Org configuration settings. For example, maybe the asset is not referenced by any other asset. This can happen with Groups, Queues, and Email Templates. Perhaps the asset is not assigned to any user. This can happen with Profiles, Permission Sets, and Roles. In addition, assets can be forgotten because they were deactivated or hidden. This can happen with Workflows, Assignment Rules, Approval Processes, and Flows.
There are analogous problems on the data side, where Fields and Picklist Values can be underutilized. Reports and Dashboards can also go stale. Admins should take advantage of the compliance information that Salesforce provides for each Custom Field. This information includes the Description, Help Text, Business Status, Security Classification, and Compliance Group. Special attention should be given to the date the asset was created and last modified, and which admins were involved. Deleting all these unused assets is a great first step towards Org cleanup and optimization!
Inactive Users
Once a User is created, they can be made inactive, but they cannot be deleted. Over time, Orgs can end up with more inactive users than active ones. There are over 50 different ways that an inactive user can remain connected to a Salesforce Org. Inactive users can still manage active users, receive system emails, remain team members, and serve as the running user for Dashboards and other important enterprise systems.
These zombie users complicate Org management and software development. They also pose multiple security threats. Tracking down and removing all the connections between inactive users and the Salesforce Org can be a very complicated process. The connections can be by username, user email, and user ID. The best practice is to identify these Users and either delete their connections to the Org or replace them with an active user.
Record-Level Security
Another common problem within Salesforce Orgs is simply having too many assets of some metadata type. We once worked with a customer that had over 100,000 User Roles. Sometimes having that many Roles is unavoidable due to Portal or Experience Cloud setup, but other times it’s just technical debt.
When the Role hierarchy changes, this triggers a recalculation of Record-Level Security. As you can imagine, this customer’s Org suffered from massive performance issues due to constant security recalculations. In this case, the answer was cleaning up the Role hierarchy and then reassigning all Users to the correct Role. Salesforce admins should pay special attention to changes that trigger Record-Level Security calculations. These include:
- Role Hierarchy
- Territory Hierarchy
- Sharing Rules
- Team Membership
- Manual Sharing
- Programmatic Sharing
- Groups and Queues
Second Generation Packages
Older applications are often unpackaged, and they tend to directly customize various aspects of the Org and add lots of unnecessary complexity to it. To remedy this, release managers should dedicate some time in each release cycle to packaging these legacy projects. Packages encapsulate functionality, isolate bugs, simplify releases, improve testing, and provide a foundation for agile development.
The great thing about managed packages is that they add an additional layer of separation with namespaces. Unmanaged packages can accomplish this as well with strong naming conventions. Developing packaged applications is the wave of the future, but even the most advanced DevOps practice can’t succeed if the Org is saddled with technical debt.
The Healthy Org
An analogy with healthcare is helpful. If your doctor gave you and MRI, and said that there were many connections in your brain, and that your brain was quite complex, would you think that was a medical problem? Or a sign of intelligence? On the other hand, if the doctor said some of your internal organs were abnormally large, or that your white blood cell count was too high, then you would be worried. At Metazoa, we have analyzed hundreds of Orgs, and we have found that the Orgs with technical debt tend to be less complex than healthy Orgs. They are dead in the water, not able to adopt new Salesforce features, and in need of repair. The conventional wisdom that technical debt is caused by too many customizations is simply wrong.
Conclusion
Salesforce has provided many powerful options for customizing an Org and solving important business problems. But without proper management, a dynamic Salesforce Org can become crippled with technical debt over time. But don’t worry, technical debt is a problem that can be fixed. Fortunately, there are some powerful tools available that can both find technical debt and fix the problems as well. Please check out our Org management product Snapshot on the AppExchange if you need to tackle technical debt!
Congratulations! 