Our Snapshot product has over 40 special purpose reports that detail the complexity found in almost any Salesforce Org. The Data Dictionary Report documents Custom Objects and Fields with all of the information available from the Data, Metadata, and Tooling API. The Profiles and Permission Sets Report shows the inner structure of Org permissions and helps identify security problems. But what about the other 200 Metadata Types? Some of these asset types are both very complex and quite important: Custom Applications, Flows, Reports, Dashboards, and Workflows all come to mind. We have specific reports that cover certain use cases, but we need general purpose reports for everything else. The new Org Dictionary report was designed to tackle this challenge. This blog post outlines some of the trials and tribulations we encountered building this technology.
Can’t Touch This
In order to create the report, we needed the ability to generate example XML envelopes for each Metadata Type. We couldn’t just look at the documentation, there are too many errors there. And we couldn’t just look at the actual XML retrieved from an Org, either. Depending on the shape of the Org, the downloaded XML might or might not contain all of the information we needed. The sheer size of the job was also daunting. There are roughly 200 different Metadata Types with over 6000 different named elements. We had to understand the generic structure of each Metadata Type before an Admin could assemble any type of report
WSDL To The Rescue
The best approach was to study the Metadata WSDL. This information can be downloaded from the Setup Menu. Unfortunately, if you have ever looked at WSDL before, these documents are difficult to understand, and this one was over one gigabyte in size. But by generating example XML envelopes from the WSDL and comparing that against known Metadata and the documentation we were able to generate example XML for almost all of the different Metadata Types. We were also able to identify the data type and enumerated values for each element.
Work In Progress
Another problem was that the Metadata WSDL depends on the Org shape to some extent. Because we could not find Orgs with certain features, there are a handful of Metadata Types that we were unable to define. For example, there are some new Metadata Types associated with Salesforce C360, but this product is not yet widely available. If a Metadata Type that you need is not available in the Org Dictionary report then please send us your Metadata WSDL and we can add that asset type to the list. As new Salesforce products become available over time we will add new Metadata Types to the Org Dictionary report.
The Really Big Picture
Once we were able to generate the XML structure of all the different Metadata Types, an interesting picture emerged. By looking at the number of elements per asset type we were able to visualize the overall complexity of all Salesforce Metadata. The chart below shows all of the Metadata Types that have more than 20 XML elements. The chart is shown with the simpler types at the top and the most complex types at the bottom. This chart provides an incredibly high-level view of Metadata complexity.
As you can see, Object Translations, Workflows, Page Layouts, Dashboards, Reports, Custom Objects and Flows are among the most complex Metadata Types. Flows are by far the most complex, with over 2000 different element names in that one structure. Flows are more than twice as complex as Custom Objects. Because of this, we truncated the horizontal axis of the graph above for the sake of presentation. Since some of these elements define arrays, any given instance of an asset could be very large in size. For example, Profiles only have about 100 different element names, but because they contain so many arrays, they can be among the largest assets in any Org.
Displaying XML Structure
Another challenge was how to display complex assets with a hierarchical XML structure in a two-dimensional report table. For example, here is the Role asset type. There are no arrays, so all the Roles stack right up in a linear fashion.
But if you have an asset like Custom Objects, there are sub arrays like Fields, and sub-sub arrays like Picklist Values, and so on. There are over 40 different types of arrays just in Custom Objects alone. If you display everything then you will probably be overwhelmed by the complexity of the resulting report. Because of this, the Org Dictionary allows the user to pick the elements to display and adjust the column order.
Here is the report table below that was generated for a Custom Object with Fields and Picklist Values. Note the labels at the top of the picture that show the structure of the nested arrays. This naming convention is similar to XPath, for you developers out there. This example was chosen to illustrate how hierarchical structures are rendered. Obviously, the more specialized Data Dictionary report could provide more information on Custom Objects.
The New Org Dictionary Report
After the WSDL work was completed, the new Org Dictionary report was pretty easy to for the engineering team to build. The interface is similar to other Snapshot reports. The different Asset Types are listed in the first column. When one is selected, the WSDL structure is displayed in the center column. Administrators can select these elements and move them into the report column on the right. We gave each asset type a nice default report to help get things started. All you need to do is select any additional fields and preview the results to be sure the report is what you wanted. Now you can document any Metadata Type in your Org!
There is an option to hide and show the camel case names in the report labels. These names match the Metadata API documentation, but might be difficult for someone like an auditor to understand. There is also an option to hide and show the element data types and enumerated values. This information helps explain the purpose behind each field so the report can be assembled quickly. There are some other nifty advantages to having this information that are discussed below.
<pclass=”blog-p”>
Friends With Benefits
There have been some interesting side effects to our research effort exploring the Metadata WSDL. Now Snapshot has the ability to really start dealing with interior characteristics of different Metadata Types in a more comprehensive manner. Developers often like to edit XML by hand. Now Snapshot can validate these documents and highlight any errors. Salesforce Administrators sometimes make changes to the Snapshot file before deployment. Now Snapshot can suggest the proper data type and enumerated values. This inside information simplifies editing operations and reduces errors.
For example, in the View Snapshot interface, an Admin might need to edit an Apex Class. If they edit the apiVersion, Snapshot will make sure that the new value is a floating-point number. If they edit the status, they are automatically prompted to choose a new enumerated value: Active, Inactive, or Deleted. This level of intelligence is possible because of the new WSDL database we developed for the Org Dictionary report.
Inside Baseball
I wrote this blog to show some of the thinking and research that goes into building new reports for Snapshot. We are trying to make very powerful capabilities very easy to use. Let us know if you like the new Org Dictionary report! If this is the type of report that you would like to run on your Org then please check out Snapshot on the AppExchange.
Congratulations! 