Metazoa has added some powerful new tools to Snapshot that move connected sets of data between Salesforce orgs. This capability is useful for backing up data, refreshing Sandboxes, merging orgs, and populating orgs with test data. Now Snapshot can move metadata to a destination org and follow up by migrating the actual data as well. Both of these transactions can be scheduled as needed.
When records are migrated between orgs, all of the internal relationships are preserved. External references in the Dataset are also connected to matching objects on the destination. You can also select the fields used for matching destination objects. For example, you might want to match Accounts by Name and BillingCity.
There are capabilities to map objects and fields to new names, and to scramble fields to obscure personal or financial information. You can also import Datasets from other systems with CSV files. Our new CSV file format provides a simple way to import, merge, and connect data to any destination org.
Snapshot uses the Salesforce Bulk API for this new feature to ensure that very large Datasets can be moved efficiently. We have tested Snapshot by moving hundreds of thousands of records. This blog presents an overview on how to build and migrate Datasets. Find more detailed technical documentation here:
https://www.metazoa.com/publications/dataset_migration_v3.pdf
Getting Started
Simply restart Snapshot and you will automatically get the new Data Migration capability. When you right-click a Deployment Arrow, the second set of options will have new commands to Build, Import, and Migrate Datasets. If you do not see these commands, then perhaps the Deployment Arrow is connected to a Developer Project. They do not have any actual data and cannot be used as a source of data or as a destination for migration.
The option to Build a Dataset will use the source Salesforce org to download multiple records in the form of XML files to your local machine. The option to Import a Dataset will use CSV files from any source to create a new Dataset as well. Lastly, the option to Migrate a Dataset will insert and update records from the selected Dataset into the destination Salesforce org.
Building Datasets
The first tab of the Build Dataset dialog allows you to select parent objects that you want to include in the Dataset. These records are available on the source Salesforce org. You can select all records, a subset of records by name, or a subset of records using a complex filter. The total number of downloaded records can be limited. This is useful for grabbing a random subset of records for acceptance testing or application development.
Selecting Children
The next tab allows the selection of connected child objects for each parent object. When a Dataset is created, the selected parent records are loaded first, followed by all of the children connected to that parent. You can specify multiple child objects in a hierarchy. The relationship field used to associate each parent and child is shown in parenthesis. The internal relationships between parent and child are always preserved when the Dataset is migrated.
Loading Fields
After that, you can select fields to load for each parent and child object. You can choose fields that need to be loaded by moving them to the list at right. Fields that cannot be created or updated on the destination do not usually need to be loaded. Removing unwanted fields makes your Dataset smaller in size and easier to migrate.
Build Datasets Button
The next tab allows you to enter the name of a new Dataset and then click the Build Datasets button at right to start the download process. All of the download results will be listed in the window pane at lower right. The Dataset will be saved in XML files to your local machine. Snapshot is an extremely secure solution, because we never move your data or metadata to any third-party cloud or Salesforce org! The last tab allows you to schedule the creating of a Dataset at a specific time in the future or as a recurring event.
Importing Datasets
There is another way to create a Dataset. The Import Dataset command is right under Build Datasets when you right-click a Deployment Arrow. When you Import a Dataset, you can select any number of CSV or XML files and add them to the list at left. The interface will show the imported fields and source records in the window panes at right.
Snapshot has introduced a new CSV file format that simplifies the process of merging and connecting CSV data with any destination Salesforce org. Check out our technical documentation for all of the details on how to prepare CSV files in this format:
https://www.metazoa.com/publications/dataset_migration_v3.pdf
Mapping Fields
The Mapping Fields tab provides an easy way to make sure that the imported object and field names match up to the object and field names in the destination org. First, select the desired destination object name at top. Then go down the list and select matching field names. The same technique applies to matching external references and fields.
Finish Importing
The last tab allows you to enter the name of a new Dataset and then click the Import Datasets button at right to start the import process. If you select an existing Dataset name from the menu then that Dataset will be replaced. When you want to migrate that data to an actual Salesforce org you will need to use the Migrate Dataset dialog, explained in the next section.
Migrating Datasets
After a Dataset has been created, you are ready to migrate these records to a destination Salesforce org. Right-click a Deployment Arrow that is connected to the correct destination org and select the Migrate Datasets option to get started. The Migrate Datasets dialog allows you to select any of the global Datasets from the list at left and see the objects and fields that are available in the list at right.
Matching Fields
Snapshot lets you select the fields used to match records on the destination org. One powerful way to match destination objects is with External Ids. Other common matching fields include object names, email addresses, and usernames. Some Salesforce sandboxes have the same Ids as production orgs. In that case you can simply use the Id field for matching destination objects.
Scrambled Fields
Datasets are often used to move records into a Salesforce Sandbox or Developer Edition for testing or application development. In these situations, you may want to scramble data records that contain sensitive information. These fields might contain financial information, such as credit cards or bank accounts, or personal information, such as email addresses or Social Security numbers. The Scramble Fields tab provides an easy way to select fields that you want to obscure in the destination org.
Deactivate Assets
When a Salesforce record is inserted or updated, various Apex Triggers, Workflow Rules, and Validation Rules might be invoked. These automated behaviors can cause potentially undesirable effects during data migration. For example, thousands of emails might be sent out, or some records might not be updated. The Deactivate Assets tab provides an easy way to deactivate Apex Triggers, Workflow Rules, and Validation Rules in the destination org before data migration is attempted. After migration, the deactivated triggers and rules will be turned back on.
Migrate Datasets Button
The next tab has the main interface for migrating datasets to the destination org. First, make sure that the migration options are set correctly. Then click the Migrate Datasets button to get started. If any errors are encountered, they will be written to the Log Files along with the Salesforce record Ids.
There You Have It!
We are stoked that Snapshot can handle Datasets as well as Metadata. This is a great new product addition. Please give this capability a test drive and let us know how Datasets are working for you.
Register for our Webinar on December 12th at 10:00 a.m. PST to see it in action.
Congratulations! 