In this webinar, our experts showcase a variety of demo use cases of how different components of the...
You can speed up your data migration using the new Scribe Insight Adapter for Dynamics CRM, Version 5.4.0. How? By doing a bulk operation! The new adapter comes with Scribe Insight 7.6, which now allows you to perform bulk insert, update, and delete operations.
A bulk operation processes a block of rows in one transaction, minimizing the number of calls between the Scribe source and the CRM destination. The block of rows (as specified in the screenshot below) is sent in a bundle once to CRM for commit.
With row-based processing, each row needs to be committed before it sends another. This does not mean that if you set number of rows per operation to 1000 that it will be 1000 times faster than row-based operations. Scribe does not work that way yet! There are a number of factors that determine how many records are pushed in one transaction, including type of connection, cross reference keys, update source, or rejected rows.
Here is how you enable bulk operation in a DTS package step:
You are all set!
There are certain things you'll want to consider before you use bulk operations in your Scribe package. Some of the very handy features of Scribe becomes unavailable when we use bulk operation. Be aware of the following:
Speed is the primary reason why we look for bulk operation. If the DTS steps are simple and need to process a large number rows (say in billions), then it is definitely a good idea to use bulk mode.
Below are performance comparison statistics compiled by PowerObjects using Microsoft Dynamics CRM Online:
Operation type | Number of threads (packages running in parallel) | Operation | Number of rows per operation(bulk) | CRM Server | Rows per minute | Result | |||
Row-Based | 1 | Insert | N/A | Online | 440 | ||||
Bulk Mode | 1 | Insert | 700 | Online | 2317 | 5.3 X Row-Based insert | |||
Row-Based | 1 | Update | N/A | Online | 170 | ||||
Bulk Mode | 1 | Update | 700 | Online | 1022 | 6.0 X Row-Based update |
Note: These are not Scribe's official numbers. These are the numbers that PowerObjects got when we tested this new feature.
The focus here is not on how big these numbers are—the actual numbers can vary based on number of processes running in parallel, number of fields updated, and so on. Instead, look at the difference between the rows per minute for these two types of operations. Our test shows that the bulk insert operation is at least 5.3 times faster than row-based inserts. Similarly, bulk update is 6 times faster than row-based updates. In either case, the results clearly show that bulk operation outperforms row-based operations by multiples of 5 or more.
In conclusion, you can get more performance using bulk operation than normal row-based operation. At the same time, bulk mode has some limitations. So, to utilize Scribe Insight's bulk mode feature of the new Adapter for Dynamics CRM 2011, consider simplifying the Scribe package steps. You can mix and match row-based or bulk mode steps in a single package.
Bulk operation is a very powerful feature and can be best utilized in migrating/integrating to CRM online where the number of rows processed is large.
Happy migrating and integrating to Dynamics CRM!
If you implemented Microsoft Dynamics CRM 2011 Update Rollup 12 and Microsoft Dynamics CRM December 2012 Service Update releases, you should be able to use the Dynamics CRM Bulk API. With this option checked, Scribe is capable of processing more than 11500 records per minute on a on-prem implementation. Provided you run multiple threads in parallel and also keeping the steps simple.
If you implemented Microsoft Dynamics CRM 2011 Update Rollup 12 and Microsoft Dynamics CRM December 2012 Service Update releases, you should be able to use the Dynamics CRM Bulk API. With this option checked, Scribe is capable of processing more than 11500 records per minute on a on-prem implementation. Provided you run multiple threads in parallel and also keeping the steps simple.
Anybody else had the issue where when using Bulk mode on a very simple update (one lookup one, data link) I get cannot find matching row errors (all rows) but in 'standard' mode every row succeeds? No other changes just ticking the bulk mode option