Hello,

I was wondering how other people are importing data to asset center daily when the number of computers scanned daily by the enterprise discovery systems (eq. Tivoli Inventory, or HP DDMi) is over 30-40K? I wrote a solution for it because our asset management team had a serious performance problem with importing such a large amount of data (we have over 12 DDMI servers) and was wondering how other developers approached it?

My solution is as follows

Quote Originally Posted by Funks
We had a similar problem, loading similar data from Tivoli Inventory (and now DDMI 7.5). With the amount of CI changes were were importing, our Connect-IT scenario took more than a day to process 30K CI's hence it was in an eternal backlog. We noted that the extract and transform phase can easily be ran in parallel which is limited/non-existant at best in Connect-IT. We wrote a JEE5 application that pulled data and transformed it to XML in parallel (think multiple threads) then submitted the complete per CI record to a message broker for delivery to our Connect-IT server (whose job at that point is strictly loading). We made sure that we calculated a crc / record checksum so that Connect-IT will just ingore those records that have the same stored checksum in the Asset Center Database.

This dropped our daily processing time from +1 day(s) with hardware only, to around 6 hours (w/ hardware AND software ). A profound improvement primarily due to parallelism, the avoidance of multiple db calls to the inventory system's database per CI (as all the info needed to update the AM record is in the XML file - processors, networkcards, drives, partitions, videocards, and SAI software), and the checksum support.

Another good thing about this approach is that we can horizontally scale our Connect-IT servers by interleaving the messages read from the JMS Topic through a JMS message selector.

Only drawback with this approach is that it brings about more complexity - nevertheless, once a solution is at limits, it usually requires a different strategy.
Other people stated that changing the table counter / key generation strategy (stored procedure used by asset center) to a sequence improves Connect-IT loading performance considerably.

Any others?


I was thinking about open sourcing the solution (GPL V3) I wrote and sharing it with the rest who have the same demands.

Thanks,
George