The Situation
A large North American based manufacturer of building materials experienced a serious data loss due to a corrupt database file on their Commvault Media Server.
The client’s data storage solution consisted of the Commvault server as well as a Media Agent under the Commcell® management system. A Commvault Simpana® 9 was used for backup and archiving. The company lost access to media files crucial to the operation of their business.
The client attempted to open one of its virtual tape library backups only to find the MediaAgent database file, the gateway to the files on the media server, was corrupted and showed zero bytes in size.
The virtual tapes and the files stored inside the media server were still intact, but were not able to be accessed. The client contacted Commvault support and they attempted to retrieve the missing data from the media agent volume residing on the deployed Dell® MD1200 storage using the Commvault Media Explorer Tool. Their efforts proved to be unsuccessful. More than 3500 virtual tapes and 25 tape-sets were lost resulting in over 230 million inaccessible files. Fortunately, the client was able to copy the entire volume from the Dell server to external disks by using the Commvault Simpana 9 and sent them to Ontrack to recover the data. Unfortunately, the metadata containing the shortcuts leading from the database towards the files was not transferred with the media agent volume; therefore the files still could not be reached. Intensive investigation was necessary in order to discover the data mapping.
The Solution
Ontrack engineers were able to rebuild the original structure of the virtual array with the information contained on the external hard disks.
To reconstruct the data mapping used by the Commvault system, Ontracks internal software development team researched how Commvault distributes files, over the whole storage system and the built-in disks. After determining the custom mapping used by Commvault and decoding the virtual array, another challenge arose when the engineers discovered the virtual tape backup sets were also de-duplicated. The Ontrack software developers and data recovery engineers were able to create custom tools to
read the files from the system, rebuild the missing catalog and mapping information, and gain access to the file data. The engineers were then able to restore the files upon extraction and remove the de-duplication.
The Resolution
In the end, Ontrack engineers were able to recover the missing tape libraries as well as tape-sets with the 230+ million files included in just a couple of weeks.
The client was amazed at the success and speed of the recovery. With the insights gained into the Commvault system and the newly developed tools, the data recovery specialists at Ontrack are able to retrieve data even faster from all Commvault-based systems.