The Situation
The customer was in the process of acquiring another company, which had most of its systems virtualised. When the company’s users left the office on Friday all of the systems were operational and when they came in Monday morning the users were not able to access the virtual machines.
The client contacted the HP and VMware support teams and after an initial investigation discovered that the virtual machines and their snapshots had been deleted from the source volumes. The volumes containing the backups had been initialised - overwriting the backup files.
The customer was then referred to Ontrack for data recovery.
The Solution
The company suspected foul play so the KLDiscovery security team was dispatched to the customer location to begin an investigation during the data recovery. The security team locked down the data centre and installed monitoring systems, then started a full investigation into the suspected breach. KLDiscovery Background Screening was also brought in to do a background check on suspected individuals.
The team soon determined that an individual had logged into a system from outside the company and then connected to a series of systems. Once the individual reached the VMware server, the individual intentionally and maliciously deleted the virtual disks and backups. This individual was able to access 27 out of the 28 logical unit numbers (LUNs).
The customer had an initial consultation with Ontrack and was offered a Remote Data Recovery solution. They connected multiple machines and upon connection, Ontrack teams were assigned to perform an evaluation of the LUNs provided. The Ontrack engineers confirmed that the data had been deleted from the volume. They then proceeded to identify all of the recoverable data on each LUN and provided the customer with reports detailing the data.
The evaluation was very challenging as the client had limited documentation, unknown virtual machine names, unknown content on the virtual disks and unknown operating systems. They also did not have a disaster recovery plan in place for this system, so it had changing priorities during the evaluation.
Having identified the sheer scale of the problem, which meant that the company was prevented from delivering vital prescriptions and other medical supplies, Ontrack constructed a virtual team based on resources from eight data recovery centres around the world. Upon review of the provided reports and approval from the customer, the Ontrack teams extracted the deleted data and as needed extracted the contents of corrupted virtual disks to new storage provided by the customer.
The Resolution
The lead system administrator from the smaller company left his employment before talks of the merger happened - taking all of the system information with him. The company was forced to rehire him as a contractor to perform a knowledge share with the remaining employees.
During the time he was a contract employee, the lead administrator set up a rogue vCenter on the company’s network without the knowledge of any of the other employees. He then finished his contract and left the company without incident. Shortly after he left the company for the second time, the business went through a merger with a larger pharmaceutical. During the merger, some employees were laid off including the lead administrator’s friend.
During the KLDiscovery security investigation, investigators found an unuthorised entry from outside the network in one of the router logs. The entry led them to the rogue vCenter and from there, after working with VMware support, they determined that it was the system used to delete all of the virtual disks and their snapshots.
The KLDiscovery team then contacted the FBI and working with their team, they were able to determine that the external IP address belonged to AT&T. The FBI contacted AT&T and learned that the IP address was registered to a McDonalds. The FBI sent field agents to the McDonalds and found evidence that one of their suspects had been there the day the incident occurred. Once the field agents had the evidence, they confronted the suspect and he confessed to the crime.
From written accounts and court records, the team learned that the lead administrator decided to get revenge for the layoff of his friend and teach the smaller company a lesson. One Sunday morning, he got in his car, drove to the McDonalds in question, purchased breakfast with his credit card and then logged into the public WiFi. From there he made a connection to the rogue vCenter and proceeded to delete all of the virtual machines as well as the backups.
Australia | English
Locations
Search
Back to listing
