Note: this is an illustrative case combining situations typical of the enterprise storage work we handle. For reasons of confidentiality, the details do not correspond to any specific client or to real dates. The aim is to show how we approach this kind of failure technically.
What happens when a RAID 5 loses two drives at once? In theory it should not survive: RAID 5 tolerates the failure of only one. This is how we approach rebuilding an array in that scenario, for a company whose dispatch management system became completely inaccessible.
The scenario: two drives, the same weekend
On a Friday night, one of the drives in a four-drive RAID 5 began to fail. The system carried on operating in degraded mode, exactly as it is designed to. In the early hours of Saturday, a second drive went down during an electrical storm. The array was completely offline, and with it the company's routing, inventory and invoicing system.
Why this is not solved with software
RAID 5 distributes a parity block across all the drives in the array. That parity makes it possible to calculate exactly what was on a missing drive: it is a mathematical operation, and it works.
When two are missing, that operation no longer has a solution. It is not that it is difficult, or that better software is needed: the information required for the calculation simply is not anywhere any more. That is why rebuilding a RAID 5 with two failed drives does not begin at the computer, but on the laboratory bench. The question stops being how to rebuild the array and becomes how to get one of those two drives reading again.
Before touching a drive: gathering the information
The first step is not to open anything, it is to understand what happened. And a good part of that answer is usually written in one of the system's own logs. Wherever possible, we extract the logs from the RAID controller or the NAS. That information, when it exists, answers questions that would otherwise have to be guessed at blind:
- The order in which each drive went down, and how far apart. That is the fact that determines which one the recovery effort should concentrate on.
- Whether the array had been running in degraded mode, and for how long, which tells us how out of date the first drive to drop out had become.
- Whether someone already attempted a rebuild after the second failure. That single fact changes the prognosis of the case completely, and it is far better to know it before starting than halfway through.
- The error counters of each unit, which tell us in advance which one will be hard to read.
This gathering is done together with the client's infrastructure or systems team, who have access to the management console, know the history of the equipment and know what was tried before we were called. It is not a formality: it is the difference between working with information and working by elimination.
When the logs are not available, because the controller died, the array was reconfigured, or the machine was switched off without leaving a trace, the same analysis has to be reconstructed afterwards by examining the structure of the data in the clones. It can be done, but it takes longer and yields fewer certainties.
Not all failed drives are worth the same
In almost all of these cases the two drives did not fail at the same moment. One failed first and the array carried on in degraded mode, sometimes for days, without anyone noticing. The second went down later, and that is when everything stopped.
That means the first drive to fail holds old data: from the moment it left the array, the system carried on writing without it. The second, by contrast, was active until the last minute.
So the work is not divided equally. With the system logs in hand we establish the order in which they failed, and we concentrate the recovery effort on the second one, which holds the current information. Without logs, that order is deduced afterwards by comparing the data structures of each clone. Recovering the wrong drive can hand back an array that mounts perfectly but with data from a week ago, which for a dispatch operation is nearly as useless as having nothing at all.
What recovering access to a drive actually means
It depends on why it died. There are three families of failure, with three different interventions.
Electronic failure. The controller board burned out, usually from a voltage spike. It is replaced with one from a donor drive, but swapping it is not enough on its own: each board stores the calibration parameters of its own drive, unique from the factory. That information has to be transferred to the new board, or the drive will not recognise its own platters.
Firmware failure. The drive powers up and spins, but does not present itself to the system, or reports zero capacity. The problem lies in the service area, a reserved region of the drive that is not accessible by ordinary means. This is worked on with specific equipment that talks directly to the manufacturer's firmware.
Mechanical failure. The most delicate of the three, and the one that means opening the drive.
Head replacement in a clean room (laminar flow chamber with positive pressure)
When the read heads fail there is no shortcut: the drive has to be opened and they have to be replaced. And that can only be done inside a clean room (laminar flow chamber with positive pressure).
The reason is one of scale. The heads of a hard drive do not touch the surface: they fly over it at a distance of a few nanometres, held up by the very current of air the platter generates as it spins. A speck of ordinary dust is hundreds of times larger than that flying height. If a single one gets in while the drive is open, then on the next start-up the head drags it against the magnetic surface and scratches the platter. A scratched platter is not recoverable: the information that was there stops existing physically.
The chamber solves that with two complementary mechanisms. Laminar flow circulates filtered air in a single direction, constant and without turbulence, so that any particle is swept outwards instead of hanging over the open drive. Positive pressure keeps the inside at a higher pressure than the surrounding room, so air always flows out through any opening and unfiltered air never gets in.
Sourcing the right donor is not a purchasing formality either. An enterprise drive from ten or fifteen years ago is no longer manufactured, and rarely turns up on the local market. That search, identifying which unit will serve and where to obtain it, can be the slowest part of the whole process, and it is invisible to the client, who only sees that the case is in the laboratory.
Inside the chamber, the procedure is:
- Locate a suitable donor drive. Not just any unit of the same model will do: it has to be compatible in the characteristics that allow its heads to read the platters of the damaged drive.
- Remove the head assembly from the damaged drive with specific tools that keep the heads separated from one another. If two heads touch, they damage each other and become unusable.
- Take the heads from the donor and fit them into the original drive, without letting them brush the surface of the platters at any point.
- Close it up and test. If the drive reads again, it goes straight into cloning.
That last point is not a minor detail. A drive with transplanted heads has a short and unpredictable working life: it may read for hours or fail within minutes. That is why nobody explores it or browses around to see what is there: it is cloned in full from the first second, and all subsequent analysis happens on that copy.
Only now does the rebuild begin
With at least one of the failed drives reading again, the process continues:
- Bit-by-bit cloning of every drive, both the healthy ones and the recovered one. From here on nobody touches the originals again.
- Determining the array parameters: drive order, stripe size, parity rotation and start offset. When the controller died or was reconfigured, those parameters are documented nowhere and have to be deduced by analysing the structure of the data itself.
- Virtual reconstruction of the RAID on top of the clones, without writing to them even once.
- Extraction of the file system and validation against whatever partial backups exist.
- Integrity checks on the critical tables and files before handover.
The technical lesson behind the case
With two drives down, the most expensive mistake is trying to force the array from the controller so that it mounts again. That operation usually rewrites the existing parity, and destroys precisely the information that made reconstruction possible.
The correct order is the reverse, and it does not allow shortcuts: the laboratory first, the software afterwards. Recover access to the failed drive, clone it, and only then rebuild.
Services related to this article
