Your RAID Array Is Degraded: 7 Checks Before You Start a Rebuild

One morning the controller interface, or a monitoring email, shows a single word: degraded. The array is still running, users have noticed nothing, and the replacement drive is sitting on the shelf. The natural move is to swap the drive and start the rebuild, and most of the time it finishes without trouble. This post is about the times it does not: why the rebuild is the most fragile moment in an array's life, and what to check before you start one.

What degraded means, and what it does not

RAID 5 tolerates the loss of one drive, RAID 6 of two. In the degraded state the array recalculates the missing drive's data from the remaining data and parity on every read. The data is still there; but tolerance has dropped to zero on RAID 5 and to one on RAID 6. On RAID 10 it depends on which drive failed: if the second drive of the same mirror pair goes, the array stops.

What the degraded warning does not tell you is this: the remaining drives are the same age, usually from the same production batch, and have spent years under the same load.

By level, the picture is this:

Why is a rebuild risky?

To calculate what goes on the new drive, a rebuild reads every remaining drive from start to finish. Sectors that have not been read for years in normal operation are read for the first time. Three things can go wrong:

1. Latent bad sectors. Manufacturer data sheets usually state the unrecoverable read error rate as at most 1 in 10¹⁴ bits for desktop-class drives and at most 1 in 10¹⁵ bits for enterprise drives. 10¹⁴ bits is about 12.5 TB. That figure is an upper bound and healthy drives usually do better; even so, meeting a single unreadable sector during an uninterrupted read of tens of terabytes is not unusual. On RAID 5 there is no redundancy left to correct that sector; the parity is already being used to calculate the missing drive.

2. A second drive failure. A rebuild is a heavy read load lasting hours, and on large drives days. A drive that is running on the edge may drop at exactly this point.

3. The wrong drive. If the drive pulled as faulty turns out to be healthy, you have lost two drives in one move.

Seven checks before starting a rebuild

The summary first; the detail follows the table.

1. Read the event log: which drive dropped, and when?

In the controller or operating system log, find the slot, serial number and time of the failed drive. What you are really looking for is this: has another drive dropped and come back earlier? If two drives dropped at different times, the data on the first one is stale. Forcing that drive online makes the array look healthy but leaves the file system inconsistent.

2. Verify the backup

"Backups are running" is not an answer. Check the date of the last successful backup and actually restore at least one folder or one database. If there is no backup, or it does not restore, take every following step as if that array were the only copy.

3. Check the health of the remaining drives

The remaining drives will carry the rebuild load. On SATA drives look at the reallocated sector (5), pending sector (197), offline uncorrectable (198) and interface CRC error (199) counters; on SAS drives look at the grown defect list and uncorrected error counts. If one of the remaining drives has pending sectors, the rebuild will very likely stall on them.

4. Confirm the drive you will pull by its serial number

Slot number, LED and operating system device name do not always point at the same drive; cables may have been reordered or the backplane mislabelled. Before pulling the drive, read its serial number and slot from the controller output, then use the controller's "locate" function to light that slot's LED and confirm the bay by eye. After pulling it, compare the serial number on the label; if it does not match, put the drive straight back into the same bay and do nothing else. Write the slot number on every drive you remove.

5. Check the hot spare and controller state

If a hot spare is defined, the rebuild may have started without you; watch its progress and the error counters. On hardware controllers check the state of the cache battery or capacitor and whether there is a "foreign configuration" warning. Do not import or clear a foreign configuration without understanding what it is.

6. If there is no backup, take the most critical data first

If the degraded array is still readable and you have no verified backup, copy the most critical data to other storage before the rebuild: first what is small and irreplaceable (database dumps, accounting data), then the rest. If you see read errors or pending sectors on the remaining drives, do not attempt the copy either; shut the array down. The safest route is to power the array down and take a sector-level copy of each member drive; if you lack the hardware or the time, copying the critical data is the minimum precaution.

7. Make sure the new drive is suitable

Its capacity must be at least that of the old one and its sector format (512e or 4Kn) must match the array. Avoid SMR drives for rebuilds: they slow down badly under long uninterrupted writes and some controllers drop the drive on a timeout. Before fitting a drive that was previously used in another array, make sure its old RAID metadata has been cleared.

When should a rebuild not be started at all?

Wherever the table says "Do not start", the array is powered down, the drives are labelled with their slot numbers, and nothing is written to them.

What not to do

Commands for reading the state

All of the commands below are read-only; they change nothing on the array.

# Linux software RAID (mdadm)
cat /proc/mdstat
mdadm --detail /dev/md0
mdadm --examine /dev/sd[a-d]1        # compare the "Events" counters of the members

# Drive health
smartctl -a /dev/sda
smartctl -a -d megaraid,0 /dev/sda   # drive behind MegaRAID; 0 = DID in storcli output

# Broadcom / LSI MegaRAID (perccli on Dell PERC uses the same syntax)
storcli /c0 show
storcli /c0/vall show
storcli /c0/eall/sall show
storcli /c0 show events

# HPE Smart Array
ssacli ctrl all show config
ssacli ctrl slot=0 pd all show status

In mdadm --examine output, the member whose "Events" value is lower than the others is the drive that dropped out of the array earlier.

If you are sending the drives to a laboratory

• Send every member drive, including the failed one.
• Write the slot number on each drive.
• Note the controller model, the RAID level and, if you know them, the stripe size and file system.
• Add a timeline: which drive dropped when, and what has been tried.
• Put each drive in an antistatic bag and pack them separately with shock-absorbing material.

How an array is rebuilt virtually in the laboratory, and how drive order and stripe size are determined, is described on the RAID data recovery page.

Frequently asked questions

How long does a rebuild take?
It depends on drive capacity, the load on the array at the time and the controller's rebuild priority setting; anything from a few hours to a few days. The longer it runs, the longer the window stays open for a second failure.

I use RAID 6. Do I still need to be careful?
With one drive lost, RAID 6 still has one drive of tolerance; an unreadable sector met during the rebuild is corrected from the second parity. That is why RAID 6 is preferred on large drives. With two drives lost, you are where RAID 5 is after losing one.

Is there a way to prevent this?
Regular consistency scans (patrol read, scrub) find and repair latent bad sectors while redundancy is still there. Test from time to time that monitoring alerts really reach someone. And RAID is not a backup: it does not solve a deleted file, ransomware or a controller failure.