software RAID 1 debian notes

Notes taken while I was converting an 1 hard drive disk system to a software RAID 1 system.

The system.
# cat /etc/issue /etc/debian_version
Debian GNU/Linux 6.0 \n \l

6.0.7



Install the MD devices --Linux Software RAID-- Manager
# apt-get install mdadm


The System is installed on a ~250GB hard drive.
# fdisk -l

Disk /dev/sda: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00096f89

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1       29933   240430080   83  Linux
/dev/sda2           29933       30402     3766273    5  Extended
/dev/sda5           29933       30402     3766272   82  Linux swap / Solaris


Shutdown the system, add one more ~250GB disk, and boot

What partitions does the system see?
cat /proc/partitions 
major minor  #blocks  name

   8        0  244198584 sda
   8        1  240430080 sda1
   8        2          1 sda2
   8        5    3766272 sda5
   8       16  244198584 sdb
Yupe, the second drive --sdb is identical to the first drive --sda.

List all active md devices and their status
# cat /proc/mdstat 
Personalities :
unused devices: 


Copy the sda partition scheme to sdb.
# sfdisk -d /dev/sda | sfdisk --force /dev/sdb


Check the partition tables with sfdisk.
# sfdisk -l

Disk /dev/sdb: 30401 cylinders, 255 heads, 63 sectors/track
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sdb1   *      0+  29932-  29933- 240430080   83  Linux
/dev/sdb2      29932+  30401-    469-   3766273    5  Extended
/dev/sdb3          0       -       0          0    0  Empty
/dev/sdb4          0       -       0          0    0  Empty
/dev/sdb5      29932+  30401-    469-   3766272   82  Linux swap / Solaris

Disk /dev/sda: 30401 cylinders, 255 heads, 63 sectors/track
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sda1   *      0+  29932-  29933- 240430080   83  Linux
/dev/sda2      29932+  30401-    469-   3766273    5  Extended
/dev/sda3          0       -       0          0    0  Empty
/dev/sda4          0       -       0          0    0  Empty
/dev/sda5      29932+  30401-    469-   3766272   82  Linux swap / Solaris


Check the partition tables with fdisk.
# fdisk -l

Disk /dev/sdb: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xdb0cbcd3

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1       29933   240430080   83  Linux
/dev/sdb2           29933       30402     3766273    5  Extended
/dev/sdb5           29933       30402     3766272   82  Linux swap / Solaris

Disk /dev/sda: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00096f89

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1       29933   240430080   83  Linux
/dev/sda2           29933       30402     3766273    5  Extended
/dev/sda5           29933       30402     3766272   82  Linux swap / Solaris


Change the sdb1 and sdb5 partitions Ox ID to 'Linux raid auto' ( fd )
# fdisk /dev/sdb

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): m
Command action
   a   toggle a bootable flag
   b   edit bsd disklabel
   c   toggle the dos compatibility flag
   d   delete a partition
   l   list known partition types
   m   print this menu
   n   add a new partition
   o   create a new empty DOS partition table
   p   print the partition table
   q   quit without saving changes
   s   create a new empty Sun disklabel
   t   change a partition's system id
   u   change display/entry units
   v   verify the partition table
   w   write table to disk and exit
   x   extra functionality (experts only)

Command (m for help): t
Partition number (1-5): 1
Hex code (type L to list codes): L

 0  Empty           24  NEC DOS         81  Minix / old Lin bf  Solaris        
 1  FAT12           39  Plan 9          82  Linux swap / So c1  DRDOS/sec (FAT-
 2  XENIX root      3c  PartitionMagic  83  Linux           c4  DRDOS/sec (FAT-
 3  XENIX usr       40  Venix 80286     84  OS/2 hidden C:  c6  DRDOS/sec (FAT-
 4  FAT16 <32M      41  PPC PReP Boot   85  Linux extended  c7  Syrinx         
 5  Extended        42  SFS             86  NTFS volume set da  Non-FS data    
 6  FAT16           4d  QNX4.x          87  NTFS volume set db  CP/M / CTOS / .
 7  HPFS/NTFS       4e  QNX4.x 2nd part 88  Linux plaintext de  Dell Utility   
 8  AIX             4f  QNX4.x 3rd part 8e  Linux LVM       df  BootIt         
 9  AIX bootable    50  OnTrack DM      93  Amoeba          e1  DOS access     
 a  OS/2 Boot Manag 51  OnTrack DM6 Aux 94  Amoeba BBT      e3  DOS R/O        
 b  W95 FAT32       52  CP/M            9f  BSD/OS          e4  SpeedStor      
 c  W95 FAT32 (LBA) 53  OnTrack DM6 Aux a0  IBM Thinkpad hi eb  BeOS fs        
 e  W95 FAT16 (LBA) 54  OnTrackDM6      a5  FreeBSD         ee  GPT            
 f  W95 Ext'd (LBA) 55  EZ-Drive        a6  OpenBSD         ef  EFI (FAT-12/16/
10  OPUS            56  Golden Bow      a7  NeXTSTEP        f0  Linux/PA-RISC b
11  Hidden FAT12    5c  Priam Edisk     a8  Darwin UFS      f1  SpeedStor      
12  Compaq diagnost 61  SpeedStor       a9  NetBSD          f4  SpeedStor      
14  Hidden FAT16 <3 63  GNU HURD or Sys ab  Darwin boot     f2  DOS secondary  
16  Hidden FAT16    64  Novell Netware  af  HFS / HFS+      fb  VMware VMFS    
17  Hidden HPFS/NTF 65  Novell Netware  b7  BSDI fs         fc  VMware VMKCORE 
18  AST SmartSleep  70  DiskSecure Mult b8  BSDI swap       fd  Linux raid auto
1b  Hidden W95 FAT3 75  PC/IX           bb  Boot Wizard hid fe  LANstep        
1c  Hidden W95 FAT3 80  Old Minix       be  Solaris boot    ff  BBT            
1e  Hidden W95 FAT1
Hex code (type L to list codes): fd
Changed system type of partition 1 to fd (Linux raid autodetect)

Command (m for help): t
Partition number (1-5): 5
Hex code (type L to list codes): fd
Changed system type of partition 5 to fd (Linux raid autodetect)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.


Check sdb partitions.
# fdisk -l /dev/sdb

Disk /dev/sdb: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xdb0cbcd3

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1       29933   240430080   fd  Linux raid autodetect
/dev/sdb2           29933       30402     3766273    5  Extended
/dev/sdb5           29933       30402     3766272   fd  Linux raid autodetect


Zero-out any remains of RAID arrays on /dev/sdb
# mdadm --zero-superblock /dev/sdb1
# mdadm --zero-superblock /dev/sdb5


Create the RAID arrays using the older 0.90 superblock format.
# mdadm --create /dev/md0 --level=1 --metadata=0 --raid-disks=2 missing /dev/sdb1
mdadm: array /dev/md0 started.
# mdadm --create /dev/md1 --level=1 --metadata=0 --raid-disks=2 missing /dev/sdb5
mdadm: array /dev/md1 started.
"The version-0.90 superblock limits the number of component devices within an array to 28, and limits each component device to a maximum size of 2TB on kernel version <3.1 and 4TB on kernel version >=3.1."

List MD devices and MD Status.
# cat /proc/mdstat 
Personalities : [raid1] 
md1 : active raid1 sdb5[1]
      3766208 blocks [2/1] [_U]
      
md0 : active raid1 sdb1[1]
      240430016 blocks [2/1] [_U]
      
unused devices: 


Create File Systems on MD devices.
# mkfs.ext3 -v -m .1 -b 4096 -E stride=32,stripe-width=64 /dev/md0
# mkswap /dev/md1


Set /etc/mdadm/mdadm.conf
# cp /etc/mdadm/mdadm.conf /etc/mdadm/mdadm.conf.0
Delete ARRAY definitions in /etc/mdadm/mdadm.conf
# mdadm --examine --scan >> /etc/mdadm/mdadm.conf


Mount md0
# mkdir /mnt/md0
# mount /dev/md0 /mnt/md0


Set
/dev/md0 to mount at /
and
/dev/md1 as swap
in /etc/fstab for now
# mkdir /etc/bak
# cp /etc/fstab /etc/bak/fstab.0
# vi /etc/fstab
# cat /etc/fstab |grep md
/dev/md0  /               ext3    errors=remount-ro 0       1
/dev/md1  none            swap    sw              0       0


Configure Grub (2).
# cp /etc/grub.d/40_custom /etc/grub.d/09_raid1
# uname -r
2.6.32-5-amd64
Set /etc/grub.d/09_raid1
# vi /etc/grub.d/09_raid1
# cat /etc/grub.d/09_raid1
#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
menuentry 'Debian GNU/Linux, with Linux 2.6.32-5-amd64 raid1' --class debian --class gnu-linux --class gnu --class os {
        insmod raid
        insmod mdraid
        insmod part_msdos
        insmod ext2
        set root='(md0)'
        echo    'Loading Linux 2.6.32-5-amd64 ....'
        linux   /boot/vmlinuz-2.6.32-5-amd64 root=/dev/md0 ro  quiet
        echo    'Loading initial ramdisk ... .'
        initrd  /boot/initrd.img-2.6.32-5-amd64
}


Uncomment GRUB_DISABLE_LINUX_UUID=true in /etc/default/grub
# vi /etc/default/grub




Back up /etc/mtab
# cp /etc/mtab /etc/bak/mtab.0
Replace /dev/sda1 with /dev/md0 in /etc/mtab.

Update Grub
# grub-mkconfig -o /boot/grub/grub.cfg
Update the initramfs image
# update-initramfs -u


Copy the system to /dev/md0
# cp -dpRx / /mnt/md0/


Install grub on both drives.
# grub-install /dev/sdb
# grub-install /dev/sda


Reboot
# shutdown -r now


Check if md is used.
# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md0              226G  1.6G  213G   1% /
tmpfs                 943M     0  943M   0% /lib/init/rw
udev                  936M  152K  936M   1% /dev
tmpfs                 943M     0  943M   0% /dev/shm




Set the Ox partition IDs on /dev/sda to "Linux RAID auto" ( fd ).
# for part in 1 5;do sfdisk --change-id /dev/sda $part fd;done


Add /dev/sda5 to the /dev/md1 RAID array
# mdadm --add /dev/md1 /dev/sda5
mdadm: added /dev/sda5
Check
# cat /proc/mdstat 
Personalities : [raid1] 
md1 : active raid1 sda5[2] sdb5[1]
      3766208 blocks [2/1] [_U]
      [>....................]  recovery =  3.3% (124800/3766208) finish=7.7min speed=7800K/sec
      
md0 : active raid1 sdb1[1]
      240430016 blocks [2/1] [_U]
      
unused devices: 


Add /dev/sda1 to /dev/md0
# mdadm --add /dev/md0 /dev/sda1
mdadm: added /dev/sda1


Check MD Status again.
# cat /proc/mdstat 
Personalities : [raid1] 
md1 : active raid1 sda5[2] sdb5[1]
      3766208 blocks [2/1] [_U]
      [=>...................]  recovery =  6.2% (234368/3766208) finish=55.2min speed=1064K/sec
      
md0 : active raid1 sda1[2] sdb1[1]
      240430016 blocks [2/1] [_U]
       resync=DELAYED
      
unused devices: 


Wait until the 're'sync is done.

Come on!
# cat /proc/mdstat 
Personalities : [raid1] 
md1 : active raid1 sda5[0] sdb5[1]
      3766208 blocks [2/2] [UU]
      
md0 : active raid1 sda1[2] sdb1[1]
      240430016 blocks [2/1] [_U]
      [===========>.........]  recovery = 59.1% (142333696/240430016) finish=23.1min speed=70622K/sec
      
unused devices: 


Finally.
# cat /proc/mdstat 
Personalities : [raid1] 
md1 : active raid1 sda5[0] sdb5[1]
      3766208 blocks [2/2] [UU]
      
md0 : active raid1 sda1[0] sdb1[1]
      240430016 blocks [2/2] [UU]
      
unused devices: 


Install Grub2 everywhere and reboot.
# grub-install /dev/sdb
# grub-install /dev/sda
# grub-install /dev/md0
# reboot


Test! What would happen in a hard drive failure?

Shutdown the system and pull out a hard disk drive.
# shutdown -h now
Pull out /dev/sdb and Boot.

The system came up with one drive and a degraded Array.
# cat /proc/mdstat
Personalities : [raid1] 
md1 : active (auto-read-only) raid1 sda5[0]
      3766208 blocks [2/1] [U_]
      
md0 : active raid1 sda1[0]
      240430016 blocks [2/1] [U_]
      
unused devices: 


Let's test how it would come up if the /dev/sda fails.
# shutdown -h now
Put in /dev/sdb, pull out /dev/sda, and boot the system.

Great, the system came up again with one drive on a degraded Array.
# cat /proc/mdstat 
Personalities : [raid1] 
md1 : active (auto-read-only) raid1 sda5[1]
      3766208 blocks [2/1] [_U]
      
md0 : active raid1 sda1[1]
      240430016 blocks [2/1] [_U]
      
unused devices: 


# shutdown -h now
Put in /dev/sda again and boot the system.

The system comes up fine.
# cat /proc/mdstat 
Personalities : [raid1] 
md1 : active (auto-read-only) raid1 sda5[0] sdb5[1]
      3766208 blocks [2/2] [UU]
      
md0 : active raid1 sdb1[1]
      240430016 blocks [2/1] [_U]
      
unused devices: 
Oups, sda1 is missing from the RAID 1 Array md0.
# mdadm -a /dev/md0 /dev/sda1
mdadm: re-added /dev/sda1
Check
# cat /proc/mdstat 
Personalities : [raid1] 
md1 : active (auto-read-only) raid1 sda5[0] sdb5[1]
      3766208 blocks [2/2] [UU]
      
md0 : active raid1 sda1[2] sdb1[1]
      240430016 blocks [2/1] [_U]
      [>....................]  recovery =  0.6% (1466496/240430016) finish=88.4min speed=45012K/sec
      
unused devices: 


Of course in case of a real hard drive failure the new drive entered is not going to have the same partition table with the system nor the superblock information, nor grub2 installed. If the new drive entered is identical to the one used by the RAID Array the steps describing how to add a drive to the RAID 1 drive Array should work fine, eg:
# sfdisk -d /dev/OLD-GOOD-DRIVE | sfdisk --force /dev/NEW-REPLACEMENT-DRIVE
# mdadm --zero-superblock /dev/NEW-REPLACEMENT-DRIVE-PARTITIONS
# mdadm -a /dev/RAID-DEVICE /dev/NEW-REPLACEMENT-DRIVE-PARTITIONS
# grub-install /dev/NEW-REPLACEMENT-DRIVE


If the new hard drive is not an identical drive you will have to work a bit more creating partitions that match the ones used by the system.

Well, it is not a good idea to use a RAID 1 Array on a swap md ... And you can get away without creating a RAID 0 array for swap. You could use /dev/sda5 and /dev/sdb5 as 2 different swap partitions.

References:
RAID superblock formats
Raid Setup
Swapping_on_RAID



Software RAID 1 on Debian GNU Linux