yet another software RAID debian

Notes taken while I was converting an 1 hard drive system to a 4 hard drives system with software RAID Arrays.

  • 2 x 1TB hard drives contributing one partition each to a software RAID 1 array and contributing one swap partition each to the system.
  • 2 x 3TB hard drives contributing two partitions each to software RAID 1 Arrays.


The system:
# cat /etc/issue /etc/debian_version
Debian GNU/Linux 6.0 \n \l

6.0.7
# uname -r
2.6.32-5-amd64


Install the Multiple Devices MD --Linux Software RAID Array-- Manager
# apt-get install mdadm


Check out the disk used by the System.
# fdisk -l

Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000a8440

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1      121133   972993536   83  Linux
/dev/sda2          121133      121602     3766273    5  Extended
/dev/sda5          121133      121602     3766272   82  Linux swap / Solaris


OK, shut it down and add the other ~ 1TB disk

Look at the disks
# sfdisk -l

Disk /dev/sda: 121601 cylinders, 255 heads, 63 sectors/track
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sda1   *      0+ 121132- 121133- 972993536   83  Linux
/dev/sda2     121132+ 121601-    469-   3766273    5  Extended
/dev/sda3          0       -       0          0    0  Empty
/dev/sda4          0       -       0          0    0  Empty
/dev/sda5     121132+ 121601-    469-   3766272   82  Linux swap / Solaris

Disk /dev/sdb: 121601 cylinders, 255 heads, 63 sectors/track

sfdisk: ERROR: sector 0 does not have an msdos signature
 /dev/sdb: unrecognized partition table type
No partitions found


Copy the partition table of /dev/sda to /dev/sdb
# sfdisk -d /dev/sda | sfdisk --force /dev/sdb


List all logical drives on all hard drives
# sfdisk -l

Disk /dev/sda: 121601 cylinders, 255 heads, 63 sectors/track
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sda1   *      0+ 121132- 121133- 972993536   83  Linux
/dev/sda2     121132+ 121601-    469-   3766273    5  Extended
/dev/sda3          0       -       0          0    0  Empty
/dev/sda4          0       -       0          0    0  Empty
/dev/sda5     121132+ 121601-    469-   3766272   82  Linux swap / Solaris

Disk /dev/sdb: 121601 cylinders, 255 heads, 63 sectors/track
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sdb1   *      0+ 121132- 121133- 972993536   83  Linux
/dev/sdb2     121132+ 121601-    469-   3766273    5  Extended
/dev/sdb3          0       -       0          0    0  Empty
/dev/sdb4          0       -       0          0    0  Empty
/dev/sdb5     121132+ 121601-    469-   3766272   82  Linux swap / Solaris


Set /dev/sdb1 0x ID to 0xfd ( Linux RAID Auto )
# sfdisk --change-id /dev/sdb 1 fd


# sfdisk -l /dev/sdb

Disk /dev/sdb: 121601 cylinders, 255 heads, 63 sectors/track
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sdb1   *      0+ 121132- 121133- 972993536   fd  Linux raid autodetect
/dev/sdb2     121132+ 121601-    469-   3766273    5  Extended
/dev/sdb3          0       -       0          0    0  Empty
/dev/sdb4          0       -       0          0    0  Empty
/dev/sdb5     121132+ 121601-    469-   3766272   82  Linux swap / Solaris


Overwrite with zeros MD superblocks --remains of RAID Arrays-- if any.
# mdadm --zero-superblock --force /dev/sdb1


Create a RAID 1 Array of 2 logical disks using the v0.90 superblock format
# mdadm --create /dev/md0 --level=1 --metadata=0 --raid-disks=2 missing  /dev/sdb1
mdadm: array /dev/md0 started.
"The version-0.90 superblock limits the number of component devices within an array to 28, and limits each component device to a maximum size of 2TB on kernel version <3.1 and 4TB on kernel version >=3.1." The version-0.90 Superblock Format

Look at proc
# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sdb1[1]
      972993472 blocks [2/1] [_U]
      
unused devices: 


Create an ext3 file-system on the Multiple Devices MD device /dev/md0
# mkfs.ext3 -v -m .1 -b 4096 -E stride=32,stripe-width=64 /dev/md0
with stripe-width=64 , it should not matter in RAID 1 Arrays.
Options explained:
     The first command makes a ext3 filesystem
     -v verbose
     -m .1 leave .1% of disk to root 
     -b 4096 block size of 4kb (recommended above for large-file systems)
     -E stride=32,stripe-width=64 see below calculation
RAID wiki

Backup mdadm.conf
# cp /etc/mdadm/mdadm.conf /etc/mdadm/mdadm.conf.0
Delete ARRAY definitions from /etc/mdadm/mdadm.conf
# grep -v ARRAY /etc/mdadm/mdadm.conf >> /etc/mdadm/mdadm.conf
Update /etc/mdadm/mdadm.conf
# mdadm --examine --scan >> /etc/mdadm/mdadm.conf


Mount md0
# mkdir /mnt/md0
# mount /dev/md0 /mnt/md0/


Save /etc/fstab for historical reasons
# cp /etc/fstab /etc/fstab.0
Remove or comment out lines starting with UUID or /dev/sd* from /etc/fstab
and make /etc/fstab look like the following.
# grep -v "#" /etc/fstab
proc            /proc           proc    defaults        0       0
/dev/md0         /              ext3    errors=remount-ro 0       1
/dev/sda5 swap  swap defaults,pri=1   0  0
/dev/sdb5 swap  swap defaults,pri=1   0  0
Swapping on RAID

Configure the bootloader --Grub2
# cp /etc/grub.d/40_custom /etc/grub.d/08_raid1
# vi /etc/grub.d/08_raid1
Here is my /etc/grub.d/08_raid1 :
#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
menuentry 'Debian GNU/Linux, with Linux 2.6.32-5-amd64 raid1' --class debian --class gnu-linux --class gnu --class os {
        insmod raid
        insmod mdraid
        insmod part_msdos
        insmod ext2
        set root='(md0)'
        echo    'Loading Linux 2.6.32-5-amd64 ...'
        linux   /boot/vmlinuz-2.6.32-5-amd64 root=/dev/md0 ro  quiet
        echo    'Loading initial ramdisk ...'
        initrd  /boot/initrd.img-2.6.32-5-amd64
}


Un-comment or add
GRUB_DISABLE_LINUX_UUID=true
at /etc/default/grub

Replace /dev/sda with /dev/md0 in /etc/mtab
and delete the line showing /dev/md0 mount on /mnt/md0

Update grub.cfg
# grub-mkconfig -o /boot/grub/grub.cfg 


Update the initramfs image
# update-initramfs -u


Copy the System to /dev/md0
# cp -dpRx / /mnt/md0


Install grub on all drives.
# grub-install /dev/sdb
# grub-install /dev/sda



Cross your fingers and reboot
# shutdown -r now


OK, the system came up as expected.
Check proc
# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sdb1[1]
      972993472 blocks [2/1] [_U]
      
unused devices: 


Set /dev/sda1 0x ID to 0xfd ( Linux RAID Auto )
# sfdisk --change-id /dev/sda 1 fd


List all partitions on all drives
# sfdisk -l

Disk /dev/sda: 121601 cylinders, 255 heads, 63 sectors/track
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sda1   *      0+ 121132- 121133- 972993536   fd  Linux raid autodetect
/dev/sda2     121132+ 121601-    469-   3766273    5  Extended
/dev/sda3          0       -       0          0    0  Empty
/dev/sda4          0       -       0          0    0  Empty
/dev/sda5     121132+ 121601-    469-   3766272   82  Linux swap / Solaris

Disk /dev/sdb: 121601 cylinders, 255 heads, 63 sectors/track
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sdb1   *      0+ 121132- 121133- 972993536   fd  Linux raid autodetect
/dev/sdb2     121132+ 121601-    469-   3766273    5  Extended
/dev/sdb3          0       -       0          0    0  Empty
/dev/sdb4          0       -       0          0    0  Empty
/dev/sdb5     121132+ 121601-    469-   3766272   82  Linux swap / Solaris

Disk /dev/md0: 243248368 cylinders, 2 heads, 4 sectors/track

sfdisk: ERROR: sector 0 does not have an msdos signature
 /dev/md0: unrecognized partition table type
No partitions found



Add /dev/sda1 to the RAID 1 md0
# mdadm --add /dev/md0 /dev/sda1
mdadm: added /dev/sda1


Check proc mdstat
# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sda1[2] sdb1[1]
      972993472 blocks [2/1] [_U]
      [>....................]  recovery =  0.4% (4773696/972993472) finish=144.5min speed=111602K/sec
      
unused devices: 


Check load
22:53:31 up 20 min,  1 user,  load average: 0.66, 0.19, 0.06
Not (very) bad ;)

Wait for the raid array to sync.

root@bob:~# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sda1[2] sdb1[1]
      972993472 blocks [2/1] [_U]
      [===========>.........]  recovery = 56.1% (546131776/972993472) finish=74.2min speed=95857K/sec
      
unused devices: 


Done.
# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sda1[0] sdb1[1]
      972993472 blocks [2/2] [UU]
      
unused devices: 


Test! What if I take off /dev/sdb ?
# shutdown -h now
take of /dev/sdb and boot

It came up fine :) , check proc
# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sda1[0]
      972993472 blocks [2/1] [U_]
      
unused devices: 


(

Yet another not-very-cheap way for a technician
to hide 'some' tracks (I think)
in a dedicated server farm.

  • shutdown the system
  • take off drive a
  • do crap
  • shutdown the system, take off drive b , put in drive a , boot
  • shutdown the system, put in drive b, and let them sync

)

Back to business ...
  • shutdown the system
  • put in sdb
  • and boot


It came up fine, check proc
# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sda1[0]
      972993472 blocks [2/1] [U_]
      
unused devices: 
Add /dev/sdb to the RAID array
# mdadm --add /dev/md0 /dev/sdb1
Check proc
# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sdb1[2] sda1[0]
      972993472 blocks [2/1] [U_]
      [>....................]  recovery =  3.6% (35851584/972993472) finish=145.8min speed=107124K/sec
      
unused devices: 


OK, Let 's create the "DATA" arrays.
  • shutdown the system
  • put in the 3TB hard drives
  • Set on BIOS to boot only from /dev/sda and /dev/sdb
  • boot


OK the system came up.
Check out proc
# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sda1[0] sdb1[2]
      972993472 blocks [2/1] [U_]
      [>....................]  recovery =  0.0% (449280/972993472) finish=15694.6min speed=1032K/sec
      
unused devices: 


List all logical and physical disks
# sfdisk -l

Disk /dev/sda: 121601 cylinders, 255 heads, 63 sectors/track
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sda1   *      0+ 121132- 121133- 972993536   fd  Linux raid autodetect
/dev/sda2     121132+ 121601-    469-   3766273    5  Extended
/dev/sda3          0       -       0          0    0  Empty
/dev/sda4          0       -       0          0    0  Empty
/dev/sda5     121132+ 121601-    469-   3766272   82  Linux swap / Solaris

Disk /dev/sdb: 121601 cylinders, 255 heads, 63 sectors/track
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sdb1   *      0+ 121132- 121133- 972993536   fd  Linux raid autodetect
/dev/sdb2     121132+ 121601-    469-   3766273    5  Extended
/dev/sdb3          0       -       0          0    0  Empty
/dev/sdb4          0       -       0          0    0  Empty
/dev/sdb5     121132+ 121601-    469-   3766272   82  Linux swap / Solaris

WARNING: GPT (GUID Partition Table) detected on '/dev/sdd'! The util sfdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sdd: 364801 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sdd1          0+ 267349- 267350- 2147483647+  ee  GPT
  start: (c,h,s) expected (0,0,2) found (0,0,1)
/dev/sdd2          0       -       0          0    0  Empty
/dev/sdd3          0       -       0          0    0  Empty
/dev/sdd4          0       -       0          0    0  Empty

WARNING: GPT (GUID Partition Table) detected on '/dev/sdc'! The util sfdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sdc: 364801 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sdc1          0+ 267349- 267350- 2147483647+  ee  GPT
  start: (c,h,s) expected (0,0,2) found (0,0,1)
/dev/sdc2          0       -       0          0    0  Empty
/dev/sdc3          0       -       0          0    0  Empty
/dev/sdc4          0       -       0          0    0  Empty

Disk /dev/md0: 243248368 cylinders, 2 heads, 4 sectors/track

sfdisk: ERROR: sector 0 does not have an msdos signature
 /dev/md0: unrecognized partition table type
No partitions found


Partition the /dev/sdc hard drive
# parted /dev/sdc
GNU Parted 2.3
Using /dev/sdc
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel gpt
Warning: The existing disk label on /dev/sdc will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? yes                                                               
(parted) mkpart data00 0GB 1501GB                                         
(parted) mkpart data01 1502BG 3001GB
Warning: You requested a partition from 1024B to 3001GB.                  
The closest location we can manage is 17.4kB to 1048kB.
Is this still acceptable to you?
Yes/No? no                                                                
(parted) quit                                                             
Information: You may need to update /etc/fstab.                           
  


Now, I could use parted again to partition /dev/sdd.
However, I really want to try to clone GPT disks with gdisk --GPT fdisk.
For debian GPT fdisk, currently is found in sid.

Install gdisk
# echo "deb http://ftp.it.debian.org/debian sid main" >> /etc/apt/sources.list
# apt-get update
# apt-get install gdiskgdisk
GPT fdisk (gdisk) version 0.8.5

Type device filename, or press  to exit: 
#
Not bad, at this moment the latest GPT fdisk version is 0.8.6.

Comment out the ftp.it.debian.org repository, add a squeeze repository and
# apt-get update


sgdisk is part of GPT fdisk --gdisk.
Copy the partition scheme of sdc to sdd and change the GUID on /dev/sdd.
# sgdisk -R=/dev/sdd /dev/sdc
The operation has completed successfully.
#  sgdisk -G /dev/sdd
The operation has completed successfully.


excerpt from the sgdisk man page.
-R, --replicate=second_device_filename
              Replicate the main device's partition table on the specified second device.  Note that the replicated partition table is
              an  exact copy, including all GUIDs; if the device should have its own unique GUIDs, you should use the -G option on the
              new disk.



List all logical and physical disks.
# parted -l
Model: ATA WDC WD1002FAEX-0 (scsi)
Disk /dev/sda: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type      File system     Flags
 1      1049kB  996GB   996GB   primary   ext3            boot, raid
 2      996GB   1000GB  3857MB  extended
 5      996GB   1000GB  3857MB  logical   linux-swap(v1)


Model: ATA WDC WD1002FAEX-0 (scsi)
Disk /dev/sdb: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type      File system  Flags
 1      1049kB  996GB   996GB   primary   ext3         boot, raid
 2      996GB   1000GB  3857MB  extended
 5      996GB   1000GB  3857MB  logical


Model: ATA WDC WD30EFRX-68A (scsi)
Disk /dev/sdc: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number  Start   End     Size    File system  Name    Flags
 1      1049kB  1501GB  1501GB               data00
 2      1502GB  3001GB  1499GB               data01


Model: ATA WDC WD30EFRX-68A (scsi)
Disk /dev/sdd: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number  Start   End     Size    File system  Name    Flags
 1      1049kB  1501GB  1501GB               data00
 2      1502GB  3001GB  1499GB               data01


Model: Linux Software RAID Array (md)
Disk /dev/md0: 996GB
Sector size (logical/physical): 512B/512B
Partition Table: loop

Number  Start  End    Size   File system  Flags
 1      0.00B  996GB  996GB  ext3



Set the raid flag to true on all logical partitions of /dev/sdc.
# parted /dev/sdc
GNU Parted 2.3
Using /dev/sdc
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print                                                            
Model: ATA WDC WD30EFRX-68A (scsi)
Disk /dev/sdc: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number  Start   End     Size    File system  Name    Flags
 1      1049kB  1501GB  1501GB               data00
 2      1502GB  3001GB  1499GB               data01

(parted) set 1 raid on                                                    
(parted) set 2 raid on
(parted) print
Model: ATA WDC WD30EFRX-68A (scsi)
Disk /dev/sdc: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number  Start   End     Size    File system  Name    Flags
 1      1049kB  1501GB  1501GB               data00  raid
 2      1502GB  3001GB  1499GB               data01  raid

(parted) quit                                                             
Information: You may need to update /etc/fstab.    


Set the raid flag to true on all logical partition of /dev/sdd.
# parted /dev/sdd
GNU Parted 2.3
Using /dev/sdd
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print                                                            
Model: ATA WDC WD30EFRX-68A (scsi)
Disk /dev/sdd: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number  Start   End     Size    File system  Name    Flags
 1      1049kB  1501GB  1501GB               data00
 2      1502GB  3001GB  1499GB               data01

(parted) set 1 raid on
(parted) set 2 raid on                                                    
(parted) print                                                            
Model: ATA WDC WD30EFRX-68A (scsi)
Disk /dev/sdd: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number  Start   End     Size    File system  Name    Flags
 1      1049kB  1501GB  1501GB               data00  raid
 2      1502GB  3001GB  1499GB               data01  raid

(parted) quit                                                             
Information: You may need to update /etc/fstab.


Look the partitions in proc
# cat /proc/partitions 
major minor  #blocks  name

   8        0  976762584 sda
   8        1  972993536 sda1
   8        2          1 sda2
   8        5    3766272 sda5
   8       16  976762584 sdb
   8       17  972993536 sdb1
   8       18          1 sdb2
   8       21    3766272 sdb5
   8       48 2930266584 sdd
   8       49 1465819136 sdd1
   8       50 1463469056 sdd2
   8       32 2930266584 sdc
   8       33 1465819136 sdc1
   8       34 1463469056 sdc2
   9        0  972993472 md0



Create the md1 data00 raid 1 Array.
# mdadm --create /dev/md1 --level=1 --metadata=0 --raid-disks=2 /dev/sdc1 /dev/sdd1
mdadm: array /dev/md1 started.


Format /dev/md1
# mkfs.ext3 -v -m .1 -b 4096 -E stride=32,stripe-width=64 /dev/md1


Create the md2 data01 raid 1 Array
#  mdadm --create /dev/md2 --level=1 --metadata=0 --raid-disks=2 /dev/sdc2 /dev/sdd2
mdadm: array /dev/md2 started.


Format /dev/md2
# mkfs.ext3 -v -m .1 -b 4096 -E stride=32,stripe-width=64 /dev/md2


Update mdadmn.conf
# mdadm --examine --scan >> /etc/mdadm/mdadm.conf
# vi /etc/mdadm/mdadm.conf
delete multiple ARRAY definitions

Create mount points and update /etc/fstab
# mkdir /data00
# mkdir /data01
# vi /etc/fstab
This is how my /etc/fstab looks like
# grep -v "#" /etc/fstab
proc            /proc           proc    defaults        0       0
/dev/md0         /              ext3    errors=remount-ro 0       1
/dev/sda5 swap  swap defaults,pri=1   0  0
/dev/sdb5 swap  swap defaults,pri=1   0  0
/dev/md1         /data00        ext3    errors=remount-ro 0       1
/dev/md2         /data01        ext3    errors=remount-ro 0       1


Look at /proc/mdstat
# cat /proc/mdstat 
Personalities : [raid1] 
md2 : active raid1 sdd2[1] sdc2[0]
      1463468992 blocks [2/2] [UU]
       resync=DELAYED
      
md1 : active raid1 sdd1[1] sdc1[0]
      1465819072 blocks [2/2] [UU]
      [=>...................]  resync =  6.8% (100567936/1465819072) finish=172.6min speed=131767K/sec
      
md0 : active raid1 sda1[0] sdb1[1]
      972993472 blocks [2/2] [UU]
      
unused devices: 



# reboot


The system came up fine
Loot at what is mounted and filesystem usage.
# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md0              914G  1.6G  911G   1% /
tmpfs                 943M     0  943M   0% /lib/init/rw
udev                  936M  168K  936M   1% /dev
tmpfs                 943M     0  943M   0% /dev/shm
/dev/md1              1.4T  198M  1.4T   1% /data00
/dev/md2              1.4T  198M  1.4T   1% /data01


Look at mdstat
# cat /proc/mdstat 
Personalities : [raid1] 
md2 : active raid1 sdc2[0] sdd2[1]
      1463468992 blocks [2/2] [UU]
       resync=DELAYED
      
md1 : active raid1 sdc1[0] sdd1[1]
      1465819072 blocks [2/2] [UU]
      [=>...................]  resync =  8.6% (127094592/1465819072) finish=168.6min speed=132290K/sec
      
md0 : active raid1 sda1[0] sdb1[1]
      972993472 blocks [2/2] [UU]
      
unused devices: 



Look the load
# w
 21:24:43 up 1 min,  1 user,  load average: 2.17, 0.75, 0.27
ouch! I 'll keep you posted on that.

Done!



Yet another Software RAID on Debian GNU Linux

Greek Gangsters with Suits

There is a country where incompetent, laughable, corrupted gangsters wearing suits become politicians , create iffy organizations , and rape on a daily basis the rest of the people.

Greece is that country ...
and today's subject is a corrupt iffy organization named AEPI--aepi.gr run by non-technical gangsters who have become experts in suing grandpas, taxi drivers , and souvlaki sellers for listening to radio or reproducing 400 year old songs.

Depending on the day of the week AEPI operates as a Collective Administration Organization or an Inc.

However, most of the time AEPI operates as a traditional mafia organization selling `the right to music` to coffee shops, bars, hair-dressers, kiosks , fast-food shops , taxi-drivers ... whoever.

AEPI, for the most part, is harassing people who cannot afford a lawyer and creates a mess in everything it touches. It is yet another factor contributing to the Greek misery. Supposedly it represents artists but the artists in that union are for the most part untalented and no-popular. These "artists" are loosers with no common sense flying many miles upon Earth. Even the scheme used to distribute the AEPI earnings to the represented untalented no-popular artists is a scam.

Recently AEPI came up with one more idea proving their technical inefficiencies, all around low IQ, and bad taste.

This time AEPI wants to force Internet Service Providers operating in Greece to cut access to a ~10 torrent directories.

greek.to
greek-team.cc
p2planet.net
tsibato.info
greekddl.eu
greek-best.com
kat.ph
isohunt.com
1337x.org
h33t.com

Will the tasteless, low-intellect, sub-humans running aepi figure out that cutting access to such web sites is impossible?

Will AEPI go bankrupt for infringing upon China's Great Firewall Idea?

Seriously though, some time soon, the rest of the people in Greece will wake up ...



Greek Gangsters with Suits

ssh_exchange_identification: Connection closed by remote host

$ ssh bingo@10.3.57.3 -vvv
OpenSSH_5.3p1 Debian-xduh, OpenSSL 0.9.8k 25 Mar 2010
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 10.3.57.3 [10.3.57.3] port 22.
debug1: Connection established.
debug1: identity file /home/from/.ssh/identity type -1
debug1: identity file /home/from/.ssh/id_rsa type -1
debug1: identity file /home/from/.ssh/id_dsa type -1
ssh_exchange_identification: Connection closed by remote host



In the SSH Server.
# /etc/init.d/fail2ban stop
# /etc/init.d/portsentry stop


To allow connections to the SSH daemon from all hosts allways
# echo "sshd: ALL" >> /etc/hosts.allow




ssh_exchange_identification hosts.allow Debian

Debian configure default editor

On relatively new debian systems nano is the `default` editor. I cannot stand nano.

To change it :
# update-alternatives --config editor
There are 7 choices for the alternative editor (providing /usr/bin/editor).

  Selection    Path                Priority   Status
------------------------------------------------------------
* 0            /bin/nano            40        auto mode
  1            /bin/nano            40        manual mode
  2            /bin/nano-tiny       0         manual mode
  3            /usr/bin/mcedit      25        manual mode
  4            /usr/bin/mg          30        manual mode
  5            /usr/bin/nvi         19        manual mode
  6            /usr/bin/vim.basic   30        manual mode
  7            /usr/bin/vim.tiny    10        manual mode

Press enter to keep the current choice[*], or type selection number: 6
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/editor (editor) in manual mode.


To see if it worked try
# file /usr/bin/editor
/usr/bin/editor: symbolic link to `/etc/alternatives/editor'
# file /etc/alternatives/editor
/etc/alternatives/editor: symbolic link to `/usr/bin/vim.basic'
or
# update-alternatives --config editor
again and look for the * or try anything that opens an editor eg.
# crontab -e






Debian Configure Default Editor

ftp server debian

# apt-get install proftpd
Select inetd or standalone, standalone

Install adduser.sh,yet another script that adds users
# wget http://kod.ipduh.com/lib/adduser.sh
# chmod 700 adduser.sh
# mv adduser.sh /bin


Add a dummy shell to the system allowed shells
# echo "/usr/bin/nologin" >> /etc/shells


Add a user
# adduser.sh 
Add User:
Enter GROUPID     : 8880
Enter GROUPNAME   : ftpusers
Enter USERID      : 8881
Enter USERNAME    : userftp
Enter USER HOME DIRECTORY ( Or hit enter for /home/userftp ): /data/userftp
Enter USERSHELL   : /usr/bin/nologin
Enter USERCOMMENT : ftpuser    
Enter new UNIX password: 
Retype new UNIX password: 
passwd: password updated successfully
 
User:userftp:x:8881:8880:ftpuser:/data/userftp:/usr/bin/nologin

Group:ftpusers:x:8880:

userftp home Dir /data/userftp long listing:
total 20
4 drwxr-xr-x 2 userftp ftpusers 4096 Apr 24 14:09 .
4 drwxr-xr-x 4 root    root     4096 Apr 24 14:09 ..
4 -rw-r--r-- 1 userftp ftpusers  220 Apr 24 14:09 .bash_logout
4 -rw-r--r-- 1 userftp ftpusers 3184 Apr 24 14:09 .bashrc
4 -rw-r--r-- 1 userftp ftpusers  675 Apr 24 14:09 .profile
 
.


Delete /data/userftp/.bash_logout , /data/userftp/.bashrc , /data/userftp/.profile
# rm /data/userftp/.bas*
# rm /data/userftp/.pro*


In /etc/proftpd/proftpd.conf
uncomment DefaultRoot set to ~
# grep DefaultRoot /etc/proftpd/proftpd.conf 
 DefaultRoot   ~
Optionally, disable ipv6
UseIPv6                         off
and enable extented logs
ExtendedLog /var/log/proftpd/extftp.log


Disallow to the rest of the users ftp access
# cat /etc/passwd |grep -v userftp | awk -F : '{print $1}' > /etc/proftpd/no-ftp-users
# cat /etc/ftpusers >> /etc/proftpd/no-ftp-users
# cat /etc/proftpd/no-ftp-users | sort | uniq > /etc/proftpd/no-ftp-users.uniq
# cp /etc/proftpd/no-ftp-users.uniq /etc/ftpusers


Restart proftpd
# /etc/init.d/proftpd restart
Stopping ftp server: proftpd.
Starting ftp server: proftpd.


Test
# touch /data/userftp/blah
# chown 8881.8880 /data/userftp/blah
 ftp localhost
Connected to localhost.
220 ProFTPD 1.3.3a Server ready.
Name (localhost:gone): userftp
331 Password required for userftp
Password:
230 User userftp logged in
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> dir
200 PORT command successful
150 Opening ASCII mode data connection for file list
-rw-r--r--   1 userftp  ftpusers        0 Apr 24 11:11 blah
226 Transfer complete
ftp> quit
221 Goodbye.


Test from another host in that same LAN eg. a routerboard powered machine.
[ipduh@ath-rox] > tool fetch address=10.3.77.3 mode=ftp password=userftpsomepassword user=userftp port=21 host="" dst-path=blah src-path=blah
scii=yes  
  status: finished

[ipduh@ath-rox] > 


Done!
However, if you don't absolutely need FTP disable-uninstall proftp and configure SFTP instead ;)



FTP server on Debian GNU Linux - proftpd

ntpclient Openwrt Kamikaze 7.09

Notes taken while installing the ntpclient on a:
# cat /etc/banner;cat /proc/cpuinfo  |egrep -i "hardw|proc"
  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 KAMIKAZE (7.09) -----------------------------------
  * 10 oz Vodka       Shake well with ice and strain
  * 10 oz Triple sec  mixture into 10 shot glasses.
  * 10 oz lime juice  Salute!
 ---------------------------------------------------
Processor : XScale-IXP42x Family rev 2 (v5b)
Hardware : Linksys NSLU2


Set the Timezone, upon fighting uci and /etc/config/system for a while I ended up reading the init scripts and changed /etc/TZ
# date
Tue Apr 23 15:54:54 UTC 2013
# echo 'EET-2EEST,M3.5.0/3,M10.5.0/4' > /etc/TZ
# date
Tue Apr 23 19:00:32 EEST 2013
That's for Athens, Greece

Install ntpclient
# cd /usr/sbin/
# wget http://downloads.openwrt.org/kamikaze/7.09/packages/armeb/ntpclient_2003_194-4_armeb.ipk
# ipkg install ntpclient_2003_194-4_armeb.ipk
# rm ntpclient_2003_194-4_armeb.ipk


Set the clock
# /usr/sbin/ntpclient -c 1 -h 10.21.241.4
41385 57983.683    9344.0    149.7      67.8 308441.2         0
# date
Tue Apr 23 19:06:26 EEST 2013
10.21.241.4 is a local timeserver

Configure ntpclient
# cd /etc/config
# vi ntpclient
This is my /etc/config/ntpclient
# cat ntpclient 
config ntpclient
 option hostname '10.21.241.4'
 option port '123'

config ntpclient
 option hostname '194.177.210.54'
 option port '123'


Make it stick
# uci commit ntpclient
check if it did stick
# /etc/init.d/network restart
# cat /etc/config/ntpclient 
config ntpclient
 option hostname '10.21.241.4'
 option port '123'

config ntpclient
 option hostname '194.177.210.54'
 option port '123'
hmm, I am not sure if that will do anything ... plan B

Enable cron
# /etc/init.d/cron enable
# ps |grep crond
 2086 root        308 S   crond -c /etc/crontabs 
21136 root        300 S   grep crond 


# vi /etc/init.d/ntpclient
# cat /etc/init.d/ntpclient 
#!/bin/sh
/usr/sbin/ntpclient -l -h 10.21.241.4 -c 1


Create the update clock cronjob
# vi /etc/crontabs/ntpclient
# cat /etc/crontabs/ntpclient
# cat /etc/crontabs/ntpclient 
*/30 * * * * /etc/init.d/ntpclient


Restart cron-Install cronjob
# killall crond
# /etc/init.d/cron start


References:
_ ntpclient _ openwrt
_ openwrt _ timezones




ntpclient OpenWrt Kamikaze

Hybrid Internet AWMN caching DNS server

Yet another, quick how-to configure a Hybrid Internet-AWMN caching nameserver,
this time on
# cat /etc/issue /etc/debian_version 
Debian GNU/Linux 6.0 \n \l

6.0.7


Install bind
# apt-get install bind9


Adjust /etc/bind/named.conf
# cat /etc/bind/named.conf
// This is the primary configuration file for the BIND DNS server named.
//
// Please read /usr/share/doc/bind9/README.Debian.gz for information on the 
// structure of BIND configuration files in Debian, *BEFORE* you customize 
// this configuration file.
//
// If you are just adding zones, please do that in /etc/bind/named.conf.local

include "/etc/bind/named.conf.options";
include "/etc/bind/named.conf.local";

I will include the default zones in the .local configuration file.

Adjust /etc/bind/named.conf.options
options {
 directory "/var/cache/bind";

 version "awmn. #g0 2012"; 

 // If there is a firewall between you and nameservers you want
 // to talk to, you may need to fix the firewall to allow multiple
 // ports to talk.  See http://www.kb.cert.org/vuls/id/800113

 // If your ISP provided one or more IP addresses for stable 
 // nameservers, you probably want to use them as forwarders.  
 // Uncomment the following block, and insert the addresses replacing 
 // the all-0's placeholder.

 //comment out zone "." hint if you choose to use forwarders
 // forwarders {
 //  0.0.0.0;
 // List of Public DNS:  http://alog.ipduh.com/2012/06/public-dns-servers.html
 //
 // };

 //listen-on { 127.0.0.1; 10.3.57.3; };
 listen-on { any; }
 auth-nxdomain no;    # conform to RFC1035
 //listen-on-v6 { any; };
};

Replace 10.3.57.3 with your awmn IP address if you choose to not use listen-on {any;}

# cat /etc/bind/named.conf.local 
//
// #g0 2013 -- awmn caching nameserver
// #http://ipduh.com/contact

acl internals {
        127.0.0.0/8;
        10.0.0.0/8;
};

view "internal" {
        match-clients { internals; };
        recursion yes;
        allow-recursion { any; };
        allow-query { any; };
        allow-query-cache { any; };

zone "localhost" {
        type master;
        file "/etc/bind/db.local";
};

zone "127.in-addr.arpa" {
        type master;
        file "/etc/bind/db.127";
};

zone "0.in-addr.arpa" {
        type master;
        file "/etc/bind/db.0";
};

zone "255.in-addr.arpa" {
        type master;
        file "/etc/bind/db.255";
};

zone "10.in-addr.arpa" IN {
        type forward;
        forwarders {
        10.0.0.1; 
//You may want to add a few more from http://awmn:awmn@nagios.ipduh.awmn/cgi-bin/nagios3/status.cgi?hostgroup=awmn-dns&style=detail
        10.19.143.13;
        10.19.143.12;
        };
};

//####################################
//# Greek Wireless Communities Zones #
//####################################
//# https://www.awmn.net/wiki/       #
//####################################
//Because awmn will go gwmn pretty soon g stands for Greek or Global ;)
//####################################

zone "awmn" IN {
        type forward;
        forwarders {
                  10.0.0.1; 
//You may want to add a few more from http://awmn:awmn@nagios.ipduh.awmn/cgi-bin/nagios3/status.cgi?hostgroup=awmn-dns&style=detail
                  10.19.143.12;
                  10.19.143.13;  
        };
};

zone "wn" IN {
        type forward;
        forwarders {
//yeah good luck with these name servers

                10.126.3.115;
                10.110.17.115;
                10.19.143.12;
                10.17.122.134;
                10.86.87.129;
                10.2.16.130;
                10.110.17.67;

//the following does wn. as much as the above which is not-at-all
                10.21.241.100;

        };
};

zone "swn" IN {
        type forward;
        forwarders {
                10.101.0.254;
                10.106.3.1;
                10.174.254.101;
                10.174.1.253;
        };
};

zone "twmn" IN {
        type forward;
        forwarders {
                10.104.76.65;
                10.122.20.70;
                10.122.3.68;
                10.122.14.72;
                10.104.1.74;
        };
};

zone "wthess" IN {
        type forward;
        forwarders {
                10.96.0.1;
                10.96.22.2;
                10.96.9.3;
        };
};

zone "ewn" IN {
        type forward;
        forwarders {
                10.145.7.150;
                10.146.210.130;
        };
};

zone "mswn" IN {
        type forward;
        forwarders {
                10.148.50.2;
        };
};

zone "cywn" IN {
        type forward;
        forwarders {
                10.215.0.125;
                10.215.2.126;
        };
};

zone "dwn" IN {
        type forward;
        forwarders {
                10.174.1.253;
                10.174.254.101;
                10.174.17.250;
        };
};

zone "wiran" IN {
        type forward;
        forwarders {
                10.230.3.133;
        };
};

zone "wana" IN {
        type forward;
        forwarders {
                10.224.3.35;
        };
};

zone "awn" IN {
        type forward;
        forwarders {
                10.198.0.130;
        };
};

zone "pwmn" IN {
        type forward;
        forwarders {
                10.140.14.67;
        };
};

// prime the server with knowledge of the root servers
// comment out if you are planning on using forwarders
zone "." {
        type hint;
        file "/etc/bind/db.root";
};


};



Set this Name-Server in /etc/resolv.conf
# echo "nameserver 127.0.0.1" > /etc/resolv.conf
# cat /etc/resolv.conf 
nameserver 127.0.0.1


Restart the Name-Server
# /etc/init.d/bind9 restart
Stopping domain name service...: bind9 waiting for pid 14979 to die.
Starting domain name service...: bind9.


Test the Name-server
# dig +short ipduh.com
85.25.242.245
# dig +short ipduh.awmn
10.21.241.4
# dig +short www.awmn
srv1.awmn.
10.19.143.13
# dig +short wind.awmn
10.19.143.12
# dig +short voip.awmn
10.67.0.17



Done.



Hybrid Internet-AWMN Caching DNS server How-To

linksys Network Storage Link NSL openwrt install

Notes taken while I was installing openwrt to a Cisco/Linksys NSL.

Get the openwrt firmware from openwrt kamikaze slug-firmware

# mkdir nsl-openwrt;mv openwrt-kamikaze-7.09-nslu2.zip nsl-openwrt;cd nsl-openwrt
# unzip openwrt-kamikaze-7.09-nslu2.zip


Install upslug2
# apt-get install upslug2


Put the NSL into upgrade MODE.

Power off - unplug the power cord.

Plug in the power cord and then while holding the reset button with a pushpin or a paper-clip hit the power button

Once the Ready/Status LED change to Dark Orange from Orange release the Reset Button

If the Ready/Status LED is flashing dark-orange/green you succeeded ,
Else you did not release the Ready/Status LED soon enough --Try again.


Figure out if the NSL is into upgrade mode.
# upslug2 -d eth0
NSLU2     00:1d:7e:aa:17:3a Product ID: 1 Protocol ID: 0 Firmware Version: R23V63 [0x2363]


Install the openwrt-kamikaze image
# upslug2 -d eth0 -t 00:1d:7e:aa:17:3b -i openwrt-kamikaze-7.09-nslu2.bin
Upgrading LKGAA173B 00:1d:7e:aa:17:3b
    . original flash contents  * packet timed out
    ! being erased             - erased
    u being upgraded           U upgraded
    v being verified           V verified 

  Display:
     
+ Status: * timeout occured + sequence error detected 7fffff+000000 ...VVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVV Rebooting... done


Now, the NSL will send DHCP Discover Broadcasts and try to get 192.168.1.77.
After watching ethereal for a few minutes, I just created an 192.168.1.0/24 network with a DHCP server for it and put the NSL there.

Once 192.168.1.77 pongs you should be able to set the password at http://192.168.1.77 or through ssh upon loging in with root:root

Set up network in /etc/config/network, shutdown and plug it into the real network.

check CPU, RAM and `Disk` Info


BusyBox v1.4.2 (2007-12-28 10:50:24 UTC) Built-in shell (ash)
Enter 'help' for a list of built-in commands.

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 KAMIKAZE (7.09) -----------------------------------
  * 10 oz Vodka       Shake well with ice and strain
  * 10 oz Triple sec  mixture into 10 shot glasses.
  * 10 oz lime juice  Salute!
 ---------------------------------------------------
root@OpenWrt:~# cat /proc/cpuinfo 
Processor : XScale-IXP42x Family rev 2 (v5b)
BogoMIPS : 266.24
Features : swp half fastmult edsp 
CPU implementer : 0x69
CPU architecture: 5TE
CPU variant : 0x0
CPU part : 0x41f
CPU revision : 2
Cache type : undefined 5
Cache clean : undefined 5
Cache lockdown : undefined 5
Cache format : Harvard
I size  : 32768
I assoc  : 32
I line length : 32
I sets  : 32
D size  : 32768
D assoc  : 32
D line length : 32
D sets  : 32

Hardware : Linksys NSLU2
Revision : 0000
Serial  : 0000000000000000
root@OpenWrt:~# cat /proc/meminfo 
MemTotal:        30472 kB
MemFree:         21412 kB
Buffers:           848 kB
Cached:           2552 kB
SwapCached:          0 kB
Active:           2464 kB
Inactive:         1784 kB
SwapTotal:           0 kB
SwapFree:            0 kB
Dirty:               0 kB
Writeback:           0 kB
AnonPages:         896 kB
Mapped:            760 kB
Slab:             3276 kB
SReclaimable:     1392 kB
SUnreclaim:       1884 kB
PageTables:        156 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:     15236 kB
Committed_AS:     2904 kB
VmallocTotal:   991232 kB
VmallocUsed:     16828 kB
VmallocChunk:   966652 kB
root@OpenWrt:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/mtdblock5            5.4M    400.0k      5.0M   7% /jffs
mini_fo:/jffs             1.0M      1.0M         0 100% /
root@OpenWrt:~# 

Not Bad!




Put OpenWrt on a Linksys Network Storage Link NSL

MAC-telnet daemon etc debian

Install mndp, MAC-telnet-server , mactelnet & macping ( mactelnet writer and source )

# mkdir mactelnet
# cd mactelnet
# wget http://ftp.de.debian.org/debian/pool/main/m/mactelnet/mactelnet_0.4.0-1.debian.tar.gz
# wget http://ftp.de.debian.org/debian/pool/main/m/mactelnet/mactelnet_0.4.0.orig.tar.gz
# tar xvzf mactelnet_0.4.0.orig.tar.gz
# cd mactelnet-0.4.0/
# make all install
# cd ..
# tar xvzf mactelnet_0.4.0-1.debian.tar.gz
# cd debian/
# cp mactelnet-server.init /etc/init.d/mactelnet-server
# chmod 755 /etc/init.d/mactelnet-server
# update-rc.d mactelnet-server defaults
# chmod 600 /etc/mactelnetd.users
add a system user and a password in /etc/mactelnetd.users

Discover mactelnet daemons in the LAN, similar to mactelnet -l
$ mndp 
Searching for MikroTik routers... Abort with CTRL+C.

MAC-Address       Identity (platform version hardware) uptime
a0:b3:cc:e9:4b:55 acube (Linux 2.6.32-5-amd64 x86_64) up 2 days 0 hours
d4:ca:6d:5d:4d:9f ATH_Rocks (MikroTik 5.24 RB750) up 2 days 0 hours



Macping
# macping d4:ca:6d:5d:4d:9f
d4:ca:6d:5d:4d:9f 56 byte, ping time 0.74 ms
d4:ca:6d:5d:4d:9f 56 byte, ping time 0.71 ms
^C
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.71/0.73/0.74 ms


Mactelnet
# mactelnet d4:ca:6d:5d:4d:9f
Login: admin
Password: 
Connecting to d4:ca:6d:5d:4d:9f...done








  MMM      MMM       KKK                          TTTTTTTTTTT      KKK
  MMMM    MMMM       KKK                          TTTTTTTTTTT      KKK
  MMM MMMM MMM  III  KKK  KKK  RRRRRR     OOOOOO      TTT     III  KKK  KKK
  MMM  MM  MMM  III  KKKKK     RRR  RRR  OOO  OOO     TTT     III  KKKKK
  MMM      MMM  III  KKK KKK   RRRRRR    OOO  OOO     TTT     III  KKK KKK
  MMM      MMM  III  KKK  KKK  RRR  RRR   OOOOOO      TTT     III  KKK  KKK

  MikroTik RouterOS 5.24 (c) 1999-2013       http://www.mikrotik.com/
[admin@ATH_Rocks] > 


Telnet into your debian system from another host in the LAN.
[admin@ATH_Rocks] > /tool mac-telnet A0:B3:CC:E9:4B:55
Login: rox
Password: 
Trying A0:B3:CC:E9:4B:55...
Connected to A0:B3:CC:E9:4B:55
Linux acube 2.6.32-5-amd64 #1 SMP Mon Feb 25 00:26:11 UTC 2013 x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
rox@acube:~$ 






MAC Telnet Debian

software RAID 1 debian notes

Notes taken while I was converting an 1 hard drive disk system to a software RAID 1 system.

The system.
# cat /etc/issue /etc/debian_version
Debian GNU/Linux 6.0 \n \l

6.0.7



Install the MD devices --Linux Software RAID-- Manager
# apt-get install mdadm


The System is installed on a ~250GB hard drive.
# fdisk -l

Disk /dev/sda: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00096f89

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1       29933   240430080   83  Linux
/dev/sda2           29933       30402     3766273    5  Extended
/dev/sda5           29933       30402     3766272   82  Linux swap / Solaris


Shutdown the system, add one more ~250GB disk, and boot

What partitions does the system see?
cat /proc/partitions 
major minor  #blocks  name

   8        0  244198584 sda
   8        1  240430080 sda1
   8        2          1 sda2
   8        5    3766272 sda5
   8       16  244198584 sdb
Yupe, the second drive --sdb is identical to the first drive --sda.

List all active md devices and their status
# cat /proc/mdstat 
Personalities :
unused devices: 


Copy the sda partition scheme to sdb.
# sfdisk -d /dev/sda | sfdisk --force /dev/sdb


Check the partition tables with sfdisk.
# sfdisk -l

Disk /dev/sdb: 30401 cylinders, 255 heads, 63 sectors/track
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sdb1   *      0+  29932-  29933- 240430080   83  Linux
/dev/sdb2      29932+  30401-    469-   3766273    5  Extended
/dev/sdb3          0       -       0          0    0  Empty
/dev/sdb4          0       -       0          0    0  Empty
/dev/sdb5      29932+  30401-    469-   3766272   82  Linux swap / Solaris

Disk /dev/sda: 30401 cylinders, 255 heads, 63 sectors/track
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sda1   *      0+  29932-  29933- 240430080   83  Linux
/dev/sda2      29932+  30401-    469-   3766273    5  Extended
/dev/sda3          0       -       0          0    0  Empty
/dev/sda4          0       -       0          0    0  Empty
/dev/sda5      29932+  30401-    469-   3766272   82  Linux swap / Solaris


Check the partition tables with fdisk.
# fdisk -l

Disk /dev/sdb: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xdb0cbcd3

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1       29933   240430080   83  Linux
/dev/sdb2           29933       30402     3766273    5  Extended
/dev/sdb5           29933       30402     3766272   82  Linux swap / Solaris

Disk /dev/sda: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00096f89

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1       29933   240430080   83  Linux
/dev/sda2           29933       30402     3766273    5  Extended
/dev/sda5           29933       30402     3766272   82  Linux swap / Solaris


Change the sdb1 and sdb5 partitions Ox ID to 'Linux raid auto' ( fd )
# fdisk /dev/sdb

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): m
Command action
   a   toggle a bootable flag
   b   edit bsd disklabel
   c   toggle the dos compatibility flag
   d   delete a partition
   l   list known partition types
   m   print this menu
   n   add a new partition
   o   create a new empty DOS partition table
   p   print the partition table
   q   quit without saving changes
   s   create a new empty Sun disklabel
   t   change a partition's system id
   u   change display/entry units
   v   verify the partition table
   w   write table to disk and exit
   x   extra functionality (experts only)

Command (m for help): t
Partition number (1-5): 1
Hex code (type L to list codes): L

 0  Empty           24  NEC DOS         81  Minix / old Lin bf  Solaris        
 1  FAT12           39  Plan 9          82  Linux swap / So c1  DRDOS/sec (FAT-
 2  XENIX root      3c  PartitionMagic  83  Linux           c4  DRDOS/sec (FAT-
 3  XENIX usr       40  Venix 80286     84  OS/2 hidden C:  c6  DRDOS/sec (FAT-
 4  FAT16 <32M      41  PPC PReP Boot   85  Linux extended  c7  Syrinx         
 5  Extended        42  SFS             86  NTFS volume set da  Non-FS data    
 6  FAT16           4d  QNX4.x          87  NTFS volume set db  CP/M / CTOS / .
 7  HPFS/NTFS       4e  QNX4.x 2nd part 88  Linux plaintext de  Dell Utility   
 8  AIX             4f  QNX4.x 3rd part 8e  Linux LVM       df  BootIt         
 9  AIX bootable    50  OnTrack DM      93  Amoeba          e1  DOS access     
 a  OS/2 Boot Manag 51  OnTrack DM6 Aux 94  Amoeba BBT      e3  DOS R/O        
 b  W95 FAT32       52  CP/M            9f  BSD/OS          e4  SpeedStor      
 c  W95 FAT32 (LBA) 53  OnTrack DM6 Aux a0  IBM Thinkpad hi eb  BeOS fs        
 e  W95 FAT16 (LBA) 54  OnTrackDM6      a5  FreeBSD         ee  GPT            
 f  W95 Ext'd (LBA) 55  EZ-Drive        a6  OpenBSD         ef  EFI (FAT-12/16/
10  OPUS            56  Golden Bow      a7  NeXTSTEP        f0  Linux/PA-RISC b
11  Hidden FAT12    5c  Priam Edisk     a8  Darwin UFS      f1  SpeedStor      
12  Compaq diagnost 61  SpeedStor       a9  NetBSD          f4  SpeedStor      
14  Hidden FAT16 <3 63  GNU HURD or Sys ab  Darwin boot     f2  DOS secondary  
16  Hidden FAT16    64  Novell Netware  af  HFS / HFS+      fb  VMware VMFS    
17  Hidden HPFS/NTF 65  Novell Netware  b7  BSDI fs         fc  VMware VMKCORE 
18  AST SmartSleep  70  DiskSecure Mult b8  BSDI swap       fd  Linux raid auto
1b  Hidden W95 FAT3 75  PC/IX           bb  Boot Wizard hid fe  LANstep        
1c  Hidden W95 FAT3 80  Old Minix       be  Solaris boot    ff  BBT            
1e  Hidden W95 FAT1
Hex code (type L to list codes): fd
Changed system type of partition 1 to fd (Linux raid autodetect)

Command (m for help): t
Partition number (1-5): 5
Hex code (type L to list codes): fd
Changed system type of partition 5 to fd (Linux raid autodetect)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.


Check sdb partitions.
# fdisk -l /dev/sdb

Disk /dev/sdb: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xdb0cbcd3

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1       29933   240430080   fd  Linux raid autodetect
/dev/sdb2           29933       30402     3766273    5  Extended
/dev/sdb5           29933       30402     3766272   fd  Linux raid autodetect


Zero-out any remains of RAID arrays on /dev/sdb
# mdadm --zero-superblock /dev/sdb1
# mdadm --zero-superblock /dev/sdb5


Create the RAID arrays using the older 0.90 superblock format.
# mdadm --create /dev/md0 --level=1 --metadata=0 --raid-disks=2 missing /dev/sdb1
mdadm: array /dev/md0 started.
# mdadm --create /dev/md1 --level=1 --metadata=0 --raid-disks=2 missing /dev/sdb5
mdadm: array /dev/md1 started.
"The version-0.90 superblock limits the number of component devices within an array to 28, and limits each component device to a maximum size of 2TB on kernel version <3.1 and 4TB on kernel version >=3.1."

List MD devices and MD Status.
# cat /proc/mdstat 
Personalities : [raid1] 
md1 : active raid1 sdb5[1]
      3766208 blocks [2/1] [_U]
      
md0 : active raid1 sdb1[1]
      240430016 blocks [2/1] [_U]
      
unused devices: 


Create File Systems on MD devices.
# mkfs.ext3 -v -m .1 -b 4096 -E stride=32,stripe-width=64 /dev/md0
# mkswap /dev/md1


Set /etc/mdadm/mdadm.conf
# cp /etc/mdadm/mdadm.conf /etc/mdadm/mdadm.conf.0
Delete ARRAY definitions in /etc/mdadm/mdadm.conf
# mdadm --examine --scan >> /etc/mdadm/mdadm.conf


Mount md0
# mkdir /mnt/md0
# mount /dev/md0 /mnt/md0


Set
/dev/md0 to mount at /
and
/dev/md1 as swap
in /etc/fstab for now
# mkdir /etc/bak
# cp /etc/fstab /etc/bak/fstab.0
# vi /etc/fstab
# cat /etc/fstab |grep md
/dev/md0  /               ext3    errors=remount-ro 0       1
/dev/md1  none            swap    sw              0       0


Configure Grub (2).
# cp /etc/grub.d/40_custom /etc/grub.d/09_raid1
# uname -r
2.6.32-5-amd64
Set /etc/grub.d/09_raid1
# vi /etc/grub.d/09_raid1
# cat /etc/grub.d/09_raid1
#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
menuentry 'Debian GNU/Linux, with Linux 2.6.32-5-amd64 raid1' --class debian --class gnu-linux --class gnu --class os {
        insmod raid
        insmod mdraid
        insmod part_msdos
        insmod ext2
        set root='(md0)'
        echo    'Loading Linux 2.6.32-5-amd64 ....'
        linux   /boot/vmlinuz-2.6.32-5-amd64 root=/dev/md0 ro  quiet
        echo    'Loading initial ramdisk ... .'
        initrd  /boot/initrd.img-2.6.32-5-amd64
}


Uncomment GRUB_DISABLE_LINUX_UUID=true in /etc/default/grub
# vi /etc/default/grub




Back up /etc/mtab
# cp /etc/mtab /etc/bak/mtab.0
Replace /dev/sda1 with /dev/md0 in /etc/mtab.

Update Grub
# grub-mkconfig -o /boot/grub/grub.cfg
Update the initramfs image
# update-initramfs -u


Copy the system to /dev/md0
# cp -dpRx / /mnt/md0/


Install grub on both drives.
# grub-install /dev/sdb
# grub-install /dev/sda


Reboot
# shutdown -r now


Check if md is used.
# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md0              226G  1.6G  213G   1% /
tmpfs                 943M     0  943M   0% /lib/init/rw
udev                  936M  152K  936M   1% /dev
tmpfs                 943M     0  943M   0% /dev/shm




Set the Ox partition IDs on /dev/sda to "Linux RAID auto" ( fd ).
# for part in 1 5;do sfdisk --change-id /dev/sda $part fd;done


Add /dev/sda5 to the /dev/md1 RAID array
# mdadm --add /dev/md1 /dev/sda5
mdadm: added /dev/sda5
Check
# cat /proc/mdstat 
Personalities : [raid1] 
md1 : active raid1 sda5[2] sdb5[1]
      3766208 blocks [2/1] [_U]
      [>....................]  recovery =  3.3% (124800/3766208) finish=7.7min speed=7800K/sec
      
md0 : active raid1 sdb1[1]
      240430016 blocks [2/1] [_U]
      
unused devices: 


Add /dev/sda1 to /dev/md0
# mdadm --add /dev/md0 /dev/sda1
mdadm: added /dev/sda1


Check MD Status again.
# cat /proc/mdstat 
Personalities : [raid1] 
md1 : active raid1 sda5[2] sdb5[1]
      3766208 blocks [2/1] [_U]
      [=>...................]  recovery =  6.2% (234368/3766208) finish=55.2min speed=1064K/sec
      
md0 : active raid1 sda1[2] sdb1[1]
      240430016 blocks [2/1] [_U]
       resync=DELAYED
      
unused devices: 


Wait until the 're'sync is done.

Come on!
# cat /proc/mdstat 
Personalities : [raid1] 
md1 : active raid1 sda5[0] sdb5[1]
      3766208 blocks [2/2] [UU]
      
md0 : active raid1 sda1[2] sdb1[1]
      240430016 blocks [2/1] [_U]
      [===========>.........]  recovery = 59.1% (142333696/240430016) finish=23.1min speed=70622K/sec
      
unused devices: 


Finally.
# cat /proc/mdstat 
Personalities : [raid1] 
md1 : active raid1 sda5[0] sdb5[1]
      3766208 blocks [2/2] [UU]
      
md0 : active raid1 sda1[0] sdb1[1]
      240430016 blocks [2/2] [UU]
      
unused devices: 


Install Grub2 everywhere and reboot.
# grub-install /dev/sdb
# grub-install /dev/sda
# grub-install /dev/md0
# reboot


Test! What would happen in a hard drive failure?

Shutdown the system and pull out a hard disk drive.
# shutdown -h now
Pull out /dev/sdb and Boot.

The system came up with one drive and a degraded Array.
# cat /proc/mdstat
Personalities : [raid1] 
md1 : active (auto-read-only) raid1 sda5[0]
      3766208 blocks [2/1] [U_]
      
md0 : active raid1 sda1[0]
      240430016 blocks [2/1] [U_]
      
unused devices: 


Let's test how it would come up if the /dev/sda fails.
# shutdown -h now
Put in /dev/sdb, pull out /dev/sda, and boot the system.

Great, the system came up again with one drive on a degraded Array.
# cat /proc/mdstat 
Personalities : [raid1] 
md1 : active (auto-read-only) raid1 sda5[1]
      3766208 blocks [2/1] [_U]
      
md0 : active raid1 sda1[1]
      240430016 blocks [2/1] [_U]
      
unused devices: 


# shutdown -h now
Put in /dev/sda again and boot the system.

The system comes up fine.
# cat /proc/mdstat 
Personalities : [raid1] 
md1 : active (auto-read-only) raid1 sda5[0] sdb5[1]
      3766208 blocks [2/2] [UU]
      
md0 : active raid1 sdb1[1]
      240430016 blocks [2/1] [_U]
      
unused devices: 
Oups, sda1 is missing from the RAID 1 Array md0.
# mdadm -a /dev/md0 /dev/sda1
mdadm: re-added /dev/sda1
Check
# cat /proc/mdstat 
Personalities : [raid1] 
md1 : active (auto-read-only) raid1 sda5[0] sdb5[1]
      3766208 blocks [2/2] [UU]
      
md0 : active raid1 sda1[2] sdb1[1]
      240430016 blocks [2/1] [_U]
      [>....................]  recovery =  0.6% (1466496/240430016) finish=88.4min speed=45012K/sec
      
unused devices: 


Of course in case of a real hard drive failure the new drive entered is not going to have the same partition table with the system nor the superblock information, nor grub2 installed. If the new drive entered is identical to the one used by the RAID Array the steps describing how to add a drive to the RAID 1 drive Array should work fine, eg:
# sfdisk -d /dev/OLD-GOOD-DRIVE | sfdisk --force /dev/NEW-REPLACEMENT-DRIVE
# mdadm --zero-superblock /dev/NEW-REPLACEMENT-DRIVE-PARTITIONS
# mdadm -a /dev/RAID-DEVICE /dev/NEW-REPLACEMENT-DRIVE-PARTITIONS
# grub-install /dev/NEW-REPLACEMENT-DRIVE


If the new hard drive is not an identical drive you will have to work a bit more creating partitions that match the ones used by the system.

Well, it is not a good idea to use a RAID 1 Array on a swap md ... And you can get away without creating a RAID 0 array for swap. You could use /dev/sda5 and /dev/sdb5 as 2 different swap partitions.

References:
RAID superblock formats
Raid Setup
Swapping_on_RAID



Software RAID 1 on Debian GNU Linux

devz How-to

devz - DEVeloper' S Stupid Servant - How to.

I wrote devz a while ago to help me out with workflows that look like;
work & test on a system -> push the changes to the production systems.

Devz was written in an evening and it has been helping me and a few friends for ~three years. Recently, I wanted to use devz on another project so I beautified it a bit and wrote this how-to.

devz has 4 basic verbs-functionalities that do 4 simple things.
  1. toprod: sends a file to a list of systems using scp
  2. ctoprod: sends a set of commands to a list of systems over ssh
  3. stor: stores a file in ./stor/file.n
  4. fromprod: fetches a file from the first production system
There are a few helper functions.
  • devz-setagent: initiates an ssh-agent session
  • devz-showconfig: see the files used in the configuration and list the production systems






updated devz howto





To list the devz commands type devz.
$devz
******
devz
DEVeloper'S Stupid Servant
g0 2010 - http://ipduh.com/contact
******
devz verbs:
*
'toprod' or 'devz toprod'
 toprod file
 scp a file to the production server(s)
*
'ctoprod' or 'devz ctoprod'
 ctoprod 'command;command;' 
 send commands to all production server(s)
*
'fromprod' or 'devz fromprod'
 fromprod file
 scp a file from the first production server here.
*
'stor' or 'devz stor'
 stor file 
 creates the directory stor in the current directory if it does not exist.
 makes a copy of the file in stor
 the file gets a version number like file.n where n [0,n]
*
'devz-setagent' or 'devz setagent'
 setagent
 start an ssh-agent login session
*
'devz-showconfig' or 'devz showconfig'
 showconfig
 See and Current devz configuration
*
'devz-setconfig' or 'devz setconfig'
 setconfig
 set production-servers list file
******


Use devz

At the beginning of each session you better start an ssh-agent session.
turor:~$devz setagent
Enter passphrase for /tutor/.ssh/id_dsa: 
Identity added: /tutor/.ssh/id_dsa (/tutor/.ssh/id_dsa)


Let's list the devz configuration.
tutor:~$devz showconfig
******
PRODUCTION SERVERS LIST from /tutor/.devzconfig/production-servers
1) - 10.19.143.6 -- 22 -- root -
2) - 10.255.9.14 -- 22 -- root -
***
IDENTITY: /tutor/.ssh/id_dsa
******


Send a command to the production servers
tutor:~$ctoprod 'cat /proc/cpuinfo |grep name' 
devz: root@10.19.143.6:22 "cat /proc/cpuinfo |grep name"
model name : Intel(R) Xeon(R) CPU           X5560  @ 2.80GHz
model name : Intel(R) Xeon(R) CPU           X5560  @ 2.80GHz
model name : Intel(R) Xeon(R) CPU           X5560  @ 2.80GHz
model name : Intel(R) Xeon(R) CPU           X5560  @ 2.80GHz
devz: root@10.255.9.14:22 "cat /proc/cpuinfo |grep name"
model name : Intel(R) Xeon(R) CPU           E5310  @ 1.60GHz
model name : Intel(R) Xeon(R) CPU           E5310  @ 1.60GHz
model name : Intel(R) Xeon(R) CPU           E5310  @ 1.60GHz
model name : Intel(R) Xeon(R) CPU           E5310  @ 1.60GHz


Send a file to the production systems
tutor:~$cd /tmp
tutor:~$echo test > test
tutor:~$toprod test
devz:/tmp/test to root@10.19.143.6:22:/tmp/test
test                                                            100%    5     0.0KB/s   00:00    
devz:/tmp/test to root@10.255.9.14:22:/tmp/test
test                                                            100%    5     0.0KB/s   00:00    
tutor:~$


Stor a file
tutor:/tmp$stor test
devz:The directory ./stor does not exist! I will create it.
devz:test is at ./stor/test.0


a bit more on stor
tutor:/tmp$stor test
devz:test is the same as ./stor/test.0
devz:I did not stor test


a bit more on stor
tutor:/tmp$echo next > test
tutor:/tmp$stor test
devz:test is at ./stor/test.1


Fetch a file from a production system
tutor:/tmp$fromprod test
devz:test exists! Please stor it and delete it or rename it or delete it.
tutor:/tmp$rm test
tutor:/tmp$fromprod test
devz:root@10.19.143.6:22:/tmp/test to /tmp/test
test                                                            100%    5     0.0KB/s   00:00    


Install devz

devz is a simple extension of your bash environment and depends on many unix utilities.
devz assumes that you have created at least one ssh identity - ssh key-pair and that copies of your public key are in the authorized keys of the production systems.

#wget http://kod.ipduh.com/lib/devz
#mv devz /bin
#chmod 755 /bin/devz
To install elsewhere eg /usr/local/bin change DEVZAT.

Setup devz for user tutor
$ whoami
tutor
$ mkdir ~/.devzconfig
$ echo "source /bin/devz" >> ~/.bashrc
$ source ~/.bashrc
A production-servers list should be in ~/.devzconfig and look like the following.
$cat ~/.devzconfig/production-servers
# production servers
# IP address , SSH TCP port, user
10.19.143.6,22,root
10.255.9.14,22,root
Lines beginning with a # are ignored.

Create an SSH key pair
$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/tutor/.ssh/id_dsa): 
Created directory '/home/tutor/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/tutor/.ssh/id_dsa.
Your public key has been saved in /home/tutor/.ssh/id_dsa.pub.
The key fingerprint is:
88:5a:f3:75:ed:c3:92:ed:04:a0:bf:bc:d7:e9:7c:68 tutor@some-ipduh-system
The key's randomart image is:
+--[ DSA 1024]----+
|                 |
|                 |
|        .        |
|     . o . .     |
|    + o S o .    |
|   o + o . =     |
|  .   . . oo*.   |
|       . +E*+.   |
|        ++ o+    |
+-----------------+

$ ls -a ~/.ssh
.  ..  id_dsa  id_dsa.pub
Now, you need to copy your public key in each
Production-System-X:/USER-HOME-DIR-PATH/.ssh/authorized_keys2

eg for root@10.19.143.16: 10.19.143.6:/root/.ssh/authorized_keys2

more on ssh-keys

devz source

devz is just a few hundred lines of easy to follow bash code and depends on many standard unix programs.

##devz DEVeloper'S Stupid Servant
##devz g0 2010 - http://ipduh.com/contact
## GNU GPLvX 
## Dependencies: bash,source,grep,awk,ssh,ssh-agent,scp,diff,pwd,expr,ps,cat 
## One of these days I 'll write it in C --not , g0 
## http://alog.ipduh.com/2013/04/devz-how-to.html

VERBOSE="1"
PRO_SRV="${HOME}/.devzconfig/production-servers"
EGO="devz"
DEVZAT=/bin/${EGO}
IDENTITY="${HOME}/.ssh/id_dsa"

function devz {
 MEAT=${DEVZAT}
 PT2="##"

if [ "${1}" = "toprod" ]
then
  toprod ${2}

elif [ "${1}" = "fromprod" ]
then
 fromprod ${2}

elif [ "${1}" = "stor" ]
then
 stor ${2}

elif [ "${1}" = "setagent" ]
then
 devz-setagent

elif [ "${1}" = "showconfig" ]
then
 devz-showconfig

elif [ "${1}" = "setconfig" ]
then
 devz-setconfig
else 
 echo "******"
 echo "devz"
 grep "${PT2}devz " ${MEAT} |awk -F "${PT2}devz " '{print $2}'
 echo "******"
 echo "devz verbs:"
 echo "*" 
 echo "'toprod' or 'devz toprod'"
 grep "${PT2}toprod" ${MEAT} |awk -F "${PT2}toprod" '{print $2}' 
 echo "*" 
        echo "'ctoprod' or 'devz ctoprod'"
        grep "${PT2}ctoprod" ${MEAT} |awk -F "${PT2}ctoprod" '{print $2}'
        echo "*"
 echo "'fromprod' or 'devz fromprod'"
 grep "${PT2}fromprod" ${MEAT} |awk -F "${PT2}fromprod" '{print $2}' 
 echo "*" 
 echo "'stor' or 'devz stor'"
 grep "${PT2}stor" ${MEAT} |awk -F "${PT2}stor" '{print $2}' 
 echo "*" 
 echo "'devz-setagent' or 'devz setagent'"
 grep "${PT2}devz-setagent" ${MEAT} |awk -F "${PT2}devz-setagent" '{print $2}'  
 echo "*" 
 echo "'devz-showconfig' or 'devz showconfig'"
 grep "${PT2}devz-showconfig" ${MEAT} |awk -F "${PT2}devz-showconfig" '{print $2}'  
 echo "*" 
 echo "'devz-setconfig' or 'devz setconfig'"
 grep "${PT2}devz-setconfig" ${MEAT} |awk -F "${PT2}devz-setconfig" '{print $2}'  
 echo "******"

fi 
}
##
##devz-setconfig setconfig
##devz-setconfig set production-servers list file
##
function devz-setconfig {
 PROSRV=""
 ls -l ${PRO_SRV}*
 read -p "Set production-servers list: " PROSRV;
 cat ${PROSRV} > ${PRO_SRV}
 devz-showconfig 
}
##
##devz-showconfig showconfig
##devz-showconfig See the Current devz configuration
##
function devz-showconfig {
 
 echo "******"
 echo "PRODUCTION SERVERS LIST from ${PRO_SRV}"
 COUNTER=1 
 for SERVER in `grep -v -E '^#|^$' ${PRO_SRV}`
        do
                PRODUCTION_SRV=`echo ${SERVER} | awk -F "," '{print $1}'` 
                PRODUCTION_SRV_PORT=`echo ${SERVER} | awk -F "," '{print $2}'`
                PRODUCTION_SRV_USER=`echo ${SERVER} | awk -F "," '{print $3}'` 
                echo "${COUNTER}) - $PRODUCTION_SRV -- $PRODUCTION_SRV_PORT -- $PRODUCTION_SRV_USER -"
  (( COUNTER += 1 ))
 done
 echo "***"
 echo "IDENTITY: ${IDENTITY}"
 echo "******"
}
##
##devz-setagent setagent
##devz-setagent start an ssh-agent login session
##
function devz-setagent {
 ssh-add -ls 2>/dev/null
 if [ $? -eq 0 ]
 then
   echo "${EGO}:It seems that active identities are held by the ssh-agent"
 else
   ssh-agent sh -c "ssh-add ${IDENTITY} && bash"
 fi
}
##
##toprod toprod file
##toprod scp a file to the production server(s)
##
function toprod {
 PWD=`pwd`
 
if [ -z "${1}" ]
then
      echo "${EGO}:toprod WHAT?"
      echo "${EGO}:Type devz for help with all commands."
elif [ ! -e ${PRO_SRV} ]
then
 echo "${EGO}:Make sure that a readable ${PRO_SRV} exists and contains at least one not null DEVZ_PROD_SRV"
else
 for SERVER in `grep -v -E '^#|^$' ${PRO_SRV}`
        do
                PRODUCTION_SRV=`echo ${SERVER} | awk -F "," '{print $1}'`
                PRODUCTION_SRV_PORT=`echo ${SERVER} | awk -F "," '{print $2}'`
                PRODUCTION_SRV_USER=`echo ${SERVER} | awk -F "," '{print $3}'`
  if [ ${VERBOSE} -eq 1 ]
  then
   echo "${EGO}:${PWD}/${1} to $PRODUCTION_SRV_USER@$PRODUCTION_SRV:$PRODUCTION_SRV_PORT:${PWD}/${1}"
  fi 
  scp -r -P ${PRODUCTION_SRV_PORT} -i ${IDENTITY} $1 ${PRODUCTION_SRV_USER}@${PRODUCTION_SRV}:${PWD}/$1
        done
fi  
}
##
##ctoprod ctoprod 'command;command;' 
##ctoprod send commands to all production server(s)
##
function ctoprod {
        PWD=`pwd`

if [ -z "${1}" ]
then
      echo "${EGO}:ctoprod WHAT?"
      echo "${EGO}:Type devz for help with all commands."
elif [ ! -e ${PRO_SRV} ]
then
        echo "${EGO}:Make sure that a readable ${PRO_SRV} exists and contains at least one not null DEVZ_PROD_SRV"
else
        for SERVER in `grep -v -E '^#|^$' ${PRO_SRV}`
        do
                PRODUCTION_SRV=`echo ${SERVER} | awk -F "," '{print $1}'`
                PRODUCTION_SRV_PORT=`echo ${SERVER} | awk -F "," '{print $2}'`
                PRODUCTION_SRV_USER=`echo ${SERVER} | awk -F "," '{print $3}'`
                if [ ${VERBOSE} -eq 1 ]
                then
                        echo "${EGO}: ${PRODUCTION_SRV_USER}@${PRODUCTION_SRV}:${PRODUCTION_SRV_PORT} \"$1\""
                fi
                ssh -p ${PRODUCTION_SRV_PORT} -i ${IDENTITY} ${PRODUCTION_SRV_USER}@${PRODUCTION_SRV} ${1}
        done
fi
}
##
##fromprod fromprod file
##fromprod scp a file from the first production server here.
##
function fromprod {

if [ -z "${1}" ]
then
      echo "${EGO}:fromprod WHAT?"
      echo "${EGO}:Type devz for help with all commands."
else 
 if [ -e $1 ]
        then
  echo "${EGO}:$1 exists! Please stor it and delete it or rename it or delete it."
        else
  SERVER=`grep -v -E '^#|^$' -m 1 ${PRO_SRV}` 
  PRODUCTION_SRV=`echo ${SERVER} | awk -F "," '{print $1}'`
                PRODUCTION_SRV_PORT=`echo ${SERVER} | awk -F "," '{print $2}'`
                PRODUCTION_SRV_USER=`echo ${SERVER} | awk -F "," '{print $3}'`

  if [ ${VERBOSE} -eq 1 ]
                then
                        echo "${EGO}:$PRODUCTION_SRV_USER@$PRODUCTION_SRV:$PRODUCTION_SRV_PORT:${PWD}/${1} to ${PWD}/${1}"
                fi
  
  scp -r -P ${PRODUCTION_SRV_PORT} -i ${IDENTITY} ${PRODUCTION_SRV_USER}@${PRODUCTION_SRV}:${PWD}/$1 .
 fi 
fi
}
##
##stor stor file 
##stor creates the directory stor in the current directory if it does not exist.
##stor makes a copy of the file in stor
##stor the file gets a version number like file.n where n [0,n]
##
function stor {

if [ -z "${1}" ]
then
 echo "${EGO}:stor WHAT?"
        echo "${EGO}:Type devz for help with all commands."
else

 
 if [ ! -d ./stor ]
 then
  echo "${EGO}:The directory ./stor does not exist! I will create it."
  mkdir ./stor 
 fi

 if [ -e ./stor/$1.0 -a -d ./stor ]
        then
                declare -a FILES=( `ls ./stor |grep ${1} |awk -F "${1}." '{print $2}'` )
  FILEV="" #file versions string 
 
  for i in $(seq 0 $((${#FILES[@]} -1)))
  do
          if [[ "${FILES[$i]}" =~ ^[0-9]+$ ]]
          then
                  FILEV="$FILEV ${FILES[$i]}"
          fi
  done
  
  declare -a FILEVL=( ${FILEV} )
  LAST=0
  
  for j in $(seq 0 $((${#FILEVL[@]} -1)))
  do
          if [ ${LAST} -lt ${FILEVL[$j]} ]
          then
                  LAST=${FILEVL[$j]}
          fi
  done

         diff ${1} ./stor/${1}.${LAST} > /dev/null
  if [ $? -eq 0 ]
  then
   echo "${EGO}:${1} is the same as ./stor/${1}.${LAST}"
   echo "${EGO}:I did not stor ${1}"
  else
   LAST=`expr ${LAST} + 1`  
   cp $1 ./stor/$1.${LAST}
   echo "${EGO}:${1} is at ./stor/${1}.${LAST}"
  fi
        fi

 if [ ! -e ./stor/$1.0 -a -d ./stor ]
 then
  cp $1 ./stor/$1.0
  echo "${EGO}:$1 is at ./stor/$1.0" 
 fi
fi
}








devz How To