Original transcript: http://www.coraid.com/support/linux/eg/raid5-lvm.txt
Modified by: John Suomi
Date: April 11, 2006 @ 9:41 AM
OVERVIEW
This is my transcript of the initial creation of the logical volume
from a RAID 5 on the shelf with address 1. I’m using CentOS4.3 and
a Coraid 1U 4 Disk SATA+RAID Ethernet storage appliance (CED1U4SS).
CONFIGURATION
The appliance is configured as a JBOD to use software RAID and LVM
on the host server. This will allow for future appliance expansion.
It has a shelf address of 1.
The RAID 5 utilizes 4 disks and 0 hot spares. Since the host server
already had two LVM volumes 0, I configured the appliance
volume as 2.
NOTES
Coraid’s original transcript utilized the jfs file system,
but CentOS (RH4) doesn’t support it out-of-the-box. I may look deeper
into the jfs file system in future configurations.
[root@photo ~]# modprobe aoe
[root@photo ~]# aoe-stat
e1.0 400.088GB eth1 up
e1.1 400.088GB eth1 up
e1.2 400.088GB eth1 up
e1.3 400.088GB eth1 up
[root@photo ~]# mdadm -C -l 5 -n 4 /dev/md0 /dev/etherd/e1.[0-3]
mdadm: array /dev/md0 started.
[root@photo ~]# pvcreate /dev/md0
Physical volume “/dev/md0″ successfully created
[root@photo ~]# vgcreate pool0 /dev/md0
Volume group “pool0″ successfully created
[root@photo ~]# vgdisplay pool0
— Volume group —
VG Name pool0
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 1
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 0
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size 1.09 TB
PE Size 4.00 MB
Total PE 286165
Alloc PE / Size 0 / 0
Free PE / Size 286165 / 1.09 TB
VG UUID S2mwnR-xIXt-4JP1-xFpT-BUGp-Zckn-vpV4B1
[root@photo ~]# lvcreate -l 286165 -n vol1 pool0
Logical volume “vol1″ created
[root@photo ~]# mkfs -t ext3 /dev/pool0/vol1
Warning! All data on device /dev/pool0/vol1 will be lost!
Continue? (Y/N) y
mke2fs 1.35 (28-Feb-2004)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
146522112 inodes, 293032960 blocks
14651648 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=293601280
8943 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 28 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
[root@photo ~]# mkdir /mnt/vol1
[root@photo ~]# mount /dev/pool0/vol1 /mnt/vol1
[root@photo ~]# rsync -a /etc/init.d /mnt/vol1
Now the logical volume, “vol1″, has a ext3 mounted on /mnt/vol1.
ADDING ANOTHER SHELF
In the future, wwe will add another shelf (with shelf address 2).
On this Fedora Core 3 system, there’s no /dev/md1 file, so we use
mdadm’s “auto” option to create it as needed.
[root@photo ~]# ls /dev/md1
ls: /dev/md1: No such file or directory
[root@photo ~]# mdadm -C -l 5 -n 4 –auto=md /dev/md1 /dev/etherd/e2.[0-4]
mdadm: array /dev/md1 started.
[root@photo ~]# cat /proc/mdstat
We make an LVM physical volume out of the new RAID 5 and then add it
to the volume group. Then the logical volume is extended to take up
the new space.
[root@photo ~]# pvcreate /dev/md1
Physical volume “/dev/md1″ successfully created
[root@photo ~]# vgextend pool0 /dev/md1
Volume group “pool0″ successfully extended
[root@photo ~]# vgdisplay pool0
[root@photo ~]# lvextend -l +88349 /dev/pool0/vol0
Extending logical volume vol0 to 651.88 GB
Logical volume vol0 successfully resized
[root@photo ~]#
The filesystem doesn’t know that the block device it’s on is bigger,
so next we grow the jfs (while it’s mounted and usable). This took
quite some time for me, since the RAIDs were still initializing.
[root@photo ~]# df -h /mnt/vol0
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pool0-vol0
307G 39M 307G 1% /mnt/vol0
[root@photo ~]# mount -o remount,resize /mnt/vol0
[root@photo ~]# df -h /mnt/vol0
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pool0-vol0
652G 82M 652G 1% /mnt/vol0
[root@photo ~]#
SHUTTING DOWN and STARTING UP
I added these to the aoe-init script located in /etc/init.d/
To shut down the storage cleanly, shut down each layer from the top
layer down.
[root@photo ~]# umount /mnt/vol1
[root@photo ~]# vgchange -an pool0
0 logical volume(s) in volume group “pool0″ now active
[root@photo ~]# mdadm -S /dev/md0
Later, to bring the storage back online, make each layer active from
the bottom up.
[root@photo ~]# mdadm -A /dev/etherd/e1.[0-3]
mdadm: /dev/etherd/e1.0 does not appear to be an md device
[root@photo ~]# mdadm -A /dev/md0 /dev/etherd/e1.[0-3]
mdadm: /dev/md0 has been started with 8 drives (out of 9) and 2 spares.
[root@photo ~]# vgscan
Reading all physical volumes. This may take a while…
Found volume group “VolGroup00″ using metadata type lvm2
Found volume group “pool0″ using metadata type lvm2
[root@photo ~]# vgchange -ay pool0
1 logical volume(s) in volume group “pool0″ now active
[root@photo ~]# mount /dev/pool0/vol0 /mnt/vol0
[root@photo ~]# ls /mnt/vol0/
init.d
[root@photo ~]#
