Setting up gvinum raid5 on FreeBSD 7 ( amd64 )

Hello, I'm trying to get gvinum raid5 running on FreeBSD 7 and have hit a road block, here's the run down.

I have an IDE HD ( 200gb ), which has the freeBSD install on it and 4 500GB hard drives ( exact same hard drives ) connected via sata2. I have thoroughly tested all 5 hard drives on other systems and they are all good.

After installing freeBSD I did the following


#bsdlabel -w /dev/ad4
#bsdlabel -w /dev/ad5
#bsdlabel -w /dev/ad6
#bsdlabel -w /dev/ad7

3) created my raid config file "/etc/raid.conf"

drive disk_1 device /dev/ad4
drive disk_2 device /dev/ad5
drive disk_3 device /dev/ad6
drive disk_4 device /dev/ad7
volume raid5
plex org 261k
sd drive disk_1
sd drive disk_2
sd drive disk_3
sd drive disk_4

3) Created the array

#gvinum create /etc/raid.conf

4) Partitioned the new array

#newfs /dev/gvinum/raid5

5) Created Directory and Mounted it

#mkdir /wmfiles
#mount /dev/gvinum/raid5 /wmfiles

And voila , the volume, plex, and subdisks were running and I could write to wmfiles perfectly fine.

I then add the new mount to fstab

/dev/gvinum/raid5                /wmfiles                 ufs                  rw                 2                 2

and add it to the /boot/loader.conf


Here's where things go wrong, I reboot and the boot fails because of the new volume. It throws the following error and puts me into a crippled single user mode where I must remount /usr, /, and /var just to use my /rescue/vi

        ufs: /dev/gvinum/raid5 (/wmfiles)
Automatic file system check failed; help!

when I run

#fsck /dev/gvinum/raid5

I get


which seems to be the first block, followed by ( after continue )

THE FOLLOWING DISK SECTORS COULD NOT BE READ: 128, 129 [etc etc] 142, 143,
ioctl (GCINFO): Inappropriate ioctl for device
fsck_ufs: /dev/gvinum/raid5: can't read disk label

#gvinum list

returns that the volume and plex are down and the subdisks are stale, if I remove the fstab line and the loader.conf line, and reboot, then try to mount and start the volume manually, nothing happens.

Anyone have any ideas as to how to fix this? I am a relative UNIX newbie, so please tell me exactly how to do something if you'd like more information ( ie if you ask for a config file print out etc, tell me how to get said printout, thanks )

Who is Participating?
gheistConnect With a Mentor Commented:
raid5 volume has no checksums initiated.

dd if=/dev/zero of=/dev/gvinum/raid5 bs=64k

will do the job.

Otherwise you will end up again in unreadable sectors in areas where checksums are not present.

Partitioning using fdisk and labeling using disklabel is highly recommended given error messages you receive (sysinstall does both easy way after plex & volume is running)

Have a look at smartmontools package for physical disk monitoring.

achaean1Author Commented:
Thanks, I will try it out when I get home from work. I'll reset the raid config and partition with fdisk instead of newfs.
achaean1Author Commented:
Ok, so I don't understand how to use sysinstall to do this, when I run disklabel the disks show up as their separate ad4 to ad7 selves rather than a single raid array. There is an ar0, is this the raid array? The size it presents is not the array size though. Only ad0 shows up in the disklabel tool.

I attempted to run the dd if=/dev/zero command , unfortunately it doesn't seem to show progress, I stopped it after 10 minutes and it had only processed about 600mb, which would estimate out to about 15 days, so I'm probably confused amid my lack of knowledge here.

Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Errrmmm.... Your setup is that slow... and gvinum is not loaded yet....
achaean1Author Commented:
I didn't think my setup was slow, athlon64 X2 5400+, 7200RPM hard drives, 2gb ram.

What do you mean gvinum is not loaded yet?
Given 1MB per minute write speed is very low.

Given your descriptopn gvinum was not loaded at time of creation, so odds are high that metatdata are not written correctly.
achaean1Author Commented:
Thanks for the assistance thus far,

Any suggestions on how to speed it up? Should I be trying anything specific aside from playing with the plex org size?

What is cpu usage of gvinum kernel module when you do dd command? (from top)

RAID has slow writes by definition:
Basically when 64k block is written gvinum module reads 261KB off the four disks and writes 261KB back to two disks.

Let me suggest RAID 10 :
Because it is highly fault tolerant in hot-swap enclosure.

Normally disk DMA size is 32KB, so stripes bigger than that do not contribute to IO performance, so let me recommend setting plex to 32K and redo IO benchmarks to see improvement.

If that is still unacceptable go with RAID10

Let me suggest iobench benchmark as it is able to simulate different loads, like random reading/writing multiple 2GB slices in 16K blocks as PostgreSQl does, reading multiple <1MB files in 32KB blocks like Apache, or writing 500 files <16MB in 16KB blocks like sendmail.

Getting blocksize below page size which is 4KB on amd64 would put unnecessary load on memory manager.
achaean1Author Commented:
Seems regardless of what I do I can't get it going faster than 7mbs, I've decided to just shell out the money for a RAID card, never really found out if your solution worked or not, but you've deserved the points for effort.
Have you tried RAID0/RAID1 versions? I guess checksumming is delay factor. I do get RAID0 nearly n times faster on read or write using old bad disk arrays. I know it is walking the fire, but sometimes helps. Thats drop-box for g4u images, so I do not care - just bring down "array" on SMART warnings, copy disk, restart array. Thats complicated only for forst time, next time it goes quite easy.

RAID cards accommodate NCQ etc and give simplet SCSI-like interface, so kernel has less job to do.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.