Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win

x
?
Solved

Setting up gvinum raid5 on FreeBSD 7 ( amd64 )

Posted on 2008-06-22
12
Medium Priority
?
1,691 Views
Last Modified: 2013-12-06
Hello, I'm trying to get gvinum raid5 running on FreeBSD 7 and have hit a road block, here's the run down.

I have an IDE HD ( 200gb ), which has the freeBSD install on it and 4 500GB hard drives ( exact same hard drives ) connected via sata2. I have thoroughly tested all 5 hard drives on other systems and they are all good.

After installing freeBSD I did the following

1) BSDLABELS

#bsdlabel -w /dev/ad4
#bsdlabel -w /dev/ad5
#bsdlabel -w /dev/ad6
#bsdlabel -w /dev/ad7

3) created my raid config file "/etc/raid.conf"

drive disk_1 device /dev/ad4
drive disk_2 device /dev/ad5
drive disk_3 device /dev/ad6
drive disk_4 device /dev/ad7
volume raid5
plex org 261k
sd drive disk_1
sd drive disk_2
sd drive disk_3
sd drive disk_4

3) Created the array

#gvinum create /etc/raid.conf

4) Partitioned the new array

#newfs /dev/gvinum/raid5

5) Created Directory and Mounted it

#mkdir /wmfiles
#mount /dev/gvinum/raid5 /wmfiles

And voila , the volume, plex, and subdisks were running and I could write to wmfiles perfectly fine.

I then add the new mount to fstab

/dev/gvinum/raid5                /wmfiles                 ufs                  rw                 2                 2

and add it to the /boot/loader.conf

geom_vinum_load="YES"

-------------------------------------------------------
Here's where things go wrong, I reboot and the boot fails because of the new volume. It throws the following error and puts me into a crippled single user mode where I must remount /usr, /, and /var just to use my /rescue/vi

THE FOLLOWING FILE SYSTEM HAD AN UNEXPECTED INCONSISTENCY:
        ufs: /dev/gvinum/raid5 (/wmfiles)
Automatic file system check failed; help!

when I run

#fsck /dev/gvinum/raid5

I get

CANNOT READ BLK: 128

which seems to be the first block, followed by ( after continue )

THE FOLLOWING DISK SECTORS COULD NOT BE READ: 128, 129 [etc etc] 142, 143,
ioctl (GCINFO): Inappropriate ioctl for device
fsck_ufs: /dev/gvinum/raid5: can't read disk label

#gvinum list

returns that the volume and plex are down and the subdisks are stale, if I remove the fstab line and the loader.conf line, and reboot, then try to mount and start the volume manually, nothing happens.

Anyone have any ideas as to how to fix this? I am a relative UNIX newbie, so please tell me exactly how to do something if you'd like more information ( ie if you ask for a config file print out etc, tell me how to get said printout, thanks )


0
Comment
Question by:achaean1
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 7
  • 5
12 Comments
 
LVL 62

Accepted Solution

by:
gheist earned 1500 total points
ID: 21842429
raid5 volume has no checksums initiated.

dd if=/dev/zero of=/dev/gvinum/raid5 bs=64k

will do the job.

Otherwise you will end up again in unreadable sectors in areas where checksums are not present.

Partitioning using fdisk and labeling using disklabel is highly recommended given error messages you receive (sysinstall does both easy way after plex & volume is running)

Have a look at smartmontools package for physical disk monitoring.

0
 

Author Comment

by:achaean1
ID: 21846986
Thanks, I will try it out when I get home from work. I'll reset the raid config and partition with fdisk instead of newfs.
0
 

Author Comment

by:achaean1
ID: 21851418
Ok, so I don't understand how to use sysinstall to do this, when I run disklabel the disks show up as their separate ad4 to ad7 selves rather than a single raid array. There is an ar0, is this the raid array? The size it presents is not the array size though. Only ad0 shows up in the disklabel tool.

I attempted to run the dd if=/dev/zero command , unfortunately it doesn't seem to show progress, I stopped it after 10 minutes and it had only processed about 600mb, which would estimate out to about 15 days, so I'm probably confused amid my lack of knowledge here.

0
Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

 
LVL 62

Expert Comment

by:gheist
ID: 21853054
Errrmmm.... Your setup is that slow... and gvinum is not loaded yet....
0
 

Author Comment

by:achaean1
ID: 21854580
I didn't think my setup was slow, athlon64 X2 5400+, 7200RPM hard drives, 2gb ram.

What do you mean gvinum is not loaded yet?
0
 
LVL 62

Expert Comment

by:gheist
ID: 21858903
Given 1MB per minute write speed is very low.

Given your descriptopn gvinum was not loaded at time of creation, so odds are high that metatdata are not written correctly.
0
 

Author Comment

by:achaean1
ID: 21859460
Thanks for the assistance thus far,

Any suggestions on how to speed it up? Should I be trying anything specific aside from playing with the plex org size?

0
 
LVL 62

Expert Comment

by:gheist
ID: 21870268
What is cpu usage of gvinum kernel module when you do dd command? (from top)

RAID has slow writes by definition:
Basically when 64k block is written gvinum module reads 261KB off the four disks and writes 261KB back to two disks.

Let me suggest RAID 10 : http://en.wikipedia.org/wiki/Nested_RAID_levels#RAID_1.2B0
Because it is highly fault tolerant in hot-swap enclosure.

Normally disk DMA size is 32KB, so stripes bigger than that do not contribute to IO performance, so let me recommend setting plex to 32K and redo IO benchmarks to see improvement.

If that is still unacceptable go with RAID10

0
 
LVL 62

Expert Comment

by:gheist
ID: 21870290
Let me suggest iobench benchmark as it is able to simulate different loads, like random reading/writing multiple 2GB slices in 16K blocks as PostgreSQl does, reading multiple <1MB files in 32KB blocks like Apache, or writing 500 files <16MB in 16KB blocks like sendmail.

0
 
LVL 62

Expert Comment

by:gheist
ID: 21870340
Getting blocksize below page size which is 4KB on amd64 would put unnecessary load on memory manager.
0
 

Author Comment

by:achaean1
ID: 21920183
Seems regardless of what I do I can't get it going faster than 7mbs, I've decided to just shell out the money for a RAID card, never really found out if your solution worked or not, but you've deserved the points for effort.
0
 
LVL 62

Expert Comment

by:gheist
ID: 21920973
Have you tried RAID0/RAID1 versions? I guess checksumming is delay factor. I do get RAID0 nearly n times faster on read or write using old bad disk arrays. I know it is walking the fire, but sometimes helps. Thats drop-box for g4u images, so I do not care - just bring down "array" on SMART warnings, copy disk, restart array. Thats complicated only for forst time, next time it goes quite easy.

RAID cards accommodate NCQ etc and give simplet SCSI-like interface, so kernel has less job to do.
0

Featured Post

VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Attention: This article will no longer be maintained. If you have any questions, please feel free to mail me. jgh@FreeBSD.org Please see http://www.freebsd.org/doc/en_US.ISO8859-1/articles/freebsd-update-server/ for the updated article. It is avail…
Introduction Regular patching is part of a system administrator's tasks. However, many patches require that the system be in single-user mode before they can be installed. A cluster patch in particular can take quite a while to apply if the machine…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
In a previous video, we went over how to export a DynamoDB table into Amazon S3.  In this video, we show how to load the export from S3 into a DynamoDB table.
Suggested Courses

610 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question