Expanding ext2 file systems

Since it fails passt 8GB.
Are you sure your blocksize is 4K ??
The maximum ext2 filesystem size is 4G * (blocksize in K).
Therefore, your problem pretty much sounds like 2K block size. Just recheck.

Output of tune2fs shows the following; Block Size: 4096

can you post your vgdisplay -v output here?

This line:
"resizing the file system on /dev/vol_grp/vol_data to -1612120064 (4K) blocks"

pretty much sounds like an integer overflow, using a signed instead of an unsigned integer.
What version of resize2fs? Redhat 3.4 is pretty old.
Try to grab a current version of resize2fs, compile it on your system and do the resize again.
rpm -qf `which resize2fs`. Unfortunately, I don't have access to a 3.4 system. Otherwise, one could run rezise2fs with a debugger and actually see what happens. I pretty much suspect the "old" version of resize2fs.
Also, 3.4 has gone up to 3.9. Is your system up to the latest revision?

The last version for Redhat 3 was:
http://mirror.centos.org/centos/3.9/os/i386/RedHat/RPMS/e2fsprogs-1.32-15.3.i386.rpm

  I'm sorry this won't be the best comment I've ever left, but...

  I encountered something like what you're seeing.  Here's my Slashdot journal post about doing it.  It was a 6.4Tb system in 2004.

  http://slashdot.org/journal/90480/64Tb-filesystem-array-we-have-a-winner?art_pos=87

  I mention using xfs there, but I don't see where I mentioned ext2 being a pain.  It didn't want to handle it, and that was a blocksize issue.
 
  I believe some of it was fixed in later kernels and fs tools.  I was using 2.6.9 in 2004.  You're using an older environment, dating back to about the same period.  What you may be running into isn't actually a problem, but a limitation of the tools.  

  If you can't upgrade the OS, you may want to try building new lvm and ext2fs tools from current sources, and upgrade your kernel.  

  Keep this as an option.  Hopefully someone else will come in with a better answer.

/usr/sbin/vgdisplay -v
Finding all volume groups
Finding volume group "vol_grp"

---Volume Group ---
VG Name vol_grp
System ID
Format LVM2
Metadata Areas 5
Metadata Sequence No 6
VG Access read/write
VG Status resizeable
MAX LV 0
Cur LV 1
Open LV 0
Max PV 0
Cur PV 5
Act PV 5
VG Size 10.00 TB
PE Size 32.00 MB
Total PE 327559
Alloc PE/Size 327558 / 10.00 TB
Free PE/Size 1 / 32.00 MB
VG UUID bHuXvK-Vej5-0IUX-Y3yQ-qRzv-ev3b-qw1s7s

---Logical Volume ---
LV Name /dev/vol_grp/vol_data
VG Name vol_grp
LV UUID
LV Write Access read/write
LV STatus available
# open 0
LV Size 10.00 TB
Current LE 327558
Segments 5
Allocation inherit
Read ahead sectors 0
Block device 254:0

---- Physical Volumes ----
PV Name /dev/sda3
PV UUID cUBLAb-P0IH-HA7X-w0Yw-abPm-8Z5c-aYUCrT
PV Status allocatable
Total PE/Free PE 65419/0

PV Name /dev/sdb
PV UUID 15HIlX-DtA5-BQ4x-v4tp-7ooM-0oeb-1FUlj8
PV Status allocatable
Total PE/Free PE 65535/0

PV Name /dev/sdc
PV UUID KaqWgh-xSv9-bINH-ehlg-Cs9m-xZxg-n6CqQl
PV Status allocatable

PV Name /dev/sdd
PV UUID 2g9r2H-mDa2-1Ngl-B59i-UvGf-BUPV-8mft5h
PV Status allocatable
Total PE/Free PE 65535/1

PV Name /dev/sde
PV UUID KlDtgD-54L3-0jiv-SbjX-zEzD-3Rie-8Hv0Kf
PV Status allocatable
Total PE/Free PE 65535/0

The system I am working on is a security/video surveillance system and the OS/kernel cannot be changed without "breaking the software".  That being said, I am using version 1.35 of resize2fs.  I would assume I can update the tools without necessarily impacting the software I am running.

So the logical volume is fine and - as I said - most likely it's a tool resize problem!
Therefore, what is:

rpm -qf `which resize2fs`

Can you upgrade? See link above.

I'm not talking about kernel upgrade. Just the latest version of the tool for Redhat 3!
And only the tools. All userland!

Understood.  I just downloaded the rpm from the location you provided and am copying to a CD.  For some reason on this system, there is no device label created for the USB CD-ROM and I have been unsuccessful in getting it to mount.  any ideas why?  Your normal sda1, etc are already assigned to devices.  I can boot from the USB CDROM but can access it through the directory structure.

That should be "cannot" access it through the directory structure.

What does dmesg say? Look for the device there. It should give you a hint as to which /dev/? it is.
Look at the last lines of dmesg right after you've plugged in the the usb plug.
Doesn't the machine have a network port?

it does have a network port but they have telnet, ssh all locked down.  

dmesg says
new high speed usb device using ehci_hcd and address 3, thats it for the cd rom

So the usb cdrom emulation does not exist in Redhat 3 ...
But you do have access to the console...you could start sshd by hand ...
Have you ever transferred any data to the machine?

no, unfortunately not, I haven't worked on these much.  

Does
/etc/init.d/sshd start
work? So you could connect over the network?
The e2fsprogs-1.32-15.3.i386.rpm package ould fit on a floppy disk ...
However there may be other dependencies on that package that you will also have to transfer to the machine.

Just by chance: is there a
/dev/scd0
If yes, as root do a
mount /dev/scd0 /mnt

Make sure nothing else is mounted on /mnt beforehand. Just type the mount command by itself.

I forgot:
Once you're on the console, you can use outgoing network connections,, can't you?

/etc/init.d/sshd start fails.
I have since been able to telent into the box from another machine but if I try to use ftp, I get connection refused.  
The system doesn't have a floppy disk either.  It is basically a server with a 12TB hardware RAID and the security software I mentioned.  It is very bare bones.  
There is no /dev/scd0 either.  I have removed and reconnected the USB CDROM and USB flash drive multiple times.  lsusb shows them with the correct manufacturer etc. but they don't generate a device name to mount to in the dmesg output.  

As for network connections, I can ping the server and can telnet to it but not in the opposite direction.  Can't ping my workstation or ftp.  

I was able to get the new package copied to the box via SMB share however the next issue is the installation results in the following error;
rpm: error while loading shared libraries: librpm-4.3.so: cannot open shared object file: No such file or directory.

That's a really strange one...
Did the people, that setup the system, delete librpm-4.3.so???
What is the output of

ldd `which rpm`

We can try to make rpm running ...

result of ldd 'which rpm' is no such file or directory

????
And

which rpm

by itself?

smathes, those are suppose to be backquotes, not singlequotes.

Attached is an example of each.  You'll see single quotes (the one by the enter key) throws the error.  The backquote (the one at the top left) doesn't.

root @ evil2 (/root)ldd 'which rpm'
ldd: ./which rpm: No such file or directory

root @ evil2 (/root)ldd `which rpm`
	linux-vdso.so.1 =>  (0x00007fffa41ff000)
	librpmbuild.so.0 => /usr/lib64/librpmbuild.so.0 (0x00007f893323a000)
	librpm.so.0 => /usr/lib64/librpm.so.0 (0x00007f8932fc8000)
	libmagic.so.1 => /usr/lib64/libmagic.so.1 (0x00007f8932dad000)
	librpmio.so.0 => /usr/lib64/librpmio.so.0 (0x00007f8932b88000)
[snip]

Select allOpen in new window

[root@vnas 15]# ldd `which rpm`
bash: which: command not found
ldd: missing file arguments
Try 'ldd --help' for more information

tedious ...

ldd /bin/rpm

and if that fails

ldd /usr/bin/rpm

Your path may not be set up right.   You'll have to play with this a little bit, to even attempt it.

BTW, I'm not taking over this one.  hemmi can help you out once we get you finding files. :)

We know ldd is in your path, so you're ok there.

He was only using which to figure out where rpm is.  It's not important for us to find it right now.

Find where RPM is.  Mine is in /bin/.  It could be /bin/ /usr/bin/ /usr/local/bin/ /sbin/ /usr/sbin/ or /usr/local/sbin/

You can also do "locate rpm | grep bin", and get an idea of where it resides.  Once you find the full path to it, have ldd check it.  For example, if it is in /bin/, you would type:  ldd /bin/rpm

Give us the output from that, and it will say what the missing libraries are.

The missing "which" is because it is Redhat 3. On old systems "which" used to be an alias to the

type

bash builtin command.
PATH, I guess, will be correct. The ldd is being found.

#ldd /bin/rpm
linux-gate.so.1 => (oxffffe000)
librpm-4.3.so => not found
librpmdb-4.3.so => not found
libselinux.so.1 => /lib/libselinux.so.1 (0xb7fd7000)
librpmio-4.3.so => not found
libpopt.so.0 => /usr/lib/libpopt.so.0 (0xb7fd0000)
libelf.so.1 => /usr/lib/libelf.so.1 (0xb7fbf000)
libbeecrypt.so.6 =>/usr/lib/libbeecrypt.so.6 (0xb7fa2000)
libm.so.6 => /lib/tls/libm.so.6 (0xb7f7f000)
libdl.so.2 => /lib/libdl.so.2 (0xb7f7b000)
librt.so.1=> /lib/tls/librt.so.1 (0xb7f67000)
libpthread.so.0 => /lib/tls/libpthread.so.0 (0xb7f55000)
libz.so.1 => /usr/lib/libbz2.so.1 (0xb7f44000)
libbz2.so.1 => /usr/lib/libbz2.so.1 (0xb7f33000)
libc.so.6 => /lib/tls/libc.so.6 (0xb7e0c000)
/lib/ld-linux.so.2 (0xb7feb000)

hemmi - What did you mean by "We can try to make rpm running ..."

I have also tried completely removing my logical volumes and volume groups and creating from scratch to see if using mkfs, I can create an ext2 or ext3 file system on the entire 10TB LV rather then extending it.  Unfortunately, the result of mkfs is always 2TB.  

  smathes,

  The ldd indicates you are missing 3 libraries for RPM to work.  They happen to be from the same package.

  I've downloaded the appropriate RPM and converted it to a tar file, so you can install it.   This is a temporary location, and I will be deleting it after you have successfully fixed your system.

  hemmi can take over once you have RPM working again, which this will do.

  The correct RPM is "rpm-libs-4.3.2-5.fc3.i386.rpm".  

  Log into the machine, and do the following.

cd /
wget http://jwsmythe.com/expertsexchange/smathes/rpm-libs-4.3.2-5.fc3.i386.tgz
tar xvpzf rpm-libs-4.3.2-5.fc3.i386.tgz
ldconfig

Now you should be able to use the rpm tool.   You may want to download and install the rpm (in the same directory, or from your favorite repo) so the system will recognize that they were installed via RPM.


 

JWSmythe,

is there any particular reason, why you wan't to install the fedora libs on the RHEL3 system, instead of using the (correct) RHEL libs??

smathes,
with "running" I mena, make the program "rpm" work again.
I guess the people, that have setup the system tried to prevent whatever rpm operation by deleting the libs. Thus, we'll reinstall them, make rpm work again, install a new resize2f and see whether this will solve your problem.
However, as stated above, unless JWSmythe has very good reasons, why he want's to install the fedora libs on the the RHEL system, you'll need libraries, that I can provide.

I am by no means an expert here so I will defer to you on the appropriate libs to use.  Can you provide location and instructions?

  I misread the original post saying the version.  :(  sorry.  It was the closet I found on rpmfind.net, but it should still work fine.  It only contains the related libraries, so it won't break anything else, and won't stomp on anything existing, since they're obviously missing.

http://www.rpmfind.net/linux/rpm2html/search.php?query=librpm-4.3.so

  If you know where to get a better match for RHEL3, I'll be more than happy to stick them in the same place, and post the updated links.

Well, this is getting very strange. The latest RHEL3 librpm (and the others) is
librpm-4.2.so
but your rpm executable needs
librpm-4.3.so

What strange system you have. It cannot be a result of a redhat update ...
Who has setup this system?
It seems that the libs JWSmythe pointed to are better.
However, I don't really know what's been done to your RHEL system ... or is it Fedora instead?
What is the result of the command

cat /etc/redhat-release

The issue is that it is somewhat a custom mix of all kinds of junk which is what makes it more difficult because I can't just upgrade or go to another file system, etc.  The result of /etc/redhat-release is indeed Fedora Core release 3, sorry for the confusion.  At bootup, it says redhat 3.4.  Very confusing.  I am downloading the libs posted above and will post as soon as I have applied them and tried to run the resize again.  

  It sounds like whoever initially built it wanted some newer features, so they laid FC3 (or parts of it) over RHEL3.4.  No big deal there, but it can cause some confusion, like what we're running into.  I wouldn't recommend it for anyone integrating, because it causes what you're seeing now.  

  I don't have an old RHEL machine handy, but the boot message is probably in /etc/issue or /etc/motd, if you feel like changing it.    If it's not in one of those, you could always do:

cd /etc
grep -i "redhat 3.4" `find .`

  It should spit something back that says it.  Sometimes those are dynamically created, so you may have to hunt deeper into the boot scripts.  That's just an aesthetic change though, so don't worry about it right now.

No! Hang on:
You're mixing things up.
The libs' purpose is to get the rpm running. The people, that have setup, seem to have "destroyed" rpm.
You need the rpm to install a later version of resize2fs.
After you've installed the libs run the command

rpm -qf /sbin/resize2fs

and next we'll see whether we have a more recent version resize2fs for your system!
After that you will install the newer version and only afterwards it makes sense to retry the resize2fs.

My last comment was a reply to smathes message not JWSmythe (overlap).

Hemmi - I got it!  Working on it now.  

I had to download the file and then use Samba to get it on the box because the server doesn't have access to the web.  It downloaded as a .tar file not tgz as in your instructions.  how does that change the steps?

  If it really uncompressed it, change the tar line from

tar xvpzf [filename]

  to

tar xvpf [filename]

  We're just removing the "z" flag, which tells it to ungzip it.  The rest is the same.   If one doesn't work, try the other.

Ok, got them extracted and attempted to install the e2fsprogs rpm again and got the following error;
"Unable to open /usr/lib/rpm/rpmrc for reading: no such file or directory"
There is no RPM directory under /usr/lib.

That missing one is part of the
rpm
package.
The libs were part of the
rpm-libs
package.

JWSmythe, do you want to produce the tgz of the fedora 3 rpm package, or shall I do it?

And:
don't reinstall the e2fsprogs package I mentioned before. That was for RHEL3 (please read what I wrote!).
When we get rpm working, the first thing will be
rpm -qf /sbin/resize2fs

And then we'll find a correct later version of e2fsprogs!

Understood, thank you.

So now I am waiting for you to provide a download, correct?

  Sorry about that.  I was going to just acquire the rpmrc, but it is probably better to do the whole thing.  I was distracted by family saying "lets go Christmas shopping".  Little do they know how important EE is to me.   (trust me, I'm serious there)

  You can collect the file at:

 http://jwsmythe.com/expertsexchange/smathes/rpm-libs-4.3.2-5.fc3.i386.tgz

  The same instructions as before apply.  Just change the filename.

That looks like the same exact thing as the first attempt.  

  Sorry, wrong link.  This is the right one.

http://jwsmythe.com/expertsexchange/smathes/rpm-4.3.2-5.fc3.i386.tgz

Done, what's next?  There were a lot of files in there.  

  Follow hemmi's instructions from here.  I'll keep an eye on this problem, to see if I can help out.  

  He's looking for the output from:

rpm -qf /sbin/resize2fs
 
  BTW, be sure to give him the points on this one, once it's resolved.  I only got you through getting a single tool to work that should have already been working.  I already see he's going to be more valuable for the rest of your problem.

The output is the same;
"error: Unable to open /usr/lib/rpm/rpmrc for reading: No such file or directory"

  It's in the tar file.   Try to extract it again.  Be sure you are root while you're doing this.

cd /
wget http://jwsmythe.com/expertsexchange/smathes/rpm-4.3.2-5.fc3.i386.tgz
tar xvpzf rpm-4.3.2-5.fc3.i386.tgz

Well, in your case, since you're copying it over a samba mount, and it's getting decompressed on the download, it would be

cd /
tar xvpf [path_to_your_file]/rpm-4.3.2-5.fc3.i386.tgz

Watch for errors.  Is it having problems writing anything?   The "cd /" at the beginning is essential.  Otherwise, it'll build the directory tree under where ever you are at that moment.

Sorry for the looong delay ... different timezone: have you installed rpm-4.3.2-5.fc3.i386.tgz (or most likely rather the tar file) in the meantime?

The file extracted fine, with no errors.  The results were as follows.  Awaiting next steps....

/# rpm -qf /sbin/resize2fs
file /sbin/resize2fs is not owned by any package

So, also the database with the information on the installed packages has been deleted...

Please run

resizre2fs -?

to find out the version of the installed resize2fs

... is this tedious ... and it is only going to be a try, whether a newer resize2fs will not show the problems...

resize2fs -?
resize2fs 1.35 (28-Feb-2004)
usage: /sbin/resize2fs [d debug_flags] [-f] [-F] [-p] device [new-size]

  I just did a search for your version of resize2fs, and found someone having a similar problem.  I'll leave it up to you two, to decide how applicable it is.

http://www.linux-archive.org/red-hat-linux/291688-segmentation-fault-when-running-resize2fs-rhel4-12tbs.html

  In that, he was going from 8Tb to 12Tb with RHEL4, and encountered the same problem.  They recommended another tool which did the job for him.  That tool should already be included on your system as well.

Ok, transfer this file to the machine:
http://archives.fedoraproject.org/pub/archive/fedora/linux/core/updates/3/i386/e2fsprogs-1.38-0.FC3.1.i386.rpm

and run
rpm -ihv e2fsprogs-1.38-0.FC3.1.i386.rpm

It will probably fail, but what is the output?
There might be other dependencies missing ...

Is your fs ext2 or ext3?
You can see it with the

mount

command.

It is ext3 and i started with ext2online originally and it repeatedly errored with a block not reserved error.  At that point, we went to resize2fs with an offline resize and ran into the issues we are discussing now.  

I am copying the e2fsprogs that you provided and will post the output shortly.  

rpm -ihv e2fsprogs-1.38-0.FC3.1.i386.rpm
warning: e2fsprogs-1.38-0.FC3.1.i386.rpm: V3 DSA signature: NOKEY, key ID 4f2a6Fd2
error: Failed dependencies:
/sbin/ldconfig is needed by e2fsprogs-1.38-0.FC3.1.i386.rpm
libc.so.6(GLIBC_2.0) is needed by e2fsprogs-1.38-0.FC3.1.i386.rpm
libc.so.6(GLIBC_2.1) is needed by e2fsprogs-1.38-0.FC3.1.i386.rpm
libc.so.6(GLIBC_2.2) is needed by e2fsprogs-1.38-0.FC3.1.i386.rpm
libc.so.6(GLIBC_2.3) is needed by e2fsprogs-1.38-0.FC3.1.i386.rpm
libdl.so.2 is needed by e2fsprogs-1.38-0.FC3.1.i386.rpm
libdl.so.2(GLIBC_2.0) is needed by e2fsprogs-1.38-0.FC3.1.i386.rpm
libdl.so.2(GLIBC_2.1) is needed by e2fsprogs-1.38-0.FC3.1.i386.rpm

Sorry for the long delay ... christmas is getting close ... times are very busy ...

Well, this following should work now:

rpm -ihv --force e2fsprogs-1.38-0.FC3.1.i386.rpm

afterwards check to see if the new version has arrived:

resize2fs -?

That should now give a different result from before (see above).

If it's really the new version, again try the expand the filesystem.

I forced the install of the new e2fsprogs and ran the resize again but the result is the same, except that it shows the new version of resize2fs.

/sbin/resize2fs /dev/vol_grp/vol_data
resize2fs 1.38 (30-June-2005)
Resizing the filesystem on dev/vol_grp/log_vol_data to -1611612160 (4k) blocks

Also tried ext2online and received the following;
ext2online v1.1.18 - 2001/03/18 for EXT2FS 0.5b
ext2online: group 0, block 385 not reserved
ext2online: unable to resize /dev/mapper/vol_grp-vol_data

When you do a
man ext2online
I think it should tell you, that it is only suited for ext2 and not for ext3 ...
How exactly did you call ext2online? What was the commandline?

As I second last resort, I could have a look at an even newer version of resize2fs. That version will, however, not be a fc3 version and I will have to check for the dependencies, as you have a running system, that you probably want to continue to run...

Have you thought about using the extra 2GB differently? Make a partition, mount it and move c. 2GB old stuff to that partition and thus arrive at a similar result?

  On the last error, are you trying ext2online on a mounted filesystem?  
  http://ramblingfoo.blogspot.com/2007/03/ext2online-is-not-always-online.html

  Which is repeated in this:  http://www.quintatec.de/informationen/linux/san/tips-und-tricks/

"ext2online: group 0, block 2 not reserved"
"ext2online: unable to resize /dev/mapper/sanvg-sanlv"

"dann muß das Dateisystem umounted werden und die Vergrößerung mit Hilfe von resize2fs durchgeführt werden."
(translation by Google)
"then the file system must be umounted and be done with the help of the magnification resize2fs."

"umount /dev/sanvg/sanlv"

  And does your kernel support it?

  From http://lists.centos.org/pipermail/centos/2006-August/025988.html

  "You may not be doing anything wrong. From the man page,"
 ...
  "you need to have the Online ext2 resize support
  (CON-FIG_EXT2_RESIZE)  feature  enabled  in  the kernel (after
  applying the appropriate patch for the 2.x series of kernels)."

  I know you said you've done this expansion before, so unless the kernel has changed, this shouldn't be an issue.

Just by chance:
Can you take the system down and boot off a cd? Or does the system have to run all the time?
Booting off some cd will solve you problem very easily!

I will try to address all of the questions in one post.
Hemmi:
How exactly did you call ext2online? What was the commandline?
ext2online /dev/vol_grp/vol_data

Have you thought about using the extra 2GB differently? Make a partition, mount it and move c. 2GB old stuff to that partition and thus arrive at a similar result?

Unfortunately, the software that runs on this server can only address a single logical volume.  I have even tried deleting the VG and LV and recreating so that I can use mkfs to format the entire volume at once rather then extending it and it always results in a 2TB file system when I am finished.  

JWSmythe:
I don't believe ext2online is a viable option given the "ext2" limitation although at this point it was worth trying.  When you say I have done the resize before, yes using resize2fs I have successfully gone from 6TB to 8TB so I would say resize support is there.  

Hemmi:
I can bring the system down, in fact, I have an iso that I reimage with after each failed attempt to ensure we are testing on a clean system that will work once the resize is successful.  I keep detailed notes for each round of tests to ensure it is repeatable if successful.  So to answer your question, I can boot with a CD.  I was of the understanding though that tools like gparted and others can't address LVM volumes, is that what you were thinking?  Or do you have another magic tool that might help me?

I think I addressed everyone's questions above.  Let me know if I missed anything.  

I wasn't aware of the possibility to shutdown the system.
Since you can bring the system down:

Produce a cd with the iso from here:
http://www.sysresccd.org/Download

Boot off the cd.
Simply start with Enter.
It'll ask you for your keyboard layout and afterwards  at the prompt type
wizard

This ususally will setup a graphical system.
Start gparted from the graphical interface. The rest will be fairly obvious. You'll be able to expand the filesystem.
You should be able to resize your existing filesystem. This really seems to be a question of ext3 and not the underlying partition/lvm. Your procedure only fails at the very end when resizing the fs.

I forgot to say:
you do not have to start the graphical interface since you know all command line tools.
Just do, what you've already done in the past. Only use the tools from the booted cd.
I should have asked earlier, whether it'll be possible to boot off a cd.

So gparted supports LVM? Or are we bypassing that function of the CD to get the new resize tools?  I just want to make sure I understand because this will have to be replicated across all the production units.  

  smathes,

  From the gpartd Wikipedia page:

http://en.wikipedia.org/wiki/GParted

"GParted does not support logical volume management (LVM) at present, although this feature has been requested by many users and may be implemented in a future release."

  Sorry.

I booted with the CD and repeated the steps I completed originally. All of the commands leading up to the resize were successful just as before.  When I attempt to resize, it shows the following;
"Resizing the filesystem on dev/vol_grp/log_vol_data to 2683363328 (4k) blocks".  
This is encouraging because previously it showed a negative number and then failed with a segmentation failure.
Unfortunately, it fails this time with the following error;
"resize2fs: Memory allocation failed while trying to resize /dev/vol_grp/vol_data"
"Please run 'e2fsck -fy /dev/vol_grp/vol_data' to fix the filesystem after the aborted resize operation"

I plan to run a few more tests today and then I will post the outcome.  I have been able to remove and recreate the LV and VG then use mkfs to format the entire 10TB.  So far the incremental addition; from 6 to 8TB and then 8 to 10TB consistently fails as well.  Stay tuned for more updates later this afternoon...

It looks like recreating the VG and LV and then using mkfs on the entire 10TB logical volume is the only consistent way to get this to work.  I have a few bugs to iron out with the software that I was building this for but your assistance was extremely helpful.  

You have definitely found a bug in the most recent resize2fs. It'd be nice to identify it by running it with a debugger.
However you'll probably want that machine running.
Thanks to JWSmythe!