kcarrim
asked on
File is not a database
Hi
I am running a Lotus Domino 8.5 server on Centos 5.2 x64 Linux. The installation is on a Intel server with Adaptec RAID controller 1.7Tb and the Notesdata directory is on an external Dell PowerVault MD1000 5.5Tb with RAID 5 configuration. I also have LVM2 installed.
On Wednesday last week we had a power surge. I have 2 x 3KVA UPS installed, but for some reason the external Powervault RAID lost the connection to the hard drives. After i rebooted, i check the RAID BIOS and found that 1 drive failed and the RAID is rebuilding.
After the process completed, i started up the server. FSCK gave me error that the volume is inconsistent and i should run fsck manually. So i ran fsck -y /dev/VolGroup02-LogVol02
This ran for almost 10hours, and supposedly fixed hundreds of errors. Once the server booted up, the file system was mounted as read only. I tried unmounting and remounting the volume, also checked fstab for valid mount points.
fdisk -l stated that the volume does not contain a valid partition table. So i bought 3 x 1tb drives and started transferring the data. I cleared and re-initialized the RAID and created a new RAID5 config.
This time i created a GPT partition table and formatted the file system to ext3, transferred the data back using Krusader and preserved ownership and permissions.
I ran an offline fixup, compact and updall.
I then fired up the Domino server to find that my names.nsf is gone. So copied one from a backup server. On startup i found many errors about files that do not exist. I copied the missing files from the backup server and all went well.
Only problem i am stuck with is user mail files. Some files when opening will give error "File is not a database". The mail files have sizes ranging from 260Mb to 4Gb.
I tried running a fixup, but got the same error that the file is not a database. I have replicas of the user mail files on my backup server but there are a couple of mails missing and i am unable to replicate databases hence the error.
What i have done for the time being is rename the problematic mail files and pushed new replicas from the backup server. The result is a loss of mail and folders for dates 12 - 22 March.
What can i do to recover/repair these mail files?
File-is-not-a-database.jpg
I am running a Lotus Domino 8.5 server on Centos 5.2 x64 Linux. The installation is on a Intel server with Adaptec RAID controller 1.7Tb and the Notesdata directory is on an external Dell PowerVault MD1000 5.5Tb with RAID 5 configuration. I also have LVM2 installed.
On Wednesday last week we had a power surge. I have 2 x 3KVA UPS installed, but for some reason the external Powervault RAID lost the connection to the hard drives. After i rebooted, i check the RAID BIOS and found that 1 drive failed and the RAID is rebuilding.
After the process completed, i started up the server. FSCK gave me error that the volume is inconsistent and i should run fsck manually. So i ran fsck -y /dev/VolGroup02-LogVol02
This ran for almost 10hours, and supposedly fixed hundreds of errors. Once the server booted up, the file system was mounted as read only. I tried unmounting and remounting the volume, also checked fstab for valid mount points.
fdisk -l stated that the volume does not contain a valid partition table. So i bought 3 x 1tb drives and started transferring the data. I cleared and re-initialized the RAID and created a new RAID5 config.
This time i created a GPT partition table and formatted the file system to ext3, transferred the data back using Krusader and preserved ownership and permissions.
I ran an offline fixup, compact and updall.
I then fired up the Domino server to find that my names.nsf is gone. So copied one from a backup server. On startup i found many errors about files that do not exist. I copied the missing files from the backup server and all went well.
Only problem i am stuck with is user mail files. Some files when opening will give error "File is not a database". The mail files have sizes ranging from 260Mb to 4Gb.
I tried running a fixup, but got the same error that the file is not a database. I have replicas of the user mail files on my backup server but there are a couple of mails missing and i am unable to replicate databases hence the error.
What i have done for the time being is rename the problematic mail files and pushed new replicas from the backup server. The result is a loss of mail and folders for dates 12 - 22 March.
What can i do to recover/repair these mail files?
File-is-not-a-database.jpg
ASKER
Thanks for your response. I had clustering setup, but due to the cost of the IBM Clustering license we were forced to disable/discontinue the clustering. What i had setup is a daily push/pull replication to the backup server. Will keep this question open for additional comments/solutions.
So you say that the cost of the clustering licences (if I calculate it right it's about $7,000) is too high compared to the cost of this outage and the havoc it created? What licence do you have, is it a Domino Collaboration Express server? In that case, yes, it's sad...
This site explains the offerings rather well: http://www.ilinktechnology.com/lotus-domino.html
And daily pull-push?? Could you increase that to (working-)hourly? Or two-hourly? It is indeed a load, but what risks you run...
This site explains the offerings rather well: http://www.ilinktechnology.com/lotus-domino.html
And daily pull-push?? Could you increase that to (working-)hourly? Or two-hourly? It is indeed a load, but what risks you run...
ASKER
We have 11 domino servers and about 450 users. We use Notes Standard and Sametime Standard.
According to the software supplier, the cost of the IBM clustering license comes to roughly R600 000 South African rand. That's about 63,379.89 USD
So are we being ripped off? Can you perhaps help with the total costs as per amount of users and servers?
I would like to get this licensing/pricing issue resolved. We can afford $7,000 easily. I think we are being misled. We paid about $65, 000 - $70, 000 for the current licenses.
I have made the changes to run hourly push/pull replication as suggested.
According to the software supplier, the cost of the IBM clustering license comes to roughly R600 000 South African rand. That's about 63,379.89 USD
So are we being ripped off? Can you perhaps help with the total costs as per amount of users and servers?
I would like to get this licensing/pricing issue resolved. We can afford $7,000 easily. I think we are being misled. We paid about $65, 000 - $70, 000 for the current licenses.
I have made the changes to run hourly push/pull replication as suggested.
ASKER
By the way, yes it is Collaboration Express License.
Oh dear, I should have added "but I'm absolutely no expert on Domino pricing, it requires a separate University degree". Even IBM reps have sometimes difficulties in calculating the right price.
What I found:
Express:
- http://www-01.ibm.com/software/lotus/notesanddomino/domino-express-licensing.html
which leads to a price of around $150 per seat.
Standard:
- https://www-112.ibm.com/software/howtobuy/buyingtools/paexpress/Express?P0=E1&part_number=D55JDLL,D55J9LL,D55MRLL,D55MTLL,D55JBLL,D55JGLL&catalogLocale=en_US&Locale=en_US&country=USA&PT=html&S_TACT=none&S_CMP=none
Domino Enterprise server costs about $40 per PVU; one processor is 100 PVU's, but there's a table somewhere showing the number of PVU's per type of processor (see http://www-01.ibm.com/software/lotus/notesanddomino/domino-pvu-licensing.html). I finally found the table (thanks to Google): http://www-01.ibm.com/software/lotus/passportadvantage/pvu_licensing_for_customers.html which says that a licence for a dual-core system costs you $4000... so if you have 11 servers... that'll be a lot of Rand...
- https://www-112.ibm.com/software/howtobuy/buyingtools/paexpress/Express?P0=E1&part_number=D5CPILL,D5CS2LL,D53T5LL&catalogLocale=en_US&locale=en_US&country=USA&PT=html
so the Notes client is probably a little over $150 in your country
Anyway, far too expensive, but please ask a real expert in this (mine)field.
Sorry to have misinformed you...
What I found:
Express:
- http://www-01.ibm.com/software/lotus/notesanddomino/domino-express-licensing.html
which leads to a price of around $150 per seat.
Standard:
- https://www-112.ibm.com/software/howtobuy/buyingtools/paexpress/Express?P0=E1&part_number=D55JDLL,D55J9LL,D55MRLL,D55MTLL,D55JBLL,D55JGLL&catalogLocale=en_US&Locale=en_US&country=USA&PT=html&S_TACT=none&S_CMP=none
Domino Enterprise server costs about $40 per PVU; one processor is 100 PVU's, but there's a table somewhere showing the number of PVU's per type of processor (see http://www-01.ibm.com/software/lotus/notesanddomino/domino-pvu-licensing.html). I finally found the table (thanks to Google): http://www-01.ibm.com/software/lotus/passportadvantage/pvu_licensing_for_customers.html which says that a licence for a dual-core system costs you $4000... so if you have 11 servers... that'll be a lot of Rand...
- https://www-112.ibm.com/software/howtobuy/buyingtools/paexpress/Express?P0=E1&part_number=D5CPILL,D5CS2LL,D53T5LL&catalogLocale=en_US&locale=en_US&country=USA&PT=html
so the Notes client is probably a little over $150 in your country
Anyway, far too expensive, but please ask a real expert in this (mine)field.
Sorry to have misinformed you...
ASKER
I have downloaded and tried about 4 NSF recovery programs. None could recover or read the NSF files.
One database however complained about corrupt file header. Is there a way of fixing file headers?
Seems corrupted file headers may be the issue and in fact the data might still be recoverable?
I have tried:
compact -c
fixup -J -L
None of these seems to be able to recover the databases.
One database however complained about corrupt file header. Is there a way of fixing file headers?
Seems corrupted file headers may be the issue and in fact the data might still be recoverable?
I have tried:
compact -c
fixup -J -L
None of these seems to be able to recover the databases.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Thanks for the prompt replies and information. I have recovered 93% of the data. What i have done is to create new replicas from local computers back to the server. Mail lost is now only for a 7 hour period.
I think your data is lost. Apart from the usual compact/fixup utilities there is nothing that can recover a Notes database when it's returned to a mere sequence of bytes, i.e. unrecognisable by the Domino server. The only way to recover is from the backup server or from backup tapes.
Is this you as well:
http://www-11.lotus.com/ldd/nd6forum.nsf/ReleaseAllThreadedweb/7fc52fab9f0acf7d8525729c0024480b?OpenDocument
What you should know is that Domino has a rather large write-behind cache. If the server is stopped right in the middle of a very busy period, you may get many errors in files. And, what adds to the misery is the fact that you have server and disks on two different servers, probably with a general network in between. Often, one single server with all disks might succeed to save everything before it stops. Finally, the configuration you run is not a fault-tolerant one.
Advice: if power-outages are not uncommon, consider to set-up a cluster of two servers, both with UPS, on a network that is private to both of them (requiring at least 2 network interfaces per system).