Solved

Foxpro 2 gig dbf limit

Posted on 2011-02-15
8
2,963 Views
Last Modified: 2012-06-27
Guys, is there a way to overcome the 2 gig limitation on Foxpro dbf files.   I have a client adding 100 meg. a month to one file and could reach the file limit size in about a year.  One way is to simply open another database, but he really likes having all the records in one place.

Frank
0
Comment
Question by:fmoore0001
  • 3
  • 2
  • 2
  • +1
8 Comments
 
LVL 12

Accepted Solution

by:
jrbbldr earned 250 total points
Comment Utility
You can use another non-VFP backend for the data tables.

If you use something like SQL Server for your tables, they will support more than 2GB size.

The only issue would be that your queries would need to return record sets less than 2GB so that VFP could work on them.

Good Luck
0
 

Author Comment

by:fmoore0001
Comment Utility
Anyone had some experience with this Woody;s DBF Resizer:
************************************************************
* ResizeDbf 
************************************************************
* Copyright 2007 by wOOdy, ProLib Software GmbH, Germany
************************************************************
* This routine allows you to correct the filesize of a DBF
* depending on the desired Recordcount 
* It automatically takes care about the absolute maximum of 
* possible records
************************************************************

SET TALK OFF
CLEAR
USE

cFileName = GETFILE("DBF")
IF NOT FILE(cFileName)
	RETURN .f.
ENDIF

? "Filename", cFilename
ADIR(aFileInfo, cFileName)
nFileSize = aFileInfo(1,2)
? "Length of file ", nFileSize


nFile = FOPEN(cFileName,12)
IF nFile < 1
	MESSAGEBOX("Error at FileOpen")
	RETURN .F.
ENDIF

* Read the File characteristics from the header area
cHeader   = FREAD(nFile, 32)
nRecCount = CTOBIN(SUBSTR(cHeader, 5, 4), "4SR")
nHeader   = CTOBIN(SUBSTR(cHeader, 9, 2), "2SR")
nRecSize  = CTOBIN(SUBSTR(cHeader,11, 2), "2SR")   	


? "RecordCount", nRecCount
? "HeaderLength", nHeader
? "RecordLength", nRecSize

* Formula for the max number of possible records:
nMaxRec = Int((2^31-nHeader-2)/(nRecsize))
? "Max.Records   ", nMaxRec

nMaxrecNew = VAL(INPUTBOX("New Reccount","Input",TRANSFORM(nMaxRec),0,TRANSFORM(nMaxRec),"0"))
*nMaxRecNew = MIN(nMaxRec, nMaxRecNew)	 && just to be save from idiots
IF nMaxRecNew >0 
	IF MESSAGEBOX(TEXTMERGE("Set file <<cFileName>> to <<nMaxRecNew>> records?"),4+32+256) = 6 
		nNewSize = nHeader + (nRecSize * nMaxRecNew) 
 		FCHSIZE(nFile, nNewSize)	&& change the filesize
		FSEEK(nFile, 4,0)			&& reposition to the header (4 bytes move from Byte 1 = position 5)
		FWRITE(nFile, BINTOC(nMaxRec, "4SR"))	&& poke new reccount in reverse intel format
	ENDIF
ENDIF

FCLOSE(nFile)

TRY 
	USE (cFileName)
	GO BOTTOM
	BROWSE NOWAIT
CATCH
	? "Error on opening"
ENDTRY

Open in new window

0
 
LVL 12

Expert Comment

by:jrbbldr
Comment Utility
Keep in mind that the 2GB limitation relates to the size for the DBF, FPT & CDX files themselves, not the record count.    
Admittedly I have not taken the time to try to understand the code above, but the text references all seem to be about 'records', not file size.

I have a client adding 100 meg. a month to one file and could reach the file limit size in about a year.

With that in mind, you can either do as I suggested above, and put the data into non-VFP database data table(s) (the preferred solution for best longevity) or you can distribute the data across multiple related VFP tables each of which would each be smaller in size.

Good Luck
0
 
LVL 12

Expert Comment

by:jrbbldr
Comment Utility
"One way is to simply open another database, but he really likes having all the records in one place."

As long as you are talking about VFP, the 'database' is very seldom the issue.  
It is the 'data tables' (the DBF, CDX, & FPT files) that most often come up against the 2GB file limitation issue.  

Yes, if your VFP   DBC,DBX,DBT files (the 'database' files) were to exceed 2GB you would indeed have problems, but that is seldom the issue.

If "he" wants to be the one to design the data architecture, then, by all means, let him do it and let him experience the consequences.   But if 'he' wants things to work, then get him to be open to advice from others.

NOTE - putting the data into one or more SQL Server data tables would not only keep "all the records in one place", it would also add security to the data via SQL Server security.

Good Luck
0
Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

 
LVL 29

Expert Comment

by:Olaf Doschke
Comment Utility
wOOdys DBF resizer is adressing the probem of a dbf reaching the 2GB limit, to fix it back to all records below that limit, it does not extend dbfs to be capable to store more than 2GB.

"takes care about the absolute maximum of possible records" means it cuts off everything in the dbf above that limit, it does not extend the maximum. It truncates all records above 2GB. And it's also usable to fix a header corruption in regard to reccount.

You commented that line: *nMaxRecNew = MIN(nMaxRec, nMaxRecNew)       && just to be save from idiots

Now the program is not safe from idiots. Sorry, not to offend you, but if you extend a dbf to contain more records than nMaxRecNew computed a few lines before that, you're making the dbf file larger than 2GB but also inaccessable and not usable to foxpro anymore.

Bye, Olaf.
0
 
LVL 29

Assisted Solution

by:Olaf Doschke
Olaf Doschke earned 250 total points
Comment Utility
correction: nMaxRec is the max possible number, nMaxRecNew is the number the user can enter via INPUTBOX.

Anyway, FCHSIZE(nFile, nNewSize) will already fail, if nNewSize is above 2GB as also the low level file handle function FOPEN FSEEK, FREAD, FPUTS, FCLOSE and FCHSIZE are limited to 2GB too.

It's possible to create and read files larger than 2GB with foxpro using Windows API or using Scripting.FilesystemObject and it's .OpenTextFile() method and .ReadLine() etc.

But also that does not help extending DBFs. (or FPT, CDX)

If the problem is with FPT, consider storing contents of the memo/blob fields as seperate files and point to that file in the DBF. If the problem is within the DBF try to determine if you're not wastin space with too large char fields, if the problem is within the CDX try to reindex and/or to put some indexes in secondary CDX or IDX files.

Other possibilities is a 1:1 vertical split of the file, eg put fields1 to n in one dbf, field n+1 to M in another dbf, this is limited up to the point each DBF has only one field of the record of course and does not help much in the case a single fpt field (memo/blob) is the source of the file growth.

Then you can of course add tables, table1,table2, table3 etc. to keep the record structure, but that makes queries complicated, as you need to union query results from each table and make sure that union doesn't grow larger than 2GB.

What can also save a few bytes is SET BLOCKSIZE TO 0 for fpt. But it's also no miracle reducing fpt lower than the data you store within.

The argument of keeping all in one (or a few) files is nonsense, as you can keep all seperate files in a subfolder, then to that directory it also does not mean bloating the file list, it's one folder in there. Of course with many files inside, but what does that matter? It will not use much more disc space. And copying the data to another place will also not take more time. Store file neames relative and the data also is portable to other folders like any DBC and it's DBFs, that also store there file position relative to the DBC. In case of storing images in Genereal fields you even save disc space despite the fact you're storing more files and each entry in the file system also takes a bit of disc space.

Bye, Olaf.
0
 
LVL 41

Expert Comment

by:pcelba
Comment Utility
All the records in one place slow the performance down. You have to convince your client what means "one place" for database stored on file server. One Folder is the entity we are talking about...

Horizontal or vetical split is the only VFP solution today. (To wait for the next version is still not worth to do.)

BTW, does your client really need the data all together in one file? We are obviously working with (or reporting) a small subset. If they need e.g. monthly cumulative data then such data should be precalculated in separate tables for closed months and the only live data query should read current month which is much faster.
0
 

Author Closing Comment

by:fmoore0001
Comment Utility
Thanks for the help guys.  I think we need to consider some sort of an SQL back end.
0

Featured Post

Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

Join & Write a Comment

Microsoft Visual FoxPro (short VFP) is a programming language with it’s own IDE and database, ranking somewhat between Access and VB.NET + SQL Server (Express). Product Description: http://msdn.microsoft.com/en-us/vfoxpro/default.aspx (http://msd…
Find out what Office 365 Transport Rules are, how they work and their limitations managing Office 365 signatures.
Internet Business Fax to Email Made Easy - With eFax Corporate (http://www.enterprise.efax.com), you'll receive a dedicated online fax number, which is used the same way as a typical analog fax number. You'll receive secure faxes in your email, fr…
Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now