Extremely large .FPT memo files being created by xBase++ app using FOXCDX database engine

I have an xBase++ app that creates a table that has memo fields. The
table is created by consolidating information from several source
tables.

When I run the app complied with Clipper 5.2 the resulting .FPT file
is about 220 MB.

When I run the same app compiled with xBase++ the resulting FPT file
is approximately 20 GB.

I have the driver set up as follows in the xBase app:

// Build DBE
DbeLoad( "FOXDBE", .T.)
DbeLoad( "CDXDBE",.T.)
DbeBuild( "FOXCDX", "FOXDBE", "CDXDBE" )
DbeSetDefault( "FOXCDX" )
 
// Set up for FOX 2.x and Comix compatability
DbeInfo(COMPONENT_DATA, FOXDBE_CREATE_2X, .T. )
DbeInfo(COMPONENT_DATA, FOXDBE_LOCKMODE, FOXDBE_LOCKMODE_CLIPPER)
DbeInfo( COMPONENT_ORDER, CDXDBE_MODE, CDXDBE_COMIX )

Any help is much appreciated.
Tim CallahanPrincipalAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

pcelbaCommented:
This is dependent on the way how you are creating these memo fields and also on the memo block size.

Xbase++ uses memo block size 512 bytes (probably) which can cause memo file large size when you have many short memos.
Clipper and VFP can use smaller blocks.

Also if you are updating existing memo fields then it can be done in two ways:
1) Reuse the space occupied by old memo blocks
2) Leave old memo blocks untouched in memo file and append the updated memo at the end - this generates large memo files

Does Xbase++ support some memo file PACKing? If yes then try it.
0
Tim CallahanPrincipalAuthor Commented:
Thanks for the input.

- by default the xBase driver should use 64 byte memo blocks but I will try setting the explicitly

- xbase does support packing but I *think* it just removes deleted records - I will give that a try

- The memos are created in the new table by copying memos from the source tables so I don't think I am appending?

- Also testing on different machines to see if it is something related to the physical disk setup

More news to come....
0
Tim CallahanPrincipalAuthor Commented:
p.s. the code is pretty simple - here's a condensed version:

do while ! comment->(eof())

   cmntout->(rlock())
   cmntout->note := comment->note
   cmntout->(dbunlock())

   comment->(dbskip())

enddo
0
Keep up with what's happening at Experts Exchange!

Sign up to receive Decoded, a new monthly digest with product updates, feature release info, continuing education opportunities, and more.

pcelbaCommented:
I don't know Xbase++ language but the above code seems to replace just one row in the cmntout table or the append just disappeared when condensing the code. :-)

If memo fields are just appended then the problem is purely in memo block size or in the space reserved in memo file for each db record.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Tim CallahanPrincipalAuthor Commented:
Oops on the code. Yes the append was omittied :)

I have set the blocksize to 64 explicitly now in the database driver setup and get the same results.

It happens on both Win7 and Win Server 2016 so I don't think it is machine/OS related.

Working with xBase++ support now - will post any results as soon as I get them.
0
pcelbaCommented:
I am also interested - 20 gigs should be explained.

Thanks.
0
Tim CallahanPrincipalAuthor Commented:
This was user error.

xBase++ database drivers have a table component (COMPONENT_DATA) and an index component (COMPONENT_ORDER.)

I was setting the parameters on the driver as follows;
 
   DbeInfo(COMPONENT_DATA, FOXDBE_MEMOBLOCKSIZE, 64 )
   DbeInfo(COMPONENT_DATA, CDXDBE_LOCKRETRY, 1000000)

The LOCKRETRY setting should be applied to the ORDER component not the DATA component (I can't say how many times I locked at the code and docs and never caught this.)

Turns out the FOXDBE_MEMOBLOCKSIZE and CDXDBE_LOCKRETRY defines have the same numeric value. Result was I was setting a block size of 1MM.
0
pcelbaCommented:
Great it is solved!  Sounds to me like points should go to Xbase++ support :-)
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
FoxPro

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.