Dynamic array data gets trashed when I increase size with Setlength

Hello Experts... I have a dynamic array of FileStream's that can get bigger only... never smaller. Reason is I need to have (possibly) many filestreams open at once as I need to write a record to the appropriate stream based on the number of fields the record has. For instance, if I read a record and it has 7 fields, I need to write it to FileStream[7]... if the next record I read has 22 fields and my Filestream array is only 0..7 in length, I then need to do a Setlength on my FileStream array to increase the length to 23 Filestream elements (ie. 0..22). My code basically looks like this:


             // check if an output file already has been opened for this particular fieldcount
                BuildFileName := FNnoext+'-'+IntToStr(numfields);

                TRY
                  SetLength(HBuffer, Length(holdrec)+2);
                  StrPCopy(PChar(HBuffer), holdrec+#13#10);
                  bytes_Wrote:=OutFilesStream[numfields].Write(HBuffer[0], StrLen(PChar(HBuffer)));
                EXCEPT
                  If numfields+1 > Length(OutFilesStream) then  // This file doesn't yet exist, increase the size of our dynamic files array
                     SetLength(OutFilesStream, numfields+1);
                  OutFilesStream[numfields]:=TGpHugeFileStream.Create(BuildFileName, accWrite);
                  bytes_Wrote:=OutFilesStream[numfields].Write(HBuffer[0], StrLen(PChar(HBuffer)));
                END;

Open in new window



(I'm using "TGpHugeFileStream" component from Primoz Gabrijelcic instead of a regular TFileStream. Shouldn't make any difference).

Somehow, my dynamic array data is getting trashed when I try to increase the size of it with the SetLength(OutFilesStream, numfields+1); statement. Stepping through it in debug mode revealed this:

- after the declaration and initial Setlength of the array to 0 (not shown), the array contains simply: ()
- after reading the first data record with 8 fields (numfields=8) and doing a Setlength(numfields+1), the array looked like: nil,nil,nil,nil,nil,nil,nil,nil,nil
- after the file "create" for OutFileStream[8], the array looked like : nil,nil,nil,nil,nil,nil,nil,nil,$1297078
- okay, so far so good, but when the next record is read... one with 9 fields this time, the trouble happens. Upon doing the Setlength(numfields+1), this is what it did to the OutFileStream array: nil,(),(),(),(),(),(),(),(),nil

  ...It lost my file handle info for element [8]. Any thoughts anyone?

Thanks!
   Shawn
shawn857Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

ste5anSenior DeveloperCommented:
I think I know what's happening. But the problem is: Your code snippet is incomplete. It does not show use the important part... Always post a complete and concise example.

I guess: It's a scope problem. The OutFilesStream array is declared inside the loop (you said next record).

btw, you're misusing the try-except construct to control your program flow. This is a very bad practice. Test whether you need to resize your array in the normal control flow.

If numfields + 1 > Length(OutFilesStream) then  
begin
	SetLength(OutFilesStream, numfields+1);
	OutFilesStream[numfields]:=TGpHugeFileStream.Create(BuildFileName, accWrite);
end;

SetLength(HBuffer, Length(holdrec)+2);
StrPCopy(PChar(HBuffer), holdrec+#13#10);
bytes_Wrote := OutFilesStream[numfields].Write(HBuffer[0], StrLen(PChar(HBuffer)));

Open in new window


Also the generic exception handling may mask the problem, thus don't use it all.

And depending on your Delphi version use Ctrl+D to get a uniform formatted code.
0
shawn857Author Commented:
Thanks Ste5an... the reason I structured it with the TRY..EXCEPT was for speed. Before that, I explicitly checked to see if the proper output file existed before attempting to write to it. While profiling the code, I noticed that this check was a big bottleneck... and found the try..except technique while googling. I agree that it's not the most elegant.
   Thanks for your suggested code, but I'm afraid it won't work. You see, what about the situation where the length of the dynamic array is say, 9 and then along comes a record with say, 4 fields that I have not yet encountered. Your code in this case, would not do the Filestream.create for OutFilesStream[4), and thus fail on the following Write statement to that particular file.
   Regarding the scope issue you touched on - I think I should be okay there as I declare the OutFilesStream dynamic array globally, and do the "SetLength(OutFilesStream, 0)" statement just a little bit earlier in the code segment I originally included.

Thanks!
   Shawn
0
ste5anSenior DeveloperCommented:
Well, it was just a guess. Without a complete and concise example...
0
Geert GOracle dbaCommented:
dynamic array ?
a stream per field of a record ?

so you want to write all colums of the record in parallel through a single network connection ?
i'd write serially ... but hey ... why not do it different ?

why not use a TList for your stream collection ?
i'd create the streams in the getter of the list if it doesn't exist for that index

you might want to override the add method of TList too

type
  TStreamClass = class of TStream;

  TStreamList = class(TList)
  private
    fStreamClass: TStreamClass;
    function GetStream(Index: Integer): TStream; 

  public
    property Stream[Index: Integer]: TStream read GetStream;
  end;


function TStreamList.GetStream(Index: Integer): TStream; 
begin
  while Count-1 < Index do 
    Add(fStreamClass.Create);
  Result := TStream(inherited Items[Index]);
end;

Open in new window


way simpler than a dynamic array.
the cleanup needs a little bit attention and grow and shrinking is already part of Tlist
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Geert GOracle dbaCommented:
ugh ... global dynamic array's with primoz otl library ?
i hope you thought of some protection within the multi threading ?
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Delphi

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.