?
Solved

Dynamic array data gets trashed when I increase size with Setlength

Posted on 2014-08-15
5
Medium Priority
?
289 Views
1 Endorsement
Last Modified: 2014-08-21
Hello Experts... I have a dynamic array of FileStream's that can get bigger only... never smaller. Reason is I need to have (possibly) many filestreams open at once as I need to write a record to the appropriate stream based on the number of fields the record has. For instance, if I read a record and it has 7 fields, I need to write it to FileStream[7]... if the next record I read has 22 fields and my Filestream array is only 0..7 in length, I then need to do a Setlength on my FileStream array to increase the length to 23 Filestream elements (ie. 0..22). My code basically looks like this:


             // check if an output file already has been opened for this particular fieldcount
                BuildFileName := FNnoext+'-'+IntToStr(numfields);

                TRY
                  SetLength(HBuffer, Length(holdrec)+2);
                  StrPCopy(PChar(HBuffer), holdrec+#13#10);
                  bytes_Wrote:=OutFilesStream[numfields].Write(HBuffer[0], StrLen(PChar(HBuffer)));
                EXCEPT
                  If numfields+1 > Length(OutFilesStream) then  // This file doesn't yet exist, increase the size of our dynamic files array
                     SetLength(OutFilesStream, numfields+1);
                  OutFilesStream[numfields]:=TGpHugeFileStream.Create(BuildFileName, accWrite);
                  bytes_Wrote:=OutFilesStream[numfields].Write(HBuffer[0], StrLen(PChar(HBuffer)));
                END;

Open in new window



(I'm using "TGpHugeFileStream" component from Primoz Gabrijelcic instead of a regular TFileStream. Shouldn't make any difference).

Somehow, my dynamic array data is getting trashed when I try to increase the size of it with the SetLength(OutFilesStream, numfields+1); statement. Stepping through it in debug mode revealed this:

- after the declaration and initial Setlength of the array to 0 (not shown), the array contains simply: ()
- after reading the first data record with 8 fields (numfields=8) and doing a Setlength(numfields+1), the array looked like: nil,nil,nil,nil,nil,nil,nil,nil,nil
- after the file "create" for OutFileStream[8], the array looked like : nil,nil,nil,nil,nil,nil,nil,nil,$1297078
- okay, so far so good, but when the next record is read... one with 9 fields this time, the trouble happens. Upon doing the Setlength(numfields+1), this is what it did to the OutFileStream array: nil,(),(),(),(),(),(),(),(),nil

  ...It lost my file handle info for element [8]. Any thoughts anyone?

Thanks!
   Shawn
1
Comment
Question by:shawn857
  • 2
  • 2
5 Comments
 
LVL 36

Expert Comment

by:ste5an
ID: 40264861
I think I know what's happening. But the problem is: Your code snippet is incomplete. It does not show use the important part... Always post a complete and concise example.

I guess: It's a scope problem. The OutFilesStream array is declared inside the loop (you said next record).

btw, you're misusing the try-except construct to control your program flow. This is a very bad practice. Test whether you need to resize your array in the normal control flow.

If numfields + 1 > Length(OutFilesStream) then  
begin
	SetLength(OutFilesStream, numfields+1);
	OutFilesStream[numfields]:=TGpHugeFileStream.Create(BuildFileName, accWrite);
end;

SetLength(HBuffer, Length(holdrec)+2);
StrPCopy(PChar(HBuffer), holdrec+#13#10);
bytes_Wrote := OutFilesStream[numfields].Write(HBuffer[0], StrLen(PChar(HBuffer)));

Open in new window


Also the generic exception handling may mask the problem, thus don't use it all.

And depending on your Delphi version use Ctrl+D to get a uniform formatted code.
0
 

Author Comment

by:shawn857
ID: 40265320
Thanks Ste5an... the reason I structured it with the TRY..EXCEPT was for speed. Before that, I explicitly checked to see if the proper output file existed before attempting to write to it. While profiling the code, I noticed that this check was a big bottleneck... and found the try..except technique while googling. I agree that it's not the most elegant.
   Thanks for your suggested code, but I'm afraid it won't work. You see, what about the situation where the length of the dynamic array is say, 9 and then along comes a record with say, 4 fields that I have not yet encountered. Your code in this case, would not do the Filestream.create for OutFilesStream[4), and thus fail on the following Write statement to that particular file.
   Regarding the scope issue you touched on - I think I should be okay there as I declare the OutFilesStream dynamic array globally, and do the "SetLength(OutFilesStream, 0)" statement just a little bit earlier in the code segment I originally included.

Thanks!
   Shawn
0
 
LVL 36

Expert Comment

by:ste5an
ID: 40265585
Well, it was just a guess. Without a complete and concise example...
0
 
LVL 38

Accepted Solution

by:
Geert Gruwez earned 2000 total points
ID: 40265869
dynamic array ?
a stream per field of a record ?

so you want to write all colums of the record in parallel through a single network connection ?
i'd write serially ... but hey ... why not do it different ?

why not use a TList for your stream collection ?
i'd create the streams in the getter of the list if it doesn't exist for that index

you might want to override the add method of TList too

type
  TStreamClass = class of TStream;

  TStreamList = class(TList)
  private
    fStreamClass: TStreamClass;
    function GetStream(Index: Integer): TStream; 

  public
    property Stream[Index: Integer]: TStream read GetStream;
  end;


function TStreamList.GetStream(Index: Integer): TStream; 
begin
  while Count-1 < Index do 
    Add(fStreamClass.Create);
  Result := TStream(inherited Items[Index]);
end;

Open in new window


way simpler than a dynamic array.
the cleanup needs a little bit attention and grow and shrinking is already part of Tlist
0
 
LVL 38

Expert Comment

by:Geert Gruwez
ID: 40265870
ugh ... global dynamic array's with primoz otl library ?
i hope you thought of some protection within the multi threading ?
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article explains how to create forms/units independent of other forms/units object names in a delphi project. Have you ever created a form for user input in a Delphi project and then had the need to have that same form in a other Delphi proj…
In my programming career I have only very rarely run into situations where operator overloading would be of any use in my work.  Normally those situations involved math with either overly large numbers (hundreds of thousands of digits or accuracy re…
this video summaries big data hadoop online training demo (http://onlineitguru.com/big-data-hadoop-online-training-placement.html) , and covers basics in big data hadoop .
Loops Section Overview
Suggested Courses

864 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question