Dynamic array data gets trashed when I increase size with Setlength

Posted on 2014-08-15
1 Ratings
Last Modified: 2014-08-21
Hello Experts... I have a dynamic array of FileStream's that can get bigger only... never smaller. Reason is I need to have (possibly) many filestreams open at once as I need to write a record to the appropriate stream based on the number of fields the record has. For instance, if I read a record and it has 7 fields, I need to write it to FileStream[7]... if the next record I read has 22 fields and my Filestream array is only 0..7 in length, I then need to do a Setlength on my FileStream array to increase the length to 23 Filestream elements (ie. 0..22). My code basically looks like this:

             // check if an output file already has been opened for this particular fieldcount
                BuildFileName := FNnoext+'-'+IntToStr(numfields);

                  SetLength(HBuffer, Length(holdrec)+2);
                  StrPCopy(PChar(HBuffer), holdrec+#13#10);
                  bytes_Wrote:=OutFilesStream[numfields].Write(HBuffer[0], StrLen(PChar(HBuffer)));
                  If numfields+1 > Length(OutFilesStream) then  // This file doesn't yet exist, increase the size of our dynamic files array
                     SetLength(OutFilesStream, numfields+1);
                  OutFilesStream[numfields]:=TGpHugeFileStream.Create(BuildFileName, accWrite);
                  bytes_Wrote:=OutFilesStream[numfields].Write(HBuffer[0], StrLen(PChar(HBuffer)));

Open in new window

(I'm using "TGpHugeFileStream" component from Primoz Gabrijelcic instead of a regular TFileStream. Shouldn't make any difference).

Somehow, my dynamic array data is getting trashed when I try to increase the size of it with the SetLength(OutFilesStream, numfields+1); statement. Stepping through it in debug mode revealed this:

- after the declaration and initial Setlength of the array to 0 (not shown), the array contains simply: ()
- after reading the first data record with 8 fields (numfields=8) and doing a Setlength(numfields+1), the array looked like: nil,nil,nil,nil,nil,nil,nil,nil,nil
- after the file "create" for OutFileStream[8], the array looked like : nil,nil,nil,nil,nil,nil,nil,nil,$1297078
- okay, so far so good, but when the next record is read... one with 9 fields this time, the trouble happens. Upon doing the Setlength(numfields+1), this is what it did to the OutFileStream array: nil,(),(),(),(),(),(),(),(),nil

  ...It lost my file handle info for element [8]. Any thoughts anyone?

Question by:shawn857
    LVL 31

    Expert Comment

    I think I know what's happening. But the problem is: Your code snippet is incomplete. It does not show use the important part... Always post a complete and concise example.

    I guess: It's a scope problem. The OutFilesStream array is declared inside the loop (you said next record).

    btw, you're misusing the try-except construct to control your program flow. This is a very bad practice. Test whether you need to resize your array in the normal control flow.

    If numfields + 1 > Length(OutFilesStream) then  
    	SetLength(OutFilesStream, numfields+1);
    	OutFilesStream[numfields]:=TGpHugeFileStream.Create(BuildFileName, accWrite);
    SetLength(HBuffer, Length(holdrec)+2);
    StrPCopy(PChar(HBuffer), holdrec+#13#10);
    bytes_Wrote := OutFilesStream[numfields].Write(HBuffer[0], StrLen(PChar(HBuffer)));

    Open in new window

    Also the generic exception handling may mask the problem, thus don't use it all.

    And depending on your Delphi version use Ctrl+D to get a uniform formatted code.

    Author Comment

    Thanks Ste5an... the reason I structured it with the TRY..EXCEPT was for speed. Before that, I explicitly checked to see if the proper output file existed before attempting to write to it. While profiling the code, I noticed that this check was a big bottleneck... and found the try..except technique while googling. I agree that it's not the most elegant.
       Thanks for your suggested code, but I'm afraid it won't work. You see, what about the situation where the length of the dynamic array is say, 9 and then along comes a record with say, 4 fields that I have not yet encountered. Your code in this case, would not do the Filestream.create for OutFilesStream[4), and thus fail on the following Write statement to that particular file.
       Regarding the scope issue you touched on - I think I should be okay there as I declare the OutFilesStream dynamic array globally, and do the "SetLength(OutFilesStream, 0)" statement just a little bit earlier in the code segment I originally included.

    LVL 31

    Expert Comment

    Well, it was just a guess. Without a complete and concise example...
    LVL 36

    Accepted Solution

    dynamic array ?
    a stream per field of a record ?

    so you want to write all colums of the record in parallel through a single network connection ?
    i'd write serially ... but hey ... why not do it different ?

    why not use a TList for your stream collection ?
    i'd create the streams in the getter of the list if it doesn't exist for that index

    you might want to override the add method of TList too

      TStreamClass = class of TStream;
      TStreamList = class(TList)
        fStreamClass: TStreamClass;
        function GetStream(Index: Integer): TStream; 
        property Stream[Index: Integer]: TStream read GetStream;
    function TStreamList.GetStream(Index: Integer): TStream; 
      while Count-1 < Index do 
      Result := TStream(inherited Items[Index]);

    Open in new window

    way simpler than a dynamic array.
    the cleanup needs a little bit attention and grow and shrinking is already part of Tlist
    LVL 36

    Expert Comment

    by:Geert Gruwez
    ugh ... global dynamic array's with primoz otl library ?
    i hope you thought of some protection within the multi threading ?

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    How to improve team productivity

    Quip adds documents, spreadsheets, and tasklists to your Slack experience
    - Elevate ideas to Quip docs
    - Share Quip docs in Slack
    - Get notified of changes to your docs
    - Available on iOS/Android/Desktop/Web
    - Online/Offline

    Objective: - This article will help user in how to convert their numeric value become words. How to use 1. You can copy this code in your Unit as function 2. than you can perform your function by type this code The Code   (CODE) The Im…
    Introduction The parallel port is a very commonly known port, it was widely used to connect a printer to the PC, if you look at the back of your computer, for those who don't have newer computers, there will be a port with 25 pins and a small print…
    Need more eyes on your posted question? Go ahead and follow the quick steps in this video to learn how to Request Attention to your question. *Log into your Experts Exchange account *Find the question you want to Request Attention for *Go to the e…
    Illustrator's Shape Builder tool will let you combine shapes visually and interactively. This video shows the Mac version, but the tool works the same way in Windows. To follow along with this video, you can draw your own shapes or download the file…

    758 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    13 Experts available now in Live!

    Get 1:1 Help Now