Solved

ultra huge array of Bits (Boolean)

Posted on 2006-11-28
6
342 Views
Last Modified: 2010-04-05
i Want to create a bit array , size [1...1E6, 1...1E6] or even a little bit bigger.  .-))  
I used a TList Object and a Bool-Pointer
max. array size is only 1E5x1E5 . Any Hint thow o blow up my record size?
the function TBitplane.Init_BitArray  should run with stepx=1E6.. and function stepy=1E6...

------------------
some trial code
------------------

type
       TPointBoolZ = record
               // x  : Integer;    {  x index = 0...n, removed, changed to lin. Address }
               // y  : Integer;    {  y index = 0...m, removed, changed to lin. Address }
               Z  : Boolean;
               end;

     TPointBoolZPtr =^TPointBoolZ;



 { 2D Array of Bool to calc the area of any arbitray gemomatical layouts }
     TBitPlane = Class

           BitArray  :  TList;

           constructor  Create;

           private
           Upp   :   FPoint;
           Low   :   FPoint;

           deltax  :  Real;
           deltay  :  Real;

           stepX :  Integer;
           stepY :  Integer;


           { Function for discrete geometric (DG) problems }

           public




           function Init_BitArray : Boolean;

           function AddTRect(MyRect :  TRect) : Boolean;
           function AddTFRect(MyFRect : TFRect) : Boolean;

           function AddTCircle(MyRect :  TRect) : Boolean;
           function AddFCircle(MyFRect : TFRect) : Boolean;

           function AddTThickline(MyLine: TThickline): Boolean;

           function GetTIndex (MyPoint :FPoint): TPoint;
           function GetFPoint (MyPoint :TPoint): FPoint;

           function VirtualPoint(I: Integer): TPoint;

           function AssignSize(UpperCorner, LowerCorner :  FPoint; Sx, Sy :  Integer) :  Boolean;

           function SetBitArray (x,y : Integer; Value : Boolean) : Boolean;

           function GetBitArea: real;

           function GetFalseCount :  Integer;
           function GetBitArray(x, y: Integer): Boolean;
           end;








{ Calculate from the 1D Array of Bool to  2d Array  A[X;Y]  }
function TBitplane.VirtualPoint(I : Integer):  TPoint;
var     V   :   TPoint;
        x,y :   Word;
begin
      // procedure DivMod(Dividend: Integer; Divisor: Word; var Result, Remainder: Word);
      DivMod(i, stepx,y,x);

      V.x :=x;
      V.y :=y;

      result := v;

end;


function TBitplane.Init_BitArray : Boolean;
var    x, y    :  Integer;

       SinglePoint : TPointBoolZPtr;
begin

      BitArray.Clear;

     try
      begin
      for y:=0 to stepy do
        for x := 0 to stepx do
        begin

          new(SinglePoint);

          //  SinglePoint^.x := x;    {  version 1 only }

         //   SinglePoint^.y := y;    {  version 1 only }

          SinglePoint^.Z := false;

          BitArray.Add(SinglePoint);
        end;
       result := true;
       end;
       except
          result := false;
       end;
end;
0
Comment
Question by:BdLm
6 Comments
 
LVL 21

Assisted Solution

by:ziolko
ziolko earned 20 total points
ID: 18026679
if You want that huge ammount of items stored give up with arrays or TList
make Your own data structure which will be limited only by system resources... something like this

unit Unit1;

interface

uses
  Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms,
  Dialogs, StdCtrls;

type

  TItem = class(TObject)
  private
    FNextItem: TItem;
    FYourData: Pointer;
  public
    constructor Create(AData: Pointer);
    property NextItem: TItem read FNextItem write FNextItem;
  end;

  TForm1 = class(TForm)
    Button1: TButton;
    procedure Button1Click(Sender: TObject);
    procedure FormCreate(Sender: TObject);
    procedure FormDestroy(Sender: TObject);
  private
    FMyDataStructure: TItem;
  public
    { Public declarations }
  end;

var
  Form1: TForm1;

implementation

{$R *.dfm}

{ TItem }

constructor TItem.Create(AData: Pointer);
begin
  inherited Create;
  FYourData := AData;
  FNextItem := nil;
end;

procedure TForm1.Button1Click(Sender: TObject);
var nIt: TItem;
begin
  nIt := TItem.Create(nil);
  nIt.NextItem := FMyDataStructure;
  FMyDataStructure := nIt;
end;

procedure TForm1.FormCreate(Sender: TObject);
begin
  FMyDataStructure := TItem.Create(nil);  
end;

procedure TForm1.FormDestroy(Sender: TObject);
var it: TItem;
begin
  repeat
    it := FMyDataStructure;
    FMyDataStructure := FMyDataStructure.NextItem;
    FreeAndNil(it);
  until FMyDataStructure = nil;
end;

end.

ziolko
0
 
LVL 26

Accepted Solution

by:
Russell Libby earned 20 total points
ID: 18028217

Lol... you do realize what you are asking, right?

1E6 (1 million) squared is 1E12, or 1 trillion bits. Even given the fact that you could compress this by 8 (8 bits / byte) and use 64 bit integer math to locate the desired byte at X,Y (and mod to determine the bit), eg:

1M rows / Each row at 125K (1M / 8) / Each byte of the row comprises 8 values

and you would still be faced with storing 125 GB of data to make up your array. Now also consider that the max address space of any user mode windows process is technically 2GB (possible to achieve up to 4gb), and you quickly see that you can only address 1.6% of the actual array in memory. This would mean using a 125 GB disk file as the actual storage backing for the array. So while theortically achievable, its not going to be fast, and you need a pretty beefy system with plenty of storage to handle this. And even on a fast disk, you would probably be looking @ roughly 1-2 hours to perform an Init(...) on your virtual array.

As far as what ziolko is recommending, it just does not apply here, as your process would hit the 2GB address limit before it even finished 0.2% of the array.

---

Russell
0
 
LVL 19

Expert Comment

by:MerijnB
ID: 18028557
I agree with rlibby.

So the big question is what you need it for, maybe there is a better solution for your problem.
0
Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

 
LVL 26

Expert Comment

by:Russell Libby
ID: 18029122
**IF** you are dealing with a very sparse array/matrix, then there may be some hope. For the education of those not familiar with what a sparse array/matrix is:

http://www.answers.com/topic/sparse-matrix

An example data structure for the handling might look like:

type
  PSparseCol     =  ^TSparseCol;
  TSparseCol     =  packed record
     Col:        Integer;
     Next:       PSparseCol;
  end;

var
  RowArray:      Array [1..1000000] of PSparseCol;

The row array is 4MB in size, and contains a singly linked list to each "column" that is in an "on" state (on/set/true/whatever you want to call it). This would allow for a theoretical maximum of 267 million bits being on at once. And by very sparse, I mean VERY sparse, as you would still only be able to address less than 1 percent of the virtual array size.

Russell
0
 
LVL 4

Assisted Solution

by:tobjectpascal
tobjectpascal earned 20 total points
ID: 18034757
What i find interesting about Boolean is, it's not a Bit, it's actually a Byte / 8 bit register.


Var
 B: Boolean;
begin
 Asm
  mov al,10;
  mov b,al;
 End;
  If Al=True Then
    ShowMessage('True');


>0 true...

0
 
LVL 8

Author Comment

by:BdLm
ID: 18035262
many thanks to all of you for the good inputs.

1)  the background of the question:  I got tons of images (~  xx  GByte) building up a map, I have to sum up cerain araes with in  these images. Step and repeat with my algo through the whole image would be one idea, but to keep the whole image in one array makes the code quite easy.


2)  a sparse matrix approach won't help completly, Im thinking about something like a compressed array , as 0 and 1 's are balanced and once I got a '1'  the will always be a certain sequence of '1'. This would also speed my area sum algorithm.


0

Featured Post

What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

Join & Write a Comment

This article explains how to create forms/units independent of other forms/units object names in a delphi project. Have you ever created a form for user input in a Delphi project and then had the need to have that same form in a other Delphi proj…
In my programming career I have only very rarely run into situations where operator overloading would be of any use in my work.  Normally those situations involved math with either overly large numbers (hundreds of thousands of digits or accuracy re…
Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…
This tutorial demonstrates a quick way of adding group price to multiple Magento products.

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now