Solved

# ultra huge array of Bits (Boolean)

Posted on 2006-11-28
Medium Priority
349 Views
i Want to create a bit array , size [1...1E6, 1...1E6] or even a little bit bigger.  .-))
I used a TList Object and a Bool-Pointer
max. array size is only 1E5x1E5 . Any Hint thow o blow up my record size?
the function TBitplane.Init_BitArray  should run with stepx=1E6.. and function stepy=1E6...

------------------
some trial code
------------------

type
TPointBoolZ = record
// x  : Integer;    {  x index = 0...n, removed, changed to lin. Address }
// y  : Integer;    {  y index = 0...m, removed, changed to lin. Address }
Z  : Boolean;
end;

TPointBoolZPtr =^TPointBoolZ;

{ 2D Array of Bool to calc the area of any arbitray gemomatical layouts }
TBitPlane = Class

BitArray  :  TList;

constructor  Create;

private
Upp   :   FPoint;
Low   :   FPoint;

deltax  :  Real;
deltay  :  Real;

stepX :  Integer;
stepY :  Integer;

{ Function for discrete geometric (DG) problems }

public

function Init_BitArray : Boolean;

function AddTRect(MyRect :  TRect) : Boolean;
function AddTFRect(MyFRect : TFRect) : Boolean;

function AddTCircle(MyRect :  TRect) : Boolean;
function AddFCircle(MyFRect : TFRect) : Boolean;

function GetTIndex (MyPoint :FPoint): TPoint;
function GetFPoint (MyPoint :TPoint): FPoint;

function VirtualPoint(I: Integer): TPoint;

function AssignSize(UpperCorner, LowerCorner :  FPoint; Sx, Sy :  Integer) :  Boolean;

function SetBitArray (x,y : Integer; Value : Boolean) : Boolean;

function GetBitArea: real;

function GetFalseCount :  Integer;
function GetBitArray(x, y: Integer): Boolean;
end;

{ Calculate from the 1D Array of Bool to  2d Array  A[X;Y]  }
function TBitplane.VirtualPoint(I : Integer):  TPoint;
var     V   :   TPoint;
x,y :   Word;
begin
// procedure DivMod(Dividend: Integer; Divisor: Word; var Result, Remainder: Word);
DivMod(i, stepx,y,x);

V.x :=x;
V.y :=y;

result := v;

end;

function TBitplane.Init_BitArray : Boolean;
var    x, y    :  Integer;

SinglePoint : TPointBoolZPtr;
begin

BitArray.Clear;

try
begin
for y:=0 to stepy do
for x := 0 to stepx do
begin

new(SinglePoint);

//  SinglePoint^.x := x;    {  version 1 only }

//   SinglePoint^.y := y;    {  version 1 only }

SinglePoint^.Z := false;

end;
result := true;
end;
except
result := false;
end;
end;
0
Question by:BdLm
[X]
###### Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

• Help others & share knowledge
• Earn cash & points

LVL 21

Assisted Solution

ziolko earned 80 total points
ID: 18026679
if You want that huge ammount of items stored give up with arrays or TList
make Your own data structure which will be limited only by system resources... something like this

unit Unit1;

interface

uses
Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms,
Dialogs, StdCtrls;

type

TItem = class(TObject)
private
FNextItem: TItem;
FYourData: Pointer;
public
property NextItem: TItem read FNextItem write FNextItem;
end;

TForm1 = class(TForm)
Button1: TButton;
procedure Button1Click(Sender: TObject);
procedure FormCreate(Sender: TObject);
procedure FormDestroy(Sender: TObject);
private
FMyDataStructure: TItem;
public
{ Public declarations }
end;

var
Form1: TForm1;

implementation

{\$R *.dfm}

{ TItem }

begin
inherited Create;
FNextItem := nil;
end;

procedure TForm1.Button1Click(Sender: TObject);
var nIt: TItem;
begin
nIt := TItem.Create(nil);
nIt.NextItem := FMyDataStructure;
FMyDataStructure := nIt;
end;

procedure TForm1.FormCreate(Sender: TObject);
begin
FMyDataStructure := TItem.Create(nil);
end;

procedure TForm1.FormDestroy(Sender: TObject);
var it: TItem;
begin
repeat
it := FMyDataStructure;
FMyDataStructure := FMyDataStructure.NextItem;
FreeAndNil(it);
until FMyDataStructure = nil;
end;

end.

ziolko
0

LVL 26

Accepted Solution

Russell Libby earned 80 total points
ID: 18028217

Lol... you do realize what you are asking, right?

1E6 (1 million) squared is 1E12, or 1 trillion bits. Even given the fact that you could compress this by 8 (8 bits / byte) and use 64 bit integer math to locate the desired byte at X,Y (and mod to determine the bit), eg:

1M rows / Each row at 125K (1M / 8) / Each byte of the row comprises 8 values

and you would still be faced with storing 125 GB of data to make up your array. Now also consider that the max address space of any user mode windows process is technically 2GB (possible to achieve up to 4gb), and you quickly see that you can only address 1.6% of the actual array in memory. This would mean using a 125 GB disk file as the actual storage backing for the array. So while theortically achievable, its not going to be fast, and you need a pretty beefy system with plenty of storage to handle this. And even on a fast disk, you would probably be looking @ roughly 1-2 hours to perform an Init(...) on your virtual array.

As far as what ziolko is recommending, it just does not apply here, as your process would hit the 2GB address limit before it even finished 0.2% of the array.

---

Russell
0

LVL 19

Expert Comment

ID: 18028557
I agree with rlibby.

So the big question is what you need it for, maybe there is a better solution for your problem.
0

LVL 26

Expert Comment

ID: 18029122
**IF** you are dealing with a very sparse array/matrix, then there may be some hope. For the education of those not familiar with what a sparse array/matrix is:

An example data structure for the handling might look like:

type
PSparseCol     =  ^TSparseCol;
TSparseCol     =  packed record
Col:        Integer;
Next:       PSparseCol;
end;

var
RowArray:      Array [1..1000000] of PSparseCol;

The row array is 4MB in size, and contains a singly linked list to each "column" that is in an "on" state (on/set/true/whatever you want to call it). This would allow for a theoretical maximum of 267 million bits being on at once. And by very sparse, I mean VERY sparse, as you would still only be able to address less than 1 percent of the virtual array size.

Russell
0

LVL 4

Assisted Solution

tobjectpascal earned 80 total points
ID: 18034757
What i find interesting about Boolean is, it's not a Bit, it's actually a Byte / 8 bit register.

Var
B: Boolean;
begin
Asm
mov al,10;
mov b,al;
End;
If Al=True Then
ShowMessage('True');

>0 true...

0

LVL 8

Author Comment

ID: 18035262
many thanks to all of you for the good inputs.

1)  the background of the question:  I got tons of images (~  xx  GByte) building up a map, I have to sum up cerain araes with in  these images. Step and repeat with my algo through the whole image would be one idea, but to keep the whole image in one array makes the code quite easy.

2)  a sparse matrix approach won't help completly, Im thinking about something like a compressed array , as 0 and 1 's are balanced and once I got a '1'  the will always be a certain sequence of '1'. This would also speed my area sum algorithm.

0

## Featured Post

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

A lot of questions regard threads in Delphi. Â  One of the more specific questions is how to show progress of the thread. Â  Updating a progressbar from inside a thread is a mistake. A solution to this would be to send a synchronized message to theâ€¦
Creating an auto free TStringList The TStringList is a basic and frequently used object in Delphi. On many occasions, you may want to create a temporary list, process some items in the list and be done with the list. In such cases, you have toâ€¦
Add bar graphs to Access queries using Unicode block characters. Graphs appear on every record in the color you want. Give life to numbers. Hopes this gives you ideas on visualizing your data in new ways ~ Create a calculated field in a query: â€¦
Visualize your data even better in Access queries. Given a date and a value, this lesson shows how to compare that value with the previous value, calculate the difference, and display a circle if the value is the same, an up triangle if it increasedâ€¦
###### Suggested Courses
Course of the Month10 days, 10 hours left to enroll