Solved

[Challenge] File-based Queue...

Posted on 2006-11-21
12
515 Views
Last Modified: 2010-04-05
I'm working on a simple solution where I need to put items in a queue and retrieve them from it again. But this list needs to be persistent so the application will continue where it left off the next time I start it. Also, this queue can become huge with over half a million items or more so my solution is to use a file for storage.
The items I will store is just a single string. The contents of it will be XML containing the exact data of what I wanted to add to the stack. The data itself is not really important but right now my first approach is the following:
The first 8 bytes (INT64) of the file will give me the address of the first item in the list.
When pushing a new item to the queue, I just write 4 bytes with the string length, followed by the string itself.
When popping an item from the queue, I use the first 8 bytes to retrieve the starting point, read the length of the string and the string itself and increase the starting points with 4+length so it will point to the next record.

Drawback... I now have a history of items in my file and I would like this to be clean...

The current solution is simple... Once in a while I will just open the file, move the data closer to the beginning and truncate it past the last item. Not very complex but it's just something you don't want to do very often with a file of half a gigabyte in size or more. This packing of the "database" isn't really fast but it works. And in general I just pack it about every 512 pops or so.

So this makes me wonder about possible faster alternatives. Are there any? (Btw, this is already bloody fast...) So here is a new challenge. Make it work faster! :-)

Starting point is this:

type
  TStack = class
  public
    procedure Push(Const Data: string);
    function Pop: string;    
  end;

Empty strings are NOT pushed to the file thus if there are no items in the queue then pop will just return an empty string indicating the end of the queue.
And no, am not providing the rest of my code. :-) It's a challenge, remember? I just described the solution.

(Btw, you could also turn this queue into a stack by writing the length TWICE to the file. Once before the string and once at the end of it. The Push function would be the same but popping items would have to read the last 4 bytes of the file to determine string length and then go back this length in the file to read the string. Then go back 4 + length + 4 bytes and truncate the file at that position. But for now I'm only interested in the Queue solution...)
0
Comment
Question by:Wim ten Brink
  • 5
  • 4
  • 2
  • +1
12 Comments
 
LVL 26

Expert Comment

by:Russell Libby
Comment Utility

It would be much more efficient to track the deleted (pop'ed) location and available size at the header of the file:

eg:
Header = [8 Byte first record location][8 Byte free location][4 byte free size][Data....]

When an item is popped, the delete location is updated: If already set, the free size is incremented, if not, the location and size is stored. This allows you to use this space to backfill the file in a round robin fashion. When pushing an item, the free location / size is checked. If enough space is available, the new pushed record backfills this position and the free location / size is updated. If not, the pushed record goes to the end of the file. This keeps the wasted space down to a minimum without rewriting / moving large amounts of data.
 
Let me know if you need to see a working example.
Russell

0
 
LVL 17

Author Comment

by:Wim ten Brink
Comment Utility
Well, this is a challenge so a working example would be a requirement. :-)

Of course, you could fill in the gaps in the file with the newer items but the problem is that it still needs to work as a queue, meaning that the first record going in is the first record going out, etc. The order is important here. If you're going to re-use the gaps then you need to add information about where the next record is located to keep them in a proper order. Furthermore, you would risk ending up with a file made out of Swiss cheese: full with tiny holes. ;-)

Reminds me on how Turbo Pascal used to manage it's memory. It would round all memory allocations upwards to a multiple of 8 bytes. Thus, when a part of memory was released again, the memory manager would just fill it with information about the size of this hole in-memory and the location of the next hole. That way the memory manager kept track of it's free space in the old Turbo Pascal times. Of course, this too could lead to situations where your memory was full with holes. Even worse, the memory manager might indicate that you still had 100 KB of memory available but it would be divided over small holes, none of them bigger than, say, 128 bytes and thus trying to allocate anything over 128 bytes would result in an out-of-memory situation...

I don't mind if the solution is a bit wasteful with resources as long as they're released once in a while. Speed is more important here. With a stack, this would not be a problem since after popping an item I would just truncate the file which would remove the last record in the file. But in this case I have to deal with removing the first records and that makes it more challenging. :-)
0
 
LVL 19

Expert Comment

by:MerijnB
Comment Utility
as you already indicate what matters here is speed and the order of the items. I think any other solution will compromise in either one or both of them. RLibby's option sounds interesting, but you only gain on not (or less often) having to pack, with cons of loosing speed and (as you mentioned) fragmentation.
0
 
LVL 17

Author Comment

by:Wim ten Brink
Comment Utility
True. Biggest problem in this exercise is trying to keep things as balanced as possible. Speed is the most important thing here but a solution that eats up large amounts of diskspace to just fill it up with items that can already be deleted is just a big waste of diskspace. Therefore, fragmentation wouldn't be that bad as long as there is an option to keep track of the empty spaces and a way to clean it up again.

The situation that I'm currently using this is the following. (This might also explain why speed is so important!) I have an out-of process COM executable and it's only purpose is to maintain this queue. All calls to the queue are synchronised so there are no two threads working on the stack at the same time. Just saves a lot of possible threading issues too.
I have several different processes that are accessing this Queue COM object. Many of them are pushing data to the queue but do this at a reasonable slow speed each. One process is popping data from the queue and is processing this as fast as it can. The Push-processes are reasonable short-lived while the Pop-process will continue to exist until the stack is empty. It needs to be reactivated again after some time to start processing again. And it must be able to process all data while other processes are pushing data. (And worse, since it is a COM object, it is possible that a process running on a different system is also pushing data to the same queue!)

Because the file access is done in a single thread, a minor problem arises. Basically a speed issue, which is why I'm looking for a faster alternative. I have plenty of disk space to waste anyways but an alternate solution which wastes a lot less diskspace is more than welcome. :-)
0
 
LVL 19

Expert Comment

by:MerijnB
Comment Utility
it sounds to me the speed bottleneck is the disk on which the data is written. If that is the case it wouldn't help if you'd had more threads. It seems to me if you really want to get (a lot) faster you will first have to get rid of that bottleneck and thus keep all the data in memory. Once you are there you can think of other ways to speed it (multiple threads, whatever).
0
 
LVL 26

Expert Comment

by:Russell Libby
Comment Utility

Agreed, speed is directly affected by disk IO, and is the major bottleneck here... so how to solve becomes the question.

The problem with queues is that they tend to be hard to do well. You normally end up with a solution that is either very fast at inserts, and slow on deletes, or vice versa (due to fragmentation). DB's try to resolve the fragmentation by allocating blocks (pages), but this also inccurs a performance hit when consolidating data on the pages, and is complex due to the page management involved. Your initial solution is very fast at new inserts, but can be very slow if you have to move a gb of data in the file to reclaim free space. It will also most likely suffer from disk fragmentation as data is inserted. Truncating the file (reclamation of space) partially addresses this, until you push more data in which case the new data will most likely end up physically further away than before. Looking at my first post, it can suffer from a great deal of fragmentation as well.

A possible alternative, providing that you can live with a fixed file size, would be to allocate one single large file (eg 1, 2, 4 gb etc) that would be the maximum size of the queue. (This can be done extremely quickly using setfilepointer / setendoffile, 16ms for 2GB file on my system). This would allow a round-robin queue to be implemented on the file, with no fragmentation of data. If the tail location is < head location, and the tail location + size of new data > head position, then the queue is considered "full".

Drawbacks:
- Allocates a large file, most of which may be unused depending on "pushed" data
- Does not provide for expansion on a "full" scenario.

Advantages:
- For a clean disk, the file allocated would be contiguous.
- No fragmentation of data
- Linear insert time
- Linear delete time.
- No chance of running into an "out of disk space" condition after the initial allocation.
- Fairly trivial to implement, especially compared to paging schemes.

I am not suggesting that this is perfect; but it is fast, is simple to implement, and does not suffer from fragmentation of data. Any other disk based solution will either be fast and fragmented, or will introduce non-linear times for inserts/deletes in order to handle the fragmentation.

--------

Regards,
Russell
0
Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

 
LVL 4

Expert Comment

by:tobjectpascal
Comment Utility
I don't believe IO is all that much of a bottleneck... I wrote a prime numbers program, the task was to determine if a given number was a prime number and then depending on params passed, it stored it to disk.., on 1gig CPU it managed to save 1 million primes includuing writing to the disk in less than 1/2 second. 100ms to write a million numbers to disk.

If it's a simple case of pushing the data, then seeking to the end of the file and seeking back 8 bytes and updating the index of where the last value is, should not be slow in the slightest.

0
 
LVL 26

Expert Comment

by:Russell Libby
Comment Utility

The problem is implementing a queue on a file, and then truncating the file at some point (after items are removed) so the file does not keep growing indefinitely. Because the data is removed from the start of the file, the data needs to be shifted before it can be truncated. THIS can be a bottleneck when your trying to keep an O(N) constant performance time.

Russell


0
 
LVL 26

Accepted Solution

by:
Russell Libby earned 500 total points
Comment Utility

Here is a first cut at my suggestion recommended above. I say first cut, because it works correctly, but has not been tweaked to the max yet (though it may be close to start with). By creating a fixed size queue, regardless of total size (4, 6, 8 gb), I was able to achieve close to 100K (thousand)  pushes **AND** pops / second writing / reading variable sized data (Random(1000) bytes). This is pretty much as close as you can get to to O(N) performance time dealing with disk I/O, albeit at the sacrificice of specifying a "fixed" queue storage size.

Russell


unit PersistQueue;
////////////////////////////////////////////////////////////////////////////////
//
//   Unit        :  PersistQueue
//   Author      :  rllibby
//   Date        :  11.21.2006
//   Description :  File based queue implementation using a round-robin scheme
//                  based on a fixed file size. When data is popped, nothing is
//                  removed from the file and only the head location is updated.
//                  This keeps an O(N) constant time for Pop(). When data is pushed,
//                  no allocation is done; the data is written and the tail location
//                  is updated, again, keeping an O(N) constant time. The results
//                  of this on an SATA 150 7500 WD system were appx 90K pushes AND
//                  pops / second using a random data size of appx 1K.
//
////////////////////////////////////////////////////////////////////////////////
interface

////////////////////////////////////////////////////////////////////////////////
//   Include units
////////////////////////////////////////////////////////////////////////////////
uses
  Windows, SysUtils;

////////////////////////////////////////////////////////////////////////////////
//   Constants
////////////////////////////////////////////////////////////////////////////////
const
  PQ_MINSIZE        = 1024;
  PQ_FILEMAGIC:     Word  =  20816;

////////////////////////////////////////////////////////////////////////////////
//   Data types
////////////////////////////////////////////////////////////////////////////////
type
  TPQHeader         =  packed record
     Magic:         Word;
     Head:          Int64;
     Tail:          Int64;
     Size:          Int64;
  end;

////////////////////////////////////////////////////////////////////////////////
//   TPersistQueue
////////////////////////////////////////////////////////////////////////////////
type
  TPersistQueue     =  class(TObject)
  private
     // Private declatations
     FFileName:     String;
     FWriter:       THandle;
     FReader:       THandle;
     FHeader:       TPQHeader;
     procedure      CheckActive;
     procedure      FlushAndClose;
  protected
     // Protected declarations
     procedure      SetLocation(Handle: THandle; Position: Int64);
     function       GetActive: Boolean;
     function       GetFreeSpace: Int64;
  public
     // Public declarations
     constructor    Create;
     destructor     Destroy; override;
     function       CanPush(const Data: String): Boolean;
     function       CanPop: Boolean;
     procedure      CreateQueue(FileName: String; Size: Int64);
     procedure      OpenQueue(FileName: String);
     function       Pop: String;
     procedure      Push(const Data: String);
     property       Active: Boolean read GetActive;
     property       FreeSpace: Int64 read GetFreeSpace;
  end;

implementation

function TPersistQueue.CanPush(const Data: String): Boolean;
var  dwSource:      DWORD;
begin

  // Check handles
  if GetActive then
  begin
     // Get length of source
     dwSource:=Length(Data);
     // Don't allow writing an empty string
     if (dwSource = 0) then
        // Success
        result:=True
     else
        // Check free space against the data we must write
        result:=not((SizeOf(DWORD) + dwSource) > GetFreeSpace);
  end
  else
     // Queue not active
     result:=False;

end;

function TPersistQueue.CanPop: Boolean;
begin

  // Check handles
  if GetActive then
     // Can pop if head <> tail
     result:=not(FHeader.Head = FHeader.Tail)
  else
     // Queue not active
     result:=False;

end;

procedure TPersistQueue.Push(const Data: String);
var  lpBuffer:      PChar;
     dwSource:      DWORD;
     dwWrap:        DWORD;
     dwWrite:       DWORD;
begin

  // Get length of source
  dwSource:=Length(Data);

  // Don't allow writing an empty string
  if (dwSource > 0) then
  begin
     // Check active
     CheckActive;
     // Check free space against the data we must write
     if ((SizeOf(DWORD) + dwSource) > GetFreeSpace) then
     begin
        // Not enough space for the operation
        SetLastError(ERROR_NOT_ENOUGH_MEMORY);
        // Raise error
        RaiseLastWin32Error;
     end
     else
     begin
        // Allocate a single buffer for the write
        GetMem(lpBuffer, SizeOf(DWORD) + dwSource);
        // Resource protection
        try
           // Move the source length first
           Move(dwSource, lpBuffer^, SizeOf(DWORD));
           // Move the string
           Move(Pointer(Data)^, lpBuffer[SizeOf(DWORD)], dwSource);
           // Update total size to write
           Inc(dwSource, SizeOf(DWORD));
           // Calculate for wrap around
           if ((FHeader.Tail + dwSource) > FHeader.Size) then
           begin
              // Calculate the first write
              dwWrap:=FHeader.Size - FHeader.Tail;
              // Write the data
              if not(WriteFile(FWriter, lpBuffer^, dwWrap, dwWrite, nil)) then RaiseLastWin32Error;
              // Set file position
              SetLocation(FWriter, 0);
              // Update remaining byte count to write
              Dec(dwSource, dwWrap);
              // Write the remaining data
              if not(WriteFile(FWriter, lpBuffer[dwWrap], dwSource, dwWrite, nil)) then RaiseLastWin32Error;
              // Update the tail loction
              FHeader.Tail:=dwSource;
           end
           else
           begin
              // Write the data
              if not(WriteFile(FWriter, lpBuffer^, dwSource, dwWrite, nil)) then RaiseLastWin32Error;
              // Update the tail location
              Inc(FHeader.Tail, dwSource);
           end;
        finally
           // Free memory
           FreeMem(lpBuffer);
        end;
     end;
  end;

end;

function TPersistQueue.Pop: String;
var  lpSize:        Array [0..3] of Byte;
     dwLength:      DWORD;
     dwRead:        DWORD;
     dwWrap:        DWORD;
begin

  // Check active
  CheckActive;

  // Check head and tail location
  if (FHeader.Head = FHeader.Tail) then
     // Nothing to pop from the queue
     SetLength(result, 0)
  else
  begin
     // We need to read the DWORD for the string length
     if ((FHeader.Head + SizeOf(DWORD)) > FHeader.Size) then
     begin
        // Calculate the first read
        dwWrap:=FHeader.Size - FHeader.Head;
        // Read the data length data
        if not(ReadFile(FReader, lpSize, dwWrap, dwRead, nil)) then RaiseLastWin32Error;
        // Set file position
        SetLocation(FReader, 0);
        // Read the remaining DWORD
        if not(ReadFile(FReader, lpSize[dwWrap], SizeOf(DWORD) - dwWrap, dwRead, nil)) then RaiseLastWin32Error;
        // Update the head loction
        FHeader.Head:= SizeOf(DWORD) - dwWrap;
     end
     else
     begin
        // Read the data length data
        if not(ReadFile(FReader, lpSize, SizeOf(DWORD), dwRead, nil)) then RaiseLastWin32Error;
        // Update the head location
        Inc(FHeader.Head, SizeOf(DWORD));
     end;
     // Cast 4 byte buffer to length
     dwLength:=Cardinal(Pointer(@lpSize)^);
     // Set result string length
     SetLength(result, dwLength);
     // Check length
     if (dwLength > 0) then
     begin
        // Now we need to read the string, which may wrap the queue file boundaries
        if ((FHeader.Head + dwLength) > FHeader.Size) then
        begin
           // Calculate the first read
           dwWrap:=FHeader.Size - FHeader.Head;
           // Read the data
           if not(ReadFile(FReader, Pointer(result)^, dwWrap, dwRead, nil)) then RaiseLastWin32Error;
           // Set file position
           SetLocation(FReader, 0);
           // Update remaining byte count to read
           Dec(dwLength, dwWrap);
           // Read the remaining data
           if not(ReadFile(FReader, result[Succ(dwWrap)], dwLength, dwRead, nil)) then RaiseLastWin32Error;
           // Update the head loction
           FHeader.Head:=dwLength;
        end
        else
        begin
           // Read the data
           if not(ReadFile(FReader, Pointer(result)^, dwLength, dwRead, nil)) then RaiseLastWin32Error;
           // Update the head location
           Inc(FHeader.Head, dwLength);
        end;
     end;
  end;

end;

procedure TPersistQueue.SetLocation(Handle: THandle; Position: Int64);
begin

  // Check active
  if (Handle = INVALID_HANDLE_VALUE) then
  begin
     // Set last error
     SetLastError(ERROR_INVALID_HANDLE);
     // Raise error
     RaiseLastWin32Error;
  end
  else
  begin
     // Update the position with the header size
     Inc(Position, SizeOf(FHeader));
     // Set file pointer for the handle
     if (SetFilePointer(Handle, LARGE_INTEGER(Position).LowPart, @LARGE_INTEGER(Position).HighPart, FILE_BEGIN) = $FFFFFFFF) then RaiseLastWin32Error;
  end;
end;

function TPersistQueue.GetFreeSpace: Int64;
var  qwDiff:        Int64;
begin

  // Calculate the difference in head / tail
  qwDiff:=FHeader.Head - FHeader.Tail;

  // Calculate the free space in the queue file
  if (qwDiff = 0) then
     // All of the file is free
     result:=FHeader.Size
  else
  begin
     // Check location
     if (qwDiff < 0) then
        // Tail is after head
        result:=FHeader.Size + qwDiff
     else
        // Tail is before head
        result:=qwDiff;
     // Decrement the result by one so we can detect an empty file
     Dec(result);
  end;

end;

procedure TPersistQueue.CheckActive;
begin

  // Check active state
  if not(GetActive) then
  begin
     // Set last error
     SetLastError(ERROR_INVALID_HANDLE);
     // Raise error
     RaiseLastWin32Error;
  end;

end;

function TPersistQueue.GetActive: Boolean;
begin

  // Determine if the queue is active
  result:=not((FWriter = INVALID_HANDLE_VALUE) or (FReader = INVALID_HANDLE_VALUE));

end;

procedure TPersistQueue.OpenQueue(FileName: String);
var  qwSize:        Int64;
     dwRead:        DWORD;
begin

  // Flush
  FlushAndClose;

  // Exception trap
  try
     // Open the existing file
     FWriter:=CreateFile(PChar(FileName), GENERIC_READ or GENERIC_WRITE, FILE_SHARE_READ, nil, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, 0);
     FReader:=CreateFile(PChar(FileName), GENERIC_READ, FILE_SHARE_READ or FILE_SHARE_WRITE, nil, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, 0);
     // Check the handles
     if (FWriter = INVALID_HANDLE_VALUE) or (FReader = INVALID_HANDLE_VALUE) then
        // Raise last error
        RaiseLastWin32Error
     else
     begin
        // Read the header in
        if not(ReadFile(FReader, FHeader, SizeOf(FHeader), dwRead, nil)) then RaiseLastWin32Error;
        // Check the size read and then compare magic header
        if (dwRead <> SizeOf(FHeader)) or (FHeader.Magic <> PQ_FILEMAGIC) then
        begin
           // Set last error
           SetLastError(ERROR_INVALID_DATA);
           // Raise error
           RaiseLastWin32Error;
        end
        else
        begin
           // Get the file size
           LARGE_INTEGER(qwSize).LowPart:=GetFileSize(FReader, @LARGE_INTEGER(qwSize).HighPart);
           // Check results
           if (LARGE_INTEGER(qwSize).LowPart = $FFFFFFFF) then RaiseLastWin32Error;
           // Now check against stored size
           if (qwSize <> (FHeader.Size + SizeOf(FHeader))) then
           begin
              // Set last error
              SetLastError(ERROR_INVALID_DATA);
              // Raise error
              RaiseLastWin32Error;
           end;
           // Set file pointers
           SetLocation(FWriter, FHeader.Tail);
           SetLocation(FReader, FHeader.Head);
        end;
     end;
  except
     // Flush
     FlushAndClose;
     // Re-raise the exception
     raise;
  end;

end;

procedure TPersistQueue.CreateQueue(FileName: String; Size: Int64);
var  qwSize:        Int64;
begin

  // Flush current
  FlushAndClose;

  // Exception trap
  try
     // Create the new file
     FWriter:=CreateFile(PChar(FileName), GENERIC_READ or GENERIC_WRITE, FILE_SHARE_READ, nil, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, 0);
     FReader:=CreateFile(PChar(FileName), GENERIC_READ, FILE_SHARE_READ or FILE_SHARE_WRITE, nil, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, 0);
     // Check the handles
     if (FWriter = INVALID_HANDLE_VALUE) or (FReader = INVALID_HANDLE_VALUE) then
        // Raise last error
        RaiseLastWin32Error
     else
     begin
        // Check minimum size
        if (Size < PQ_MINSIZE) then Size:=PQ_MINSIZE;
        // Set header buffer
        FHeader.Magic:=PQ_FILEMAGIC;
        FHeader.Head:=0;
        FHeader.Tail:=0;
        FHeader.Size:=Size;
        // Set total size
        qwSize:=Size + SizeOf(FHeader);
        // Set the file pointer
        if (SetFilePointer(FWriter, LARGE_INTEGER(qwSize).LowPart, @LARGE_INTEGER(qwSize).HighPart, FILE_BEGIN) = $FFFFFFFF) then RaiseLastWin32Error;
        // Set end of file
        if not(SetEndOfFile(FWriter)) then RaiseLastWin32Error;
        // Set file pointers
        SetLocation(FWriter, FHeader.Tail);
        SetLocation(FReader, FHeader.Head);
     end;
  except
     // Flush
     FlushAndClose;
     // Re-raise the exception
     raise;
  end;

end;

procedure TPersistQueue.FlushAndClose;
var  dwWrite:       DWORD;
begin

  // Close reader handle
  if not(FReader = INVALID_HANDLE_VALUE) then
  begin
     // Resource protection
     try
        // Close the file handle
        CloseHandle(FReader);
     finally
        // Clear the handle
        FReader:=INVALID_HANDLE_VALUE;
     end;
  end;

  // Check handle
  if not(FWriter = INVALID_HANDLE_VALUE) then
  begin
     // Resource protection
     try
        // Seek to start of file
        if (SetFilePointer(FWriter, 0, nil, FILE_BEGIN) = $FFFFFFFF) then RaiseLastWin32Error;
        // Write the header to disk
        if not(WriteFile(FWriter, FHeader, SizeOf(FHeader), dwWrite, nil)) then RaiseLastWin32Error;
     finally
        // Resource protection
        try
           // Close the file handle
           CloseHandle(FWriter);
        finally
           // Clear the handle
           FWriter:=INVALID_HANDLE_VALUE;
        end;
     end;
  end;

end;

constructor TPersistQueue.Create;
begin

  // Perform inherited
  inherited Create;

  // Set initial defaults
  FWriter:=INVALID_HANDLE_VALUE;
  FReader:=INVALID_HANDLE_VALUE;
  FillChar(FHeader, SizeOf(FHeader), 0);
  SetLength(FFileName, 0);

end;

destructor TPersistQueue.Destroy;
begin

  // Resource protection
  try
     // Flush the header and close the file
     FlushAndClose;
  finally
     // Perform inherited
     inherited Destroy;
  end;

end;

end.
0
 
LVL 17

Author Comment

by:Wim ten Brink
Comment Utility
About disk usage... I have no problems with allocating a few gigabytes of diskspace for the queue if this will improve the performance. Calculating the proper amount to do so would be tricky, though.
After playing around a bit with my current solution, I did discover that I could easily truncate the file right after the header if the stack counter reaches zero. That means that any data in the stack has become void. And the stack will reach a zero count once in a while.

I have also experimented a bit with my current solution and it is extremely fast already. Because it's done in an out-of-process executable, I also have a timer which reads the count once in a while and displays it in some label. The highest count so far went up to 5,000 records with records being around 200 bytes each. But this was just random testdata. The final solution might have records varying from 10 bytes or less to a few megabytes per record. It is just XML data and my Pop-process is reading it and deciding which actions should follow based upon this XML data. (For example, write to a log file, start another process, push data to a database, whatever.)

However, the Persistent Queue solution looks very fast too and happens to be slightly closer to the Windows API than my current solution. :-) (Yes, I just use the File type from Pascal.) Then again, mine was just a quick thought. :-) Seems a very good solution to me.

Well, now I have to deal with the other bottleneck which is the whole COM handling. But that will be the only bottleneck that I have to deal with for now. At least, for this Queue thing. :-)
0
 
LVL 26

Expert Comment

by:Russell Libby
Comment Utility

Your biggest bottleneck at this point is going to be the COM server, as the COM cross-process marshalling is very slow compared with inproc COM libraries. And testing 5000 writes with 200 bytes each on TPersistQueue ran @ ~appx 16-20ms. (100,000 cps avg). So if possible, use early binding over late binding, as late binding will incur a double hit; one for GetIDsOfNames and the other for Invoke. With out of proc servers, the rule tends to be; do more in fewer calls.

Russell
0
 
LVL 17

Author Comment

by:Wim ten Brink
Comment Utility
No worries, I am experienced enough with COM and will be using early binding for this project. :-) It's just that this is the easiest solution to implement since I have about a dozen or so processes that will be using one and the same stack/queue. (Especially since this code will be running on a dual-core system.) If it's just the COM-part that happens to be the biggest bottleneck then that's fine by me. With an in-process COM server I would still have to deal with the file being accessed from multiple processes, which I definitely want to prevent.
Btw, the stack might even be used through DCOM in connection with several other computers.
0

Featured Post

Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

Join & Write a Comment

In this tutorial I will show you how to use the Windows Speech API in Delphi. I will only cover basic functions such as text to speech and controlling the speed of the speech. SAPI Installation First you need to install the SAPI type library, th…
In my programming career I have only very rarely run into situations where operator overloading would be of any use in my work.  Normally those situations involved math with either overly large numbers (hundreds of thousands of digits or accuracy re…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

763 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

7 Experts available now in Live!

Get 1:1 Help Now