We help IT Professionals succeed at work.

Fastest Way to Extract Record from a Long Semicolon Delimited WideString?

coole
coole asked
on
Medium Priority
407 Views
Last Modified: 2010-04-05
I am writing a function to get data from server and process accordingly. The data receive from server is in a semi colon delimited widestring format.

For example,

var
  Input: WideString;
  i, TotalRec: Integer;

begin
  TotalRec := GetData(Input);
  // Input will have values like 'leon,24,89,100;jimmy,30,77,56;'
  for i := 0 to TotalRec do
  // TotalRec could range from 5,000 - 20,000
  // I need to extract each record to process
  // What's the fastest way to do this?
end;

Comment
Watch Question

Commented:
unit Unit_Q_20946183;

interface

uses
  Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms,
  Dialogs, StdCtrls, Buttons;

type
  TForm1 = class(TForm)
      SpeedButton1: TSpeedButton;
      ListBox1: TListBox;
      procedure SpeedButton1Click(Sender: TObject);
    private   { Private declarations }
    public    { Public declarations }
      procedure ExtractFromString(Input: WideString);
  end;

var
  Form1: TForm1;

implementation

{$R *.dfm}

procedure TForm1.ExtractFromString(Input: WideString);
var
//  i, TotalRec: Integer;
  P:      Integer;
  S:      string;
  procedure Extract(S: string);
  begin
    ListBox1.Items.Add(S);
    Delete(Input, 1, P);
    P := Pos(';',Input);
  end;
begin
  {
  TotalRec := GetData(Input);
  // Input will have values like 'leon,24,89,100;jimmy,30,77,56;'
  for i := 0 to TotalRec do
  // TotalRec could range from 5,000 - 20,000
  // I need to extract each record to process
  // What's the fastest way to do this?
  }
  P := Pos(';',Input);
  while (P>0) do
  begin
    S := Copy(Input, 1, P-1);
    Extract(S);
  end;
  if (Input<>'') then
    Extract(Input);
end;

procedure TForm1.SpeedButton1Click(Sender: TObject);
begin
  ExtractFromString('Leon,24,89,100;Jimmy,30,77,56');
end;

end.

emil

Commented:
object Form1: TForm1
  Left = 199
  Top = 114
  Width = 696
  Height = 480
  Caption = 'Form1'
  Color = clBtnFace
  Font.Charset = DEFAULT_CHARSET
  Font.Color = clWindowText
  Font.Height = -11
  Font.Name = 'MS Sans Serif'
  Font.Style = []
  OldCreateOrder = False
  PixelsPerInch = 96
  TextHeight = 13
  object SpeedButton1: TSpeedButton
    Left = 36
    Top = 22
    Width = 103
    Height = 22
    Caption = 'Extract'
    OnClick = SpeedButton1Click
  end
  object ListBox1: TListBox
    Left = 150
    Top = 2
    Width = 531
    Height = 441
    ItemHeight = 13
    TabOrder = 0
  end
end

Commented:
You can do as follows:

procedure ParseRecData;
var
 ml:TStrings;
 Input:WideString;
begin
  // Input:=  ----> Method to get Input from server
  ml:=TStringList.Create;
  try
    if Input[1]=';' then Delete(Input,1,1);
    while Pos(';',Input) <> 0 do begin
      ml.Add(copy(Input,1,Pos(';',Input) - 1));
      Delete(Input,1,Pos(';',Input));
    end;
    if length(Input) <> 0 then ml.Add(Input);
    Memo1.Lines.Assign(ml);
  finally
    ml.Free;
  end;
end;

Commented:
boy, one has to be quick if he wishes to be the first :-)

Commented:
Next code is better:

procedure TForm1.ExtractFromString(Input: WideString);
var
  P:      Integer;
  S:      string;
  procedure Extract(S: string);
  begin
    ListBox1.Items.Add(S);
    Delete(Input, 1, P);
    P := Pos(';',Input);
  end;
begin
  P := Pos(';',Input);
  while (P>0) do
  begin
    S := Copy(Input, 1, P-1);
    Extract(S);
  end;
  if (Input<>'') then
    ListBox1.Items.Add(S);
end;

emil

Commented:
 if (Input<>'') then
    ListBox1.Items.Add(Input);

Commented:
Forget any solution using Delete on the very large 'Input' string or Input:= copy( Input, xxx).
This will take a huge amount of time since the string is copied in memory over and over.

Point is to keep the Input string intact so that it isn't moved around in memory.
You could scan the string for the position of the delimiter and take the data out of it.
You can do this using AnsiStrPos(Str, SubStr: PChar) where you move the begin pointer in the string until you reach the end of the string.

Another solution could be to replace the delimiter by a CRLF and paste that string in the text property of a stringlist.
Then you can directly browse the stringlist.
Ferruccio AccalaiCEO and Co-founder
CERTIFIED EXPERT

Commented:
what about this?

procedure TForm1.Button1Click(Sender: TObject);
procedure GetRecords(var Records: TStrings;Input: WideString);
  begin
    Records.Clear;
    Records.Delimiter := ';';
    Records.DelimitedText := Input;
  end;
var
List: TStrings;
begin
  List := TStringList.Create;
  try
    GetRecords(List,'leon,24,89,100;jimmy,30,77,56;');
    ListBox1.Items.Assign((list));//do here whatever you want with your record list
  finally
    List.Free;
  end;
end;

Commented:
uses
  StrUtils;

procedure TForm1.Extract(S: WideString);
var
  P,P1: integer;
begin
  P1 := 0;
  P := Pos(';',S);
  Memo1.Lines.BeginUpdate;
  while P > 0 do
    begin
    Memo1.Lines.Add(Copy(S,P1+1,P-P1-1));
    P1 := P;
    P := PosEx(';',S,P+1);
    end;
  if P1 < Length(S) then
    Memo1.Lines.Add(Copy(S,P1+1,Length(S)-P1));
  Memo1.Lines.EndUpdate;
end;

Commented:
And here is some test

my solution: 17665 ms

and

Feruccio: 671 ms

Well done.

Commented:
This is why I love to read all the questions, once in a while I get my eyes opened ...

The best solution in my opinion: Ferruccio68
Commented:
Unlock this solution and get a sample of our free trial.
(No credit card required)
UNLOCK SOLUTION

Commented:
New unofficial test results
-----------------------------

my solution: 17665 ms
Feruccio: 671 ms
mocarts: 30ms

Well done.

It looks like I must improve time resolution (10ms now) or change tested data. ;-)

Commented:
I think Feruccio's and mocarts's solutions are equivalent each-other. They are the best !!!!

emil

Author

Commented:
Thank you guys. Excellent work!
Unlock the solution to this question.
Thanks for using Experts Exchange.

Please provide your email to receive a sample view!

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.