stringtokenizer in Delphi/Object Pascal

hi there,

I was wondering if there is a sort of StringTokenizer in Delphi. I know that Java has a StringTokenizer, but I can't find it in Delphi, can anyone help me?

A SringTokenizer is an object which divides a string into parts. You can give up a delimiter like:

String deString = "hi,there,this,are,commas";
StringTokenizer someName = new StringTokenizer(deString, ",");

And if i want the next token/element i simply do:
someName.nexttoken();

But this is JAVA, but to implement this in Delphi? Plz hELP!!!

Regards and thanks in advance,
Ruben Bouman,
The Netherlands
rubenboumanAsked:
Who is Participating?

Improve company productivity with a Business Account.Sign Up

x
 
Tomas Helgi JohannssonConnect With a Mentor Commented:
Hi!

You can create your own Tokenizer easelly. This is how I did it (coded in 3 minutes in D5 ;)


unit Unit1;

interface

uses
  Windows, Messages, SysUtils, Classes, Graphics, Controls, Forms, Dialogs,
  StdCtrls;

type
  TMyClass = class
  private
    TokList: TStrings;
  public
    constructor Create(const s, t: string);
    destructor Destroy;
    procedure AddLists(s: TStrings);
  published
    property thelist: TStrings read TokList write AddLists;
  end;

type
  TForm1 = class(TForm)
    ListBox1: TListBox;
    Edit1: TEdit;
    Button1: TButton;
    procedure Button1Click(Sender: TObject);
  private
    { Private declarations }
  public
    { Public declarations }
  end;

var
  Form1: TForm1;

implementation

{$R *.DFM}

{ TMyClass }

procedure TMyClass.AddLists(s: TStrings);
begin
  TokList.Assign(s);
end;

constructor TMyClass.Create(const s, t: string);
var
  i, j: Integer;
  temp : String;
begin
  temp := s;
  TokList := TStringList.Create;
  i := 1;
  j := 1;
  if Pos(t, temp) > 0 then
    while Pos(t, temp) > 0 do
    begin
      i := Pos(t, temp);
      Toklist.add(Copy(temp, j, i - j));
      delete(temp,j,i);
    end;
    Toklist.Add(Copy(temp, j, i - j));
end;

destructor TMyClass.Destroy;
begin
  TokList.Free;
end;

procedure TForm1.Button1Click(Sender: TObject);
var
  myTokl: TMyClass;
begin
  myTokl := TMyClass.Create(Edit1.Text, '/');
  ListBox1.Items.Assign(myTokl.thelist);
  myTokl.Free;
end;

Regards,
Tomas Helgi
0
 
EpsylonCommented:
Check out TStringList and its CommaText property....
0
 
EpsylonCommented:
Here is an example:

var s: String;
    sl: TStringList;
    i: Integer;
begin
  sl := TStringList.Create;
  s := 'hi,there,this,are,commas';
  sl.CommaText := s;
  for i := 0 to sl.Count -1 do
    ShowMessage(sl[i]);
  sl.Free;
end;
0
Get your problem seen by more experts

Be seen. Boost your question’s priority for more expert views and faster solutions

 
VSFCommented:
This is a simple example...
Add:
1 Editbox - Tedit
1 Button - TButton
1 Listbox - TlistBox

//Add this procedure
function sBreakApart(BaseString, BreakString: string; StringList: TStringList): TStringList;
var
EndOfCurrentString: byte;
TempStr: string;
begin
repeat
EndOfCurrentString := Pos(BreakString, BaseString);
if EndOfCurrentString = 0 then
StringList.add(BaseString)
else
StringList.add(Copy(BaseString, 1, EndOfCurrentString - 1));
BaseString := Copy(BaseString, EndOfCurrentString + length(BreakString), length(BaseString) - EndOfCurrentString);

until EndOfCurrentString = 0;
result := StringList;
end;

//Add this code to the OnClick event of the button
procedure TForm1.Button1Click(Sender: TObject);
var
t: TStringList;
begin
t := TStringList.create;
ListBox1.Items.Assign(sBreakApart(Edit1.Text, ' ', t));
t.free;
end;

Run your program and type your name in the Editbox
Press the button
and the list box should show the your name Tokenized using the blank spaces as the string separator.

Adjust this function to any string u want!
0
 
VSFCommented:
To adjust the function just send to the
BreakString: string
any separator u want!

Hope this helps!

E-mail me if u want a compiled sample or leave your e-mail and I'll send it to you!

VSF
www.enge.cjb.net
www.victory.hpg.com.br
UIN:14016999
0
 
CesarioCommented:
Hi rubenbouman,

you can use the TPARSER-Object from Delphi. You will find it in the CLASSES Unit.

Best Regards

Cesario
0
 
Tomas Helgi JohannssonCommented:
Hi again!

I forgot to put in the "NextToken" function.
In TMyClass
under private write:
ix : Integer;
under public write :

function NextToken : String;

and the implementation is

function NextToken : String;
begin
    if ix < TokList.Count then
    begin
       result := TokList[ix];
       Inc(ix);
    end
    else
      result := '';
end;

add to the Constructor

ix := 0;

To prevent the user of this class to access the TokList outside the class you simply delete the property line. ;)

Regards,
Tomas Helgi
0
 
VSFCommented:
No need to post answers! : )

Just lock his question!

VSF
0
 
kretzschmarCommented:
//this simple function, does what you need
//it returns the first part as result, and shorten the given string
//for next item simple call again
//if there are no items, then the given string becomes empty
function SplitStr(Var AString : String; ADelim : String) : String;
var APos : Integer;
begin
  Result := '';
  APos := Pos(ADelim,AString);
  if APos > 0 then
  begin
    Result := Copy(AString,1,APos-1);
    AString := copy(Astring,APos+length(Adelim),MaxLongInt);
  end
  else
  begin
    Result := AString;
    AString := '';
  end;
end;

//a usage sample
procedure TForm1.Button1Click(Sender: TObject);
var s : String;
begin
 ListBox1.Clear;
 s := edit1.text;
 while s <> '' do
   listbox1.items.add(SplitStr(s,edit2.text));
end;

meikl ;-)
0
 
VSFCommented:
so rubenbouman... have u tested any code?!
Feel free to ask if u have any doubts!
0
 
kretzschmarCommented:
hi vsf,

who are you, the boy or the girl ?  :-))

sorry to be off topic

meikl ;-)
0
 
geobulCommented:
Hi,

I've written a small piece of code about dividing a sentence into words. The sentence has to be entered
in the Edit1. Pressing the button passes it as a string to the Parser function and the result is returned
in SList. You can add additional symbols as word delimiters (space, tab and comma here) if you want.
A symbol for the end of the sentence is not mandatory because it ends at the end of the passed string
by default. If such a symbol exists, the rest of the string will be ignored.

unit Unit1;

interface

uses
 Windows, Messages, SysUtils, Classes, Graphics, Controls, Forms, Dialogs,
 StdCtrls;

type
 TForm1 = class(TForm)
   Button1: TButton;
   Edit1: TEdit;
   Memo1: TMemo;
   procedure Button1Click(Sender: TObject);
   procedure FormCreate(Sender: TObject);
   procedure FormDestroy(Sender: TObject);
 private
   { Private declarations }
 public
   { Public declarations }
 end;

var
 Form1: TForm1;
 SList : TStringList;

implementation

{$R *.DFM}

procedure Parser(Sent: string);
const
 delimiters = [CHR(32),CHR(8),','];
 FullStops = ['.','!','?'];
var
 ch : char;
 i, cnt : integer;
 token : string;
begin
 SList.Clear;
 cnt := Length(Sent);
 token := '';
 if (cnt > 0) then begin
   for i := 1 to cnt do begin
     // Next character
     ch := Sent[i];
     // new token = word
     if ch in Delimiters then begin
       if length(token) > 0 then SList.Add(token);
       token := '';
     end else begin
       // end of sentence
       if ch in FullStops then begin
         if length(token) > 0 then SList.Add(token);
         token := '';
         break;
       end else begin
         // regular character
         token := token + ch;
       end;
     end;
   end;
   // last token
   if length(token) > 0 then SList.Add(token);
 end;
end;

procedure TForm1.Button1Click(Sender: TObject);
var
 i: integer;
begin
 Parser(Edit1.Text);
 // you can analize the words here:
 // for i := 0 to SList.Count - 1 do begin
 //   ...
 // end;  
 // The next line is for visualisation only.
 Memo1.Lines.Assign(SList);
end;

procedure TForm1.FormCreate(Sender: TObject);
begin
 SList := TStringList.Create;
end;

procedure TForm1.FormDestroy(Sender: TObject);
begin
 SList.Free;
end;

end.

Regards, Geo
0
 
rubenboumanAuthor Commented:
Perfect solution! I already used it many many many times!!!

Regards,
Ruben Bouman
0
 
VSFCommented:
Hi kretzschmar,

I'm the Boy... the girl is my girlfriend Carol. We're togather for almost 3 years now!


0
 
kretzschmarCommented:
I guessed that, but ... i wasn't sure :-))

nice foto
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.