Go Premium for a chance to win a PS4. Enter to Win

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 572
  • Last Modified:

Faster string concatenation?

Dear Experts, my app manipulates a lot of text strings and by using a line profiler program, I see that a good portion of time is on a couple of key lines in my program that do the string concatenation operation:

str1 := str1 + str2;

I already use FastMM4 in my project and my understanding is that it does help to speed up string operations... but is there any component or unit that would do a more efficient/faster string concatenation?

Thanks!
    Shawn

P.S: For that matter, the string "Delete" procedure seems to be a bit of a bottleneck for me too. As does the "StringReplace" function.
0
shawn857
Asked:
shawn857
  • 12
  • 5
  • 4
  • +4
3 Solutions
 
ste5anSenior DeveloperCommented:
There is no general advice.

It strongly depends on your problem and you solution. The key maybe to avoid StringReplace().
0
 
shawn857Author Commented:
Thanks. I thought there was a package called "Jedi" or something, that had faster string routines... no? Would that work?

Thanks
   Shawn
0
 
aikimarkCommented:
There are many great routines in the JEDI project.  Here are links to their pages, both project and download.
http://sourceforge.net/projects/jvcl/
http://www.delphi-jedi.org/
http://jvcl.delphi-jedi.org/

=============
Are you trying to concatenate ASCII strings or wide strings/unicode?
What version of Delphi are you using?
How many string concatenation operations are you doing?
How long are your final resulting strings?

It will help if you read this article from one of the BorCon2004 sessions.
http://conferences.embarcadero.com/article/32120

I have found the string builder approach to be the best.  This is a native class in the .Net framework and I've written my own in different environments.  Basically, each string you add to the object is added to a list.  Only when you want the entire list does the actual concatenation occur.
In the BorCon session, they mentioned Halvard's class.  Here are the links to his pas files.
https://code.google.com/p/omnithreadlibrary/source/browse/trunk/src/HVStringBuilder.pas
https://code.google.com/p/omnithreadlibrary/source/browse/trunk/src/HVStringData.pas

Delphi2009 and later have a TStringBuilder class that is worth testing before you start adding other libraries to your project.

Also, you might open up a stream object, replacing your concatenation operations with stream write operations.
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
ste5anSenior DeveloperCommented:
They have faster string routines, but only for certain cases like character based replaces.. you need to compare them in your actual use-case.
0
 
shawn857Author Commented:
Hi guys, I'm using Delphi 7 and what I'm doing is basically parsing a CSV file character-by-character. The statements that seem to take the most time are these:

HoldString := Holdstring + c;

... where HoldString is a string variable that accumulates the string that gets added to it character-by-character by the "c" variable (declared as Char).


Thanks!
   Shawn
0
 
Sinisa VukCommented:
Agree with others - there is no general approach. Best is to use buffer memory to read portion of file in and analize it, keep integer indexes of start and current position in this buffer. Current index is last good character (valid for you) - when you detect field separator - then do copy of all chars between start and current index - so no char by char manipulation.
0
 
shawn857Author Commented:
Thanks Sinisa, you're probably right, but this would mean major major surgery on my program to create a whole new parsing engine. For now, I was just looking (hoping?) for a faster way to do some basic string manipulations - particularly the concatenate operation. So I guess there's really nothing faster than the standard old '+' operator then eh?

Cheers
   Shawn
0
 
ste5anSenior DeveloperCommented:
@Sinisa is right. I had also this kind of processing in mind as I've read your "HoldString := Holdstring + c;"..

btw, when your parsing engine is well structured, then reengineering is not a huge or complex task.

And for the +: nope.
0
 
aikimarkCommented:
Why are you parsing a CSV file, character-by-character?!?

Please answer the questions I posted, http:#a39981695
0
 
Sinisa VukCommented:
Made example in your another EE question.
In short I use global buffer, track start pointer of a String (PChar), look for delimiter (;) and replace it with #0 (zero) - this way I'm get zero terminated string (value) without concatenating.
0
 
MerijnBSr. Software EngineerCommented:
As long as you use non unicode strings you can use QStrings, which is old but is fast with string manipulation, you can still download it here: http://www.torry.net/vcl/vcltools/text/adqstrings.zip

It won't help with concatenation, but it does have a fast StringReplace() alternative.
0
 
shawn857Author Commented:
Thanks for all the responses guys...


Aikimark, you asked:

(1) Are you trying to concatenate ASCII strings or wide strings/unicode?

Just simple ASCII strings.


(2) What version of Delphi are you using?

D7


(3) How many string concatenation operations are you doing?

Well, as I mentioned, I'm looping through each line of data parsing for fields so I'm concatenating each "good" character on to the end of a string, and once I hit another delimiter, I write out the string. So I'm doing a *lot* of these little concatenations... millions.


(4) How long are your final resulting strings?

Not too long - just as long as one "field" in a CSV record might be. Maybe on average 15-20 characters.

Thanks
   Shawn
0
 
aikimarkCommented:
Can we safely assume that you are doing line-at-a-time reading of the file and not character-by-character reading?

Not knowing what constitutes a 'good' character, I would suggest that you use a reasonably long pchar variable, initialized to spaces (or some non-null character).  As you loop through your record, to a direct value assignment of the next available position in the pchar array of the good character you have just found.
0
 
shawn857Author Commented:
Yes aikimark - I'm reading line-by-line from my file on disk.

I really should be thinking "bigger"... it must be lack of sleep, but here I am wasting time trying to optimize string manipulation routines, when what I really should be doing is finding the fastest method to parse a CSV file - and it's certainly not by concatenating characters one-at-a-time to the end of another string!
   I'd like to open a new question addressing just that, and I'll close out this one and award points. Sorry....

Thanks
    Shawn
0
 
Geert GruwezOracle dbaCommented:
you need a combo of unit faststrings and one of russel's comments
unfortunately i haven't found russel's comment yet

but ... Peter Morris was the one who wrote FastStrings while he was stuckindoors.
this babbling of mine probably doesn't make sense at the moment, but hang on ...
just imagine if someone had the same problem as you did and decided to take the proverbial bull by the horns and create a unit for very fast string manipulation
the naming of that unit would be obvious ... yes ... FastStrings
I knew that unit existed ... due to have come across it on this website the previous decade ... ugh, don't be resented at the code being that old yet  ... it was originally written in Delphi 7 ...

without further ado and without going into details about how to use the search box on this site with the words "Delphi FastStrings Peter Morris", which likely will give you a few hits with my comments too, here is a link for the updated XE2 version

https://code.google.com/p/ryulib4delphi/source/browse/trunk/XE2/FastStrings.pas
I hope the newer version works on your XE2, it took me a long while (at least 20 seconds) to find that newer version.

That unit off course, is just the very basic string handling, and off course, you're looking for that itsy pitsy tad more, namely concatenation of strings
SO, again without further ado, here is another unit which uses the earlier unit
https://code.google.com/p/ryulib4delphi/source/browse/trunk/XE2/FastStringFuncs.pas

After checking that secondary unit, you will probably wonder where in all that code do i find what i'm looking for, and why would i use assembler code ? The first answer would be, I have to admit: "it's not in that code", but you might now remember that the quote "unfortunately i haven't found russel's comment yet" ...
i already hear all those minds going "aah, now it's coming ..."

but alas, i haven't found that comment yet
0
 
Geert GruwezOracle dbaCommented:
hey, i'm back again.
no i haven't found russel's comment yet.
sorry, very disappointing, i know. but i found an appetizer
there are a few experts on this site who aren't "genuis" yet, at least on this site, but ...
actually they are legend level, by my standards, but i'm not a genius

here is one routine to "move" characters from 1 string to another
you could call it concatenation if you "move" characters from 1 string to the end of another string
it's written in asm, but a true legend, he's got 662000 points at the moment
here is a link to the assembler concatenation comment:
http://www.experts-exchange.com/Programming/Languages/Pascal/Delphi/Q_20106634.html

yes, i know it's a decade old, but i stated that in my earlier comment

and no, i still haven't found russel's comment yet
0
 
Geert GruwezOracle dbaCommented:
aha. good ol russel, i knew he would come around in the end and give me his comments
here it is

http://www.experts-exchange.com/Programming/Languages/Pascal/Delphi/Q_20615170.html

you might be wondering where "russel" comes from ?
0
 
Sinisa VukCommented:
where do you store parsed values from csv? how do you work whit them (after you've got values)?
0
 
shawn857Author Commented:
Geert: thank you for all this, particularly Russel's concat method as show here:

http://www.experts-exchange.com/Programming/Languages/Pascal/Delphi/Q_20615170.html

I made a small project to try it, but I don't know how to really use it - I get an access violation error. Is it as simple as declaring a TCharStream variable:

var
   newCharStream : TCharStream

... and then just appending to it like this? :

newCharStream.Append(SomeChar);



Sinisa: you asked "where do you store parsed values from csv?". As I parse the record, I store each field to a TStringList. Then I do some checks on each of these StringList values, modify some of them if needed, then immediately write them out to an output file and start again reading the next record from the input file.

Thanks
    Shawn
0
 
shawn857Author Commented:
H Geert, sorry to bug you on a weekend but if you have a second could you kindly quickly show how to utilize Russel's TCharStream method? I tried and get an access violation error... :-(

Thanks!
   Shawn
0
 
Geert GruwezOracle dbaCommented:
just read your mail ...
after a weekend of masonry and just before going to bed as i'm rather burnt out now
it'll be at least a week before i'm back on EE
too busy ... i'll post when i get back
0
 
shawn857Author Commented:
of course Geert... get some rest!  :-)

Cheers
   Shawn
0
 
shawn857Author Commented:
Hi Geert, are you back in business?  :-)

Cheers
   Shawn
0
 
shawn857Author Commented:
Hi Geert, are you still with me...?

Cheers
   Shawn
0
 
shawn857Author Commented:
Can anyone help me on this please??

Thanks
    Shawn
0
 
mlmccCommented:
I've requested that this question be deleted for the following reason:

The question has either no comments or not enough useful information to be called an "answer".
0
 
aikimarkCommented:
I think several comments would qualify as solutions.  In order of their "goodness":
Geert: http:#a39985215
aikimark: http:#a39981695
Geert: http:#a39985209
0
 
shawn857Author Commented:
Sorry guys... I was just waiting for Geert (or somebody) to reply with an example on how to implement the CharStream append method, as I had mentioned here:

"Geert: thank you for all this, particularly Russel's concat method as show here:

http://www.experts-exchange.com/Programming/Languages/Pascal/Delphi/Q_20615170.html"

Thanks
   Shawn
0
 
aikimarkCommented:
Thanks for responding, Shawn.

For future readers of this thread, here is an article about different (instrinsic) stream objects you might use, depending on your version of Delphi.
http://delphi.about.com/od/vclusing/l/aa110803a.htm

If Geert isn't available to help you with his posted (rllibby) solution, you might try one of the other methods.  Another participating expert may also be able to help you.  What specific help do you need on "how to implement the CharStream"?

By the way, Russel posted an IMPRESSIVE body of Delphi code when he was active.  I could almost write a book solely based on his posted code.  If you have the time, do a search on solution comments by rllibby.
0

Featured Post

Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

  • 12
  • 5
  • 4
  • +4
Tackle projects and never again get stuck behind a technical roadblock.
Join Now