Solved

Faster string concatenation?

Posted on 2014-04-05
30
429 Views
Last Modified: 2014-11-24
Dear Experts, my app manipulates a lot of text strings and by using a line profiler program, I see that a good portion of time is on a couple of key lines in my program that do the string concatenation operation:

str1 := str1 + str2;

I already use FastMM4 in my project and my understanding is that it does help to speed up string operations... but is there any component or unit that would do a more efficient/faster string concatenation?

Thanks!
    Shawn

P.S: For that matter, the string "Delete" procedure seems to be a bit of a bottleneck for me too. As does the "StringReplace" function.
0
Comment
Question by:shawn857
  • 12
  • 5
  • 4
  • +4
30 Comments
 
LVL 32

Expert Comment

by:Stefan Hoffmann
Comment Utility
There is no general advice.

It strongly depends on your problem and you solution. The key maybe to avoid StringReplace().
0
 

Author Comment

by:shawn857
Comment Utility
Thanks. I thought there was a package called "Jedi" or something, that had faster string routines... no? Would that work?

Thanks
   Shawn
0
 
LVL 45

Assisted Solution

by:aikimark
aikimark earned 166 total points
Comment Utility
There are many great routines in the JEDI project.  Here are links to their pages, both project and download.
http://sourceforge.net/projects/jvcl/
http://www.delphi-jedi.org/
http://jvcl.delphi-jedi.org/

=============
Are you trying to concatenate ASCII strings or wide strings/unicode?
What version of Delphi are you using?
How many string concatenation operations are you doing?
How long are your final resulting strings?

It will help if you read this article from one of the BorCon2004 sessions.
http://conferences.embarcadero.com/article/32120

I have found the string builder approach to be the best.  This is a native class in the .Net framework and I've written my own in different environments.  Basically, each string you add to the object is added to a list.  Only when you want the entire list does the actual concatenation occur.
In the BorCon session, they mentioned Halvard's class.  Here are the links to his pas files.
https://code.google.com/p/omnithreadlibrary/source/browse/trunk/src/HVStringBuilder.pas
https://code.google.com/p/omnithreadlibrary/source/browse/trunk/src/HVStringData.pas

Delphi2009 and later have a TStringBuilder class that is worth testing before you start adding other libraries to your project.

Also, you might open up a stream object, replacing your concatenation operations with stream write operations.
0
 
LVL 32

Expert Comment

by:Stefan Hoffmann
Comment Utility
They have faster string routines, but only for certain cases like character based replaces.. you need to compare them in your actual use-case.
0
 

Author Comment

by:shawn857
Comment Utility
Hi guys, I'm using Delphi 7 and what I'm doing is basically parsing a CSV file character-by-character. The statements that seem to take the most time are these:

HoldString := Holdstring + c;

... where HoldString is a string variable that accumulates the string that gets added to it character-by-character by the "c" variable (declared as Char).


Thanks!
   Shawn
0
 
LVL 25

Expert Comment

by:Sinisa Vuk
Comment Utility
Agree with others - there is no general approach. Best is to use buffer memory to read portion of file in and analize it, keep integer indexes of start and current position in this buffer. Current index is last good character (valid for you) - when you detect field separator - then do copy of all chars between start and current index - so no char by char manipulation.
0
 

Author Comment

by:shawn857
Comment Utility
Thanks Sinisa, you're probably right, but this would mean major major surgery on my program to create a whole new parsing engine. For now, I was just looking (hoping?) for a faster way to do some basic string manipulations - particularly the concatenate operation. So I guess there's really nothing faster than the standard old '+' operator then eh?

Cheers
   Shawn
0
 
LVL 32

Expert Comment

by:Stefan Hoffmann
Comment Utility
@Sinisa is right. I had also this kind of processing in mind as I've read your "HoldString := Holdstring + c;"..

btw, when your parsing engine is well structured, then reengineering is not a huge or complex task.

And for the +: nope.
0
 
LVL 45

Expert Comment

by:aikimark
Comment Utility
Why are you parsing a CSV file, character-by-character?!?

Please answer the questions I posted, http:#a39981695
0
 
LVL 25

Expert Comment

by:Sinisa Vuk
Comment Utility
Made example in your another EE question.
In short I use global buffer, track start pointer of a String (PChar), look for delimiter (;) and replace it with #0 (zero) - this way I'm get zero terminated string (value) without concatenating.
0
 
LVL 19

Expert Comment

by:MerijnB
Comment Utility
As long as you use non unicode strings you can use QStrings, which is old but is fast with string manipulation, you can still download it here: http://www.torry.net/vcl/vcltools/text/adqstrings.zip

It won't help with concatenation, but it does have a fast StringReplace() alternative.
0
 

Author Comment

by:shawn857
Comment Utility
Thanks for all the responses guys...


Aikimark, you asked:

(1) Are you trying to concatenate ASCII strings or wide strings/unicode?

Just simple ASCII strings.


(2) What version of Delphi are you using?

D7


(3) How many string concatenation operations are you doing?

Well, as I mentioned, I'm looping through each line of data parsing for fields so I'm concatenating each "good" character on to the end of a string, and once I hit another delimiter, I write out the string. So I'm doing a *lot* of these little concatenations... millions.


(4) How long are your final resulting strings?

Not too long - just as long as one "field" in a CSV record might be. Maybe on average 15-20 characters.

Thanks
   Shawn
0
 
LVL 45

Expert Comment

by:aikimark
Comment Utility
Can we safely assume that you are doing line-at-a-time reading of the file and not character-by-character reading?

Not knowing what constitutes a 'good' character, I would suggest that you use a reasonably long pchar variable, initialized to spaces (or some non-null character).  As you loop through your record, to a direct value assignment of the next available position in the pchar array of the good character you have just found.
0
 

Author Comment

by:shawn857
Comment Utility
Yes aikimark - I'm reading line-by-line from my file on disk.

I really should be thinking "bigger"... it must be lack of sleep, but here I am wasting time trying to optimize string manipulation routines, when what I really should be doing is finding the fastest method to parse a CSV file - and it's certainly not by concatenating characters one-at-a-time to the end of another string!
   I'd like to open a new question addressing just that, and I'll close out this one and award points. Sorry....

Thanks
    Shawn
0
How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

 
LVL 36

Expert Comment

by:Geert Gruwez
Comment Utility
you need a combo of unit faststrings and one of russel's comments
unfortunately i haven't found russel's comment yet

but ... Peter Morris was the one who wrote FastStrings while he was stuckindoors.
this babbling of mine probably doesn't make sense at the moment, but hang on ...
just imagine if someone had the same problem as you did and decided to take the proverbial bull by the horns and create a unit for very fast string manipulation
the naming of that unit would be obvious ... yes ... FastStrings
I knew that unit existed ... due to have come across it on this website the previous decade ... ugh, don't be resented at the code being that old yet  ... it was originally written in Delphi 7 ...

without further ado and without going into details about how to use the search box on this site with the words "Delphi FastStrings Peter Morris", which likely will give you a few hits with my comments too, here is a link for the updated XE2 version

https://code.google.com/p/ryulib4delphi/source/browse/trunk/XE2/FastStrings.pas
I hope the newer version works on your XE2, it took me a long while (at least 20 seconds) to find that newer version.

That unit off course, is just the very basic string handling, and off course, you're looking for that itsy pitsy tad more, namely concatenation of strings
SO, again without further ado, here is another unit which uses the earlier unit
https://code.google.com/p/ryulib4delphi/source/browse/trunk/XE2/FastStringFuncs.pas

After checking that secondary unit, you will probably wonder where in all that code do i find what i'm looking for, and why would i use assembler code ? The first answer would be, I have to admit: "it's not in that code", but you might now remember that the quote "unfortunately i haven't found russel's comment yet" ...
i already hear all those minds going "aah, now it's coming ..."

but alas, i haven't found that comment yet
0
 
LVL 36

Assisted Solution

by:Geert Gruwez
Geert Gruwez earned 334 total points
Comment Utility
hey, i'm back again.
no i haven't found russel's comment yet.
sorry, very disappointing, i know. but i found an appetizer
there are a few experts on this site who aren't "genuis" yet, at least on this site, but ...
actually they are legend level, by my standards, but i'm not a genius

here is one routine to "move" characters from 1 string to another
you could call it concatenation if you "move" characters from 1 string to the end of another string
it's written in asm, but a true legend, he's got 662000 points at the moment
here is a link to the assembler concatenation comment:
http://www.experts-exchange.com/Programming/Languages/Pascal/Delphi/Q_20106634.html

yes, i know it's a decade old, but i stated that in my earlier comment

and no, i still haven't found russel's comment yet
0
 
LVL 36

Accepted Solution

by:
Geert Gruwez earned 334 total points
Comment Utility
aha. good ol russel, i knew he would come around in the end and give me his comments
here it is

http://www.experts-exchange.com/Programming/Languages/Pascal/Delphi/Q_20615170.html

you might be wondering where "russel" comes from ?
0
 
LVL 25

Expert Comment

by:Sinisa Vuk
Comment Utility
where do you store parsed values from csv? how do you work whit them (after you've got values)?
0
 

Author Comment

by:shawn857
Comment Utility
Geert: thank you for all this, particularly Russel's concat method as show here:

http://www.experts-exchange.com/Programming/Languages/Pascal/Delphi/Q_20615170.html

I made a small project to try it, but I don't know how to really use it - I get an access violation error. Is it as simple as declaring a TCharStream variable:

var
   newCharStream : TCharStream

... and then just appending to it like this? :

newCharStream.Append(SomeChar);



Sinisa: you asked "where do you store parsed values from csv?". As I parse the record, I store each field to a TStringList. Then I do some checks on each of these StringList values, modify some of them if needed, then immediately write them out to an output file and start again reading the next record from the input file.

Thanks
    Shawn
0
 

Author Comment

by:shawn857
Comment Utility
H Geert, sorry to bug you on a weekend but if you have a second could you kindly quickly show how to utilize Russel's TCharStream method? I tried and get an access violation error... :-(

Thanks!
   Shawn
0
 
LVL 36

Expert Comment

by:Geert Gruwez
Comment Utility
just read your mail ...
after a weekend of masonry and just before going to bed as i'm rather burnt out now
it'll be at least a week before i'm back on EE
too busy ... i'll post when i get back
0
 

Author Comment

by:shawn857
Comment Utility
of course Geert... get some rest!  :-)

Cheers
   Shawn
0
 

Author Comment

by:shawn857
Comment Utility
Hi Geert, are you back in business?  :-)

Cheers
   Shawn
0
 

Author Comment

by:shawn857
Comment Utility
Hi Geert, are you still with me...?

Cheers
   Shawn
0
 

Author Comment

by:shawn857
Comment Utility
Can anyone help me on this please??

Thanks
    Shawn
0
 
LVL 100

Expert Comment

by:mlmcc
Comment Utility
I've requested that this question be deleted for the following reason:

The question has either no comments or not enough useful information to be called an "answer".
0
 
LVL 45

Expert Comment

by:aikimark
Comment Utility
I think several comments would qualify as solutions.  In order of their "goodness":
Geert: http:#a39985215
aikimark: http:#a39981695
Geert: http:#a39985209
0
 

Author Comment

by:shawn857
Comment Utility
Sorry guys... I was just waiting for Geert (or somebody) to reply with an example on how to implement the CharStream append method, as I had mentioned here:

"Geert: thank you for all this, particularly Russel's concat method as show here:

http://www.experts-exchange.com/Programming/Languages/Pascal/Delphi/Q_20615170.html"

Thanks
   Shawn
0
 
LVL 45

Expert Comment

by:aikimark
Comment Utility
Thanks for responding, Shawn.

For future readers of this thread, here is an article about different (instrinsic) stream objects you might use, depending on your version of Delphi.
http://delphi.about.com/od/vclusing/l/aa110803a.htm

If Geert isn't available to help you with his posted (rllibby) solution, you might try one of the other methods.  Another participating expert may also be able to help you.  What specific help do you need on "how to implement the CharStream"?

By the way, Russel posted an IMPRESSIVE body of Delphi code when he was active.  I could almost write a book solely based on his posted code.  If you have the time, do a search on solution comments by rllibby.
0

Featured Post

Highfive Gives IT Their Time Back

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Suggested Solutions

Title # Comments Views Activity
Intraweb submit form as a POST request 4 227
Best Firemonkey component pack 1 73
Delphi selector screen 2 57
How to debug For loops? 3 33
Introduction Raise your hands if you were as upset with FireMonkey as I was when I discovered that there was no TListview.  I use TListView in almost all of my applications I've written, and I was not going to compromise by resorting to TStringGrid…
In my programming career I have only very rarely run into situations where operator overloading would be of any use in my work.  Normally those situations involved math with either overly large numbers (hundreds of thousands of digits or accuracy re…
This video discusses moving either the default database or any database to a new volume.
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

8 Experts available now in Live!

Get 1:1 Help Now