Solved

Data Storing.

Posted on 2006-10-31
8
182 Views
Last Modified: 2010-04-05
At the moment, my server uses Paradox, problem is it's quite slow and there's people logging in and out all the time and with each login it hits almost 30% CPU time on a 1gig machine.. so what i want to do is dump it and for something a bit more simplistic..

Ideally, I'd love to be able hash a username, seek to the hashed value on the hard disk (Could be a 2 gig or more file i don't care about disk space, just the speed) And then write out the data as a Record, I used to use this method for a really old program but i came to the problem where

cat and tac would produce the same value and then the record would overwrite it..

the record cosists of  

Record...
Username: String[40];
Password: String[40];
Age: Integer;
Location: String[255];

/etc etc...

File of TRecord ;

Then i'd blockwrite the record out to hard disk, but as i say, once i hashed the username (the rest of the details might not be known so all i have is the username) and i would often overwrite old records...

So i'm looking for a Solution, it can be really simplisitic as it gets i just need to store data fast and retrieve it, i'm open to using DLL's i'm not interested in SQL fast and simple...

Any Suggestions?
0
Comment
Question by:tobjectpascal
8 Comments
 

Expert Comment

by:PAG_Promax
ID: 17847876
You're not going to get much speed doing it this way.  In fact, I'd bet with index tables, Paradox would kick that flat file method you mentioned's butt.  I know you don't wan to use SQL, but I'd strongly recommend it.  There's plenty of embedded SQL solutions for Delphi which will be lightning fast.  Some of them are free of charges and royalties as well.  If you're interested at all, I can give you a few things to start looking at.  But I'd strongly recommend throwing the flat file idea out.  Especially in multi-user environments :)
0
 
LVL 4

Author Comment

by:tobjectpascal
ID: 17847882
Yeah i'm open to anything, I just don't like supplying 12 meg of runtimes just so i can store data to the HD. i just need a way of getting data in and out as fast as possible.
0
 
LVL 1

Accepted Solution

by:
cerdal earned 125 total points
ID: 17856652
I'm moving from Paradox to DBISAM as it compiles into the delphi program so you can just distribute the exe file and it works without anything else.
It's fast (if you define the right indexes), reliable, and can index free text if required.
You can use it with Paradox-like tables and prior/next/gotokey etc, or with SQL.
It's pretty simple to modify your Paradox-based program to DBISAM, and if you need to improve speed you can do that later on, little by little.
You need to pay for a license to develop with it, but not for each run-time you install or sell.
It also provides a server program you can install on a separate machine, good for up to 5 users without extra cost. That gives you an inherently safer multi-user system, as only one program (the DBISAM server) accesses the database at a time, rather than all the copies of your program using file-sharing over a network.
If you're talking about a web-server that's getting too slow, perhaps you'd find it worthwhile setting up a separate PC as a database server and doing all the web stuff on one PC, accessing the database on the other by a client-server setup. DBISAM can do this with only very few changes in your program.
0
Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

 
LVL 45

Assisted Solution

by:aikimark
aikimark earned 125 total points
ID: 17877435
@tobjectpascal

<<Any Suggestions?>>
First some questions:
1. What kind of "server" is this?  (a more complete description of the environment and place your Paradox database serves within it, please)

Example:
This is a Delphi application running on your users' desktops, talking over a (LAN) network connection to a file server that has one or more Paradox tables.

2. What happens when a user logs in?

3. What kind of data is stored in this/these Paradox table(s)?

4. What methods and layers are involved in a database call?

5. Is your file of record merely a log of activity or is this data used for other purposes?

6. When and by whom is this data retrieved?

<<i'm open to using DLL's i'm not interested in SQL fast and simple...>>
7. Does this mean that you are willing to abandon the BDE but not willing to use SQL?  Please ellaborate.

================================
8. I see two major potential bottlenecks.
  * concurrency and integrity of writes (lack of locking and transaction mechanisms)
  * read/write performance

9. What is the volume of data you are working with?

10. What is the activity rate of your application upon the database?

11. You are asking us for specific solutions to a very poorly defined problem.
0
 
LVL 4

Author Comment

by:tobjectpascal
ID: 17886675
the data just contains their username, their location and just typical sign up data to get access to the server... all i really need to do is store a record and get it out again as fast as possible, if that's  using a hashing technique writing data to disk or using a database like access / paradox, it does not matter too much, i don't need to restructure data

i will not be planning on adding more fields or removing them, it's just as basica as you get, I could simply iterate through  through the records 1. is username whoever? 2 is username whoever... but i'd prefer to hash a username

craig becomes 34234 which then

Seek(File,34234);
BlockWrite(File,RecordData,SizeOf(RecordData),bc);

when i the username craig is accessed

Seek(File,34234);
BlockRead(file,RecordData

What i originally did was turn each Char into it's Ascii value

A = 65
B = 66

I added them all up, but problem with that was CAT gives the same value as TAC.

for I=1 to length(tmp) do
    gennum:=gennum+((ord(tmp[I])-30)*I);

This way i got a better value, but still, I would end up with Certain usernames giving the same value as others, so what i really could do with is to improve on this routine, and if i do come across a record already in use..

Make another file to store all the hashed values that give the same result

sdfsdjkfh = drstrtnme in which case I make a File hashed4324234.dat

and then store each record with that value and then iterated through the list looking for the username..

That's my idea, i just need a decent hashing routine, obviously the file size is limited to 2gig, so a routine to produce 3242345345345435 is not going to work..

Chars Userd 0..9 'a..z' 'A..Z'  along with $ . @ chars, the rest will not be allowed.

Maybe that'll explain what i mean better.
0
 
LVL 45

Expert Comment

by:aikimark
ID: 17887029
@tobjectpascal

** It would help us if you'd answer more of my questions, especially related to the volume of data and frequency of R/W access.

12. Since it seems you are storing and locating character data, your hashing idea has some merit.  However, it would be easier to store/retrieve a hash value from within a Paradox database with appropriate indexes rather than trying to create your own database.

13. You need to decide what you want to do in order to uniquely distinguish people with the same name.

14. (fixed length) Numeric data certainly compares much faster than (variable length) character data.

15. The hashing algorithm you have tried fails to properly translate the characters into a valid numeric value and maintain a collating sequence.  Rod Stephens has a good chapter on this (Key Combination and Compression) in his "Ready-to-Run Delphi Algorithms" book in the Sorting section.

16. There are several well established hashing algorithms that you might employ to create a numeric hash of the LastnameFirstnamePassword (whatever) data.  If you need a larger hash, you might combine them into multiple integer columns.

17. Look at TurboPower's TPLockBox on http://Sourceforge.net for Delphi hashing examples.

17. Does this data need to persist on some hard drive or is it only a runtime requirement?
0
 
LVL 1

Expert Comment

by:Computer101
ID: 21133090
Forced accept.

Computer101
EE Admin
0

Featured Post

Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

Join & Write a Comment

A lot of questions regard threads in Delphi.   One of the more specific questions is how to show progress of the thread.   Updating a progressbar from inside a thread is a mistake. A solution to this would be to send a synchronized message to the…
Introduction I have seen many questions in this Delphi topic area where queries in threads are needed or suggested. I know bumped into a similar need. This article will address some of the concepts when dealing with a multithreaded delphi database…
It is a freely distributed piece of software for such tasks as photo retouching, image composition and image authoring. It works on many operating systems, in many languages.
This video shows how to remove a single email address from the Outlook 2010 Auto Suggestion memory. NOTE: For Outlook 2016 and 2013 perform the exact same steps. Open a new email: Click the New email button in Outlook. Start typing the address: …

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now