[x]
Posted via EE Mobile

Search, ask, and monitor your questions on the go with EE Mobile. Visit Experts Exchange from your mobile device and never be out of touch again.

Question
[x]
Attachment Details
[x]
The Solution Rating System

With so many solutions, how can you tell which solutions are most likely to help you and which ones are not? To provide you with a tool to use, we rate our solutions based on various elements that most accurately determine if a solution is a quality solution. To explain what factors affect the solution rating, here are the elements we take into consideration when formulating our solution rating.

  • The Grade of the Solution
  • The Zone Rank of the Expert Providing the Solution
  • The Number of Author and Expert Comments
  • The Number of Experts Contributing
  • The Feedback of the Community

Your Input Matters
Because of the way the system is set up, the most important variable in this equation is you. As a member of Experts Exchange, you are able to cast your vote on the quality of the solutions in regard to how complete, accurate, helpful and easy to understand each solution is. When you provide your feedback, each rating is adjusted accordingly. So, if you see a solution that has a poor rating that you think is a good solution, let us know by rating it. As you do, the rating will be adjusted and will become more accurate for other members of our site.

If you have any suggestions that you would like to make for our rating system, please ask a question in the Suggestions Zone of Community Support.

Thank you!

5.0

KbmMemTable too slow

Asked by Jaymol in Delphi Programming

Tags: delphi kbmmemtable loadfromdataset performance

Hi.

I've got an app that needs to load data from a dataset into a KbmMemTable, which is then streamed over a tcp connection and lots of other stuff done with it.  The application is working fine, except for performance.  There's a couple of issues, such as encryption and decryption, that I'm currently working on as they slow the process down, but loading the data from a dataset into a KbmMemTable is simply too slow.

I've included code below (dfm & pas) for a delphi test app, and an sql script that will create the test table I've used in my testing.

It's a messy table, on purpose, but it's representative of the type of data I'll be looking at importing regularly into a memory table.

The initial query to get all data from the test table takes anywhere up to 0.5 seconds.  That's obviously fine.  The problem is that KbmMemTable.LoadDataSet takes about 3.5 seconds with the test data, which is about a third of the size of the real tables.  It takes 12-13 seconds for the real data to be imported, and this simply is too slow.

Any suggestions would be greatly appreciated, but this is a very old and long-standing system, so suggestions that relate to the database are no good to me.  I need to get a dataset into a memory stream as quickly as possible.

The demo app will expect the test table to exist on a local instance of MASTER, so change that if required.

Thanks in advance.

John.
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:
41:
42:
43:
44:
45:
46:
47:
48:
49:
50:
51:
52:
53:
54:
55:
56:
57:
58:
59:
60:
61:
62:
63:
64:
65:
66:
67:
68:
69:
70:
71:
72:
73:
74:
75:
76:
77:
78:
79:
80:
81:
82:
83:
84:
85:
86:
87:
88:
89:
90:
91:
92:
93:
94:
95:
96:
97:
98:
99:
100:
101:
102:
103:
104:
105:
106:
107:
108:
109:
110:
111:
112:
113:
114:
115:
116:
117:
118:
119:
120:
121:
122:
123:
124:
125:
126:
127:
128:
129:
130:
131:
132:
133:
134:
135:
136:
137:
138:
139:
140:
141:
142:
143:
144:
145:
146:
147:
148:
149:
150:
151:
152:
153:
154:
155:
156:
// Start of DFM --------------------------------------------------------------------------------------------------
object Form1: TForm1
  Left = 312
  Top = 103
  BorderIcons = [biSystemMenu]
  BorderStyle = bsSingle
  Caption = 'Form1'
  ClientHeight = 377
  ClientWidth = 505
  Color = clBtnFace
  Font.Charset = DEFAULT_CHARSET
  Font.Color = clWindowText
  Font.Height = -11
  Font.Name = 'MS Sans Serif'
  Font.Style = []
  OldCreateOrder = False
  PixelsPerInch = 96
  TextHeight = 13
  object Button1: TButton
    Left = 16
    Top = 16
    Width = 161
    Height = 25
    Caption = 'Test '#39'LoadFromDataSet'#39
    TabOrder = 0
    OnClick = Button1Click
  end
  object ListBox1: TListBox
    Left = 192
    Top = 16
    Width = 297
    Height = 345
    ItemHeight = 13
    TabOrder = 1
  end
end
// Start of PAS --------------------------------------------------------------------------------------------------
unit kbmst01;
 
interface
 
uses
  Windows, Messages, SysUtils, Classes, Graphics, Controls, Forms, Dialogs, StdCtrls,
  ADODB, DB, KbmMemTable, ActiveX;
 
type
  TForm1 = class(TForm)
    Button1: TButton;
    ListBox1: TListBox;
    procedure Button1Click(Sender: TObject);
  private
    procedure UpdateStatus(Value: String);
    { Private declarations }
  public
    { Public declarations }
  end;
 
var
  Form1: TForm1;
 
implementation
 
{$R *.DFM}
 
procedure TForm1.Button1Click(Sender: TObject);
var
	sQuery,
  sAlias: String;
  adoDataSet: TADODataSet;
  kbmMemTable: TkbmMemTable;
begin
 
	sQuery := 'SELECT * FROM DUMMY_TABLE';
 
  try
    CoInitialize(nil);
 
    adoDataSet := TADODataSet.Create(Self);
    adoDataSet.ConnectionString :=
    	'Provider=SQLOLEDB.1;' +
      'Integrated Security=SSPI;' +
      'Persist Security Info=False;' +
      'Initial Catalog=MASTER;' +
      'Data Source=localhost';
 
    UpdateStatus('PreQuery');
    adoDataSet.CursorType := ctOpenForwardOnly;
    adoDataSet.CommandText := sQuery;
    adoDataSet.Open;
    UpdateStatus('PostQuery');
 
    kbmMemTable := TkbmMemTable.Create(Self);
 
    UpdateStatus('PreLoad');
    kbmMemTable.LoadFromDataSet(adoDataSet, [mtcpoStructure]);
    UpdateStatus('PostLoad');
    UpdateStatus(Format('RecordCount (%d)', [kbmMemTable.RecordCount]));
 
    FreeAndNil(adoDataSet);
    FreeAndNil(kbmMemTable);
 
    CoUninitialize;
 
  except
  	on E:Exception do
    begin
    	CoUninitialize;
    	raise Exception.Create(E.Message);
    end;
  end;
end;
 
procedure TForm1.UpdateStatus(Value: String);
begin
	ListBox1.Items.BeginUpdate;
	ListBox1.Items.Add(Format('[%s] %s', [FormatDateTime('hh:mm:ss:zzz', Now), Value]));
  ListBox1.TopIndex := ListBox1.Items.Count - 1;
	ListBox1.Items.EndUpdate;
end;
 
end.
// Start of SQL --------------------------------------------------------------------------------------------------
SET NOCOUNT ON
 
IF EXISTS (SELECT 1 FROM SYSOBJECTS WHERE [NAME] = 'DUMMY_TABLE' AND TYPE = 'U')
	DROP TABLE DUMMY_TABLE
 
DECLARE @CREATE_TABLE VARCHAR(8000),
	@COUNTER INT
 
SET @CREATE_TABLE = 'CREATE TABLE DUMMY_TABLE (RECORD_ID INT,'
SET @COUNTER = 0
 
WHILE @COUNTER < 100
BEGIN
	SET @COUNTER = @COUNTER + 1
	SET @CREATE_TABLE = @CREATE_TABLE + 'FIELD' + CAST(@COUNTER AS VARCHAR) + ' VARCHAR(36) DEFAULT NEWID()'
	IF @COUNTER < 100
		SET @CREATE_TABLE = @CREATE_TABLE + ','
	ELSE
		SET @CREATE_TABLE = @CREATE_TABLE + ')'
END
 
EXEC(@CREATE_TABLE)
 
SET @COUNTER = 0
 
WHILE @COUNTER < 2000
BEGIN
	SET @COUNTER = @COUNTER + 1
	INSERT INTO DUMMY_TABLE (RECORD_ID) VALUES (@COUNTER)
END
 
SELECT * FROM DUMMY_TABLE
 
SET NOCOUNT OFF
[+][-]06/15/09 02:36 PM, ID: 24633272Expert Comment

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]06/16/09 01:15 AM, ID: 24635803Author Comment

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 30-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]06/16/09 01:25 AM, ID: 24635865Author Comment

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 30-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]06/16/09 01:30 AM, ID: 24635897Expert Comment

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]06/16/09 01:51 AM, ID: 24636025Author Comment

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 30-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]06/16/09 01:52 AM, ID: 24636038Expert Comment

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]06/16/09 01:57 AM, ID: 24636070Author Comment

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 30-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]06/16/09 02:30 AM, ID: 24636226Author Comment

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 30-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]06/16/09 02:37 AM, ID: 24636247Expert Comment

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]06/16/09 02:50 AM, ID: 24636315Author Comment

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 30-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]06/16/09 03:02 AM, ID: 24636387Author Comment

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 30-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]06/16/09 04:30 AM, ID: 24636910Expert Comment

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]06/16/09 04:33 AM, ID: 24636926Author Comment

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 30-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]06/17/09 01:40 PM, ID: 24652363Expert Comment

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]06/23/09 02:45 AM, ID: 24690318Author Comment

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 30-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]06/25/09 03:27 AM, ID: 24709854Accepted Solution

View this solution now by starting your 30-day free trial. Setting up your free trial is quick, easy, and secure. We will return you to this solution, unlocked, when you're done.

About this solution

Zone: Delphi Programming
Tags: delphi kbmmemtable loadfromdataset performance
Sign Up Now!
Solution Provided By: Jaymol
Participating Experts: 2
Solution Grade: A
 
[+][-]06/25/09 03:29 AM, ID: 24709869Expert Comment

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]06/25/09 06:11 AM, ID: 24710987Author Comment

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 30-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]06/25/09 06:28 AM, ID: 24711146Expert Comment

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]06/26/09 04:32 AM, ID: 24719945Author Comment

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 30-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]06/26/09 05:24 AM, ID: 24720222Expert Comment

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]06/26/09 06:39 AM, ID: 24720797Expert Comment

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]06/28/09 12:04 PM, ID: 24732060Author Comment

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 30-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]06/28/09 12:05 PM, ID: 24732063Author Comment

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 30-day free trial to view this Author Comment or ask the Experts your question.

 
 
Loading Advertisement...
20091111-EE-VQP-89 - Hierarchy / EE_QW_3_20080625