I was wondering if there was a better method of creating a SQL search parameters than what I have used.
In my application written in C# connected to an SQLite database, I have provided a single column datagrid for users to enter values, my original intention was for small to medium scale queries for quick results. As I am sure everyone has found out that a developers expectations and what a user does is often very different.
The program currently opens a connection to the DB and loops through the values checking for a scalar value, if no scalar, it reports the selected value doesn't exist and continues to the end of the loop and closes the connection, alerting the user to the erroneous value/s.
It then opens a new connection to the DB where it creates an SQL query string in a list, with a new item added for each non-blank entry in the datagrid , because the values exist in two separate tables (for unfortunate legacy reasons) Views are created and destroyed in the statement.
A sample query for 8 items would look like the following:
CREATE VIEW view1 AS SELECT * FROM table1, table2;
CREATE VIEW view2 AS SELECT * FROM table3, table4;
CREATE VIEW collection AS SELECT * FROM view2
SELECT * FROM view1;
SELECT * FROM collection WHERE collection.Column1 LIKE "%value1%"
or collection.Column1 like "%value2%"
or collection.Column1 like "%value3%"
or collection.Column1 like "%value4%"
or collection.Column1 like "%value5%"
or collection.Column1 like "%value6%"
or collection.Column1 like "%value7%"
or collection.Column1 like "%value8%"
DROP VIEW view1;
DROP VIEW view2;
DROP VIEW collection;
Whilst the process time for this example in Navicat is 0.304 seconds, when users enter hundreds or even thousands of items, the program becomes slow, unresponsive and it turns out that the SQLite engine can only handle a max of 1000 requests per query.
For each value in the datagrid a "or collection like value.." is created in a list which is joined together using using a list.toarray ie: string s1 = string.Join(string.empty, list.ToArray()).
Is there a way to make this part of the program more robust and efficient?
Why do you use "SELECT * FROM table1, table2"? or "FROM table3, table4"
Do you know that creates a "Cartesian product" (all rows of table1 multiplied by all rows of table2)?
Why create a views and then drop them?
I think you really need to provide sample data (per source table, not your views) and then an expected result.
btw: If you have simplified your real sql I would suggest just giving us the real deal.