Link to home
Start Free TrialLog in
Avatar of Murray Brown
Murray BrownFlag for United Kingdom of Great Britain and Northern Ireland

asked on

VB.net Select/Copy very large tables

Hi

I have a VB.net Windows forms application where I have to count the records in SQL tables of 65 to 100 million records. I also have to pull data into a DataGridView.

Is there a class that I can use to make this faster? I have used the SQLBulkCopy class to upload data. I am looking
for something similar to pull data faster
Avatar of Guy Hengel [angelIII / a3]
Guy Hengel [angelIII / a3]
Flag of Luxembourg image

if you want to display 100M rows on the client, there will be no fast thing.
unless you want to "page" (and obviously filter) the data, so the end-user can reduce the number of records that need to be transferred from the server over the whole networkwork infrastructure and to be put into the end-users computer RAM. I see many times where a developer forgets that resources are limited, especially on normal end-user computers.
the lesser rows you bring to the user, the better (for EVERYbody)

you will need to clarify exactly what are the needs for this form, and how to reduce the data transfer between the machines.
just consider 100M rows, each how having 1KB size each (which can be considered a short row, actually).
total size of all those records would be around 100GB. how much RAM does your machine have? how much disk space does your machine have?

next point is that anything above 100 records (at a time) for a normal end-user is unusable, he will just be scrolling "all day".

note: there are exceptions, of course, but those are very rare.
Avatar of Murray Brown

ASKER

Thanks Guy. What about counting records
ASKER CERTIFIED SOLUTION
Avatar of Guy Hengel [angelIII / a3]
Guy Hengel [angelIII / a3]
Flag of Luxembourg image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Thanks very much
Great answer!