Comparing Large Files in Excel and SQL Server

AlHal2
AlHal2 used Ask the Experts™
on
Suppose I have 2 large files.  If I want to compare them using a MATCH function in Excel it could take hours.
If I load them into a SQL Database and do an inner join it can take minutes or seconds.

I want to compare some large files in Visual Studio 2010 (C#) and was wondering if I could compare them in minutes rather than hours without actually uploading them to a SQL Database.
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Most Valuable Expert 2012
Top Expert 2014
Commented:
Depends on what you're comparing. A database engine line SQL Server is optimized for these sorts of things. A programming language - any language - is not. C# (or any .NET language) has methods you can use to compare data/records, but those will always be slower than a database.

If you want the most performance, then load them into a database table and use Joins and such as you mention. You have a bit less control over the process, but it's easily the fastest.

If you need finer control, then moving this into a datatable (or something along those lines) would give you that. It would be slower, of course.
AndyAinscowFreelance programmer / Consultant
Commented:
You also need to think just what 'compare' actually means.
consider the following two fictional file contents:
file1
A
B
C
D

and file2
A

B
C
D

Note the second has one blank line in it, otherwise identical contents BUT only the first line matches between the two.
Pawan KumarDatabase Expert
Awarded 2016
Top Expert 2016
Commented:
>>.I want to compare some large files in Visual Studio 2010 (C#) and was wondering if I could compare them in minutes rather than hours without actually uploading them to a SQL Database.

Always better to upload the files in SQL ( DATABASE ) and then compare. With this you can compare N number of columns and rows. Any other tool/language will take time. Data Truncate and conversion issues with also occur.

Author

Commented:
Thanks.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial