Solved

Is there a size limit for .csv files?

Posted on 2014-02-26
8
21,730 Views
Last Modified: 2014-02-27
We have an application that is exporting 1,254,236 records to a .csv file.  The file size is 295,788.  Excel brings up an error msg that it was unable to load the full file.  I have a batch job that successfully splits the file into smaller ones which works ok.  However, when I try to reconcile the counts, it doesn't match the number exported by the application.

Is there a size limit for .csv files?

Thanks for your help.
je
0
Comment
Question by:aeolianje
8 Comments
 
LVL 92

Expert Comment

by:John Hurst
ID: 39890186
I do not think you are hitting a size limit, but a record limit. For Excel 2010, the number of records is limited to 1048576 rows and you exceed that.
0
 
LVL 35

Expert Comment

by:Kimputer
ID: 39890190
There's no limit for the .csv file itself (other than the max file size of the OS), it's Excel who has to read the data. And therefore the limits of Excel come into play

Excel 2003 and before: max 65,536 rows
Excel 2007 and above: max 1,048,576 rows

I therefore think you might be using 2003 or earlier and the split files are over 65,536 rows, and you lose the amount over that, times the number of files you imported.
0
 
LVL 3

Expert Comment

by:costanos
ID: 39890192
No, there is no hard size limit on .csv files.. The limit in any scenario would be the file system / hdd size.  However, the application used to open these files MIGHT give you a problem.  ie opening a 10gb .csv file in excel.
0
 
LVL 18

Expert Comment

by:Jerry Miller
ID: 39890350
If you want to open csv files of a large size you can use a different program to open them. I use Notepad++. It is a free open source program that is very easy to use.

http://notepad-plus-plus.org/
0
Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

 
LVL 25

Accepted Solution

by:
Tony Giangreco earned 500 total points
ID: 39890374
The CSV file has no limitation other than free disk space on your drive or an O/S file size limitation.

Excel has a record limit which you are hitting. If you split those files and try importing them into one Excel spreadsheet, you will still have the same problem. I suggest splitting the CSV file into multiple files that are under the Excel record limit providing you room to insert or add records to that spreadsheet without being close to the limit again.

I ran into that a few years ago. Hope this helps!
0
 
LVL 92

Expert Comment

by:John Hurst
ID: 39890429
Using an arbitrary number of characters per record of 256, the file would be 321Mb in size. This is large for most systems, but if you need a text editor that will handle this size of file, I use UltraEdit. It has no size limit.

Otherwise, with Excel, you are limited by the Excel limit (1048576) as I noted much further back.
0
 
LVL 25

Expert Comment

by:Tony Giangreco
ID: 39890450
FYI, I've been using Ultra Edit for 15 years. It's the best text editor I've ever seen.
0
 
LVL 25

Expert Comment

by:Tony Giangreco
ID: 39891694
Glad I could help.
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
MS Office applications not loading on Windows 8i 10 67
Unable to boot to USB 5 91
AS400 Logging Print device 2 74
DNS Dynamic Update 7 63
Introduction How to create multiboot configuration with XP\Vista and Windows 7 on it? And most important question - how to do this correctly so not to have any kind of nightmares we get when system gets screwed? First of all one should realize t…
As the title indicates, I have done this before. It chills me everytime I update the OS on my phone, (http://www.experts-exchange.com/articles/18084/Upgrading-to-Android-5-0-Lollipop.html) because one time I did this and I essentially had a bricked …
This is used to tweak the memory usage for your computer, it is used for servers more so than workstations but just be careful editing registry settings as it may cause irreversible results. I hold no responsibility for anything you do to the regist…
Hi friends,  in this video  I'll show you how new windows 10 user can learn the using of windows 10. Thank you.

863 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

24 Experts available now in Live!

Get 1:1 Help Now