Solved

How to edit (or split) a Text File -> close to 8 GB?

Posted on 2006-11-24
8
1,104 Views
Last Modified: 2013-11-13
Hi,

I have to process the contents of a text file programmatically in VB.NET ... any code I've yet tried worked up to a file size of apx. 999 MB without crashing the server ... as soon as the file size reaches more the 1 GB the systems runs for hours and then just goes to sleep ... so, what I "always" did was to split the files (manually!) into sizes that could afterwards easily be processed ... but ... now I have to deal with a file size close to 8 GB and to tell, there's no way of splitting, processing or anything else ... I'm able to cut some chunks but after the 15th or some more chunk I have a "memory leak" message ... which seems to be astonishing since I have 4 GB physical memory on that server and another 10 GB virtual memory ...

... to shorten this a little ... there's no way to handle this with programming, nor with using the "editor" ... and not even a way to perform this with UltraEdit32 ... so, what to do?

... well, this is my question ... ;-)) ... what to do?


Best regards,
Raisor
0
Comment
Question by:Raisor
  • 4
  • 2
  • 2
8 Comments
 
LVL 41

Expert Comment

by:HonorGod
Comment Utility
 What kind of editing do you need to do?  Can you use sed? http://www.cornerstonemag.com/sed/
0
 
LVL 41

Expert Comment

by:HonorGod
Comment Utility
 How well can you describe what needs to be done?
  If not sed, how about perl?  http://www.perl.org/about.html
  You can retrieve a free implementation of it from http://www.activestate.com/store/productdetail.aspx?prdGuid=81fbce82-6bd5-49bc-a915-08d58c2648ca
0
 
LVL 15

Author Comment

by:Raisor
Comment Utility
Hi,

Thanks for your suggestion!

To be truth ... I'm not at all into UNIX and Perl ... it's not that I'm not having had a lot of experiences with both ... it's just that I’d prefer a way that offers me an entry to a .NET kind of thing ... the needs are “infact” that I have to import a 8 GB text file with a terrible specification into a SQL Server database ... I don't mind about the interface ... it's the file size that bothers!


Best regards,
Raisor
0
 
LVL 8

Accepted Solution

by:
YoungBonzi earned 500 total points
Comment Utility
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 
LVL 15

Author Comment

by:Raisor
Comment Utility
Hi,

This looks very promising on first sight ... it's currently running on the 8 GB file ... will let you know the result!


Thanks a lot so far!
Best regards,
Raisor
0
 
LVL 15

Author Comment

by:Raisor
Comment Utility
Hi,

It does not only look promising ... it's just perfect!

I've first used the "largest" option ... after only eight minutes the first part was done ... UltraEdit32 even had a problem to open it (665.000 MB) ... I've then killed all related processes and restarted with the 1.140 KB option ... and checked some of the outcomes ... files are not cut in a "structured" way ... but who cares! ... ;-)) ... the files are readable and the files are still in a code page that kept all included languages (Arabic, Russian, Chinese and all other languages!) ... and I can even open them in the "editor" ...


Excellent answer, excellent hint!
Thanks a lot!!!
Best regards,
Raisor
0
 
LVL 8

Expert Comment

by:YoungBonzi
Comment Utility
Very nice. I'm going to download this myself.
0
 
LVL 15

Author Comment

by:Raisor
Comment Utility
Hi,

... if you're dealing with large files you surely should ... ;)


Best regards,
Raisor
0

Featured Post

Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

Join & Write a Comment

RIA (Rich Internet Application) tools are interactive internet applications which have many of the characteristics of desktop applications. The RIA tools typically deliver output either by the way of a site-specific browser or via browser plug-in. T…
This is an explanation of a simple data model to help parse a JSON feed
Viewers will learn how to properly install Eclipse with the necessary JDK, and will take a look at an introductory Java program. Download Eclipse installation zip file: Extract files from zip file: Download and install JDK 8: Open Eclipse and …
In this fourth video of the Xpdf series, we discuss and demonstrate the PDFinfo utility, which retrieves the contents of a PDF's Info Dictionary, as well as some other information, including the page count. We show how to isolate the page count in a…

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now