Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win

x
?
Solved

How to edit (or split) a Text File -> close to 8 GB?

Posted on 2006-11-24
8
Medium Priority
?
1,112 Views
Last Modified: 2013-11-13
Hi,

I have to process the contents of a text file programmatically in VB.NET ... any code I've yet tried worked up to a file size of apx. 999 MB without crashing the server ... as soon as the file size reaches more the 1 GB the systems runs for hours and then just goes to sleep ... so, what I "always" did was to split the files (manually!) into sizes that could afterwards easily be processed ... but ... now I have to deal with a file size close to 8 GB and to tell, there's no way of splitting, processing or anything else ... I'm able to cut some chunks but after the 15th or some more chunk I have a "memory leak" message ... which seems to be astonishing since I have 4 GB physical memory on that server and another 10 GB virtual memory ...

... to shorten this a little ... there's no way to handle this with programming, nor with using the "editor" ... and not even a way to perform this with UltraEdit32 ... so, what to do?

... well, this is my question ... ;-)) ... what to do?


Best regards,
Raisor
0
Comment
Question by:Ralf Klatt
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 2
  • 2
8 Comments
 
LVL 41

Expert Comment

by:HonorGod
ID: 18009840
 What kind of editing do you need to do?  Can you use sed? http://www.cornerstonemag.com/sed/
0
 
LVL 41

Expert Comment

by:HonorGod
ID: 18009864
 How well can you describe what needs to be done?
  If not sed, how about perl?  http://www.perl.org/about.html
  You can retrieve a free implementation of it from http://www.activestate.com/store/productdetail.aspx?prdGuid=81fbce82-6bd5-49bc-a915-08d58c2648ca
0
 
LVL 15

Author Comment

by:Ralf Klatt
ID: 18009920
Hi,

Thanks for your suggestion!

To be truth ... I'm not at all into UNIX and Perl ... it's not that I'm not having had a lot of experiences with both ... it's just that I’d prefer a way that offers me an entry to a .NET kind of thing ... the needs are “infact” that I have to import a 8 GB text file with a terrible specification into a SQL Server database ... I don't mind about the interface ... it's the file size that bothers!


Best regards,
Raisor
0
Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

 
LVL 8

Accepted Solution

by:
YoungBonzi earned 2000 total points
ID: 18009997
0
 
LVL 15

Author Comment

by:Ralf Klatt
ID: 18010015
Hi,

This looks very promising on first sight ... it's currently running on the 8 GB file ... will let you know the result!


Thanks a lot so far!
Best regards,
Raisor
0
 
LVL 15

Author Comment

by:Ralf Klatt
ID: 18010042
Hi,

It does not only look promising ... it's just perfect!

I've first used the "largest" option ... after only eight minutes the first part was done ... UltraEdit32 even had a problem to open it (665.000 MB) ... I've then killed all related processes and restarted with the 1.140 KB option ... and checked some of the outcomes ... files are not cut in a "structured" way ... but who cares! ... ;-)) ... the files are readable and the files are still in a code page that kept all included languages (Arabic, Russian, Chinese and all other languages!) ... and I can even open them in the "editor" ...


Excellent answer, excellent hint!
Thanks a lot!!!
Best regards,
Raisor
0
 
LVL 8

Expert Comment

by:YoungBonzi
ID: 18010075
Very nice. I'm going to download this myself.
0
 
LVL 15

Author Comment

by:Ralf Klatt
ID: 18010147
Hi,

... if you're dealing with large files you surely should ... ;)


Best regards,
Raisor
0

Featured Post

Enroll in October's Free Course of the Month

Do you work with and analyze data? Enroll in October's Course of the Month for 7+ hours of SQL training, allowing you to quickly and efficiently store or retrieve data. It's free for Premium Members, Team Accounts, and Qualified Experts!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Today, the web development industry is booming, and many people consider it to be their vocation. The question you may be asking yourself is – how do I become a web developer?
Q&A with Course Creator, Mark Lassoff, on the importance of HTML5 in the career of a modern-day developer.
Simple Linear Regression
Introduction to Processes
Suggested Courses

598 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question