c# decompress a really large file

hi

i am using c# 3.5
i have a file that is just under 1GB.
 I am trying to unzip it (it has been Gzipped) and write the bytes to a new file.

I am getting an outofmemory exception when i try and do this.

here is a sample of my code (which works fine on small files):

using System.IO;
using System.IO.Compression;
using System.IO.Packaging;

//##############################################
//step 1
//first i read the bytes out the zipped file (this bit works fine)
//###############################################

byte[] zippedBytes;
using (var fileStream = new FileStream(fileName, FileMode.Open, FileAccess.Read))
{
       using (var binaryReader = new BinaryReader(fileStream))
       {
          var numBytes = new FileInfo(fileName).Length;
          zippedBytes = binaryReader.ReadBytes((int)numBytes);
       }
}

//################################################################
//step 2
//then i try and decompress to a new memorystream
//and i get repeated "Exception of type 'System.OutOfMemoryException' was thrown."    
//in the following code...        
//################################################################

using (var gZipStream = new GZipStream(new MemoryStream(zippedBytes), CompressionMode.Decompress))
{
         const int size = 4096;
         var buffer = new byte[size];
         using (var memoryStream = new MemoryStream())
             {
               var count = 0;
                do
                {
                  count = gZipStream.Read(buffer, 0, size);
                  if (count > 0)
                    {
                      memoryStream.Write(buffer, 0, count);
                        }
                    } while (count > 0);

                    return memoryStream.ToArray();
               }
}


can anyone help please?!

thankyou for your time
LVL 1
MrKevorkianAsked:
Who is Participating?
 
Jaime OlivaresSoftware ArchitectCommented:
MrKevorkian,
This is not a good approach, you are reading the entire file contents into memory, which overloads the memory. What will happen if you have a file of 4 GB? Also, consider that you are creating a decompressing buffer which is of the size of compressed file. It would be better to have a dynamic buffer.

The standard technique is to read in chunks and write to new file, like in the example shown in MSDN:
http://msdn.microsoft.com/en-us/library/system.io.compression.gzipstream(v=VS.90).aspx

If you are working with .net 4.0, then the solution is quiet straightforward with the new CopyTo method:


var ms = new MemoryStream(zippedBytes), 

using (GZipStream decompress = new GZipStream(new FileStream(fileName, FileMode.Open, FileAccess.Read), CompressionMode.Decompress))
                    {
decompress.CopyTo(ms);
return ms.ToArray();

// Notice you will have at least two buffers here: the 'ms' stream and the array generated before returning.
// To avoid this, you can decompress to a temporary filestream and read the contents into an array with ReadAllBytes

Open in new window

0
 
vusovCommented:
Please try to use Ionic.Zip library. I've attached the GZipHelper sample of using.
Compress.zip
0
 
MrKevorkianAuthor Commented:
hi sorry for the delay!  im just looking at these two answers now. thanks
0
 
MrKevorkianAuthor Commented:
hi i used the msdn example (well actually one of the comments at the bottom of the article)

heres my final code

 public void DecompressAndWriteToFile(FileDetails fileDetails)
        {
            const int bufferSize = 4096;
            var compressedfileInfo = new FileInfo(fileDetails.FullPath);

            using (var compressedFileStream = compressedfileInfo.OpenRead())
            {
                var newDecompressedPath = Path.Combine(directoryProvider.MyDropDirectory, fileDetails.UncompressedName);
                using (var decompressedFileSream = File.Create(newDecompressedPath))
                {
                    using (var gZipStream = new GZipStream(compressedFileStream, CompressionMode.Decompress))
                    {
                        var buffer = new byte[bufferSize];
                        int numRead;
                        while ((numRead = gZipStream.Read(buffer, 0, buffer.Length)) != 0)
                        {
                            decompressedFileSream.Write(buffer, 0, numRead);
                        }
                    }
                }
            }
        }

it works great. thanks very much
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.