Link to home
Create AccountLog in
Avatar of ppittle
ppittleFlag for United States of America

asked on

C# Write File larger than 64 MB to network (UNC)

I have a basic File Upload web site that allows users to upload files.  The files are then written out to two separate servers.  I've been using code similar to this:

byte[] byteArrayContainingUploadedFileBinary;

File.WriteAllBytes("\\server1\FileShare\", byteArrayContainingUploadedFileBinary);
File.WriteAllBytes("\\server2\FileShare\", byteArrayContainingUploadedFileBinary);

This works fine if the user uploads a file less than 64 MB.  If the file is larger than 64 MB I receive an IOException.  Microsoft indicates this is "by design", as this error is described in the method documentation.  http://msdn.microsoft.com/en-us/library/system.io.file.writeallbytes.aspx

I also tried using FileStream, but that doesn't work.  Interestingly enough, File.WriteAllBytes uses a FileStream internally.  Even though the msdn documentation for FileStream doesn't explicitly state the same 64 MB limit when writing to a UNC location, the FileStream class still fails with files over 64mb.

Does anyone know of a good way to write a file larger than 64MB to a UNC location?

Thanks,

PJ

Sorry about the low points for this question.  I only have 120 left.
ASKER CERTIFIED SOLUTION
Avatar of algen
algen
Flag of United States of America image

Link to home
membership
Create an account to see this answer
Signing up is free. No credit card required.
Create Account
SOLUTION
Link to home
membership
Create an account to see this answer
Signing up is free. No credit card required.
Create Account
In response to johnbouma, their solution does work around the issue. The reason for the issue to be present in the first place though, is mainly because it's inefficient and you should use a proper technique, such as asynchronous chunk-based movement. Using synchronous and blocking methods that allocate a lot of data into RAM causes things to be placed on the large object heap and in the end makes for a higher degree of fragmentation in the managed memory pool. To avoid this, you can use the buffering technique as I explained before. You reduce memory consumption and fragmentation but use a bit more of the CPU with stack overhead from the callbacks. RAM is more expensive than CPU in most cases, as far as resource management goes. You can really use either method, but it depends on your experience and whether or not your application needs to perform well.

Additionally, using the asynchronous methods will stop your application from hanging when doing large reads or writes.
Avatar of ppittle

ASKER

Al & Josh,

Thank you for the quick responses.  I'm going to try both approaches and get back to you.

PJ
Great! Let us know how they work for you and if you discover anything new. I'm sure both of us would be happy to continue to assist you, should you require it.
Avatar of ppittle

ASKER

Al & Josh,

Thank you both again for your input.  I have reviewed and implemented both of your responses and both do indeed work.  I attached the code I use to used in case anyone needs it in the future.

I found using an Async technique did on occasion provide far superior performance (writing a 140 MB file to a network share in under 50ms).  However, I also found the performance to be inconsistent (writing a 140 mb file to a network share could take up to 900ms).  The Temp write and move technique didn't hit the same peak performance as the async technique (250-500 ms to write a 140 mb file to a network share).

In the end I will probably end up using the "temp write and move" technique because of the simplicity of the code.  I agree with Al that this will consume more resources, but in my particular application only a small percentage of users will be uploading files and they will already be expecting large page load times as they have to wait to upload the file.

Again, I'm sorry I couldn't award more points, I ran out of them.
<%@ Page Language="C#" AutoEventWireup="true" CodeBehind="WriteLargeFile.aspx.cs"
    Inherits="Phil_SandBox.DocPubSandBox.WriteLargeFile" %>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head runat="server">
    <title></title>
</head>
<body>
    <form id="form1" runat="server">
    <div>
        <table>
            <tr>
                <td>
                    File (Relative to Server):
                </td>
                <td>
                    <asp:TextBox ID="txtFileInputLocation" runat="server" />
                </td>
            </tr>
            <tr>
                <td>
                    File Write location:
                </td>
                <td>
                    <asp:TextBox ID="txtFileOutputLocation" runat="server" />
                </td>
            </tr>
        </table>
        <br />
        <asp:Button ID="btnAsyncWrite" Text="Async Write" runat="server" OnClick="btnAsyncWrite_Click" />
        &nbsp;&nbsp;&nbsp;
        <asp:Button ID="btnTempWriteAndCopy" Text="Temp Write and Copy" runat="server" OnClick="btnTempWriteAndCopy_Click" />
        <br />
        <br />
        Results:
        <asp:Label ID="lblResults" runat="server" />
    </div>
    </form>
</body>
</html>

<%-- Code Behind --%>
using System;
using System.Collections.Generic;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.IO;
using System.Threading;
using System.Security.Principal;

namespace Phil_SandBox.DocPubSandBox
{
    public partial class WriteLargeFile : System.Web.UI.Page
    {
        protected void Page_Load(object sender, EventArgs e)
        {
            lblResults.Text = string.Empty;
        }

        protected void btnAsyncWrite_Click(object sender, EventArgs e)
        {
            writeFile(new AsyncFileWriter(false));
        }       

        protected void btnTempWriteAndCopy_Click(object sender, EventArgs e)
        {
            writeFile(new TempWriteAndCopy());
        }

        private void writeFile(ILargeFileWriter fileWriter)
        {  
            string fileReadPath = txtFileInputLocation.Text;
            string fileWritePath = txtFileOutputLocation.Text;

            byte[] bytesOfFile;

            #region Read in File
            try
            {
                DateTime fileReadStartTime = DateTime.Now;
                bytesOfFile = File.ReadAllBytes(fileReadPath);

                lblResults.Text += "Read file at [" + fileReadPath + "] in [" +
                    DateTime.Now.Subtract(fileReadStartTime).Milliseconds + "] ms. " +
                    "File Size:  " + bytesOfFile.Length + " bytes <br /><br />";
            }
            catch (Exception exc)
            {
                lblResults.Text += "Failed Reading File [" + fileReadPath + "]: <br />" +
                    exc.Message + "<br /> " + exc.Source + "<br />" + exc.StackTrace;
                return;
            }
            #endregion

            #region Write File
            try
            {
                DateTime fireWriteStartTime = DateTime.Now;
                fileWriter.WriteFile(fileWritePath, bytesOfFile);

                lblResults.Text += "Using [" + fileWriter.GetType().FullName + "] <br/>" + 
                    "Wrote file at [" + fileWritePath + "] in [" +
                   DateTime.Now.Subtract(fireWriteStartTime).Milliseconds + "] ms. " +
                   "<hr />";
            }
            catch (Exception exc)
            {
                lblResults.Text += "Failed Writing File [" + fileWritePath + "]: <br />" +
                    exc.Message + "<br /> " + exc.Source + "<br />" + exc.StackTrace;
                return;
            }
            #endregion

        }
    }   

    public class AsyncFileWriter : ILargeFileWriter
    {
        public AsyncFileWriter() { }
        public AsyncFileWriter(bool openFileStreamWithUseAsyncFlag)
        {
            OpenFileStreamWithUseAsyncFlag = openFileStreamWithUseAsyncFlag;
        }

        public bool OpenFileStreamWithUseAsyncFlag { get; set; }

        private bool IsWriting { get; set; }        

        /// <summary>
        /// Method based on example at 
        /// http://msdn.microsoft.com/en-us/library/7db28s3c.aspx
        /// </summary>        
        public void WriteFile(string path, byte[] bytesOfFile)
        {
            FileStream fs = new FileStream(
                path,
                FileMode.Create,
                FileAccess.ReadWrite,
                FileShare.None,
                4096,
                OpenFileStreamWithUseAsyncFlag);            

            fs.BeginWrite(bytesOfFile, 0, bytesOfFile.Length,
                new AsyncCallback(EndWriteCallback),
                fs);

            this.IsWriting = true;

            while (IsWriting)
            {
                //wait for fs.BeginWrite to finish
                Thread.Sleep(500);
            }
        }

        private void EndWriteCallback(IAsyncResult asyncResult)
        {
            FileStream fs = (FileStream)asyncResult.AsyncState;
            fs.EndWrite(asyncResult);

            this.IsWriting = false;
        }

    }

    public class TempWriteAndCopy : ILargeFileWriter
    {
        public void WriteFile(string path, byte[] bytesOfFile)
        {
            string temfilePath = Path.GetTempPath() + Path.GetFileName(path);
            File.WriteAllBytes(temfilePath, bytesOfFile);
            File.Move(temfilePath, path);
        }     
    }
    public interface ILargeFileWriter
    {
        void WriteFile(string path, byte[] bytesOfFile);
    }
}

Open in new window

Avatar of ppittle

ASKER

See my last post in this thread.