Solved: C# Write File larger than 64 MB to network (UNC)

algen

2/18/2010

In response to johnbouma, their solution does work around the issue. The reason for the issue to be present in the first place though, is mainly because it's inefficient and you should use a proper technique, such as asynchronous chunk-based movement. Using synchronous and blocking methods that allocate a lot of data into RAM causes things to be placed on the large object heap and in the end makes for a higher degree of fragmentation in the managed memory pool. To avoid this, you can use the buffering technique as I explained before. You reduce memory consumption and fragmentation but use a bit more of the CPU with stack overhead from the callbacks. RAM is more expensive than CPU in most cases, as far as resource management goes. You can really use either method, but it depends on your experience and whether or not your application needs to perform well.

Additionally, using the asynchronous methods will stop your application from hanging when doing large reads or writes.

ppittle

2/18/2010

ASKER

Al & Josh,

Thank you for the quick responses. I'm going to try both approaches and get back to you.

PJ

algen

2/18/2010

Great! Let us know how they work for you and if you discover anything new. I'm sure both of us would be happy to continue to assist you, should you require it.

ppittle

2/19/2010

ASKER

Al & Josh,

Thank you both again for your input. I have reviewed and implemented both of your responses and both do indeed work. I attached the code I use to used in case anyone needs it in the future.

I found using an Async technique did on occasion provide far superior performance (writing a 140 MB file to a network share in under 50ms). However, I also found the performance to be inconsistent (writing a 140 mb file to a network share could take up to 900ms). The Temp write and move technique didn't hit the same peak performance as the async technique (250-500 ms to write a 140 mb file to a network share).

In the end I will probably end up using the "temp write and move" technique because of the simplicity of the code. I agree with Al that this will consume more resources, but in my particular application only a small percentage of users will be uploading files and they will already be expecting large page load times as they have to wait to upload the file.

Again, I'm sorry I couldn't award more points, I ran out of them.

<%@ Page Language="C#" AutoEventWireup="true" CodeBehind="WriteLargeFile.aspx.cs"
    Inherits="Phil_SandBox.DocPubSandBox.WriteLargeFile" %>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head runat="server">
    <title></title>
</head>
<body>
    <form id="form1" runat="server">
    <div>
        <table>
            <tr>
                <td>
                    File (Relative to Server):
                </td>
                <td>
                    <asp:TextBox ID="txtFileInputLocation" runat="server" />
                </td>
            </tr>
            <tr>
                <td>
                    File Write location:
                </td>
                <td>
                    <asp:TextBox ID="txtFileOutputLocation" runat="server" />
                </td>
            </tr>
        </table>
        <br />
        <asp:Button ID="btnAsyncWrite" Text="Async Write" runat="server" OnClick="btnAsyncWrite_Click" />
        &nbsp;&nbsp;&nbsp;
        <asp:Button ID="btnTempWriteAndCopy" Text="Temp Write and Copy" runat="server" OnClick="btnTempWriteAndCopy_Click" />
        <br />
        <br />
        Results:
        <asp:Label ID="lblResults" runat="server" />
    </div>
    </form>
</body>
</html>

<%-- Code Behind --%>
using System;
using System.Collections.Generic;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.IO;
using System.Threading;
using System.Security.Principal;

namespace Phil_SandBox.DocPubSandBox
{
    public partial class WriteLargeFile : System.Web.UI.Page
    {
        protected void Page_Load(object sender, EventArgs e)
        {
            lblResults.Text = string.Empty;
        }

        protected void btnAsyncWrite_Click(object sender, EventArgs e)
        {
            writeFile(new AsyncFileWriter(false));
        }       

        protected void btnTempWriteAndCopy_Click(object sender, EventArgs e)
        {
            writeFile(new TempWriteAndCopy());
        }

        private void writeFile(ILargeFileWriter fileWriter)
        {  
            string fileReadPath = txtFileInputLocation.Text;
            string fileWritePath = txtFileOutputLocation.Text;

            byte[] bytesOfFile;

            #region Read in File
            try
            {
                DateTime fileReadStartTime = DateTime.Now;
                bytesOfFile = File.ReadAllBytes(fileReadPath);

                lblResults.Text += "Read file at [" + fileReadPath + "] in [" +
                    DateTime.Now.Subtract(fileReadStartTime).Milliseconds + "] ms. " +
                    "File Size:  " + bytesOfFile.Length + " bytes <br /><br />";
            }
            catch (Exception exc)
            {
                lblResults.Text += "Failed Reading File [" + fileReadPath + "]: <br />" +
                    exc.Message + "<br /> " + exc.Source + "<br />" + exc.StackTrace;
                return;
            }
            #endregion

            #region Write File
            try
            {
                DateTime fireWriteStartTime = DateTime.Now;
                fileWriter.WriteFile(fileWritePath, bytesOfFile);

                lblResults.Text += "Using [" + fileWriter.GetType().FullName + "] <br/>" + 
                    "Wrote file at [" + fileWritePath + "] in [" +
                   DateTime.Now.Subtract(fireWriteStartTime).Milliseconds + "] ms. " +
                   "<hr />";
            }
            catch (Exception exc)
            {
                lblResults.Text += "Failed Writing File [" + fileWritePath + "]: <br />" +
                    exc.Message + "<br /> " + exc.Source + "<br />" + exc.StackTrace;
                return;
            }
            #endregion

        }
    }   

    public class AsyncFileWriter : ILargeFileWriter
    {
        public AsyncFileWriter() { }
        public AsyncFileWriter(bool openFileStreamWithUseAsyncFlag)
        {
            OpenFileStreamWithUseAsyncFlag = openFileStreamWithUseAsyncFlag;
        }

        public bool OpenFileStreamWithUseAsyncFlag { get; set; }

        private bool IsWriting { get; set; }        

        /// <summary>
        /// Method based on example at 
        /// http://msdn.microsoft.com/en-us/library/7db28s3c.aspx
        /// </summary>        
        public void WriteFile(string path, byte[] bytesOfFile)
        {
            FileStream fs = new FileStream(
                path,
                FileMode.Create,
                FileAccess.ReadWrite,
                FileShare.None,
                4096,
                OpenFileStreamWithUseAsyncFlag);            

            fs.BeginWrite(bytesOfFile, 0, bytesOfFile.Length,
                new AsyncCallback(EndWriteCallback),
                fs);

            this.IsWriting = true;

            while (IsWriting)
            {
                //wait for fs.BeginWrite to finish
                Thread.Sleep(500);
            }
        }

        private void EndWriteCallback(IAsyncResult asyncResult)
        {
            FileStream fs = (FileStream)asyncResult.AsyncState;
            fs.EndWrite(asyncResult);

            this.IsWriting = false;
        }

    }

    public class TempWriteAndCopy : ILargeFileWriter
    {
        public void WriteFile(string path, byte[] bytesOfFile)
        {
            string temfilePath = Path.GetTempPath() + Path.GetFileName(path);
            File.WriteAllBytes(temfilePath, bytesOfFile);
            File.Move(temfilePath, path);
        }     
    }
    public interface ILargeFileWriter
    {
        void WriteFile(string path, byte[] bytesOfFile);
    }
}

Open in new window

ppittle

2/19/2010

ASKER

See my last post in this thread.