Avatar of ppittle
ppittle
Flag for United States of America asked on

C# Write File larger than 64 MB to network (UNC)

I have a basic File Upload web site that allows users to upload files.  The files are then written out to two separate servers.  I've been using code similar to this:

byte[] byteArrayContainingUploadedFileBinary;

File.WriteAllBytes("\\server1\FileShare\", byteArrayContainingUploadedFileBinary);
File.WriteAllBytes("\\server2\FileShare\", byteArrayContainingUploadedFileBinary);

This works fine if the user uploads a file less than 64 MB.  If the file is larger than 64 MB I receive an IOException.  Microsoft indicates this is "by design", as this error is described in the method documentation.  http://msdn.microsoft.com/en-us/library/system.io.file.writeallbytes.aspx

I also tried using FileStream, but that doesn't work.  Interestingly enough, File.WriteAllBytes uses a FileStream internally.  Even though the msdn documentation for FileStream doesn't explicitly state the same 64 MB limit when writing to a UNC location, the FileStream class still fails with files over 64mb.

Does anyone know of a good way to write a file larger than 64MB to a UNC location?

Thanks,

PJ

Sorry about the low points for this question.  I only have 120 left.
C#Windows Server 2003ASP.NET

Avatar of undefined
Last Comment
ppittle

8/22/2022 - Mon
ASKER CERTIFIED SOLUTION
algen

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
GET A PERSONALIZED SOLUTION
Ask your own question & get feedback from real experts
Find out why thousands trust the EE community with their toughest problems.
SOLUTION
johnbouma

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
GET A PERSONALIZED SOLUTION
Ask your own question & get feedback from real experts
Find out why thousands trust the EE community with their toughest problems.
algen

In response to johnbouma, their solution does work around the issue. The reason for the issue to be present in the first place though, is mainly because it's inefficient and you should use a proper technique, such as asynchronous chunk-based movement. Using synchronous and blocking methods that allocate a lot of data into RAM causes things to be placed on the large object heap and in the end makes for a higher degree of fragmentation in the managed memory pool. To avoid this, you can use the buffering technique as I explained before. You reduce memory consumption and fragmentation but use a bit more of the CPU with stack overhead from the callbacks. RAM is more expensive than CPU in most cases, as far as resource management goes. You can really use either method, but it depends on your experience and whether or not your application needs to perform well.

Additionally, using the asynchronous methods will stop your application from hanging when doing large reads or writes.
ppittle

ASKER
Al & Josh,

Thank you for the quick responses.  I'm going to try both approaches and get back to you.

PJ
algen

Great! Let us know how they work for you and if you discover anything new. I'm sure both of us would be happy to continue to assist you, should you require it.
All of life is about relationships, and EE has made a viirtual community a real community. It lifts everyone's boat
William Peck
ppittle

ASKER
Al & Josh,

Thank you both again for your input.  I have reviewed and implemented both of your responses and both do indeed work.  I attached the code I use to used in case anyone needs it in the future.

I found using an Async technique did on occasion provide far superior performance (writing a 140 MB file to a network share in under 50ms).  However, I also found the performance to be inconsistent (writing a 140 mb file to a network share could take up to 900ms).  The Temp write and move technique didn't hit the same peak performance as the async technique (250-500 ms to write a 140 mb file to a network share).

In the end I will probably end up using the "temp write and move" technique because of the simplicity of the code.  I agree with Al that this will consume more resources, but in my particular application only a small percentage of users will be uploading files and they will already be expecting large page load times as they have to wait to upload the file.

Again, I'm sorry I couldn't award more points, I ran out of them.
<%@ Page Language="C#" AutoEventWireup="true" CodeBehind="WriteLargeFile.aspx.cs"
    Inherits="Phil_SandBox.DocPubSandBox.WriteLargeFile" %>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head runat="server">
    <title></title>
</head>
<body>
    <form id="form1" runat="server">
    <div>
        <table>
            <tr>
                <td>
                    File (Relative to Server):
                </td>
                <td>
                    <asp:TextBox ID="txtFileInputLocation" runat="server" />
                </td>
            </tr>
            <tr>
                <td>
                    File Write location:
                </td>
                <td>
                    <asp:TextBox ID="txtFileOutputLocation" runat="server" />
                </td>
            </tr>
        </table>
        <br />
        <asp:Button ID="btnAsyncWrite" Text="Async Write" runat="server" OnClick="btnAsyncWrite_Click" />
        &nbsp;&nbsp;&nbsp;
        <asp:Button ID="btnTempWriteAndCopy" Text="Temp Write and Copy" runat="server" OnClick="btnTempWriteAndCopy_Click" />
        <br />
        <br />
        Results:
        <asp:Label ID="lblResults" runat="server" />
    </div>
    </form>
</body>
</html>

<%-- Code Behind --%>
using System;
using System.Collections.Generic;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.IO;
using System.Threading;
using System.Security.Principal;

namespace Phil_SandBox.DocPubSandBox
{
    public partial class WriteLargeFile : System.Web.UI.Page
    {
        protected void Page_Load(object sender, EventArgs e)
        {
            lblResults.Text = string.Empty;
        }

        protected void btnAsyncWrite_Click(object sender, EventArgs e)
        {
            writeFile(new AsyncFileWriter(false));
        }       

        protected void btnTempWriteAndCopy_Click(object sender, EventArgs e)
        {
            writeFile(new TempWriteAndCopy());
        }

        private void writeFile(ILargeFileWriter fileWriter)
        {  
            string fileReadPath = txtFileInputLocation.Text;
            string fileWritePath = txtFileOutputLocation.Text;

            byte[] bytesOfFile;

            #region Read in File
            try
            {
                DateTime fileReadStartTime = DateTime.Now;
                bytesOfFile = File.ReadAllBytes(fileReadPath);

                lblResults.Text += "Read file at [" + fileReadPath + "] in [" +
                    DateTime.Now.Subtract(fileReadStartTime).Milliseconds + "] ms. " +
                    "File Size:  " + bytesOfFile.Length + " bytes <br /><br />";
            }
            catch (Exception exc)
            {
                lblResults.Text += "Failed Reading File [" + fileReadPath + "]: <br />" +
                    exc.Message + "<br /> " + exc.Source + "<br />" + exc.StackTrace;
                return;
            }
            #endregion

            #region Write File
            try
            {
                DateTime fireWriteStartTime = DateTime.Now;
                fileWriter.WriteFile(fileWritePath, bytesOfFile);

                lblResults.Text += "Using [" + fileWriter.GetType().FullName + "] <br/>" + 
                    "Wrote file at [" + fileWritePath + "] in [" +
                   DateTime.Now.Subtract(fireWriteStartTime).Milliseconds + "] ms. " +
                   "<hr />";
            }
            catch (Exception exc)
            {
                lblResults.Text += "Failed Writing File [" + fileWritePath + "]: <br />" +
                    exc.Message + "<br /> " + exc.Source + "<br />" + exc.StackTrace;
                return;
            }
            #endregion

        }
    }   

    public class AsyncFileWriter : ILargeFileWriter
    {
        public AsyncFileWriter() { }
        public AsyncFileWriter(bool openFileStreamWithUseAsyncFlag)
        {
            OpenFileStreamWithUseAsyncFlag = openFileStreamWithUseAsyncFlag;
        }

        public bool OpenFileStreamWithUseAsyncFlag { get; set; }

        private bool IsWriting { get; set; }        

        /// <summary>
        /// Method based on example at 
        /// http://msdn.microsoft.com/en-us/library/7db28s3c.aspx
        /// </summary>        
        public void WriteFile(string path, byte[] bytesOfFile)
        {
            FileStream fs = new FileStream(
                path,
                FileMode.Create,
                FileAccess.ReadWrite,
                FileShare.None,
                4096,
                OpenFileStreamWithUseAsyncFlag);            

            fs.BeginWrite(bytesOfFile, 0, bytesOfFile.Length,
                new AsyncCallback(EndWriteCallback),
                fs);

            this.IsWriting = true;

            while (IsWriting)
            {
                //wait for fs.BeginWrite to finish
                Thread.Sleep(500);
            }
        }

        private void EndWriteCallback(IAsyncResult asyncResult)
        {
            FileStream fs = (FileStream)asyncResult.AsyncState;
            fs.EndWrite(asyncResult);

            this.IsWriting = false;
        }

    }

    public class TempWriteAndCopy : ILargeFileWriter
    {
        public void WriteFile(string path, byte[] bytesOfFile)
        {
            string temfilePath = Path.GetTempPath() + Path.GetFileName(path);
            File.WriteAllBytes(temfilePath, bytesOfFile);
            File.Move(temfilePath, path);
        }     
    }
    public interface ILargeFileWriter
    {
        void WriteFile(string path, byte[] bytesOfFile);
    }
}

Open in new window

ppittle

ASKER
See my last post in this thread.