Asp.Net Fileupload - The process cannot access the file 'filename' because it is being used by another process.

ASP.NET C#
Visual Studio 2017

Description:

I am creating a form where the user clicks an image (image has runat='server' attribute), and then uploads a new image that replaces the old image.
( Note: that when the user uploads a new image it is renamed to exactly the same name as the old image. )

I'm using a hidden Fileupload control for uploading images.

Problem:
Initially ( when I upload the first version of the image ) everything works fine, but once I start uploading new images ( to replace the old ones ) I start getting the error:

System.IO.IOException: The process cannot access the file 'filename' because it is being used by another process.

Code Sample:
I do some validation checks and then something like:

try
{
    fileUpload.PostedFile.SaveAs(path);
}
catch (Exception ex)
{
     lblMessage.InnerText = ex.ToString();
}
finally
{
    fileUpload.Dispose();
}

Select allOpen in new window

I tried adding this to see if it works but I still have the same issue:

private void SaveFile(FileUpload fileupload, string path)
        {
            fileupload.PostedFile.SaveAs(path);
            const int numberOfRetries = 10;
            const int delayOnRetry = 1000;

            for (int i = 1; i <= numberOfRetries; ++i)
            {
                try
                {
                    fileupload.PostedFile.SaveAs(path);
                    break; 
                }
                catch (IOException e) when (i <= numberOfRetries)
                {
                    Thread.Sleep(delayOnRetry);
                }
            }
        }

Select allOpen in new window

I've also tried toggling images to see if that helps.  For example: If the current image is xxxxa.jpg then save as xxxxb.jpg.  But I still get the same error message.

Final Notes:
I am able to upload images but it seems to be random, sometimes it works and sometimes ( often ) I get the error message.  I've tested the same code on my desktop and everything works great but when I put it on the live server I start seeing the error message.