We help IT Professionals succeed at work.

Application error on Windows Server 2008 64-bit SP2

I have two Windows Server 2008 64 bit SP2 terminal servers where users run an application written in C++ (the application is made by our customer, for whom we house the solution).
The users are running the application from a file server (Windows Server 2008 64 bit SP1), where they map a folder to a drive letter.
They used to run this application on 2003 terminal servers and a 2003 file server, where it worked without error.
The error we're seeing happens from 1-10 times each day, the application suddenly terminates.
According to the customer, they have many of their customers running this application on 2008 servers without any problems (although I doubt they're running the application from a separate file server).
The application is fairly light, just a 10MB exe-file. The data is stored in databases on a database server.

We have a firewall between the terminal servers and the file server, the firewall rules are exactly the same. We have set the sessions to be open for eight hours without closing.

The file server is used by many other customers, and ourselves, and no one else is experiencing similar problems.

No NLB is set up.

Trend Antivirus is running on the servers, with the firewall and proxy services disabled.

Windows Firewall is disabled.

DEP is turned off.

I have tried to modify the application compatibility level to Server 2003 SP1 by adding the registry setting below to all the users when they log on:
 'HKEY_CURRENT_USER\Software\Microsoft\WindowsNT\CurrentVersion\AppCompatFlags\Layers]"
w:\\<path>\\<executable name>"="WINSRV03SP1"
I'm not certain this has any effect.

GPOs are run when users log on, limiting their access to resources, control panel and other things. The file server is added to Trusted Sites with a GPO setting.

The servers run on VMware, and have 8GB of RAM and 2CPUs, no resource problems.

I get this event in the application log (event id 1000):
Faulting application <executable name>, version 1.0.0.5, time stamp 0x4ba7811c, faulting module ntdll.dll, version 6.0.6002.18005, time stamp 0x49e03824, exception code 0xc0000006, fault offset 0x000444c3, process id 0x2ab8, application start time 0x01cb08a1cdf881b1.

Immediately after the event above, this event is logged in the application log (event id 1005)
Windows cannot access the file  for one of the following reasons:  there is a problem with the network connection, the disk that the file is stored on, or the storage  drivers installed on this computer; or the disk is missing.  Windows closed the program <program name> because of this error.

Program: <program name>
File:

The error value is listed in the Additional Data section.
User Action
1. Open the file again.  This situation might be a temporary problem that corrects itself when the program runs again.
2.  If the file still cannot be accessed and
      - It is on the network,  your network administrator should verify that there is not a problem with the network and that the server can be contacted.
      - It is on a removable disk, for example, a floppy disk or CD-ROM, verify that the disk is fully inserted into the computer.
3. Check and repair the file system by running CHKDSK. To run CHKDSK, click Start, click Run, type CMD, and then click OK. At the command prompt, type CHKDSK /F, and then press ENTER.
4. If the problem persists, restore the file from a backup copy.
5. Determine whether other files on the same disk can be opened. If not, the disk might be damaged. If it is a hard disk, contact your administrator or computer hardware vendor for  further assistance.

Additional Data
Error value: C00000C4
Disk type: 0
-------------
I  have tried a lot of things like deleting user profiles, etc, but this error persists.
Does anyone have any suggestions?

Regards,
Christian
Comment
Watch Question

Top Expert 2005

Commented:
Sounds like an intermittant network issue.  The error indicates IO_ERROR: (NTSTATUS) 0xc00000c4 - An unexpected network error occurred (if you were to open the debug log).Disable Offloading on the NICs and see if it goes away.
Top Expert 2012

Commented:
Here are some settings that I would disable.

I have seen issues with TCP Chimney causes isses like this. http://support.microsoft.com/kb/951037

Disable TCP Offload, IP Offload, TCP Checksum, and IP Checksum on the network card properties.
Several questions here:
1. Was the 2003 server before 64-bit as well? If not, was the application fully tested/certified for 64-bit?
2. Does the application work properly on 64-bit, meaning did you confirm this with the developer? If they say they are not sure you may be into something.
3. What happens if you bring the application to the local C: drive of the server for testing? Does it work in this case? If it does the issue may be on the VMWare NIC driver or on a setting as pointed by dariusg.
4. Remember this is a VMWare VM and unexpected things may happen with TS. :-) So the obvious question is, if you try from a physical server with 2008 64-bit SP2, does it work? If it does VMWare and/or their drivers is the issue.

My 2c.

Cláudio Rodrigues
Citrix CTP

Author

Commented:
Thanks for the suggestions, guys.
I have already tried disabling TCP Chimney and RSS, and some of the settings dariusg suggests. I have now disabled the rest, according to dariusg's suggestion.

More suggestions are welcome!

Author

Commented:
Cláudio, I didn't see your post before I posted my previous reply.

1) The 2003 servers are 32-bit. This is an application by a small company, with one developer. He says it works on 64-bit servers, but I'm having second thoughts as well (I think the original source code is fairly old).
2)  They have other customers running the application on 64-bit servers, claiming there are no problems.
3) I have tried the application from the C-drive, but since this is happening randomly we haven't seen the error. But the application can work fine for a day or two, and then suddenly crashes. Maybe I'll have to arrange for a real user to start the program from the C-drive. It might be the VMware NIC, although i changed type from the e1000 adapter to the new vmxnet3 adapter, with no luck.
4) There's always a chance that this might be related to VMware, we have other TS running on VMware, and although it might not be my favorite solution, we're not experiencing any problems, as long as the servers have enough resources. I'd like to try it from a physical server, but I'm not sure I'll be doable at this time...

I have the feeling this happens when users try to perform a certain action in the application, but according to the developer this is not the case...
Unfortunately I don't know the application well enough myself to make it crash.

Thanks a bunch for the advice so far!

Regards,
Christian
Exclude the exe location from real time virus scan and check
a better test would be to completely uninstall AV and see if the problem happens

Top Expert 2005

Commented:
Good point regarding the AV idea.  We had issues when we allowed the AV software to scan network drives and realtime scanning touched anything that was opened.

Exclude network drives and see if you can exclude the folder where the EXE is run from realtime scanning.

Author

Commented:
After I did the adjustments to the network cards the problem got even worse, so I reset them back to defaults.

I've now disabled opportunistic locking on the terminal servers, it looks good so far, but it's a bit early to say.

If anyone is interested I added this to the registry on the terminal servers:
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MRXSmb\Parameters\
OplocksDisabled REG_DWORD Value: 1

Default: 0 (not disabled)

Cut from this KB article:
http://support.microsoft.com/kb/296264
NBF

Commented:
Wow we are having these EXACT same issues with a customer order system application written in house in Delphi.

Running VMWARE vsphere 4 on a FC SAN.  2008 Terminal Server 32 BIT.  Users run the application in remoteapp mode and are getting these exact disk errors intermittently.  We are also having weird issues where the title of the application in a remoteapp will change from the name of the app to just (REMOTE) and the users are unable to figure out what applciation they are ALT-TABBING to.  Closing the app and reopening fixes the labeling issue.  This is also intermittent.

Have you had any other issues after disabling opportunistic locking?
NBF

Commented:
pragmadrift: has disabling opportunisic locking resolved the problem?
NBF

Commented:
Any update on this?  Please don't abandon this topic!
Sorry for the extremely late reply.

If anyone is still reading, disabling opportunistic locking did not help.
The application is now copied to a local folder on the server when the user logs on,  and it's being started from there.
It's a crappy workaround for a strange problem...but it resulted in fewer crashes.

Still seeing some strange disconnects from clients though...

NBF

Commented:
That was our solution as well.  We make lots of updates so it required lots of scipts to copy updates to the application to local folders on the RDP server.  Ridiculous that there is so many problems running applications from network drives or shares over 2008 RDP.

Commented:
I know this is an old question but i created my account solely to post this answer.
The problem lies in Windows 2008. Microsoft recently created a fast publish article about this problem. I have tested the described situation and result are that this is indeed the problem.

http://support.microsoft.com/kb/2536487

The article describes the exact problem that you and i (and many others) experience. When running your application from a network drive in a terminal server environment the application crashes randomly.
This occurs due to the way in which the redirector handles the FCB (File Control Block) for the binary in question. In Windows Server 2008, the FCB is owned by the user who first opened the file and this FCB is used by subsequent users. When the first user logs off, the FCB is orphaned which caused the subsequent users of the application to crash or become unresponsive. In Windows Server 2008 R2, the FCB is owned by the last user to open the file and prior users experience the issue if the last user logs off.

The article further suggests to run your applications from a local drive instead of a (mapped) network drive.

This "bug" is probably by design so do not expect a fix. Instead, expect that all newer versions of Windows will have the same problem.