APC_INDEX_MISMATCH BSOD - Mulitple XenApp 6.5 Windows 2008 R2 Servers

Jacob78
Jacob78 used Ask the Experts™
on
I have a new XenApp 6.5 farm with 14 Dell Servers. All of them are randomly giving the blue screen of death with APC_INDEX_MISMATCH.

Servers XenApp 01-12 are all Dell PowerEdge M610's with identical hardware and software installs.

Servers XenApp18 and XenApp19 were added by me last night. These are R610's with different hardware (except for the Broadcom NIC) and I eliminated most software from them like anti-virus, backup software, etc).

Has anyone seen this??? We've eliminated hardware issues and I'm running into a wall here.
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Carl WebsterCitrix Technology Professional - Fellow
Top Expert 2010

Commented:
Sounds serious.  Looks like a driver error that is crashing the Windows Kernel.

http://msdn.microsoft.com/en-us/library/ff557419%28v=vs.85%29.aspx

Author

Commented:
I thought that too, which is why I brought two additional servers online that had dissimilar hardware. The only thing they share in common is Citrix (and its print drivers) and the Broadcom NIC. I guess it could be the NIC.

What are the chances that this is print driver related?
Carl WebsterCitrix Technology Professional - Fellow
Top Expert 2010

Commented:
If you are talking about the Citrix Universal Print Driver, I would say zero percent it could be the issue.

Have you checked with Dell and made sure you are using the latest, BIOS, firmware and drivers?
Ensure you’re charging the right price for your IT

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

Author

Commented:
If chances are slim to none that it's the UPD, the only common driver to both server models is the Broadcom NIC. I'll go ahead and try installing the latest driver directly from Broadcom (we're using the latest available from Dell, which is 6 months old).

Carl WebsterCitrix Technology Professional - Fellow
Top Expert 2010

Commented:
If you have a Citrix support contract, they can help you get a system dump to them and they should be able to figure out what drive is causing the crash.

The Citrix UPD is so well tested and implemented, I seriously doubt it is the issue.

Author

Commented:
So much for updating the Broadcom drivers. The Dell package included the latest drivers from Broadcom so I'm up to date there as well.

Any other ideas?
Carl WebsterCitrix Technology Professional - Fellow
Top Expert 2010

Commented:
Using windbg and having it analyze the minidump to see what driver it sees as causing the crash.

Author

Commented:
I don't have one but at this point, I'm willing to purchase one.
Carl WebsterCitrix Technology Professional - Fellow
Top Expert 2010

Commented:
Before popping $400 to Citrix support, I would attempt to use windbg and analyze the crash dump myself!  
Commented:
Confirmed by Citrix Support - this is caused by a bug in Special Folder Redirection. Once I have more information I'll post it here.
Carl WebsterCitrix Technology Professional - Fellow
Top Expert 2010

Commented:
Ahhh, never use that feature/option.

Author

Commented:
Will post link to hotfix as soon as Citrix creates one.

Commented:
Did you ever get a hotfix? We're currently experiencing this issue as well, and have calls open with both Microsoft and Citrix, but we're getting bounced between the two.

We are able to reproduce the error if we enable Special Folder Redirection, but Citrix maintains it is a Microsoft fault due to the APC Mismatch error within the memory.dmp file.

Do you have a Citrix call reference number, or job ID we can mention to see if it rings any bells with Citrix??

Regards.

Adrian..

Author

Commented:
We have not, though Citrix is confident that the problem is on their end. Once you open a case with them, you can reference our case number - 60673440.

Author

Commented:
We finally have a private binary to fix the problem. Send me a message if you need it.

Commented:
Hi, if you could send us a copy to try, we're still not getting any headway with Citrix on this yet. I've been trying to find out how to message you, but can't see any options, so if you could email it to my account at "voodoo_people"_at_"hotmail.com" i'll pick it up there and we'll try it out and see how we get on. Much Appreciated.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial