APC_INDEX_MISMATCH BSOD - Mulitple XenApp 6.5 Windows 2008 R2 Servers

I have a new XenApp 6.5 farm with 14 Dell Servers. All of them are randomly giving the blue screen of death with APC_INDEX_MISMATCH.

Servers XenApp 01-12 are all Dell PowerEdge M610's with identical hardware and software installs.

Servers XenApp18 and XenApp19 were added by me last night. These are R610's with different hardware (except for the Broadcom NIC) and I eliminated most software from them like anti-virus, backup software, etc).

Has anyone seen this??? We've eliminated hardware issues and I'm running into a wall here.
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Carl WebsterCitrix Technology Professional - FellowCommented:
Sounds serious.  Looks like a driver error that is crashing the Windows Kernel.

Jacob78Author Commented:
I thought that too, which is why I brought two additional servers online that had dissimilar hardware. The only thing they share in common is Citrix (and its print drivers) and the Broadcom NIC. I guess it could be the NIC.

What are the chances that this is print driver related?
Carl WebsterCitrix Technology Professional - FellowCommented:
If you are talking about the Citrix Universal Print Driver, I would say zero percent it could be the issue.

Have you checked with Dell and made sure you are using the latest, BIOS, firmware and drivers?
Determine the Perfect Price for Your IT Services

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!

Jacob78Author Commented:
If chances are slim to none that it's the UPD, the only common driver to both server models is the Broadcom NIC. I'll go ahead and try installing the latest driver directly from Broadcom (we're using the latest available from Dell, which is 6 months old).

Carl WebsterCitrix Technology Professional - FellowCommented:
If you have a Citrix support contract, they can help you get a system dump to them and they should be able to figure out what drive is causing the crash.

The Citrix UPD is so well tested and implemented, I seriously doubt it is the issue.
Jacob78Author Commented:
So much for updating the Broadcom drivers. The Dell package included the latest drivers from Broadcom so I'm up to date there as well.

Any other ideas?
Carl WebsterCitrix Technology Professional - FellowCommented:
Using windbg and having it analyze the minidump to see what driver it sees as causing the crash.
Jacob78Author Commented:
I don't have one but at this point, I'm willing to purchase one.
Carl WebsterCitrix Technology Professional - FellowCommented:
Before popping $400 to Citrix support, I would attempt to use windbg and analyze the crash dump myself!  
Jacob78Author Commented:
Confirmed by Citrix Support - this is caused by a bug in Special Folder Redirection. Once I have more information I'll post it here.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Carl WebsterCitrix Technology Professional - FellowCommented:
Ahhh, never use that feature/option.
Jacob78Author Commented:
Will post link to hotfix as soon as Citrix creates one.
Did you ever get a hotfix? We're currently experiencing this issue as well, and have calls open with both Microsoft and Citrix, but we're getting bounced between the two.

We are able to reproduce the error if we enable Special Folder Redirection, but Citrix maintains it is a Microsoft fault due to the APC Mismatch error within the memory.dmp file.

Do you have a Citrix call reference number, or job ID we can mention to see if it rings any bells with Citrix??


Jacob78Author Commented:
We have not, though Citrix is confident that the problem is on their end. Once you open a case with them, you can reference our case number - 60673440.
Jacob78Author Commented:
We finally have a private binary to fix the problem. Send me a message if you need it.
Hi, if you could send us a copy to try, we're still not getting any headway with Citrix on this yet. I've been trying to find out how to message you, but can't see any options, so if you could email it to my account at "voodoo_people"_at_"hotmail.com" i'll pick it up there and we'll try it out and see how we get on. Much Appreciated.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.