We have a pretty hairy issue at the moment, and I'm looking for some outside eyes and expert thoughts (hoping that I missed something obvious).
We are working on a problem in the following environment:
Windows Active Directory Domain (circa 2003, some 2k8 servers)>
Citrix XenApp 4.5 and 5 Farm>
Published Healthcare Application (PMS software) (running on SQL Server, 2005 I think)>
The PMS Vendor does regular updates for Security / HIPPA and Insurance adjustments. The latest update lastweek broke certain features for a subset of our users (fortunately, only a very select group). Specifically, since this happened, the Billing department has been unable to access a certain section of the app (which only they use - it's a Client Billing section). When they try to open this section from the menu, they receive a generic RunTime error, and the command fails (though the Application doesn't crash entirely).
I have a hunch that we could be looking at an SQL table permission issue, but I'm not SQL pro - and I have more testing to confirm the viability of that (we are using Windows Authentication on SQL). The beauty of this problem is that the Senior Tech (who knew all of this stuff pretty well) left the company right before this happened. We DID ask him for some input (and he graciously offered a few thoughts), but no results.
To date, we have looked hard at Citrix, and the App itself (vendor remoted in to review and confirm that the Update completed fine and that wasn't our problem), as well as User Account permissions (in AD) and GPO's. Lots of RSOP!
The WEIRD thing is that the issue ONLY affects users in this specific "Billing" Organizational Unit in AD (at the same level). I systematically copied an affected user account, and removed all special Roles/Permissions and Account settings (down to Account Expiration), with no change. However, Recreating an IDENTICAL test account in Another OU (as a test, we created a "Business Office" OU) FIXES the problem.
- We can log into Citrix with ANY OTHER USER and have no problem, even when an affected user enters their Application Credentials for the App (users have separate user Accounts within the PMS)
- I checked Policies applied specifically to the Billing OU - there is only one, and it only contains some pre-added IE zone settings.
- Moving the created account from Business Office (working) to Billing (not working), breaks the account.
- Moving the account BACK to Business Office (previously working) from Billing (not working) does NOT fix the account. Neither does moving the Original accounts from Billing to Business Office fix those. Basically, and account that ever enters "Billing", dies.
I have more tests to run (when I get to it) - I'm going bypass Citrix entirely and run directly from an App Server with affected accounts to test. Any bright ideas from the Pros? Anything I'm missing or should look at more closely?