Citrix server freezing up between 8-8:30 am and problem goes away after 2 pm

Yesterday after 8 am both our Citrix servers froze at same time.  I couldn't get our Citrix support company out there until 1 pm.  They thought it was a printing problem so the set the Policy to only use Citrix Universal Printing.  Everything worked for the remainder of the day.

Today at the same approximate time it happened again.  The following are the steps taken

Force server off because they are unresponsive
Restart

After restarting within a few minutes as the users logged on it would freeze.  When the freeze happened it was to both servers.

Restarted them again.

We use an program called Simplify Printing from Tricerat.  I shut this service down so the only printing was Citrix Universal.  They still froze.  So that eliminated the possibility of Simplify Printing causing the problem.

Removed one of the servers from the Farm.  So there was only one.  Still froze.

I had all the users stop using Citrix.  Then I began to log on as them one by one.  I'd wait 2-3 minutes and then start using the Access Database we use to manage the animals.  I logged on with 7 users and no problem.  I logged on with the 8th and it froze in two minutes.  I thought maybe the users computer was causing the problem.  I restarted the 1 server.  Logged back on as the same user and it never froze.

After 2:14 pm, there was never a freeze.  It is now 6:55 pm pst, and the servers are fine.  There are still users logged in.  

We have two remote locations that we adopt are animals from (spcaLA) and this has completely shut them down.  If I could I award 10,000 points to fix this I would.  I praying someone has an idea what the cause is.  We would also be MORE than glad to pay to solve this.  I don't know if I can offer that on EE, but I hope I can.  THIS IS CRITICAL

Note the support company came back today and between them and I we could find any cause.   HELP!!!!!
J.R. SitmanIT DirectorAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Sekar ChinnakannuStaff EngineerCommented:
Did you checked antivirus tools? some time antivirus tools run the update frequently. Try to disable the same and check. Also check the memory\cpu usage details for the specific time, it will help to find the cause. Did this issue occurring recently? if so check any update you installed means remove the same and check.
0
J.R. SitmanIT DirectorAuthor Commented:
We've been using Eset for two years and no problems.  How do I check the memory\cpu usage for past days?
It started yesterday.  I checked any updates and the last ones were 7/16 and it was Windows Defender.
0
Sekar ChinnakannuStaff EngineerCommented:
is this virtual machine? if so you can get the cpu\memory details, check with server team for the same, try to uninstall all the updates you installed recently. Also sometime antivirus tool updates the version so may it cause the cause check the event logs and make sure no update\errors at specific time
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

J.R. SitmanIT DirectorAuthor Commented:
No two physical servers.  I'll disable AV in the morning as a test.
0
Sekar ChinnakannuStaff EngineerCommented:
for physical server you can user perfmon to get cpu/memory usgate. Check this for more details about perfmon http://rickardnobel.se/using-performance-monitor-effectively/
0
J.R. SitmanIT DirectorAuthor Commented:
Thanks
0
Daniel BorgerSenior Citrix Engineer- CCEECommented:
I'd start by rolling back the last round of updates.  Are they 2003 or 2008 servers?
0
J.R. SitmanIT DirectorAuthor Commented:
Both are 2008 R2.  The last update was 7/9.  The problem started 7/15.  The servers are used everyday, so I don't believe that is the cause.  
One very puzzling item is when it happens, they both freeze.
I've been monitoring it since 6:15 this morning.  There are now 13 users and no problems so far.  I currently only have 1 server in the farm.  
We are also going to add a 3rd Virtual Citrix server today.
0
Daniel BorgerSenior Citrix Engineer- CCEECommented:
Make sure you have the listed Citrix and Microsoft hotfies noted here.  I don't think the MS one come with Windows updates.

http://support.citrix.com/article/CTX129229
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
J.R. SitmanIT DirectorAuthor Commented:
My support company said they did it a few weeks ago.
0
J.R. SitmanIT DirectorAuthor Commented:
What is Citrix Edge Site?
0
Daniel BorgerSenior Citrix Engineer- CCEECommented:
performance and monitoring mostly. it  requires a server and agents to be installed on XenApp. in your case the freeze up may keep any information from being written to the database.
0
J.R. SitmanIT DirectorAuthor Commented:
Thx
0
J.R. SitmanIT DirectorAuthor Commented:
Latest updates helped
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Citrix

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.