rgmckenz
asked on
Citrix Presentation Server v4.5 Stops Servicing Published Apps After Reboot
I have a nasty situation going on with a Citrix Presentation Server (v4.5). This server is hosted on a Dell PowerEdge 2900 running Windows Server 2003 R2 with all service packs and Windows updates (except for the last few days updates). One day earlier this week, I got a call that no users could get into published apps on the Citrix server. I immediately jumped online to check it out. I could get into the server via RDP and through a Program Neighborhood desktop session, but none of my published apps would run. It mentioned that the ICA browser could not be contacted and I was having network problems. The other symptoms were a missing TCP address on the Zone/Servers screen in the Access Management Console and an error with lmgrd.exe (memory read error) when shutting down and rebooting the server.
There was an update to Goldmine and an installation of Mappoint by another group the week before, and the server had not yet been rebooted until this problem occurred. An automatic Windows update (not the normal setting) caused an update, and the server was not able to service Published Apps after that. I tried my best to bring it back - scanned for virus/other activity (finding none), stopping and restarting services, cleaning up temp files, etc., but nothing worked. Finally, I restored an image of the volumes from Backup Exec System Recovery and I was back to square one, working again. However, I've had to do two more recoveries of the server, once due to an automatic update reboot (didn't turn it off after a restore) and once due to a strange mid-day reboot of the server, from what looked like a svchost process.
I know this is going to happen again, and I can't keep restoring the server's boot image every time. Has anyone ever seen this type of activity before? Any idea of where to go next for troubleshooting? Users of this server all use Application Set Settings from Program Neighborhood to launch the applications - and it's worked fine for almost 3 years now. In the mean time, I'm building another Citrix server - XenApp 5 FR3 - as a potential replacement. Thanks ahead for any help or advice!
There was an update to Goldmine and an installation of Mappoint by another group the week before, and the server had not yet been rebooted until this problem occurred. An automatic Windows update (not the normal setting) caused an update, and the server was not able to service Published Apps after that. I tried my best to bring it back - scanned for virus/other activity (finding none), stopping and restarting services, cleaning up temp files, etc., but nothing worked. Finally, I restored an image of the volumes from Backup Exec System Recovery and I was back to square one, working again. However, I've had to do two more recoveries of the server, once due to an automatic update reboot (didn't turn it off after a restore) and once due to a strange mid-day reboot of the server, from what looked like a svchost process.
I know this is going to happen again, and I can't keep restoring the server's boot image every time. Has anyone ever seen this type of activity before? Any idea of where to go next for troubleshooting? Users of this server all use Application Set Settings from Program Neighborhood to launch the applications - and it's worked fine for almost 3 years now. In the mean time, I'm building another Citrix server - XenApp 5 FR3 - as a potential replacement. Thanks ahead for any help or advice!
ASKER
Thank you for the response. Sounds like good advice! A quick question, though. After I restore the System/Boot volume to a state a few days prior to the first problem, it seems to run fine and accept all connections to Published Apps. My concern is this: if it IS getting corrupted, what is doing the corrupting....and why would it happen after the first reboot (legitimate or otherwise) after a restore?
I'll sweat it out today and hope that the newly restored volume will keep us going during their production hours. When they close for the day, I'll recreate the ICA Listener. If it gets stuck again and I have to restore, I'll do so and recreate the ICA Listener immediately after the restore. I'll let you know what happens. Thanks again!
I'll sweat it out today and hope that the newly restored volume will keep us going during their production hours. When they close for the day, I'll recreate the ICA Listener. If it gets stuck again and I have to restore, I'll do so and recreate the ICA Listener immediately after the restore. I'll let you know what happens. Thanks again!
I have no idea what corrupts the ICA Listener. The customer site I am working at, I have had to recreate the ICA Listener on 4 serevrs and the RDP Listener on 3 servers. No connection issues on those servers since the listeners were deleted and recreated.
ASKER
OK, thanks again. I'll let you know how it works out and will award points accordingly. Have an excellent weekend!
ASKER
CarlWebster,
Here's a quick update. I restored the main volume of the Citrix server once again. I deleted the ICA Listener and rebuilt it from scratch. Unfortunately, it didn't make any difference to my published applications access - it still can't find the ICA Browser and denies connection.
I think I may have some kind of virus, trojan or exploit going on here. I found some interesting things in the error logs and with a couple of users right before the system lost the ICA browser. The remote users was told that Windows updates had completed and asked them to reboot. Since they were running published apps, they thought it was their local machine and rebooted. HOwever, it rebooted the Citrix server, which brings on the problem. Also, on shutdown, there's a lmgrd.exe error message referencing address 0x00000000 - a memory cannot be read. If regular Windows update was on and rebooted the server, the same thing happens.
I may have to build a totally new server (a process which won't be enjoyable in their busy season), but may have no choice. Any other ideas? Thanks again!
Here's a quick update. I restored the main volume of the Citrix server once again. I deleted the ICA Listener and rebuilt it from scratch. Unfortunately, it didn't make any difference to my published applications access - it still can't find the ICA Browser and denies connection.
I think I may have some kind of virus, trojan or exploit going on here. I found some interesting things in the error logs and with a couple of users right before the system lost the ICA browser. The remote users was told that Windows updates had completed and asked them to reboot. Since they were running published apps, they thought it was their local machine and rebooted. HOwever, it rebooted the Citrix server, which brings on the problem. Also, on shutdown, there's a lmgrd.exe error message referencing address 0x00000000 - a memory cannot be read. If regular Windows update was on and rebooted the server, the same thing happens.
I may have to build a totally new server (a process which won't be enjoyable in their busy season), but may have no choice. Any other ideas? Thanks again!
ASKER
One more update...... I went back after the last volume restore and pulled the Citrix licenses off (returned them), updated the license manager, created a new license file (split up between the original license server and a license server on a new farm), and installed the new license file. After that, the IP address appeared in the Zone node for the server and the server started servicing Published Apps again. I will wait until a more quiet time to reboot the server and see if the configuration holds, or whether it reverts back to its former state. I hope that we can close this out with a successful reboot.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Well, I'm not sure where to go with this one. Even after reinstalling the license manager and associated programs, the Citrix server continued to exhibit the same symptoms after a reboot. I must conclude that there was some kind of bug or worm that made it to the server and it was directing reboots affecting the license manager in some way. We scrapped the server and moved users over to a new XenApp v5 server we just built (virtual server on Hyper-V, which runs extremely well). We're wiping out the operating system, etc. on the old physical box and will reload it with XenApp 5, so that we have two servers in the farm.
Carl, I'm not sure how to assign the points here, but since you were the one that gave the best advice and got back to me with the license issue (which I think it partly was), I'll assign you the full amount. Thanks for your help. Much appreciated!
Carl, I'm not sure how to assign the points here, but since you were the one that gave the best advice and got back to me with the license issue (which I think it partly was), I'll assign you the full amount. Thanks for your help. Much appreciated!
ASKER
The solutions presented alleviated the problem, allowing us time to build another server, but we never did find the root cause. Thanks for the help, however.
http://support.citrix.com/article/CTX115152