On production servers, I don't want to install X11 but sometimes, we need to give admins a GUI. Some admins are simply better with GUIs while others are better at CLI.
Searching the net for a solution isn't leading me to anything which seems all that simple to install and use so thought I would ask here as I am sure there are others who want a low resource cost GUI solution to manage VMs.
I leave xauth enabled on the target display and use an expect script to authorise the system I'm using - can post if you would be interested (it uses rsh but I guess ssh should work)
As well as xterm, you could use any installed X application - I guess that's what your admins want to do?