GPO Install

I have a gpo that installs an application on your computer if you are a user in a certain AD group. The gpo and install work fine. However, there is a problem when a user in the group rdp's into a server and the application tries to install (it eventually fails everytime anyway) - we dont want the application to install on the server since it is a client app. We only want it to install when on the users computer when that user logs in.
It is a user enabled gpo. the problem with making it a computer related gpo would mena that it would have to constantly be edited with adding and removing users computers everytime a computer is reimaged, new computers, etc.

Is there anything I can do to make sure it only installs on the clients computer that belongs to the AD group and not the servers? The servers are in a seperate AD OU if that helps.