RDS User being redirected to wrong session host

I'm not trying to run a "farm", and this is my first go at the new terminal services model of a gateway, broker, and hosts.  I'm trying to run each session host server as it's own box (due to software installed). We are a small shop so HA/clustering isn't a desire.

I've installed the RDS Licenses on one box, the Broker and Gateway on another, and 5 separate session servers (all hyper V VM's).

The issue I'm having is when a user logs on to a specific server IP or name, they are getting redirected to a different one.  Is it load balancing that is doing this? If so how can I turn it off, also I've read that the broker "knows" what server you last logged on to and puts you back on it. I need the same user ID to be able to log on to different session hosts at the same time.