Untangle on ESX

the only thing i can think of is that the virtual nics are pointing to the wrong vswitches. Must admit I've never used untangle but can't see how it would be different on a virtual box over a physical one.
Are you pinging by ipaddress or name (dns settings?)

Just to confirm I understand what your connections look like:
untangle VM ---> vnic1 172.17.0.2 -----> vswitch1 -----> Phsical switch (172.17.0.0/16) ----> 172.17.0.1 firewall
untangle VM ---> vnic2 (ip not given) -----> vswitch2 -----> Phsical switch (on subnet for network) ----> network pc's

When setting up the NICs in untangle the first one(external) is given an IP address and the second one(internal) is set to bridging mode.

I am pretty sure the virtual nics are assigned correctly but I have tried to switch them with no success.

I am pinging the ip addresses of some hosts on the network

untangle VM ---> vnic1 -----> vswitch1 -----> 172.17.0.1(firewall)
untangle VM ---> vnic2 -----> vswitch2 -----> Phsical switch (172.17.0.0/16) ----> network pc's

Aslo when setting up the VM I selected Linux Other 32bit.

long shot but :- are the physical nics in use gigabit or 100mb. The reason I ask is because I assume you are running a direct cable to the firewall from the ESX box and I'm wondering if you're using a crossover cable. If so is there a chance that you have gotten them the wrong way around (ie crossover into switch and straight though into firewall).
If its a gigabit connection on either end (likely these days) then never mind at the nics will auto switch anyway and sort themselves out.
Sorry can't be more help but I've never used untangle

Thanks but I have a crossover cable

And you're sure that the crossover cable is connected to the correct nic interface for the correct vswitch.
Don't mean to sound pedantic but have seen simalr issues where the nics are not mapped to the vswitch that the admin thinks they are. i.e your firewall vswitch is connected to the lan and the lan vswitch is connected to the firewall.
if no other vm's are using the esx server you could confirm by unplugging one of the network cables and then check which one is disconnected.

Due to the fact that there are other VM's on the Server I am almost 100% certain that the correct cable is pluged in to the correct vswitch as I have neatley labed them External and Internal the other VM's are connected to internal and continue to be able to access internal recourses and not the firewall when plugged into the Exteranl vswitch.

I am also able to see what mac addresses are assigned to the virtual nic's in untangled and reference them to the VM's config in ESX and verify which vswitch they are connected to in the ESX server.

I appreciate your suggestions as these things are often overlooked and I have double checked them after your posts but more and more I think there is a software configuration issue with this.

Thanks for posting this question.

I had never heard of Untangle so I downloaded it so I go give it a go.

I am now running it on my ESX 3.5 box with no problems.

I am actually using it on a single ESX Physical NIC, which I present to Untangle as two NIC's.
I have a static IP on the LAN side and I am using PPPoE on the External (WAN) side.

So anyway, there doesn't appear to be any problems (so far) that I have seen with running Untange on ESX.

Not sure if this helps you much, but I think you can discount the ESX being a problem, unless it relates to bridge mode.

I was thinking about using bridge mode, rather than routed mode, but I couldn't plug a nic directly into my firewall, so I don't think I can do that.

Is there any reason why you can't use routed mode.  I know I was discouraged from this as the setup wizard wanted to enable DHCP, and I already had one of those, but I discovered you can disable it in the network config section.

did you set the permiscuous mode for the network adapters in ESX to Reject or Allow?

Hi bevhost
I do not see anywhere to apply promiscuous mode settings for the physical network adapters. But both vSwitches are set to reject and both the Virtual Networks(Ports) are not set(unticked).

Internal
Virtual Switch: vSwitch0 Promiscuous Mode:Reject
Virtual Machine Port Group: Internal Promiscuous Mode: Unticked
                      ------------------------------
External
Virtual Switch: vSwitch1 Promiscuous Mode:Reject
Virtual Machine Port Group: External Promiscuous Mode: Unticked

It is not really an option to set this up in routed mode we defnitley need this set up in bridging mode.

Thanks that worked.

After setting both Vswitches to promiscuos mode the Untangled software worked as it should. Thanks