Load balancing with Solaris zones

Here the setup:

Layer 2/3 loadbalanced Webservers. (load balancing done by manipulation of IP packets and data)

Loadbalancer ip 212.47.171.152 MAC-ADDR:00:01:02:72:a7:23
One Server   ip 212.47.171.145 MAC-ADDR 08:00:20:ed:13:66 with 2 Solaris 10 zones on it.
Each zone has one IP address and a separate Network Adapter.
On each Zone is a seperate apache web server.

zone1        ip 212.47.171.151 MAC-ADDR:08:00:20:ed:13:66  (same MAC as global zone)
zone2        ip 212.47.171.153 MAC_ADDR:08:00:20:ed:44:d5

The customer sends all his HTTP request to the load balancer (the dns of the requested domain resolves the ip of the loadbalancer) The Loadbalancer detects the Webservers buy sending a request every 20s to a certain page on each of the webservers. If the page contains a certain keyword the load balancer includes the webserver in the cluster and it will receive requests from the internet.

In order for this to work each webserver has an additional loopback adapter configured with the IP (.152) of the loadbalancer. This is necessary so that apache can be configured (2 Listen entries, one with .152 the other with the IP of the Solaris zone) to answer to request which are sent to the IP of the loadbalancer. ARP is disabled for the loopback adapter so that the loadbalancer is the only visible 212.47.171.152 IP in the network thus no conflicts.

So summarized a request from the internet will hit the Loadbalancer. It will then remove his IP in the TCP/IP request and replace with the IP of the customer and send it to one of the web servers on the zones by also changing the MAC address. The Apache web server sees a request for the IP .152 coming from the internet, The apache generates a response and sends it DIRECTLY back to the customer without knowing that it came from the load balancer.

So far so good. This works fine on a n-machine scale with physical machines. But because of the setup using Solaris zones there is a problem:

No matter where the loadbalancer sends the IP packet (zone1 or zone2) it is allways the first zone that was started up that sends the response.

Is there a bug in my concept or a fault in the config or a error in solaris zones or might this be a feature?

Example 1:
boot zone1
boot zone2
remove the webserver on zone1 from the loabalancer cluster by changing the keyword
add the webserver on zone2 from the loabalancer cluster by changing the keyword
send the request to .152 (load balancer)
the response comes from zone1

Example 2:
boot zone2
boot zone1
add the webserver on zone1 from the loabalancer cluster by changing the keyword
remove the webserver on zone2 from the loabalancer cluster by changing the keyword
send the request to .152 (load balancer)
the response comes from zone2

Here the snoop for example 1:

Request from loadbalancer to zone2 (check the MAC-ADDR)
ETHER:  ----- Ether Header -----
ETHER:  
ETHER:  Packet 11 arrived at 23:31:34.52007
ETHER:  Packet size = 296 bytes
ETHER:  Destination = 8:0:20:ed:44:d5, Sun
ETHER:  Source      = 0:1:2:72:a7:23,
ETHER:  Ethertype = 0800 (IP)
ETHER:  
IP:   ----- IP Header -----
IP:  
IP:   Version = 4
IP:   Header length = 20 bytes
IP:   Type of service = 0x00
IP:         xxx. .... = 0 (precedence)
IP:         ...0 .... = normal delay
IP:         .... 0... = normal throughput
IP:         .... .0.. = normal reliability
IP:         .... ..0. = not ECN capable transport
IP:         .... ...0 = no ECN congestion experienced
IP:   Total length = 282 bytes
IP:   Identification = 29885
IP:   Flags = 0x4
IP:         .1.. .... = do not fragment
IP:         ..0. .... = last fragment
IP:   Fragment offset = 0 bytes
IP:   Time to live = 116 seconds/hops
IP:   Protocol = 6 (TCP)
IP:   Header checksum = 9ff5
IP:   Source address = 85.2.28.97, 97-28.2-85.cust.bluewin.ch
IP:   Destination address = 212.47.171.152, 212.47.171.152
IP:   No options
IP:  
TCP:  ----- TCP Header -----
TCP:  
TCP:  Source port = 12901
TCP:  Destination port = 80 (HTTP)
TCP:  Sequence number = 1582209661
TCP:  Acknowledgement number = 1285440570
TCP:  Data offset = 20 bytes
TCP:  Flags = 0x18
TCP:        0... .... = No ECN congestion window reduced
TCP:        .0.. .... = No ECN echo
TCP:        ..0. .... = No urgent pointer
TCP:        ...1 .... = Acknowledgement
TCP:        .... 1... = Push
TCP:        .... .0.. = No reset
TCP:        .... ..0. = No Syn
TCP:        .... ...0 = No Fin
TCP:  Window = 65535
TCP:  Checksum = 0x656e
TCP:  Urgent pointer = 0
TCP:  No options
TCP:  
HTTP: ----- HyperText Transfer Protocol -----
HTTP:
HTTP: GET /lb/loytest.html HTTP/1.1
HTTP: Accept: */*
HTTP: Accept-Language: de-ch
HTTP: Accept-Encoding: gzip, deflate
HTTP: User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
HTTP: [...]
HTTP:


Response from zone1 instead of zone2! (check the MAC-ADDR) 0:0:d1:ed:96:30 is the default gateway.
ETHER:  ----- Ether Header -----
ETHER:  
ETHER:  Packet 13 arrived at 23:31:34.52115
ETHER:  Packet size = 449 bytes
ETHER:  Destination = 0:0:d1:ed:96:30, Adaptec Inc. Nodem product
ETHER:  Source      = 8:0:20:ed:13:66, Sun
ETHER:  Ethertype = 0800 (IP)
ETHER:  
IP:   ----- IP Header -----
IP:  
IP:   Version = 4
IP:   Header length = 20 bytes
IP:   Type of service = 0x00
IP:         xxx. .... = 0 (precedence)
IP:         ...0 .... = normal delay
IP:         .... 0... = normal throughput
IP:         .... .0.. = normal reliability
IP:         .... ..0. = not ECN capable transport
IP:         .... ...0 = no ECN congestion experienced
IP:   Total length = 435 bytes
IP:   Identification = 48798
IP:   Flags = 0x4
IP:         .1.. .... = do not fragment
IP:         ..0. .... = last fragment
IP:   Fragment offset = 0 bytes
IP:   Time to live = 64 seconds/hops
IP:   Protocol = 6 (TCP)
IP:   Header checksum = 897b
IP:   Source address = 212.47.171.152, 212.47.171.152
IP:   Destination address = 85.2.28.97, 97-28.2-85.cust.bluewin.ch
IP:   No options
IP:  
TCP:  ----- TCP Header -----
TCP:  
TCP:  Source port = 80
TCP:  Destination port = 12901
TCP:  Sequence number = 1285440570
TCP:  Acknowledgement number = 1582209903
TCP:  Data offset = 20 bytes
TCP:  Flags = 0x18
TCP:        0... .... = No ECN congestion window reduced
TCP:        .0.. .... = No ECN echo
TCP:        ..0. .... = No urgent pointer
TCP:        ...1 .... = Acknowledgement
TCP:        .... 1... = Push
TCP:        .... .0.. = No reset
TCP:        .... ..0. = No Syn
TCP:        .... ...0 = No Fin
TCP:  Window = 50400
TCP:  Checksum = 0xf96c
TCP:  Urgent pointer = 0
TCP:  No options
TCP:  
HTTP: ----- HyperText Transfer Protocol -----
HTTP:
HTTP: HTTP/1.1 200 OK
HTTP: Date: Wed, 28 Mar 2007 21:31:34 GMT
HTTP: Server: Apache
HTTP: Last-Modified: Wed, 21 Mar 2007 13:54:59 GMT
HTTP: ETag: "1b196-7e-2b0a72c0"
HTTP: [...]
HTTP:

loy-sw00:/root% ifconfig -a
lo0: flags=20010008c9<UP,LOOPBACK,RUNNING,NOARP,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
lo0:1: flags=20010008c9<UP,LOOPBACK,RUNNING,NOARP,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        zone loy-sz00
        inet 212.47.171.152 netmask ffffff00
lo0:2: flags=20010008c9<UP,LOOPBACK,RUNNING,NOARP,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        zone loy-sz00
        inet 127.0.0.1 netmask ff000000
lo0:3: flags=20010008c9<UP,LOOPBACK,RUNNING,NOARP,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        zone loy-sz01
        inet 212.47.171.152 netmask ffffff00
lo0:4: flags=20010008c9<UP,LOOPBACK,RUNNING,NOARP,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        zone loy-sz01
        inet 127.0.0.1 netmask ff000000
hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet 212.47.171.145 netmask fffffff0 broadcast 212.47.171.159
        ether 8:0:20:ed:13:66
hme0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        zone loy-sz00
        inet 212.47.171.151 netmask fffffff0 broadcast 212.47.171.159
hme0:2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        zone loy-sz00
        inet 192.168.121.151 netmask ffffff00 broadcast 192.168.121.255
hme1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
        inet 10.168.121.145 netmask ffffff00 broadcast 10.168.121.255
        ether 8:0:20:ed:38:d
hme2: flags=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
        inet 0.0.0.0 netmask 0
        ether 8:0:20:ed:44:d5
hme2:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
        zone loy-sz01
        inet 192.168.121.153 netmask ffffff00 broadcast 192.168.121.255
hme2:2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
        zone loy-sz01
        inet 212.47.171.153 netmask fffffff0 broadcast 212.47.171.159

Zoneconfig zone1
create -b
set zonepath=/opt/zones/loy-sz00
set autoboot=true
add inherit-pkg-dir
set dir=/lib
end
add inherit-pkg-dir
set dir=/platform
end
add inherit-pkg-dir
set dir=/sbin
end
add inherit-pkg-dir
set dir=/usr
end
add fs
set dir=/opt/tomcat
set special=/opt/apache-tomcat-5.5.12
set type=lofs
end
add net
set address=212.47.171.151/28
set physical=hme0
end
add net
set address=192.168.121.151
set physical=hme0
end
add net
set address=212.47.171.152/32
set physical=lo0
end

Zoneconfig zone2
create -b
set zonepath=/opt/zones/loy-sz01
set autoboot=true
add inherit-pkg-dir
set dir=/lib
end
add inherit-pkg-dir
set dir=/platform
end
add inherit-pkg-dir
set dir=/sbin
end
add inherit-pkg-dir
set dir=/usr
end
add fs
set dir=/opt/tomcat
set special=/opt/apache-tomcat-5.5.12_app2
set type=lofs
end
add net
set address=192.168.121.153
set physical=hme2
end
add net
set address=212.47.171.153/28
set physical=hme2
end
add net
set address=212.47.171.152/32
set physical=lo0
end