Solved

Load balancing with Solaris zones

Posted on 2007-03-28
4
2,626 Views
Last Modified: 2013-12-27
Here the setup:

Layer 2/3 loadbalanced Webservers. (load balancing done by manipulation of IP packets and data)

Loadbalancer ip 212.47.171.152 MAC-ADDR:00:01:02:72:a7:23
One Server   ip 212.47.171.145 MAC-ADDR 08:00:20:ed:13:66 with 2 Solaris 10 zones on it.
Each zone has one IP address and a separate Network Adapter.
On each Zone is a seperate apache web server.

zone1        ip 212.47.171.151 MAC-ADDR:08:00:20:ed:13:66  (same MAC as global zone)
zone2        ip 212.47.171.153 MAC_ADDR:08:00:20:ed:44:d5

The customer sends all his HTTP request to the load balancer (the dns of the requested domain resolves the ip of the loadbalancer) The Loadbalancer detects the Webservers buy sending a request every 20s to a certain page on each of the webservers. If the page contains a certain keyword the load balancer includes the webserver in the cluster and it will receive requests from the internet.

In order for this to work each webserver has an additional loopback adapter configured with the IP (.152) of the loadbalancer. This is necessary so that apache can be configured (2 Listen entries, one with .152 the other with the IP of the Solaris zone) to answer to request which are sent to the IP of the loadbalancer. ARP is disabled for the loopback adapter so that the loadbalancer is the only visible 212.47.171.152 IP in the network thus no conflicts.

So summarized a request from the internet will hit the Loadbalancer. It will then remove his IP in the TCP/IP request and replace with the IP of the customer and send it to one of the web servers on the zones by also changing the MAC address. The Apache web server sees a request for the IP .152 coming from the internet, The apache generates a response and sends it DIRECTLY back to the customer without knowing that it came from the load balancer.

So far so good. This works fine on a n-machine scale with physical machines. But because of the setup using Solaris zones there is a problem:

No matter where the loadbalancer sends the IP packet (zone1 or zone2) it is allways the first zone that was started up that sends the response.

Is there a bug in my concept or a fault in the config or a error in solaris zones or might this be a feature?

Example 1:
boot zone1
boot zone2
remove the webserver on zone1 from the loabalancer cluster by changing the keyword
add the webserver on zone2 from the loabalancer cluster by changing the keyword
send the request to .152 (load balancer)
the response comes from zone1

Example 2:
boot zone2
boot zone1
add the webserver on zone1 from the loabalancer cluster by changing the keyword
remove the webserver on zone2 from the loabalancer cluster by changing the keyword
send the request to .152 (load balancer)
the response comes from zone2

Here the snoop for example 1:

Request from loadbalancer to zone2 (check the MAC-ADDR)
ETHER:  ----- Ether Header -----
ETHER:  
ETHER:  Packet 11 arrived at 23:31:34.52007
ETHER:  Packet size = 296 bytes
ETHER:  Destination = 8:0:20:ed:44:d5, Sun
ETHER:  Source      = 0:1:2:72:a7:23,
ETHER:  Ethertype = 0800 (IP)
ETHER:  
IP:   ----- IP Header -----
IP:  
IP:   Version = 4
IP:   Header length = 20 bytes
IP:   Type of service = 0x00
IP:         xxx. .... = 0 (precedence)
IP:         ...0 .... = normal delay
IP:         .... 0... = normal throughput
IP:         .... .0.. = normal reliability
IP:         .... ..0. = not ECN capable transport
IP:         .... ...0 = no ECN congestion experienced
IP:   Total length = 282 bytes
IP:   Identification = 29885
IP:   Flags = 0x4
IP:         .1.. .... = do not fragment
IP:         ..0. .... = last fragment
IP:   Fragment offset = 0 bytes
IP:   Time to live = 116 seconds/hops
IP:   Protocol = 6 (TCP)
IP:   Header checksum = 9ff5
IP:   Source address = 85.2.28.97, 97-28.2-85.cust.bluewin.ch
IP:   Destination address = 212.47.171.152, 212.47.171.152
IP:   No options
IP:  
TCP:  ----- TCP Header -----
TCP:  
TCP:  Source port = 12901
TCP:  Destination port = 80 (HTTP)
TCP:  Sequence number = 1582209661
TCP:  Acknowledgement number = 1285440570
TCP:  Data offset = 20 bytes
TCP:  Flags = 0x18
TCP:        0... .... = No ECN congestion window reduced
TCP:        .0.. .... = No ECN echo
TCP:        ..0. .... = No urgent pointer
TCP:        ...1 .... = Acknowledgement
TCP:        .... 1... = Push
TCP:        .... .0.. = No reset
TCP:        .... ..0. = No Syn
TCP:        .... ...0 = No Fin
TCP:  Window = 65535
TCP:  Checksum = 0x656e
TCP:  Urgent pointer = 0
TCP:  No options
TCP:  
HTTP: ----- HyperText Transfer Protocol -----
HTTP:
HTTP: GET /lb/loytest.html HTTP/1.1
HTTP: Accept: */*
HTTP: Accept-Language: de-ch
HTTP: Accept-Encoding: gzip, deflate
HTTP: User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
HTTP: [...]
HTTP:


Response from zone1 instead of zone2! (check the MAC-ADDR) 0:0:d1:ed:96:30 is the default gateway.
ETHER:  ----- Ether Header -----
ETHER:  
ETHER:  Packet 13 arrived at 23:31:34.52115
ETHER:  Packet size = 449 bytes
ETHER:  Destination = 0:0:d1:ed:96:30, Adaptec Inc. Nodem product
ETHER:  Source      = 8:0:20:ed:13:66, Sun
ETHER:  Ethertype = 0800 (IP)
ETHER:  
IP:   ----- IP Header -----
IP:  
IP:   Version = 4
IP:   Header length = 20 bytes
IP:   Type of service = 0x00
IP:         xxx. .... = 0 (precedence)
IP:         ...0 .... = normal delay
IP:         .... 0... = normal throughput
IP:         .... .0.. = normal reliability
IP:         .... ..0. = not ECN capable transport
IP:         .... ...0 = no ECN congestion experienced
IP:   Total length = 435 bytes
IP:   Identification = 48798
IP:   Flags = 0x4
IP:         .1.. .... = do not fragment
IP:         ..0. .... = last fragment
IP:   Fragment offset = 0 bytes
IP:   Time to live = 64 seconds/hops
IP:   Protocol = 6 (TCP)
IP:   Header checksum = 897b
IP:   Source address = 212.47.171.152, 212.47.171.152
IP:   Destination address = 85.2.28.97, 97-28.2-85.cust.bluewin.ch
IP:   No options
IP:  
TCP:  ----- TCP Header -----
TCP:  
TCP:  Source port = 80
TCP:  Destination port = 12901
TCP:  Sequence number = 1285440570
TCP:  Acknowledgement number = 1582209903
TCP:  Data offset = 20 bytes
TCP:  Flags = 0x18
TCP:        0... .... = No ECN congestion window reduced
TCP:        .0.. .... = No ECN echo
TCP:        ..0. .... = No urgent pointer
TCP:        ...1 .... = Acknowledgement
TCP:        .... 1... = Push
TCP:        .... .0.. = No reset
TCP:        .... ..0. = No Syn
TCP:        .... ...0 = No Fin
TCP:  Window = 50400
TCP:  Checksum = 0xf96c
TCP:  Urgent pointer = 0
TCP:  No options
TCP:  
HTTP: ----- HyperText Transfer Protocol -----
HTTP:
HTTP: HTTP/1.1 200 OK
HTTP: Date: Wed, 28 Mar 2007 21:31:34 GMT
HTTP: Server: Apache
HTTP: Last-Modified: Wed, 21 Mar 2007 13:54:59 GMT
HTTP: ETag: "1b196-7e-2b0a72c0"
HTTP: [...]
HTTP:

loy-sw00:/root% ifconfig -a
lo0: flags=20010008c9<UP,LOOPBACK,RUNNING,NOARP,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
lo0:1: flags=20010008c9<UP,LOOPBACK,RUNNING,NOARP,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        zone loy-sz00
        inet 212.47.171.152 netmask ffffff00
lo0:2: flags=20010008c9<UP,LOOPBACK,RUNNING,NOARP,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        zone loy-sz00
        inet 127.0.0.1 netmask ff000000
lo0:3: flags=20010008c9<UP,LOOPBACK,RUNNING,NOARP,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        zone loy-sz01
        inet 212.47.171.152 netmask ffffff00
lo0:4: flags=20010008c9<UP,LOOPBACK,RUNNING,NOARP,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        zone loy-sz01
        inet 127.0.0.1 netmask ff000000
hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet 212.47.171.145 netmask fffffff0 broadcast 212.47.171.159
        ether 8:0:20:ed:13:66
hme0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        zone loy-sz00
        inet 212.47.171.151 netmask fffffff0 broadcast 212.47.171.159
hme0:2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        zone loy-sz00
        inet 192.168.121.151 netmask ffffff00 broadcast 192.168.121.255
hme1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
        inet 10.168.121.145 netmask ffffff00 broadcast 10.168.121.255
        ether 8:0:20:ed:38:d
hme2: flags=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
        inet 0.0.0.0 netmask 0
        ether 8:0:20:ed:44:d5
hme2:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
        zone loy-sz01
        inet 192.168.121.153 netmask ffffff00 broadcast 192.168.121.255
hme2:2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
        zone loy-sz01
        inet 212.47.171.153 netmask fffffff0 broadcast 212.47.171.159

Zoneconfig zone1
create -b
set zonepath=/opt/zones/loy-sz00
set autoboot=true
add inherit-pkg-dir
set dir=/lib
end
add inherit-pkg-dir
set dir=/platform
end
add inherit-pkg-dir
set dir=/sbin
end
add inherit-pkg-dir
set dir=/usr
end
add fs
set dir=/opt/tomcat
set special=/opt/apache-tomcat-5.5.12
set type=lofs
end
add net
set address=212.47.171.151/28
set physical=hme0
end
add net
set address=192.168.121.151
set physical=hme0
end
add net
set address=212.47.171.152/32
set physical=lo0
end

Zoneconfig zone2
create -b
set zonepath=/opt/zones/loy-sz01
set autoboot=true
add inherit-pkg-dir
set dir=/lib
end
add inherit-pkg-dir
set dir=/platform
end
add inherit-pkg-dir
set dir=/sbin
end
add inherit-pkg-dir
set dir=/usr
end
add fs
set dir=/opt/tomcat
set special=/opt/apache-tomcat-5.5.12_app2
set type=lofs
end
add net
set address=192.168.121.153
set physical=hme2
end
add net
set address=212.47.171.153/28
set physical=hme2
end
add net
set address=212.47.171.152/32
set physical=lo0
end
0
Comment
Question by:benmathis
  • 3
4 Comments
 
LVL 22

Expert Comment

by:blu
ID: 18816649
The difficulty you are having is due to the fact that at this time, zones use only a single network stack. So, when it comes time to send a packet to the router the routing tables are searched and the first entry
is used, focusing all the traffic through a single interface. There are a couple of possible ways to address this. You don't say what version of Solaris you are using. The latest version of Solaris Express  
has the virtualized IP stack in it so that this will "just work" if configured. This same feature is expected to be released in the next Solaris 10 update later this year.

Now, my understanding is that the system should be balancing the outbound traffic across routes. Could you post the output of "netstat -rn" from the global zone?
0
 
LVL 22

Expert Comment

by:blu
ID: 18816692
By the way, you should not be configuring logical interfaces on lo0 as loop back in this manner. The proper way to do this on Solaris is to configure those addresses  against a virtual network interface, (vni). This is the supported way on Solaris.
0
 

Author Comment

by:benmathis
ID: 18824612
Aha, I am currently using Solaris 10 1/06 (Generic_118822-25). So a upgrade to the newest Solaris 10 wouldnt fix the problem. By the way, i was not able to config vni on the zones. As soon as I wanted to start the seccond zone i got a error message:

loy-sw00:/etc/zones% ifconfig vni0 plumb
loy-sw00:/etc/zones% zoneadm -z loy-sz00 boot
loy-sw00:/etc/zones% zoneadm -z loy-sz01 boot
zoneadm: zone 'loy-sz01': vni0:2: could not bring interface up: address in use by zone 'loy-sz00': Cannot assign requested address
zoneadm: zone 'loy-sz01': call to zoneadmd failed

Here The netstat
loy-sw00:/etc/zones% netstat -rnv
IRE Table: IPv4
  Destination             Mask           Gateway          Device Mxfrg  Rtt  Ref Flg  Out  In/Fwd
-------------------- --------------- -------------------- ------ ----- ----- --- --- ----- ------
192.168.121.1        255.255.255.255 212.47.171.145       hme0    1500*    0   1 UH     330     0
212.47.171.144       255.255.255.240 212.47.171.145       hme0    1500*    0   1 U        1     0
172.23.24.0          255.255.255.0   10.168.121.1                 1500*    0   1 UG       3     0
10.168.10.0          255.255.255.0   10.168.121.1                 1500*    0   1 UG      14     0
10.168.121.0         255.255.255.0   10.168.121.145       hme1    1500*    0   1 U       11     0
224.0.0.0            240.0.0.0       212.47.171.145       hme0    1500*    0   1 U        0     0
default              0.0.0.0         192.168.121.1                1500*    0   1 UG    3884     0
127.0.0.1            255.255.255.255 127.0.0.1            lo0     8232*    0  21 UH    3084     0

Now I need more explaning to do. Our routing is not standard.
This machine is in a datacenter with about 40 dedicated Firewalls for each customer.
To save public IPs we give the Firewalls Private IPs
Default GW is 192.168.121.1 (2nd last rule) and will be reached via hme0 (first rule)
The servers do not have (need)  a private IP as the ifconfig sent earlier shows.
172.x and 10.x is for backup

I can see a problem with the first rule. I guess i will have to use source based routing.
0
 
LVL 22

Accepted Solution

by:
blu earned 500 total points
ID: 18836156
The problem with your outbound balancing is that you only have a single default route set up. Since the default route points to gateway 192.168.121.1, and you have a host route set up that points this address to the hme0 interface, the system has no choice but to send the packets out that interface. If the hme1 interface can likewise reach that gateway, then if you add a second host route for the gateway through the hme1 interface, then the system will balance the outbound packets between them. Of course if the hme1 interface cannot reach the gateway, then the discussion is moot.

I don't know why you had trouble with the vni interfaces, it does work and is used by many people. However, since you don't show hwo you configured them, I don't know where you went wrong. Along as you do not configure the loopback address as point-to-point, the way you are doing should work, so carry on and don't worry about the vni thing if you don't need to.
0

Featured Post

Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

Join & Write a Comment

What’s a web proxy server? A proxy server is a server that goes between clients and web servers, used in corporate to enforce corporate browsing policy and ensure security. Proxy servers are commonly used in three modes. A)    Forward proxy …
When it comes to security, there are always trade-offs between security and convenience/ease of administration. This article examines some of the main pros and cons of using key authentication vs password authentication for hosting an SFTP server.
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

706 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now