Hm, I think you need to do this with routing; not with iptables. At least I would never think of doing this with iptables...

Try:

ip route add default scope global \
nexthop via <wan1 gateway ip> dev wan1 weight 1 \
nexthop via <wan2 gateway ip> dev wan2 weight 1

Select allOpen in new window

Because of weight 1 we have 50:50 load balancing in our default gateway. If the WANs are not running equal seeds, adjust the weight to the correct ratio. Eg. if you have a 50Mbit wan1 and a 10 Mbit wan2, wan1 weight would be 1 and wan2 weight would be 5.

My wans are DHCP, is it possible with the method you mentioned?

Interesting, helge000

I have been sticking with iptables for now, here is what I came up with but sleep calls.


This doesn't seem to work, but almost...  meaning, pages load some, then hang...  then load... it's weird. At least I got connmark to work!



$IPT -t mangle -N CM1
$IPT -t mangle -A CM1 -j MARK --set-mark 1
$IPT -t mangle -A CM1 -j CONNMARK --save-mark

$IPT -t mangle -N CM2
$IPT -t mangle -A CM2 -j MARK --set-mark 2
$IPT -t mangle -A CM2 -j CONNMARK --save-mark


$IPT -t mangle -A POSTROUTING -o wan1 -p tcp -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark

$IPT -t mangle -A POSTROUTING -o wan2 -p tcp -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark

$IPT -t mangle -A PREROUTING -i wan1 -p tcp -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark

$IPT -t mangle -A PREROUTING -i wan2 -p tcp -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark


$IPT -t mangle -A POSTROUTING -o wan1 -p tcp -m state --state NEW -m statistic --mode nth --every 2 --packet 0 -j CM1

$IPT -t mangle -A POSTROUTING -o wan2 -p tcp -m state --state NEW -m statistic --mode nth --every 2 --packet 1 -j CM2

$IPT -t mangle -A PREROUTING -i wan1 -p tcp -m state --state NEW -m statistic --mode nth --every 2 --packet 0 -j CM1

$IPT -t mangle -A PREROUTING -i wan2 -p tcp -m state --state NEW -m statistic --mode nth --every 2 --packet 1 -j CM2

The mangle chain would be the correct one, was my first guess to. But keep in mind the following:

If you open a page in a browser, the connection always goes through the default GW (of your Router!). Then, the answer comes back from the web server, which replies to address he got.

If you "mangle" it, NAT would not know the package, because it comes back through a connection never opened by a client. The firewall discards the packet. Thats why the pages only open partly.

The correct way is to through the routing table of your router. Then, balancing works.

If you try my script above; I have to apologize: I forgot to mention that the cron job would also have to set the default route anew. The correct script would be:

#!/bin/bash
export GW1=$(ip route show dev wan1 | awk '/default/ {print $3}')
export GW2=$(ip route show dev wan2 | awk '/default/ {print $3}')

ip route add default scope global \
nexthop via $GW1 dev wan1 weight 1 \
nexthop via $GW2 dev wan2 weight 1

Select allOpen in new window

And if you prefer the easy way (I always do!) have a look at:
http://pfsense.org

This the rirewall appliance I always use. Can all the stuff your box can do and much more, plus you can configure everything from a handy web GUI :)

I always learn more by doing it the hard way lol. Thanks for your help, I really appreciate it. I'll look into all of this later tonight :-)

dr34m3r

The nexthop solution didn't work for whatever reason.

I'm continuing my research and development.

You need to setup centos as a router.
Two WAN ports and 1 LAN NATed port
Mangle is the correct table to handle packets leaving LAN port.
Your mark setting deals with incoming traffic. This is needed for loadbalancing services on your LAN accessed from outside.
In your case, you want to load balance outgoing requests which suggests you need to mark within the postrouting table to make sure outgoing requests maintain state.
This is important for secure communication connections, VPN, HTTPS, imaps, pop3s, SMTPS.

Your routing table is set by DHCP.
The setup you have will not detect a failure of a WAN port without an additional ping type test since the cable connection is terminated on the cable router.

I got to the point where I can mark the packets and restore the packets. I am not able to get them to go through any interface but the default gateway however.

Any thoughts on that?

Here are my rules.

ten1 is my 10GbE switch
wan1 and wan2 are my broadband connections


# LOAD BALANCE MARKS

# Restore a MARKed Connection
$IPT -N RESMARK -t mangle
$IPT -A RESMARK -t mangle -j CONNMARK --restore-mark
#$IPT -A RESMARK -t mangle -j LOG --log-prefix 'restore-mark: ' --log-level info
$IPT -A RESMARK -t mangle -j RETURN

# Create MARKING chains
$IPT -N CN1 -t mangle
$IPT -A CN1 -t mangle -j MARK --set-mark 1
$IPT -A CN1 -t mangle -j CONNMARK --save-mark
$IPT -A CN1 -t mangle -j LOG --log-prefix 'set-mark 1: ' --log-level info
$IPT -A CN1 -t mangle -j RETURN

$IPT -N CN2 -t mangle
$IPT -A CN2 -t mangle -j MARK --set-mark 2
$IPT -A CN2 -t mangle -j CONNMARK --save-mark
$IPT -A CN2 -t mangle -j LOG --log-prefix 'set-mark 2: ' --log-level info
$IPT -A CN2 -t mangle -j RETURN

# Set Marks Based On Statistics
$IPT -A FORWARD -t mangle -i ten1 -o wan1 -m state --state NEW -m statistic --mode nth --every 2 --packet 0 -j CN1
$IPT -A FORWARD -t mangle -i ten1 -o wan1 -m state --state NEW -m statistic --mode nth --every 2 --packet 0 -j CN2

# Restore Mark If Marked And Established Or Related
$IPT -A FORWARD -t mangle -i wan1 -o ten1 -m state --state ESTABLISHED,RELATED -j RESMARK
$IPT -A FORWARD -t mangle -i wan2 -o ten1 -m state --state ESTABLISHED,RELATED -j RESMARK
$IPT -A FORWARD -t mangle -i ten1 -o wan1 -m state --state ESTABLISHED,RELATED -j RESMARK
$IPT -A FORWARD -t mangle -i ten1 -o wan2 -m state --state ESTABLISHED,RELATED -j RESMARK

Forward rule seems to be the only place I can set -i and -o together that's why I was testing it this way.

The packets seem to be getting marked and restored correctly though.

They do not seem to be going through the correct routing table even though they seem to be setup


/etc/iproute2/rt_tables

#
# reserved values
#
255     local
254     main
253     default
0      unspec
#
# local
#
#1      inr.ruhep
200     wr1
201     wr2



[root@fwrtr apf]# ip route show table wr1
default via 10.1.0.1 dev wan1

[root@fwrtr apf]# ip route show table wr2
default via 10.2.0.1 dev wan2



[root@fwrtr sh]# cat balance.rules.sh
#!/bin/bash

ip route flush table wr1
ip route flush table wr2

#ip route add table wr1 default dev wan1:gw
#ip route add table wr2 default dev wan2:gw

ip rule del from all fwmark 2 2>/dev/null
ip rule del from all fwmark 1 2>/dev/null

ip rule add fwmark 1 table wr1 prio 1024
ip rule add fwmark 2 table wr2 prio 1025

ip route add default via 10.1.0.1 table wr1
ip route add default via 10.2.0.1 table wr2

ip route flush cache


10.1.0.1 is a virtual ip routing to wan1 and the internet
10.2.0.1 is a virtual ip routing to wan2 and the internet

wan1:gw is the default gateway ip of 10.1.0.1

Everything seems to be routing through 10.1.0.1 (the default gateway)



There are probably just a few things I'm missing or incorrect on, please correct!! :-)

Thanks for any help!

dr34m3r


PS I tried with SNAT but that didn't seem to do the trick...

[root@fwrtr network-scripts]# cat /etc/sysctl.conf
# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled.  See sysctl(8) and
# sysctl.conf(5) for more details.

# Controls IP packet forwarding
net.ipv4.ip_forward = 1

# Controls source route verification
net.ipv4.conf.default.rp_filter = 0
#net.ipv4.conf.wan1.rp_filter=0
#net.ipv4.conf.wan2.rp_filter=0


# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0

# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0

# Controls whether core dumps will append the PID to the core filename.
# Useful for debugging multi-threaded applications.
kernel.core_uses_pid = 1

# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1

# Disable netfilter on bridges.
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

# Controls the maximum size of a message, in bytes
kernel.msgmnb = 65536

# Controls the default maxmimum size of a mesage queue
kernel.msgmax = 65536

# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736

# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296




# Increased ARP Sizes
## works best with <= 500 client computers ##
# Force gc to clean-up quickly
net.ipv4.neigh.default.gc_interval = 3600
 
# Set ARP cache entry timeout
net.ipv4.neigh.default.gc_stale_time = 3600
 
# Setup DNS threshold for arp
net.ipv4.neigh.default.gc_thresh3 = 4096
net.ipv4.neigh.default.gc_thresh2 = 2048
net.ipv4.neigh.default.gc_thresh1 = 1024

You are on the wrong side of the iptables.

You need mangle on the output rule
See if the below helps.

http://blog.khax.net/2009/12/01/multi-gateway-balancing-with-iptables/
Using the -s IP1 for wan1 and -s IP2 for wan2 on each respective outgoing rule will direct the response to the correct destination.

I have tried a number of things but nothing is working yet.

I can get it to almost work sometimes, but it's almost like there is something missing, a little thing somewhere... I'll keep working on it.

I have another issue I have to resolve that *may* be related....

I'll do that first and post back if I figure it out.

Thanks!

dr34m3r

By "almost work" I mean, I can see the marks and the connections being restored from wan1 and wan2 in my debug log but I think there is an issue with wan2...  so things will start to load, then stop, load half a page, then not load the images....

Also, my ifconfig shows no bandwidth increase, just the usual packets of a connection with no data being send through it.....

I see: I am unable to ping my wan2 from the outside world. This is definitely an odd issue and contributing to the load balancing issue I'm having with this thread....

Haha, wow. What a week! :)

I'll post back here once I figure out everything

Posting a new thread on the inability to ping wan2 on EE...

dr34m3r

The same issue the sends everything through wan1 is the same reason an ICMP received on the WAN2 is not being sent back.

Yes. That is definitely the issue holding me back.

I'm attempting this or similar.

http://kindlund.wordpress.com/2007/11/19/configuring-multiple-default-routes-in-linux/

I assumed linux would just understand that if an incoming packet that is from a gateway link, and there were multiple gateways, and no rules to change gateway links available, then it would just go back the same way it came in... but what was I thinking, that would make SENSE! ;-)

Ok I'll post back once I figure this out.

iptables is the one that will be distributing the routes and not the routing table.

The iptables rules on the outgoing side will need to alter the source IP to match the interface through which the request is being sent out.

The two wan ports you have use private IPs space which sounds that when you ping WAN2 it terminates on the device that ISP provided and not getting to your centos box.

The default routes rules often deal with load distribution by assigning both default gateway the same weight/metric.

Is your centos 6 box configured as a router where the ten1 interface is natted by iptables?

Hi arnold,

Yes, all the lans and tens are just forward nat rules, one forwarding through to the other.

I have taken out the middle man for my testing and the virtual IP's are not an issue right now (10.2.0.1 and 10.1.0.1 / wan1:gw and wan2:gw) I will add those back later if possible to make my internal rules more permanent.

I was able to route through my ping now with the ip route rules, but I am unable to connect to any services I would assume because IP tables isn't setup correctly for it yet. How would I do that?

It seems data is coming into wan2 -> ping gets returned -> services such as forwarded 25 do not return, only work on wan1 still...  where do I jump to now? ;-)

As a router you need a rule on the INPUT side and a forward that directs the packet to a LAN IP and port.

IMHO, you should first complete the outgoing functionality.

The mangle on the INPUT/FORWARD with mark is needed to maintain response paths.

I'm trying to keep it simple but not being a network guru it's hard for me to figure out the little details sometimes.

Do you have a quick example I could work from?

I would like to handle outbound balancing first, then move to inbound. Inbound is actually easier. I'm not worried about inbound.

Question:

Does this kernel support what I'm attempting to do?

Linux 2.6.32-358.14.1.el6.x86_64 #1 SMP Tue Jul 16 23:51:20 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

Yes the kernel supports.  The question is whether you setup the system as a router?
If you disconnect wan2, are systems behind the ten1 interface have access to the outside?  You should start with this.

Centos has that it adds a chain into INPUT and FORWARD
This deals/simplifies things such that you add a single rule into the chain rather than having to add one rule on the INPUT side to accept the specific type of request and then you need another rule on the forward side to pass it to a system behind the NAT.

Similarly on the outgoing side.

You look at anything leaving the ten1 interface from the LAN and redirect them out through nth, with the change in the source through one of the gateways. Etc.
You do not leave it upto the routing tables.

--to-destination defines where the packet should be going when dealing with i

I've tried about everything I can for tonight.

Night everyone :)

Hope I wake up to a solution tomorrow...

This didn't work:

$IPT -A POSTROUTING -t nat -m state --state NEW -m statistic --mode nth --every 2 --packet 0 -j SNAT --to-source x.x.x.x
$IPT -A POSTROUTING -t nat -m state --state NEW -m statistic --mode nth --every 2 --packet 1 -j SNAT --to-source x.x.x.x

But I didn't expect it would.

This code seems to mark packets properly, but the marks are not going through the correct interfaces even when set correctly in the ip rules and ip routes

# Restore a MARKed Connection
$IPT -N RESW1 -t mangle
$IPT -A RESW1 -t mangle -j CONNMARK --restore-mark
$IPT -A RESW1 -t mangle -m mark ! --mark 0 -j LOG --log-prefix 'restore-mark wan1: ' --log-level info
$IPT -A RESW1 -t mangle -j RETURN

$IPT -N RESW2 -t mangle
$IPT -A RESW2 -t mangle -j CONNMARK --restore-mark
$IPT -A RESW2 -t mangle -m mark ! --mark 0 -j LOG --log-prefix 'restore-mark wan2: ' --log-level info
$IPT -A RESW2 -t mangle -j RETURN

$IPT -N REST1 -t mangle
$IPT -A REST1 -t mangle -j CONNMARK --restore-mark
$IPT -A REST1 -t mangle -m mark ! --mark 0 -j LOG --log-prefix 'restore-mark ten1: ' --log-level info
$IPT -A REST1 -t mangle -j RETURN

$IPT -N RESFW1 -t mangle
$IPT -A RESFW1 -t mangle -j CONNMARK --restore-mark
$IPT -A RESFW1 -t mangle -m mark ! --mark 0 -j LOG --log-prefix 'restore-mark forward 1: ' --log-level info
$IPT -A RESFW1 -t mangle -j RETURN

$IPT -N RESFW2 -t mangle
$IPT -A RESFW2 -t mangle -j CONNMARK --restore-mark
$IPT -A RESFW2 -t mangle -m mark ! --mark 0 -j LOG --log-prefix 'restore-mark forward 2: ' --log-level info
$IPT -A RESFW2 -t mangle -j RETURN

# Create MARKING chains
$IPT -N CN1 -t mangle
$IPT -A CN1 -t mangle -j MARK --set-mark 1
$IPT -A CN1 -t mangle -j CONNMARK --save-mark
$IPT -A CN1 -t mangle -j LOG --log-prefix 'set-mark 1: ' --log-level info
$IPT -A CN1 -t mangle -j RETURN

$IPT -N CN2 -t mangle
$IPT -A CN2 -t mangle -j MARK --set-mark 2
$IPT -A CN2 -t mangle -j CONNMARK --save-mark
$IPT -A CN2 -t mangle -j LOG --log-prefix 'set-mark 2: ' --log-level info
$IPT -A CN2 -t mangle -j RETURN

# Set Marks Based On Statistics
$IPT -A PREROUTING -i ten1 -t mangle -m state --state NEW -m statistic --mode nth --every 2 --packet 0 -j CN2
$IPT -A PREROUTING -i ten1 -t mangle -m state --state NEW -m statistic --mode nth --every 2 --packet 1 -j CN1

# Restore Mark If Marked And Established Or Related
$IPT -A PREROUTING -t mangle -i ten1 -m state --state ESTABLISHED,RELATED -j REST1
$IPT -A PREROUTING -t mangle -i wan1 -m state --state ESTABLISHED,RELATED -j RESW1
$IPT -A PREROUTING -t mangle -i wan2 -m state --state ESTABLISHED,RELATED -j RESW2

$IPT -A POSTROUTING -t mangle -o wan1 -m state --state ESTABLISHED,RELATED -j RESW1
$IPT -A POSTROUTING -t mangle -o wan2 -m state --state ESTABLISHED,RELATED -j RESW2

$IPT -A FORWARD -t mangle -i ten1 -o wan1 -m state --state ESTABLISHED,RELATED -j RESFW1
$IPT -A FORWARD -t mangle -i ten1 -o wan2 -m state --state ESTABLISHED,RELATED -j RESFW2
$IPT -A FORWARD -t mangle -i wan1 -o ten1 -m state --state ESTABLISHED,RELATED -j RESFW1
$IPT -A FORWARD -t mangle -i wan2 -o ten1 -m state --state ESTABLISHED,RELATED -j RESFW2

Maybe fwmark is simply broken.

[root@fwrtr sh]# ip rule show
0:      from all lookup local
1:      from all to x.x.x.x lookup wio2
2:      from x.x.x.x lookup wio2
3:      from all to x.x.x.x lookup wio1
4:      from x.x.x.x lookup wio1
5:      from all fwmark 0x1 lookup wio1
6:      from all fwmark 0x2 lookup wio2
32766:      from all lookup main
32767:      from all lookup default


I may as well just go with pfsense or an option that actually WORKS :-)

could you output iptables -L -t filter --line-numbers
iptables -L -t nat --line-numbers
iptables -L -t mangle --line-numbers

You are using -A which adds an entry, but it is not clear what order the entry is in.

use -I "TABLE" number
where number is the line number on which this new rule will appear.

[root@fwrtr apf]# iptables -L -t filter --line-numbers
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination        
1    ACCEPT     all  --  anywhere             anywhere            
2    DROP       all  --  100.64.0.0/10        anywhere            
3    DROP       all  --  loopback/8           anywhere            
4    DROP       all  --  link-local/16        anywhere            
5    DROP       all  --  192.0.0.0/24         anywhere            
6    DROP       all  --  192.0.2.0/24         anywhere            
7    DROP       all  --  198.18.0.0/15        anywhere            
8    DROP       all  --  198.51.100.0/24      anywhere            
9    DROP       all  --  203.0.113.0/24       anywhere            
10   DROP       all  --  base-address.mcast.net/4  anywhere            
11   DROP       all  --  240.0.0.0/4          anywhere            
12   TMP_DROP   all  --  anywhere             anywhere            
13   TALLOW     all  --  anywhere             anywhere            
14   TDENY      all  --  anywhere             anywhere            
15   TGALLOW    all  --  anywhere             anywhere            
16   TGDENY     all  --  anywhere             anywhere            
17   DROP       tcp  --  anywhere             anywhere            tcp dpts:epmap:netbios-ssn
18   DROP       udp  --  anywhere             anywhere            udp dpts:epmap:netbios-ssn
19   DROP       tcp  --  anywhere             anywhere            tcp dpt:sunrpc
20   DROP       udp  --  anywhere             anywhere            udp dpt:sunrpc
21   DROP       tcp  --  anywhere             anywhere            tcp dpt:login
22   DROP       udp  --  anywhere             anywhere            udp dpt:who
23   DROP       tcp  --  anywhere             anywhere            tcp dpt:efs
24   DROP       udp  --  anywhere             anywhere            udp dpt:router
25   DROP       tcp  --  anywhere             anywhere            tcp dpt:microsoft-ds
26   DROP       udp  --  anywhere             anywhere            udp dpt:microsoft-ds
27   DROP       tcp  --  anywhere             anywhere            tcp dpt:ms-sql-s
28   DROP       udp  --  anywhere             anywhere            udp dpt:ms-sql-s
29   DROP       tcp  --  anywhere             anywhere            tcp dpt:ms-sql-m
30   DROP       udp  --  anywhere             anywhere            udp dpt:ms-sql-m
31   DROP       tcp  --  anywhere             anywhere            tcp dpt:search-agent
32   DROP       udp  --  anywhere             anywhere            udp dpt:search-agent
33   DROP       tcp  --  anywhere             anywhere            tcp dpt:ingreslock
34   DROP       udp  --  anywhere             anywhere            udp dpt:ingreslock
35   DROP       tcp  --  anywhere             anywhere            tcp dpt:ctx-bridge
36   DROP       udp  --  anywhere             anywhere            udp dpt:ctx-bridge
37   IN_SANITY  all  --  anywhere             anywhere            
38   FRAG_UDP   all  --  anywhere             anywhere            
39   PZERO      all  --  anywhere             anywhere            
40   P2P        all  --  anywhere             anywhere            
41   ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:EtherNet/IP-1
42   ACCEPT     icmp --  anywhere             anywhere            icmp destination-unreachable limit: avg 30/sec burst 5
43   ACCEPT     icmp --  anywhere             anywhere            icmp redirect limit: avg 30/sec burst 5
44   ACCEPT     icmp --  anywhere             anywhere            icmp time-exceeded limit: avg 30/sec burst 5
45   ACCEPT     icmp --  anywhere             anywhere            icmp echo-reply limit: avg 30/sec burst 5
46   ACCEPT     icmp --  anywhere             anywhere            icmp type 30 limit: avg 30/sec burst 5
47   ACCEPT     icmp --  anywhere             anywhere            icmp echo-request limit: avg 30/sec burst 5
48   DROP       tcp  --  anywhere             anywhere            tcp flags:!FIN,SYN,RST,ACK/SYN state NEW
49   ACCEPT     tcp  --  anywhere             anywhere            state RELATED,ESTABLISHED
50   ACCEPT     udp  --  anywhere             anywhere            state RELATED,ESTABLISHED
51   ACCEPT     udp  --  a.resolvers.level3.net  anywhere            udp spt:domain dpts:1023:65535
52   ACCEPT     tcp  --  a.resolvers.level3.net  anywhere            tcp spt:domain dpts:1023:65535
53   DROP       tcp  --  anywhere             anywhere            tcp spt:domain dpts:1023:65535
54   DROP       udp  --  anywhere             anywhere            udp spt:domain dpts:1023:65535
55   ACCEPT     udp  --  b.resolvers.Level3.net  anywhere            udp spt:domain dpts:1023:65535
56   ACCEPT     tcp  --  b.resolvers.Level3.net  anywhere            tcp spt:domain dpts:1023:65535
57   DROP       tcp  --  anywhere             anywhere            tcp spt:domain dpts:1023:65535
58   DROP       udp  --  anywhere             anywhere            udp spt:domain dpts:1023:65535
59   ACCEPT     tcp  --  anywhere             anywhere            tcp spt:EtherNet/IP-1 dpts:login:65535 state RELATED,ESTABLISHED
60   ACCEPT     tcp  --  anywhere             anywhere            tcp spts:1024:65535 dpt:EtherNet/IP-1 flags:FIN,SYN,RST,ACK/SYN state RELATED,ESTABLISHED
61   ACCEPT     udp  --  anywhere             anywhere            udp dpt:EtherNet/IP-1 state ESTABLISHED
62   ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpts:traceroute:33534
63   DROP       tcp  --  anywhere             anywhere            
64   DROP       udp  --  anywhere             anywhere            
65   DROP       all  --  anywhere             anywhere            

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination        
1    ACCEPT     all  --  anywhere             anywhere            state NEW
2    ACCEPT     all  --  anywhere             anywhere            state NEW
3    ACCEPT     all  --  anywhere             anywhere            state NEW
4    ACCEPT     all  --  anywhere             anywhere            state NEW
5    ACCEPT     all  --  anywhere             anywhere            state NEW
6    ACCEPT     all  --  anywhere             anywhere            state NEW
7    ACCEPT     all  --  anywhere             anywhere            state NEW
8    ACCEPT     all  --  anywhere             anywhere            state NEW
9    ACCEPT     all  --  anywhere             anywhere            state NEW
10   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
11   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
12   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
13   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
14   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
15   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
16   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
17   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
18   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
19   ACCEPT     all  --  anywhere             anywhere            state NEW
20   ACCEPT     all  --  anywhere             anywhere            state NEW
21   ACCEPT     all  --  anywhere             anywhere            state NEW
22   ACCEPT     all  --  anywhere             anywhere            state NEW
23   ACCEPT     all  --  anywhere             anywhere            state NEW
24   ACCEPT     all  --  anywhere             anywhere            state NEW
25   ACCEPT     all  --  anywhere             anywhere            state NEW
26   ACCEPT     all  --  anywhere             anywhere            state NEW
27   ACCEPT     all  --  anywhere             anywhere            state NEW
28   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
29   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
30   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
31   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
32   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
33   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
34   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
35   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
36   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
37   ACCEPT     all  --  anywhere             anywhere            state NEW
38   ACCEPT     all  --  anywhere             anywhere            state NEW
39   ACCEPT     all  --  anywhere             anywhere            state NEW
40   ACCEPT     all  --  anywhere             anywhere            state NEW
41   ACCEPT     all  --  anywhere             anywhere            state NEW
42   ACCEPT     all  --  anywhere             anywhere            state NEW
43   ACCEPT     all  --  anywhere             anywhere            state NEW
44   ACCEPT     all  --  anywhere             anywhere            state NEW
45   ACCEPT     all  --  anywhere             anywhere            state NEW
46   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
47   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
48   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
49   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
50   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
51   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
52   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
53   ACCEPT     all  --  anywhere             anywhere            state NEW
54   ACCEPT     all  --  anywhere             anywhere            state NEW
55   ACCEPT     all  --  anywhere             anywhere            state NEW
56   ACCEPT     all  --  anywhere             anywhere            state NEW
57   ACCEPT     all  --  anywhere             anywhere            state NEW
58   ACCEPT     all  --  anywhere             anywhere            state NEW
59   ACCEPT     all  --  anywhere             anywhere            state NEW
60   ACCEPT     all  --  anywhere             anywhere            state NEW
61   ACCEPT     all  --  anywhere             anywhere            state NEW
62   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
63   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
64   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
65   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
66   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
67   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
68   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
69   ACCEPT     all  --  anywhere             anywhere            state NEW
70   ACCEPT     all  --  anywhere             anywhere            state NEW
71   ACCEPT     all  --  anywhere             anywhere            state NEW
72   ACCEPT     all  --  anywhere             anywhere            state NEW
73   ACCEPT     all  --  anywhere             anywhere            state NEW
74   ACCEPT     all  --  anywhere             anywhere            state NEW
75   ACCEPT     all  --  anywhere             anywhere            state NEW
76   ACCEPT     all  --  anywhere             anywhere            state NEW
77   ACCEPT     all  --  anywhere             anywhere            state NEW
78   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
79   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
80   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
81   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
82   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
83   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
84   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
85   ACCEPT     all  --  anywhere             anywhere            state NEW
86   ACCEPT     all  --  anywhere             anywhere            state NEW
87   ACCEPT     all  --  anywhere             anywhere            state NEW
88   ACCEPT     all  --  anywhere             anywhere            state NEW
89   ACCEPT     all  --  anywhere             anywhere            state NEW
90   ACCEPT     all  --  anywhere             anywhere            state NEW
91   ACCEPT     all  --  anywhere             anywhere            state NEW
92   ACCEPT     all  --  anywhere             anywhere            state NEW
93   ACCEPT     all  --  anywhere             anywhere            state NEW
94   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
95   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
96   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
97   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
98   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
99   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
100  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
101  ACCEPT     all  --  anywhere             anywhere            state NEW
102  ACCEPT     all  --  anywhere             anywhere            state NEW
103  ACCEPT     all  --  anywhere             anywhere            state NEW
104  ACCEPT     all  --  anywhere             anywhere            state NEW
105  ACCEPT     all  --  anywhere             anywhere            state NEW
106  ACCEPT     all  --  anywhere             anywhere            state NEW
107  ACCEPT     all  --  anywhere             anywhere            state NEW
108  ACCEPT     all  --  anywhere             anywhere            state NEW
109  ACCEPT     all  --  anywhere             anywhere            state NEW
110  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
111  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
112  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
113  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
114  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
115  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
116  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
117  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
118  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
119  ACCEPT     all  --  anywhere             anywhere            state NEW
120  ACCEPT     all  --  anywhere             anywhere            state NEW
121  ACCEPT     all  --  anywhere             anywhere            state NEW
122  ACCEPT     all  --  anywhere             anywhere            state NEW
123  ACCEPT     all  --  anywhere             anywhere            state NEW
124  ACCEPT     all  --  anywhere             anywhere            state NEW
125  ACCEPT     all  --  anywhere             anywhere            state NEW
126  ACCEPT     all  --  anywhere             anywhere            state NEW
127  ACCEPT     all  --  anywhere             anywhere            state NEW
128  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
129  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
130  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
131  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
132  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
133  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
134  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
135  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
136  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
137  ACCEPT     all  --  anywhere             anywhere            state NEW
138  ACCEPT     all  --  anywhere             anywhere            state NEW
139  ACCEPT     all  --  anywhere             anywhere            state NEW
140  ACCEPT     all  --  anywhere             anywhere            state NEW
141  ACCEPT     all  --  anywhere             anywhere            state NEW
142  ACCEPT     all  --  anywhere             anywhere            state NEW
143  ACCEPT     all  --  anywhere             anywhere            state NEW
144  ACCEPT     all  --  anywhere             anywhere            state NEW
145  ACCEPT     all  --  anywhere             anywhere            state NEW
146  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
147  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
148  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
149  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
150  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
151  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
152  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
153  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
154  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
155  ACCEPT     all  --  anywhere             anywhere            state NEW
156  ACCEPT     all  --  anywhere             anywhere            state NEW
157  ACCEPT     all  --  anywhere             anywhere            state NEW
158  ACCEPT     all  --  anywhere             anywhere            state NEW
159  ACCEPT     all  --  anywhere             anywhere            state NEW
160  ACCEPT     all  --  anywhere             anywhere            state NEW
161  ACCEPT     all  --  anywhere             anywhere            state NEW
162  ACCEPT     all  --  anywhere             anywhere            state NEW
163  ACCEPT     all  --  anywhere             anywhere            state NEW
164  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
165  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
166  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
167  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
168  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
169  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
170  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
171  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
172  ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination        
1    ACCEPT     all  --  anywhere             anywhere            
2    TCPMSS     tcp  --  anywhere             anywhere            tcp flags:SYN,RST/SYN TCPMSS clamp to PMTU
3    DROP       all  --  anywhere             100.64.0.0/10      
4    DROP       all  --  anywhere             loopback/8          
5    DROP       all  --  anywhere             link-local/16      
6    DROP       all  --  anywhere             192.0.0.0/24        
7    DROP       all  --  anywhere             192.0.2.0/24        
8    DROP       all  --  anywhere             198.18.0.0/15      
9    DROP       all  --  anywhere             198.51.100.0/24    
10   DROP       all  --  anywhere             203.0.113.0/24      
11   DROP       all  --  anywhere             base-address.mcast.net/4
12   DROP       all  --  anywhere             240.0.0.0/4        
13   TMP_DROP   all  --  anywhere             anywhere            
14   TALLOW     all  --  anywhere             anywhere            
15   TDENY      all  --  anywhere             anywhere            
16   TGALLOW    all  --  anywhere             anywhere            
17   TGDENY     all  --  anywhere             anywhere            
18   DROP       tcp  --  anywhere             anywhere            tcp dpts:epmap:netbios-ssn
19   DROP       udp  --  anywhere             anywhere            udp dpts:epmap:netbios-ssn
20   DROP       tcp  --  anywhere             anywhere            tcp dpt:sunrpc
21   DROP       udp  --  anywhere             anywhere            udp dpt:sunrpc
22   DROP       tcp  --  anywhere             anywhere            tcp dpt:login
23   DROP       udp  --  anywhere             anywhere            udp dpt:who
24   DROP       tcp  --  anywhere             anywhere            tcp dpt:efs
25   DROP       udp  --  anywhere             anywhere            udp dpt:router
26   DROP       tcp  --  anywhere             anywhere            tcp dpt:microsoft-ds
27   DROP       udp  --  anywhere             anywhere            udp dpt:microsoft-ds
28   DROP       tcp  --  anywhere             anywhere            tcp dpt:ms-sql-s
29   DROP       udp  --  anywhere             anywhere            udp dpt:ms-sql-s
30   DROP       tcp  --  anywhere             anywhere            tcp dpt:ms-sql-m
31   DROP       udp  --  anywhere             anywhere            udp dpt:ms-sql-m
32   DROP       tcp  --  anywhere             anywhere            tcp dpt:search-agent
33   DROP       udp  --  anywhere             anywhere            udp dpt:search-agent
34   DROP       tcp  --  anywhere             anywhere            tcp dpt:ingreslock
35   DROP       udp  --  anywhere             anywhere            udp dpt:ingreslock
36   DROP       tcp  --  anywhere             anywhere            tcp dpt:ctx-bridge
37   DROP       udp  --  anywhere             anywhere            udp dpt:ctx-bridge
38   OUT_SANITY  all  --  anywhere             anywhere            
39   FRAG_UDP   all  --  anywhere             anywhere            
40   PZERO      all  --  anywhere             anywhere            
41   P2P        all  --  anywhere             anywhere            
42   ACCEPT     tcp  --  anywhere             anywhere            tcp dpts:1024:65535 state RELATED,ESTABLISHED
43   ACCEPT     udp  --  anywhere             anywhere            udp dpts:1024:65535 state RELATED,ESTABLISHED
44   ACCEPT     udp  --  anywhere             a.resolvers.level3.net udp spts:1023:65535 dpt:domain
45   ACCEPT     tcp  --  anywhere             a.resolvers.level3.net tcp spts:1023:65535 dpt:domain
46   ACCEPT     udp  --  anywhere             a.resolvers.level3.net udp spts:1023:65535 dpt:domain
47   ACCEPT     tcp  --  anywhere             a.resolvers.level3.net tcp spts:1023:65535 dpt:domain
48   ACCEPT     udp  --  anywhere             b.resolvers.Level3.net udp spts:1023:65535 dpt:domain
49   ACCEPT     tcp  --  anywhere             b.resolvers.Level3.net tcp spts:1023:65535 dpt:domain
50   ACCEPT     udp  --  anywhere             b.resolvers.Level3.net udp spts:1023:65535 dpt:domain
51   ACCEPT     tcp  --  anywhere             b.resolvers.Level3.net tcp spts:1023:65535 dpt:domain
52   ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpts:traceroute:33534
53   ACCEPT     all  --  anywhere             anywhere            

Chain FRAG_UDP (2 references)
num  target     prot opt source               destination        
1    DROP       udp  -f  anywhere             anywhere            

Chain IN_SANITY (1 references)
num  target     prot opt source               destination        
1    DROP       tcp  --  anywhere             anywhere            tcp flags:FIN,SYN,RST,PSH,ACK,URG/NONE
2    DROP       tcp  --  anywhere             anywhere            tcp flags:FIN,SYN/FIN,SYN
3    DROP       tcp  --  anywhere             anywhere            tcp flags:SYN,RST/SYN,RST
4    DROP       tcp  --  anywhere             anywhere            tcp flags:FIN,RST/FIN,RST
5    DROP       tcp  --  anywhere             anywhere            tcp flags:FIN,ACK/FIN
6    DROP       tcp  --  anywhere             anywhere            tcp flags:ACK,URG/URG
7    DROP       tcp  --  anywhere             anywhere            tcp flags:PSH,ACK/PSH
8    DROP       tcp  --  anywhere             anywhere            tcp flags:FIN,SYN,RST,PSH,ACK,URG/FIN,PSH,URG
9    DROP       tcp  --  anywhere             anywhere            tcp flags:FIN,SYN,RST,PSH,ACK,URG/FIN,SYN,RST,ACK,URG
10   DROP       tcp  --  anywhere             anywhere            tcp flags:FIN,SYN,RST,PSH,ACK,URG/FIN,SYN,RST,PSH,ACK,URG
11   DROP       tcp  --  anywhere             anywhere            tcp flags:FIN,SYN,RST,PSH,ACK,URG/FIN

Chain OUT_SANITY (1 references)
num  target     prot opt source               destination        
1    DROP       tcp  --  anywhere             anywhere            tcp flags:FIN,SYN,RST,PSH,ACK,URG/NONE
2    DROP       tcp  --  anywhere             anywhere            tcp flags:FIN,SYN/FIN,SYN
3    DROP       tcp  --  anywhere             anywhere            tcp flags:SYN,RST/SYN,RST
4    DROP       tcp  --  anywhere             anywhere            tcp flags:FIN,RST/FIN,RST
5    DROP       tcp  --  anywhere             anywhere            tcp flags:FIN,ACK/FIN
6    DROP       tcp  --  anywhere             anywhere            tcp flags:PSH,ACK/PSH
7    DROP       tcp  --  anywhere             anywhere            tcp flags:ACK,URG/URG

Chain P2P (2 references)
num  target     prot opt source               destination        
1    REJECT     tcp  --  anywhere             anywhere            tcp dpt:kazaa reject-with icmp-port-unreachable
2    REJECT     tcp  --  anywhere             anywhere            tcp spt:kazaa dpts:1024:65534 reject-with icmp-port-unreachable
3    REJECT     udp  --  anywhere             anywhere            udp spts:1024:65534 dpt:kazaa reject-with icmp-port-unreachable
4    REJECT     udp  --  anywhere             anywhere            udp spt:kazaa dpts:1024:65534 reject-with icmp-port-unreachable
5    REJECT     tcp  --  anywhere             anywhere            tcp dpt:3d-nfsd reject-with icmp-port-unreachable
6    REJECT     tcp  --  anywhere             anywhere            tcp spt:3d-nfsd dpts:1024:65534 reject-with icmp-port-unreachable
7    REJECT     udp  --  anywhere             anywhere            udp spts:1024:65534 dpt:3d-nfsd reject-with icmp-port-unreachable
8    REJECT     udp  --  anywhere             anywhere            udp spt:3d-nfsd dpts:1024:65534 reject-with icmp-port-unreachable
9    REJECT     tcp  --  anywhere             anywhere            tcp spts:1024:65534 dpts:smaclmgr:traversal reject-with icmp-port-unreachable
10   REJECT     tcp  --  anywhere             anywhere            tcp spts:smaclmgr:traversal dpts:1024:65534 reject-with icmp-port-unreachable
11   REJECT     udp  --  anywhere             anywhere            udp spts:1024:65534 dpts:smaclmgr:traversal reject-with icmp-port-unreachable
12   REJECT     udp  --  anywhere             anywhere            udp spts:smaclmgr:traversal dpts:1024:65534 reject-with icmp-port-unreachable
13   REJECT     tcp  --  anywhere             anywhere            tcp dpt:6257 reject-with icmp-port-unreachable
14   REJECT     tcp  --  anywhere             anywhere            tcp spt:6257 dpts:1024:65534 reject-with icmp-port-unreachable
15   REJECT     udp  --  anywhere             anywhere            udp spts:1024:65534 dpt:6257 reject-with icmp-port-unreachable
16   REJECT     udp  --  anywhere             anywhere            udp spt:6257 dpts:1024:65534 reject-with icmp-port-unreachable
17   REJECT     tcp  --  anywhere             anywhere            tcp dpt:6699 reject-with icmp-port-unreachable
18   REJECT     tcp  --  anywhere             anywhere            tcp spt:6699 dpts:1024:65534 reject-with icmp-port-unreachable
19   REJECT     udp  --  anywhere             anywhere            udp spts:1024:65534 dpt:6699 reject-with icmp-port-unreachable
20   REJECT     udp  --  anywhere             anywhere            udp spt:6699 dpts:1024:65534 reject-with icmp-port-unreachable
21   REJECT     tcp  --  anywhere             anywhere            tcp dpt:gnutella-svc reject-with icmp-port-unreachable
22   REJECT     tcp  --  anywhere             anywhere            tcp spt:gnutella-svc dpts:1024:65534 reject-with icmp-port-unreachable
23   REJECT     udp  --  anywhere             anywhere            udp spts:1024:65534 dpt:gnutella-svc reject-with icmp-port-unreachable
24   REJECT     udp  --  anywhere             anywhere            udp spt:gnutella-svc dpts:1024:65534 reject-with icmp-port-unreachable
25   REJECT     tcp  --  anywhere             anywhere            tcp dpt:gnutella-rtr reject-with icmp-port-unreachable
26   REJECT     tcp  --  anywhere             anywhere            tcp spt:gnutella-rtr dpts:1024:65534 reject-with icmp-port-unreachable
27   REJECT     udp  --  anywhere             anywhere            udp spts:1024:65534 dpt:gnutella-rtr reject-with icmp-port-unreachable
28   REJECT     udp  --  anywhere             anywhere            udp spt:gnutella-rtr dpts:1024:65534 reject-with icmp-port-unreachable
29   REJECT     tcp  --  anywhere             anywhere            tcp spts:1024:65534 dpts:6881:6889 reject-with icmp-port-unreachable
30   REJECT     tcp  --  anywhere             anywhere            tcp spts:6881:6889 dpts:1024:65534 reject-with icmp-port-unreachable
31   REJECT     udp  --  anywhere             anywhere            udp spts:1024:65534 dpts:6881:6889 reject-with icmp-port-unreachable
32   REJECT     udp  --  anywhere             anywhere            udp spts:6881:6889 dpts:1024:65534 reject-with icmp-port-unreachable
33   REJECT     tcp  --  anywhere             anywhere            tcp dpt:gnutella-svc reject-with icmp-port-unreachable
34   REJECT     tcp  --  anywhere             anywhere            tcp spt:gnutella-svc dpts:1024:65534 reject-with icmp-port-unreachable
35   REJECT     udp  --  anywhere             anywhere            udp spts:1024:65534 dpt:gnutella-svc reject-with icmp-port-unreachable
36   REJECT     udp  --  anywhere             anywhere            udp spt:gnutella-svc dpts:1024:65534 reject-with icmp-port-unreachable
37   REJECT     tcp  --  anywhere             anywhere            tcp dpt:interwise reject-with icmp-port-unreachable
38   REJECT     tcp  --  anywhere             anywhere            tcp spt:interwise dpts:1024:65534 reject-with icmp-port-unreachable
39   REJECT     udp  --  anywhere             anywhere            udp spts:1024:65534 dpt:interwise reject-with icmp-port-unreachable
40   REJECT     udp  --  anywhere             anywhere            udp spt:interwise dpts:1024:65534 reject-with icmp-port-unreachable

Chain PROHIBIT (0 references)
num  target     prot opt source               destination        
1    REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited

Chain PZERO (2 references)
num  target     prot opt source               destination        
1    DROP       tcp  --  anywhere             anywhere            tcp dpt:0
2    DROP       udp  --  anywhere             anywhere            udp dpt:0
3    DROP       tcp  --  anywhere             anywhere            tcp spt:0
4    DROP       udp  --  anywhere             anywhere            udp spt:0

Chain RESET (0 references)
num  target     prot opt source               destination        
1    REJECT     tcp  --  anywhere             anywhere            reject-with tcp-reset

Chain TALLOW (2 references)
num  target     prot opt source               destination        
1    ACCEPT     all  --  10.0.0.0/8           anywhere            
2    ACCEPT     all  --  anywhere             10.0.0.0/8          

Chain TDENY (2 references)
num  target     prot opt source               destination        
1    DROP       all  --  50.115.166.129       anywhere            
2    DROP       all  --  anywhere             50.115.166.129      
3    DROP       all  --  118.186.208.122      anywhere            
4    DROP       all  --  anywhere             118.186.208.122    
5    DROP       all  --  50.115.166.253       anywhere            
6    DROP       all  --  anywhere             50.115.166.253      
7    DROP       all  --  10.7.7.3             anywhere            
8    DROP       all  --  anywhere             10.7.7.3            
9    DROP       all  --  222.186.34.162       anywhere            
10   DROP       all  --  anywhere             222.186.34.162      

Chain TGALLOW (2 references)
num  target     prot opt source               destination        

Chain TGDENY (2 references)
num  target     prot opt source               destination        

Chain TMP_DROP (2 references)
num  target     prot opt source               destination

[root@fwrtr apf]# iptables -L -t nat --line-numbers
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination        
1    DNAT       tcp  --  x.x.x.x         anywhere            tcp dpt:smtp to:x.x.x.x:xx
   #A LOT OF FORWARDED DNAT RULES HERE ALL CORRECT AND OK
75   DNAT       udp  --  x.x.x.x        anywhere            udp dpt:xxxxx to:x.x.x.x:xxxxx

Chain POSTROUTING (policy ACCEPT)
num  target     prot opt source               destination        
1    SNAT       all  --  10.0.0.0/8           anywhere            to:<wan1 gateway>
2    SNAT       all  --  10.0.0.0/8           anywhere            to:<wan2 gateway>
3    MASQUERADE  all  --  anywhere             anywhere            
4    MASQUERADE  all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination

[root@fwrtr apf]# iptables -L -t mangle --line-numbers
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination        
1    TOS        tcp  --  anywhere             anywhere            tcp spt:ftp TOS set 0x08/0xff
2    TOS        udp  --  anywhere             anywhere            udp spt:ftp TOS set 0x08/0xff
3    TOS        tcp  --  anywhere             anywhere            tcp spt:ftp-data TOS set 0x08/0xff
4    TOS        udp  --  anywhere             anywhere            udp spt:ftp-data TOS set 0x08/0xff
5    TOS        tcp  --  anywhere             anywhere            tcp spt:http TOS set 0x08/0xff
6    TOS        udp  --  anywhere             anywhere            udp spt:http TOS set 0x08/0xff
7    TOS        tcp  --  anywhere             anywhere            tcp spt:smtp TOS set 0x10/0xff
8    TOS        udp  --  anywhere             anywhere            udp spt:smtp TOS set 0x10/0xff
9    TOS        tcp  --  anywhere             anywhere            tcp spt:pop3 TOS set 0x10/0xff
10   TOS        udp  --  anywhere             anywhere            udp spt:pop3 TOS set 0x10/0xff
11   TOS        tcp  --  anywhere             anywhere            tcp spt:imap TOS set 0x10/0xff
12   TOS        udp  --  anywhere             anywhere            udp spt:imap TOS set 0x10/0xff
13   TOS        tcp  --  anywhere             anywhere            tcp spts:exec:65535 TOS and 0x00
14   TOS        udp  --  anywhere             anywhere            udp spts:biff:65535 TOS and 0x00
15   CN2        all  --  anywhere             anywhere            state NEW statistic mode nth every 2
16   CN1        all  --  anywhere             anywhere            state NEW statistic mode nth every 2 packet 1
17   REST1      all  --  anywhere             anywhere            state RELATED,ESTABLISHED
18   RESW1      all  --  anywhere             anywhere            state RELATED,ESTABLISHED
19   RESW2      all  --  anywhere             anywhere            state RELATED,ESTABLISHED

Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination        

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination        
1    RESFW1     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
2    RESFW2     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
3    RESFW1     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
4    RESFW2     all  --  anywhere             anywhere            state RELATED,ESTABLISHED

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination        

Chain POSTROUTING (policy ACCEPT)
num  target     prot opt source               destination        
1    RESW1      all  --  anywhere             anywhere            state RELATED,ESTABLISHED
2    RESW2      all  --  anywhere             anywhere            state RELATED,ESTABLISHED
3    TOS        tcp  --  anywhere             anywhere            tcp dpt:ftp TOS set 0x08/0xff
4    TOS        udp  --  anywhere             anywhere            udp dpt:ftp TOS set 0x08/0xff
5    TOS        tcp  --  anywhere             anywhere            tcp dpt:ftp-data TOS set 0x08/0xff
6    TOS        udp  --  anywhere             anywhere            udp dpt:ftp-data TOS set 0x08/0xff
7    TOS        tcp  --  anywhere             anywhere            tcp dpt:http TOS set 0x08/0xff
8    TOS        udp  --  anywhere             anywhere            udp dpt:http TOS set 0x08/0xff
9    TOS        tcp  --  anywhere             anywhere            tcp dpt:smtp TOS set 0x10/0xff
10   TOS        udp  --  anywhere             anywhere            udp dpt:smtp TOS set 0x10/0xff
11   TOS        tcp  --  anywhere             anywhere            tcp dpt:pop3 TOS set 0x10/0xff
12   TOS        udp  --  anywhere             anywhere            udp dpt:pop3 TOS set 0x10/0xff
13   TOS        tcp  --  anywhere             anywhere            tcp dpt:imap TOS set 0x10/0xff
14   TOS        udp  --  anywhere             anywhere            udp dpt:imap TOS set 0x10/0xff
15   TOS        tcp  --  anywhere             anywhere            tcp dpts:exec:65535 TOS and 0x00
16   TOS        udp  --  anywhere             anywhere            udp dpts:biff:65535 TOS and 0x00

Chain CN1 (1 references)
num  target     prot opt source               destination        
1    MARK       all  --  anywhere             anywhere            MARK set 0x1
2    CONNMARK   all  --  anywhere             anywhere            CONNMARK save
3    LOG        all  --  anywhere             anywhere            LOG level info prefix `set-mark 1: '
4    RETURN     all  --  anywhere             anywhere            

Chain CN2 (1 references)
num  target     prot opt source               destination        
1    MARK       all  --  anywhere             anywhere            MARK set 0x2
2    CONNMARK   all  --  anywhere             anywhere            CONNMARK save
3    LOG        all  --  anywhere             anywhere            LOG level info prefix `set-mark 2: '
4    RETURN     all  --  anywhere             anywhere            

Chain RESFW1 (2 references)
num  target     prot opt source               destination        
1    CONNMARK   all  --  anywhere             anywhere            CONNMARK restore
2    LOG        all  --  anywhere             anywhere            mark match !0x0 LOG level info prefix `restore-mark forward 1: '
3    RETURN     all  --  anywhere             anywhere            

Chain RESFW2 (2 references)
num  target     prot opt source               destination        
1    CONNMARK   all  --  anywhere             anywhere            CONNMARK restore
2    LOG        all  --  anywhere             anywhere            mark match !0x0 LOG level info prefix `restore-mark forward 2: '
3    RETURN     all  --  anywhere             anywhere            

Chain REST1 (1 references)
num  target     prot opt source               destination        
1    CONNMARK   all  --  anywhere             anywhere            CONNMARK restore
2    LOG        all  --  anywhere             anywhere            mark match !0x0 LOG level info prefix `restore-mark ten1: '
3    RETURN     all  --  anywhere             anywhere            

Chain RESW1 (2 references)
num  target     prot opt source               destination        
1    CONNMARK   all  --  anywhere             anywhere            CONNMARK restore
2    LOG        all  --  anywhere             anywhere            mark match !0x0 LOG level info prefix `restore-mark wan1: '
3    RETURN     all  --  anywhere             anywhere            

Chain RESW2 (2 references)
num  target     prot opt source               destination        
1    CONNMARK   all  --  anywhere             anywhere            CONNMARK restore
2    LOG        all  --  anywhere             anywhere            mark match !0x0 LOG level info prefix `restore-mark wan2: '
3    RETURN     all  --  anywhere             anywhere

Also, I tried a nexthop ip rule and route set, but it didn't work either.

May have something inherently wrong with this box somewhere. May have to build and test with another box.

You have so much duplication.  You also need to make sure to add the -i <interface> to which rules apply.
i.e. wan1/wan2

Look into getting fwbuilder http://www.fwbuilder.org/ this may help in building your iptables rules. You need to use logdrop, logaccept, logreject to see which rules hit.

http://www.linuxquestions.org/questions/linux-security-4/iptables-logging-385165/

Part of the reason is because I build it with Advanced Policy Firewall. My favorite till recently. I really am considering pfsense, it's pretty amazing.

I'm trying bonding at some point, and if bonding doesn't work, I'm going to build a new dedicated fwrtrbal (firewall & router & balancer) box with a cheap intel atom and a dual wan nic card.

Thanks for all the help everyone who wrote, I REALLY appreciate all your time and energy. God knows I've spent about 30 hours on this lol. Boot camp for linux ip route dummies with ip tables on top.... w00t!

You can not bond interface from different vendors.

It's a dual nic card, so bonding should work. I will find out soon.

It can bond, you just need to decide the type of bond you want.
load balance (there are two types 802.3ad and kernel), failover, etc.

http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding

It's easier to use PFSense my centos doesn't do it "out of the box"...  I'm sure there is a way. :)

Bond is a manual.
Create an entry in modprobe.d/bond.conf to define a bond interface
Then copy ifcfg-eth0 script to ifcfg-bond0
Change the settings within including HWADDRESS which defines the MAC address that the two subordinated interfaces will use.
The follow is written up:

http://wiki.centos.org/TipsAndTricks/BondingInterfaces

Yes I did everything correctly and set the bond up.

I cat /proc/net/bonding/bond0 and it said bonding was up and links were included but I couldn't ping out from the box.

Maybe bonding doesn't work with DHCP connections. I tried changing to "static" using my DHCP information, but it didn't work.

Bond works with DHCP.  Without looking at your configuration it is hard to tell what may have gone wrong. The only possible issue might be that you did not include HWADDRESS from one of the NICs (eth0/eth1)
ifconfig -a
HWADDR=

This way both slave (eth0/eth1) will have the same MAC Address and will get the same IP passed from the bond0 interface.

Yes the macs were setup correctly and then changed automatically by bonding to the same mac.

I'm providing my own comment as the solution because no solution was found: My findings indicate that NO load balancing solution of any kind exists for my software / hardware setup.

I think it is a good move to use pfSense. It works so well you find setting up advanced stuff like policy routing, BGP and QoS a real joy.
You can even build powerful embedded solutions based on PCengies ALIX boards.

I see it this way:
At the end I know I could do all this with editing config files and manually installing packages or even compile my own stuff; but I will be the only one who will be able to administer such a custom solution in the end.
I would need to write a real long manual and revision it all the time I make changes, since I normally do not use WAN load balancing at home.

Right! My parts came in today, put it together... Just have to install PFSense and start the balancing.

The issue with CentOS was that I should have reinstalled and chosen "load balancing" options during the install phase.

I think my kernel didn't have the correct modules or compilation options to do what I needed to do with it.

I built a PFSense box with 2 dedicated WANs connecting to 2 broadband connections and 1 LAN that connects to my linux router box. This works GREAT! I was able to setup DHCP on the wans and use OUTBOUND LOAD BALANCING perfectly with it!

IN BOUND load balancing is handled by my forward router (held in a data center a few states away from me)

Thanks for all the help on this! SO happy it's resolved now :-)