CentOS firewall issue

I am running the latest version of CentOS with the latest version of Apache on 2 boxes as a reverse proxy.

My Cisco load balancer is not sending any traffic to box 1, while box 2 gets all the traffic.

In troubleshooting I see this:

DMZ10-1#telnet 192.168.10.52 80
Trying 192.168.10.52, 80 ...
% Connection refused by remote host
 
DMZ10-1#telnet 192.168.10.52 443
Trying 192.168.10.52, 443 ...
% Connection refused by remote host
 
DMZ10-1#telnet 192.168.10.53 80
Trying 192.168.10.53, 80 ... Open
^C
DMZ10-1#telnet 192.168.10.53 443
Trying 192.168.10.53, 443 ... Open


The first 2 attempts above are to box 1 & the 2nd 2 attempts to box 2.

Both boxes should be configured exactly the same, they both have the same settings under firewall options: SSH, HTTPS & HTTP are trusted services & no other ports are enabled on either box. SELinux is turned off on both boxes.

where to start the troubleshooting?
someITGuyAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

eleibowitzCommented:
This may be a routing issue.  While there are several potential causes, the most likely cause is the default route on the CentOS servers.  Unless you are doing something unusual, the default route should be the load balancer.  Also look for any potential causes of asymetric routing.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
someITGuyAuthor Commented:
The default route is the same for both:

Box 1:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.10.0    *               255.255.255.0   U     0      0        0 eth0
169.254.0.0     *               255.255.0.0     U     0      0        0 eth0
default         192.168.10.1    0.0.0.0         UG    0      0        0 eth0


Box 2:

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.10.0    *               255.255.255.0   U     0      0        0 eth0
169.254.0.0     *               255.255.0.0     U     0      0        0 eth0
default         192.168.10.1    0.0.0.0         UG    0      0        0 eth0

They both point to the router at .1 not the Cisco load balancer in the same subnet, but only box 2 gets any traffic....

My incoming traffic comes to these boxes from the Cisco load balancer, are you saying out-bound traffic (to my internal LAN) should also go through the Cisco load balancer?

someITGuyAuthor Commented:
My biggest issue is that I am not getting any inbound traffic to Box 1
Starting with Angular 5

Learn the essential features and functions of the popular JavaScript framework for building mobile, desktop and web applications.

someITGuyAuthor Commented:
Box 1 worked until recently, I can't find any changes though.
TRW-ConsultingCommented:
what is the output of 'iptables -L' and 'netstat -na' on each box?
Kerem ERSOYPresidentCommented:
Hi,

First of all to veirfy that this is not a connection issue:

just try to ping both boxes from the load balancer. If it succeeds go to the next step.

Then check if firewall is active with this command

service iptable status

The outpur should contain:
12   ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:80
15   ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:443

(12 and 15 are the sequence numbers and could be different than the numbers listed here)

Please check both boxes for /etc/sysconfig/iptables and see if both have these:

-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT

If not add these lines and restart firewall:

service iptables restart.

Since it was working until recently it gets me thinking that the problem is the Apache daemon is not working on the Box 1..

Will you please check if it is working with these command and post the output here:

netstat -anpt | grep http

Do you see anything like :

tcp        0      0 :::80                       :::*                        LISTEN      3717/httpd          
tcp        0      0 :::443                      :::*                        LISTEN      3717/httpd  

in the output?

If not please try to restart the daemon :

service httpd restart

If this fails please attach the output from :

/var/log/messages
/var/log/httpd/access_log and /var/log/httpd/error_log


Cheers,
K.


someITGuyAuthor Commented:
KeremE:

Everything checked out except the netstat, it looks like httpd is not running. When I try to start it I get this error:

[root@exxxpxv1 ~]# service httpd start
Starting httpd: Syntax error on line 115 of /etc/httpd/conf/httpd.conf:
Invalid command 'DirectoryIndex', perhaps misspelled or defined by a module not included in the server configuration
                                                           [FAILED]
 


I checked the httpd.conf file on both boxes & other than IP's & hostnames everything is the same...
TRW-ConsultingCommented:
This has changed from a firewall issue to an httpd.conf issue.  When you say the only difference between a good one and the bad one is the IP's and hostnames, was this determination based on visual inspection, or did you run the 'diff' command on them?  You may have to post the httpd.conf file, or at least the offending portion of it (removing or masking private information), for it to be debugged any further.
someITGuyAuthor Commented:
Agreed, the firewall is not the issue.

Do you folks feel I should create a new question? I can give all of you credit for this one since you all had great troubleshooting advice?
Kerem ERSOYPresidentCommented:
It seems that the problem is the HTTPD service is not working as I've indicated earlier,  but not a firewalling issue. So you'd better close the question and open another one.

But you might as well try copying the working configuration from the running one assuming that both servers are identical in configuration and save your time instead :))

cheers,
K
someITGuyAuthor Commented:
Here is the offending section (according to the error message) of httpd.conf from Box 1:


DirectoryIndex index.html index.html.var
AccessFileName .htaccess
<Files ~ "^\.ht">
    Order allow,deny
    Deny from all
</Files>
TypesConfig /etc/mime.types
DefaultType text/plain
<IfModule mod_mime_magic.c>
    MIMEMagicFile conf/magic
</IfModule>

HostnameLookups Off
ErrorLog logs/owa-error_log

LogLevel info
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

CustomLog logs/access_log combined
ServerSignature On

Alias /icons/ "/var/www/icons/"
<Directory "/var/www/icons">
    Options Indexes MultiViews
    AllowOverride None
    Order allow,deny
    Allow from all
</Directory>

<IfModule mod_dav_fs.c>
    # Location of the WebDAV lock database.
    DAVLockDB /var/lib/dav/lockdb
</IfModule>

ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"
<Directory "/var/www/cgi-bin">
    AllowOverride None
    Options None
    Order allow,deny
    Allow from all
</Directory>

IndexOptions FancyIndexing VersionSort NameWidth=* HTMLTable
AddIconByEncoding (CMP,/icons/compressed.gif) x-compress x-gzip





Here is the same section of httpd.conf from Box 2:

DirectoryIndex index.html index.html.var
AccessFileName .htaccess
<Files ~ "^\.ht">
    Order allow,deny
    Deny from all
</Files>
TypesConfig /etc/mime.types
DefaultType text/plain
<IfModule mod_mime_magic.c>
    MIMEMagicFile conf/magic
</IfModule>

HostnameLookups Off
ErrorLog logs/owa-error_log

LogLevel info
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

CustomLog logs/access_log combined
ServerSignature On

Alias /icons/ "/var/www/icons/"
<Directory "/var/www/icons">
    Options Indexes MultiViews
    AllowOverride None
    Order allow,deny
    Allow from all
</Directory>

<IfModule mod_dav_fs.c>
    # Location of the WebDAV lock database.
    DAVLockDB /var/lib/dav/lockdb
</IfModule>

ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"
<Directory "/var/www/cgi-bin">
    AllowOverride None
    Options None
    Order allow,deny
    Allow from all
</Directory>

IndexOptions FancyIndexing VersionSort NameWidth=* HTMLTable
AddIconByEncoding (CMP,/icons/compressed.gif) x-compress x-gzip
Kerem ERSOYPresidentCommented:
It seems that you have a problem with your mod_dir.so. Yo might need to resintall whole apache. using yum.

There's no syntax error on the configuration excerpt you've posted.

Cheers,
K.
someITGuyAuthor Commented:
someITGuyAuthor Commented:
It looks like KeremE's suggestion to copy the good httpd.conf over to box 1 & edit as appropriate did work. I am up & running.
Kerem ERSOYPresidentCommented:
Wish I had given this advice on the other question :)
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Software Firewalls

From novice to tech pro — start learning today.