Upgrade Squid v2 to Squid v3
Hi
I have some troubling after upgrading Squid from version 2 to version 3. As I have read it is som changes to be made to the config, but i dont know what.
The old server has no iptables-rules and this is the working config:
I have tried this config but replaced
with
But it is still not working. The traffic is getting from the client this way:
Client -> Checkpoint FW (HTTP_mapped rule, SRV_REDIRECT(80,10.235.16.
Any idea?
I have some troubling after upgrading Squid from version 2 to version 3. As I have read it is som changes to be made to the config, but i dont know what.
The old server has no iptables-rules and this is the working config:
http_port 8080 transparent
cache_dir ufs /var/spool/squid3 100 16 256
cache_mem 512 MB
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
acl manager url_regex -i ^cache_object:// +i ^https?://[^/]+/squid-internal-mgr/
acl localhost src 127.0.0.1/32 ::1
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 ::1
acl localnet src 10.0.0.0/8 # RFC 1918 possible internal network
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
acl fm dstdomain intranett.fm.no
acl nrk dstdomain tv.nrk.no
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost
http_access allow localnet
always_direct allow fm
always_direct allow nrk
http_access deny all
I have tried this config but replaced
http_port 8080 transparent
with
http_port 9090
http_port 8080 intercept
But it is still not working. The traffic is getting from the client this way:
Client -> Checkpoint FW (HTTP_mapped rule, SRV_REDIRECT(80,10.235.16.
Any idea?
Does the request to port 8080 seen, reported by squid v3 or is it not being passed?
Presumably you added the iptables rule on the system where squid is installed to allow incoming port 8080 traffic in
iptables -t filter -L
Iptables -t filter -I INPUT 4 -p TCP -dport 8080 -j ACCEPT
THE rule would apply when added, iptables-save to commit and make it permanent.
https://wiki.squid-cache.org/ConfigExamples#Interception
Make sure you allow the system where the new version is installed through the checkpoint fw on port 80.
Presumably you added the iptables rule on the system where squid is installed to allow incoming port 8080 traffic in
iptables -t filter -L
Iptables -t filter -I INPUT 4 -p TCP -dport 8080 -j ACCEPT
THE rule would apply when added, iptables-save to commit and make it permanent.
https://wiki.squid-cache.org/ConfigExamples#Interception
Make sure you allow the system where the new version is installed through the checkpoint fw on port 80.
The client have IP: 10.235.1.85
The clients gw is checkpoint: 10.235.1.1
The squid box is 10.235.16.2 - connected to the same CheckPoint. Internet access is through the CP FW.
I have no iptables rules at all. Everything on the squid box is allowed.
The squid is reporting the following:
Debug:
access.log
iptables
The clients gw is checkpoint: 10.235.1.1
The squid box is 10.235.16.2 - connected to the same CheckPoint. Internet access is through the CP FW.
I have no iptables rules at all. Everything on the squid box is allowed.
The squid is reporting the following:
Debug:
2017/11/23 09:00:48| WARNING: Forwarding loop detected for:
GET /Versions/ HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: nb-NO,nb;q=0.9,no-NO;q=0.8,no;q=0.6,nn-NO;q=0.5,nn;q=0.4,en-US;q=0.3,en;q=0.1
Accept-Encoding: gzip, deflate
Referer: https://www.google.no/
Upgrade-Insecure-Requests: 1
Via: 1.1 fmheproxy (squid/3.5.12)
X-Forwarded-For: 10.235.1.85
Cache-Control: max-age=0
Connection: keep-alive
Host: www.squid-cache.org
access.log
1511424048.992 0 10.235.16.2 TCP_MISS/403 4413 GET http://www.squid-cache.org/Versions/ - HIER_NONE/- text/html
1511424048.992 1 10.235.1.85 TCP_MISS/403 4515 GET http://www.squid-cache.org/Versions/ - ORIGINAL_DST/10.235.16.2 text/html
iptables
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
The loop error means your checkpoint fw is redirecting the request from the proxy back to itself, make sure you exempt the 10.235.16.2 ip by allowing it to pass directly to the Internet for port 80 requests.
You shoukd have a rule that says if 10.235.16.2 is requesting port 80 to pass such that the rule destination port 80 refirect to 10.235.16.2:8080 would not apply.
You shoukd have a rule that says if 10.235.16.2 is requesting port 80 to pass such that the rule destination port 80 refirect to 10.235.16.2:8080 would not apply.
Thanks. This fixed the first error. Now I am getting this:
Thought this maybe was because the new version of squid maybe requires the NAT to happen in the kernel. Then tried to use this lines of iptables rules:
Anything to get me in the right direction
2017/11/23 15:55:25| ERROR: NF getsockopt(ORIGINAL_DST) failed on local=10.235.16.2:8080 remote=10.235.1.85:55099 FD 13 flags=33: (92) Protocol not available
2017/11/23 15:55:25| ERROR: NAT/TPROXY lookup failed to locate original IPs on local=10.235.16.2:8080 remote=10.235.1.85:55099 FD 13 flags=33
Thought this maybe was because the new version of squid maybe requires the NAT to happen in the kernel. Then tried to use this lines of iptables rules:
iptables -t nat -A PREROUTING -p tcp --dport 8080 -j ACCEPT
iptables -t nat -A PREROUTING -p tcp --dport 8080 -j REDIRECT --to-port 8081
iptables -t nat -A POSTROUTING -j MASQUERADE
The following error was encountered while trying to retrieve the URL: http://www.h-a.no/
Connection to 10.235.16.2 failed.
The system returned: (111) Connection refused
Anything to get me in the right direction
iptables is a software firewall runnning on the linux system where squid is? or your iptables is from somewhere else.
please post after reduction of PUBLIC IP if any or any identifying markings
iptable -t filter -L
You merely need to allow the request to come through. not sure why you are
You do not have NAT so the POSTROUTING which is the outgoing traffic is hitting ......an issue.
the only thing you need for iptables on the system where squid proxy is running, is
iptables -t filter -I INPUT 4 -p tcp -m tcp --dport 8080 -j ACCEPT
I am working based on the premise you have:
Internet <=> CHECKPOINT FW <=> LAN SWITCH <=> Linux SQUID BOX (iptables running)
disable iptables for a second and see whether the proxy starts to work
your ACL rules double check whether you allow your LAN to make requests.
to test while on the system where squid is running
telnet localhost 8080
GET https://www.experts-exchange.com/ HTTP/1.1
Host:
Referrer:
see if you get a response with the data from experts exchange to confirm the squid is functional
Trying to address the following:
NAT/TPROXY lookup failed to locate original IPs on local=10.235.16.2:8080 remote=10.235.1.85:55099 FD 13 flags=33
look at the netstat -rn on the linux system looking at the routing table.
please post after reduction of PUBLIC IP if any or any identifying markings
iptable -t filter -L
You merely need to allow the request to come through. not sure why you are
You do not have NAT so the POSTROUTING which is the outgoing traffic is hitting ......an issue.
the only thing you need for iptables on the system where squid proxy is running, is
iptables -t filter -I INPUT 4 -p tcp -m tcp --dport 8080 -j ACCEPT
I am working based on the premise you have:
Internet <=> CHECKPOINT FW <=> LAN SWITCH <=> Linux SQUID BOX (iptables running)
disable iptables for a second and see whether the proxy starts to work
your ACL rules double check whether you allow your LAN to make requests.
to test while on the system where squid is running
telnet localhost 8080
GET https://www.experts-exchange.com/ HTTP/1.1
Host:
Referrer:
see if you get a response with the data from experts exchange to confirm the squid is functional
Trying to address the following:
NAT/TPROXY lookup failed to locate original IPs on local=10.235.16.2:8080 remote=10.235.1.85:55099 FD 13 flags=33
look at the netstat -rn on the linux system looking at the routing table.
Yeah - forget what I was doing with iptables. Restored to only 1 accept rule. The connection i broken before squid handles the query. Am i missing something on the server?
Some more information:
Some more information:
root@fmheproxy:~# iptables -t filter -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT tcp -- anywhere anywhere tcp dpt:http-alt
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
root@fmheproxy:~#
root@fmheproxy:~# telnet localhost 8080
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Connection closed by foreign host.
root@fmheproxy:~#
root@fmheproxy:/etc/squid# squid3 -N -d 100
2017/11/24 01:54:19| Current Directory is /etc/squid
2017/11/24 01:54:19| Starting Squid Cache version 3.5.12 for x86_64-pc-linux-gnu...
2017/11/24 01:54:19| Service Name: squid
2017/11/24 01:54:19| Process ID 1434
2017/11/24 01:54:19| Process Roles: master worker
2017/11/24 01:54:19| With 1024 file descriptors available
2017/11/24 01:54:19| Initializing IP Cache...
2017/11/24 01:54:19| DNS Socket created at [::], FD 8
2017/11/24 01:54:19| DNS Socket created at 0.0.0.0, FD 9
2017/11/24 01:54:19| Adding nameserver 10.235.1.23 from /etc/resolv.conf
2017/11/24 01:54:19| Logfile: opening log stdio:/var/log/squid/access.log
2017/11/24 01:54:19| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec
2017/11/24 01:54:19| Logfile: opening log stdio:/var/log/squid/store.log
2017/11/24 01:54:19| Swap maxSize 0 + 262144 KB, estimated 20164 objects
2017/11/24 01:54:19| Target number of buckets: 1008
2017/11/24 01:54:19| Using 8192 Store buckets
2017/11/24 01:54:19| Max Mem size: 262144 KB
2017/11/24 01:54:19| Max Swap size: 0 KB
2017/11/24 01:54:19| Using Least Load store dir selection
2017/11/24 01:54:19| Current Directory is /etc/squid
2017/11/24 01:54:19| Finished loading MIME types and icons.
2017/11/24 01:54:19| HTCP Disabled.
2017/11/24 01:54:19| Pinger socket opened on FD 15
2017/11/24 01:54:19| Squid plugin modules loaded: 0
2017/11/24 01:54:19| Adaptation support is off.
2017/11/24 01:54:19| Accepting HTTP Socket connections at local=[::]:9090 remote=[::] FD 12 flags=9
2017/11/24 01:54:19| Accepting NAT intercepted HTTP Socket connections at local=0.0.0.0:8080 remote=[::] FD 13 flags=41
2017/11/24 01:54:20| storeLateRelease: released 0 objects
2017/11/24 01:54:28| ERROR: NF getsockopt(ORIGINAL_DST) failed on local=127.0.0.1:8080 remote=127.0.0.1:53002 FD 14 flags=33: (92) Protocol not available
2017/11/24 01:54:28| ERROR: NAT/TPROXY lookup failed to locate original IPs on local=127.0.0.1:8080 remote=127.0.0.1:53002 FD 14 flags=33
root@fmheproxy:/etc/squid# squid -v
Squid Cache: Version 3.5.12
Service Name: squid
Ubuntu linux
configure options: '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=${prefix}/lib/squid3' '--srcdir=.' '--disable-maintainer-mode' '--disable-dependency-tracking' '--disable-silent-rules' 'BUILDCXXFLAGS=-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' '--libexecdir=/usr/lib/squid' '--mandir=/usr/share/man' '--enable-inline' '--disable-arch-native' '--enable-async-io=8' '--enable-storeio=ufs,aufs,diskd,rock' '--enable-removal-policies=lru,heap' '--enable-delay-pools' '--enable-cache-digests' '--enable-icap-client' '--enable-follow-x-forwarded-for' '--enable-auth-basic=DB,fake,getpwnam,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB' '--enable-auth-digest=file,LDAP' '--enable-auth-negotiate=kerberos,wrapper' '--enable-auth-ntlm=fake,smb_lm' '--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,unix_group,wbinfo_group' '--enable-url-rewrite-helpers=fake' '--enable-eui' '--enable-esi' '--enable-icmp' '--enable-zph-qos' '--enable-ecap' '--disable-translation' '--with-swapdir=/var/spool/squid' '--with-logdir=/var/log/squid' '--with-pidfile=/var/run/squid.pid' '--with-filedescriptors=65536' '--with-large-files' '--with-default-user=proxy' '--enable-build-info=Ubuntu linux' '--enable-linux-netfilter' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wall' 'LDFLAGS=-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security'
root@fmheproxy:/etc/squid# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 10.235.16.1 0.0.0.0 UG 0 0 0 ens160
10.235.16.0 0.0.0.0 255.255.255.0 U 0 0 0 ens160
root@fmheproxy:/etc/squid#
Tried to connecy directly via telnet to port 9090 which is the normal proxy port in my config. This is working, but only from telnet and in browser settings.
The resolv.conf is including my internal dns. Tried nslookup, and is can resolv both local and external addresses.
This is the cfg:
How do i enable more debugging?
root@fmheproxy:/etc/squid# tail /var/log/squid/access.log
1511511574.852 234 ::1 TCP_MISS/301 907 GET http://www.experts-exchange.com/ - HIER_DIRECT/104.20.169.10 -
The resolv.conf is including my internal dns. Tried nslookup, and is can resolv both local and external addresses.
This is the cfg:
root@fmheproxy:/etc/squid# cat squid.conf
http_port 9090
http_port 0.0.0.0:8080 intercept
cache_access_log stdio:/var/log/squid/access.log
cache_store_log stdio:/var/log/squid/store.log
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
acl manager url_regex -i ^cache_object:// +i ^https?://[^/]+/squid-internal-mgr/
acl localnet src 10.0.0.0/8 # RFC 1918 possible internal network
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
acl fm dstdomain intranett.DOMAIN.no
acl nrk dstdomain tv.nrk.no
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost
http_access allow localnet
always_direct allow fm
always_direct allow nrk
http_access deny
How do i enable more debugging?
Debug_options="ALL,1"
Increasing the number will make the verbosity of data in a log.
Squid-cache.org has the reference
Not sure whether it has to be assigned, or in the following format.
Debug_options "ALL,1"
Besides browser, where do you need it to work?
Not sure I understand the current issue, did it not start to work? Or you have other devices for which it does not work?
Increasing the number will make the verbosity of data in a log.
Squid-cache.org has the reference
Not sure whether it has to be assigned, or in the following format.
Debug_options "ALL,1"
Besides browser, where do you need it to work?
Not sure I understand the current issue, did it not start to work? Or you have other devices for which it does not work?
If i remove the intercept and only use "http_port 8080" it is working when putting the proxy-address direct under the settings in the browser. I dont want settings in the browser, but using checkpoint fw to redirect (ps. redirect with http_access rule - not NAT) all traffic on port 80 to 8080 on the squid box. As shown in the first post. I do need the intercept option on, and thats where the "Protocol not available" is coming. I have searched all the web for this but cant find a solution. It seems to me it maybe has something with the kernel to do.
This is working on the old server. The FW-rules is not changed, the ip-address of the new server is the same as the old was, the config is the same. Just replaced "transparent" with "intercept" as this was deprecated in the new version.
So the only two thing that is diffent from the old working server and the new server is:
* Newer version of Ubuntu
* Newer version of Squid
As I can see from forum posts the error is coming because the kernel has no clue of the NAT source.
Quote: The first error means the kernel NAT tables do not have any record of
the connection that arrived on the Squid intercept port.
Just to be clear, this is still the problem:
If I do not use intercept I get invalid URL when redirecting with FW. I ask for www.test.com/test and the log telling me URL: /test is invalid. So it does not keep the domain after redirecting.
This is working on the old server. The FW-rules is not changed, the ip-address of the new server is the same as the old was, the config is the same. Just replaced "transparent" with "intercept" as this was deprecated in the new version.
So the only two thing that is diffent from the old working server and the new server is:
* Newer version of Ubuntu
* Newer version of Squid
As I can see from forum posts the error is coming because the kernel has no clue of the NAT source.
Quote: The first error means the kernel NAT tables do not have any record of
the connection that arrived on the Squid intercept port.
Just to be clear, this is still the problem:
2017/11/23 15:55:25| ERROR: NF getsockopt(ORIGINAL_DST) failed on local=10.235.16.2:8080 remote=10.235.1.85:55099 FD 13 flags=33: (92) Protocol not available
2017/11/23 15:55:25| ERROR: NAT/TPROXY lookup failed to locate original IPs on local=10.235.16.2:8080 remote=10.235.1.85:55099 FD 13 flags=33
If I do not use intercept I get invalid URL when redirecting with FW. I ask for www.test.com/test and the log telling me URL: /test is invalid. So it does not keep the domain after redirecting.
Not sure, transparent deals with redorected packets by the edge firewall/rputer of requests matching ...
You may wish to define what you want this setup to do and work towards that
Difficulty, until you cut over you will not see.
One option, add a rule on the firewall to refirect a requests from your workstation's ip to the new transparent proxy and text/work it this way.
Adding the proxy into the browser, is not the same as the redirected requests.......
You may wish to define what you want this setup to do and work towards that
Difficulty, until you cut over you will not see.
One option, add a rule on the firewall to refirect a requests from your workstation's ip to the new transparent proxy and text/work it this way.
Adding the proxy into the browser, is not the same as the redirected requests.......
I have now replaced the firewall rule redirecting to the new proxy and have a hit percent in cache after a while for around 40%. I think this problem is solved. I do believe that with the intercept option it requires use of NAT on the firewall and not a plain REDIRECT. That is not an option in my case. But finally it is working as it should. Thanks
Gain unlimited access to on-demand training courses with an Experts Exchange subscription.
Get AccessWhy Experts Exchange?
Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.
Jim Murphy
Programmer at Smart IT Solutions
When asked, what has been your best career decision?
Deciding to stick with EE.
Mohamed Asif
Technical Department Head
Being involved with EE helped me to grow personally and professionally.
Carl Webster
CTP, Sr Infrastructure Consultant
Did You Know?
We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE
Ask ANY Question
Connect with Certified Experts to gain insight and support on specific technology challenges including:
- Troubleshooting
- Research
- Professional Opinions