Layer 7 Transparent cookie checker

We need a software which detect the cookies like httptestcookie and block the bots automatically and transparently for 80 port requests
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

You can strip cookies on webserver/load balancer/reverse proxy
It is too much weight to carry to do that in passing packet.
FireBallITAuthor Commented:
Is there any alternative method to block Botnet transparently.
On nginx proxy yes it is easy and there some modules or htaccess is a way of cleaning up botnets

but what about a general transparent prxy
It is not meant to be like that. You can try to drop such requests according to signature in packet.
IT Pros Agree: AI and Machine Learning Key

We’d all like to think our company’s data is well protected, but when you ask IT professionals they admit the data probably is not as safe as it could be.

FireBallITAuthor Commented:
Signature should change any time like
...... random algorithms which is same with normal search queries or same with some other requests.

the best way is http test cookie but we need to do it on a proxy without creating config files for each site.
Because we have hosting servers. and reseller account should open a host any time without our knowladge.
We need a general solution which works transparently
But why? Those bot scans are part of normal internet noise...
FireBallITAuthor Commented:
we are serving 100.000+ web sites from our network and some of our customers renting servers also.
So server controls are not in our hands. We are trying to put a centrilized proxy which we forwrd 80  port to on it and it check for client's browsers / location / image downloads  / and most important is accepting http cookies or not.

so our customers won't need any additional setups on their servers to block botnet
Make reverse proxy for all sites and cut them for direct access.
FireBallITAuthor Commented:
yes but for nginx reverse proxy we need to write a conf file for each web site.
actually http test cookie is a perfect tool it waits the visitor for 1 second and check for cookie no bots can pass to the php.
But it does not work for general requests.
Playnly enforcing HTTP Host: header and blacklisting all IPs connecting without one also keeps pain down.
FireBallITAuthor Commented:
we have seen 400K+ bots last week that is not a good solution for us we are getting huge attacks
400k/7/24/60/60 = request every 1.5s
Just leave it alone. nginx or apache or IIS can cope with such noise. even tomcat and heavier servers are fine.
FireBallITAuthor Commented:

that module solved the problem . We need to use asystem like it transparently
with packet every couple of seconds I think nginx is the correct final lair for countermeasure.
You should make your own conntrack module to inspect and patch packets on the fly
FireBallITAuthor Commented:
have you ever heard any thing like that before ?
If you follow EU laws you need to ask user's acknowledgment to store cookies....
So it is on slippery side of IT, especially if site behind stores no cookies.
FireBallITAuthor Commented:
Does it possible to wait on a blank page for 1 miliseconds then redirect to the visitors to the page on first get requests ?
that block %99 of bot attacks
It is called walled garden and commonly used for free access points at mcdonalds (if you look for firewall rules)
I dont think the disease is so bad to put all medicine on it. Simple syn proxy or even syncookies on the host will keep webservers happy.
FireBallITAuthor Commented:
Actually SRX 3K working perfect for SYN proxy or SYN cookie check but ref get attacks commonly comes from bot pc's without a browser supplied source.

should you redirect me to a source for that :(
Redirect all to minimal webserver that redirects to itself. in the meantime punch the hole in fw and direct it to right destination ?
FireBallITAuthor Commented:
Yes that is good to start but there is one more thing to do :)
1. is there any way to google / yahoo ...... etc ips to let inside directly without hitting proxy  ?
2. is there any way to let an ip hit this place for once ?
3. is there any way to put a page that wait and redirect to host without bring a finger print ?
1. you cannot intercept https
2. yes, iptables "tables" can sit forever
3. No - user sees that and some will complain. You can delay HTTP requests the way you do with syn proxy (though I have no clue how to accomplish such DoS prevention in Linux iptables)

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
FireBallITAuthor Commented:
I will check it today
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux Security

From novice to tech pro — start learning today.