IIS 8.5 Restrict Requests When No Referrer
We have a public facing web site from which users query data pulled from our Oracle database. It uses IIS v8.5 (pretty sure). I'm the Oracle DBA who notices that on a weekly basis, the temp tablespace fills up because of what I believe are robots sucking data out. This goes on for a few hours then quits.
Using a Log Parser, I see the pattern is:
* No cs(referer) information
* It is pulling directly from the PrintDatalet.aspx
* It's using several different IP addresses that are registered in various countries including U.S. but through this leasing company LEASEWEB-NL-MNT
* The pattern is that from each IP address (there are many) it increments the number of connections before moving on to the next IP in the range.
Some may be legitimate and so I don't think I should completely block IP ranges. Plus I imagine the IP range will change for these annoying users.
I see in IIS there's a way to restrict the number of concurrent requests and I thought I'd like to experiment with this.
I'm very embryonic with IIS and would require some handholding if any suggestions are made.
Is there a way to throttle the requests based on the following?
1. Null value for cs(referer)
2. The "PrintDatalet.aspx" value for (cs-uri-stem)
Thank you.
Using a Log Parser, I see the pattern is:
* No cs(referer) information
* It is pulling directly from the PrintDatalet.aspx
* It's using several different IP addresses that are registered in various countries including U.S. but through this leasing company LEASEWEB-NL-MNT
* The pattern is that from each IP address (there are many) it increments the number of connections before moving on to the next IP in the range.
Some may be legitimate and so I don't think I should completely block IP ranges. Plus I imagine the IP range will change for these annoying users.
I see in IIS there's a way to restrict the number of concurrent requests and I thought I'd like to experiment with this.
I'm very embryonic with IIS and would require some handholding if any suggestions are made.
Is there a way to throttle the requests based on the following?
1. Null value for cs(referer)
2. The "PrintDatalet.aspx" value for (cs-uri-stem)
Thank you.
ASKER
Thanks Kimputer.
It appears that this rejects any connection that has an empty referer that uses any "aspx" page. Am I interpreting it correctly?
It appears that this rejects any connection that has an empty referer that uses any "aspx" page. Am I interpreting it correctly?
Not fully reject, just serve the page you created.
ASKER
Not every request with a null referer is one I want to block. There are legitimate companies that directly pull from the PrintDatalet.aspx page but they don't send hundreds of requests at once.
There's this feature in IIS where you can restrict the concurrent requests made by an IP, IP Range or domain. "IP Address and Domain Restrictions". I'll call it throttling. But I don't know if the IP ranges change for these invasive users. So rather than plug in a bunch of IP addresses (or a range), I wondered if I could throttle based on the following two factors:
1) a call to the PrintDatalet.aspx
2) a null referer
There's this feature in IIS where you can restrict the concurrent requests made by an IP, IP Range or domain. "IP Address and Domain Restrictions". I'll call it throttling. But I don't know if the IP ranges change for these invasive users. So rather than plug in a bunch of IP addresses (or a range), I wondered if I could throttle based on the following two factors:
1) a call to the PrintDatalet.aspx
2) a null referer
No, it's quite simple. It's all described here: https://docs.microsoft.com/en-us/iis/get-started/whats-new-in-iis-8/iis-80-dynamic-ip-address-restrictions
Basically, it's IP based. Period. Not file based, not referrer based.
If you want it to be, you will have to write your own solution.
Though at no cost (since writing your own solution still costs money, because even if it's not, it still takes time, which equals to money), you can just use the IP range throttling anyway. The IP ranges really do NOT contain normal user requests. Add IP ranges as needed (after analyzing it's really from a bot)
Basically, it's IP based. Period. Not file based, not referrer based.
If you want it to be, you will have to write your own solution.
Though at no cost (since writing your own solution still costs money, because even if it's not, it still takes time, which equals to money), you can just use the IP range throttling anyway. The IP ranges really do NOT contain normal user requests. Add IP ranges as needed (after analyzing it's really from a bot)
ASKER
Thank you again Kimputer.
I don't understand your statement "The IP ranges really do NOT contain normal user requests". Would you please explain.
I don't understand your statement "The IP ranges really do NOT contain normal user requests". Would you please explain.
It's virtually not possible you're excluding normal users access to your server. The IP range you figured out from the logs, are really ONLY used by those bots, not by normal users. Whenever you detect such activity, just block the range you're seeing.
ASKER
Thanks. For further clarification.... are you saying that this is a normal thing that these particular IPs I am seeing are used only for robots? Is this a "standard"?
I'm assuming that if the bots don't get what they need, they'll switch to another range. Then I'll be seeing the same situation again soon and will have to add another range. Is that an accurate assumption?
I'm assuming that if the bots don't get what they need, they'll switch to another range. Then I'll be seeing the same situation again soon and will have to add another range. Is that an accurate assumption?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Thanks so much!
We blocked several ranges of IPs and our problem stopped right away.
As expected though, we have gotten a couple calls from legitimate customers saying they are getting 403 Forbidden errors. We see that we were too broad with one of the IP ranges simply because we don't have a great grasp on the mask part.
We set a deny for 45.0.0.0 with a mask of 255.0.0.0. But we need to adjust it to only block this range of IPs:
45.128.0.0 - 45.159.255.255
Do I specify this in IIS:
IP Address Range: 45.128.0.0
Mask or Prefix: 255.128.0.0
We blocked several ranges of IPs and our problem stopped right away.
As expected though, we have gotten a couple calls from legitimate customers saying they are getting 403 Forbidden errors. We see that we were too broad with one of the IP ranges simply because we don't have a great grasp on the mask part.
We set a deny for 45.0.0.0 with a mask of 255.0.0.0. But we need to adjust it to only block this range of IPs:
45.128.0.0 - 45.159.255.255
Do I specify this in IIS:
IP Address Range: 45.128.0.0
Mask or Prefix: 255.128.0.0
That's more like 255.224.0.0
With about 8 million IP's blocked, it's safe to say the crawler company does NOT own that many IP nr.
Even 255.255.0.0 is quite a lof for a company to own.
With about 8 million IP's blocked, it's safe to say the crawler company does NOT own that many IP nr.
Even 255.255.0.0 is quite a lof for a company to own.
ASKER
Can you help me to understand why 255.224.0.0 blocks range: 45.128.0.0 - 45.159.255.255 ?
I've googled and read some explanations on how to figure it but I need it dumbed down more.
I've googled and read some explanations on how to figure it but I need it dumbed down more.
The mask, is what DOES NOT CHANGE (converted to bits, the ones means can't change)
So having a mask 255.255.255.255 means you don't change anything, meaning 1 IP number.
A mask of 0.0.0.0 means you can change EVERYTHING, so it's 255*255*255*255 IP numbers
So a mask of 255.0.0.0 is QUITE a lot, it's from 45.0.0.0 and then 255*255*255 IP numbers, which is why so many clients complained.
A mask of 255.224.0.0 means for the 224, remains a subnet availability of 32, And that's why it's 45.128.0.0. to 45.(128+32).0.0 is 45.160.0.0 (or as you wanted it rounded down, 45.159.255.255)
So having a mask 255.255.255.255 means you don't change anything, meaning 1 IP number.
A mask of 0.0.0.0 means you can change EVERYTHING, so it's 255*255*255*255 IP numbers
So a mask of 255.0.0.0 is QUITE a lot, it's from 45.0.0.0 and then 255*255*255 IP numbers, which is why so many clients complained.
A mask of 255.224.0.0 means for the 224, remains a subnet availability of 32, And that's why it's 45.128.0.0. to 45.(128+32).0.0 is 45.160.0.0 (or as you wanted it rounded down, 45.159.255.255)
ASKER
Thank you so much Kimputer. This was very helpful. My database is happy again and those annoying bots are being blocked. So far no additional calls from customers but I know how to handle them with the masking now.
<rule name="Prevent image hotlinking">
<match url=".*\.(aspx)$"/>
<conditions>
<add input="{HTTP_REFERER}" pattern="^$" negate="true" />
</conditions>
<action type="Rewrite" url="/empty_or_warning_pag
</rule>
Obviously, add this "empty_or_warning_page.htm