cwaterbury
asked on
Unblock IP Addresses for Vendor
Greetings,
I'm really just looking for advice. I've been asked by our marketing department to allow an outside vendor to "crawl" our website. In order to do this, they require over 200 IP addresses be given full access to our web server on all ports. To me, this is extremely excessive. However, the usual response from marketing is "really big companies do this why can't we?" Perhaps, but it seems like a huge breach of security to me.
I'm just checking to see if I'm being to strict. Would you do this on your network?
Thanks in advance for any advice you can give me.
Chris
I'm really just looking for advice. I've been asked by our marketing department to allow an outside vendor to "crawl" our website. In order to do this, they require over 200 IP addresses be given full access to our web server on all ports. To me, this is extremely excessive. However, the usual response from marketing is "really big companies do this why can't we?" Perhaps, but it seems like a huge breach of security to me.
I'm just checking to see if I'm being to strict. Would you do this on your network?
Thanks in advance for any advice you can give me.
Chris
Dave Baldwin
I wouldn't and I don't believe that "really big companies do this" is true. Marketing departments tend to say what they think will get them what they think they want which is just as often not what is actually needed. My main customer has asked things like this several times and changed his mind after I explained the consequences to him. I think he went ahead and did it a couple of times anyway.
what do you mean by "crawl" your website?
What does your web server listens on externally port/protocol?
Doesn't make any sense to allow your external vendor on all ports.
What does your web server listens on externally port/protocol?
Doesn't make any sense to allow your external vendor on all ports.
The recommendation from your marketing department is a BAD idea, dealing with marketing is difficult and they would say anything to get their things moving... Just saying.
ASKER
Here's the email chain between myself and the vendor (bottom up):
Vendor Response
Our servers are working simultaneously and there are not specific ports. In order to use
our sevices and for our system to crawl your content, the necessary IP Addresses would be to be
unblocked.
My Response
Thanks for the explanation. However, that doesn't explain why I need to open up all these IP
addresses with ALL ports. What ports specifically are needed? I'm not willing to open up my
network to such a wide range of IPs. If I can narrow it by port, I might consider it.
Vendor Response
Thanks for your patience here!
We uses a special user agent so that it's identifiable by site owners. This serves two
purposes:
1. It makes it clear we aren't spoofing another user agent in order to scrape sites, and 2. it lets
site administrators know that we are there so they can tailor their site's response to our crawlers.
Our crawlers come from one of our three data centers. They collect your site's content so that
we can serve the most relevant recommendations based on that content. We also use that
content if you have a content discovery campaign with us. It helps us know better where to
place your content in our ad network. Our data centers are large, and they are located in
different parts of the country. As a result they use a rather large block of IP addresses. If you
enable this block of IP addresses we should be able to successfully crawl your content.
I sent URL
Vendor Response
Thanks for reaching out.
We ask customers to whitelist our IPs so we can properly crawl your content and allow you to
run your content in our network and use our service. Otherwise, our system will not be able
to categorize the content and recommend it to the audience that would be most interested.
Feel free to send over the URL so I can take a further look!
My initial question to the vendor
Greetings,
I'm being asked to unblock/whitelist over 200 IP addresses but the marketing folks can't tell
me why. I need to know why you want these unblocked before I will do that. I've read your
help text about Crawler Block but that doesn't have any info in it either. Why is it necessary to
open these IP addresses?
Vendor Response
Our servers are working simultaneously and there are not specific ports. In order to use
our sevices and for our system to crawl your content, the necessary IP Addresses would be to be
unblocked.
My Response
Thanks for the explanation. However, that doesn't explain why I need to open up all these IP
addresses with ALL ports. What ports specifically are needed? I'm not willing to open up my
network to such a wide range of IPs. If I can narrow it by port, I might consider it.
Vendor Response
Thanks for your patience here!
We uses a special user agent so that it's identifiable by site owners. This serves two
purposes:
1. It makes it clear we aren't spoofing another user agent in order to scrape sites, and 2. it lets
site administrators know that we are there so they can tailor their site's response to our crawlers.
Our crawlers come from one of our three data centers. They collect your site's content so that
we can serve the most relevant recommendations based on that content. We also use that
content if you have a content discovery campaign with us. It helps us know better where to
place your content in our ad network. Our data centers are large, and they are located in
different parts of the country. As a result they use a rather large block of IP addresses. If you
enable this block of IP addresses we should be able to successfully crawl your content.
I sent URL
Vendor Response
Thanks for reaching out.
We ask customers to whitelist our IPs so we can properly crawl your content and allow you to
run your content in our network and use our service. Otherwise, our system will not be able
to categorize the content and recommend it to the audience that would be most interested.
Feel free to send over the URL so I can take a further look!
My initial question to the vendor
Greetings,
I'm being asked to unblock/whitelist over 200 IP addresses but the marketing folks can't tell
me why. I need to know why you want these unblocked before I will do that. I've read your
help text about Crawler Block but that doesn't have any info in it either. Why is it necessary to
open these IP addresses?
it does not sound like the vendor wants you to open all of the ports -- just to allow 200 IPs to get your web site data and put it into their network.
ASKER
I do think that's what they are saying. They have tried "crawling" our site but say they can't. All IPs can access our website on port 80 so why do they need me to unblock over 200 IPs to do this?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
if you have completely open access to port 80, then you need to do nothing.
verify that you don't have:
1) a restriction in firewall throttling incoming connections to port 80
2) that the webserver configuration does not have restricted access on subdirectories (and if it does, then need content provider should enable those retrictions, we well).
i'd ask them what kind of error they're getting.
verify that you don't have:
1) a restriction in firewall throttling incoming connections to port 80
2) that the webserver configuration does not have restricted access on subdirectories (and if it does, then need content provider should enable those retrictions, we well).
i'd ask them what kind of error they're getting.
I suspect this is an advertising / marketing service because all this sounds familiar. I would never do what they are asking. In particular, there is no one with technical info or competence that is talking to you.
ASKER
Very true. I don't even have an error message. They haven't shared that with me.
We're looking for another vendor. They obviously aren't interested in working with any company that doesn't unblock all those IP's.
Thanks for all your input!
We're looking for another vendor. They obviously aren't interested in working with any company that doesn't unblock all those IP's.
Thanks for all your input!