Unblock IP Addresses for Vendor


I'm really just looking for advice.  I've been asked by our marketing department to allow an outside vendor to "crawl" our website.  In order to do this, they require over 200 IP addresses be given full access to our web server on all ports.  To me, this is extremely excessive.  However, the usual response from marketing is "really big companies do this why can't we?"  Perhaps, but it seems like a huge breach of security to me.

I'm just checking to see if I'm being to strict.  Would you do this on your network?

Thanks in advance for any advice you can give me.

Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Dave BaldwinFixer of ProblemsCommented:
I wouldn't and I don't believe that "really big companies do this" is true.  Marketing departments tend to say what they think will get them what they think they want which is just as often not what is actually needed.  My main customer has asked things like this several times and changed his mind after I explained the consequences to him.  I think he went ahead and did it a couple of times anyway.
Satyendra SharmaMicrosoft UC Technical ArchitectCommented:
what do you mean by "crawl" your website?
What does your web server listens on externally port/protocol?

Doesn't make any sense to allow your external vendor on all ports.
Satyendra SharmaMicrosoft UC Technical ArchitectCommented:
The recommendation from your marketing department is a BAD idea, dealing with marketing is difficult and they would say anything to get their things moving... Just saying.
Discover the Answer to Productive IT

Discover app within WatchGuard's Wi-Fi Cloud helps you optimize W-Fi user experience with the most complete set of visibility, troubleshooting, and network health features. Quickly pinpointing network problems will lead to more happy users and most importantly, productive IT.

cwaterburyAuthor Commented:
Here's the email chain between myself and the vendor (bottom up):

Vendor Response
Our servers are working simultaneously and there are not specific ports.  In order to use
our sevices and for our system to crawl your content, the necessary IP Addresses would be to be

My Response
Thanks for the explanation.  However, that doesn't explain why I need to open up all these IP
addresses with ALL ports.  What ports specifically are needed?  I'm not willing to open up my
network to such a wide range of IPs.  If I can narrow it by port, I might consider it.

Vendor Response
Thanks for your patience here!

We uses a special user agent so that it's identifiable by site owners. This serves two

1. It makes it clear we aren't spoofing another user agent in order to scrape sites, and 2. it lets
site administrators know that we are there so they can tailor their site's response to our crawlers.

Our crawlers come from one of our three data centers. They collect your site's content so that
we can serve the most relevant recommendations based on that content. We also use that
content if you have a content discovery campaign with us. It helps us know better where to
place your content in our ad network. Our data centers are large, and they are located in
different parts of the country. As a result they use a rather large block of IP addresses. If you
enable this block of IP addresses we should be able to successfully crawl your content.  

I sent URL

Vendor Response
Thanks for reaching out.

We ask customers to whitelist our IPs so we can properly crawl your content and allow you to
run your content in our network and use our service.  Otherwise, our system will not be able
to categorize the content and recommend it to the audience that would be most interested.

Feel free to send over the URL so I can take a further look!

My initial question to the vendor

I'm being asked to unblock/whitelist over 200 IP addresses but the marketing folks can't tell
me why.  I need to know why you want these unblocked before I will do that.  I've read your
help text about Crawler Block but that doesn't have any info in it either.  Why is it necessary to
open these IP addresses?
Jan SpringerCommented:
it does not sound like the vendor wants you to open all of the ports -- just to allow 200 IPs to get your web site data and put it into their network.
cwaterburyAuthor Commented:
I do think that's what they are saying.  They have tried "crawling" our site but say they can't.  All IPs can access our website on port 80 so why do they need me to unblock over 200 IPs to do this?
Satyendra SharmaMicrosoft UC Technical ArchitectCommented:
To be honest your vendor is not making sense and you would need to talk with someone who actually knows how this works technically.

Is your websites hosted on this vendors datacenter or is it on-prem and this vendor is suppose to scan your websites to make sure it doesn't have any invalid content?  Also a published website normally listens on particular port/protocol (TCP 80 for HTTP or TCP 443 for HTTPS) and if you want to protect your website then i have seen configurations where customer do port redirection on their external facing firewall or load balance.  Anyway opening 200 ip's on all ports is not correct.

Does your vendor has any published configuration whitepaper? If they have many large datacenter i would imaging they have their configuration steps published for public.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Jan SpringerCommented:
if you have completely open access to port 80, then you need to do nothing.

verify that you don't have:
  1) a restriction in firewall throttling incoming connections to port 80
  2) that the webserver configuration does not have restricted access on subdirectories (and if it does, then need content provider should enable those retrictions, we well).

i'd ask them what kind of error they're getting.
Dave BaldwinFixer of ProblemsCommented:
I suspect this is an advertising / marketing service because all this sounds familiar.  I would never do what they are asking.  In particular, there is no one with technical info or competence that is talking to you.
cwaterburyAuthor Commented:
Very true.  I don't even have an error message.  They haven't shared that with me.

We're looking for another vendor.  They obviously aren't interested in working with any company that doesn't unblock all those IP's.

Thanks for all your input!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.