How to identify if a bot is indexing a web page


Since we started with a paid campaign in google adwords, we have noticed that the visits from countries like Russia have been increased exponentially (represent almost the 50% of the whole visits), even with the fact that the campaign is only targeted at North America.

Is this a normal situation?

Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Dave BaldwinFixer of ProblemsCommented:
Yes.  Any advertising brings in people from all over the world.  The Russians are looking to see if you have anything worth taking.  You can look in your server logs to see which of the 'legitimate' bots are indexing your site.  The not-so-legitimate bots look just like web browsers and you will only be able to tell they're bots if they try too hard and download too many files at once.  But then, you competition will do that too.

This looks like a good list of 'legitimate' bots:

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Scott Fell, EE MVEDeveloper & EE ModeratorCommented:
First, verify that those visits are directly related to the advertising campaign.  Double check your campaign as it is is easy to be confused.  Google Adwords is not as simple and straightforward as you would think and many people end up targeting the wrong thing without realizing it.    I think they will offer live support for free within the first 30 days and you can allow somebody from Google to log into your campaign and verify everything is set up properly.
David H.H.LeeCommented:
dimensionav, you may revise again your adwords campaign by adding more negative keywords to prevent unwanted traffic. You may follow through further Google Adwords best practices to make sure you're targeting the right audience with the right budget.  

Lastly, If you believe your site's ranking is being harmed by low-quality links you do not control, you can ask Google not to take them into account when assessing your site.
Lucas BishopClick TrackerCommented:
When you configure the location settings in your Adwords account, you can configure who is shown your ads. This is the default setting:
You can see that people "who show interest in my targeted location" is part of the default setting. Hence, if you have a product/page that is of interest to people in Russia, even if you've selected "United States" as your target location, you will still be marketing to Russia.

In Google Analytics you can drill down at the Adwords-Campaign level to see where your ads are showing. For example, navigate in Google Analytics to:
Acquisition > All Traffic > Channels > Paid Search

On this report, you can select "Secondary Dimension = Users>Country" (or Users>City, etc.)

Now you'll see a breakout of keyword and country. You can change the primary dimension to something else if you'd prefer not to filter by keyword.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Internet Marketing

From novice to tech pro — start learning today.