[x]
Posted via EE Mobile

Search, ask, and monitor your questions on the go with EE Mobile. Visit Experts Exchange from your mobile device and never be out of touch again.
Question
[x]
Attachment Details

Robots.txt and Twiceler-0.9

Asked by dmat1 in Search Engines, Stores & Carts

I have the following robots.txt file on a site

User-agent: *
Disallow: /cdgcart/insertcdg.asp
Disallow: /dodcart/insertdod.asp
Disallow: /rentalcart/insertrental.asp
Disallow: /hardwarecart/inserthardware.asp

I dont really know if it is working or not. I am trying to prevent the crawlers from inserting items into a shopping cart.

I see a lot of entires in the past few days from Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twic...

Am I doing something wrong in my robots.txt. Is this a rogue robot that I need to deal with?

What's a poor boy to do?

[+][-]12/16/08 01:28 AM, ID: 23182139Expert Comment

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]12/16/08 01:44 AM, ID: 23182204Author Comment

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 30-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]12/16/08 01:48 AM, ID: 23182220Author Comment

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 30-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]12/16/08 03:43 AM, ID: 23182783Expert Comment

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]12/16/08 06:26 AM, ID: 23183797Expert Comment

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]01/01/09 11:36 AM, ID: 23275836Expert Comment

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]06/30/09 12:23 AM, ID: 24743011Expert Comment

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
 
Loading Advertisement...
20091111-EE-VQP-89 - Hierarchy / EE_QW_2_20070628