[x]
Posted via EE Mobile

Search, ask, and monitor your questions on the go with EE Mobile. Visit Experts Exchange from your mobile device and never be out of touch again.
12/15/2008 at 10:43PM PST, ID: 23987699 | Points: 500
[x]
Attachment Details

Robots.txt and Twiceler-0.9

Asked by dmat1 in Search Engines, Stores & Carts

I have the following robots.txt file on a site

User-agent: *
Disallow: /cdgcart/insertcdg.asp
Disallow: /dodcart/insertdod.asp
Disallow: /rentalcart/insertrental.asp
Disallow: /hardwarecart/inserthardware.asp

I dont really know if it is working or not. I am trying to prevent the crawlers from inserting items into a shopping cart.

I see a lot of entires in the past few days from Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twic...

Am I doing something wrong in my robots.txt. Is this a rogue robot that I need to deal with?

What's a poor boy to do?

[+][-]12/16/08 01:28 AM, ID: 23182139

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]12/16/08 01:44 AM, ID: 23182204

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 30-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]12/16/08 01:48 AM, ID: 23182220

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 30-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]12/16/08 03:43 AM, ID: 23182783

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]12/16/08 06:26 AM, ID: 23183797

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]01/01/09 11:36 AM, ID: 23275836

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]06/30/09 12:23 AM, ID: 24743011

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
 
Loading Advertisement...
20091028-EE-VQP-86 - Hierarchy / EE_QW_2_20070628