Experts Exchange connects you with the people and services you need so you can get back to work.
# -- bingbot, microsoft indexer
# -- msnbot, microsoft indexer
# -- Googlebot
# -- Slurp, Yahoo indexer
# -- Default for all others
Open in new window
Twitter’s crawler respects Google’s robots.txt specification when scanning URLs. If a page with card markup is blocked, no card will be shown. If an image URL is blocked, no thumbnail or photo will be shown.
Twitter uses the User-Agent of Twitterbot (with version, such as Twitterbot/1.0), which can be used to create an exception in the robots.txt file.
Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.
Have a better answer? Share it in a comment.
Please enter a first name
Please enter a last name
Must be at least 4 characters long.
Join and Comment
From novice to tech pro — start learning today.
Premium members can enroll in this course at no extra cost.