I am trying to use preg_match and trying to find if a page contains the meta tag of noindex. problem is that there are several possible tags used. Here are a list of them
All Search Engines
<meta name="robots" content="noindex" />
<meta name="googlebot" content="noindex" />
<meta name="Slurp" content="noindex" />
<meta name="msnbot" content="noindex" />
Also some meta tags might have content="noindex, follow" or content="follow, noindex"
anyone have an example preg_match that would help me find these?