404 Errors but pages exsist and appear after user hits refresh


I am experiencing some strange 404 errors with IIS. A 404 appears randomly when you try to view pages throughout the site. If you hit refresh in the browser, the page will appear. These are all ASP pages. Also, the 404 error page that appears is a default page (see below). However, IIS is suppose to be displaying a custom error page everytime. The custom page only shows up on pages that DO NOT actually exist but not for pages that are randomly not visible.  

The only thing that I know of that has changed is our traffic. Certain staff member have registered the site with free services that ping the site for uptime. The only thing I can think of is that it's almost a DOS attack.

Any thoughts or ideas would be greatly appreciated.

Here is the error:

The page cannot be found
The page you are looking for might have been removed, had its name changed, or is temporarily unavailable.


Please try the following:

If you typed the page address in the Address bar, make sure that it is spelled correctly.

Open the home page, and then look for links to the information you want.
Click the  Back button to try another link.
Click  Search to look for information on the Internet.

HTTP 404 - File not found
Internet Explorer  
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Sounds like you may have network problems, packet loss issues, dns issues, proxy server issues, etc.
I think it's more likely that you have genuine 404 issues that you just don't understand yet.

What are you seeing in your IIS access logs?  We have found IIS to be very reliable in offering custom error pages and in giving 404 page not found when the page is not found.

You've had one good idea about the cause of 404s, if you're seeing them in your logs:  incorrect URLs posted to external sites.  

One way to debug this is to google for the URLs which are giving 404s in your logs.  It might help you locate an external site with the wrong link to your site.

>>The custom page only shows up on pages that DO NOT actually exist but not for pages that are randomly not visible.  

This sounds as if the custom error page for 404 is showing up when you expect.

What do you mean by pages "that are randomly not visible"?  Do you mean that in your IIS log you check the URL and it's for a valid page, but IIS responded with 404?  Or do you mean that in your testing from a browser, sometimes you click on a link to your page and it's empty?  Or you click on a link to a page and it gives you a 404 when you are sure that it's a correct URL?

It would help if you could post the URL you are trying which exhibits the wrong behavior, and the IIS access log line which corresponds to it.  Then we might be able to help you more with trouble-shooting.

>>that it's almost a DOS attack.

This is almost certainly not the case.  When you have a real DOS attack, you know because your Web server shuts down -- is completely inaccessible.  It's much more likely that there are a series of human errors which are cascading -- maybe because of webcrawlers -- and causing a lot of 404s.
peazakAuthor Commented:
Thanks mrcoffee365 for taking the time. If you want to click a few links on the site and let me know what you see, that would be awesome.
The site is: www.export.gov 
Clicking links on this page will bring up the 404: http://www.export.gov/archive/index.asp 
To see the custom 404: www.export.gov/showmethe404 (or something of that nature)

I've contacted our network team to see if they can see the problem on the routers etc. The IIS logs don't give any useful information.
Cloud Class® Course: Python 3 Fundamentals

This course will teach participants about installing and configuring Python, syntax, importing, statements, types, strings, booleans, files, lists, tuples, comprehensions, functions, and classes.

Okay, I don't see a 404 when I click on http://www.export.gov/archive/index.asp  .  I see what looks like a correct page.

I do see a custom 404 on this page: www.export.gov/showmethe404 .  So that's good, your custom 404 is working, as you said.

It doesn't help in debugging to have random clients like me click on the URLs.  You need to see the IIS logs to see what IIS is serving when you  see the 404 in your test browser that you don't think you should see.  Naturally, as an external client browser, I can't see the IIS logs when I click on a URL, so my clicking isn't helpful.

It is not likely that a client browser would display a 404 message if your servers are not available to the Internet.  The only circumstance when a 404 should be displayed is when contact is made with the Web server, but it responds with a 404.  Therefore, you should see something in your IIS logs.

If you do not see anything, my guess is that you aren't looking in the right place in your IIS logs, or the requested URL that you think is going to your IIS server is actually going somewhere else, which is responding with the 404.

I assume that you have eliminated the possibility of a browser-cached 404 for a page which is still served, even after the page exists.  That scenario doesn't seem to match your description, but it's always good to mention.


Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
peazakAuthor Commented:
I couldn't find anything in the logs. I believe it's a network issue. Thanks for your help.
I did what I suggested you do, and googled for references to www.export.gov.  I found a Word doc online that might be the source of your 404s.  This URL:

has a Word doc which has hyperlinks to pages on your site, and one of them is:

When I click on the link in the newsletter Word doc I downloaded from the Web (living dangerously, I know), I got this message from Word:

Unable to open http://www.export.gov/archive/index.asp.  The Internet site reports that the item you requested could not be found. (HTTP/1.0 404)

I can't tell what's wrong with the link.  When I added a link to this Word doc myself, the Word hyperlink worked fine.  So -- there's something odd about this link in this Word doc.  It's true for all of the links in this Word doc to your site, in fact.  Other links in the Word doc seem to work fine.

So the Word docs from one of your sites  might be the source of the odd 404 messages you're seeing.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft IIS Web Server

From novice to tech pro — start learning today.