Send those users a debug version of the program ... one containing symbol info.   You will have useful debugging information within 10 days :)

-- Dan

There is no way i can send debug versions to customers !!
i did a version check things look normal. They have recreated the machine several times...

There seems to be something else going on...

It's not as unusual as you make it sound.  Out of 40,000 customers, two have a problem.  You let them know how rare that is and you have a crew working night and day on it, but  cannot recreate the problems.  When I've done that I usually find somebody on the other end who is very savvy and we become lifelong friends.  Right cookre?

Anyway, the code you send does not need to be a full debug version!.  Do a release build that contains debug info.  Then when it crashes, it will show a source code filenames, and line numbers, and a readable Call stack... rather than random numbers.

-- Dan

Ain't that the truth.  We have to deal with about 120,000 desktops across the country.  Even an impossible 99.5% success rate with any given release of something is going to generate 600 trouble calls.  

We'd be lost without the local contacts we keep stroked and fed with the latest goodies.

Guys,
I know what you are saying. I have been their before in different companies !! But this is kind of an unusual customer. When i said Debug version i meant a Release version with pdb stuff. One thing I have a version with the SetUnhandledexception hook and i was talking to these guys to put it on there for 3 weeks. They just did not agree till today morn'. They are huge bank and you know how banks security is... and they use this on 5 production servers. just 1 crashes and it is production. so they were very hesitant to put anything other than the release.

And you know what today they put it on and when it crashes it goes and creates a dmp file but beleive it or not it is a 0 byte file !!

im totally shocked. i have never seen this happen before. So.. Now i have asked them to see if they can install Dr Watson there. May be Dr Watson's hook can do something ??!!

Any other ideas will be helpful

Do the two addresses in the error message vary?

Do you do a static or dynamic link?
Do  you use any non-Window's DLLs?
Do you do GetProcAddress()s?

static linking...
as you see below the read error is always repeated. there is a write error which changes.


Application popup: xxx.exe - Application Error : The instruction at "0x77fc95a3" referenced memory at "0x0000302b". The memory could not be "read".

Application popup: xxx.exe - Application Error : The instruction at "0x77fc8fe3" referenced memory at "0x00003037". The memory could not be "written".

Application popup: xxx.exe - Application Error : The instruction at "0x77fcb5bf" referenced memory at "0x34393234". The memory could not be "written".

There actually *is* some usable information there!

A pointer somehow has gotten set to these values:

0x0000302b    "0#"   or maybe "#0"
0x00003037    "07"   or maybe  "70"
0x34393234    "4924" or           "4294"

Since they are all human-readable ASCII character values, there is some likelihood that the root problem is a buffer overflow.  

Buffer overlow happens when you do (for instance) a strcpy() of, say a 50-byte string into a buffer that this say, 40 bytes long.  The result is that the end of the string overwrites the next variable in memory.  If that variable happens to be a pointer, then the next use of that pointer will access an invalid memory location.

It may be possible to isolate the specific error locus.  
1) Generate a release version identical to that of the customer's, but build it with debugging symbols.

2) Manually place breakpoints at all three of those locations:
     0x77fc95a3
     0x77fc8fe3
     0x77fcb5bf

Run the program, and if (and when) it stops, you can see the code or the call stack to gain the next clue.  Odds are, it is in a low-level runtime library call, but the call stack will provide a locust in your own code.

-- Dan

Dan,
tried that before. the damn thing never comes to that instruction on my machine. having tried all the various things... the surprising thing is the dump file not getting created on the customer's machine.
ive convinced the customer to rebuild his machine. hopefully the problem will go away after he does that. hoping it goes away. but i still have my doubts. im really suspicious of the code that ive written and have gone thru' i dont know how many lines for the past 20 days.

one thing do you think the access violation will not be caught by the exception handler ?

Thanks

If you generate and examine the linker Map file, you might be able to get at least some idea where the offending program instructions are located.  If you can narrow it down to a general area, then you can (and should) eyeball each line of code in that part of the program and look for potential buffer overflows.  These happen when you assume that an incoming string will never exceed a certain length, but something unforseen allowed it to happen.  

Sometimes you can get a clue by learning, in general what was happening at the time of the error.  Did I forget to mention that?  That is in fact, the PRIME way to to start to fix these problems.

Starting with *no* location clue, the way you track these down is to examine all places where you process outside data that has entered into your program... user input, data from files that you generate or others generate, data obtained from the Internet, etc.

But there are many many variables that you cannot control.  The code could be part of an ActiveX or the customer might have API interception happening, or, well anything.

-- Dan

The map file does not have any indication of this address.
ntdll.dll is loaded in that address space. so it is failing on an ntdll.dll instruction. this is a huge server environment with 1000s of clients. so itl be tought to know what someone was doing but our logs dont indicate anything unusual. anyways ive asked them to install dr watson on there. let me hope if that can give me any information because the setunhandledexception filter does not seem to be doing anything...
the other thing im suspecting is something to do with our use of DAO 3.6. but i dont know if mdac 2.6 has any bug like this which can crash the system...

We had unexplained error like this when using an old version of Sybase database.  We upgraded the ODBC driver and they went away.   THis was *very* hard to track down because the error ws in a system DLL -- the one that was calling into the db driver.  

In our specific case, it turned out that we had a database field that was bigger than the buffer allocated to hold it.  I believe that the ancient DAO technolgy allocates and binds fixed-length string buffers just as MFC's CRecordset does.  It shows up as an intermittent error becasue this particular client has entered a string that is longer than normal (in our case, they had a person who lived in a city with a name like
     East Saint Patricks Snake River, IL  
which was 31 characters long where we had provided a generous 30 character buffer. :)

-- Dan

MDAC upgrade did not help. Still working on it...

MyName1,

Here are a few ideas.  You said somewhere above that they have 5 systems and it is failing on 1.  You've checked your app's DLL's and MDAC across the boxes, but there might be some more to check to0.  Check ntdll.dll and check the drivers for the particular type of database your are writing too (i.e. Sybase) for consistency across all 5.  Are you using a DSN for your database call?  If so, make sure they are identical across all 5.  Where you ever able to get a Dr. Watson log?

Tim

One more thing.  If you are using JET compare the JET versions between systems.

Tim

The dr watson log is not getting created. All the Jet versions look fine. .. Still working on it no luck-

Ok i have the drwtsn log now. This is the information. As I see it looks like the ntdll.dll has a problem...

Application exception occurred:
        App:  (pid=956)
        When: 4/10/2003 @ 08:12:21.210
        Exception number: c0000005 (access violation)

 956 xxx.exe
   0 _Total.exe

{{ 1.8 MB comment text deleted -- Dan Rollins / EE Page Editor }}

HA HA!!! Looks like your customers are missing a bitmap image

wow, mods have all the power :)

Hey,

I don't know if this will help, but I'm going to put it out there for you.  

I was having exact same crash in a program that was calling a DLL that I had written.  It appears that the problem is fixed now so let me tell you what I did to fix it.  The functions in my DLL were being called by a multithreaded program.  I had compiled my DLL with the multithreaded option, but at some point along the way, I had set it to use the runtime "Debug Multithreaded".  I changed this to "Debug Multithreaded DLL." and the program has been running for 6 hours without a crash (it was crashing as often as every two or three minutes).   I also had to copy msvcrtd.dll to my test server as it wasn't installed on it by anything else.

Here are some of the crashes this issue was causing:

Application Error : The instruction at "0x77fc911e" referenced memory at "0x0b578795". The memory could not be "read".
Application Error : The instruction at "0x77fc8eff" referenced memory at "0x00000000". The memory could not be "read".
Application Error : The instruction at "0x77fc9234" referenced memory at "0x00000000". The memory could not be "written".
Application Error : The instruction at "0x77fcb5bf" referenced memory at "0x00000000". The memory could not be "written".
Application Error : The instruction at "0x77fc8dbd" referenced memory at "0x00000000". The memory could not be "written".
Application Error : The instruction at "0x77fc8da5" referenced memory at "0x0053002a". The memory could not be "read".

These settings are in the Project Settings Dialog on the C/C++ tab in the Category "Code Generation".

Note also that it's not enough for your code to use the proper Multithreaded libraries--all of the code, including DLLs, in your program need to use the proper libraries.  So it might not be your code but a third party DLL you are using.

Hope this helps, or at least kickstarts your debugging grey matter.

Dan,
I also hate it when my support staff does this to me. But correct me if im wrong there is no way to attach a file to the comment . right ?
Again, sorry for the entire contents pasted...

Mine is a release build and use the Mutithreaded-DLL option... and what I dont get is this on over 400000 locations and 1 fails !!

Besides Dan's question as to the why Kavar thinks it is a bitmap problem, MyName1 can you tell us how you got the dr. watson logging enabled for your customer?

Tim

For yet another straw to grasp at, the event logs sometimes will show other errors that happend at the same time that may be related.

ok so you split it. that is interesting !! but the problem is still not resolved. anyways have fun with the points .

Not sure if you'll get this, but check to make sure the following exchange 2000 patches are loaded:

SP3
Exchange2000-KB870540-v2-x86-ENU.exe
Exchange2000-KB894549-x86-ENU.exe

The last of the three patches requires the previous two and it fixes some denial of service buffer overruns (aka junior programmer coding errors) in exchange/smtp associated with DNS name resolution calls.

Cheers,
Sam