release build: getting an assert on CDialog::DoModal within a dialog-based app


I have a dialog-based app that causes assert whenever I try to do DoModal on any dialog. This happens in the release build only - debug works fine.

A bit of poking around pointed to the source of assert in AfxGetResourceHandle():

    ASSERT(afxCurrentResourceHandle != NULL)

The confusing part is that I call AfxGetResourceHandle() just before doing dlg.DoModal() that is failing, and returned HINSTANCE is fine.

So, something changes from the start of CDialog::DoModal to the point where AfxGetResourceHandle() is called, that I just can't see:

int CDialog::DoModal()
      // can be constructed with a resource template or InitModalIndirect
      ASSERT(m_lpszTemplateName != NULL || m_hDialogTemplate != NULL ||
            m_lpDialogTemplate != NULL);

      // load resource as necessary
      LPCDLGTEMPLATE lpDialogTemplate = m_lpDialogTemplate;
      HGLOBAL hDialogTemplate = m_hDialogTemplate;
      HINSTANCE hInst = AfxGetResourceHandle();

Any help appreciated!!

Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Is the dialog resource included in your release build?
AndyAinscowFreelance programmer / ConsultantCommented:
<I have a dialog-based app that causes assert whenever I try to do DoModal on any dialog. This happens in the release build only - debug works fine.>

?? - ASSERT in release build is a null statement - how can a non-existant line of code fail?
aquarius003Author Commented:
Yes, dialog resource is included. There are no resource conflicts or anything like that.

Andy: I got the location of the assert when I changed the project settings to produce a PDB with debug information. Maybe ASSERT is not empty for that case??
Microsoft Azure 2017

Azure has a changed a lot since it was originally introduce by adding new services and features. Do you know everything you need to about Azure? This course will teach you about the Azure App Service, monitoring and application insights, DevOps, and Team Services.

aquarius003Author Commented:
Actually, afxCurrentResourceHandle is defined as:

 #define afxCurrentResourceHandle    AfxGetModuleState()->m_hCurrentResourceHandle

So, my assert is an access violation within AfxGetModuleState(). Unfortunately, it won't let me step in to see, but it could be that pState (_afxThreadState) is NULL, and being de-referenced, rather than afxBaseModuleState.

      _AFX_THREAD_STATE* pState = _afxThreadState;
      AFX_MODULE_STATE* pResult;
      if (pState->m_pModuleState != NULL)
            // thread state's module state serves as override
            pResult = pState->m_pModuleState;
            // otherwise, use global app state
            pResult = _afxBaseModuleState.GetData();
      ASSERT(pResult != NULL);
      return pResult;
Is this taking place inside of code in a DLL?  If so, make sure you are using:

       AFX_MANAGE_STATE( AfxGetStaticModuleState() );

at the start of any code that may be called by the main EXE.  Some entry points are kinda sneaky...
aquarius003Author Commented:
Thanks for the suggestion, Dan. I already had a similar issue with a DLL, so I know about doing AFX_MANAGE_STATE for every exported function.

However, my current problem has nothing to do with a DLL. There are some functions called by the application itself, from another DLL, but everything related to the dialogs that I'm seeing issues with, is witin the app.

This is really, really annoying, since everything appears to be kosher.
mahesh1402IT ProfessionalCommented:
I would recommend calling AfxSetResourceHandle with the Instance handle of your extension dll before calling your Dialog class.  like this

HINSTANCE hOldInst = AfxGetResourceHandle();

CMyDialog dlg;



AndyAinscowFreelance programmer / ConsultantCommented:
PDB files - I'm no expert with them.

DoModal is a virtual function.  That means you could, temporarily, make a copy of the standard CDialog DoModal and create your own CMyDialog::DoModal.  In the copy put some AfxMessageBox calls to track how far it gets before the failure and also what values are present in a second iteration.


then when you know between which lines it fails at
CString s;
s.Format("something is %ld", lSomething);
aquarius003Author Commented:
Mahesh, I tried that already: doing AfxGetResourceHandle() and using the HINSTANCE to do AfxSetResourceHandle() just before calling DoModal. Doesn't make any diff.

Andy: good idea, will try and get back to you.

Thanks !!!
aquarius003Author Commented:
I resolved the issue and it had to do with a message handler added for a new message between two app components.


When I removed sending of this message, WM_MY_BLAH,  to the target window, and replaced it with a direct method invocation, no problems were happening any more.

I initially tried changing the base for user-defined message IDs, but that didn't do much good. I'm still not sure what exactly is happening (the problem seemed very 'slick' - as soon as I'd trace it to one location, it would skip to another, so to speak), since there are other user-defined message IDs that cause no problem.

I've read in a posting that user-defined message IDs can cause issues in release mode, if they overlap with IDs already used by the system, and this led me to poke around them. Not that I understand what's really going on. But hey, at least it's not crashing any longer.

Thanks for your help and suggestions!
aquarius003Author Commented:
Btw, if anyone can point to the connection between the user-defined messages and the crash I was getting, I'd be more than happy to assign points to them. :)
It would have to relate to whatever the UDM handler was doing and and what point in the normal sequence of events that it was being sent.  

In general (and to avoid hard-to-debug message-related errors) I prefer to use PostMessage rather than SendMessage.  If there is any quirky "unfinished business" then posting the message makes it much more likely that things will be handled correctly.  For instance, when setting the current selection for a CListCtrl there are messages flying all over the place for a while and interrupting the sequence can end up in recursion or other problems.  Same thing if messages happen early in the CDialog::Create() cycle.  

Given the nature of this error, I'd say that creation of some control was not complete at the time you sent a message to a handler that expected the dialog and all of its controls to be fully created.

If you defined your UDM using
then you should have no problem of conflicts with standard Windows messages.  If you are sending private messages to another application program of a suite of interacting programs, then you should use values returned by RegisterWindowMessage()

-- Dan

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
AndyAinscowFreelance programmer / ConsultantCommented:
For custom messages I usually use
const UINT AA_WM_XYZ = ::RegisterWindowMessage(_T("AA_WM_XYZ"));
which guarantees a unique ID for AA_WM_XYZ BUT, very important, any call to RegisterWindowMessage with the same string returns the same unique value - very good for passing between apps.

message map entry

and function
mahesh1402IT ProfessionalCommented:
aquarius003Author Commented:
Thanks for the answers! Dan was right on the spot - changing SendMessage to PostMessage solved the issue.

It's much better to understand what's going on, not just having it working. :)

AndyAinscowFreelance programmer / ConsultantCommented:
<It's much better to understand what's going on, not just having it working.>

You do understand the VERY LARGE difference between SendMessage and PostMessage I hope.
aquarius003Author Commented:
AndyAinscowFreelance programmer / ConsultantCommented:
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
System Programming

From novice to tech pro — start learning today.