Solved

How do I Hook IDXGISwapChain::Present in existing app?

Posted on 2011-02-14
15
3,045 Views
Last Modified: 2012-05-11
Hi experts,

I'm trying to render a D3D overlay in an injected app using function prologue hooks after looking up entries in the vTables of IDirect3ddevice9 and IDXGISwapChain.

I've no problem creating a temporary d3d device to modify the function prologues, however, my overlay rendering is performed in IDirect3ddevice9::Present which is bypassed in some d3d9 apps (calls are fast tracked to IDXGISwapChain::Present).  So, I'm trying to hook  IDXGISwapChain::Present using the same vTable method (first creating a temporary IDXGISwapChain object to get the vTable address), but the creation of IDXGISwapChain is failing with E_NOINTERFACE.  Incidentally, API hooking IDXGIFactory::CreateSwapChain to catch the creation of the existing object is not an option for me, I need to be able to hook existing apps at runtime without finiky windows hooks.

Could someone please sanity check the attached code for me and tell me if you can spot anything?

Many thanks in advance,

Chris J





/**
 * Creates a temporary D3D device.
 */
IDirect3DDevice9* CD3D9RenderDevice::CreateD3DDevice(void* d3d, HWND hWnd)
{
	D3DDISPLAYMODE d3ddm;

	if(FAILED(((LPDIRECT3D9) d3d)->GetAdapterDisplayMode(D3DADAPTER_DEFAULT, &d3ddm)))
	{
		DEBUG_PRINT("CD3D9RenderDevice::CreateD3DDevice:- GetAdapterDisplayMode failed.\n");
		return NULL;
	}

	D3DCAPS9 d3dCaps;

	if(FAILED(((LPDIRECT3D9) d3d)->GetDeviceCaps(D3DADAPTER_DEFAULT, D3DDEVTYPE_HAL, &d3dCaps)))
	{
		DEBUG_PRINT("CD3D9RenderDevice::CreateD3DDevice:- GetDeviceCaps failed.\n");
		return NULL;
	}

	DWORD dwBehaviorFlags = 0;

	if( d3dCaps.VertexProcessingCaps != 0 )
		dwBehaviorFlags |= (d3dCaps.VertexProcessingCaps != 0 ? D3DCREATE_HARDWARE_VERTEXPROCESSING : D3DCREATE_SOFTWARE_VERTEXPROCESSING);

	D3DPRESENT_PARAMETERS d3dpp;
	memset(&d3dpp, 0, sizeof(d3dpp));
	d3dpp.BackBufferFormat       = d3ddm.Format;// D3DFMT_UNKNOWN;
	d3dpp.SwapEffect             = D3DSWAPEFFECT_DISCARD;
	d3dpp.Windowed               = TRUE;
	d3dpp.EnableAutoDepthStencil = TRUE;
	d3dpp.AutoDepthStencilFormat = D3DFMT_D16;
	d3dpp.PresentationInterval   = D3DPRESENT_INTERVAL_IMMEDIATE;

	LPDIRECT3DDEVICE9 pd3dDevice = NULL;

	HRESULT res = ((LPDIRECT3D9) d3d)->CreateDevice( D3DADAPTER_DEFAULT, D3DDEVTYPE_HAL, hWnd,
									  dwBehaviorFlags, &d3dpp, &pd3dDevice );

	if(FAILED( res ))
	{
		res = ((LPDIRECT3D9) d3d)->CreateDevice( D3DADAPTER_DEFAULT, D3DDEVTYPE_REF, hWnd,
											D3DCREATE_SOFTWARE_VERTEXPROCESSING, &d3dpp, &pd3dDevice );
	}
				
	if(FAILED( res ))
	{
		DEBUG_PRINT_INT("\nCD3D9RenderDevice::CreateD3DDevice:- CreateDevice failed with error code ", res);
		return NULL;
	}

	return (IDirect3DDevice9*) pd3dDevice; 
}

/**
 * This code is in my init...
 */

         IDirect3DDevice9* d3dDevice = CreateD3DDevice(d3d, hWnd); // this succeeds and I can hook the vTable of this device without issue.

         IDXGIFactory * pFactory;
	HRESULT hr = CreateDXGIFactory(__uuidof(IDXGIFactory), (void**)(&pFactory) ); // This succeeds

	if(hr != S_OK)
	{
		DEBUG_PRINT("Error:- DXGI factory creation failed.\n");
		return;
	}
	  
	DXGI_SWAP_CHAIN_DESC swapChainDesc;
	ZeroMemory( &swapChainDesc, sizeof(swapChainDesc) );
	swapChainDesc.BufferCount = 1;
	swapChainDesc.BufferDesc.Width = 300;
	swapChainDesc.BufferDesc.Height = 300;
	swapChainDesc.BufferDesc.Format = DXGI_FORMAT_R8G8B8A8_UNORM;
	swapChainDesc.BufferUsage = DXGI_USAGE_RENDER_TARGET_OUTPUT;
	swapChainDesc.OutputWindow = hWnd;
	swapChainDesc.SampleDesc.Count = 1;
	swapChainDesc.SampleDesc.Quality = 0;
	swapChainDesc.Windowed = TRUE;

	IDXGISwapChain* swapChain;
	hr = pFactory->CreateSwapChain((IDirect3DDevice9*) d3dDevice, &swapChainDesc, &swapChain);	

	if(hr != S_OK)
	{
	    // FAILS HERE WITH E_NOINTERFACE
	}

Open in new window

0
Comment
Question by:chrispauljarram
  • 7
  • 6
  • 2
15 Comments
 
LVL 86

Assisted Solution

by:jkr
jkr earned 100 total points
ID: 34890635
Just an idea - what about using 'CoTreatAsClass()' (http://msdn.microsoft.com/en-us/library/ms693452(VS.85).aspx) to replace IDXGISwapChain with your own interface and then foward the calls properly?
0
 

Author Comment

by:chrispauljarram
ID: 34891768
Thanks jkr, I wasn't aware of that method and shall explore it soon... I /think/ the reason my code is failing is I'm passing an IDirect3dDevice9 object to pFactory->CreateSwapChain, when I should be creating and passing a DXGI object.

As an aside (and looking into some other pages following your suggestion), do you think is it possible for vTable hooking to just call CoCreateInstance with the COM CLSID of whichever interface it is to hook, in order to get an uninitialized object which is just a pointer to a vTable?  Or is more required than this to get a valid vTable pointer?

Cheers,
Chris
0
 
LVL 86

Expert Comment

by:jkr
ID: 34892293
Let me put it like this - I would rather not tamper with the vtable ;o)
0
 

Author Comment

by:chrispauljarram
ID: 34892394
Understood :o)  However, I'm not really tampering with the vTable - I'm only using it to look up function locations and install prologue hooks similar to how detours does it, as opposed to changing its own entries - this is said to be a better method as vTables can change.  Problem is I have a product launch in 2 weeks so I'm sort of stuck with this method at the moment, however 98% of the time it does work, and the installing app can configure what the plugin does according to which .exe is injected (it knows exactly what the executable is by MD5).. one thing I'd like to avoid is borking other people's hooks, that's a very angry cat just waiting to be let out of a small bag...
0
 
LVL 12

Expert Comment

by:satsumo
ID: 34893030
Could you call IDirect3DDevice::GetSwapChain () for the application's device?  Would that solve the problem?  This question rattles my teeth.
0
 

Author Comment

by:chrispauljarram
ID: 34910286
Hi satsumo,

No because that returns an IDirect3DSwapChain9, which is different to an IDXGISwapChain.. but thanks anyway.  Still welcoming comments on this one! :)

Cheers,
Chris.
0
 
LVL 12

Expert Comment

by:satsumo
ID: 34912606
OK then, another comment.  You mention using D3D9 and yet I think DXGI is a D3D10-Vista-Windows 7 thing.  I think the reason the D3D9 device won't give you an interface to IDXGISwapChain is because it dosen't have one.  The online docs are quite explict that D3D9 dosen't support DXGI.  There is no DXGI on any PC that is running XP.

http://msdn.microsoft.com/en-us/library/bb205075%28v=VS.85%29.aspx

Look at the box with differences between D3D9 and D3D10, about halfway down the page.
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 

Author Comment

by:chrispauljarram
ID: 34914575
Hi again,

Hmm, I was actually starting to think that myself as I trawled around, however what confused me about this is I've heard of other developers doing this on the msdn forums (http://forums.create.msdn.com/forums/t/5324.aspx), and I've had instances where hooking the device present or device swapchain present functions didn't work (my content was not being rendered), but hooking endscene did (so clearly my resources were present and correct) - however, obviously hooking endscene is no good as there are usually multiple calls per frame.  So, another question then - what is the lowest level function I can hook that is guranteed to be called on a frame by frame basis, where I can still render my overlay to the frame buffer?

Thanks
Chris
0
 
LVL 12

Expert Comment

by:satsumo
ID: 34915469
The link you posted to appears to support the idea that DXGI dosen't work in D3D9.  The question is about a DX10 program and mentions that thinks work differently in DX9 .

Hooking EndScene is a bad idea.  It gets called many times, rendering a realtime environment map, a shadow map or a prodcedural texture.  If anything changes the target buffer the change would appear in the shadows,  reflections, all over the palce.  Actually, that sounds quite amusing.

Present looks like the only sensible place to hook.  Are you certain that hooking Present didn't fail because of the state of the device.  Could it have been targeting the wrong surface when present was called?  Could a renderstate have been enabled that stopped that output?

A quote from the DX9 documentation of IDirect3DDevice9::Present
Present will fail, returning D3DERR_INVALIDCALL, if called between BeginScene and EndScene pairs unless the render target is not the current render target (such as the back buffer you get from creating an additional swap chain). This is a new behavior for Microsoft® DirectX® 9.0.
0
 

Author Comment

by:chrispauljarram
ID: 34915647
Hi again,

There is actually a comment in that thread that says:-

"hook the DXGI Present and D3D9device calls on DX9 and the DXGI Present call on DX10 (hint not all applications need to call d3d9 present since it can be fast tracked by DX into the DXGI call as it is a shortcut in the API.  MSDN has an article on this."

I've not actually found this so-called article but I had no reason to disbelieve what this guy was saying as he claimed to have already implemented it, maybe he just made a typo but I ruled that out because he subsequently claimed there was an article about it.

I already know hooking EndScene is not a good idea, btw I have tried it previously to see what would happen, the results are indeed highly amusing! :)

It is possible I could be targeting the wrong surface when present is called... Basically I intecept this call, then my overlay code is wrapped between its own BeginScene and EndScene constructs before I call back to the original Present.  I'm not really a d3d expert, can you possibly tell me how I ensure the rendering surface is set to be the main frame buffer before I draw my sprite, so my overlay is visible?  I think by the sounds of it therein lies the key, the last render target to be set by the problem app is not the correct surface...

My code is as follows (some pseudo obviously):

MyPresent(LPDIRECT3DDEVICE9 pDevice)
{
...

// Render 2d layer

pDevice->BeginScene();
pDevice->CreateStateBlock( D3DSBT_ALL, &pStateBlock );
pDevice->SetVertexShader(0);
pDevice->SetPixelShader(0);
pDevice->SetRenderState(D3DRS_ZENABLE, false);
pDevice->SetRenderState(D3DRS_CULLMODE, D3DCULL_NONE);
pDevice->SetRenderState(D3DRS_LIGHTING, 0);

// Draw a sprite

DrawSprite(...); // see code below.

// Exit 2d layer

pStateBlock->Apply();
pStateBlock->Release();      
pDevice->EndScene();

// Call original Present...

OrigPresent(...);
}


// My entire sprite drawing function (if it helps):

/**
 */
void CD3D9RenderDevice::DrawSprite(HBITMAP hBmp, int sx, int sy, int sw, int sh, int x, int y, int w, int h, float vis, bool updateRequired)
{
      static char str[512];

      LPDIRECT3DDEVICE9 pDevice = (LPDIRECT3DDEVICE9) m_pD3DDevice;

      std::map<HBITMAP, DXTexturePair*>::iterator iter = m_mTextures.begin();
      iter = m_mTextures.find(hBmp);

      DXTexturePair* tp;
      if (iter != m_mTextures.end())
      {
            tp = iter->second;
      }
      else
      {
            // No texture found for this HBITMAP, create one.
            DEBUG_PRINT_INT("\nCreating texture for hBmp ", hBmp);
            BITMAP BM;
            GetObject(hBmp, sizeof(BM), &BM);      
            tp = new DXTexturePair();            
            pDevice->CreateTexture(BM.bmWidth, BM.bmHeight, 1, 0, D3DFMT_A8R8G8B8, D3DPOOL_MANAGED, &(tp->texture), NULL);
            D3DXCreateSpriteImported((LPDIRECT3DDEVICE9) m_pD3DDevice, &(tp->sprite));
            m_mTextures.insert(std::pair<HBITMAP, DXTexturePair*>(hBmp, tp));
      }

      if(updateRequired)
      {
            BITMAP BM;
            GetObject(hBmp, sizeof(BM), &BM);
            D3DLOCKED_RECT locked;
            tp->texture->LockRect(0, &locked, 0, 0);
            unsigned char* imageData = (unsigned char*) locked.pBits;
            unsigned char* pixels = (unsigned char*) BM.bmBits;
            memcpy(imageData, BM.bmBits, BM.bmWidth * BM.bmHeight * 4);
            // Set alpha as 100% for each pixel (internal format seems to be BGRA, but declared as ARGB when it was created!?
            for(int i = 0; i < BM.bmWidth * BM.bmHeight; i++)
                  imageData[(i * 4) + 3] = 0xFF; //a

            tp->texture->UnlockRect(0);            
      }

      D3DSURFACE_DESC surfaceDesc;
      tp->texture->GetLevelDesc(0, &surfaceDesc);
      int col = 0x00FFFFFF | ((int) (255.0 * vis) << 24);
      tp->sprite->Begin(D3DXSPRITE_ALPHABLEND);
      D3DXVECTOR2 spriteCentre = D3DXVECTOR2(surfaceDesc.Width / 2.0, surfaceDesc.Height / 2.0);
      D3DXVECTOR2 trans = D3DXVECTOR2(x, y + h);
      D3DXMATRIX mat;

      float xScale = (float) w / surfaceDesc.Width;
      float yScale = (float) h / surfaceDesc.Height;
      xScale *= (float) ((float)surfaceDesc.Width / (float) sw);
      yScale *= (float) ((float)surfaceDesc.Height / (float) sh);

      D3DXVECTOR2 scaling(xScale, -yScale);
      float rotation = 0;
      D3DXMatrixTransformation2DImported(&mat,NULL,0.0,&scaling,&spriteCentre,rotation,&trans);
      tp->sprite->SetTransform(&mat);
      RECT r; r.left = sx; r.right = sx + sw; r.top = sy + (surfaceDesc.Height - sh); r.bottom = sy + surfaceDesc.Height;

      tp->sprite->Draw(tp->texture, &r, NULL, NULL, col);
      tp->sprite->End();
}


Thanks again, if you can point me in the right direction it would be amazing :)

Cheers,
Chris.

0
 
LVL 12

Expert Comment

by:satsumo
ID: 34916641
My best guess is that the comment you is talking about a D3D9 app running on Vista or Windows 7, with DX10 installed.  Perhaps MS made the DX10 implementation of DX9 use DXGI?

So how to handle the wrong target surface.  I would try SwapChain::GetBackBuffer and Device::GetRenderTarget.  If they are different surfaces, call Device::SetRenderTarget to render on the SwapChain's back buffer.  Then restore the Device's original render target after doing your thing.

Is it possible that the hooked program is in the middle of a scene when SwapChain::Present is called?  You could check by calling Device::EndScene (before setting a new render target).  If it does not return D3DERR_INVALIDCALL, then the program was in a scene, so you'd have to call Device::BeginScene (after you restore the original render target).  You'd have to save and restore any changes to the device state (looks like you do that anyway).

The DrawSprite function uses an object called 'tp'.  It looks like that is wrapping an ID3DXSprite interface, is that from the the hooked device?
0
 
LVL 12

Expert Comment

by:satsumo
ID: 34917138
That said, there are more likely issues.  That code does set some render states before drawing the sprite, but not enough.

Any of the render states could be a problem, D3DRS_ALPHABLENDENABLE  could be on, D3DRS_SRCBLEND and D3DRS_DESTBLEND could be set to anything.  D3DRS_ALPHATESTENABLE could be active, D3DRS_COLORWRITEENABLE could be set to zero.  These are just some of the possiblities.  To make sure it work all states that might affect the output have to be set.

Perhaps the program could record the render state it needs in a block and apply that before rendering?
0
 

Author Comment

by:chrispauljarram
ID: 34917485
Doesn't the render state block I have in my code above already do that?  I've seen the pattern for rendering a 2d overlay over and over again, and always these are the only states that are set.

The solution with SetRenderTarget worked btw (did this before you posted back :), I'll award the points shortly.

Thanks,
Chris
0
 
LVL 12

Accepted Solution

by:
satsumo earned 400 total points
ID: 34918609
Yay!  It worked, great.

The StateBlock in the code saves the render state before rendering the sprite and then restores it afterward.  Saving a StateBlock dosen't change anything in the existing render state, only applying it does.

Still, it works.  The D3DXSprite will be setting the blend state because D3DXSPRITE_ALPHABLEND is used, you've disabled depth testing.  You could try deliberately setting one of the other states before drawing the sprite to see what effect it has.

I think there's a still the possibility of a render state causing a problem, even though it isn't right now.  For example, some games pre-render the depth buffer with D3DRS_COLORWRITEENABLE disabled.
0
 

Author Closing Comment

by:chrispauljarram
ID: 34918728
Cheers Satsumo - just what I needed :)  Awarding some of the points to jkr also for his earlier help.

Thanks both,
Chris J
0

Featured Post

Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

Join & Write a Comment

With most software applications trying to cater to multiple user needs nowadays, the focus is to make them as configurable as possible. For e.g., when creating Silverlight applications which will connect to WCF services, the service end point usuall…
Recently, in one of the tech-blogs I usually read, I saw a post about the best-selling video games through history. The first place in the list is for the classic, extremely addictive Tetris. Well, a long time ago, in a galaxy far far away, I was…
The viewer will be introduced to the technique of using vectors in C++. The video will cover how to define a vector, store values in the vector and retrieve data from the values stored in the vector.
The viewer will be introduced to the member functions push_back and pop_back of the vector class. The video will teach the difference between the two as well as how to use each one along with its functionality.

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now