How do I Hook IDXGISwapChain::Present in existing app?

Thanks jkr, I wasn't aware of that method and shall explore it soon... I /think/ the reason my code is failing is I'm passing an IDirect3dDevice9 object to pFactory->CreateSwapChain, when I should be creating and passing a DXGI object.

As an aside (and looking into some other pages following your suggestion), do you think is it possible for vTable hooking to just call CoCreateInstance with the COM CLSID of whichever interface it is to hook, in order to get an uninitialized object which is just a pointer to a vTable?  Or is more required than this to get a valid vTable pointer?

Cheers,
Chris

Let me put it like this - I would rather not tamper with the vtable ;o)

Understood :o)  However, I'm not really tampering with the vTable - I'm only using it to look up function locations and install prologue hooks similar to how detours does it, as opposed to changing its own entries - this is said to be a better method as vTables can change.  Problem is I have a product launch in 2 weeks so I'm sort of stuck with this method at the moment, however 98% of the time it does work, and the installing app can configure what the plugin does according to which .exe is injected (it knows exactly what the executable is by MD5).. one thing I'd like to avoid is borking other people's hooks, that's a very angry cat just waiting to be let out of a small bag...

Could you call IDirect3DDevice::GetSwapChain () for the application's device?  Would that solve the problem?  This question rattles my teeth.

Hi satsumo,

No because that returns an IDirect3DSwapChain9, which is different to an IDXGISwapChain.. but thanks anyway.  Still welcoming comments on this one! :)

Cheers,
Chris.

OK then, another comment.  You mention using D3D9 and yet I think DXGI is a D3D10-Vista-Windows 7 thing.  I think the reason the D3D9 device won't give you an interface to IDXGISwapChain is because it dosen't have one.  The online docs are quite explict that D3D9 dosen't support DXGI.  There is no DXGI on any PC that is running XP.

http://msdn.microsoft.com/en-us/library/bb205075%28v=VS.85%29.aspx

Look at the box with differences between D3D9 and D3D10, about halfway down the page.

Hi again,

Hmm, I was actually starting to think that myself as I trawled around, however what confused me about this is I've heard of other developers doing this on the msdn forums (http://forums.create.msdn.com/forums/t/5324.aspx), and I've had instances where hooking the device present or device swapchain present functions didn't work (my content was not being rendered), but hooking endscene did (so clearly my resources were present and correct) - however, obviously hooking endscene is no good as there are usually multiple calls per frame.  So, another question then - what is the lowest level function I can hook that is guranteed to be called on a frame by frame basis, where I can still render my overlay to the frame buffer?

Thanks
Chris

The link you posted to appears to support the idea that DXGI dosen't work in D3D9.  The question is about a DX10 program and mentions that thinks work differently in DX9 .

Hooking EndScene is a bad idea.  It gets called many times, rendering a realtime environment map, a shadow map or a prodcedural texture.  If anything changes the target buffer the change would appear in the shadows,  reflections, all over the palce.  Actually, that sounds quite amusing.

Present looks like the only sensible place to hook.  Are you certain that hooking Present didn't fail because of the state of the device.  Could it have been targeting the wrong surface when present was called?  Could a renderstate have been enabled that stopped that output?

A quote from the DX9 documentation of IDirect3DDevice9::Present

Present will fail, returning D3DERR_INVALIDCALL, if called between BeginScene and EndScene pairs unless the render target is not the current render target (such as the back buffer you get from creating an additional swap chain). This is a new behavior for Microsoft® DirectX® 9.0.

Hi again,

There is actually a comment in that thread that says:-

"hook the DXGI Present and D3D9device calls on DX9 and the DXGI Present call on DX10 (hint not all applications need to call d3d9 present since it can be fast tracked by DX into the DXGI call as it is a shortcut in the API.  MSDN has an article on this."

I've not actually found this so-called article but I had no reason to disbelieve what this guy was saying as he claimed to have already implemented it, maybe he just made a typo but I ruled that out because he subsequently claimed there was an article about it.

I already know hooking EndScene is not a good idea, btw I have tried it previously to see what would happen, the results are indeed highly amusing! :)

It is possible I could be targeting the wrong surface when present is called... Basically I intecept this call, then my overlay code is wrapped between its own BeginScene and EndScene constructs before I call back to the original Present.  I'm not really a d3d expert, can you possibly tell me how I ensure the rendering surface is set to be the main frame buffer before I draw my sprite, so my overlay is visible?  I think by the sounds of it therein lies the key, the last render target to be set by the problem app is not the correct surface...

My code is as follows (some pseudo obviously):

MyPresent(LPDIRECT3DDEVICE9 pDevice)
{
...

// Render 2d layer

pDevice->BeginScene();
pDevice->CreateStateBlock( D3DSBT_ALL, &pStateBlock );
pDevice->SetVertexShader(0);
pDevice->SetPixelShader(0);
pDevice->SetRenderState(D3DRS_ZENABLE, false);
pDevice->SetRenderState(D3DRS_CULLMODE, D3DCULL_NONE);
pDevice->SetRenderState(D3DRS_LIGHTING, 0);

// Draw a sprite

DrawSprite(...); // see code below.

// Exit 2d layer

pStateBlock->Apply();
pStateBlock->Release();      
pDevice->EndScene();

// Call original Present...

OrigPresent(...);
}


// My entire sprite drawing function (if it helps):

/**
 */
void CD3D9RenderDevice::DrawSprite(HBITMAP hBmp, int sx, int sy, int sw, int sh, int x, int y, int w, int h, float vis, bool updateRequired)
{
      static char str[512];

      LPDIRECT3DDEVICE9 pDevice = (LPDIRECT3DDEVICE9) m_pD3DDevice;

      std::map<HBITMAP, DXTexturePair*>::iterator iter = m_mTextures.begin();
      iter = m_mTextures.find(hBmp);

      DXTexturePair* tp;
      if (iter != m_mTextures.end())
      {
            tp = iter->second;
      }
      else
      {
            // No texture found for this HBITMAP, create one.
            DEBUG_PRINT_INT("\nCreating texture for hBmp ", hBmp);
            BITMAP BM;
            GetObject(hBmp, sizeof(BM), &BM);      
            tp = new DXTexturePair();            
            pDevice->CreateTexture(BM.bmWidth, BM.bmHeight, 1, 0, D3DFMT_A8R8G8B8, D3DPOOL_MANAGED, &(tp->texture), NULL);
            D3DXCreateSpriteImported((LPDIRECT3DDEVICE9) m_pD3DDevice, &(tp->sprite));
            m_mTextures.insert(std::pair<HBITMAP, DXTexturePair*>(hBmp, tp));
      }

      if(updateRequired)
      {
            BITMAP BM;
            GetObject(hBmp, sizeof(BM), &BM);
            D3DLOCKED_RECT locked;
            tp->texture->LockRect(0, &locked, 0, 0);
            unsigned char* imageData = (unsigned char*) locked.pBits;
            unsigned char* pixels = (unsigned char*) BM.bmBits;
            memcpy(imageData, BM.bmBits, BM.bmWidth * BM.bmHeight * 4);
            // Set alpha as 100% for each pixel (internal format seems to be BGRA, but declared as ARGB when it was created!?
            for(int i = 0; i < BM.bmWidth * BM.bmHeight; i++)
                  imageData[(i * 4) + 3] = 0xFF; //a

            tp->texture->UnlockRect(0);            
      }

      D3DSURFACE_DESC surfaceDesc;
      tp->texture->GetLevelDesc(0, &surfaceDesc);
      int col = 0x00FFFFFF | ((int) (255.0 * vis) << 24);
      tp->sprite->Begin(D3DXSPRITE_ALPHABLEND);
      D3DXVECTOR2 spriteCentre = D3DXVECTOR2(surfaceDesc.Width / 2.0, surfaceDesc.Height / 2.0);
      D3DXVECTOR2 trans = D3DXVECTOR2(x, y + h);
      D3DXMATRIX mat;

      float xScale = (float) w / surfaceDesc.Width;
      float yScale = (float) h / surfaceDesc.Height;
      xScale *= (float) ((float)surfaceDesc.Width / (float) sw);
      yScale *= (float) ((float)surfaceDesc.Height / (float) sh);

      D3DXVECTOR2 scaling(xScale, -yScale);
      float rotation = 0;
      D3DXMatrixTransformation2DImported(&mat,NULL,0.0,&scaling,&spriteCentre,rotation,&trans);
      tp->sprite->SetTransform(&mat);
      RECT r; r.left = sx; r.right = sx + sw; r.top = sy + (surfaceDesc.Height - sh); r.bottom = sy + surfaceDesc.Height;

      tp->sprite->Draw(tp->texture, &r, NULL, NULL, col);
      tp->sprite->End();
}


Thanks again, if you can point me in the right direction it would be amazing :)

Cheers,
Chris.

My best guess is that the comment you is talking about a D3D9 app running on Vista or Windows 7, with DX10 installed.  Perhaps MS made the DX10 implementation of DX9 use DXGI?

So how to handle the wrong target surface.  I would try SwapChain::GetBackBuffer and Device::GetRenderTarget.  If they are different surfaces, call Device::SetRenderTarget to render on the SwapChain's back buffer.  Then restore the Device's original render target after doing your thing.

Is it possible that the hooked program is in the middle of a scene when SwapChain::Present is called?  You could check by calling Device::EndScene (before setting a new render target).  If it does not return D3DERR_INVALIDCALL, then the program was in a scene, so you'd have to call Device::BeginScene (after you restore the original render target).  You'd have to save and restore any changes to the device state (looks like you do that anyway).

The DrawSprite function uses an object called 'tp'.  It looks like that is wrapping an ID3DXSprite interface, is that from the the hooked device?

That said, there are more likely issues.  That code does set some render states before drawing the sprite, but not enough.

Any of the render states could be a problem, D3DRS_ALPHABLENDENABLE  could be on, D3DRS_SRCBLEND and D3DRS_DESTBLEND could be set to anything.  D3DRS_ALPHATESTENABLE could be active, D3DRS_COLORWRITEENABLE could be set to zero.  These are just some of the possiblities.  To make sure it work all states that might affect the output have to be set.

Perhaps the program could record the render state it needs in a block and apply that before rendering?

Doesn't the render state block I have in my code above already do that?  I've seen the pattern for rendering a 2d overlay over and over again, and always these are the only states that are set.

The solution with SetRenderTarget worked btw (did this before you posted back :), I'll award the points shortly.

Thanks,
Chris

Cheers Satsumo - just what I needed :)  Awarding some of the points to jkr also for his earlier help.

Thanks both,
Chris J