Cursor click on object detection

Hi,

I didn't get much luck with my last question so am re-wording it as I am closing in on the project deadline.

I have several objects (PrimitiveType.TriangleStrip) and have a cursor which i obviously can get the on screen co-ordinates of x and y although when i try and detect whether the cursor is over an object it doesn't work.  I'm guessing obviously because my objects are being translated etc. about in the world.

Any way to find out the on screen co-ordinates of the objects?
I've been doing lots of googling and it looks like i need to project a ray from my 2d co-ordinates of the mouse through the world and check for intersections with my object!

Thanks for any help.

James
LVL 4
James_h1023Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

ikeworkCommented:
hi James,

right, you have to send a ray down the camera direction from the 3d-cursor-position and see if it intersects an object.
the thing is you have to calculate the 3d-world-point from the 2d-screen-point of the mouse-position. you can do it by calculation the screen-to-world-matrix. here is an example:

http://www.mvps.org/directx/articles/improved_ray_picking.htm


ike
0
James_h1023Author Commented:
Thanks for the link i will have a look at that :)

Just looking through my code again and have realised something i didn't know before (taking i'm adapting someone else's code).  The cursor is also a mesh.

The target and cursor i want to find if are intersecting are both vertexBuffers which are then drawn through DrawPrimitives.  Although the targets are translated as follows:
device.Transform.World *= Matrix.Translation(targetPositions[i]);
and the cursor as:
device.Transform.World = Matrix.Translation(new Vector3(X, Y, 1.0f));

I don't quite get why one is '=' and one is '*=' my matrices math is amazing!

I was thinking maybe i could just do a mesh intersection between the two! - I have several targets at different z co-ordinates so was thinking of cycling the mouse mesh backwards until intersects and the first one it intersects with would be the top target and if it reaches the back without intersecting then no collision!  Would that work maybe?

Don't know if that will affect the method because they are both VertexBuffer / meshs (my terms is also rusty)!

Thanks
0
ikeworkCommented:
>> so was thinking of cycling the mouse mesh backwards until intersects and the first one it intersects with would be the top target

that might work .. although sending a ray (or rays from cursor's edges) might be easier and cleaner .. doing the mesh-collision test might be overkill for that case, isn't it .. and btw you have to make steps down the camer's z-axis .. you might skip an object doing those steps ..
0
Get expert help—faster!

Need expert help—fast? Use the Help Bell for personalized assistance getting answers to your important questions.

James_h1023Author Commented:
Ah ok fair enough i will try the ray attempt then.

I am not very proficient in DirectX and the example went a bit over my head.
So; I have the a two vectors describing the positions of the target and the cursor (x,y,z).  I have to produce a ray from the position of the camera through the cursor, and check whether this intersects with the target; is this correct?
0
ikeworkCommented:
right .. if you have bounding volumes (boxes/spheres or something) for your meshes, you can first test the ray with those boxes, if you have a hit, test the ray with the triangles of the mesh ..
0
James_h1023Author Commented:
I'm sorry i'm being very dumb - Ok i've got that although my target and cursor are are defined in a vertex buffer and then drawn as primitives.  Should i convert them to a mesh or somethign first, although because they are already in a vertex buffer couldn't i simply check for collisions with the triangles in the buffer.  My targets are simply a circle which a texture is drawn to, 2d facing towards the camera always.  And the cursor is simply like a target in a shooting game - a circle with two lines going through it!

I'm also in C# and having difficulty transferring from C++.  From the example i've attached one of Obj.cpp from the example, have I understood the following correctly?
vNear - the position of the ray at the camera.
vDir - the position of the ray far away.
m_conbinedMat - I found as the combination of matrices multiplication for the object being drawn.  So this would be akin my target vertices.
It then converts the ray to model space.
And checks for intersection although checks with a mesh, which it turns out i don't have, because I simply have a vertexBuffer with values in which represent my 2d circle facing towards the camera.

I have attached a screen shot hoping it might help!

Thanks for your help.
// yes, convert ray to model space
D3DXVECTOR3 vNear,vDir;
D3DXMATRIX invMat;
D3DXMatrixInverse(&invMat,NULL,&m_combinedMat);
D3DXVec3TransformCoord(&vNear,pvNear,&invMat);
D3DXVec3TransformNormal(&vDir,pvDir,&invMat);
 
// test for intersection
BOOL bHit;
DWORD dwIndex;
float u,v;
float dist;
D3DXIntersect(m_pMesh,&vNear,&vDir,&bHit,&dwIndex,&u,&v,&dist,NULL,NULL);

Open in new window

example.jpg
0
ikeworkCommented:
>>  couldn't i simply check for collisions with the triangles in the buffer

excactly, you only need the mesh, if you want to use the D3DXIntersect-function. you can make the ray-triangle-intersection-tests yourself, you will find a lot of example-code with a google search for it ..
0
James_h1023Author Commented:
Ok, i've redone my targets so they are meshs, no i'm trying to do the intersect, and have the following so far!

                Vector3 near = new Vector3(cursorX, cursorY, 0);
                Vector3 far = new Vector3(cursorX, cursorY, 1);
                Matrix invMat;
                float test;
                //invMat = Matrix.Invert(targetMesh.
                Vector3 nearInv, farInv;
                nearInv = Vector3.TransformCoordinate(near, invMat);
                farInv = Vector3.TransformNormal(far, invMat);

                IntersectInformation closestIntersection;
                hit = targetMesh.Intersect(near, far, out closestIntersection);

Except i don't know how to make the inverse matrix that is needed, some websites say its the inverse matrix of the mesh I am checking for intersection with and others half say its the inverse of the world matrix!

Thanks for your continued help.

James
0
James_h1023Author Commented:
edit:
i've not got this far, although it doesn't seem to work all the time, and sometimes it apparently registers a hit when the cursor coordinates aren't over the target mesh.

Am i doing something obviously wrong here?
float cursorX = mouseCursor.getCursorX();
float cursorY = mouseCursor.getCursorY();
 
Vector3 near = new Vector3(cursorX, cursorY, 0);
Vector3 far = new Vector3(cursorX, cursorY, 1);
 
// Transform points to world space 
near.Unproject(device.Viewport, device.Transform.Projection, device.Transform.View, device.Transform.World);
far.Unproject(device.Viewport, device.Transform.Projection, device.Transform.View, device.Transform.World);
far.Normalize();
 
// Retrieve intersection information 
IntersectInformation closestIntersection;
hit = targetMesh.Intersect(near, far, out closestIntersection);

Open in new window

0
ikeworkCommented:
why dont you use the 3d-position of the cursor as startingpoint of the ray? you said you already have the 3d-position of the cursor...

Vector3 rayOrigin = 3d_cursor_position;
Vector3 rayDirection = 3d_cursor_position - 3d_camera_position;
rayDirection.Normalize();

// Retrieve intersection information
IntersectInformation closestIntersection;
hit = targetMesh.Intersect(rayOrigin, rayDirection, out closestIntersection);
0
James_h1023Author Commented:
I've tried that, and now it returns true all the time, no matter where I move the cursor.
Do i still need the transformation to world space with lines 4 & 5.
I tried with it commented and not, and with it uncommented the results go true, false, true, false repetitively quickly as the mouse moves, and with lines 4 & 5 uncommented the result is true all the time.
float cursorX = mouseCursor.getCursorX();
float cursorY = mouseCursor.getCursorY();
// yes, convert ray to model space
Vector3 near = new Vector3(cursorX, cursorY, 1);
Vector3 far = (new Vector3(cursorX, cursorY, 1)) - (new Vector3(headX, headY, headDist));
 
// Transform points to world space 
near.Unproject(device.Viewport, device.Transform.Projection, device.Transform.View, device.Transform.World);
far.Unproject(device.Viewport, device.Transform.Projection, device.Transform.View, device.Transform.World);
far.Normalize();
 
// Retrieve intersection information 
IntersectInformation closestIntersection;
hit = targetMesh[1].Intersect(near, far, out closestIntersection);

Open in new window

0
James_h1023Author Commented:
EDIT: i mean lines 8 & 9 not lines 4 & 5.
0
ikeworkCommented:
are "mouseCursor.getCursorX()" screen-coordinates or 3d-world-coordinates?
0
ikeworkCommented:
this implies "mouseCursor.getCursorX()" is in worldspace:

  (new Vector3(cursorX, cursorY, 1)) - (new Vector3(headX, headY, headDist));

otherwise you could not substract them, since camera is in world-space
0
ikeworkCommented:
pseudocode:

Vector3 screen_mouse(screen_x, screen_y, 1 );
Vector3 world_mouse = screen2world_matrix * screen_mouse;

Vector3 rayOrigin = world_mouse;
Vector3 rayNormal = world_mouse - world_camera;
rayNormal.Normalize();

// Retrieve intersection information
IntersectInformation closestIntersection;
hit = targetMesh[1].Intersect(rayOrigin, rayNormal, out closestIntersection);
0
ikeworkCommented:
are you sure "headDist" is the camera z-position. the name sounds not like that ..
0
ikeworkCommented:
ok .. is it a big project? if not send it to my email and i'm gonna have a look at it when i'm home .. my mail is in my profile ..
0
James_h1023Author Commented:
I do believe they are 3d world coordinates because i draw my cursor including:
device.Transform.World = Matrix.Translation(new Vector3(X, Y, 1.0f));
where X, and Y are the same variables i return with my getter methods.

So ok, that would mean no lines 8 & 9 because no need to swap to 3d world coordinates right?
So them uncommented returns true where ever i move the cursor.
0
James_h1023Author Commented:
That is very nice of you.
Thanks.
0
ikeworkCommented:
>> So ok, that would mean no lines 8 & 9 because no need to swap to 3d world coordinates right?

yes, if X and Y is in world-space

ah ok .. you have to transform the ray into meshspace. you can do that by inversing the mesh-matrix and multiply your ray with it before intersection-test

but without seeing the entire code .. its just guesses
0
ikeworkCommented:
no prob .. you're welcome .. ;)
0
ikeworkCommented:
hey james,

i had a look at your project and fixed it .. i took the original project, because in your project some things did not work anymore

- i switched off the wiiremote, i dont have one ;)  you can move the cursor with the mouse, when it hits a target you can see it on the rendered text
- i rendered the mesh above each target, you see it is a rectangle actually which was used by the author, the texture lets it just look like an circle
- i just used one mesh, i used targetSizes[] and targetPositions[] to put the mesh for each target in worldspace and render it


here is the most important part, the collision check:

       private bool CheckTargetForHits(int target)
       {
           // transform ray in target-space, we need the inverted target-matrix
           Matrix mTargetTransInv = Matrix.Translation(-targetPositions[target].X, -targetPositions[target].Y, -targetPositions[target].Z);
           Matrix mTargetScaleInv = Matrix.Scaling(1.0f / targetSizes[target].X, 1.0f / targetSizes[target].Y, 1.0f / targetSizes[target].Z);
           Matrix mTargetInv = mTargetTransInv * mTargetScaleInv;

           // calc ray in worldspace
           Vector3 rayOrigin = new Vector3(mouseCursor.X, mouseCursor.Y, 0.0f);
           Vector3 camPosition = new Vector3(headX, headY, headDist);
           Vector3 rayDirection = rayOrigin - camPosition;
           rayDirection.Normalize();
           float a = rayDirection.Length();

           // transform ray into targetspace
           rayOrigin.TransformCoordinate(mTargetInv);
           rayDirection.TransformNormal(mTargetInv);
           // needs to be normalized again, because of the scaling
           rayDirection.Normalize();

           return targetMesh.Intersect(rayOrigin, rayDirection);
       }

       private bool CheckForHits()
       {
           for (int i = 0; i < numTargets; ++i)
           {
               if (CheckTargetForHits(i)) return true;
           }
           return false;
       }


thats it .. i'm gonna send you the project back .. if you have any questions feel free to ask .. :)


ike
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
James_h1023Author Commented:
Thats is absolutely wonderful thank you.
Works perfectly.
Although one thing, in the original version the cursor was drawn at 1.0f and the targets started at 1.0f although here they are both 0.0f it appears.  This is where i don't understand the math, because if i put the cursor to 1.0f the rays don't work anymore?  Is it not possible to have the cursor at 1.0f anymore?
0
ikeworkCommented:
what do you mean by "cursor was drawn at 1.0f" and "put the cursor to 1.0f"  do you mean the z-value, where the cursor is rendered?
0
ikeworkCommented:
and where did you "put the cursor to 1.0f" ? can you show the code?
0
James_h1023Author Commented:
where the cursor was rendered:
device.Transform.World = Matrix.Translation(new Vector3(X, Y, 0.0f));
which then meant that the targets could move forward and the cursor would still be drawn on top of them:
float startDepth = 0.9f;
instead of
float startDepth = numInFront* depthStep;

This is probably completely wrong though!
0
James_h1023Author Commented:
Wonderful, so patient, you provided everything I needed :)
0
ikeworkCommented:
>> device.Transform.World = Matrix.Translation(new Vector3(X, Y, 0.0f));

then you have to change CheckTargetForHits as well:

       private bool CheckTargetForHits(int target)
       {
           // transform ray in target-space, we need the inverted target-matrix
           Matrix mTargetTransInv = Matrix.Translation(-targetPositions[target].X, -targetPositions[target].Y, -targetPositions[target].Z);
           Matrix mTargetScaleInv = Matrix.Scaling(1.0f / targetSizes[target].X, 1.0f / targetSizes[target].Y, 1.0f / targetSizes[target].Z);
           Matrix mTargetInv = mTargetTransInv * mTargetScaleInv;

           // calc ray in worldspace
           Vector3 rayOrigin = new Vector3(mouseCursor.X, mouseCursor.Y, 0.0f); // <<<<<< here
           Vector3 camPosition = new Vector3(headX, headY, headDist);
           Vector3 rayDirection = rayOrigin - camPosition;
           rayDirection.Normalize();
           float a = rayDirection.Length();

           // transform ray into targetspace
           rayOrigin.TransformCoordinate(mTargetInv);
           rayDirection.TransformNormal(mTargetInv);
           // needs to be normalized again, because of the scaling
           rayDirection.Normalize();

           return targetMesh.Intersect(rayOrigin, rayDirection);
       }

if you plan to change cursor's z value in your app, you better add a member to class CrosshairCursor:

class CrosshairCursor
    {
        public float lastX = 0;
        public float lastY = 0;
        public bool wasDown = false;
        public bool isDown = false;

        public float X = 0;
        public float Y = 0;
        public float Z = 0;   // << new member


and take that member instead of hardcoded 0.0f or 1.0f

0
James_h1023Author Commented:
Oh right yeh obviously, i'm sorry.
Thank you so much for all your help.
I owe you a drink.
James
0
ikeworkCommented:
youre very welcome .. gonna take something with whiskey .. ;)
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Game Programming

From novice to tech pro — start learning today.