Solved

# Cursor click on object detection

Posted on 2008-02-07
Medium Priority
1,324 Views
Hi,

I didn't get much luck with my last question so am re-wording it as I am closing in on the project deadline.

I have several objects (PrimitiveType.TriangleStrip) and have a cursor which i obviously can get the on screen co-ordinates of x and y although when i try and detect whether the cursor is over an object it doesn't work.  I'm guessing obviously because my objects are being translated etc. about in the world.

Any way to find out the on screen co-ordinates of the objects?
I've been doing lots of googling and it looks like i need to project a ray from my 2d co-ordinates of the mouse through the world and check for intersections with my object!

Thanks for any help.

James
0
Question by:James_h1023
• 17
• 13

LVL 20

Expert Comment

ID: 20842798
hi James,

right, you have to send a ray down the camera direction from the 3d-cursor-position and see if it intersects an object.
the thing is you have to calculate the 3d-world-point from the 2d-screen-point of the mouse-position. you can do it by calculation the screen-to-world-matrix. here is an example:

http://www.mvps.org/directx/articles/improved_ray_picking.htm

ike
0

LVL 4

Author Comment

ID: 20842974
Thanks for the link i will have a look at that :)

Just looking through my code again and have realised something i didn't know before (taking i'm adapting someone else's code).  The cursor is also a mesh.

The target and cursor i want to find if are intersecting are both vertexBuffers which are then drawn through DrawPrimitives.  Although the targets are translated as follows:
device.Transform.World *= Matrix.Translation(targetPositions[i]);
and the cursor as:
device.Transform.World = Matrix.Translation(new Vector3(X, Y, 1.0f));

I don't quite get why one is '=' and one is '*=' my matrices math is amazing!

I was thinking maybe i could just do a mesh intersection between the two! - I have several targets at different z co-ordinates so was thinking of cycling the mouse mesh backwards until intersects and the first one it intersects with would be the top target and if it reaches the back without intersecting then no collision!  Would that work maybe?

Don't know if that will affect the method because they are both VertexBuffer / meshs (my terms is also rusty)!

Thanks
0

LVL 20

Expert Comment

ID: 20843360
>> so was thinking of cycling the mouse mesh backwards until intersects and the first one it intersects with would be the top target

that might work .. although sending a ray (or rays from cursor's edges) might be easier and cleaner .. doing the mesh-collision test might be overkill for that case, isn't it .. and btw you have to make steps down the camer's z-axis .. you might skip an object doing those steps ..
0

LVL 4

Author Comment

ID: 20843602
Ah ok fair enough i will try the ray attempt then.

I am not very proficient in DirectX and the example went a bit over my head.
So; I have the a two vectors describing the positions of the target and the cursor (x,y,z).  I have to produce a ray from the position of the camera through the cursor, and check whether this intersects with the target; is this correct?
0

LVL 20

Expert Comment

ID: 20844028
right .. if you have bounding volumes (boxes/spheres or something) for your meshes, you can first test the ray with those boxes, if you have a hit, test the ray with the triangles of the mesh ..
0

LVL 4

Author Comment

ID: 20844145
I'm sorry i'm being very dumb - Ok i've got that although my target and cursor are are defined in a vertex buffer and then drawn as primitives.  Should i convert them to a mesh or somethign first, although because they are already in a vertex buffer couldn't i simply check for collisions with the triangles in the buffer.  My targets are simply a circle which a texture is drawn to, 2d facing towards the camera always.  And the cursor is simply like a target in a shooting game - a circle with two lines going through it!

I'm also in C# and having difficulty transferring from C++.  From the example i've attached one of Obj.cpp from the example, have I understood the following correctly?
vNear - the position of the ray at the camera.
vDir - the position of the ray far away.
m_conbinedMat - I found as the combination of matrices multiplication for the object being drawn.  So this would be akin my target vertices.
It then converts the ray to model space.
And checks for intersection although checks with a mesh, which it turns out i don't have, because I simply have a vertexBuffer with values in which represent my 2d circle facing towards the camera.

I have attached a screen shot hoping it might help!

// yes, convert ray to model space
D3DXVECTOR3 vNear,vDir;
D3DXMATRIX invMat;
D3DXMatrixInverse(&invMat,NULL,&m_combinedMat);
D3DXVec3TransformCoord(&vNear,pvNear,&invMat);
D3DXVec3TransformNormal(&vDir,pvDir,&invMat);

// test for intersection
BOOL bHit;
DWORD dwIndex;
float u,v;
float dist;
D3DXIntersect(m_pMesh,&vNear,&vDir,&bHit,&dwIndex,&u,&v,&dist,NULL,NULL);
example.jpg
0

LVL 20

Expert Comment

ID: 20844315
>>  couldn't i simply check for collisions with the triangles in the buffer

excactly, you only need the mesh, if you want to use the D3DXIntersect-function. you can make the ray-triangle-intersection-tests yourself, you will find a lot of example-code with a google search for it ..
0

LVL 4

Author Comment

ID: 20846296
Ok, i've redone my targets so they are meshs, no i'm trying to do the intersect, and have the following so far!

Vector3 near = new Vector3(cursorX, cursorY, 0);
Vector3 far = new Vector3(cursorX, cursorY, 1);
Matrix invMat;
float test;
//invMat = Matrix.Invert(targetMesh.
Vector3 nearInv, farInv;
nearInv = Vector3.TransformCoordinate(near, invMat);
farInv = Vector3.TransformNormal(far, invMat);

IntersectInformation closestIntersection;
hit = targetMesh.Intersect(near, far, out closestIntersection);

Except i don't know how to make the inverse matrix that is needed, some websites say its the inverse matrix of the mesh I am checking for intersection with and others half say its the inverse of the world matrix!

James
0

LVL 4

Author Comment

ID: 20846901
edit:
i've not got this far, although it doesn't seem to work all the time, and sometimes it apparently registers a hit when the cursor coordinates aren't over the target mesh.

Am i doing something obviously wrong here?
float cursorX = mouseCursor.getCursorX();
float cursorY = mouseCursor.getCursorY();

Vector3 near = new Vector3(cursorX, cursorY, 0);
Vector3 far = new Vector3(cursorX, cursorY, 1);

// Transform points to world space
near.Unproject(device.Viewport, device.Transform.Projection, device.Transform.View, device.Transform.World);
far.Unproject(device.Viewport, device.Transform.Projection, device.Transform.View, device.Transform.World);
far.Normalize();

// Retrieve intersection information
IntersectInformation closestIntersection;
hit = targetMesh.Intersect(near, far, out closestIntersection);
0

LVL 20

Expert Comment

ID: 20848857
why dont you use the 3d-position of the cursor as startingpoint of the ray? you said you already have the 3d-position of the cursor...

Vector3 rayOrigin = 3d_cursor_position;
Vector3 rayDirection = 3d_cursor_position - 3d_camera_position;
rayDirection.Normalize();

// Retrieve intersection information
IntersectInformation closestIntersection;
hit = targetMesh.Intersect(rayOrigin, rayDirection, out closestIntersection);
0

LVL 4

Author Comment

ID: 20849613
I've tried that, and now it returns true all the time, no matter where I move the cursor.
Do i still need the transformation to world space with lines 4 & 5.
I tried with it commented and not, and with it uncommented the results go true, false, true, false repetitively quickly as the mouse moves, and with lines 4 & 5 uncommented the result is true all the time.
float cursorX = mouseCursor.getCursorX();
float cursorY = mouseCursor.getCursorY();
// yes, convert ray to model space
Vector3 near = new Vector3(cursorX, cursorY, 1);

// Transform points to world space
near.Unproject(device.Viewport, device.Transform.Projection, device.Transform.View, device.Transform.World);
far.Unproject(device.Viewport, device.Transform.Projection, device.Transform.View, device.Transform.World);
far.Normalize();

// Retrieve intersection information
IntersectInformation closestIntersection;
hit = targetMesh[1].Intersect(near, far, out closestIntersection);
0

LVL 4

Author Comment

ID: 20849615
EDIT: i mean lines 8 & 9 not lines 4 & 5.
0

LVL 20

Expert Comment

ID: 20849662
are "mouseCursor.getCursorX()" screen-coordinates or 3d-world-coordinates?
0

LVL 20

Expert Comment

ID: 20849675
this implies "mouseCursor.getCursorX()" is in worldspace:

otherwise you could not substract them, since camera is in world-space
0

LVL 20

Expert Comment

ID: 20849696
pseudocode:

Vector3 screen_mouse(screen_x, screen_y, 1 );
Vector3 world_mouse = screen2world_matrix * screen_mouse;

Vector3 rayOrigin = world_mouse;
Vector3 rayNormal = world_mouse - world_camera;
rayNormal.Normalize();

// Retrieve intersection information
IntersectInformation closestIntersection;
hit = targetMesh[1].Intersect(rayOrigin, rayNormal, out closestIntersection);
0

LVL 20

Expert Comment

ID: 20849699
are you sure "headDist" is the camera z-position. the name sounds not like that ..
0

LVL 20

Expert Comment

ID: 20849708
ok .. is it a big project? if not send it to my email and i'm gonna have a look at it when i'm home .. my mail is in my profile ..
0

LVL 4

Author Comment

ID: 20849710
I do believe they are 3d world coordinates because i draw my cursor including:
device.Transform.World = Matrix.Translation(new Vector3(X, Y, 1.0f));
where X, and Y are the same variables i return with my getter methods.

So ok, that would mean no lines 8 & 9 because no need to swap to 3d world coordinates right?
So them uncommented returns true where ever i move the cursor.
0

LVL 4

Author Comment

ID: 20849787
That is very nice of you.
Thanks.
0

LVL 20

Expert Comment

ID: 20849808
>> So ok, that would mean no lines 8 & 9 because no need to swap to 3d world coordinates right?

yes, if X and Y is in world-space

ah ok .. you have to transform the ray into meshspace. you can do that by inversing the mesh-matrix and multiply your ray with it before intersection-test

but without seeing the entire code .. its just guesses
0

LVL 20

Expert Comment

ID: 20849811
no prob .. you're welcome .. ;)
0

LVL 20

Accepted Solution

ikework earned 2000 total points
ID: 20861047
hey james,

i had a look at your project and fixed it .. i took the original project, because in your project some things did not work anymore

- i switched off the wiiremote, i dont have one ;)  you can move the cursor with the mouse, when it hits a target you can see it on the rendered text
- i rendered the mesh above each target, you see it is a rectangle actually which was used by the author, the texture lets it just look like an circle
- i just used one mesh, i used targetSizes[] and targetPositions[] to put the mesh for each target in worldspace and render it

here is the most important part, the collision check:

private bool CheckTargetForHits(int target)
{
// transform ray in target-space, we need the inverted target-matrix
Matrix mTargetTransInv = Matrix.Translation(-targetPositions[target].X, -targetPositions[target].Y, -targetPositions[target].Z);
Matrix mTargetScaleInv = Matrix.Scaling(1.0f / targetSizes[target].X, 1.0f / targetSizes[target].Y, 1.0f / targetSizes[target].Z);
Matrix mTargetInv = mTargetTransInv * mTargetScaleInv;

// calc ray in worldspace
Vector3 rayOrigin = new Vector3(mouseCursor.X, mouseCursor.Y, 0.0f);
Vector3 rayDirection = rayOrigin - camPosition;
rayDirection.Normalize();
float a = rayDirection.Length();

// transform ray into targetspace
rayOrigin.TransformCoordinate(mTargetInv);
rayDirection.TransformNormal(mTargetInv);
// needs to be normalized again, because of the scaling
rayDirection.Normalize();

return targetMesh.Intersect(rayOrigin, rayDirection);
}

private bool CheckForHits()
{
for (int i = 0; i < numTargets; ++i)
{
if (CheckTargetForHits(i)) return true;
}
return false;
}

thats it .. i'm gonna send you the project back .. if you have any questions feel free to ask .. :)

ike
0

LVL 4

Author Comment

ID: 20861338
Thats is absolutely wonderful thank you.
Works perfectly.
Although one thing, in the original version the cursor was drawn at 1.0f and the targets started at 1.0f although here they are both 0.0f it appears.  This is where i don't understand the math, because if i put the cursor to 1.0f the rays don't work anymore?  Is it not possible to have the cursor at 1.0f anymore?
0

LVL 20

Expert Comment

ID: 20861431
what do you mean by "cursor was drawn at 1.0f" and "put the cursor to 1.0f"  do you mean the z-value, where the cursor is rendered?
0

LVL 20

Expert Comment

ID: 20861432
and where did you "put the cursor to 1.0f" ? can you show the code?
0

LVL 4

Author Comment

ID: 20861439
where the cursor was rendered:
device.Transform.World = Matrix.Translation(new Vector3(X, Y, 0.0f));
which then meant that the targets could move forward and the cursor would still be drawn on top of them:
float startDepth = 0.9f;
float startDepth = numInFront* depthStep;

This is probably completely wrong though!
0

LVL 4

Author Closing Comment

ID: 31428877
Wonderful, so patient, you provided everything I needed :)
0

LVL 20

Expert Comment

ID: 20861461
>> device.Transform.World = Matrix.Translation(new Vector3(X, Y, 0.0f));

then you have to change CheckTargetForHits as well:

private bool CheckTargetForHits(int target)
{
// transform ray in target-space, we need the inverted target-matrix
Matrix mTargetTransInv = Matrix.Translation(-targetPositions[target].X, -targetPositions[target].Y, -targetPositions[target].Z);
Matrix mTargetScaleInv = Matrix.Scaling(1.0f / targetSizes[target].X, 1.0f / targetSizes[target].Y, 1.0f / targetSizes[target].Z);
Matrix mTargetInv = mTargetTransInv * mTargetScaleInv;

// calc ray in worldspace
Vector3 rayOrigin = new Vector3(mouseCursor.X, mouseCursor.Y, 0.0f); // <<<<<< here
Vector3 rayDirection = rayOrigin - camPosition;
rayDirection.Normalize();
float a = rayDirection.Length();

// transform ray into targetspace
rayOrigin.TransformCoordinate(mTargetInv);
rayDirection.TransformNormal(mTargetInv);
// needs to be normalized again, because of the scaling
rayDirection.Normalize();

return targetMesh.Intersect(rayOrigin, rayDirection);
}

if you plan to change cursor's z value in your app, you better add a member to class CrosshairCursor:

class CrosshairCursor
{
public float lastX = 0;
public float lastY = 0;
public bool wasDown = false;
public bool isDown = false;

public float X = 0;
public float Y = 0;
public float Z = 0;   // << new member

and take that member instead of hardcoded 0.0f or 1.0f

0

LVL 4

Author Comment

ID: 20861467
Oh right yeh obviously, i'm sorry.
Thank you so much for all your help.
I owe you a drink.
James
0

LVL 20

Expert Comment

ID: 20861493
youre very welcome .. gonna take something with whiskey .. ;)
0

## Featured Post

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

What is RenderMan: RenderMan is a not any particular piece of software. RenderMan is an industry standard, defining set of rules that any rendering software should use, to be RenderMan-compliant. Pixar's RenderMan is a flagship implementation of …
As game developers, we quickly learn that Artificial Intelligence (AI) doesn’t need to be so tough.  To reference Space Ghost: “Moltar, I have a giant brain that is able to reduce any complex machine into a simple yes or no answer. (http://www.youtu…
In this video I will demonstrate how to set up Nine, which I now consider the best alternative email app to Touchdown.
Get the source code for a fully functional Access application shell with several popular security features that Access VBA application developers desire, but find difficult or impossible to figure out how to code. You get the source code for managi…
###### Suggested Courses
Course of the Month7 days, 6 hours left to enroll