OpenGL picking (hit list)

I'm trying to create a hit list for selecting objects in OpenGL.  Problem is, if an object is on the viewport, it's being recognized as a hit, whether I'm clicking the object or not.  Any help is appreciated.
void startPicking(int x, int y) {
	GLint viewport[4];
 
	glSelectBuffer(512, selectBuf);
	glRenderMode(GL_SELECT);
 
	glMatrixMode(GL_PROJECTION);
	glPushMatrix();
	glLoadIdentity();
 
	glGetIntegerv(GL_VIEWPORT, viewport);
	gluPickMatrix(x, viewport[3]-y, 5, 5, viewport);
	gluPerspective(45, xy_aspect, 0.1, 1000);
	glMatrixMode(GL_MODELVIEW);
	glInitNames();
}
 
void processHits(GLint hits, GLuint buffer[])
{
   int i, j;
   int names, *ptr, minZ, *ptrNames, numberOfNames;
 
   printf ("hits = %d\n", hits);
   ptr = (int *) buffer;
   minZ = 0xffffffff;
   for (i = 0; i < hits; i++) {	
      names = *ptr;
	  ptr++;
	  if (*ptr < minZ) {
		  numberOfNames = names;
		  minZ = *ptr;
		  ptrNames = ptr+2;
	  }
	  
	  ptr += names+2;
	}
  printf ("The closest hit names are ");
  ptr = ptrNames;
  for (j = 0; j < numberOfNames; j++,ptr++) {
     printf ("%d ", *ptr);
  }
  printf ("\n");
   
}
 
void stopPicking(int x, int y) {
	int hits;
	
	// restoring the original projection matrix
	glMatrixMode(GL_PROJECTION);
	glPopMatrix();
	glMatrixMode(GL_MODELVIEW);
	glFlush();
	
	// returning to normal rendering mode
	hits = glRenderMode(GL_RENDER);
	
	// if there are hits process them
	if (hits != 0)
		processHits(hits,selectBuf);
}
 
void mouse(int button, int state, int x, int y) {
	if(button==GLUT_LEFT_BUTTON) {
		switch(state) {
			case GLUT_DOWN:
				startPicking(x, y);
				break;
			case GLUT_UP:
				stopPicking(x, y);
				break;
		}
	}
}
 
void display(void) {
	glClearColor( .9f, .9f, .9f, 1.0f );
	glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
 
	glMatrixMode(GL_PROJECTION);
	glLoadIdentity();
	gluPerspective(45, xy_aspect, 0.1, 1000);
 
	glMatrixMode(GL_MODELVIEW);
	...
	glPushName(0);
	glutSolidSphere(...);
	glPopName();
	...
}

Open in new window

palehoseAsked:
Who is Participating?
 
palehoseConnect With a Mentor Author Commented:
Okay, I found the problem and a solution.  Here it is to possibly save someone some time:

It appears this is probably a bug with GLUI, but using a BOTTOM (or TOP) subwindow screws up the PROJECTION matrix.  That's why my program thought the object was above where it actually renders.  My fix is listed, but it requires a static size, which is fine for what I'm trying to implement.  I'm translating the SIZE*(height of the GLUI subwindow) in the negative direction.  You need to make sure you acquire the viewport again after doing this translation, like my code shows.

void startPicking(int x, int y) {
	GLint viewport[4];
 
	glSelectBuffer(512, selectBuf);
	glRenderMode(GL_SELECT);
 
	glInitNames();
   	glPushName(0);
 
	glMatrixMode(GL_PROJECTION);
	glPushMatrix();
	glLoadIdentity();
 
	glGetIntegerv(GL_VIEWPORT, viewport);
	glTranslatef(0, (float)(-SIZE*viewport[1]), 0);
 
	glGetIntegerv(GL_VIEWPORT, viewport);
	gluPickMatrix(x, viewport[3]-y, 5, 5, viewport);
 
	gluPerspective(45, xy_aspect, 0.1, 1000);
 
	display();
 
	glMatrixMode(GL_PROJECTION);
	glPopMatrix();
	glMatrixMode(GL_MODELVIEW);
	glFlush();
 
   	int hits = glRenderMode(GL_RENDER);
	processHits(hits, selectBuf);
}

Open in new window

0
 
palehoseAuthor Commented:
I've made some progress on this, and actually have somewhat of a solution.  The only problem is the coordinates somehow appear to be off.  It says I'm picking an object outside of the actual object.  Anyone know why?
0
 
ikeworkCommented:
hi palehose,

can you post the entire code?

ike
0
Cloud Class® Course: Certified Penetration Testing

This CPTE Certified Penetration Testing Engineer course covers everything you need to know about becoming a Certified Penetration Testing Engineer. Career Path: Professional roles include Ethical Hackers, Security Consultants, System Administrators, and Chief Security Officers.

 
palehoseAuthor Commented:
Hi ikework,
I uploaded it here: http://www.tomsisk.com/Modeler.cpp if that's okay.

Thanks for the help.
0
 
ikeworkCommented:
you dont pass the mouse-x and -y position to gluPickMatrix here:

  gluPickMatrix(x, viewport[3]-y, 5, 5, viewport);

should be something like:

  gluPickMatrix((GLdouble) mouse_x, (GLdouble) (viewport[3]-mouse_y), 1.0f, 1.0f, viewport);

have a look at:

  http://nehe.gamedev.net/data/lessons/lesson.asp?lesson=32

for details
0
 
ikeworkCommented:
this code, you call twice, once in startPicking and once in stopPicking

      glMatrixMode(GL_PROJECTION);
      glPopMatrix();
      glMatrixMode(GL_MODELVIEW);
      glFlush();

         int hits = glRenderMode(GL_RENDER);
      if(hits != 0) processHits(hits, selectBuf);

it should only be called once per picking-loop
0
 
palehoseAuthor Commented:
I am passing the mouse information, that's what x and y is.  It's passed from the mouse click.

And my mistake on the second issue.  stopPicking isn't even being used, I just haven't deleted it.
0
 
palehoseAuthor Commented:
Again, my problem is what is considered clicking the object.  It's a little off (clicking higher than the object is considered picking, and just below the middle of the object, or so, is not considered picking.
0
 
ikeworkCommented:
>> I am passing the mouse information, that's what x and y is.  It's passed from the mouse click.

right, sorry .. too early .. need some coffee .. ;)

>> It's a little off (clicking higher than the object is considered picking, and just below the middle of the object, or so, is not considered picking.

ok .. i see .. gonna check that
0
 
palehoseAuthor Commented:
Any luck finding what's wrong?  I'm getting frustrated here.
0
 
ikeworkCommented:
im at work right now, gonna have a look at it later.. can you make a smaller sample-app without gui, which has the same issue? if you can upload it, i can have a deeper look at it
0
 
palehoseAuthor Commented:
Haven't had time to do that, but I'm still troubleshooting this and it appears that the object is in a different place then it is rendered.  I currently have it set up so it returns the color of the pixel that I click, and it is off, just as it is when I try using the hit list.  Would that have to do with where my camera is located?  I thought I knew how gluPerspective worked, but now I'm not so sure.
0
 
palehoseAuthor Commented:
Alright, I did what you wanted me to do and removed all the GLUI stuff, and it works correctly.  I don't know what was throwing it off, but I guess I'm going to have to re-do all that (ugh).  I'll let you know if I have any other issues.
0
 
palehoseAuthor Commented:
So I'm convinced the problem is when the GLUI has elements added (and thus making the viewport smaller, to make room for the new elements).  When the viewport is shifted upwards (I have a glui subwindow on the bottom), the selection buffer (I'm guessing) does not shrink, but simply shifts up.  Is this not the correct thing to do with reshape?

void reshape(int w, int h) {
	width = w;
	height = h;
 
	int vx, vy, vw, vh;
	GLUI_Master.get_viewport_area(&vx, &vy, &vw, &vh);
	glViewport(vx, vy, vw, vh);
 
	xy_aspect = (float)width / (float)height;
	glutPostRedisplay();
}

Open in new window

0
 
ikeworkConnect With a Mentor Commented:
hey palehose, glad to hear that .. :) lets not delete it, this can be useful to others ..
0
 
ikeworkCommented:
and btw. im not quite done with it, are you? we should debug it better, to understand what happens and why.
normally the projection-matrix *cant* be off, because it is used by the renderer, then it wouldnt be rendered in the right way, would it?
0
 
palehoseAuthor Commented:
I was not deleting it, just closing it.

Personally, yes, I am done with it.  This is a school project, and I don't have the time to get into the details of WHY it works the way it does.   I will give you some points for helping me move in the right direction.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.