how to find coordinates on window to click on button displayed in active x control

add those declarations:
Private Const HC_ACTION As Integer = 0
Private Const WH_MOUSE_LL As Integer = 14
Private Const WM_LBUTTONDOWN As Integer = &H201
Private Const WM_MOUSEWHEEL As Integer = &H20A

and in MouseHookProc():
If (nCode = HC_ACTION) Then

If wParam = WM_LBUTTONDOWN Then

''do something on left mouse click

End If

any progress mate?

Hello, just got up and on the way to work now. Thx for the feedback.  I'll look at this asap.

Modalot and experts, apologies if this creates confusion.  I will be more conscious of this in the future.   thanks.

sedgewick, thank you again.   i haven't had the opportunity to try out the code you posted as I think (or hoping at least) that some progress is being made with the current code base.   I will look more in depth at what you provided as soon as possible and make use of it if needed ... thank u very much for providing this code.

In the meantime, since the last post I made some additional modifications to the code -

I first find the internet explorer window using the WindowsEnumerator class.  
- I find the appropriate top level window as you can see in the code I try to find a match on the string, "crystal", to find the appropriate IE popup window.  

- Once I find the match, the program finds the child window where the class name = ATL:07868280 (I used spy++ to find the lower level window and get the class name of where the export button is located.)

When running the code, I see now that it is able to find the window with that export button on it so it seems I've narrowed the area that needs to be clicked on now so the code wouldn't be thrown off by the toolbar, etc.

Now, I am trying to figure out how to click on the appropriate coordinates.

thanks for this additional info.  I will review this in more depth.
 
In the meantime, since my last post I tried something different which is to find the internet explorer window using the, WindowsEnumerator class.   
- I find the appropriate top level window as you can see in the code I try to find a match on the string, crystal to find the IE popup.   
 
- Once I find the match, find the child window where the class name = ATL:07868280 (I used spy++ to find the lower level window and get the class name of where the export button is located.)
 
When I run the code it is able to find the window with that export button on it so it seems Ive narrowed the area that needs to be clicked on.   
 
Now, I am trying to figure out how to click on the appropriate coordinates.
                                              

1:
2:
3:
4:
5:
6:
7:
8:
9:
10:

sorry, so you can see what I've done here's that code

Dim DialogWindows As List(Of ApiWindow)
            Dim windowflag As Boolean = False
 
 
While windowflag <> True
     Application.DoEvents()
     DialogWindows = we.GetTopLevelWindows("IEFrame")
     
     If DialogWindows.Count > 1 Then
          windowflag = True
          Debug.Print("windowflagloop count:" & DialogWindows.Count.ToString)
     End If
End While
           
Dim DialogRect As RECT
 
For Each aw In DialogWindows
   If aw.Text.ToLower.Contains("crystal") Then
       For Each ctl In we.GetChildWindows(aw.Handle)
       
            If ctl.ClassName = ("ATL:07868280") Then
                 Debug.Print(" Got ATL:07868280")
            End If
           
            GetWindowRect(aw.Handle, DialogRect)
            ClickAt(DialogRect.Left + 5, DialogRect.Top + 2, True, False)
            Debug.Print("clicked on window")
            
            Exit Sub
       Next
      
   Else
           Debug.Print("not the right window")
   End If
 
Next
                                              

1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:

Ok...so if you've got a handle to the ToolBar area then just use Spy++ to show you the bounding box with "highlight window" feature (or is it "show window"...not sure).  Anyhoo, it will show you the are that belongs to that window.  You can calculate the needed offset by taking a screenshot and then cropping the image so just that toolbar bounding box is visible.  Once you have the offset you pass the toolbar handle to GetWindowRect() as before and add the offset values to the x, y coords.

I am passing the handle of the childwindow (the toolbar area) to GetWindowRect.

So for example, using ClickAt(DialogRect.Left + 35, DialogRect.Top - 5, True, False)

DialogRect.Left + 15 thru + 35 appear to be clicking on the refresh button.   I can see the the window refreshing... there is a bar on the right side of that window that shows the refresh taking place.

Since the export button is to the left of the refresh button I have tried decreasing the #  + 0 to + 15 but it's not clicking on the export button.

Note: for DialogRect.Top I used a negative number because .Top gets the y-axis position of the top of the rectangle.   If I use a positive # I'm not able to make any contact with the buttons.
ClickAt(DialogRect.Left + 35, DialogRect.Top - 5, True, False)

I still a bit unclear on how to get the "offset".   When I enter + 50, how far to the right is this of the left side of the rectangle?    Does +50 mean 50 pixels and how much is that .. is it a millimeter, inch?

Does +50 move the position or coordinates from 771 to 821 on the x axis?

.Top - Gets the y-axis position of the top of the rectangle.  - Since the button is below the top, should this # be negative?

.Left - Gets the x-axis value of the left side of the rectangle.

   GetWindowRect(aw.Handle, DialogRect)
                        Debug.Print("left " & DialogRect.Left.ToString)
                        Debug.Print("top " & DialogRect.Top.ToString)
                        Debug.Print("top max val: " & DialogRect.Top.MaxValue.ToString())
                        Debug.Print("top min val: " & DialogRect.Top.MinValue.ToString())
                        Debug.Print("Left min val: " & DialogRect.Left.MinValue.ToString)
                        Debug.Print("Left max val: " & DialogRect.Left.MaxValue.ToString)

                        ClickAt(DialogRect.Left + 200, DialogRect.Top + 100, True, False)

I don't know if I'm interpreting this correctly or if it matters really but seems the range for that small rectangular tool bar is between -2147483648 and 2147483647?

Debug.print results:
left 771
top 174
top max val: 2147483647
top min val: -2147483648
Left min val: -2147483648
Left max val: 2147483647

It's all in PIXELS and the Offsets should be positive to move the cursor to the right and down from the Top Left corner of the Rectangle.

I would take half of the Height and add that to the Top value to get the middle of the Toolbar.  Then it's just a matter of figuring out the X offset value.

Try 70 for the Left Offset:

    GetWindowRect(aw.Handle, DialogRect)
    ClickAt(DialogRect.Left + 70, DialogRect.Top + ((DialogRect.Bottom - DialogRect.Top) / 2), True, False)

I took a screenshot as you suggested and then cropped the image so just that toolbar bounding box is visible.

I took the screenshot in my snag it tool - when I highlighted over the image I can see the coordinates at the bottom of the screen

63,13
169X19X24 bit

I tried changing the code as follows:

        ClickAt(DialogRect.Left + 60, DialogRect.Top + 13, True, False)

but it doesn't appear to be clicking the right spot.

the 63,13 is when I put the mouse arrow over the "export" button

In the ClickAt() sub, after the "Cursor.Position = New Point(x, y)" line, is the cursor physically in the correct position onscreen?  If not, how far off is it?

I'm on 32 bit.  

I took the full screenshot and when i put the mouse over the envelope it shows 918, 438

When i ran the code with the code in the msg box you provided it shows 889, 245.

>>>In the ClickAt() sub, after the "Cursor.Position = New Point(x, y)" line, is the cursor physically in the >>>correct position onscreen?  If not, how far off is it?
sorry, I did not understand how to do this or what to check for here.   since the values above didn't match, I'm presuming this could be the problem?

I tried changing the values below until that message box popup you advised to put in displayed 918, 438 (these are the x, y values for the envelope on the full screenshot)
Dim pt As New Point(DialogRect.Left + 60, DialogRect.Top + 13) - CHANGED THESE VALUES

Used the new values  below to make the values in the message box popup match up w/ the coordinates of the full screenshot (918,438)
Dim pt As New Point(DialogRect.Left + 89, DialogRect.Top + 206)
- the message box popup displays 918, 438

fyi - sorry for any confusion, the moderator split the points on this because I have another thread on the same issue.   for your reference the link is below:

http://www.experts-exchange.com/Programming/Languages/.NET/Visual_Basic.NET/Q_24660287.html#a25735511

idlemind,I don't know how I missed this comment - 25735008

Anyhow, I tried what you suggested:

Try 70 for the Left Offset:

    GetWindowRect(aw.Handle, DialogRect)
    ClickAt(DialogRect.Left + 70, DialogRect.Top + ((DialogRect.Bottom - DialogRect.Top) / 2), True, False)

This didn't work so I tried different values for DialogRect.Left ranging from 0 - 145 (0, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, etc. etc.) but for some reason it's still not clicking that button

DialogRect as I understand it should have the screen coordinates of the child window right since I passed it to the GetWindowRect method.

GetWindowRect(aw.handle, DialogRect)

With the code you provided, It seems as simple as finding the X coordinate now but i tried with what seems to be the logical left offsets ... many of them but none work? : /

May need to tweak the mouse_event() calls to make sure it's actually hitting what we want.  We could add the MOUSEEVENTF_ABSOLUTE flag back in.  I think it's:

    Public Const MOUSEEVENTF_ABSOLUTE As Integer = &H8000
    Public Const MOUSEEVENTF_MOVE As Integer = &H1
    ...
       mouse_event(MOUSEEVENTF_ABSOLUTE Or MOUSEEVENTF_MOVE, x, y, 0, 0)
       mouse_event(MOUSEEVENTF_LEFTDOWN, 0, 0, 0, 0)
       mouse_event(MOUSEEVENTF_LEFTUP, 0, 0, 0, 0)

We could also try adding a small delay (100 ms) between LeftDown and LeftUp to see if that makes a difference.

idlemind, thank you again for helping me out on this!  Will try it out tomm and let you know how it goes.

this morning I added the code you have there above.   IRan the code many times using values a range of values 0 - 145 for DialogRect.Left

ClickAt(DialogRect.Left + 145, DialogRect.Top + ((DialogRect.Bottom - DialogRect.Top) / 2), True, False)

In debug mode, i checked the x,y values when Clickat method is run.   For example, when using the code above, x value will be the offset or 145, and the y value is always 0.  

Also, added a sleep as you suggested.   Still no response though from the button.

 Private Sub ClickAt(ByVal x As Integer, ByVal y As Integer, ByVal LeftClick As Boolean, ByVal RestorePosition As Boolean)
        Dim pt As Point = Cursor.Position
        Cursor.Position = New Point(x, y)
 
        If LeftClick Then
            mouse_event(MOUSEEVENTF_ABSOLUTE Or MOUSEEVENTF_MOVE, x, y, 0, 0)
            mouse_event(MOUSEEVENTF_LEFTDOWN, 0, 0, 0, 0)
            System.Threading.Thread.Sleep(100)
            mouse_event(MOUSEEVENTF_LEFTUP, 0, 0, 0, 0)
        Else
            mouse_event(MOUSEEVENTF_RIGHTDOWN, 0, 0, 0, 0)
            mouse_event(MOUSEEVENTF_RIGHTUP, 0, 0, 0, 0)
        End If
        If RestorePosition Then
            Cursor.Position = pt
        End If
    End Sub

                                              

1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:

is the problem problem related what thelearnedone mentioned,
"It looks like you are trying to use relative coordinates, but that won't be from the Cursor.Position, but from the last call to mouse_event.  It is tough to know whether the event needs client or screen coordinates (client are relative to the form's position).  My guess is that it needs screen coordinates."

i'm having difficulty making sense of the types of coordinates
client coordinates - relative to form's position
screen coordinates
relative coordinates - looks like we are trying to use this one.

My understanding is that we got the dimensions of the bounding rectangle (assuming this is the child window - that tool bar with the export button in it?) using GetWindowRectFunction we passed the handle of the childwindow to it along with DialogRect which becomes a pointer to a structure that receives the screen coordinates.

So when we call the clickat method it presses where the cursor position is set.

cursor.position = new point (x,y)

Is the mouse then pressing relative to coordinates on the child window or the whole screen?   I'm a bit confused conceptually understanding this.

We are working in screen coords as that is what is returned by GetWindowRect().

We also moved the cursor with mouse_event():

    mouse_event(MOUSEEVENTF_ABSOLUTE Or MOUSEEVENTF_MOVE, x, y, 0, 0)

So that the last position seen by mouse_event() is the correct position in screen coords.

Not sure why it isn't working...either it's hitting the wrong spot or he actual button doesn't see the MOUSEEVENTF_LEFTDOWN() and MOUSEEVENTF_LEFTUP() actions as a "click".

You could try switching to SendInput() instead and see if that makes a difference:
http://msdn.microsoft.com/en-us/library/ms646310(VS.85).aspx
 

i did some more troubleshooting and it looks like the code is not getting the child window so I think this is why it's not able to click

Apparently the ctl.ClassName isn't a static value

I'm not sure why earlier the code was able to find the child window which mislead me to think it was finding it all the time .... but now the class name for that child window appears to change every time.

I used spy++ to see if I could find another window where the classname doesn't change.   I'm using classname "ATL:STATIC" and made the code changes below.

Still not finding it when I run the code but that's where i'm at right now....

 For Each aw In DialogWindows
                Debug.Print(aw.Text.ToString)
                If aw.Text.ToLower.Contains("crystal") Or aw.Text.ToLower.Contains("xxxxs.rpt") Then
                    For Each ctl In we.GetChildWindows(aw.Handle)
                        If ctl.ClassName.ToLower = ("ATL:STATIC") Then
                            Debug.Print(" Got ATL:STATIC")
                            GetWindowRect(aw.Handle, DialogRect)
                            ClickAt(DialogRect.Left + 70, DialogRect.Top + ((DialogRect.Bottom - DialogRect.Top) / 2), True, False)
                            Debug.Print("got to childwindow and clicked")
                            'Exit Sub
                        Else
                            Debug.Print("Not right child window")
                        End If
                    Next
                Else
                    Debug.Print("not the right Top level window")
                End If
            Next

                                              

1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:

here is ATL:STATIC

and

Spy++ properties for ATL:STATIC

Not sure if this could provide additional insight...

ok sorry in advance for any confusion on the previous msg, I should've waited to post

I went back to getting the child window by using class name for the window as follows:

  For Each aw In DialogWindows
                Debug.Print(aw.Text.ToString)
                If aw.Text.ToLower.Contains("crystal") Or aw.Text.ToLower.Contains("xxx.rpt") Then
                    For Each ctl In we.GetChildWindows(aw.Handle)
                        If ctl.ClassName.Contains("TabWindowClass") Then

"TabWindowClass" is the classname I'm using to retrieve a child window within the internet browser where the active x viewer displays the report.   It's a larger area (essentially the entire active x viewer area) vs. trying to get the smaller window (just the tool bar section).   It's Parent's classname is the IEFrame Class.

Maybe the problem before was I was trying to retrieve a child of a child window....

When I run the code, I can successfully get the child window again each time I run it.   Now again, it seems to be a matter of clicking on the right coordinates.

Still not working with Sendinput either.  

I'm using the following now:
'Click on button
 ClickAt(DialogRect.Left + 85, DialogRect.Top + 32, True, False)

'Display coordinates
 Dim pt As New Point(DialogRect.Left + 85, DialogRect.Top + 32)
 MessageBox.Show(pt.ToString())

When running the code, the message box appears and displays the coordinates. I take a full screenshot, compare and then see that the coordinates in the msgbox {X=118,Y=237} match exactly the location of the export button on the full screenshot.   In the program that I view the full screenshot in, can see the coordinates of the export button as I put my mouse over it.   They match.

I am now using SendInput as suggested.   You can see the changes I made in the "ClickAt" method as shown in code snippet below but still no luck.

Private Sub ClickAt(ByVal x As Integer, ByVal y As Integer, ByVal LeftClick As Boolean, ByVal RestorePosition As Boolean) 
        Dim inputEvents(0) As INPUT_TYPE 
        Dim pt As Point = Cursor.Position
        Cursor.Position = New Point(x, y) 
        If LeftClick Then
            Debug.Print("Clicking export button now")
            inputEvents(0).xi.dx = 0
            inputEvents(0).xi.dy = 0
            inputEvents(0).xi.mouseData = 0
            inputEvents(0).xi.dwFlags = M_MOVE + M_LD + M_LU
            inputEvents(0).xi.dwtime = 0
            inputEvents(0).xi.dwExtraInfo = 0
            inputEvents(0).dwType = INPUT_MOUSE 
            Dim SI_returnval As Integer 
   SI_returnval = SendInput(1, inputEvents(0), Len(inputEvents(0)))
  
'SEND INPUT RETURNS SUCCESSFUL WITH A 1  
        Else
            mouse_event(MOUSEEVENTF_RIGHTDOWN, 0, 0, 0, 0)
            mouse_event(MOUSEEVENTF_RIGHTUP, 0, 0, 0, 0)
        End If
        If RestorePosition Then
            Cursor.Position = pt
        End If
    End Sub

                                              

1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:

Another relevant point I just realized is that after running the code my mouse arrow moves itself to the "export report" (the envelope)  .... it's pointing exactly to the right spot so this confirms to me that the coordinates I used are correct.

Somehow though, the button isn't getting pressed.

this behavior with the mouse arrow moving to the button is not happening with the "mouse_event" approach,only with "SendInput" so sendinput looks more promising.

In the above approach, the cursor seems to be in the right place.   The actual button though doesn't appear to see the MOUSEEVENTF_LEFTDOWN() and MOUSEEVENTF_LEFTUP() actions as a "click".

Is there anything else here that I can try or insight anyone has about why this doesn't work?   I've been searching the for web similar issues, examples, etc. but everything I have found so far has either been not specific enough, too confusing and or just doesn't work.

Note, I also tried another approach using PostMessage below but that didn't work either.

Dim pt As New Point(DialogRect.Left + 85, DialogRect.Top + 32)
MessageBox.Show(pt.ToString()) 
I assigned values to x, y below using the values that show in MessageBox.Show above. 

'USING POSTMESSAGE INSTEAD OF SENDINPUT 
Dim x As Integer = 321
Dim y As Integer = 440 
Dim pt As Point = Cursor.Position
Cursor.Position = New Point(x, y) 
Dim strResult As String 
strResult = PostMessage(ctl.Handle, MOUSEEVENTF_LEFTDOWN, 0, IntPtr.Zero) 
strResult = PostMessage(ctl.Handle, MOUSEEVENTF_LEFTUP, 0, IntPtr.Zero)
                                              

1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:

I'm still struggling with this .... please see my latest update below and advise if there is anything you can think of.   I feel like I'm getting closeR here.

Before calling the "ClickAt" method to click on the export button, I retrieve the popup window and pass the handle of the window to GetWindowRect

Again, when the code runs, I can see the mouse arrow or cursor move itself to the correct coordinates, putting itself right on top of the export button to click it.  

If I can click on the button myself using a left click, there's no reason I shouldn't be able to do the same programatically right?

Would someone please look at my code below and provide feedback on possible other approaches or something I did wrong?   Thanks in advance.

*** code before calling the ClickAt method ***  
GetWindowRect(ctl.Handle, DialogRect)
ClickAt(DialogRect.Left + 85, DialogRect.Top + 32, True, False) 
*** Here is the ClickAt method ****** 
Private Sub ClickAt(ByVal x As Integer, ByVal y As Integer, ByVal LeftClick As Boolean, ByVal RestorePosition As Boolean) 
        Dim inputEvents(0) As INPUT_TYPE 
        Dim pt As Point = Cursor.Position
        'IF DURING DEBUG I STOP HERE THE VALUES FOR ARE X=89, Y=340
        Cursor.Position = New Point(x, y) 
        If LeftClick Then 
            Debug.Print("Clicking export button now")
            inputEvents(0).xi.dx = 0
            inputEvents(0).xi.dy = 0
            inputEvents(0).xi.mouseData = 0
       inputEvents(0).xi.dwFlags = MOUSEEVENTF_ABSOLUTE + M_LD + M_LU
            inputEvents(0).xi.dwExtraInfo = 0
            inputEvents(0).dwType = INPUT_MOUSE 
            SendInput(1, inputEvents(0), Len(inputEvents(0))) 
        Else
            mouse_event(MOUSEEVENTF_RIGHTDOWN, 0, 0, 0, 0)
            mouse_event(MOUSEEVENTF_RIGHTUP, 0, 0, 0, 0)
        End If
        If RestorePosition Then
            Cursor.Position = pt
        End If
    End Sub 
HERE ARE THE RELEVANT STRUCTURE USED IN CLICKAT METHOD: 
Public Structure MOUSEINPUT
        Public dx As Integer
        Public dy As Integer
        Public mouseData As Integer
        Public dwFlags As Integer
        Public dwtime As Integer
        Public dwExtraInfo As Integer
    End Structure 
    Public Structure INPUT_TYPE
        Public dwType As Integer
        Public xi As MOUSEINPUT
    End Structure

                                              

1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:
41:
42:
43:
44:
45:
46:
47:
48:
49:
50:
51:

Sorry the code above is not formatted nicely.  Some additional information:

I noticed the SendInput return value only returns back 1.  

inputEvents(0).xi.dwFlags = MOUSEEVENTF_ABSOLUTE + M_LD + M_LU

Should it be returning 3 since there are 3 flags set as above in the MOUSE_INPUT structure

Also, note that although articles I have read say that it's rare for focus to be an issue I tried using both SetFocus,  SetForegroundWindow(ctl.Handle) & SetActiveWindow(ctl.Handle) and this didn't do anything.

Previously I had tried putting a sleep delay up to 500 ms between using M_LD AND M_LU (abbreviated) when using mouse_event

 Const M_LD = &H2
 Const M_LU = &H4

Here's an article with someone who had a similar issue that the Cursor needs to be above the button for the click to work.   I tried that also decreasing the offset for the y coordinate from 32 down to 17 or so but it didn't make a difference.

http://bytes.com/topic/c-sharp/answers/575589-simulate-mouse-click-without-focus


The article here suggests to use Send or PostMessage but the problem with that I can't get the button itself as a window.   I can only get a child window that contains the button.   If I'm using screen coordinates is there even a need for the child window of the window popup?

I am so stumped, this is really a tough one for me.

   

1: