• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 159
  • Last Modified:

Robot to navigate and control windows desktop

Hi

I'm trying to writing a 'robot' program that will interact with the windows desktop in a similar way to a user would.  

Creating mouse movements and key presses is trivial using the java.awt.Robot class, but I want the program to react to windows that appear on the fly.  Essentially i want the program to 'see' whats going on on the screen and then use this to make a descision about what to do.  For example if a message box pops up with 'yes', 'no' and 'cancel' buttons and some message text, i'd like the program to parse the message text, identify there are some buttons, make a desicion based on the message (and/or title bar text), navigate to the appropriate button and click it.

I really don't want to have to take screen grabs every second with the robot class and write some sort of graphics identification to recognise windows, buttons and do OCR on text, that would just be mad if I can grab the data directly.

I have choosen java to implement this in because the robot class seems nice, but i'm savvy with a handful of languages so if anyone thinks another language would do the job better i'm open to ideas.

So, in short, how do i capture text from windows title bars, text boxes, buttons and the like in java?

Thanks in advance for your thoughts and ideas.

Ady
0
ady_at_bl
Asked:
ady_at_bl
  • 2
  • 2
1 Solution
 
girionisCommented:
What exactly will this application do? Will you be controlling the windows appearing on the fly up or not? Because if you will be controlling them then I cannot see why you would need to parse input and stuff, since you will already know in advance what the windows will be and you can take proper actions.

If you are talking about windows that you do not have control then things get complicated. YOu will need a way your Robot class to interact with them, via the network probably. Maybe this link will be of help: http://www.javaspecialists.co.za/archive/Issue091.html 
0
 
ady_at_blAuthor Commented:
Hi girionis

I'm wanting to control windows which I have not created, which are not part of my java application.

The program in the link is essentially what i already have, after a quick look at the example  it appears a screen shot is sent back to a person and the person generates the clicks and key presses which are sent to the robot to recreate on the other machine.  I want to take human interaction out of the picture, I want my program decide what to click and where to go next based on what options are available to it.  The problem is getting these options to my program, i essentially have a 'blind' robot at the moment.  Your right, this could get complicated :'-(
0
 
girionisCommented:
> I'm wanting to control windows which I have not created, which are not
> part of my java application.

I guess these windows are also Java windows, otherwise you will have to use JNI.

It looks complicated though. You can obtain the references to the other windows and then use the java.awt.Robot class to initiate the clicks. But you will have to think issues as to how will your application run alongside with the other Java windows. You will need to send the events to the interesting windows and notify them of button clicks and stuff. But you will need to hjave a reference to the other windows somehow and this is what troubles me. How will you get a reference to the instance of the windows running? You will need to communicate somehow if they are a completely different application, I'd say via some sort of server that will be listening for Robot events and taking the appropriate actions.
0
 
ady_at_blAuthor Commented:
0
 
moduloCommented:
PAQed with points refunded (125)

modulo
Community Support Moderator
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

  • 2
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now