Robot to navigate and control windows desktop


I'm trying to writing a 'robot' program that will interact with the windows desktop in a similar way to a user would.  

Creating mouse movements and key presses is trivial using the java.awt.Robot class, but I want the program to react to windows that appear on the fly.  Essentially i want the program to 'see' whats going on on the screen and then use this to make a descision about what to do.  For example if a message box pops up with 'yes', 'no' and 'cancel' buttons and some message text, i'd like the program to parse the message text, identify there are some buttons, make a desicion based on the message (and/or title bar text), navigate to the appropriate button and click it.

I really don't want to have to take screen grabs every second with the robot class and write some sort of graphics identification to recognise windows, buttons and do OCR on text, that would just be mad if I can grab the data directly.

I have choosen java to implement this in because the robot class seems nice, but i'm savvy with a handful of languages so if anyone thinks another language would do the job better i'm open to ideas.

So, in short, how do i capture text from windows title bars, text boxes, buttons and the like in java?

Thanks in advance for your thoughts and ideas.

Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

What exactly will this application do? Will you be controlling the windows appearing on the fly up or not? Because if you will be controlling them then I cannot see why you would need to parse input and stuff, since you will already know in advance what the windows will be and you can take proper actions.

If you are talking about windows that you do not have control then things get complicated. YOu will need a way your Robot class to interact with them, via the network probably. Maybe this link will be of help: 
ady_at_blAuthor Commented:
Hi girionis

I'm wanting to control windows which I have not created, which are not part of my java application.

The program in the link is essentially what i already have, after a quick look at the example  it appears a screen shot is sent back to a person and the person generates the clicks and key presses which are sent to the robot to recreate on the other machine.  I want to take human interaction out of the picture, I want my program decide what to click and where to go next based on what options are available to it.  The problem is getting these options to my program, i essentially have a 'blind' robot at the moment.  Your right, this could get complicated :'-(
> I'm wanting to control windows which I have not created, which are not
> part of my java application.

I guess these windows are also Java windows, otherwise you will have to use JNI.

It looks complicated though. You can obtain the references to the other windows and then use the java.awt.Robot class to initiate the clicks. But you will have to think issues as to how will your application run alongside with the other Java windows. You will need to send the events to the interesting windows and notify them of button clicks and stuff. But you will need to hjave a reference to the other windows somehow and this is what troubles me. How will you get a reference to the instance of the windows running? You will need to communicate somehow if they are a completely different application, I'd say via some sort of server that will be listening for Robot events and taking the appropriate actions.
ady_at_blAuthor Commented:
PAQed with points refunded (125)

Community Support Moderator

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.