Robot to navigate and control windows desktop
Posted on 2004-11-25
I'm trying to writing a 'robot' program that will interact with the windows desktop in a similar way to a user would.
Creating mouse movements and key presses is trivial using the java.awt.Robot class, but I want the program to react to windows that appear on the fly. Essentially i want the program to 'see' whats going on on the screen and then use this to make a descision about what to do. For example if a message box pops up with 'yes', 'no' and 'cancel' buttons and some message text, i'd like the program to parse the message text, identify there are some buttons, make a desicion based on the message (and/or title bar text), navigate to the appropriate button and click it.
I really don't want to have to take screen grabs every second with the robot class and write some sort of graphics identification to recognise windows, buttons and do OCR on text, that would just be mad if I can grab the data directly.
I have choosen java to implement this in because the robot class seems nice, but i'm savvy with a handful of languages so if anyone thinks another language would do the job better i'm open to ideas.
So, in short, how do i capture text from windows title bars, text boxes, buttons and the like in java?
Thanks in advance for your thoughts and ideas.