Robot to navigate and control windows desktop

Posted on 2004-11-25
Last Modified: 2010-03-31

I'm trying to writing a 'robot' program that will interact with the windows desktop in a similar way to a user would.  

Creating mouse movements and key presses is trivial using the java.awt.Robot class, but I want the program to react to windows that appear on the fly.  Essentially i want the program to 'see' whats going on on the screen and then use this to make a descision about what to do.  For example if a message box pops up with 'yes', 'no' and 'cancel' buttons and some message text, i'd like the program to parse the message text, identify there are some buttons, make a desicion based on the message (and/or title bar text), navigate to the appropriate button and click it.

I really don't want to have to take screen grabs every second with the robot class and write some sort of graphics identification to recognise windows, buttons and do OCR on text, that would just be mad if I can grab the data directly.

I have choosen java to implement this in because the robot class seems nice, but i'm savvy with a handful of languages so if anyone thinks another language would do the job better i'm open to ideas.

So, in short, how do i capture text from windows title bars, text boxes, buttons and the like in java?

Thanks in advance for your thoughts and ideas.

Question by:ady_at_bl
    LVL 35

    Expert Comment

    What exactly will this application do? Will you be controlling the windows appearing on the fly up or not? Because if you will be controlling them then I cannot see why you would need to parse input and stuff, since you will already know in advance what the windows will be and you can take proper actions.

    If you are talking about windows that you do not have control then things get complicated. YOu will need a way your Robot class to interact with them, via the network probably. Maybe this link will be of help:

    Author Comment

    Hi girionis

    I'm wanting to control windows which I have not created, which are not part of my java application.

    The program in the link is essentially what i already have, after a quick look at the example  it appears a screen shot is sent back to a person and the person generates the clicks and key presses which are sent to the robot to recreate on the other machine.  I want to take human interaction out of the picture, I want my program decide what to click and where to go next based on what options are available to it.  The problem is getting these options to my program, i essentially have a 'blind' robot at the moment.  Your right, this could get complicated :'-(
    LVL 35

    Expert Comment

    > I'm wanting to control windows which I have not created, which are not
    > part of my java application.

    I guess these windows are also Java windows, otherwise you will have to use JNI.

    It looks complicated though. You can obtain the references to the other windows and then use the java.awt.Robot class to initiate the clicks. But you will have to think issues as to how will your application run alongside with the other Java windows. You will need to send the events to the interesting windows and notify them of button clicks and stuff. But you will need to hjave a reference to the other windows somehow and this is what troubles me. How will you get a reference to the instance of the windows running? You will need to communicate somehow if they are a completely different application, I'd say via some sort of server that will be listening for Robot events and taking the appropriate actions.

    Author Comment


    Accepted Solution

    PAQed with points refunded (125)

    Community Support Moderator

    Featured Post

    Courses: Start Training Online With Pros, Today

    Brush up on the basics or master the advanced techniques required to earn essential industry certifications, with Courses. Enroll in a course and start learning today. Training topics range from Android App Dev to the Xen Virtualization Platform.

    Join & Write a Comment

    Suggested Solutions

    Title # Comments Views Activity
    countClumps  challenge 10 70
    IT Company 5 57
    Safari, Firefox, IE, Java, Plugin, Java Security error 13 27
    countAbc challenge 9 35
    For customizing the look of your lightweight component and making it look lucid like it was made of glass. Or: how to make your component more Apple-ish ;) This tip assumes your component to be of rectangular shape and completely opaque. (COD…
    Java contains several comparison operators (e.g., <, <=, >, >=, ==, !=) that allow you to compare primitive values. However, these operators cannot be used to compare the contents of objects. Interface Comparable is used to allow objects of a cl…
    Viewers will learn about if statements in Java and their use The if statement: The condition required to create an if statement: Variations of if statements: An example using if statements:
    This theoretical tutorial explains exceptions, reasons for exceptions, different categories of exception and exception hierarchy.

    746 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    14 Experts available now in Live!

    Get 1:1 Help Now