EE QA: Developing and Running Automated Tests for Our Website

Published on
6,830 Points
4 Endorsements
Last Modified:
This article is the last of three articles that explain why and how the Experts Exchange QA Team does test automation for our web site. This article covers our test design approach and then goes through a simple test case example, how to add it to a test suite, then running and debugging it.

This article doesn’t aspire to be The Way To Do Automation and it doesn’t go into deep details on processes or tools, but if you read it you should come away with a reasonably clear picture of one way to efficiently and effectively develop and execute automation tests for a website. This process helps us achieve our test automation goals.

Test Design Approach
Experts Exchange website consists of a large set of web pages. There are groups of pages with the same format. For example, all the pages that display a question have the same layout. We refer to this as the view question layout. Likewise, we refer to the view article layout and the view video layout for the pages where articles and videos are displayed.

To test our site EE QA uses something called the Page Object model. Basically, this is an encapsulation of a web page into a class. For example, we have ViewQuestion, ViewArticle and ViewVideo classes. Our test cases for the view question layout call ViewQuestion methods, rather than have each test try to directly interact with items (buttons, text boxes, etc) on the page. The page’s attributes and actions are encapsulated in ViewQuestion page object class.

Using this model has some nice benefits. If parts of a page get redesigned or if new components (buttons, text boxes, etc) are added to a page, only the page’s class needs to be updated. All the test cases that use the page class are insulated from the change. It also means over time we can add more methods to the page class. When we want to expand our testing by taking advantage of these new page class methods we can do this without impact on existing class methods or the test cases that use those existing methods. This is all just good, basic object-oriented design, applied to test case automation.

Test Case Example
Three of the critical features of the EE website are the ability of our members to ask questions, comment on questions, and selection comments as solutions to questions. We have an automated Critical Test Suite that verifies these features, along with other critical features. The suite is run continuously. This ensures no functional regressions in the critical areas.

We keep most of our test cases simple and discrete. Even our workflow tests are implemented as a sequence of test cases, rather than one big test. We use TestNG to control the execution sequence for workflow suites.

The three test cases, called runAsk, runAnswer, and  runAccept, are in a single class file called QuestionTest. Together they use five different page objects; Login, Header, AskQuestion, ViewQuestion and AcceptQuestion. The tests are fairly simple. Let's take a look at the test code.

public class QuestionTest extends EeBaseTest
   final String COMMENT = "This is an expert answer.";
   String m_questionUrl = "";  // will hold url of new question

    * Ask a question. Verify title text, body text, topic names
    * @throws Exception
   @Test  (groups = {"Critical"}, priority=0)
   public void runAsk() throws Exception
      String currentDateTime = TestUtilities.getTimeMmDdHhMmSsSss();
      final String TITLE = "Critical WebDriver Question Title " + currentDateTime; 
      final String BODY = "Critical WebDriver Body Text " + currentDateTime; 
      final String TOPIC = "Quality Assurance"; 
      LoginPage lp = new LoginPage(m_driver, m_eeUrl);
      lp.getAndLogin(Config.asker, Config.PASSWORD);
      // ask the question
      AskQuestionPage aqp = new AskQuestionPage(m_driver, m_eeUrl);   // Get AskQuestion page object
      ViewQuestionPage vqp = aqp.submitQuestion(TITLE, BODY, TOPIC);  // Submit Ask form and get back ViewQuestion object
      m_questionUrl = m_driver.getCurrentUrl();    // set instance var so other tests can get to new question url

      //verify the data on the view question page
      assertEquals(TITLE, vqp.getTextTitle());
      assertEquals(BODY, vqp.getTextBodyWhenZeroCodeSnippets());
      ArrayList<String> topicNames = vqp.getTextTopics();
      assertEquals(TOPIC, topicNames.get(0));
      assertEquals(1, topicNames.size());

      HeaderPage hp = new HeaderPage(m_driver, m_eeUrl);

    * Answer the question from the previous test. Verify comment text.
    * @throws Exception
   @Test  (groups = {"Critical"}, priority=1)
   public void runAnswer() throws Exception
      // log in as different user and go to question's page
      LoginPage lp = new LoginPage(m_driver, m_eeUrl);
      lp.getAndLogin(Config.expert, Config.PASSWORD); 

      // comment on question
      ViewQuestionPage vqp = new ViewQuestionPage(m_driver, m_eeUrl);

      // verify comment was added to question
      ArrayList <String> answers = vqp.getTextAnswers();
      assertEquals(1, answers.size());
      assertEquals(COMMENT, answers.get(0));

      HeaderPage hp = new HeaderPage(m_driver, m_eeUrl);

    * Accept as solution the answer from the previous test. 
    * Verify the answer in the previous test is the accepted solution.
    * @throws Exception
   @Test  (groups = {"Critical"}, priority=2)
   public void runAccept() throws Exception
      // log in as question owner
      LoginPage lp = new LoginPage(m_driver, m_eeUrl);
      lp.getAndLogin(Config.asker, Config.PASSWORD);

      // accept comment as answer
      ViewQuestionPage vqp = new ViewQuestionPage(m_driver, m_eeUrl);
      AcceptAnswerPage aap = vqp.clickAcceptAsAnswerButton();
      vqp = aap.clickSubmitButton();
      assertEquals(COMMENT, vqp.getTextAcceptedSolution());

Open in new window

The test cases are easy to read, in part because of the page object model, but also because we use assert statements and our class methods and variable names are descriptive. You may notice a few odd things in the code. The three @Test lines are TestNG tags. Also, a few variables are referenced, but not declared nor set in this code snippet. Both m_driver and m_eeUrl are declared and set in EeBaseTest class. Later on there are some more details about all of these. For now, let's take a look at some of the page object code. 


import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.support.FindBy;
import org.openqa.selenium.support.PageFactory;

 * Login page
public class LoginPage extends Page
   WebElement m_UsernameTextInput;
   WebElement m_PasswordTextInput;
   public LoginPage(WebDriver driver, String baseUrl)
      super(driver, baseUrl, "/login.jsp");
      PageFactory.initElements(driver, this);
   public LoginPage sendKeysUsername(String username)
      return this;
   public LoginPage sendKeysPassword(String password)
      return this;
   public ViewHomePage submit()
      return new ViewHomePage(m_driver, m_baseUrl);
   public void login(String username, String password)
   public void getAndLogin(String username)
      getAndLogin(username, "test");
   public void getAndLogin(String username, String password)
      login(username, password);


Open in new window

For this code to make sense we need to dive a bit deeper into the page object model and the PageFactory class. If you are not familiar with the PageFactory class then the first four lines after the class declaration may look odd. Basically, these work with the PageFactory to find elements on the page and assign them to variables. For example, the first two lines will look for an element on the page with an id of login-id2-loginForm-loginName and sets m_UsernameTestInput to that element. This is the login field. Class methods, such as the sendKeysUsername(), can use the variable. The PageFactory.initElements() sets this up.

We use Selenium's IDE to help us with the FindBy parameter. We also put all the@FindBy statements right under the class declaration. If a page changes or if we want to test some new element on a page, we use the IDE to get the best locator of the page element. We update the existing @FindBy() or add a new one, then write the methods that will access that element. All of this is done at the page object level and is hidden from the test cases. If, for example, the id of the login field changed from login-id2-loginForm-loginName to something else, all we need to do is update the @FindBy() to the new id. Of course if the id changed then our tests that try to log in would fail and we would investigate it, but the actual fix for such a change is very well encapsulated. We update the @FindBy() and our tests are back to passing.

The QuestionTest class extends our base test class, EeBaseTest. EeBaseTest class uses a few TestNG tags and sets m_eeUrl and m_driver variables. Let's take a  look at the code.

public class EeBaseTest
   protected String m_eeUrl;
   protected String m_driverType;
   protected WebDriver m_driver;

    * Set parameters for the test
    * @param eeUrl
    * @param driverType use htmlunit or phantomjs; otherwise you get FirefoxDriver
   @Parameters ({"eeUrl", "driverType"})  // Can be set in TestNG XML file
   public void setSite(@Optional("http://not_really_url.com") String eeUrl
                     , @Optional("firefox") String driverType) 
      m_eeUrl = getValueOrProperty(eeUrl, "eeUrl");
      m_driverType = getValueOrProperty(driverType, "driverType");

      newDriver();    // create a new driver of m_driverType, assign it to m_driver

    * If a property was passed on the command line, return it.
    * Otherwise return the XML value
   public String getValueOrProperty(String xmlValue, String propertyName)
      String propertyValue = System.getProperty(propertyName);
      if (null == propertyValue)
         return xmlValue;
      return (null == propertyValue) ? xmlValue : propertyValue;

Open in new window

The @BeforeClass(alwaysRun=true) line causes the setSite() method to be called before the first test method in the current class is invoked. The combination of this tag and the setSite() method enables us to set m_eeUrl  and m_driver before any of our test cases run. As you saw in the QuestionTest code snippet, our tests make copious use of these variables.

The @Parameters ({"eeUrl", "driverType"}) line is a way to pass variables from a TestNG xml file into the run time environment. There are more details in the Run Time section, below.

The @Optional("http://not_really_url.com") String eeUrl and @Optional("firefox") String driverType  entries are a handy way to have defaults for a parameter. These get used if nothing is specified in the  TestNG xml file.

Test Suite Example
We define a test suite as a group of test case files, annotated test cases in those files, a TestNG xml file and a TestNG group name. As you saw in the test case code above, all three test cases were annotated with  @Test  (groups = {"Critical"}, ...).  To add other tests to the critical test suite we give them the groups = {"Critical"}  annotation. In the Run Time section I'll go into more detail on the TestNG xml file and group name.

A test case can be in more than one test suite. If we had a critical suite and a question page suite we could put the tests in both suites by using groups = {"Critical", "Question"}.

Run Time
Test suites can be launched from Eclipse or the command line. When we are developing or debugging our test code, we launch the target suite from within Eclipse, thanks to the TestNG plug-in. For our continuous integration testing we launch the suite from a command line. More specifically, we have a build environment tool that runs the critical test suite as part of a build-deploy-test sequence.

The TestNG xml file defines parts of the run time environment.  We use it to pass parameters into test classes, to specify the suites test name and to list which test class files to consider. Let's take a look at our CriticalSuite.xml


<suite name="Critical">
    <test verbose="0" name="Critical" annotations="JDK">
        <parameter name="eeUrl" value="http://some_bogus_url.com/"></parameter>
        <parameter name="driverType" value="phantomjs"></parameter>

        <groups><run><include name="Critical"></include></run></groups>
            <class name="com.ee.tests.views.QuestionTest"></class>
            <class name="com.ee.tests.views.VideosTest"></class>
            <class name="com.ee.tests.views.ArticlesTest"></class>

Open in new window

Back in the EeBaseTest code snippet you saw this line:    
          @Parameters ({"eeUrl", "driverType"})  // Can be set in TestNG XML file
Now in the xml file you can see how these parameters are defined. TestNG passes these xml parameters into the code as part of the run time environment.

When the EeBaseTest.setSite() method is invoked (before the first test method in the current class is invoked due to the method having the @BeforeClass(alwaysRun=true) annotation) it will look to see if eeUrl and driverType are passed in as parameters (coming from the TestNG xml file, CriticalSuite.xml in this case). If the parameters are passed in, they are used to set the m_eeUrl and m_driver variables, but if the parameters are not set then the @Optional values will be used. This gives us a lot of control over the run time environment. We have a default test environment, as specified by the @Optional values, but we can  override that in the xml file. This makes it easy to run the tests on different test systems.

The <groups>.... line specifies what tags to look for when identifying the test cases to run. The <classes> lines define which test source code files to look through. With CriticalSuite.xml, the three files; QuestionTest, VideoTest and ArticleTest  will be inspected. All test cases in those three files that have the Critical tag will be executed. The combination of the <groups>... and <classes>... define which source files to investigate and what group tag to use to select the test cases for execution. 

The xml file gives us control over run time parameters, which files to inspect and which test cases to execute from those files.

This xml file is part of the automation project in Eclipse. To execute the suite we just right mouse click on the suite, go to Run As and select the TestNG Suite option.
EEQA_3RunTime.pngOur command line execution is straightforward, too. Basically it's
java -cp <selenium jar>;<our classes jar>;<more 3rd party jars> org.testng.TestNG <our TestNG xml file>

Open in new window

Debug Time
Eclipse does a fine job of pointing out static errors in our java code. For run time errors, we  find the Eclipse Java debugger very easy to use. Mostly we use it for setting breakpoints, inspecting variables, and stepping through pieces of code.

Here's a quick example of using breakpoints to diagnose a failure. To start with, I'll point out the problem. Line 82 has quotes around COMMENT. It should not have the quotes. COMMENT is a static variable. This causes the string "COMMENT" to be passed into the method, rather than the value of variable COMMENT.

When I run the test suite I see the failure in runAnswer test case. When I click on the test case I see the exception stack. Clicking on the exception highlights the line of code where the exception occurred.

To debug this I will set a breakpoint on line 86. I just double click on the line number and the breakpoint is added.
EEQA_3_Break.pngThen I run the critical suite, but instead of selection 'Run As' I select 'Debug As'. This executes the test suite, but will cause execution to halt when a breakpoint is encountered. When the breakpoint is hit, I go to Eclipse's debug perspective. I'm able to see which breakpoint the code stopped at. In this case it's line 86 so I'm where I want to be for debugging.

Now I can inspect the available variables. In the Variables window I can expand 'this' to see more details. I see that variable COMMENT is set to the expected string. Then I take a look answers object. I'm expecting it to be set to the same string as COMMENT. I see it is set to COMMENT, rather than the "This is an expert answer" string. How the heck did that happen ? Looking at where the comment should be set I see the problem. I click the continue button in eclipse so the run finishes. Then, I fix line 83. I remove the quotes around COMMENT and re-run the suite.
The tools and processes described in the three articles of this series give EE QA the ability to efficiently and effectively design, develop and deploy automated test cases. Much of the focus has been on these tools, and they really do enable us to achieve our automation goals. The other big win for us is using the page object model. Previously we had a more procedural approach to test case development, but we are seeing improvements in our coding efficiency, execution reliability (fewer failures due to bad test code), and lower test code maintenance costs.

Whatever tool set you use for your test case automation, consider the page object model as a way to develop your test code.

First Article: EE QA: How Selenium, Java, Eclipse, and TestNG help us achieve our test automation goals
Second Article: EE QA: Install and Configure Selenium, Java, Eclipse, and TestNG

Ask questions about what you read
If you have a question about something within an article, you can receive help directly from the article author. Experts Exchange article authors are available to answer questions and further the discussion.
Get 7 days free