?
Solved

Testing on an emulated OS versus the real thing

Posted on 2011-10-17
13
Medium Priority
?
435 Views
Last Modified: 2012-05-12
Setting up a web browser testing lab...
We are concerned solely with mimicking end-user machines.  End-Users are defined as people outside our firewall that interact with our website via http.

Our long term goal is to build a Library of client OS images available in our VM Farm that our testers can use to create whichever OS/Browser combination they need for their testing, however, it will take awhile to get that up and running.

In the meantime, we are looking at some 3rd party providers as a stop-gap.

From what we have found so far, the vendors that emulate the Operating Systems (e.g., they provide remote access to a machine that emulates Windows XP using Windows Server 2003, or they emulate Vista and 7 using Server 2008) are *significantly* less expensive than those that provide actual Operating Systems.

Is anyone aware of the drawbacks of using emulated Operating Systems instead of "the real thing"?

Thanks!
0
Comment
Question by:nap0leon
  • 7
  • 6
13 Comments
 
LVL 124
ID: 36981703
Our developers ALWAYS test on actual OS, rather than emulated. If you want a TRUE Test for Quality Purposes you should do the same, if you shortcut, you are not completing correct End to End testing, and therefore it is not a true test of the clients machine.

Also you are trusting your testing environment to a third party vendor? How good is their emulation.

e.g.

Internet Explorer 6.0 SP1 on XP SP1, SP2, SP3 (32bit and 64 bit)
Intel Explorer 7.0 on Windows Vista 64 bit SP1, SP2 etc


It does not take much today, to setup this testing envionment using free VMware Player 4.0 on a Developers PC.

0
 
LVL 18

Author Comment

by:nap0leon
ID: 36981746
Since this relationship will be thrown away within 6 months while the VM Farm is built up with the necessary client images, and since we currently test nothing on the combinations that it will be used for... what are the actual shortcomings of using an emulated OS versus the real thing (what types of bugs could we miss)?

Keep in mind, that we are only looking at cross-browser testing for end-users that hit our website.  Since our website does not directly interact with the user's OS (unlike, say, SharePoint, which has known issues kicking off various Office applications on certain OSs), is there any particular thing that makes the emulated OS different than the real thing - from a web browser perspective?
0
 
LVL 124
ID: 36982335
If it's browser testing, why not just install multiple browsers on PCs?
0
NFR key for Veeam Agent for Linux

Veeam is happy to provide a free NFR license for one year.  It allows for the non‑production use and valid for five workstations and two servers. Veeam Agent for Linux is a simple backup tool for your Linux installations, both on‑premises and in the public cloud.

 
LVL 18

Author Comment

by:nap0leon
ID: 36982667
Security team won't allow user installed browsers, the only OS on our desktops is XP, physical lab space for multiple Machines for random OS with various browsers is limited, etc.
0
 
LVL 124
ID: 36982872
Okay, well you've got Company Policies in place, which your Developers need to resolve, if they need to do a job!
0
 
LVL 18

Author Comment

by:nap0leon
ID: 36983102
That's why we are building out a VMFarm for out-of-the-box OSs that can be used on a (also being built) Isolated Test Network.
But, those things take time. In the meantime, I'm still looking for a concrete reason to not use the emulated OS providers as a shortterm stop-gap. As a QA guy, I want a true end-user OS to work with. As a BA, I need to be able to justify the extra expense.
0
 
LVL 124

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE^2) earned 2000 total points
ID: 36983135
We can only advise, that for our testing against our policies and standards, our Developers run QA tests on like for like Home And Business Operating Systems to ensure compliance testing is as close as possible.

However, its impossible to ensure that you are going to have the exact same configuration as the end user, this is where you must self certify the tests yourselves, and agree this is the nearest testing you can achieve.

I really do not know what an emulated OS is? Is this just the Agent string changed to make it appear as Windows XP, when the browser is installed on Windows 2008.

I think it really depends on if you are just testing browser based functions, code should operate within browser context, so risk is low?
0
 
LVL 18

Author Comment

by:nap0leon
ID: 36983307
I too presume the risk is low, but I have no clue what an "emulated OS" really is.

Just trying to get a handle from anyone who actually understands Operating Systems better than I do who might educate me on how Windows XP emulated on Server 2003 is different than a 'real" Windows XP machine (or similar).

Talking with the vendors that do this sort of hosting, of course they say it as "the same as running the real OS" (or, as one of them said, "we have seen no differences" - which to me means they have not run the right tests).  They are sales people and they are trained to answer questions that way.  But, I've been around IT long enough to know that nothing is "the same as" unless it actually is "the real thing" (e.g., IE6 thin client is not the same as IE6 installed,  "IE Tab" in FF is not the same as viewing the page in IE, etc.)
0
 
LVL 124
ID: 36983318
Do you have a website, We would like to know what an emulated OS is?
0
 
LVL 18

Author Comment

by:nap0leon
ID: 36983415
A couple vendors that offer these "emulated Operating Systems" that I've spoken to are:
rackspace.com
browserstack.com

There are several others... I don't recall all of their names.
0
 
LVL 18

Author Comment

by:nap0leon
ID: 36985540
From my dialogue with BrowserStack.com... my conversation with rackspace.com was basically the same.

Upon initial contact, I asked them several questions.  Quotes below are from a series of emails back and forth (I ask, they answer, I ask, they answer...)

Me: (Please provide) a complete list of OSs and browsers that you currently support. Are there any mobile devices included? Are these true OSs, or are they , for example, emulated under Windows Server 2008?
Browserstack: OS : Windows, Browsers : FF 3, 3.6, 4, 5, 6, 7; IE 6, 7, 8, 9; Chrome 12, 13, 14, 15; Safari: 4, 5, 5.1; Opera: 10, 11.5.  We use Windows server editions. We find no difference in using specific windows OS

Me: Which versions of Windows do you support?
Browserstack: Windows 2003 and 2008 server edition.

Me: My question is more geared towards which Windows OSs your system emulates.  E.g., Vista, XP, 7, 2k, etc.
Browserstack: We provide Windows Server OS which are 2003 and 2008. Windows XP is created over 2003 server OS and Windows Vista and 7 is created over 2008 server OS.
0
 
LVL 124
ID: 36985610
Okay, to be it sounds like they are Streaming Browsers to Servers, or have published applications in Silos' of different browsers.

We have a similar arrangement, we have a Win2k3 Terminal Server with IE6 which publishes IE6, and then we have another server Win2k3 with IE7 etc

but the flavour today is to stream applications to the same server at login.

But because we also have a Lab Manager environment, and vSphere, Engineers and Developers can deploy a specific Build, e.g. pick a browser and OS to test.
0
 
LVL 18

Author Comment

by:nap0leon
ID: 37032803
With no one else offering other input in the topic, I'll presume that the best answer is
"I think it really depends on if you are just testing browser based functions, code should operate within browser context, so risk is low?"

Not the confident sort of answer I was hoping for, but is along the lines of what I expected.

With our risk based testing, we will not be regularly testing every single Browser/OS combination... especially when you break down OSs into the variety of SPs and other configurable aspects that users can do to customize their desktops... no hosted solution (emulated or real OS) would give us that degree of specificity.  This will simply become one of the documented "Risk Assumptions" in our planning document.
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Giving access to ESXi shell console is always an issue for IT departments to other Teams, or Projects. We need to find a way so that teams can use ESXTOP for their POCs, or tests without giving them the access to ESXi host shell console with a root …
The article covers five tools all IT professionals should know about, as they up productivity by a great deal!
This Micro Tutorial walks you through using a remote console to access a server and install ESXi 5.1. This example is showing remote access and installation using a Dell server. The hypervisor is the very first component of your virtual infrastructu…
Hi friends,  in this video  I'll show you how new windows 10 user can learn the using of windows 10. Thank you.

850 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question