Testing on an emulated OS versus the real thing

Setting up a web browser testing lab...
We are concerned solely with mimicking end-user machines.  End-Users are defined as people outside our firewall that interact with our website via http.

Our long term goal is to build a Library of client OS images available in our VM Farm that our testers can use to create whichever OS/Browser combination they need for their testing, however, it will take awhile to get that up and running.

In the meantime, we are looking at some 3rd party providers as a stop-gap.

From what we have found so far, the vendors that emulate the Operating Systems (e.g., they provide remote access to a machine that emulates Windows XP using Windows Server 2003, or they emulate Vista and 7 using Server 2008) are *significantly* less expensive than those that provide actual Operating Systems.

Is anyone aware of the drawbacks of using emulated Operating Systems instead of "the real thing"?

Thanks!
LVL 18
nap0leonAsked:
Who is Participating?
 
Andrew Hancock (VMware vExpert / EE MVE^2)Connect With a Mentor VMware and Virtualization ConsultantCommented:
We can only advise, that for our testing against our policies and standards, our Developers run QA tests on like for like Home And Business Operating Systems to ensure compliance testing is as close as possible.

However, its impossible to ensure that you are going to have the exact same configuration as the end user, this is where you must self certify the tests yourselves, and agree this is the nearest testing you can achieve.

I really do not know what an emulated OS is? Is this just the Agent string changed to make it appear as Windows XP, when the browser is installed on Windows 2008.

I think it really depends on if you are just testing browser based functions, code should operate within browser context, so risk is low?
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Our developers ALWAYS test on actual OS, rather than emulated. If you want a TRUE Test for Quality Purposes you should do the same, if you shortcut, you are not completing correct End to End testing, and therefore it is not a true test of the clients machine.

Also you are trusting your testing environment to a third party vendor? How good is their emulation.

e.g.

Internet Explorer 6.0 SP1 on XP SP1, SP2, SP3 (32bit and 64 bit)
Intel Explorer 7.0 on Windows Vista 64 bit SP1, SP2 etc


It does not take much today, to setup this testing envionment using free VMware Player 4.0 on a Developers PC.

0
 
nap0leonAuthor Commented:
Since this relationship will be thrown away within 6 months while the VM Farm is built up with the necessary client images, and since we currently test nothing on the combinations that it will be used for... what are the actual shortcomings of using an emulated OS versus the real thing (what types of bugs could we miss)?

Keep in mind, that we are only looking at cross-browser testing for end-users that hit our website.  Since our website does not directly interact with the user's OS (unlike, say, SharePoint, which has known issues kicking off various Office applications on certain OSs), is there any particular thing that makes the emulated OS different than the real thing - from a web browser perspective?
0
Cloud Class® Course: Microsoft Office 2010

This course will introduce you to the interfaces and features of Microsoft Office 2010 Word, Excel, PowerPoint, Outlook, and Access. You will learn about the features that are shared between all products in the Office suite, as well as the new features that are product specific.

 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
If it's browser testing, why not just install multiple browsers on PCs?
0
 
nap0leonAuthor Commented:
Security team won't allow user installed browsers, the only OS on our desktops is XP, physical lab space for multiple Machines for random OS with various browsers is limited, etc.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Okay, well you've got Company Policies in place, which your Developers need to resolve, if they need to do a job!
0
 
nap0leonAuthor Commented:
That's why we are building out a VMFarm for out-of-the-box OSs that can be used on a (also being built) Isolated Test Network.
But, those things take time. In the meantime, I'm still looking for a concrete reason to not use the emulated OS providers as a shortterm stop-gap. As a QA guy, I want a true end-user OS to work with. As a BA, I need to be able to justify the extra expense.
0
 
nap0leonAuthor Commented:
I too presume the risk is low, but I have no clue what an "emulated OS" really is.

Just trying to get a handle from anyone who actually understands Operating Systems better than I do who might educate me on how Windows XP emulated on Server 2003 is different than a 'real" Windows XP machine (or similar).

Talking with the vendors that do this sort of hosting, of course they say it as "the same as running the real OS" (or, as one of them said, "we have seen no differences" - which to me means they have not run the right tests).  They are sales people and they are trained to answer questions that way.  But, I've been around IT long enough to know that nothing is "the same as" unless it actually is "the real thing" (e.g., IE6 thin client is not the same as IE6 installed,  "IE Tab" in FF is not the same as viewing the page in IE, etc.)
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Do you have a website, We would like to know what an emulated OS is?
0
 
nap0leonAuthor Commented:
A couple vendors that offer these "emulated Operating Systems" that I've spoken to are:
rackspace.com
browserstack.com

There are several others... I don't recall all of their names.
0
 
nap0leonAuthor Commented:
From my dialogue with BrowserStack.com... my conversation with rackspace.com was basically the same.

Upon initial contact, I asked them several questions.  Quotes below are from a series of emails back and forth (I ask, they answer, I ask, they answer...)

Me: (Please provide) a complete list of OSs and browsers that you currently support. Are there any mobile devices included? Are these true OSs, or are they , for example, emulated under Windows Server 2008?
Browserstack: OS : Windows, Browsers : FF 3, 3.6, 4, 5, 6, 7; IE 6, 7, 8, 9; Chrome 12, 13, 14, 15; Safari: 4, 5, 5.1; Opera: 10, 11.5.  We use Windows server editions. We find no difference in using specific windows OS

Me: Which versions of Windows do you support?
Browserstack: Windows 2003 and 2008 server edition.

Me: My question is more geared towards which Windows OSs your system emulates.  E.g., Vista, XP, 7, 2k, etc.
Browserstack: We provide Windows Server OS which are 2003 and 2008. Windows XP is created over 2003 server OS and Windows Vista and 7 is created over 2008 server OS.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Okay, to be it sounds like they are Streaming Browsers to Servers, or have published applications in Silos' of different browsers.

We have a similar arrangement, we have a Win2k3 Terminal Server with IE6 which publishes IE6, and then we have another server Win2k3 with IE7 etc

but the flavour today is to stream applications to the same server at login.

But because we also have a Lab Manager environment, and vSphere, Engineers and Developers can deploy a specific Build, e.g. pick a browser and OS to test.
0
 
nap0leonAuthor Commented:
With no one else offering other input in the topic, I'll presume that the best answer is
"I think it really depends on if you are just testing browser based functions, code should operate within browser context, so risk is low?"

Not the confident sort of answer I was hoping for, but is along the lines of what I expected.

With our risk based testing, we will not be regularly testing every single Browser/OS combination... especially when you break down OSs into the variety of SPs and other configurable aspects that users can do to customize their desktops... no hosted solution (emulated or real OS) would give us that degree of specificity.  This will simply become one of the documented "Risk Assumptions" in our planning document.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.