Compatibility Testing in a Nutshell

Few weeks back I went to Shridi (pretty near to Pune, India) by train. Supposed to be two days (to & fro) trip which extended to three most awful days. Train service in India needs to be improved, a lot. “Quality & customer service” are verses which must be implanted in Indian Railway service motto. Do not want brief about my bad experience, since it is of no use. Neither people nor government is ready for transformations. L

Now to the post, let’s start by defining compatibility testing – “A Non-functional testing which often boils down to functional testing, targeted to confirm potential configurations; where software under test functions harmoniously”. To enhance,

• Compatibility Testing ~ Configuration Testing
• Certification Testing “part of” Compatibility Testing (not mandatory)

Hope I have defined it acceptably (at-least better than testing text books!). Not convinced with above definition? Click here to read Wikipedia’s version

Exploring Envisioned Ecosystem

So foremost step is to identify environment or platforms in which AUT (Application under Test) has to be operated. Then come up with parameter-list (a list of elements/parameters) that constitutes typical customer environment. And these governing elements, hinge on type (web, mobile, desktop etc...) of AUT being developed.

Few hints to derive parameter- list

• Ideally possible environmental parameter has to be cited in requirements documents, which won’t happen in many organizations!
• Customers – if not widely distributed, then get their inputs as well. For products with huge customer base usually this task is leveraged to marketing team, sometimes outsourced.Not that easy, so better be aware.
• Software release notes – Software used for developing AUT (For Instance: Java, .Net for web based applications) can co-exist with like-minded applications, to figure out offbeat ones refer corresponding release notes. Also do remember these keywords while constructing parameter-list: “Version”, “Patches”.
• If AUT is a scratch new product then probe for products with similar characteristics. I would highly recommend competitor products for such analysis.
• Finally, apply your(tester’s) heuristics

Focus On

It may not be possible to validate AUT in all configurations you have identified. Prioritize elements on hand, by following any technique (like House of Quality) which suits you. By this time, you should have essential parameters and its corresponding elements. Now figure out the combinations or basic configuration matrix that will cover up all items in parameter list. (Hint: Try orthogonal array testing/pair-wise testing while coming up with test domains). Share your configuration matrix with stakeholders and proceed with execution.

The Next Step

There are many tools in market to aid compatibility testing. To list few

•  Browser – Free online tools to check layouts & rendering
•  PC – Virtual machines
• Emulators & Simulators for mobiles, hardware, carriers/networks

Ensure Beta users are part of compatibility testing. Backward & Forward compatibility issues are likely to be unearthed mostly by beta users. Report your observations and modify AUT as required. Remember AUT may not excel in all environments at given time, so be ready for another version :)

Points to summon up

• Compatibility testing is not limited to soft wares, applicable even for human blood
• Certification/Standard compliance – get experts help to identify which ones are applicable for AUT (waive this pointer if you are an expert!!)
• Opt for crowd testing. If your configuration matrix then is huge consider outsourcing.
• One round of testing has to be done on real environment without using any tools/simulators

Have any feedback? Please comment it.