Remember, Those old school's days ?

These old days' automation techquies should be replaced with #ML based automation, 

Isn't It ? :)

If You have such automations that causing lots of issues then let's talk on how You can come out from your nightmare pains.

Test Environment Setup

Objective
How to make your automation framework accessible through out all Test Environments in other sense how to make your automation objects ( scripts \ business objects ) reusable with other platforms with Automation Tools such as QTP \ Silk Test.

Problem
You have different test platforms such as
qa:6080 like build in qa 6080 port.
qa:7080 like build in integration qa 7080 port.
qa:8080 like build in staging qa 8080 port.
Live environment (like any production site)

We need to have centralized automation script that can get access to all above platforms.

Approach for problem solution
• We need to create
1.Function that can read current global environment configuration which may be in .xml.
2.Start-up Script that can add / update / set current environment from .xml or any other format – you can make GUI tool or use VBScript + Excel to maintain easily.

•We need to normalize all automation objects which are pointing to specific platform like filtering out all redundancy from automation objects – you can here use regular expression and concate current environment by calling function.

Execution
Execute your start-up script to set up current environment we want to go on.
Associate function to the script and call it as mention above.

QA Automation in V Model

QA Automation in V-Model

Low level Testing

• Unit Testing with NUnit (and MS Visual Studio), xUnit and HTMLUnit on Java and C# / ASP.Net plateform.
• HTTP Request-Response Testing with Selenium, Canoo WebTest, SOAPUI and other free-ware tools.

Integration Level Testing.

• Components Interface Testing.
• Web-Service Testing with SOAPUI
  
• API Testing with WebTest Canoo \ QTP \ SilkTest
• Function \ Smoke Testing \ Regression Testing

High Level Testing.

• Cross Browser \ Cross Platform Testing.
• GUI Testing with Tools such as Mercury QTP, Silk Test, WinRunner.
• Load \ Performance testing with LoadRunner.

~NV

QuickTest Pro and Traceability Matrix

I like to share one automation experience or in other sense the challenge I had in automation.

I joined one company as an Automation Quality Analyst Consultant. It is fast growing Company. I was as a tool smith like person recommending tools for QA team to make manual testing tasks more easier\reliable and fast way. We had established freeware tools for testcase management, defect repository and qa\project process and developed
test-suites for regression and smoke testing in QTP.

As we all know regression test-suite is bulky one and at that time, mine was including more then 617 testcases automated and that’s why taking more than 3 hours to finish execution. I have split test scripts on functionality vise and created test batch but the problem is test batch report does not give status for each testcases in automation execution and that encourage me to develop “TM live” tool interfaced with QTP 9.1.

“TM live” means Traceability Matrix on live (parallel with automation suite execution)



Live Graphical Report - Demo

Increase ROI in your organization with Automation Testing

This article offers answers for below questions with case-study

Why do we need automation s/w testing in organization? 

What if organization does not have fund to purchase license
for expensive automation tools?

In simple way, we can say the core objective of automation is helping to finish any testing\QA task more accurate, easier and faster way with less risk involved for failure. Say for example Tools like
# Functional automation tools QTPro, SilkTest, Canoo, Selenium, VBMacro.
# Load/Stress automation tools LoadRunner, SilkPerformer.
# Cookies reader tools.
# HTTP request / response reader tools.
# File comparison tools.
# Tesecase Management Tools.
# Web Page link / image finder Tools.
# Broken link finder Tools.
and many other tools we can categorize in automation testing fashion to make QA Process more smoother; Automation Testing also used for

Repetitive testing task activities…
As a human nature, manual tester can not perform repetitive tasks more accurately comparing to machine or software automation program. Test data-driven approach is one of the best solutions of automation for repetitive testing. Automation makes "cross browser testing" a peace of cake, you just change configuration file and you are ready to verify features on other browsers.

Is More Reliable…
In the sense of giving feedback more faster and cheaper way. Automation provides excellent feedback in well HTML / XML forms which are also open to plug into our QA statistical system like System that generates Traceability Metrix say for Mercury's Quality Center or Borland's SilkCenter and we can make qa process more transparent toward organization. Anyone can execute automation scripts at anytime, most of the times they are triggered at night. Automation always pays back; when new build comes in qa, we execute smoke automation scripts to check build health and trigger regression automation scripts to verify old features work with new features as expected in more faster and cheaper way comparing to manual testing – It works like safety net where not only qa team but development team also gets more confidence and open doors to feedback SDLC loop earlier.

Is the best candidate for some testing tasks like…
Simulate virtual users that generates load / stress on server to check page throughputs, bandwidth,CPU utilization, Memory leaks and other quality parameters.
To Compare large files / database objects like Validate pictures or binary files with their expected ones.
Gathering QA statistics like Metrix and Millstones.

There are many other reasons why organization should have automation testing, we will discuss later on. One other thought I have in my mind if organization does not have fund to purchase expensive license for tools (like qtp, silktest winrunner..)
then what qa people can do? In this scenarios, It is a good opportunity to use freeware tools. My past articles can help you guys to evaluate freeware tools

"Automation Test Strategies" -- find tool selection quality parameters
"Tool Evaluation"

You can find freeware tools on http://www.opensourcetesting.org/ which sure be helpful for qa process in your organization, couple of good tools I like to recommend are as below

Functional \ Regression Automation test tools
In my past projects experience, I have successfully established an automation framework with freeware tools – WebTest Canoo and Selenium.

WebTest Canoo is on XML\Ant based tool understands HTTP request and response of the site. Everything is xml based; your test-cases, test-driver and even it produces result in xml file.

Selenium is also a good tool available as freeware.It comes with browser extension; It has its own api function library; record/play utility in built. Selenium tests run directly in a browser, just as real users. It has in built script language conversion utility that can currently support writing tests in Java, .NET, Perl, Python and Ruby.

Test Management Tools
TestLink(http://testlink.org/wordpress),
QA Track(http://www.qatrack.org/),
Bugzilla Test Runner(http://sourceforge.net/projects/testrunner/)
I like TestLink tool most as it offers functionalities such as
# Test Case history/ versioning
# Unlimited structure of Test Specification
# Support for other databases (SQL standard, generalized DB interdace)
# Improved user management (editable)
# Attachments
# Revision of Standard terminology
# Enhanced reports
# XML import/export
# Custom fields for Test Cases
# Get Traceability Metrix and Milestones
# Can create suites like regression suite or smoke suite for the test
and then create build in testlink that allow us to attach testcases
in qa, we can also assign to team for execution and testlink gives
statistics for testing progress as well.

Defect Tracking Tools
Bugzilla(www.bugzilla.org/),
BugNet(http://www.bugnetproject.com/),
Jira(http://www.atlassian.com/software/jira/)
Jira is an award winner professional issue tracking web tool. Although
it is not freeware tool but it is less expensive.

Miscellaneous tools
FreeMind is a premier free mind-mapping software written in Java. You can create test-scenarios in more easy way and publish them into hierarchical presentation in XML/HTML format.
Eclipse to compare files.
HTTPWatch tool can read HTTP headers, cookies, caching and POST data.
IECookiesView v1.70 can read IE cookies details.
Markup Validation W3 Service checks the markup validity of Web documents in HTML, XHTML, SMIL, MathML, etc.
Link Checker - Checks anchors (hyperlinks) in a HTML/XHTML document. Useful to find broken links, etc.

In my future post, I will share my customized automation tools that made to satisfy more specific testing task.

Testing Priciples & Vocabulary

The principles can be defined as
1. a rule or code of conduct;
2. a general or fundamental, law, doctrine or assumption;

In software domain, these principles focus lights to test engineers on software systems, how suppose to build them and how they expect to behave.

Principle #1
Testing is the process of exercising a software component using a selected set of test cases, with the intent of
(a) revealing defects,
(b) evaluating quality.

The term “defects” used in this and in subsequent principles represents any deviations in the software that have negative impact on its functionality, performance, reliability, security and or any other of its specified quality attributes.


Principle #2
When the test objective is to detect defects, then a good test case is one that has a high probability of revealing a yet undetected defect(s).

The test engineer expects to work as scientist does like he creates hypotheses to prove or disprove them; that is determine if the specific defect is present or absent. Base on hypotheses, tester must select specific inputs; determines correct outputs and then they will be executed. Test results are analyzed to prove / disprove the hypotheses.

Principle #3
The test results should be inspected meticulously.
Get more detail from article.

Principle #4
A test case must contain the expected output or result.
Expected outputs allow the tester to determine
(1) whether a defect has been revealed, and
(2) pass/fail status for the test. It is very important to have a correct statement of the output so that needless time is not spent due to misconceptions about the outcome of a test. The specification of test inputs and outputs should be part of test design activities.

Principle #5
Test cases should be developed for both valid and invalid input conditions.
This principle opens room for negative testing. Tester should create positive as well as negative test cases; he or she must not assume that the software under test will always be provided with valid inputs. Inputs may be incorrect for several reasons. The developer of the software component may be biased in the selection of test inputs and specify only valid inputs in the test cases to demonstrate that the software works correctly but good tester is one who is more apt to select invalid inputs as well.


Principle #6 The probability of the existence of additional defects in a software component is proportional to the number of defects already detected in that component.

QA Process implemented in 0 leveled organizations

Test Plan IEEE 829-1983 explained by me

Automation Test Plan Strategies

Tools Evaluation

Hello Guys,

This is my first post on this blog; hoping you all enjoy.

Freeware Test Tool Evaluation for Functional\Regression Testing.


















(a = Active featured)

Freeware Test Tool for Web Content Accessibility Testing.




















(a = Active featured)

* Above evaluation is an personal evaluation based on my experience and practices.

Thank you & Enjoy Testing;
~NV.