What is Test Automation?
Test automation is the process of automating test steps, reducing the effort required when compared to manual testing. Besides saving time and cost, test automation is also more reliable when it comes to performing repetitive actions, something that is not a fair expectation of most humans. However, this doesn’t necessarily mean that automation will fully displace the need for manual testing. In fact, it's a great way to ease the burden of manual UI testing, allowing testers to use their creativity, knowledge, and instincts to refine test cases as features change over time. Keep in mind that automated testing is not always the best approach, something we'll detail further below.There are 3 levels of test automation: Unit Testing, API Testing and UI Testing. At Photobook, our test automation team, named Autobots (you see what we did there?) is focused on UI testing, ensuring all of the features our users utilize are working properly at various points in our development process.
So, in short, some common benefits would be:
- to save resources (time and money) when running tests by investing upfront on creating the automation;
- to improve test consistency;
- flexibility in timing and frequency of test runs;
- efficiency, tests can be run in parallel, and completed quicker than manual testing; and,
- unattended execution, no need to monitor as tests run, results can be delivered or picked up later.
When Does Test Automation NOT Make Sense?
There's never a time when, given the choice, a development team would rather go slower than faster, all things being equal. And there's usually plenty of motivation from our business partners to be nimble, delivering features to market faster than our competitors to capture market share or increase marketing metrics. With a manual-only approach, even with a huge investment in testers across your teams, you will still see issues with churn, consistency, resource constraints, etc. Automation can address many, if not all, of these concerns. But there are limits to its effectiveness.Generally, the following types of testing won't benefit from test automation. Most of the time it just doesn't make sense, as it might be too complicated, too costly, or too slow to automate.
- For unreleased, still under development applications or applications that undergo frequent large feature changes, you’re may be wasting your time creating automation; the value is in repeatability, and if the tests require constant enhancement to continue to be relevant, you won't enjoy this benefit.
- Subjective validations: things that are more opinion-based, like "this color clashes with that color."
- Any process or operations that requires complex, unrepeatable installation, configuration, and maintenance. These types of tests usually involve physical installation, adding or removing hardware and other operations that are impossible or expensive to automate.
- One-off / short-lived tests. Does it make sense to do automation for one-off tests? Even Dora knows the answer!
The Test Automation Journey at Photobook
The test automation journey in Photobook began with two simple statements:- Every company has QA; if you're not doing it, your customers are!
- Don't do something manually if it can be done by writing software.
A month after I started on this journey, I was joined by a new hire, Hatim, and after that, our team lead, Salman, who had finished up commitments to his former team and was able to fully focus on test automation with us. After months of hard work we managed to research and review several frameworks, choosing one that fulfills our needs. We completed a set of test scripts for our website, and integrate these tests with our CI/CD framework.
The journey has always been full of fun, excitement and challenges. The ever-present blockers, the joys of finding solutions, and the taste of victory had focused us and intensified our passion and commitment to make this project a success, while we stayed hungry for more.
Do you have the same passion and enthusiasm as us in this automation journey? We may be hiring (we expand as the rest of tech and Photobook expand), check out our job openings, and apply! Join us and be part of the cool Autobots team! We can't wait to welcome new members into the family :)
Do We Really Need a Framework?
A lot of the consistency and efficiency we've been highlighting really comes from the Framework, which is defined as "a basic outline or structure, a set of guidelines which when followed produce coherent and beneficial results."Break it down
Before discussing the Test Automation framework, let’s first understand some concepts.
- AUT: The application under test
- Test Automation Tool such as Selenium
- Test Management Tool where the test cases and screenshots are stored. In its absence a common shared drive can also be used.
- Test environment: A stable environment on which application is deployed for test purpose
- Application Objects: Different elements within application like textbox, checkbox, radio button, dropdown, etc.
- Application modules: Specific functional flows that needs to be executed within different test cases. E.g. User login.
- Test Data: Input values (valid or invalid) for the application fields such as Login credentials
- Functions: A set of reusable statements (for a particular functional flow) that needs to be executed within different scripts. E.g. User login.
- Test script: A standalone test case coded in a particular programming language using the test automation tool
- Results: This includes the error logs, execution status, screenshots, formal reports, etc.
Officially, a Test Automation Framework is a set of protocols, standards and guidelines (programmed) that sets the rules of testing specific software – how different automation components will interact to run automation scripts and capture the results in a coherent and beneficial manner (i.e. how these different components, or building blocks, need to be assembled).
Common approaches
Confused yet? To help clarify, let’s see some combinations of how these different components can interact, i.e. different ways these components can be utilized.
- User uses a record-and-playback feature wherein user records each step of the test case (including navigation and data inputs) >> script is auto-generated >> edit the script to insert checkpoints >> Playback the script. Environment, Objects and Test data are all captured during recording in the test script.
- Instead of recording each and every step, club common tasks like User login into functions which can then be called from within different test scripts.
- We can create a separate object repository to store details of all application objects and then use these objects in test scripts, instead of identifying the object every time in every script.
- How about we keep the Test data outside the script? Say in a separate excel file? And then load these values in script variables at run-time?
- Say we create different functions for every application flow/action and then read the sequence of steps (along with the data) from an external file like excel sheet? E.g. OpenHomePage >> EnterUsername >> EnterPassword >> ClickLogin
- What if we define a common Configuration File containing all the environment settings like application URL, browser specific information, login credentials, etc.?
- Capturing the screenshots for only the failed steps and creating a cumulative Pass/Fail report at the end of test execution which is saved in a shared drive
- Test results are logged into a separate log file which can then be mailed to stakeholders
The Test Automation Framework plays a key role in automation testing. But before you start building a framework, keep in mind that there is a divergent range of Test Automation Framework options available now-a-days – make sure you are building a framework which is best suited for the project or application under test!
OK, Yes, We Need a Framework
You got this far, so you're asking "what should I look for when considering various framework options?"Here are some things to consider:
-
Free Open Source Software
Who doesn’t want freebies, right? And freebies that are maintained for you, on top of that? Thats awesome! Add to that the availability of the source code, so you could figure out just what is going on in tough situations, or even contribute back yourself. -
Multiple Platforms
Your chosen framework shouldn't just be a single tool, but an entire toolbox, enabling you to tackle automation for systems with differing technology (e.g. mobile vs. tiered web vs. desktop vs. API vs. serverless vs. ...) -
Able to Run Tests in Parallel
This is even more important if you envision wiring test automation into your CI/CD/deployment process. Even if not, it's still a great time saving feature when you're running ad hoc. -
Test Grouping
What if you want to execute a subset of your tests? You should be able to organize tests however you want, by functional area, by system, etc. And during development, you should be able to run a single test whenever you need. -
Rerun on Fail
Given some fallibility in your local / non-production environments, you may want certain tests to be configured to be run multiple times in case the failure is not really due to a root cause that would ever manifest in production. -
BDD
Behaviour Driven Development is centered on the principal that some human-readable domain specific language can be used by non-engineers to define valid and invalid behaviours, and by formalizing this DSL, your tests should be able to run based on these definitions. -
Meaningful Reports
Placing substance over style, reports that quickly surface issues and allow efficient investigation and rectification are key to a well built framework. -
Community / Vendor Support
Picking a FOSS option which hasn't seen updates in years might be a bit risky; you're likely not going to find support when needed. Same goes to vendor based options if their market share is in decline. -
Programming / Scripting Language(s)
The learning curve, and the cost of ownership in the long term, can both be reduced if you can use languages you already know to build your tests. Even if you can't, then at least popular languages would be better than obscure or out-of-favor ones.