While these days most professional software developers agree that QA is important, we are still working out the best way to implement it. What seems clear, however, is that the ideal QA strategy should fit right into a modern continuous integration pipeline, and therefore be fast and reliable. Today, we’ll discuss how to identify test cases that are ready for testing automation as part of your QA strategy, and how to gain the most from its benefits while minimizing its downsides.
CREATING A BALANCED TEST AUTOMATION STRATEGY
One of the choices you have to make when thinking about your QA strategy is between automatic and manual testing, and it’s almost always a balance; there are few situations really suitable for a 100% approach one way or the other.
First, to get obvious out of the way: some things, like unit tests, are unambiguous. You should always have some form of automated unit tests! Other things, like penetration testing and usability research, almost always require a human touch. The boundary between these two is where things get interesting.
So what are the trade-offs? Well, I’m glad you asked! That’s something I tend to think about a lot, so let me try to give you a breakdown of what we’ve got.
FUNCTIONAL TESTS: TO AUTOMATE, OR NOT TO AUTOMATE?
Let’s focus on functional testing, which in the world of web applications, usually means testing the UI and flow of your app. Once you have a working app with a UX flow you are comfortable with and ship it to production, you’ll want it to have some functional test coverage. This coverage is designed to make sure you can safely iterate and improve it over time without breaking the ability of your users to achieve things. Should you test it using an automated or manual approach?
Automated tests are great, because executing them is free and they are usually fast. These are huge bonuses when you are trying to save development time, which is usually one of your most precious resources. Once you have, for example, a working Selenium script, you can run it as often as you want and get the results fairly quickly. The problem is that these tempting savings only come in after the initial (and often significant) investment of — you guessed it — dev time to write such scripts. Worse still, some automated tests tend to break unexpectedly as you change your application, so you need to continually re-invest into keeping them up-to-date. Manual testing doesn’t have this problem. It will generally work as expected, giving you reliable results without you having to specify every tiny detail. The drawbacks, however, are significant: manual testing generally creates high, persistent costs and long turnaround times.
From this, some clear pictures start to emerge. If you have a fairly stable component that you need to test often, automation is probably the way to go. Every time you execute your test suite, you will rake in a return on your initial investment. Assuming the unexpected breakages don’t happen too often, the continuous maintenance will not hurt you either, because you only need to fix your test scripts when things break. On the other hand, if you are testing something that tends to change frequently and especially if you don’t test often (which you should be doing, but life is not always simple) you might be better off going through a manual testing phase before every release.
By: MACIEJ GRYKA
LEAD SCIENTIST, RAINFOREST QA
Spanish 
English 
