Testability Is Always Good Thing

The title may seem like a stretch or wild opinion, but I don’t think it is.

Suppose a piece of software is developed. What does making it testable mean? What’s the benefit?

Testability is having a way of reducing uncertainty around what a piece of software does. It provides a valuable method of answering the question “How do you know what you know?” about your code.

Bluntly, most problems with software arise when developers don’t completely understand how and why a piece of software does what it does. How do you know what a function does? How do you know a particular feature does what you think it does?

Testability is a feature. It helps with these questions and others, and it always a good thing.

In Defense of Java

A couple of weeks ago, I found a Wikipedia article on - basically - why Java is a bad language.

I’m no Java advocate, but sometimes Java bashing can get a little out of hand. So I thought I’d post some good things about Java (based on my experience).

Full disclosure: I love my wife, I love my cats, but I do not love Java.

In no particular order, here are some of the good parts of Java:

Packaging: Creating and maintaining packages of Java code is straightforward. Make a directory/subdirectories, create classes in .java files with correct imports, then jar it all up! Managing packages is even easier with modern IDEs like Eclipse. Creating reusable libraries works more or less exactly as it should without any strict structures or external tools. In turn, adding and removing libraries from other projects also tends to be straightforward. Adding or removing libraries is as simple as adding or removing jars from a project.

Packaging is one of the underappreciated aspects of Java as it allows for making “production-ready” code easier to build and maintain. It’s also quite helpful when working with projects that are larger than a couple of files. Even in Python - one of my favourite languages - projects can be difficult to build, or dependencies not easy to manipulate (such problems is why virtualenv exists).

Testing Frameworks: When it comes to unit testing frameworks, JUnit is the benchmark. Written by Kent Beck (and a great example of test-driven development), JUnit really is an excellent unit testing framework. It strikes a great balance between using Java’s native language features and being a separate testing library. Concepts like annotations for test methods, setup/teardown methods and running test suits via XML are good practices that are emulated elsewhere in other languages. Going a step further, TestNG is a test framework written by Cedric Beust that is built on top of JUnit. TestNG extends JUnit’s features in logical ways, allowing for setup/teardowns before classes, test suites and groups in addition to test methods. It also provides easy functionality to run tests in parallel, as well as valiantly providing a way to give methods default parameter values. JUnit is a great framework for writing unit tests and TestNG is a great framework for writing any other kind of automated tests.

Cross-Platform: Say what you will, but this is at least trivially true. Even C code needs to be recompiled on different platforms to produce an executable.

Consistent Paradigm: Java was designed as an object-oriented (OO) language which is continues to be. Even if you’re not a big fan of OO programming, there is value in a language having a consistent underlying paradigm. Almost every class is kept in its own file, and every object (probably) extends java.lang.Object. This makes projects and classes easier to grok and aids in producing usable design patterns. Even though Java has made lambda expressions a first-class language feature and using Java sometimes meaning getting a gorilla along with your banana, at least with Java you know what you’re getting.

public static void main(): Honestly, being able to turn any class into a executable by adding a single static method is a surprisingly great feature. It provides a quick and easy way to check output or what a piece of code is doing. I’m not sure if Java did this first, but it’s something I use a fair bit.

That’s it for now for my sort-of defense of Java. Like it or not, it’s here to stay.

Being Cool About Best Practices

In software testing, much ink has been spilled on the topic of “best practices”. This tweet really sums things up well for me.

Although I believe that much of software testing and quality is highly context-dependent and context-driven, some cool ideas do often work well in many situations and contexts. Note I didn’t say these ideas always work in all situations. That is important.

The closest I get to using “best practice” is “good practice”, usually written as in this sentence: “A good practice when working with widgets is to build a new sprocket for them first”. Instead of insisting there is a single best practice for testing scenarios (suggesting a unique, globally optimal approach), there are often good practices that work well (not unique and maybe only locally optimal some of the time). Hence a good practice instead of the best practice.

I think the “cool idea” expression takes things a step forward by removing any sense of formality of the circumstance. Cool ideas are pretty cool, but can become uncool as well. Of course, to appreciate this point, you have to be paying attention to the coolness of the situation. Paying attention is quite helpful when testing software.

Let me be among the first to promote this phrasing as part of the context-driven tester’s vocabulary, coined by Michael Bolton. Get your team thinking about cool ideas in their work.

The “Build It After They Come” Anti-Pattern

Lately I’ve been thinking a lot about UI automation. There’s plenty to think about.

On the one hand, I’ve thought that UI automation kind of sucks. It’s an opinion that’s slightly radical but not completely unfounded.  

One the other hand, UI automation can be a great tool for developers, testers and software development teams. It can be helpful for a variety of reasons. Plus, there are some great tools and resources out there for UI automation.

I think part of the problem is that, like a lot of things in software development, UI automation isn’t itself bad but may be used badly or in improper situations. Like pointers in C++, using them and using them in an appropriate way can be very, very different things.

One way that UI automation can be used inappropriately, I think, is to delay it until late in the development cycle. The situation ends up like this:

  1. Software is developed with a specific release date in mind
  2. Development and testing proceeded until close to the release
  3. At this point, automated UI tests are requested to run looking for issues.

The reasoning for following such a pattern is to have the UI tests act as a last check before shipping. Automated tests are straightforward and defined ahead of time, and so act as a good final check.

Sounds pretty reasonable, right? Except that, in practice, there’s a number of problems with this situation such as:

Automated UI tests can be quite sensitive to small changes in the UI, leading to unreliable results: This is a classic case of a radio list becoming a checkbox. Small changes in the UI like this can cause unreliable results in automated UI tests. Trying to locate problems and repair them can be difficult, even more under time pressure.

Test code is susceptible to code rot if not run often: Automated tests may be written and refactored a head of time, but can degrade over time if they are run regularly. This can lead to issues that need to be fixed when the tests are actually needed. See the previous point.

End-to-End UI tests can be relatively complex, meaning they may not be ready to be run in time if they’re not started early enough: One of the arguments for employing UI automation at any point in the development cycle is to save time. Machines are generally faster and more efficient than human beings. However, some scenarios are still complicated enough that machines take a lot of time to complete them, in addition to any setup and initialization that is required.

Automated tests in general can miss subtle bugs a human would find more easily: Computers can only do what they are told to do. It is often valuable to open up an app and taking a human look before committing to a release.

From this perspective, running automated UI tests (or any automated tests) as a last step before shipping is a form of release testing, which is a development anti-pattern. Waiting until just before release to do any form of testing is not a terrific strategy, since finding issues can become precarious. Does a show-stopper bug push back the release? What about less critical issues? Or could these issues have been found earlier?

In this sense, instead of having automated UI tests run at the end of a production cycle, it makes sense to start them as soon as possible (possibly even the first thing that gets done). Not only do the test runs become more helpful at providing information, but the process of automating the app can be a source of information as well.

Yes, automated checks can help provide critical information in an efficient way, but there may often be a better way.

Actual Behaviour: It Stinks!

An important skill in software testing is being able to write a good bug report. Here’s a situation that’s been on my mind related to bug reports for the past while.

Is this a valid (albeit fictitious) bug report?

"After opening TestApp in my browser, I logged into my account and did the following:

-Went to the Widgets list
-Clicked on the Create Widget button
-Waited for the New Widget dialogue to appear
-Attempted to create a new widget

The expected result was that I created a new widget easily in a straightforward manner.

The actual result was that creating a new widget sucked, even if I was able to create the widget successfully.

I was able to reproduce this bug using other user accounts on IE, Chrome and Fire fox. In all cases creating a new widget basically sucked.”

This may seem frivolous; I even thought so myself when I first thought of it. After some reflection, I wondered if this is actually a good bug report.

I’ve often tested apps or features and been left frustrated. Even if the feature works as expected, I find myself being upset, or thinking things like “This work flow is so shitty. Why does it have to be like this?”. In these cases, I find myself thinking that yes, there is a problem here.

What’s interesting is that, by some conventional reasoning, there is no bug. The app works according to the spec. It follows the documentation as expected. The output is correct for the given input. There are no performance issues. Security is not an issue. In some cases, even automated approaches can be applied to this area without problem. Following this line of logic, there is problem here. Hence, the bug report is closed as “Not a bug”.

But it still sucks.

I do believe that good software testing involves information gathering. Drawing attention to a particular area can be highly beneficial and have value even if there is no clear “problem”. Even if the above bug report is posted, it could trigger a discussion. Maybe there is a usability issue, or something even more subtle like localization or accessibility. There may also be misunderstanding: a particular app may have to be a particular way for reasons the bug report’s author hadn’t thought of. Or it could just be something to improve.

Starting these kinds of conversations can be difficult. Bug reports like the one above may or may not help depending on the team or the culture. However, it could also be a way to express something that is slightly intangible at first. It may even be a shared feeling.

For full disclosure, I’ve never posted a bug report at work like the example above. But I have considered it, and I think in some cases I could defend it. Would it be professional or acceptable is really the $64 000 question. I think it can be absolutely acceptable in the right environment. But that’s just me.

GUI Automation Still Sucks

It’s been a busy few days in the testing blogosphere.

Noah Sussman starting things off by tweeting a post about why GUI test automation is so difficult. It’s not an Earth-shattering argument, but he correctly points out that trying to account for the kinds of errors that appear in a GUI-based test can be difficult to automate, leading to fragile tests. I have similar opinions, but I didn’t spell things out as clearly as Noah did.

In response, Alan Page took things one step further by suggesting that testers stop writing (GUI) automation all together. This post generated a lot of attention, and rightfully so. GUI automation is clearly a topic near and dear to many people in the software world. I just wanted to add some more commentary to the mix.

First of all, both Noah and Alan’s posts are very good. Definitely read them and any comments around them in full.

With that out of the way, here is my opinion on GUI automation: a lot of it sucks. And this is coming from a person who’s written a fair bit of it.

On top of the reasons that others have stated, I’d like to point out a couple of more reasons for why GUI automation is less than optimal:

Automation is a development activity: Writing good automated tests will almost always mean programming of some sort. Using only record and playback tools and having testers with little to know coding experience write automation can’t lead to good results. The best GUI automation scripts are written and maintained by developers, who treat automated tests as any other kind of code. If you want GUI automation to succeed at all, you need developers.

GUI automation as a dumping ground: A more insidious problem with GUI automation is when it’s treated as a catch-all for test teams. Since GUI automation is so general, the theory goes, you can apply it to testing any aspect of an application. Difficult-to-test features or long, complex regression tests are sent to automation where they are expected to suddenly get more useful and reliable. Don’t know how to test it? Automation to the rescue! The only problem is that this doesn’t actually help product quality. Just like quality can’t be “tested in”, it also can’t be “automated in”. I admit this as much a problem with team organization as it is with test methodology, but it can still happen on test teams.

GUI automation devalues other automation: A surprising number of people in software development think only of GUI automation when they think of automation. This is kind of interesting, as GUI automation is such a special case in terms of automation since it requires specialized tools and so much maintenance for a rather general task. There are so many other forms of automation that are highly beneficial (automated builds, continuous integration, unit testing, code generation and static analysis, and so on) that to focus on just GUI automation is a bit unfortunate.

What continues to amaze me how tightly people cling to using GUI automation as business as usual. People obviously don’t want to stop using it, but why remains a mystery to me. In my experience, automation doesn’t necessarily lead to faster testing, more thorough testing or even easier testing. Automating the UI can be highly effective in some contexts, but also downright annoying in others.

I still hold that automation is a tool. Sometimes you have to decide whether or not it’s worth it to use such a tool.

Tools in Your Toolbox

I like to cook. If you opened up one of my kitchen drawers or took a look at the shelves in my kitchen, you’ll find a whole bunch of tools and gadgets.

Some of these tools I use daily or more (spatula, tongs, chef knife).

Some of these tools I use less than daily, but still pretty frequently (can opener, vegetable peeler, microwave).

Some of these tools I don’t use very often, but are very helpful when I use them (lobster claw crackers).

And some of these tools I barely use. I’m not even sure why I have them (avocado slicer).

The point of all of these tools, however, is to help me make great meals. When I’m thinking about preparing dinner, I haver never once thought “How can I use all of the items in this drawer?”. In fact, that idea just seems silly. Even “How can I use most of the items in this drawer?” is kind of a weird approach to cooking.

I’ve also thought things like “What can I make using my new food processor?”, and followed through on them, usually with mixed success (hummus, or variants thereof). Usually these ideas are more experimental, and are treated as such.

Most often, I have thought “How can make a tasty meal?”. That’s why I have this kitchen full of tools. And that’s the goal these tools are meant to accomplish.

Using Selenium Locators the Smart Way

One of the first concepts developers encounter when working with Selenium scripts are element locators. Working with locators is a core skill when it comes to Selenium, but it’s also one with a surprising amount of subtlety.

A little while ago, I was part of a nice discussion of good ideas for defining locators in scripts.  There’s some good ideas there from everyone, but I thought I could elaborate here.

Let’s assume you’ve found a good attribute to locate an element by (which is a topic covered here) and you want to write a script working with this element. This may sound pretty specific but something like 95% of all Selenium webdriver tests and scripts involve doing this so it’s fairly important. How would you code this up?

To click an element with an id locator, here’s a first approximation in Python:

driver.find_element_by_id(“good_element_id”).click()


Looks innocent enough but it actually isn’t very good. The biggest problem is that there’s no easy way to identify the element again. This may seem like a quibble, but in my experience an element is rarely worked with only once in a test or script. Finding the element using a hard-coded locator value means copy-pasting similar code for multiple actions. This means you’re repeating yourself, making maintenance and refactoring more difficult.

A big improvement is put the locator in a variable which is set ahead of time. In our case, that could look like this:

good_element_locator = “good_element_id”
driver.find_element_by_id(good_element_locator).click()

Now, it’s easy to add other find_element calls as well as easy to change your locator in one place instead of several places. This is a big win for maintenance of scripts which should never be underestimated. You can also move around the locator as needed now, meaning more options for organizing your scripts. Again, a big win for maintenance.

This is going so well, you might go a step further and define the entire element in a variable instead of just the locator. So you write this:

good_element = driver.find_element_by_id(“good_element_id”)
good_element.click()

Seems good right? Saving even more typing and making things even less redundant.

However, you’ve hit a pitfall.

When a find_element() call is made, the webdriver will try to find that element in the current DOM. Sometimes you’re intersted in working with elements that are not in the DOM, such as confirming an element is not displayed or present. Defining the element instead of just the locator leads to problematic situations. Consider this test:

good_element = driver.find_element_by_id(“good_element_id”)
assert !good_element.isDisplayed()

If the element is not displayed, this test will fail. It doesn’t fail at the assert, however, but at the previous line. Since the element is not present, the webdriver does not find it and raises an error, which is what fails the test. For beginners this can seem counter-intuitive, particularly since the element doesn’t appear, which is the expected result. Subtle but deadly.

I might have a second part to this post, but for now the essential idea is here. When it comes to locators, use variables and use them well.

UI Automation Is Hard

Today I was reading this post about issues with automated tests taking to long. It finally got me to write this post.

Here’s the dirty secret of UI automation: it’s very difficult. Difficult as in “finding a needle in haystack”, not difficult as in “solving the Riemann Hypothesis”.

What makes UI automation so difficult? The U in UI stands for “User”. This typically is understood as “the person using the application”. By trying to automate an app through its UI, a machine is made to interface with something meant for a person. Machines are generally good at things people aren’t, and vice versa. Hence, this task is very difficult.

This is the core reason why UI automation efforts require so much maintenance, take so long to prepare and execute, and lead to complex set-ups. It means using a tool to for a task it is not designed for.

I’m not saying UI automation isn’t beneficial. It can be in many contexts. But it’s also a challenging problem with a possibly low benefit-cost ratio. UI automation often isn’t the right tool for the job. Testers, developers and other individuals in tech would do well to keep this in mind.

What Not To Do With Selenium: Site-Wide Format Testing

Continuing on an earlier theme, here’s a situation I’ve been thinking of recently.

Suppose you’re testing a web app that has some site-wide setting like a date format. For example, a field in the app may allow the user to change a date format from “01/02/2014” to “Jan 2, 2014” across the app. Test cases here are fairly straightforward; login to the app, change the site-wide setting, navigate to a page with fields that employ this setting, and verify the format changes accordingly. Being a good test developer, you think to yourself, “This could be a good place to use Selenium! Simple and quick.”. It’s all good, right?

Not so fast.

Automating testing of site-wide settings might seem like an easy win but there’s some subtle problems that arise.

The main one I’ve seen is interference with other tests. In the case above, if your automated tests only tested these site-wide settings, there’s really no problem. However, once you add any other sort of tests to the mix, you might run into problems. Tests may begin to fail unexpectedly because checks fail due to formatting. For example, consider checking a post on a blogging app appears as expected for a given date. Posting might functionally work as expected, but checks might fail when checking the posted date if the formats don’t match up and are not reset properly. The expected date may be correct but its format might not be.

This problem can be compounded further if test are run concurrently. If one test changes a site-wide setting at some point during a run, other tests could be affected. At this point, debugging might get pretty tricky and test results may be skewed to false failures.

One way around these problems is to set formatting to a known, fixed format before every non-formatting test. The downsides are additional test setup and potentially longer test execution time, not to mention that you are now implicitly assuming that the site-wide formatting works correctly in some cases. Another approach would be to isolate the site formatting tests so that they do not interfere with other tests. Isolating these tests may limit their utility in addition to being difficult to do effectively. Lastly, you could convert all fields that are formatted to some common format, removing formatting from the picture entirely. This might work but could also mask bugs and may require quite a bit of upkeep.

Not so simple and quick.

I’m going to outright say that Selenium isn’t the right tool for checking site-wide formatting, but I would consider it with caution. As test suites grow, such issues can cause headaches for the maintainers, especially when tests seem to randomly fail. Things may make sense in your context, so just be aware of what can go wrong.