One of the first concepts developers encounter when working with Selenium scripts are element locators. Working with locators is a core skill when it comes to Selenium, but it’s also one with a surprising amount of subtlety.
A little while ago, I was part of a nice discussion of good ideas for defining locators in scripts. There’s some good ideas there from everyone, but I thought I could elaborate here.
Let’s assume you’ve found a good attribute to locate an element by (which is a topic covered here) and you want to write a script working with this element. This may sound pretty specific but something like 95% of all Selenium webdriver tests and scripts involve doing this so it’s fairly important. How would you code this up?
To click an element with an id locator, here’s a first approximation in Python:
Looks innocent enough but it actually isn’t very good. The biggest problem is that there’s no easy way to identify the element again. This may seem like a quibble, but in my experience an element is rarely worked with only once in a test or script. Finding the element using a hard-coded locator value means copy-pasting similar code for multiple actions. This means you’re repeating yourself, making maintenance and refactoring more difficult.
A big improvement is put the locator in a variable which is set ahead of time. In our case, that could look like this:
good_element_locator = “good_element_id”
Now, it’s easy to add other find_element calls as well as easy to change your locator in one place instead of several places. This is a big win for maintenance of scripts which should never be underestimated. You can also move around the locator as needed now, meaning more options for organizing your scripts. Again, a big win for maintenance.
This is going so well, you might go a step further and define the entire element in a variable instead of just the locator. So you write this:
good_element = driver.find_element_by_id(“good_element_id”)
Seems good right? Saving even more typing and making things even less redundant.
However, you’ve hit a pitfall.
When a find_element() call is made, the webdriver will try to find that element in the current DOM. Sometimes you’re intersted in working with elements that are not in the DOM, such as confirming an element is not displayed or present. Defining the element instead of just the locator leads to problematic situations. Consider this test:
good_element = driver.find_element_by_id(“good_element_id”)
If the element is not displayed, this test will fail. It doesn’t fail at the assert, however, but at the previous line. Since the element is not present, the webdriver does not find it and raises an error, which is what fails the test. For beginners this can seem counter-intuitive, particularly since the element doesn’t appear, which is the expected result. Subtle but deadly.
I might have a second part to this post, but for now the essential idea is here. When it comes to locators, use variables and use them well.
Today I was reading this post about issues with automated tests taking to long. It finally got me to write this post.
Here’s the dirty secret of UI automation: it’s very difficult. Difficult as in “finding a needle in haystack”, not difficult as in “solving the Riemann Hypothesis”.
What makes UI automation so difficult? The U in UI stands for “User”. This typically is understood as “the person using the application”. By trying to automate an app through its UI, a machine is made to interface with something meant for a person. Machines are generally good at things people aren’t, and vice versa. Hence, this task is very difficult.
This is the core reason why UI automation efforts require so much maintenance, take so long to prepare and execute, and lead to complex set-ups. It means using a tool to for a task it is not designed for.
I’m not saying UI automation isn’t beneficial. It can be in many contexts. But it’s also a challenging problem with a possibly low benefit-cost ratio. UI automation often isn’t the right tool for the job. Testers, developers and other individuals in tech would do well to keep this in mind.
Continuing on an earlier theme, here’s a situation I’ve been thinking of recently.
Suppose you’re testing a web app that has some site-wide setting like a date format. For example, a field in the app may allow the user to change a date format from “01/02/2014” to “Jan 2, 2014” across the app. Test cases here are fairly straightforward; login to the app, change the site-wide setting, navigate to a page with fields that employ this setting, and verify the format changes accordingly. Being a good test developer, you think to yourself, “This could be a good place to use Selenium! Simple and quick.”. It’s all good, right?
Not so fast.
Automating testing of site-wide settings might seem like an easy win but there’s some subtle problems that arise.
The main one I’ve seen is interference with other tests. In the case above, if your automated tests only tested these site-wide settings, there’s really no problem. However, once you add any other sort of tests to the mix, you might run into problems. Tests may begin to fail unexpectedly because checks fail due to formatting. For example, consider checking a post on a blogging app appears as expected for a given date. Posting might functionally work as expected, but checks might fail when checking the posted date if the formats don’t match up and are not reset properly. The expected date may be correct but its format might not be.
This problem can be compounded further if test are run concurrently. If one test changes a site-wide setting at some point during a run, other tests could be affected. At this point, debugging might get pretty tricky and test results may be skewed to false failures.
One way around these problems is to set formatting to a known, fixed format before every non-formatting test. The downsides are additional test setup and potentially longer test execution time, not to mention that you are now implicitly assuming that the site-wide formatting works correctly in some cases. Another approach would be to isolate the site formatting tests so that they do not interfere with other tests. Isolating these tests may limit their utility in addition to being difficult to do effectively. Lastly, you could convert all fields that are formatted to some common format, removing formatting from the picture entirely. This might work but could also mask bugs and may require quite a bit of upkeep.
Not so simple and quick.
I’m going to outright say that Selenium isn’t the right tool for checking site-wide formatting, but I would consider it with caution. As test suites grow, such issues can cause headaches for the maintainers, especially when tests seem to randomly fail. Things may make sense in your context, so just be aware of what can go wrong.
Automated testing has a lot of benefits, and can reveal a lot about a piece of software. One of the more interesting aspects of automation is when such tests find issues that are totally unexpected.
Here’s one such story.
At my company, we have a team of test developers who write and maintain WebDriver-based functional tests, which I’m a part of. These tests focus on basic functional aspects of the web app under test, such as logging in and straightforward business logic. There’s a fair number of tests that cover different areas of the app which run both nightly and on-demand. Results are reported after tests are run, providing information on the overall functionality.
In other words, these tests were designed to look for issues in the UI and business logic aspects of the app.
However, one day we found it revealed something else.
One morning we came in to find all of our tests skipped. All of them. Puzzled, we checked the test servers with the recent build of the app under test. They were up but not connecting to the back-end correctly. It turns out the databases for these test servers were taken down. Why? Because they blew up. There was a surge of data sent into them, causing them to crash. The underlying cause of the crashes was a bug in the client-side that caused large stacktraces to be recorded to the database with every login attempt, valid or not. Essentially, the app was experiencing a sort of denial-of-service attack.
Definitely not expected.
The key to this story is that our WebDriver tests make use of (SOAP) server APIs and parallelization. Using APIs allows for quick and easy data generation, while parallelization helps speed up test execution time. These are both good approaches that I recommend considering for UI test suites. In this case, however, they combined together to crease a situation where there are lots of login attempts happening in a short period of time, resulting in a database crash. This was not something our tests were designed to do, nor was it something we considered when we started parallelizing.
The bug was obviously identified, and eventually fixed.
For me, this is one of the most fascinating parts of working in test development. Whenever software it built, developers should be wary of side effects. In a production development setting, side-effects and unexpected behaviour can be quite undesirable; for automated test development, side effects can be quite beneficial. I would’ve never considered looking for such issues despite the fact that automation would be a good tool for a scenario like this.
I’ve always felt that the process of automation produces the most benefits for testing while actually automating an app, but this case also showed how automation can lead to happy accidents as well.
This may be an unpopular opinion, but I think it’s the start of a good conversation.
I currently work as a test developer. As a test developer, I am a developer. I am a not tester.
Let me explain.
Test developers write test code. This includes (but is not limited to): writing automated tests, developing test frameworks, building automated tools for production developers and/or testers, recording and reporting automated test results and working on Continuous Integration-related projects. I’ve seen at least a little bit of each of these things in my work. These are activities that are beneficial for building good software. I’d also say these activities help improve the overall quality of a piece of software if done appropriately.
But it’s not testing, at least in the way I think of what testing is.
I don’t sit down and perform exploratory testing of an app, nor do I thoroughly investigate the app under test. That’s not to say that I shouldn’t do these things, especially since testing is an activity should be done by everyone on a team. I do report bugs and work with automating test cases, which could be considered testing activities, so in that sense I’m a tester at least some of the time.
This mirrors my opinion on what test development is (and isn’t), which has been bugging me for the past while. Test development is development, but it isn’t testing, or at least not completely. I think this is the start of sorting out the value and benefit of automation in software development.
Like Rodney Dangerfield, I feel like source control gets no respect.
Good source control systems (Git, Subversion, Mercurial and so on) are widely used in software development. These systems are generally used, as the name implies, to keep track of changes made to the source by different team members. That allows developers to see the progress of new changes and potentially roll back bad changes to avoid problems.
Developers widely understand source control as this and only this. It is a tool but a simple, limited one. Some feel that source control can get in the way of producing good code by breaking the “flow”. Others feel that source control is something only professional developers on professional projects need to use.
Modern source control can do so, so much more than that.
I’ve been working with Git for about a year now, and I think it’s an excellent tool for any developer. In fact, I’d say it’s an excellent tool for anyone working with text of any sort. Source control can do handy things if you learn how to use it. Source control systems don’t just save your work; they make make it easier and more efficient to produce good code. For some reason, some programmers just don’t feel this way. I’m going to give some examples of what source control like Git can do for you.
Conduct small, one-off experiments: Sometimes looking at code, you might want to try out something small. Maybe it’s not even directly relevant to what you’re working on or just a nagging question about the language you’re working with. What happens if I make x a float instead of an int? What if I delete this line of code? What does that constructor actually do? With good source control, you can easily undertake these experiments. Just make the change, figure out whatever you want to know, and repeat. Once you’re done, you may end up with something irrelevant to your work. You’ll avoid losing your place while coding or introducing annoying errors with superfluous code.
Conduct large, complex experiments: Suppose you want to re-write a large chunk of code but are not sure how it will turn out. Or you want to start a “test-first” approach to writing code for an existing project. These experiments could take days or weeks, affecting significant parts of your code base. How do you do this while not breaking current work? Again, there are good approaches to doing this. Workflows like feature branching allows parallel development between the mainline of a project and an experimental subproject. Feature branches can also be used to share such experiments among team members for commenting or additional work, again without disrupting the mainline of development.
Portability: This one seems basic but can be quite helpful. Having a repository in a central location makes getting source code a breeze. Setting up a machine for a new developer? How about working with your code on an arbitrary test machine? Getting code is as easy as a pull command.
Basic Continuous Integration: CI is all the rage these days, for good reason. CI helps developers and teams build and release products faster and with higher confidence. But instead of diving into some hardcore CI setup, some source control systems have similar functionality in a slightly stripped down form. In Git and Subversion, you can add hooks to perform particular actions like post-committing or pre-pushing. For example, you could put your commit messages through a formatter or trigger automated tests. Hooks can be as simple or involved as you want, making a good firs step toward a larger scale CI system.
Personal Projects: With distributed source control systems like Git or Mercurial, you don’t need to connect to a server at all. You can set up a new repo in a single command. Because of this, you can use “professional” source control for your personal projects, no matter how small. You’ll have all of the above benefits even when you’re hacking around in your free time.
There are more benefits, obviously, to using source control well, but these are the ones that I’ve seen in practice. Honestly, the first two points alone are worth the price of admission in my opinion.
Don’t just passively push and pull your changes. Give your source control system the respect is deserves.
A quick thought.
Should everyone learn to code? I don’t see why not, so yes.
Should everyone learn to develop software? No.
What’s the difference, you ask? Writing code is exactly that: opening up a shell or text editor or IDE and just hacking away. Some folks would be interested in doing this, for hobby or otherwise, so let them at it.
Software development involves that, but there’s a lot more to it.
Developing software means taking into account critical aspects of a program or app. It means allowing customers or other stakeholders into the process. It means actually caring about details that appear small to some, but aren’t to others. It means providing real value, and achieving actual goals not always set by the programmer. It’s about concern for quality in the context of what you’re building.
In short, there’s some additional purpose in software development. In coding, there isn’t, necessarily.
As I’ve mentioned before, coding is just one part of the software development process. It’s all too easy to conflate the two.
Right now, I write test code for a living. I mostly write UI tests using Selenium. I rather enjoy it, and try to improve my abilities, a little at a time.
So when I read an article about writing good unit tests, it got me thinking about the test code that I write. Specifically, I thought about UI tests versus unit tests. Unit tests should be short and easy to write. Why can’t UI tests be the same?
In the link above, Bob Martin describes writing unit tests that have exactly three lines of code. He further goes on to argue that all unit tests should be exactly three lines of code. An example he gives, for testing a stack object, looks something like this:
Stack stack = makeEmptyStack();
There’s obviously some benefits to writing tests in this way. Each line has a specific meaning (initialize/instantiate, perform the action, check), making tests more easily understandable. Tests become simpler to write because they’re more structured. Plus the code is easier to maintain, which is a big win for making tests usable. These kind of considerations turn unit tests into something closer to code specs. In short, there’s lots to like.
So why not write UI tests in the same way? Instead of code like this
PageObject testPage = new AccountsPage();
take a look at code like this
PageObject testPage = new AccountsPage();
assert testPage.isAccountPresent(“101”, “abc”);
Code is cleaner, intentions are clear, and it’s easier to extend tests to be data-driven. Imagine replacing the hard-coded strings above with a AccountData object. These can then be pre-populated and tests run based on this data. This functionality is built-in to some test frameworks like TestNG. Plus we get the benefits mentioned above.
Code like this also gets us thinking of UI tests in terms of unit tests. In other words, instead of just stringing together UI actions, tests can represent single cohesive units of functionality within the app. This could assist in nailing down acceptance checks when building an application, in addition to helping translate acceptance criteria into automated tests and vice versa.
Of course, there are some hurdles to writing code like this (hat tip here and here). Unlike creating and destroying objects, starting and stopping an app can be costly, and in turn opening and/or closing the app after every individual test may not be desirable. This can also increase overall test execution time, making long tests yet longer. Of course, there are ways around this, usually making judicious use of setup and teardown methods. In our case, where I work we have methods that open and close a browser once per set of tests in the case of web UI tests instead of after every tests. This particularly helps when tests are run on weaker VMs with limited processing. But with other aspects of UI tests, it’s all about making it work behind the scenes in the page object layer.
Above all, in my opinion, thinking about UI tests like unit tests put both on the same level of importance (or at least helps). Unit tests can assist building classes and methods, and in a similar way, UI tests can help build a final application, at least from the UI perspective. If nothing else, these ideas might make writing and maintaining UI tests a little more helpful and less fragile.
When it comes to testing, I think it’s important to have a strategy. That is, come up with a plan of action of what and how to test, preferably with input from all individuals involved in the project.
"Test Everything" is such a strategy. And it’s a bad one.
I’ve seen this strategy executed in practice, usually at the request of a product manager. I’ve also heard similar stories from other testers. The results I’ve seen have been less than optimal.
There’s a few reasons why I think this.
Test Everything is intellectually lazy. It requires no thinking, since it can apply to any product or feature in any team on any project. It also completely neglects team priorities and dynamics. But because it’s so general, it totally neglects context. Risk analysis is completely neglected; every aspect of the app is seen as equally risky. Is a given bug or issue that’s raised of high or low importance? We don’t know, we’re just to test everything. Are there any regressions or outstanding issues that we should be aware of? We don’t know, we’re just to test everything. Yes, people can be difficult and unpredictable but you can’t use that as a excuse to be lazy with test planning and strategy. It robs testers of opportunities to do some interesting and valuable testing.
Test Everything also difficult for testers cognitively. Since there’s not many details to go on, it can be daunting to start. Even simple apps or features can generate lots of possible directions for testing. Without any concepts of risky areas or priorities, it may take additional work to determine starting points or goals. For novice testers, this can be highly intimidating, even stressful. Again, not a great situation for helping testers do their jobs well.
One of the more insidious results of Test Everything approaches is that it contributes to cover-you-ass and “blame QA” culture. When one individual or team can make a such a request on another team, it can also shifts the responsibility of this testing. If bugs “get through” and into production, it’s because the testers didn’t test everything as expected. This is despite the earlier problems I’ve noted which is due to “testing everything” in the first place. The original individual or team can’t be blamed, however, because they asked for the app or feature to be tested. While this may all be fair, it sucks for testers.
I think this is the main problem I have with Test Everything: it doesn’t really contribute much to product quality, but can hard test and development teams. It’s a testing anti-pattern.
So what can be done in situations like this?
First, you could take the task at face value. Test everything. Come up with a plan, analyse the app or feature and determine some timelines. After coming up with some estimates for time and resources, provide the individuals who asked for the testing the estimates. Usually this provokes some discussion, particularly if this “simple” task of testing the app could now take weeks or even months. It also helps testers get a grasp on what kind of valuable testing may be done instead of going crazy trying to cover all the bases.
Similarly, you could start a dialogue with the original team. Ask: are there any particularly risky areas that should be tested first? Where are some strong areas that may have been tested mostly by developers? What are some weak areas that need more attention? and so on. Dialogue can be good, and help both teams figure out what’s really needed for the testing task. It also may help break down walls and silos between test teams and others, which is a good direction for development teams and companies to go in. I think this is a good thing.
Also, one way to avoid this whole situation completely is get testers involved in the development process early. This also seems to have a bunch of benefits, such as preventing bugs from happening in the first place instead of simply finding them and fixing them after the fact. It also prevents problems from “handing off” work from one team to another such a the Test Everything mindset. I think this is a win for everyone.
Test Everything can lead to bad situations, but it doesn’t have to.
This past week, among all the holiday cheer and seeing good friends and family, I had an interesting thought: learning to code is important even if you don’t actually ever write code.
Here’s what triggered this epiphany.
I was talking to a university friend I haven’t seen in a while. He did an arts undergrad (history, I think) and it pretty well spoken and well written. This past year, he started a web development diploma program in Ottawa. He has no previous coding experience. He said he’s finding some parts challenging but overall he likes it and it looking forward to getting more into the development side of things.
So here we have an intelligent person who previously had no idea what went into building computer applications suddenly understanding the complexity of simply logging into Reddit. This would also give some appreciation of the processes behind creating and maintaining web applications. It may even provide some appreciation for software quality.
That’s when I started to understand the importance of learning to code. Some have suggested not everyone should learn coding because not everyone needs to code as part of their daily life, but I think this is a bit backwards. Knowing how to code means you know a bit of how apps are created. It means knowing the difference between a 404 error and a 500 error when you try accessing a web page. It means not getting scared when you accidentally look at the page source. And this is just the web; the same principles would apply to desktop or mobile apps.
As computers become more of a regular part of life, it’s important to understand how they work. Learning coding is one way to do this.
Happy 2014 Everyone!