Continuing on an earlier theme, here’s a situation I’ve been thinking of recently.
Suppose you’re testing a web app that has some site-wide setting like a date format. For example, a field in the app may allow the user to change a date format from “01/02/2014” to “Jan 2, 2014” across the app. Test cases here are fairly straightforward; login to the app, change the site-wide setting, navigate to a page with fields that employ this setting, and verify the format changes accordingly. Being a good test developer, you think to yourself, “This could be a good place to use Selenium! Simple and quick.”. It’s all good, right?
Not so fast.
Automating testing of site-wide settings might seem like an easy win but there’s some subtle problems that arise.
The main one I’ve seen is interference with other tests. In the case above, if your automated tests only tested these site-wide settings, there’s really no problem. However, once you add any other sort of tests to the mix, you might run into problems. Tests may begin to fail unexpectedly because checks fail due to formatting. For example, consider checking a post on a blogging app appears as expected for a given date. Posting might functionally work as expected, but checks might fail when checking the posted date if the formats don’t match up and are not reset properly. The expected date may be correct but its format might not be.
This problem can be compounded further if test are run concurrently. If one test changes a site-wide setting at some point during a run, other tests could be affected. At this point, debugging might get pretty tricky and test results may be skewed to false failures.
One way around these problems is to set formatting to a known, fixed format before every non-formatting test. The downsides are additional test setup and potentially longer test execution time, not to mention that you are now implicitly assuming that the site-wide formatting works correctly in some cases. Another approach would be to isolate the site formatting tests so that they do not interfere with other tests. Isolating these tests may limit their utility in addition to being difficult to do effectively. Lastly, you could convert all fields that are formatted to some common format, removing formatting from the picture entirely. This might work but could also mask bugs and may require quite a bit of upkeep.
Not so simple and quick.
I’m going to outright say that Selenium isn’t the right tool for checking site-wide formatting, but I would consider it with caution. As test suites grow, such issues can cause headaches for the maintainers, especially when tests seem to randomly fail. Things may make sense in your context, so just be aware of what can go wrong.
Automated testing has a lot of benefits, and can reveal a lot about a piece of software. One of the more interesting aspects of automation is when such tests find issues that are totally unexpected.
Here’s one such story.
At my company, we have a team of test developers who write and maintain WebDriver-based functional tests, which I’m a part of. These tests focus on basic functional aspects of the web app under test, such as logging in and straightforward business logic. There’s a fair number of tests that cover different areas of the app which run both nightly and on-demand. Results are reported after tests are run, providing information on the overall functionality.
In other words, these tests were designed to look for issues in the UI and business logic aspects of the app.
However, one day we found it revealed something else.
One morning we came in to find all of our tests skipped. All of them. Puzzled, we checked the test servers with the recent build of the app under test. They were up but not connecting to the back-end correctly. It turns out the databases for these test servers were taken down. Why? Because they blew up. There was a surge of data sent into them, causing them to crash. The underlying cause of the crashes was a bug in the client-side that caused large stacktraces to be recorded to the database with every login attempt, valid or not. Essentially, the app was experiencing a sort of denial-of-service attack.
Definitely not expected.
The key to this story is that our WebDriver tests make use of (SOAP) server APIs and parallelization. Using APIs allows for quick and easy data generation, while parallelization helps speed up test execution time. These are both good approaches that I recommend considering for UI test suites. In this case, however, they combined together to crease a situation where there are lots of login attempts happening in a short period of time, resulting in a database crash. This was not something our tests were designed to do, nor was it something we considered when we started parallelizing.
The bug was obviously identified, and eventually fixed.
For me, this is one of the most fascinating parts of working in test development. Whenever software it built, developers should be wary of side effects. In a production development setting, side-effects and unexpected behaviour can be quite undesirable; for automated test development, side effects can be quite beneficial. I would’ve never considered looking for such issues despite the fact that automation would be a good tool for a scenario like this.
I’ve always felt that the process of automation produces the most benefits for testing while actually automating an app, but this case also showed how automation can lead to happy accidents as well.
This may be an unpopular opinion, but I think it’s the start of a good conversation.
I currently work as a test developer. As a test developer, I am a developer. I am a not tester.
Let me explain.
Test developers write test code. This includes (but is not limited to): writing automated tests, developing test frameworks, building automated tools for production developers and/or testers, recording and reporting automated test results and working on Continuous Integration-related projects. I’ve seen at least a little bit of each of these things in my work. These are activities that are beneficial for building good software. I’d also say these activities help improve the overall quality of a piece of software if done appropriately.
But it’s not testing, at least in the way I think of what testing is.
I don’t sit down and perform exploratory testing of an app, nor do I thoroughly investigate the app under test. That’s not to say that I shouldn’t do these things, especially since testing is an activity should be done by everyone on a team. I do report bugs and work with automating test cases, which could be considered testing activities, so in that sense I’m a tester at least some of the time.
This mirrors my opinion on what test development is (and isn’t), which has been bugging me for the past while. Test development is development, but it isn’t testing, or at least not completely. I think this is the start of sorting out the value and benefit of automation in software development.
Like Rodney Dangerfield, I feel like source control gets no respect.
Good source control systems (Git, Subversion, Mercurial and so on) are widely used in software development. These systems are generally used, as the name implies, to keep track of changes made to the source by different team members. That allows developers to see the progress of new changes and potentially roll back bad changes to avoid problems.
Developers widely understand source control as this and only this. It is a tool but a simple, limited one. Some feel that source control can get in the way of producing good code by breaking the “flow”. Others feel that source control is something only professional developers on professional projects need to use.
Modern source control can do so, so much more than that.
I’ve been working with Git for about a year now, and I think it’s an excellent tool for any developer. In fact, I’d say it’s an excellent tool for anyone working with text of any sort. Source control can do handy things if you learn how to use it. Source control systems don’t just save your work; they make make it easier and more efficient to produce good code. For some reason, some programmers just don’t feel this way. I’m going to give some examples of what source control like Git can do for you.
Conduct small, one-off experiments: Sometimes looking at code, you might want to try out something small. Maybe it’s not even directly relevant to what you’re working on or just a nagging question about the language you’re working with. What happens if I make x a float instead of an int? What if I delete this line of code? What does that constructor actually do? With good source control, you can easily undertake these experiments. Just make the change, figure out whatever you want to know, and repeat. Once you’re done, you may end up with something irrelevant to your work. You’ll avoid losing your place while coding or introducing annoying errors with superfluous code.
Conduct large, complex experiments: Suppose you want to re-write a large chunk of code but are not sure how it will turn out. Or you want to start a “test-first” approach to writing code for an existing project. These experiments could take days or weeks, affecting significant parts of your code base. How do you do this while not breaking current work? Again, there are good approaches to doing this. Workflows like feature branching allows parallel development between the mainline of a project and an experimental subproject. Feature branches can also be used to share such experiments among team members for commenting or additional work, again without disrupting the mainline of development.
Portability: This one seems basic but can be quite helpful. Having a repository in a central location makes getting source code a breeze. Setting up a machine for a new developer? How about working with your code on an arbitrary test machine? Getting code is as easy as a pull command.
Basic Continuous Integration: CI is all the rage these days, for good reason. CI helps developers and teams build and release products faster and with higher confidence. But instead of diving into some hardcore CI setup, some source control systems have similar functionality in a slightly stripped down form. In Git and Subversion, you can add hooks to perform particular actions like post-committing or pre-pushing. For example, you could put your commit messages through a formatter or trigger automated tests. Hooks can be as simple or involved as you want, making a good firs step toward a larger scale CI system.
Personal Projects: With distributed source control systems like Git or Mercurial, you don’t need to connect to a server at all. You can set up a new repo in a single command. Because of this, you can use “professional” source control for your personal projects, no matter how small. You’ll have all of the above benefits even when you’re hacking around in your free time.
There are more benefits, obviously, to using source control well, but these are the ones that I’ve seen in practice. Honestly, the first two points alone are worth the price of admission in my opinion.
Don’t just passively push and pull your changes. Give your source control system the respect is deserves.
A quick thought.
Should everyone learn to code? I don’t see why not, so yes.
Should everyone learn to develop software? No.
What’s the difference, you ask? Writing code is exactly that: opening up a shell or text editor or IDE and just hacking away. Some folks would be interested in doing this, for hobby or otherwise, so let them at it.
Software development involves that, but there’s a lot more to it.
Developing software means taking into account critical aspects of a program or app. It means allowing customers or other stakeholders into the process. It means actually caring about details that appear small to some, but aren’t to others. It means providing real value, and achieving actual goals not always set by the programmer. It’s about concern for quality in the context of what you’re building.
In short, there’s some additional purpose in software development. In coding, there isn’t, necessarily.
As I’ve mentioned before, coding is just one part of the software development process. It’s all too easy to conflate the two.
Right now, I write test code for a living. I mostly write UI tests using Selenium. I rather enjoy it, and try to improve my abilities, a little at a time.
So when I read an article about writing good unit tests, it got me thinking about the test code that I write. Specifically, I thought about UI tests versus unit tests. Unit tests should be short and easy to write. Why can’t UI tests be the same?
In the link above, Bob Martin describes writing unit tests that have exactly three lines of code. He further goes on to argue that all unit tests should be exactly three lines of code. An example he gives, for testing a stack object, looks something like this:
Stack stack = makeEmptyStack();
There’s obviously some benefits to writing tests in this way. Each line has a specific meaning (initialize/instantiate, perform the action, check), making tests more easily understandable. Tests become simpler to write because they’re more structured. Plus the code is easier to maintain, which is a big win for making tests usable. These kind of considerations turn unit tests into something closer to code specs. In short, there’s lots to like.
So why not write UI tests in the same way? Instead of code like this
PageObject testPage = new AccountsPage();
take a look at code like this
PageObject testPage = new AccountsPage();
assert testPage.isAccountPresent(“101”, “abc”);
Code is cleaner, intentions are clear, and it’s easier to extend tests to be data-driven. Imagine replacing the hard-coded strings above with a AccountData object. These can then be pre-populated and tests run based on this data. This functionality is built-in to some test frameworks like TestNG. Plus we get the benefits mentioned above.
Code like this also gets us thinking of UI tests in terms of unit tests. In other words, instead of just stringing together UI actions, tests can represent single cohesive units of functionality within the app. This could assist in nailing down acceptance checks when building an application, in addition to helping translate acceptance criteria into automated tests and vice versa.
Of course, there are some hurdles to writing code like this (hat tip here and here). Unlike creating and destroying objects, starting and stopping an app can be costly, and in turn opening and/or closing the app after every individual test may not be desirable. This can also increase overall test execution time, making long tests yet longer. Of course, there are ways around this, usually making judicious use of setup and teardown methods. In our case, where I work we have methods that open and close a browser once per set of tests in the case of web UI tests instead of after every tests. This particularly helps when tests are run on weaker VMs with limited processing. But with other aspects of UI tests, it’s all about making it work behind the scenes in the page object layer.
Above all, in my opinion, thinking about UI tests like unit tests put both on the same level of importance (or at least helps). Unit tests can assist building classes and methods, and in a similar way, UI tests can help build a final application, at least from the UI perspective. If nothing else, these ideas might make writing and maintaining UI tests a little more helpful and less fragile.
When it comes to testing, I think it’s important to have a strategy. That is, come up with a plan of action of what and how to test, preferably with input from all individuals involved in the project.
"Test Everything" is such a strategy. And it’s a bad one.
I’ve seen this strategy executed in practice, usually at the request of a product manager. I’ve also heard similar stories from other testers. The results I’ve seen have been less than optimal.
There’s a few reasons why I think this.
Test Everything is intellectually lazy. It requires no thinking, since it can apply to any product or feature in any team on any project. It also completely neglects team priorities and dynamics. But because it’s so general, it totally neglects context. Risk analysis is completely neglected; every aspect of the app is seen as equally risky. Is a given bug or issue that’s raised of high or low importance? We don’t know, we’re just to test everything. Are there any regressions or outstanding issues that we should be aware of? We don’t know, we’re just to test everything. Yes, people can be difficult and unpredictable but you can’t use that as a excuse to be lazy with test planning and strategy. It robs testers of opportunities to do some interesting and valuable testing.
Test Everything also difficult for testers cognitively. Since there’s not many details to go on, it can be daunting to start. Even simple apps or features can generate lots of possible directions for testing. Without any concepts of risky areas or priorities, it may take additional work to determine starting points or goals. For novice testers, this can be highly intimidating, even stressful. Again, not a great situation for helping testers do their jobs well.
One of the more insidious results of Test Everything approaches is that it contributes to cover-you-ass and “blame QA” culture. When one individual or team can make a such a request on another team, it can also shifts the responsibility of this testing. If bugs “get through” and into production, it’s because the testers didn’t test everything as expected. This is despite the earlier problems I’ve noted which is due to “testing everything” in the first place. The original individual or team can’t be blamed, however, because they asked for the app or feature to be tested. While this may all be fair, it sucks for testers.
I think this is the main problem I have with Test Everything: it doesn’t really contribute much to product quality, but can hard test and development teams. It’s a testing anti-pattern.
So what can be done in situations like this?
First, you could take the task at face value. Test everything. Come up with a plan, analyse the app or feature and determine some timelines. After coming up with some estimates for time and resources, provide the individuals who asked for the testing the estimates. Usually this provokes some discussion, particularly if this “simple” task of testing the app could now take weeks or even months. It also helps testers get a grasp on what kind of valuable testing may be done instead of going crazy trying to cover all the bases.
Similarly, you could start a dialogue with the original team. Ask: are there any particularly risky areas that should be tested first? Where are some strong areas that may have been tested mostly by developers? What are some weak areas that need more attention? and so on. Dialogue can be good, and help both teams figure out what’s really needed for the testing task. It also may help break down walls and silos between test teams and others, which is a good direction for development teams and companies to go in. I think this is a good thing.
Also, one way to avoid this whole situation completely is get testers involved in the development process early. This also seems to have a bunch of benefits, such as preventing bugs from happening in the first place instead of simply finding them and fixing them after the fact. It also prevents problems from “handing off” work from one team to another such a the Test Everything mindset. I think this is a win for everyone.
Test Everything can lead to bad situations, but it doesn’t have to.
This past week, among all the holiday cheer and seeing good friends and family, I had an interesting thought: learning to code is important even if you don’t actually ever write code.
Here’s what triggered this epiphany.
I was talking to a university friend I haven’t seen in a while. He did an arts undergrad (history, I think) and it pretty well spoken and well written. This past year, he started a web development diploma program in Ottawa. He has no previous coding experience. He said he’s finding some parts challenging but overall he likes it and it looking forward to getting more into the development side of things.
So here we have an intelligent person who previously had no idea what went into building computer applications suddenly understanding the complexity of simply logging into Reddit. This would also give some appreciation of the processes behind creating and maintaining web applications. It may even provide some appreciation for software quality.
That’s when I started to understand the importance of learning to code. Some have suggested not everyone should learn coding because not everyone needs to code as part of their daily life, but I think this is a bit backwards. Knowing how to code means you know a bit of how apps are created. It means knowing the difference between a 404 error and a 500 error when you try accessing a web page. It means not getting scared when you accidentally look at the page source. And this is just the web; the same principles would apply to desktop or mobile apps.
As computers become more of a regular part of life, it’s important to understand how they work. Learning coding is one way to do this.
Happy 2014 Everyone!
You’re in walking around Toronto. You meet someone and he says “Hi, I’m John Smith”. What do you know about him?
Probably not too much. Perhaps name and probable gender. Not too helpful.
Now suppose you’re walking around Toronto and you meet someone who says “Hi, I’m John Smith, Chartered Accountant”. Now what you can say about him?
You now can assume that he’s met all the requirements of being a chartered accountant.
This is helpful. It gives you some direction for more questions. You could ask about his experience with auditing, if he works with one of the Big Four, whether he’s assisted with any large scale corporate embezzlement, and so on.
In other words, you’ve raised the bar a bit higher for what you can assume about this person. Instead of asking just basic questions, you can start to ask more advanced, interesting questions.
For me, this is the way to think about automated test suites of unit, integration and UI tests.
Here’s a possibly radical opinion about software development: programming is usually not the bottleneck. In fact, sometimes programming the easy part of the software development process. The hard(est) part is usually somewhere else.
Somewhere I recently noticed that someone wrote that Python is a bit boring. This may have come out as sort of an insult to the language but to me, this is a strength. It shows that Python is a well-made tool that has been proven time and again. Really this is a good thing for a language because it means it’s ready for the prime-time. And this means that the language itself can’t be a roadblock. It’s boring like a monkey wrench is boring.
Furthermore, the past decade or so has been a boon for coding in general. Code is everywhere. People blog about coding practises and algorithms all over the place. Code hosting services like Github make fully working code samples a few commands away. Stack Overflow provides specific answers answers context. And of course, there’s plenty of readily available utilities available for download and free as in beer. Programming may still take skill, but you can get a surprising distance just on online sources. At the very least, it’s possible to get something thrown together that kind of works.
But if coding isn’t the bottleneck in software development, what is?
I’ve seen some discussion this week about the role of testing in the software development process. In this tweet, Alan Page sums things up nicely.
Basically, testing is more of an activity than a phase or checkbox. Testing should take into account the fluid nature of software projects and provide what is actually needed instead of ideal of what should be needed. This could mean getting rid of testing/QA departments, or other radical ideas.
But then I thought, “This might be hard in some places because people won’t like such changes”.
Then it hit me: software problems are people problems. That’s the bottleneck. Deciding what your overall coding style is like is fairly straightforward. Communicating between teams with different perspectives and biases isn’t.
This is where the bottleneck is, and where teams should focus their attention.