Exploratory Testing at Glean

“Much of what I stumbled into by following my curiosity and intuition turned out to be priceless later on.”
— Steve Jobs

At Glean, we test in many different ways which include having unit tests, component tests, end-to-end tests, and more. 

But, we often like to do things the old-fashioned way - using our own curiosity, asking the right questions, and exploring our own features just like our users would. This is often referred to as Exploratory Testing. 

For this, we must ask questions which might be outside of the box, what if I click here? What if I try opening this before I do that step? It’s in this mode of thinking where we can expose flaws that our normal tests might not cover, with the objective to add value to our product experience that satisfies our users' needs.

James Bach sums this up perfectly in the 2003 paper called, “Exploratory Testing Explained” where he says:
“Exploratory testing is simultaneous learning, test design, and test execution”


What can Exploratory Testing help answer?

Known Knowns:
These are things we’re aware of, and understand. For example, how a feature should work, and any known feature requirements. We can easily find bugs related to known knowns, through defining the scope of what we’re building, with test coverage failure..

Known Unknowns:
Things that we’re aware of, but don’t understand. Now this is where questioning becomes important, as we mentioned earlier, exploratory testing is all about being curious. To change things from Known Unknowns to Known Knowns, all we need to do is ask a question. What happens if we click on this button? Once we’re equipped with the answer to this question, it becomes a known known.

Unknown Knowns: 
These are things that we understand, but we are not aware that they might be a possible outcome. 

For example, a bug may exist, and a developer may be aware of this. However, this information is not communicated to another team developing an app. As a result this app crashes, due to this existing bug. The team developing the app understands why the app crashed, but could not have planned for this. 

Unknown knowns are often a result of poor communication, or the right stakeholders not being consulted. Observability can play a big role in helping identify these kinds of risks early on, and plays a big role in incident management. 

Unknown Unknowns:
Now this is where it gets a bit tricky, these are things that we’re neither aware of, nor understand. We cannot make assumptions about them, because we don’t know which questions to ask. As a result, bugs that exist due to this premise, can be very difficult to spot. This is where exploratory testing can really show its value.

How can Exploratory Testing be done?

In the book “Explore It!” Elisabeth Hendrickson likens Exploring Software to “exploring territory” like some of the famous explorers of the past. In order to do this effectively, we can use something called Test Charters. Just as you would plot where you have been, and what you have discovered, you would also have a rough idea of where you want to go, and what you want to discover, if you were visiting a new place. Test Charters can help you do the same whilst carrying out Exploratory Testing.

Test Charters:
A simple test charter for Exploratory Testing would consist of a target - what are we trying to explore? That could be a particular feature on an app for example. We then need our resources - what will we use to explore? This could be the device we are using. Finally, what information are we trying to discover? We may want to explore the functionality of the app and discover any usability issues. 

Why are Test Charters useful?
By using a Test Charter, we can define the scope of the exploration by giving it direction, we can even timebox it to a specific amount of time. However, we don’t limit the scope by overspecifying what we are trying to explore. This opens us up to experimenting, trying things outside of the box, and perhaps uncovering some of those unknown unknowns.

So to recap, the reason why Test Charters can be useful is because they can provide us with scope, objectives, and strategy for our testing. We want to understand how a bit of software behaves, in order to find out how it should work, and what can be improved. Test Charters allow us to extract value from our Exploratory Testing session in order to achieve this.

This allows us to extract value from Exploratory Testing, by being able to explore software with a purpose to understand how it works

What does a good Test Charter look like?
So to recap, we don’t want to make the Test Charter so specific that we lose the ability to think outside the box, and really explore how a particular feature functions. But we also don’t want to make it so broad that we lose the scope of what we are trying to explore and discover. 

Example:
‘Explore the file upload feature (target) with different file formats and sizes (resources) to discover what format is acceptable’ (information)
Timeboxed to 30 mins.

We can now begin to plot each of the actions we took during our exploration, and what the outcome of each activity was. This way, our journey is clearly recorded and we can retrace our steps easily. We can do this by simply using a pen or paper, on a shared interactive board, or even screen record the session.

Where can Test Charters come in especially handy?
We sometimes might have product requirements that are a bit ambiguous, and where the dependencies are not clearly defined, or we might not have any product requirements at all. This is a great indication that this area needs exploration!

It’s a great opportunity to use Exploratory Testing to add new requirements, to uncover risks and problems, and to review suggestions with stakeholders in order to improve the quality of the end product. 

Debriefing

So now we have spoken about Test Charters, and how they can be useful for Exploratory Testing, what do we do with the information we’ve gathered? It’s time to debrief from our session and assess what we have discovered vs our Test Charter. If we have discovered potential issues along our journey we can now begin to disseminate this information with other stakeholders. 

Product Requirements:
Perhaps we have highlighted some areas that are not well defined, and we can have a chat with stakeholders to find out answers to questions. How should this work? Is this the expected result? Why does it do x but not y? 

Bugs:
Test Charters are also great for recording bugs in an effective way. Since we have recorded our actions along the journey so far, this information is extremely valuable in creating effective bug tickets, where the issue can be reproduced easily by the reader.

Risks:
We also may have uncovered potential risks and weaknesses in the features we’re exploring. We may need to consider potential security flaws when exploring a sign-in page, or performance risks when testing with a high volume of requests. 

Conclusion

Exploratory testing requires the tester's creativity, knowledge, and experience to discover defects and issues in the software / feature they’re testing. A tester's intuition is often relied upon as they navigate through this exercise. Test Charters can help with this.

The goal of exploratory testing is to find defects that are difficult to uncover through traditional testing methods, and to gain a deeper understanding of the software and its behaviour. It’s a great way to play around with existing software, see how it reacts, as if you were an actual user looking at it for the first time.

Why not give it a try yourself! Set up a Test Charter, and feel free to ask developers, testers, Product and UX/UI humans to join in, and see what you uncover. You might reveal more about how your features work, than you ever previously thought.

Could you be our next dev?

At Team Glean, we're always on the lookout for talented people to join us.