“Exploratory testing” – as the name suggests, is a simultaneous learning, test design, and test execution process. We can say that in this testing test planning, analysis, design, and test execution, are all done together and instantly.
This testing is about exploring the system and encouraging real-time and practical thinking of a tester.
In layman’s terms, exploratory testing involves concurrent test case design and test execution of an application or system under test. The tester will create or write down a test idea to give direction and explore the system while testing to further create critical, practical, and useful tests for the successful testing of an application.
This requires minimal planning. Testers continuously decide on their next step of action. It completely depends upon the tester’s thought process.
For the purpose of identifying some minor flaws that formal testing misses, this testing may occasionally be more advantageous than the formal testing methodology.
Exploratory testing is something that every tester has done at some point in their career, whether consciously or unintentionally.
We all know that practical experience is more effective for learning than memorizing theory.
In a similar vein, a tester will only become more familiar with the program by investigating and learning about all of its features on their own. To guarantee that an application is successfully tested, it is usually beneficial to have a consumer and business perspective during the testing process.
For Example, if you open a shopping website, you have a general idea that this shopping website will let you shop by selecting a product of your choice and then paying for the same.
During this process, you might learn that the website provides you with a virtual human look-alike which helps you in the product selection process. You also found that you can order several products for a home trial or that you can make payment through rewards points of some banks, etc.
As a tester, you not only need to verify whether a system is working as expected but also check if that system is not behaving in a way that is not expected.
Few things to remember while performing this testing:
- Your mission should be clear.
- Make sure to create notes and reports on what you are doing and how a system is behaving, which could be a potential bug.
- Learn, observe, and then come up with new test cases.
Exploratory Testing Examples
Example #1:
A home care service provider website with the following components:
- Login
- Services
- Cart
- Payment
- Order History
- Technician allotment
A general idea to start exploratory testing will be to log in or book a service.
How to Cover Test Cases?
In the above example, the idea is to start with functionality based on your knowledge. As you learn and observe more about the application, you can govern your next set of test cases.
Example# 2:
I was once included in a small project which involved the addition of a new Mutual fund in the application. My task was to test the application to make sure that the new Mutual fund is available for users to buy and check if the associated valuation is correct. I had only 2 days to complete my testing.
Being given the tight deadline and severity of testing, I used the exploratory approach of testing. My goal was to test new features and to find violations of compatibility requirements.
The above-mentioned goal became my charter for this test session.
The following test cases evolved during this testing:
- Testing to make sure that the new Mutual fund has been added to the application.
- New MF is bought successfully.
- The valuation of the new MF is correct.
- Tried to buy a new MF for an existing portfolio.
- Can a new MF be added to all Portfolios?
- Impact of New MF on a valuation of existing.
- So in other test cases were evolved.
I prepared notes and reports during my testing to discuss my observation with the BA and the client involved.
The fundamental strategy of exploratory testing is to have a plan of attack. Start testing with your idea and improvise new test cases based on your knowledge and observation.
Example #3:
Exploratory Testing of the IRCTC Website
=> Click here to download the sample test cases of Exploratory Testing from the IRCTC website.
Testing Approach
- Make use of heuristics to guide testing.
- Test case execution and test case creation go hand in hand.
- Test cases keep on evolving based on tester observation and learning.
- Different testing techniques like Boundary value analysis, equivalence testing, etc. can be applied to ET.
- Session-based ET can be used to make it more structured and focused.
- Testers can branch out their ideas but never stray from your mission.
- ET testing does not use scripts, instead depends on the tester’s intuition, skill, and experience.
Benefits
Benefits of this testing include:
- Promote real-time thinking and helps in uncovering more defects.
- Promote use cases and scenario-based testing.
- Minimal documentation, maximum testing.
- Emphasis is more on learning and broadening the horizon of a tester.
- Avoid duplicate work.
- Useful when you want to audit other tester’s work.
Demerits
Demerits are enlisted below:
- Testing depends on the tester’s experience, skill, and knowledge.
- Require time to learn the application. Testers are more likely to miss if they know less about the application.
- Not appropriate for projects with long execution times.
Session-based Exploratory Testing
It can be quite challenging for testers to describe how much they have tested and on what basis while conducting exploratory testing.
The amount of time and effort required is hard to measure. Nonetheless, we must give team leads and managers KPIs, projections, and progress updates for each project. As they say, “you cannot manage it if you cannot quantify it.”
A time-based method for conducting this testing that aids in tracking and management is session-based testing. A specific time-box testing period free from phone calls, emails, messages, and other distractions is part of it.
Approach:
Testing tasks are divided into sessions.
The following are the components of session-based testing (SBT):
- Mission: The mission shouts out the purpose of the session and in a way provides the focus for the tester. It will also include a session time duration.
- Charter: Includes the scope of the testing. An agenda detailing the goals that need to be completed during the session.
Example of Test Charter for Login Functionality of Home Care Service website:
- Session: Pre-defined time-boxed testing session without any interruption. Each session can have the following duration:
- “Short” (60min)
- “Normal” (90min)
- “Long” (120 min)
- Session report: Include notes and a lightweight report to provide metrics to the leaders and managers. It gives details about the charter session remaining or done session setup time, scenario tested, the testing process, a list of bugs and the issues found, and other information for the metrics.
- Session De-brief: A short meeting or stand-up between the tester and Test Lead/Manager to review the test session findings.
Managers can get hands-on following metrics based on session reports:
- The number of sessions completed and remaining.
- The number of bugs reported.
- Time spent on session setup.
- Time spent on testing.
- Time spent on analyzing issues or problems.
- Features covered.
To summarize the above:
SBT allows for accountability in exploratory testing and offers better management of time spent on testing. It also increases productivity and provides a better grasp of bug detection. It is a great way to provide team leads and managers with metrics to check the project’s progress.
Pair Based Exploratory Testing
Pair Testing is an approach in which two people test the same thing/feature of the application at the same time by sharing a PC. They continuously share their thoughts and ideas. During this testing, one person takes charge of the keyboard whereas the other person suggests test cases and takes notes.
It is always helpful to have good communication between the partners so that both are aware of what is being done and why. A pair in which the strength of the testers mutually complements their weakness is considered a strong grouping.
Such pairing benefits both parties and each can learn something from their partner. It is also a good way to train new resources by pairing them with experienced resources.
Benefits of Pair Testing
- Helps a tester to focus on the task at hand.
- Encourage mutual trust and respect among partners.
- Brainstorming between paired testers usually, lead to more constructive ideas.
- Avoid tunnel vision.
- There is a lesser chance of others interrupting them.
Exploratory Testing Techniques
Tours: It is a simple technique that allows a tester to use his imagination and think of himself as a tourist exploring a city he visits. Here an application to test is the city and the testers are the tourists. It is very difficult to explore the entire city unless you have a lot of time and money in your hand, so a tourist needs to have a plan with a certain goal in mind.
A tourist can take the following tours:
- Guidebook tour – Testing highlighted features of the application. Use user-based scenarios.
- Exploring the history of the city – Test old features of an application.
- Money tour, which means making sure that all the critical features of the customer or client are tested and working successfully.
- Crime spree tour – Enter invalid input and test negative scenarios.
- Back alley tour – Test the least used features of the application.
- Boring tour – Spend minimum time on each screen of the application, fill in minimum fields, and take the shortest path. This will help with the default value and validation testing.
While taking a tour, you always have the choice of taking any route. You can navigate through the software and find a unique path to test the feature.
Below are some tips/tricks that you can use in ET:
- Divide the application into modules and bifurcate modules into different pages. Start your ET from the pages. This will give the right coverage.
- Make a checklist of all the features and put a checkmark when that is covered.
- Start with a basic scenario and then gradually enhance it to add more features to test it.
- Test all the input fields.
- Test for the error message
- Test all the negative scenarios.
- Check the GUI against the standards.
- Check the integration of the application with other external applications.
- Check for complex business logic.
- Try to do the ethical hacking of the application.
Factors that affect ET are as follows:
- The objective of the project
- Testing strategy
- The testing goal of a particular phase
- Available tools and facilities
- Testers role and skills
- Available time
- Management support
- Peer support
- Available resources (study materials, test conditions, etc)
- Clients interest
- Understandability of the product.
- The UI of the application
- The functionality of the application
- Previous test results
- Risks associated with the application
- Previous defects
- Recent changes
- Types of data to use for testing
- Type of user who will be using it
Instead of asking the testers what to run, we leave it to the testers’ judgment to decide what they want to test and how they want to test it.
Difference between Exploratory Testing and Ad-hoc Testing
Do not confuse ET with Ad-hoc testing.
- Ad-hoc testing refers to a process of unscripted, unplanned, and impromptu defect searching whereas exploratory testing is a thoughtful methodology for Ad-hoc testing.
- Ad-hoc testing is a hit-and-trial method of finding a bug whereas ET is not. In the ET approach, a tester learns about the system as they explore and eventually evolve the tests using the acquired knowledge.
- Ad-hoc testing is an unstructured activity whereas ET is somewhat a structured activity.
Exploratory Automated Testing (EAT)
Exploratory Automated Testing is a method that helps a tester streamline bug reporting & reproduction, snapshots gathering, and in preparation for future regression suits. It’s a process that combines automation testing with Exploratory testing.
There are two types of EAT approach:
- Passive EAT
- Active EAT
Passive EAT
A single tester or a pair of testers can conduct passive EAT. This practice often involves installing a tool on the testing resource’s computer that records and captures every action the resource performs.
With the exception of creating the test result based on the recorded session, passive EAT is identical to manually conducted ET in that it does not alter the way the tests are carried out. Later on, these test findings can be utilized to report and reenact recorded actions.
A tester can record test cases and submit defects with the aid of the installed video tool.
It also has a few other benefits like:
- Provides clear steps to reproduce the bugs.
- Reproducing defects is easier even when the defect reporter is not available.
- Do away with the conflicts between the testing and development team when an intermittent bug is reported.
- Helps in performance testing by getting the system response time at a specific point in time.
Here are a few other points to be taken into consideration before Passive EAT:
- It is advised to perform a pilot test before completely adapting the tool for Automated EAT. This is to ensure that the time required for re-designing of the test logs created during the test session is not more than test execution. If so, then the team needs to take a mutual decision on the following:
- If at all test automation is required for the particular project.
- If the tool being used needs to be changed.
- If the performance of the tool being used can be optimized.
- The tool used for performing automated EAT needs to be installed on every testing resource involved in testing. It is also a good idea to involve the developers which can be achieved by either giving the developers VPN or remote access to test machines or by installing the tool in the development environment.
- It is always a good idea to have the GUI object of the application organized in the test tool so that when the time comes to analyze the bug or an issue, the object is recognizable due to a meaningful name.
- It is a great practice to give a meaningful name to the GUI object used in AUT and keep it organized for later use.
Now, let’s move on to the second approach.
Active EAT
It is advisable to perform Active EAT with Pair Testing. In this approach, the keyword-driven testing is used in sync with Session testing. One tester creates the automated test script and the second tester executes the test scripts created by the first tester.
The creation of automation test scripts in this approach takes a different path than in conventional testing. Automated test scripts are made during testing and what has been discovered in the previous tests determines their design.
A closure phase is executed at the end of the testing session. And it should have the following tasks:
- Testers involved should swap roles so that the testing resource who created the test script has a chance to re-execute the scripts to confirm the reliability & robustness of the created suite.
- A brief description along with a few identifying characteristics should be provided for every automated test script.
- A criterion needs to be defined to identify which Automated test scripts can be used for the Regression tests.
Benefits of EAT
- At the start of each session, already created automated test scripts are executed thus enhancing the test coverage every time.
- Better bug reporting and documentation for defect reproduction.
- EAT provides enough evidence and documentation for a stakeholder to see the progress.
Types of Exploratory Testing
Given below are a few types of ET:
1) Freestyle ET:
Exploration of application in ad-hoc style.
In this type of ET, there are no rules, no account for coverage, etc. However, this type of testing is good when you need to familiarize yourself with the application quickly, when you want to verify the work of the other testers, and when you want to investigate a defect or want to do a quick smoke test.
2) Scenario-based ET:
As the name itself suggests, testing done is scenario-based. It starts with real user scenarios, end-to-end scenarios, or test scenarios. After initial testing, testers can inject variation as per their learning and observation.
Scenarios are like a generic guide for what to do during ET. Testers are encouraged to explore multiple possible paths while executing a scenario to ensure all possible paths to feature work. Testers should also ensure to gather as many scenarios as possible from different categories.
3) Strategy based ET:
Known testing techniques such as Boundary value analysis, equivalence technique, and risk-based technique are combined with exploratory testing. An experienced tester or a tester who is familiar with the application is appointed for this type of testing.
Agile Exploratory Testing
Because of its increasing popularity, I’m sure you’ve heard of or read about agile, even if you haven’t worked in one. A team has a few weeks to complete planning, estimating, development, coding, testing, and deployment thanks to the agile methodology’s brief sprints and strict deadlines.
Because exploratory testing places a strong emphasis on producing rapid and practical results, it can be helpful when deadlines are tight. You can begin testing using your expertise and experience after you have a clear understanding of the requirement.
You can create more test cases to verify the application’s operation and find unforeseen flaws after you are comfortable with its features and behavior. You must document everything because it’s a freestyle testing strategy. But , you need to maintain notes and a brief report on what you have tested, bugs and issues found, etc
Merits of Exploratory in Agile
- Proving feedback to the developers as soon as possible.
- A broader variety of defects are uncovered.
- A diverse group of a resources such as a developer, tester, BA, and designers can perform ET as there are no scripted test cases and each brings a different perspective.
- Scouting done in ET helps in exploring new territories and exposing critical bugs.
- In the case of Iterative coding of an application, ET can focus on testing new features while automation does regression and backward compatibility testing.
- In case of unstable requirements, ET can help in testing new requirements within a limited time.
Points to Remember:
1. Requires different skills: The testers performing ET need to have good listening, reading, thinking, and reporting skills. Domain experience is required as there are no scripts and test cases.
2. Sometimes it’s difficult to report a bug: While in an ET flow, we may encounter a defect but we may not be able to reproduce it. This is because we are not tracking the testing steps and we may forget the exact steps to reproduce that issue.
3. Can be done as a recreation activity: I do ET when I want a break from my regular test execution cycle. However many teams have ET as a separate phase of their testing cycle.
4. It can be done for all testing phases: We can implement ET before the beginning of any testing phase. You can perform ET even before the functional testing phase.
5. Rapid feedback: ET requires rapid feedback on the issues and any anomalies encountered.
6. Critical Thinking and diverse ideas: This testing requires critical thinking. Testers should be able to reproduce, review and logically express their ideas. A tester can implement her experience in the various technologies and domains they worked on.
How to Think Beyond Traditional Testing Boundaries in Exploratory Testing
“I really appreciate your concern for the product and making us helpful in understanding end-user perspective. It’s going to be very helpful. Thanks for the good work and keep it up!!!”
This was the last e-mail of an email chain with 21 emails from our client. It was midnight, and our product release was delayed due to a critical bug we found. You might think, what is new in that? It may happen many times. But, this was really different as the critical bug we reported was not a result of any documented test case.
After completing regression testing for the last time that evening, I was just playing with the product. What does that mean? You are free to do what you are not supposed to do. Based on my experience and project knowledge, I had some ideas on how to test the product apart from our typical test repository, called Exploratory Testing.
The exploratory testing done found a critical bug related to a server hang issue while doing something unexpected.
Being a fan of exploratory testing, I love to explore the product in different ways. For me, the definition of software is:
“It should do what it is supposed to do, and it should not do what it is not supposed to do.”
You are an incomplete tester if you restrict your testing to see if goods that should work are functioning. Actually, the completion of recorded regression testing and the updating of results marks the beginning of a tester’s life. It makes a significant difference to view things from many angles and comprehend end-user needs in various contexts. So let’s work together to discover how that change can be created today:
How to look at a product from different perspectives?
#1. Understand customer/end-user
Software testing is all about checking the quality of the product in terms of customer satisfaction. How do you know a customer’s viewpoint? The answer is simple – you have to be the customer. OK, let me make a correction. Being a customer will not be enough.
You need to understand how the customer wants to handle the product. No two customers who bought the same raw materials will prepare the same recipe. Yes, the product we develop/deliver is a raw material for customers’ businesses and they have a different mindset while using it.
As software testers, we need to check the purpose of the product and not the object or aspect of it.
Let me give you some real-life practical examples:
- Scissors were never limited to cutting paper only. Cutting is the purpose and not the paper (object).
- Cell phones were never limited to only calling, but “able to call” has always been the basic purpose.
- Storage boxes are used to store, but the safety of the material stored is as important as storage.
Understanding stakeholders and a wide range of their expectations should be the baseline of exploratory testing.
#2. A mindset
While looking for (let’s say) a job employment ad, do you see that jackpot and in between the pages with the bold font? Most of us do not (believe me, it’s true). Because we have instructed our minds to look for what is useful or to be checked. Anything else is of no use, so the mind denies us recognizing it.
Open your mind, and do not set any expectations when you start exploring a product. Always remember, it’s not OK if the product is doing what it is supposed to do. It is also important that it should not do what it is not supposed to do.
I remember one classic example:
In Linux, the ‘cat’ command is used to check the content of a file and the ‘ls’ command is to check the content of the directory. Working with Linux and being in software testing for five years, I never thought to do cat <dir name> because my mind was set; if I needed dir content, I needed to use ‘ls’.
That worked, but the reverse side of the expectation is that the product was not supposed to behave the way it was not supposed to, which was wrong. One of our customers, who did not know Linux well, did cat <dir name> by mistake and the system crashed. We paid for this mindset.
Always be ready to make mistakes with the software because that is what the end user is going to do. To test the software, you have been trained, but the end-user will not be as trained as you or he/she will not be as much of a technical expert as you. Also, he/she will do anything with the software when they are in trouble.
Think about those scenarios, and provide testing feedback. The life of the software and yours (as a tester) will rock.
#3. Know the competitors
While testing any software application for your client, did you ever attempt to know and understand other software with the same purpose? Did you ever suggest some useful functionality you observed in a competitor’s product? It does not fall in our job description, which is the typical answer. But do you know the benefits of doing it?
Here are a few real-life examples to make you understand the point:
- Don’t you like the designer who not only stitches your dress but also provides you input about matching accessories most?
- Don’t you like the pizza brand that not only makes great pizzas but home delivers on time most?
- Don’t you like the photographer who not only takes good photographs but suggests different kinds of frames for the photoshoot?
Everyone wants to have something extra for what they pay for. Our analysis of competitive software can work the same way for us. The customer always likes to hear valuable suggestions – mainly comparative suggestions to make the product more useful or marketable.
Also, this kind of comparison and analysis of the same range of products makes our analysis more powerful, and eventually, we create a treasure to which we can go back at any moment and find something useful.
Conclusion
Although exploratory testing is not a traditional method, it is nonetheless a very effective one.
It challenges testers to think creatively and inspires them to develop realistic, real-time test scenarios in order to identify flaws. It has an advantage over other testing methods because to its freestyle nature, and it can be carried out anywhere, whether it’s an Agile or Waterfall project or any other project with less documentation needs.
Numerous intangible factors, such as a tester’s talent, ability to write compelling test cases, experience, and capacity to trust their intuition, are crucial to the success of exploratory testing.
It’s critical to keep in mind that ET is an adaptive process as opposed to a essential to maintain a healthy balance between exploratory and scripted or regular testing.
YOU MAY BE INTERESTED IN
The Art of Software Testing: Beyond the Basics
Automation testing course in Pune