Software Testing

Software testing is the process of checking an application or parts of it for functionality and errors.

There are two main reasons for software testing:

1. To prove that the developed software does what it was designed to do
2. To locate any bugs before releasing the product to the users.

Testing your software is so vital that it has its own phase in the Software Development Life Cycle (SDLC) as shown in the Waterfall SDLC image below.

However, you will want to start testing as early in the life cycle as is possible and practical. This is where the agile SDLC pays off, being able to test smaller iterations and ensure each new addition to the application works with the last, means the pathway to a successful application is a lot more secure. Waiting until the end to do your testing, as in the Waterfall SDLC means there is a lot more on the line at testing time. Risks include:

• discovering defects which need to be resolved because they block functionality and fixing these could in turn break other (already working) functionality.
• the end product is found not to actually achieve its intended purpose.

At testing time, you will be using a Test environment, as discussed in Topic 1, where the application will be executed using fake data. You might have artificial users such as Lucy Lemon and Ruby Raspberry.

The tricky thing with testing, is that you will pick up a lot of things that are present but aren’t right, but what you won’t pick up are the things that are missing, for example, if the intended functionality was not captured in the acceptance criteria, then testing will not pick this up. So, bear this in mind.

Testing goals

Let’s investigate in a bit more depth the reasons (or goals) for software testing that we mentioned earlier:

1. To prove that the developed software does what it was designed to do.
   
   The way to achieve this goal is by using validation testing – it shows the developers and customers that the application does what it was required to do. So, for custom software test every item in the requirements document that is covered by the upcoming release. And for generic software test every system feature and combination of features that is to be incorporated in the upcoming product release.
   
   In validation testing we expect the application to perform in a certain way and produce certain outputs when tested against defined test cases based on the expected use of the application.
2. To locate any bugs before releasing the product to the users.
   
   The way to achieve this goal is by using defect testing – this is where we hope to identify all the ways in which the behaviour of the software is not what was expected, whether undesirable or simply not as specified. We are looking for data corruption, errors/bugs in calculations or communications with other systems, and system crashes among other things.
   
   In defect testing we are designing our test cases to help identify errors/bugs. This is where we stray from the way we expect the application to be used, we intentionally create abstract test cases. We are thinking about alternate ways users might use the system rather than following exactly what we intended.

Levels of testing

There are four main levels to testing.

Unit testing

At the lowest level, developers perform unit testing. Everything is broken down into the smallest pieces, essentially testing at an object level.

Integration/component testing

Developers/testers now test combinations of units and how they integrate and work together.

System/release testing

Developers/testers test the whole package, the entire application, a certain release of the application.

Acceptance/user testing

Users test the application against their requirements. Ensuring that the customer is happy with the end product.

Verification and Validation

All that we’ve discussed in this introduction so far can be summarised by the above infographic showing the verification steps on the left and the validation steps on the right.

Validation can be expressed as “Are we building the right product?” The product should do what the users require.

and

Verification can be expressed as “Are we building the product, right?” The product needs to meet all specifications.

Let’s use a real-world example to bring this home, imagine you have been asked to buy a blue dress with long sleeves and a high neckline in size 12.

It’s much easier to verify that the specifications are met, than it is to validate that the product is right, because the validating requires that you know the model or have the model to be able to try it on. Therefore, all four levels of testing must be completed, to ensure proper validation of the product.

Example

Verification:

• 2 sleeves?
• long sleeves?
• royal blue?
• correct size?
• high neck?

Validation:

• Is it comfortable?
• does it suit me?
• can I afford it?
• is it good quality?
• will my date approve?

Methods of testing

The two main methods of testing are manual testing and automation testing.

Manual testing

Manual testing is performed by a software tester, physically using all the functions of the software, checking that it works and identifying bugs. However, the cost of having humans do all your testing can be prohibitive, so this is where automation testing comes in.

Automation testing

Automation or automated testing is where the developers or testers write scripts to test the software. It’s a bit like having a robot test your application.

Sosna (2022) explains that “this can save a lot of time and money if done right. But you [must] keep in mind that you cannot automate everything, and that automation will only find defects in what it was designed to test.

Automation will never stray from the path; therefore, it can become obsolete, or if badly designed it will never capture defects.

Automation comes with [a] cost of maintenance also, so it is not always worth automating every possible test case or scenario.”

Listen to Dave explain the benefits of using automated testing in What is Automated Testing? (The Startup Lab, 2019).

Types of testing

We have functional testing and non-functional testing.

Functional testing

Functional testing is where we test that the software works, checking all the expected functions of the application etc. Looking for broken links, unexpected outputs, dead ends, errors, and anomalies. It also includes user acceptance, smoke, unit, integration testing and more.

Non-functional testing

In non-functional testing, we are looking at how the application performs, load-testing, scalability, security, reliability, recoverability, portability, and more.

Stages of testing

Let’s continue our software testing journey by unpacking the three stages of testing:

Development testing

Development testing occurring during the development phase of the SDLC. It is done by the developers to identify bugs and defects early on. Includes unit testing, integration/component testing, and sometimes system testing as well.

Release testing

Release testing is performed by dedicated testers and includes system testing and sometimes integration testing.

User testing

User testing, which is otherwise known as acceptance or user acceptance testing (UAT), is performed on the final system by end users or the customers.

In this section we will discuss all testing activities performed by the development team during the ‘develop’ phase of the SDLC. Specifically,

• Unit testing
  Testing at the individual object or class level, checking the functionality of objects or methods.
• Component/integration testing
  Combining units into functional user flows, to ensure the composite component interface operates as expected. For example, user logs in and lands on home page, user navigates to view their dashboard, user clicks on Enter time to go to timesheet entry page, they enter their time for the day and save their changes.
• System testing
  Units are integrated to form a whole system, and then composite components and user workflows are tested as a part of the whole. We are also checking how these composite components work together with others in the whole system.

Unit testing

In unit testing we are testing the smallest parts of the software code that can be isolated.

Units could be:

• Object functions/methods, which have one or more input/s and provides a single output.
• Classes, with several attributes and methods

Unit testing is a defect testing process.

Unit tests are grouped together to form ‘test suites’. These suites can be rerun whenever your code has been altered to ensure that your new code hasn’t broken another part of the application.

“As soon as you write a function or other block of application code, create unit tests that verify the behaviour of the code in response to standard, boundary, and incorrect cases of input data, and that check any explicit or implicit assumptions made by the code. With test driven development, you create the unit tests before you write the code, so you use the unit tests as both design documentation and functional specifications.” (Mikejo5000, 2022)

We will learn more about test-driven development further on in this section.

Read Write unit tests for C/C++ in Visual Studio (corob-msft, 2022) to get an idea of what it takes to perform unit testing.

Bug (defect) tracking system

A bug tracking system is similar to an issues register, it’s basically a system for recording, prioritising, assigning, and tracking all defects discovered in the application through testing or other means. If you are using a Test Management Tool (TMT) it should have a bug tracking system built in.

The system you choose should have the following features to be effective (softwaretestinghelp, 2022):

• Bug reporting facility – must allow for environment, description, module, severity, screenshots, etc
• Assigning bugs to dev team members
• Bug processing workflow/lifecycle
• History – status, work logs, comments
• Reporting - graphs, charts.

Unit testing test cases

Test cases are simply different scenarios that you would like to test. They should tell you what you expect the system to do in each given scenario. This means that performing test cases should identify where the system doesn’t perform as expected (bugs/defects).

There are two kinds of test cases:

• Those that show the system is working as expected when used in the normal way
• Those that show how the system behaves when users do not use the system in the expected way. Use abnormal inputs to see how the program responds – with appropriate processing/warnings, or by crashing.

What should be in a test case?

A test case should contain at minimum:

• Test case ID
• Purpose
• Preconditions
• Inputs
• Results

Example

Test-driven development

Test-driven development (TDD) is a popular method of development and development testing where you begin with writing your test-cases before you even start coding.

At first, it seems quite backwards, but this style or technique of development can help you produce the cleanest, highest quality code, as it helps you to keep your coding effort focused on only the functionality that is necessary and would be tested.

It is often called “Red, Green, Refactor” because when you first write the test it should fail (‘Red’), because there is no code yet, then you write the minimum code to ensure the test will pass next time, then you run the test again to get a pass (‘Green’), and then you ‘Refactor’ the code, just take a minute or two to ensure the code is as clean and tidy as possible before you move on to the next.

Watch Simon’s Software Development Tutorial - What is test-driven development (TDD)? (LinkedIn Learning, 2013) for a clear and concise explanation of how TDD works.

Choosing Unit Test Cases

Equivalence Partition Technique

Partition testing is a testing method which helps to ensure that you are covering all scenarios when you are testing, and that all variations of a valid or invalid input have the same output. By processing these in groups/categories it helps to cut down on the number of tests that need to be performed.

You begin by splitting up potential user inputs into categories of data, so that you can design tests to cover each category. For example, if your input required an 8-digit ID number, then you will need to determine what would make an input valid and what would make an input invalid.

In this case:

A valid input would be:

• an 8-digit number.

An invalid input would be:

• less than or equal to 7 digits (this should include negative numbers as well, which is often overlooked)
• bigger than or equal to 9 digits.

Once you have determined these possible categories, then you can design/choose test cases to cover each category, you don’t need to create one for each possible entry, for example, 1-digit input, 2-digit input, 3-digit input and so on.

Note that your tests will need to ensure that the system produces an error for both categories of invalid inputs – these error messages could be different for each category though. And if there were more than one category of valid inputs they would also need to trigger equivalent outputs. That is where the ‘equivalence’ part comes in.

Read Equivalence Partitioning Technique (javaTpoint, n.d.) for a more thorough explanation with examples and advantages and disadvantages of using this method for choosing your test cases.

Equivalence Partition Technique is a part of the Black box testing method, which you will learn more about in the following activity.

White box testing and black box testing – blog activity

For this activity, do your own research to learn about the difference between white box testing and black box testing, and then write a blog explaining them both and their differences. Please give examples and reference your sources using APA 7 style.

General testing guidelines

• Aim to create test cases that:
  • would cause the system to produce all errors and warnings.
  • will make the input buffers overflow.
  • will create invalid outputs.
  • will make results/outputs too big or too small.
• Don’t just run an input or series of inputs only once, run them multiple times, things change.
• Think of different ways you can force errors or failures.
• Don’t forget to include “happy path” scenarios, where the output should be as expected.

What is a software release and release testing?

A software release is the distribution of the final version or the newest version of a software application (Hanna, 2022).

Each ‘release’ to the customer needs thorough testing to ensure it fully meets the requirements of the customer.

Release testing is conducted to convince the customer the release is good for use in the following areas:

• Bug-free
• Meets agreed customer specifications and requirements in terms of:
  • Functionality
  • Performance
  • Reliability
  • Security
• System operates as expected during normal use.

Regression testing is a part of release testing and is done before each release. It may also be done each time a new build is received from the developer with added functionality or bug fixes, this way errors can be found and dealt with sooner. Sosna (2022) says the point of regression testing is to “cover all existing functionality and ensure that the new functionality hasn’t negatively affected anything” – ensuring the latest changes to the code hasn’t broken any other part of the system. Regression testing usually involves re-running test suites that have been created for existing functionality, checking all the original functionality is still good to go.

As time progresses however, those test suites may get bigger and bigger, and potentially outdated/redundant, as new functionality is added to the application – making these tests take longer and longer to run. In this situation testers may need to select which test cases to leave out to reduce the time it takes to perform regression testing – this can be quite the challenge as it increases the risk of not finding defects!

Different customers and different types of applications will have different needs, so release testing will look different from application to application or from situation to situation.

How is release testing different from system testing?

System testing is more on the white-box testing end of the scale – checking the system against all specifications, testing the coding, checking that the ‘product is right’.

Release testing is at the black-box testing end of the scale – testing function and performance and whether it is the ‘right product’. Does it do what it was intended to do, without breaking?

One of the best methods for release testing is using requirements-based testing. Let’s unpack that now.

Requirements-based testing

Designing and creating test cases, conditions, and data specifically to evaluate whether the release meets each requirement, one-by-one. You will do this by writing up the minimum number of test cases necessary to ensure each requirement is met within the code.

In the process of requirements-based testing you will also validate that the customer requirements are suitable and correct, consistent and not lacking.

Below is an example based on one in Ian Sommerville’s book – Software Engineering (Sommerville, 2016).

Example

Imagine you are developing a GP’s patient management system. Using this system, a GP can look up the health history and records of their patients and assign scripts and charges to them among other things.

Requirement:

If a patient has a known allergy to a medicine, then, when the GP tries to prescribe that medicine to the patient, the system must alert the GP on the screen.

If the GP wants to prescribe it anyway, they will have to input a valid reason first.

Test cases based on the requirement:

• Set up a patient record with no known allergies. Prescribe medication that is known to contain an allergen. Check that a warning message is not issued by the system.
• Set up a patient record with a known allergy. Prescribe the medication that the patient is allergic to, and check that the warning is issued by the system.
• Set up a patient record with allergies to two or more drugs. Prescribe both drugs separately and check that the correct warning for each drug is issued.
• Prescribe two drugs that the patient is allergic to. Check that two warnings are correctly issued.
• Prescribe a drug that issues a warning and overrule that warning. Check that the system requires the user to provide information explaining why the warning was overruled.

As you can see, the list covers valid and invalid responses to test that the system behaves appropriately no matter what the user does.

Similar to test-driven development, we are talking about starting this part of the development process much earlier in the SDLC to ensure the impact of requirements-related errors are minimised.

Read Requirements Based Testing – Process Overview (Bender RBT Inc., 2009) and then answer the questions in the quiz that follows.

When it comes time to perform these requirements-based tests, the tester acts in place of the future users to pre-empt their actions in order to identify all the system defects before it ever reaches the hands of the users.

To complete this section, watch A day in the life of a QA tester at a software development company (Silverchip Ltd, 2022) to get a feel for the day to day work of a software tester.

What is user testing?

This is when the users or customers get involved with testing. They will be providing input and advice on the system testing as well as testing the system. User testing is the final stage in software testing as it should eventuate in the application being accepted by the customer for release into production. It is therefore otherwise known as ‘acceptance testing’.

Why do we do user testing?

User testing is a vital part of the software testing process and cannot be skipped, even if you have performed extensive system and release testing. That is because having the system installed in the user’s environment could have major impacts on its performance and usability.

Nielsen (2018) explains in his video, which you will watch a bit later, that one of the main reasons for user testing is because you (the developer or tester) are not the user, and you know too much about technology in general and for sure you know too much about your own project, and you simply cannot like wipe your brain and pretend that you don't know the things that you do know. So, you can never have an experience of coming for the very first time to a website and how does that feel when you haven't used it before.”

The main goal of course is to validate your application and confirm that it meets the customer’s needs and does what it was designed and expected to do.

Types of user testing

There are four decisions you’ll need to make to settle on the type of user testing you will use.

• Moderated or unmoderated
  Will the developer be running the session?
• In person or remote?
• Explorative (qualitative) or Guided (quantitative)
  
  Explorative is a more open-ended style, where you really want to hear the user’s unguided observations, emotions, thoughts, and experience. With custom software, you want real users to try to do what they will be doing with system when it is released, not simply what you want them to do.
  This style will provide mostly qualitative results.
  
  Guided is when you provide all the test cases and expect them to perform specific tasks. This style will provide mostly quantitative results, metrics such as task success and time on task, and results that can be measured.

Stages of user testing

This series of stages describes a typical user testing process for custom software.

Alpha testing

Customer/users work with the developer at the developer’s location to test the software. Sometimes also used to describe the first/earliest customer/user testing, used to pick up the more glaring mistakes/errors.

Beta testing

A release of the system is made available for the customer/users to test and experiment with in their own time, to raise issues and log defects with the developers. Sometimes also used to describe more thorough customer/user testing where smaller bugs are expected to be found.

Acceptance testing

A release of the system is made available for the customers/users test and determine whether it is ready to be 1) accepted and 2) deployed, in the customer environment.

Acceptance testing can be split into three parts:

• Contract acceptance testing
• Regulation acceptance testing
• Operational acceptance testing

How do you find participants?

This is going to be different depending on whether you are developing custom or generic software.

Custom software

For custom software, it is easier to find users to test your system - you should be able to use the future users/end-users.

Generic software

For generic software, you will be looking to find people who represent your future users. You will need to create user personas that describe the type of people who are most likely to use your system in the future.

The fastest way to find people who represent your user personas is to use social media platforms like LinkedIn or Facebook. Log in and find groups that represent your user personas, join the groups, and then post in the group about your study and your need for users to help with testing. To ensure you get the best usable responses:

• Describe the tests - group or alone, observed or not? Will there be an interview? Any recordings required?
• How much time and when?
• Can they do it from home, or do they have to go somewhere?
• What reward are you offering as payment for participation?

User testing tips

Before you get started, here are a few tips from those who have gone before:

• Be prepared
  • Plan how you will capture all this information. Written forms, or will the users have access to log errors straight into your defect tracking system?
  • Some user testing is done with video or voice recordings, where the user is asked to speak their thoughts out loud so these can be captured. This is very helpful for finding pain points, points of confusion, or for example when users experience delays in their workflows because something is not located intuitively.
• Test integration with the customer’s other systems
• Test at different times of the day, especially peak times
• Have users from all different organisation levels and/or roles participate

Although this video was designed around user experience testing, the same principles apply, and Jakob has some great tips and reasons to support the user testing process. Watch Jakob discuss User Testing: Why & How (NNgroup, 2018).

User testing blog activity

To complete this topic, you will write another blog post.

For this activity, you have been hired to write a mobile communication app for a school.

• Write up three personas, and include how they will each use the system:
  • A teacher
  • A parent
  • A student
• Write 4 requirements
• Write up valid and invalid categories for these requirements using the equivalence partitioning technique (EPT)
• Write at least two test cases for each requirement, covering each of the EPT categories.