The QPath Blog
QTester APP beta 1: test manager for quantum software
Quantum gate circuits
One of the priority objectives of Software Engineering is the development of quality software products, and in the case of quantum computing, the emerging Quantum Software Engineering must also ensure software quality in all its forms.
This involves evaluating the correctness and performance of quantum algorithms from a software perspective, identifying the bugs, to ensure that the software behaves as expected.
Quantum systems are inherently different from classical systems and require different testing approaches. This fact, combined with the verification challenges presented by the inherent characteristics of quantum computers (such as superposition or entanglement), creates the need for new systematic methods for verifying the quality of quantum programs. Quantum software testing is, like everything related to quantum computing, a relevant new field of study, full of challenges to address and solve.
Motivated by the novelty of the subject and the importance of quantum software being professionally tested before being released to industry, despite the enormous difficulties that this implied given the almost inexistent scientific and technical experience on quantum software testing in 2019, based on a conceptual design by QST we took on the challenge of addressing research on quantum software testing. Since then, QST and a group of aQuantum scientific researchers, of the Alarcos Research Group, has been working on these tasks with the aim of determining the feasibility of developing qtestware (a set of test case engines and generators) that will ensure the proper functioning of the software in different quantum environments[1].
The research contemplates, according to the best practices of Software Engineering, the feasibility study of quantum software testing in the different technological approaches and, in each one of them, the investigation of different techniques and testing procedures to be able to evaluate and select those that provide the best results in each technological approach.
Among the tasks of this research, focused on the testing of quantum gates circuits, the following can be highlighted:
· Search for typical errors in quantum circuits.
· Definition of the most appropriate techniques for testing quantum gates circuits.
· Design of quantum mutation operators.
· Design of quantum mutant generation engine.
· Design of test cases.
· Execution engine.
· UML profile construction for quantum testing.
Once the first results of the scientific research were consolidated, the aQuantum team tackled the no less complex task of extending these results to the field of applied research, which began the analysis, design and development of the most appropriate tools to carry out the testing of the quantum software which, after overcoming more than one technical hurdle, allows us to present and share the use of the first beta version of QTester (Illustration 1), an APP of the QuantumPath® platform that provides different options for testing circuits developed with the QPath® platform for the technological approach of quantum gates.
Illustration 1: QTester Tool, QPath® APP
Introduction to quantum circuit testing with QTester
The QTester APP tool is part of the QuantumPath® platform suite of applications. In the commercial versions of QTester APP, the Administrator of each QPath® organization will be the one who will give permission to the members of their testing teams to access the APP, but to start working with beta 1 version of the QTester APP you must have a QuantumPath® user account. In this beta 1 of the application, access is supported for all types of subscription, including “Free Developer”. If you do not have a QuantumPath® account, you must create a Free Developer user account.
To simplify access to QTester APP beta 1, this can be done from the QPath® web application itself, in the QAPPS > QTester section.
Illustration 2: Access to QTester APP from QuantumPath®
If you already have a QuantumPath® user account, you can also access directly through the following link with your QPath® credentials:
· https://core.quantumpath.app/QAPPSTESTING
On the login page of the application, the user will simply have to provide the same credentials with which they access QuantumPath ®.
Illustration 3: Login page
Once authenticated, the application allows the user to work on different testing projects, quantum circuit containers on which a testing process will be launched.
To create a new project, the user just has to click on the “New Project” button and provide a name and description for it.
Illustration 4: New testing project form
Once inside a test project, the user can add a Circuit or System Under Test (SUT) from the QuantumPath® circuit catalogue to which the user has access.
Illustration 5: Form used to add a SUT to a testing project
On the selected circuit, which is now linked to the testing project, QTester APP allows you to perform a testing process based on test cases. This technique, following the philosophy of classical software testing, consists in the definition of test use cases, with an input and an expected output to the circuit to be tested. By carrying out a high number of executions of each use case, the system allows obtaining a sample of results large enough to eliminate the intrinsic indeterminism of quantum circuits and to be able to validate, based on percentages configured by the tester, the validity or not of every use case.
In addition, the tool allows to evaluate the quality of a set of test cases, through the technique of testing based on mutants. This technique consists of generating “mutant” circuits, applying small intentional changes to the original circuit, and executing test cases on each mutant, with the aim of evaluating the effectiveness of the test suite.
Illustration 6: The test type selection form
Testing based on test use cases
This type of testing consists of the definition of test cases, where the user provides a set of inputs and a set of expected outputs. The system executes each test case in isolation in a simulator or quantum device and analyzes the results, comparing them with the expected output to establish the success or failure of the test case.
To launch a test of this type, the user must first select if s/he wants to launch one from the data of a previous test, or if s/he wants to initialize it from scratch.
Illustration 7: Testing based on test cases – Initialization
Next, the user must indicate the use cases to be executed by the system. Each use case consists of:
· Input: An array of values 0 or 1 that represents the initialization value of each qubit of the circuit.
· Expected output: An array of values 0 or 1 that represents the expected output value of each qubit of the circuit. When the output value of some qubit is indifferent, it can be marked with an ‘X’ value.
· Maximum and minimum probability: Probability range (%) accepted in which the expected result should appear, calculated on the set of 1000 executions of the circuit that the system will carry out.
Illustration 8: Testing based on test cases – Test cases definition
Finally, the test case execution platform is selected. Due to the high execution load that the test can entail, it is recommended to select a quantum simulator.
Illustration 9: Testing based on test cases – Selection of the execution platform
At the end of the execution, the system will display the result of each test case, allowing the generation of a PDF report with the complete test result.
Illustration 10: Testing based on test cases – Execution resul
Illustration 11: Testing based on test cases – Test case result
Mutant-based testing
Software mutation testing is a technique that aims to improve the quality of a test suite and detect new errors that the test cases have not been able to identify.
Illustration 12: Overview of the mutation-based quantum software testing process [2]
As shown in the previous ilustration, the mutation-based testing process involves the following steps:
· Generation of a set of copies of the circuit or system under test (SUT), called “mutants. These mutants are created by introducing bugs or errors into the original code using mutation operators that make changes to certain gates and qubits.
· Execution of the set of test cases against each mutant in a quantum device through the QuantumPath® platform and its qSOA® API.
· Analysis of the mutants. Once each test case has been executed against the set of mutants, the system collects the result and analyses it, generating what is known as the “killing matrix” in which each mutant is labelled with one of the following states:
o Killed: a mutant is considered “dead” when at least one of the test cases executed against it fails.
o Alive: a mutant is considered “alive” when it passes all the test cases executed against it.
Calculating the percentage of mutants that the test suite is capable of “killing”, what is known as “mutation coverage” is obtained, which is a metric used to measure the quality of the set of test cases: high mutation coverage indicates that the test suite is effective in detecting flaws or bugs, while low mutation coverage suggests that the test suite may need to be improved. Analysis of mutants that have not been killed by any test cases can further help to identify new bugs that have not been previously detected by the original set of test cases.
This testing process can be executed iteratively, refining and improving the set of test cases until the mutation coverage is as close as possible to 100%, thus achieving a sufficiently robust and reliable test suite, capable of detecting most of the mutations. errors that the original circuit might contain.
From the QTester App, on a test in which all the defined test cases have passed, the user can check the quality of the test suite by starting the mutation testing process.
Illustration 13: Mutant-based testing – Initialization
Next, the tester must select the set of mutation operators that s/he considers appropriate to test the circuit.
Illustration 14: Mutant-based testing – Selection of mutation operators
From the set of generated mutants, the user can select those that he considers appropriate to test.
Ilustración 15: Mutant-based testing – Selection of mutants
Finally, the tester selects the simulator or quantum machine available on the QPath® platform on which the execution of the mutants will be launched.
Illustration 16: Mutant-based testing – Execution platform selection
Illustration 17: Mutant-based testing – Execution in process
Once the execution of the test cases against each mutant is finished, the system performs the analysis of each result in which the status (Killed or Alive) of each mutant is established and calculates the mutation coverage, dividing the number of “killed” mutants by the total number of mutants generated. The closer this percentage is to 100%, the more robust the test suite will be considered.
Illustration 18: Mutant-based testing – Killing matrix
Illustration 19: Mutant-based testing – Mutant execution result
As a utility, the user can generate a report in PDF format with the test result.
The system allows the user to launch this process as many times as desired, making corrections to the circuit, modifying the set of test cases and analysing the quality of the test suite, until it is guaranteed that the circuit under test does not contain errors and that the test cases are effective in detecting errors in the future, eventually being able to save the changes in QuantumPath® as a tested circuit without errors.
Summary
As shown, with QTester APP beta 1 it is possible to test quantum circuits with two different types of tests: based on test use cases and based on mutants. This makes it easier for the user to choose the type of tests that he considers most appropriate for each circuit or, even, subject complex circuits to both types of tests to achieve greater reliability of the quality of the quantum circuit to be implemented in critical software solutions.
Carrying out quantum software tests with QPath®’s QTester APP has a positive impact on the quality of the quantum software developed and also has a positive economic effect if the tests are carried out before running the solutions on the quantum computers, not only because in this way, the execution time of a solution with errors is avoided, but also the cost that the use of paid quantum computers can entail to carry out executions with software that still presents defects in its development.
Having implemented the architecture and the fundamental requirements that we have defined in beta 1 to test the quantum gate circuits, the QTester APP development project continues in progress. Its roadmap foresees gradually expanding its processes, progressively adding new types of quantum software testing for different technological approaches and, above all, incorporating advanced functionalities and specialized services that, applying the principles of the Talavera Manifesto[3], make it easier for developers to offer quantum programs with zero defects and ensures the quality of quantum software created with QuantumPath®.
With QTester APP beta 1, we take the first practical steps to provide tools to developers of quantum gate circuits based on the good practices of Quantum Software Engineering. With public access to the beta, we make it easier for the software, based on extensive user experience, various test cases, etc., to undergo a thorough practical test of the APP that will allow us to verify, through its use, not only its usefulness for testing quantum software, but also that the APP is free of possible bugs and, thus, contribute to ensure an adequate level of quality. In addition, sharing the use of beta 1 with users will help us, with the anonymous use metric that this will generate, to improve QPath®’s QTester APP, incorporating the most relevant user experiences into it. We will regularly report on these developments.
[1] Polo, M. García, I. García, A. Serrano, M.A. Piattini, M. Martínez, A. y Peterssen, G. (2022) Chapter 7, Quantum Software Testing. On Peterssen, G. Hevia, J. L. Piattini, M. y otros (Ed.), Quantum Software Engineering & QuantumPath®. aQuantum.
[2] Idem.
[3] The Talavera Manifesto for Quantum Software Engineering and Programming, 2020. https://www.aquantum.es/manifesto/
