Hardware Testing with Pytest
Learn how to build reliable automation software for hardware testing using Python's powerful pytest framework.
This article covers how FixturFab approaches test development for hardware test automation. We talk about high-level concepts about writing automation software for hardware tests and walk through examples of writing software using Python and pytest. When developing automated test systems for hardware, choosing the right tools and approach is crucial for long-term success.
When developing a turnkey test system, getting everything working together in the software development phase is where you’ll find the holes in your system. If those holes are big enough, it could mean one or multiple system redesigns, which can get expensive once your physical hardware has already been purchased and integrated. Making sure you carefully plan out your software and testing plan in the early phases is the most important step in mitigating significant issues.
Taking a detailed test specification and turning it into working software can still add a lot of complexity. The specifics of how your software interacts with your hardware are important for ensuring your fixture runs reliably. And how you collect and store measurement data is very important if you want to effectively use that data to improve your products or manufacturing pipeline in the future.
For any test or set of hardware tests, there is a general test flow that your tests will follow:
In the setup phase, you will run code to interface with your hardware resources and set them into the state your board needs for a test. If you’re writing many firmware tests, you will want to turn on the power to the Device Under Test (DUT) via a power supply. You may want to open a serial communication interface or simulate electrical input. None of the actions will return a result that indicates working firmware but are necessary precursors to any test measurement.
In the test phase, you will collect data from one or multiple measurement points and compare them to a set of test limits. The simplest of tests can be measuring a DC voltage at a voltage rail and verifying that the DUT’s onboard supplies are within a set of tolerances. But your test could be more complicated than that. You may want to test firmware by verifying a set of commands are sent and received over a serial interface. In that case, your measurement result might be a byte string you are comparing to an expected result.
In the teardown phase, you will close any open hardware interfaces. This could mean removing power to the DUT via a power supply. It could also mean closing a serial communication interface or removing a stimulated electrical input. You need to be the most careful writing your teardown code, because this code should run after your test whether or not your measurement was a passing or failing value.
These three phases are relevant for both individual tests and sets of tests. A lot of the time, you may want to set up and teardown some hardware only once. For example, you may want to switch power to the DUT and then run many tests that require it to be on. In this case, you may have a nested setup and teardown structure like so:
Once we build up more realistic and complicated test specifications and software requirements, you might start to recognize that maintaining your hardware resources might be hard to do with just a bunch of standalone test scripts.
Making sure that when things fail in hardware, they fail where you want them to in software can be difficult with so many layered dependencies. For example, if you forget to teardown a GPIO that you turned on for one test, it may affect the next test and cause it to fail. If you are not careful about these dependencies, your test software can easily become rigid, and moving tests around can break things unexpectedly.
As you can imagine, handling the software to interface with the setup and teardown of your test equipment throughout a test run can get pretty complicated. We’ve found that pytest is a great tool for managing these dependencies.
Pytest is a Python-based test framework. We prefer using Python, a widely used software language that is straightforward to develop. It has a vast collection of open-source libraries. Big test equipment manufacturers like Rigol frequently provide Python libraries to interface with their equipment. And if not, you can usually find open-source libraries or resources that are free and easy to use.
There are a couple of features that have made pytest the go-to testing framework for us at Fixturfab:
We’ve found that the pytest framework natively handles many difficulties of managing different hardware resources without making the test code rigid. Moving tests around and running only sets of tests from the command line makes it a powerful tool for developer and production-level hardware testing.
There are a lot of resources to learn how to get started with pytest, as it is one of the most commonly used test frameworks in the software world. I recommend looking at this article if you’re unfamiliar with the pytest framework and want to learn more about how software engineers use pytest to test their code.
But we want to use pytest to test hardware, not software. So, let us go through the basics of using pytest from a hardware testing standpoint.
Consider the very simple example test plan below:
Let’s run through this example using pytest to share how to start.
The first step is developing the code for all of your hardware interfaces.
Consider a simple test system with a power supply and a digital multimeter. Let’s write a class that will access that power supply to enable or disable power to the DUT (Device Under Test) and read the voltage from your digital multimeter:
"""hardware_interface.py"""
from power_supply import Supply
from multimeter import Multimeter
class Interface:
pinmap = {"5V": 7, "3V3": 3, "1V1": 5} # Maptest point net name to multimeter pin
def __init__(self):
"""Initialize anytest and measurementdevices in thetest system"""
self.supply = Supply()
self.multimeter = Multimeter()
def dut_en_5v(self, state: bool):
"""Enable or disable 5V to the DUT via mul"""
self.supply.set_voltage(state, voltage=5.0)
def check_supply_shorted(self):
"""Check if supply is shorted to ground"""
return self.supply.get_voltage() < 4.0:
def get_voltage(self, test_point: str) -> float:
channel = self.pinmap[test_point]
return self.multimeter.get_voltage(channel=channel)
def close(self):
"""Teardown all thehardware in thetest system"""
self.supply.close()
self.multimeter.close()
Now we have built a Python class that will let us instantiate our hardware as an Interface object like so:
from hardware_interface import Interface
interface = Interface()To use and test this new hardware interface with pytest, you can create a pytest fixture to set up and pass this to tests in your test suite. I’d look at the pytest documentation to learn how to use fixtures and how they get passed around by pytest.
To test with our new hardware interface, we will define it as a pytest fixture in our conftest.py file like so:
"""conftest.py"""
import pytest
from hardware_interface import Interface
@pytest.fixture(scope="session")
def hardware_interface():
"""Fixture used to interface with thehardware in thetest system """
print("SETUP Interface")
hardware_interface = Interface()
yield hardware_interface
print("TEARDOWN Interface")
hardware_interface.close()
You can now pass this fixture as a parameter to any tests in the repository that use it. When you do so, pytest will automatically handle the setup and teardown of all the necessary hardware resources during the test run.
The first time it finds this fixture called by a test, it will set up that hardware interface and hold it open to use it for any tests that fixture is scoped to use. In this case, the hardware interface is scoped for “session”, so it will only be setup and torn down once for the entire test session.
Now that our hardware interface is set up in pytest, we can write our test cases. If you want pytest to run a test, add “test” to the start of your function, class, and/or filename, and pytest will automatically find and collect it. Any available pytest fixtures can be passed into a test case by passing the fixture name as an argument for your method.
With this in mind, let’s write two pytest test cases: First, a test to check that the power supply can successfully power the DUT without shorting to ground. Then, another test will be conducted to measure and verify that a given voltage rail is within limits.
This can be done with two simple pytest cases shown below:
"""test_voltages.py"""
def test_dut_en_5v(hardware_interface):
"""DUT PowerTest
Send 5V to DUT and verify DUT properly powered
"""
print("TESTING test_dut_en_5v")
hardware_interface.dut_en_5v(True)
shorted = hardware_interface.check_supply_shorted()
hardware_interface.dut_en_5v(False)
assert not shorted
def test_3v3_voltage(hardware_interface):
"""DUT 3.3V Voltage RailTest.
Verify that the DUT is receiving 3.3V from the 5V supply.
"""
print("TESTING test_3v3_voltage")
# Measure
hardware_interface.dut_en_5v(True)
meas = hardware_interface.get_voltage(test_point="3V3")
hardware_interface.dut_en_5v(False)
assert 3.2 < meas < 3.4
If we run this file with pytest, and all our voltages measure as we expect them to, our output will look something like this:
(venv) $ pytest -s
=============================test session starts =============================
platform linux --Python 3.10.12, pytest-8.1.1, pluggy-1.4.0
rootdir: /home/marzieh/Documents/Test_Software/example
configfile: pyproject.toml
collected 2 items
test_voltages.py
SETUP Interface
TESTING test_dut_en_5v
.
TESTING test_3v3_voltage
.
TEARDOWN Interface
============================== 2 passed in 0.00s ==============================
But let’s say our voltage rail is measuring higher than expected. Then we might see something like this:
(venv) $ pytest -s
=============================test session starts =============================
platform linux --Python 3.10.12, pytest-8.1.1, pluggy-1.4.0
rootdir: /home/marzieh/Documents/Test_Software/example
configfile: pyproject.toml
collected 2 items
test_voltages.py
SETUP Interface
TESTING test_dut_en_5v
.
TESTING test_3v3_voltage
F
TEARDOWN Interface
================================== FAILURES ===================================
______________________________ test_3v3_voltage _______________________________
hardware_interface = <hardware_interface.Interface object at 0x7c3fa0cec520>
def test_3v3_voltage(hardware_interface):
"""DUT 3.3V Voltage RailTest.
Verify that the DUT is receiving 3.3V from the 5V supply.
"""
print("\\nTESTING test_3v3_voltage")
# Measure
hardware_interface.dut_en_5v(True)
meas = hardware_interface.get_voltage(test_point="3V3")
hardware_interface.dut_en_5v(False)
> assert 3.2 < meas < 3.4
E assert 3.6 < 3.4
test_voltages.py:28: AssertionError
=========================== shorttest summary info ===========================
FAILED test_voltages.py::test_3v3_voltage - assert 3.6 < 3.4
========================= 1 failed, 1 passed in 0.01s =========================
And just like that, with a few simple files, we have a fully functional test software test suite. You can run all your hardware tests with a simple call to the command line. Lots of great information is available in pytest’s documentation on how to set up and run your tests.
If you want to stop testing as soon as your first piece of hardware fails, call the pytest command with ‘-x.’ If you want to connect dependent tests so that some will not run if a piece of relevant hardware fails, simply set up pytest markers to flag and skip tests. When you set up your tests correctly, you shouldn’t need to touch your source code to run or debug these tests in whatever order or configuration you want.
Another important consideration in writing a good test suite is the efficiency of the test software. You may have noticed that in the previous example, when we called test_3v3_voltage, we turned the power supply on and off for the individual test. If we wanted to add more voltage checks and turn the supply on and off for each test, that would be a pretty inefficient way of running the test sequence, especially if the DUT has a long or significant bootup time.
This is where fixture scoping comes in. When you instantiate a pytest fixture, you can manage the range of its scope, and it will run its setup and teardown at the start and end of that scope. Say you want to group many tests with a common hardware setup and teardown. You could write a fixture with a “class” or “module” level scope. If you’ve written a bunch of tests in a module (ex/ a “test_voltages.py” file) or a class, the fixture will only setup and teardown once for a set of tests run within a given scope.
Let’s add another fixture to provide our hardware interface in the state where power is being provided to the DUT:
"""conftest.py"""
import pytest
from hardware_interface import Interface
@pytest.fixture(scope="session")
def hardware_interface():
"""Fixture used to interface with thehardware in thetest system """
print("SETUP Interface")
hardware_interface = Interface()
yield hardware_interface
print("TEARDOWN Interface")
hardware_interface.close()
@pytest.fixture(scope="class")
def hardware_interface_powered(hardware_interface):
"""Fixture used to manage afixture state where power needs to be enabled. """
#Hardware setup beforetests in thissequence:
print("SETUP of alltests in TestMultipleVoltages")
hardware_interface.dut_en_5v(True)
yield hardware_interface
#Hardware teardown aftertests in thissequence:
print("TEARDOWN of alltests in TestMultipleVoltages")
hardware_interface.dut_en_5v(False)
So, if we wanted to rewrite our test_voltages.py file to group these tests together and handle the group’s setup and teardown at once, we could rewrite the file like so:
"""test_voltages.py"""
def test_dut_en_5v(hardware_interface):
"""DUT PowerTest
Send 5V to DUT and verify DUT properly powered
"""
hardware_interface.dut_en_5v(True)
shorted = hardware_interface.check_supply_shorted()
hardware_interface.dut_en_5v(False)
assert not shorted
class TestMultipleVoltages:
def test_3v3_voltage(hardware_interface_powered):
"""DUT 3.3V VoltageTest.
Verify that the DUT is receiving 3.3V from the 5V supply.
"""
print("TESTING test_3v3_voltage")
# Measure
voltage = hardware_interface_powered.get_voltage(test_point="3V3")
assert 3.2 < voltage < 3.4
def test_1v1_voltage(hardware_interface_powered):
"""DUT 1.1V VoltageTest.
Verify that the DUT is receiving 1.1V from the 5V supply.
"""
print("TESTING test_1v1_voltage")
# Measure
voltage = hardware_interface_powered.get_voltage(test_point="1V1")
assert 1.0 < voltage < 1.2
Because this new fixture is only scoped to the class, it will set up and teardown once for the entire set of class tests. It also won’t be set up until the first test where it is called, so if we want to make sure it isn’t opened unless the supply is not shorted, we have to order the class tests after test_dut_en_5v.
Now, if we re-run this file, our new test output will look something like this:
(venv) $ pytest -s
=============================test session starts =============================
platform linux --Python 3.10.12, pytest-8.1.1, pluggy-1.4.0
rootdir: /home/marzieh/Documents/Test_Software/example
configfile: pyproject.toml
collected 2 items
test_voltages.py
SETUP Interface
TESTING test_dut_en_5v
.
SETUP of alltests in TestMultipleVoltages
TESTING test_3v3_voltage
.
TESTING test_1v1_voltage
.
TEARDOWN of alltests in TestMultipleVoltages
TEARDOWN Interface
============================== 3 passed in 0.00s ==============================
When hardware dependencies are handled with properly scoped fixtures, pytest can handle the layered setup sequences under the hood. This helps modularize our test code, even when the physical hardware interface needs to be in a very particular state for any given test.
Writing automation software for hardware can be daunting for a test engineer. There are infinite ways to approach test development, and managing the various hardware dependencies can become a tangled and confusing process if you are trying to write all your code from scratch. We’ve found that pytest is a powerful framework for writing and running hardware tests. Pytest fixtures allow for easy hardware dependency management, and Python is a widely supported language for interfacing with hardware testing resources such as digital multimeters and power supplies.
If you’d like to learn more about FixturFab’s test software, the next article in this series covers how to use our open-source pytest-f3ts library, a free plugin that simplifies some of the above examples.
The second part of the DEV260 Fixture Series covers the BOM, laser-cutting the plates, and assembling the mechanical fixture.
Transform manufacturing data into actionable insights by automating test analytics with pytest-f3ts and Tofu Pilot.
