ADR 17: End-to-end testing strategy for Plutus, cardano-ledger-api and cardano-node-client¶
End-to-end testing of Plutus Core functionality is
currently performed by a combination of automation and exploratory approaches, both are performed
on public preview and pre-prod testnets using a real node.
Automation test scenarios for Plutus are currently being run as part of the wider cardano-node-tests test suite, which uses a Python wrapper
Those tests focus on general ledger/node/cli functionality and only cover a few key scenarios for
Plutus functionality, such as TxInfo and SECP256k1 builtins.
There is also ongoing development work to separate the functionality of cardano-api out into two packages:
cardano-ledger-api handles the building and balancing of transactions.
cardano-node-clientwill live in cardano-node and handle the submitting of balanced transactions and querying the ledger state.
Both of these packages are in early stage development and will require end-to-end test coverage.
This document outlines the decisions and arguments for an additional approach to end-to-end test automation using a framework written in Haskell.
The exploratory testing approach is not in the scope of this document.
We will create a new end-to-end testing framework written in Haskell called
plutus-e2e-teststhat will initially be a package in plutus-apps, see argument 1.
We will use cardano-testnet for configuring and initialising local test network environments, see argument 2.
We will initially use
cardano-apifor building and balancing transactions, and to submit balanced transactions and for querying the ledger, see argument 3.
When available, we will use
cardano-apifor building and balancing transactions, see argument 4.
When available, we will use
cardano-apito submit balanced transactions and for querying the ledger state to make test assertions, see argument 5.
We will prioritise
Plutustest coverage over
cardano-node, see argument 6.
We will start by creating a few tests with the node/ledger apis without depending on
plutus-appsand then assess whether we want to use the Contract API and other off-chain tooling going forwards, see argument 7.
We will continue adding a subset of Plutus tests to
cardano-node-tests, see argument 8.
Types of Plutus tests for the
plutus-e2e-tests Haskell framework¶
Plutus end-to-end testing requirements will be covered by
In summary, with access to the Haskell and Plutus interfaces and reduced friction from using a
single programming language we are likely improve test coverage at this level.
For example, builtin functions and error scenarios.
Although we will be building out the end-to-end test coverage, it is more efficient to have fewer and broader test scenarios at this level and a greater number of tests at the lower unit and integration levels for stressing particular features and covering negative scenarios and edge cases.
Any Plutus Core builtin function. These may already be tested extensively in the lower unit/property/integration levels but there’s value in having some coverage at the end-to-end level too.
Use cases that go beyond testing features in isolation. Bringing together various functionality helps demonstrate the capability of more realistic Plutus applications.
Functionality introduced by a new Plutus version. This could mean that
ScriptContextchanges to accommodate an extended transaction body.
Functionality of old Plutus versions tested with each supporting script version. Tests for old Plutus version functionality will also be run using the newer script versions.
The primary aim is to satisfy all of
Plutus(core) end-to-end testing requirements, although, this is an opportunity to also get coverage of other packages being developed such as
cardano-node-client. We can configure external packages and their versions using CHaPs so there is no need for
plutus-e2e-teststo have its own repo. It will initially be be a package in
There are a few options for configuring and starting a private testnet (see local testnet notes section). We will use
cardano-testnetto enable dynamic configuration in Haskell, which makes it easier to design tests that can also run in the emulated environment. Also, this is the approach officially supported by the node team.
The plan is to start building tests with
cardano-node-clientare at the required stage of development. This allows us to immediately proceed with building out the framework and test suite.
The ambition of
cardano-ledger-apiis to be the go-to api for building transactions for application developers. The UX and overall quality of this component will benefit from being included in these end-to-end tests because of the high-level perspective applied during its design and development. When ready we will begin incorporating it as a replacement for
cardano-node-clientwill eventually replace
cardano-apias a interface with consensus. As the expected means to submit and query for application developers, it is a vital we include it under test in
plutus-e2e-tests. When ready we will begin incorporating it as a replacement for
cardano-node-clientare under test it isn’t feasible to expect thorough coverage of all ledger and node functionality, such as staking and update proposals, because the primary focus is to satisfy end-to-end testing requirements for
Plutus. Fortunately, much of that functionality is already being covered by
Initially, a few tests will be created without depending on Plutus-apps. This entails building transactions with
cardano-ledger-apiand waiting for on-chain events using
cardano-node-clientwithout use of the Contract api or the constraints library. This approach allows us to build specific transactions, which is especially useful for testing edge-cases and error scenarios that off-chain tooling may prohibit. However, this approach will require more boilerplate code and this could negatively impact readability of the tests. Having assessed this approach, we may then decide to depend on
Plutus-appsfor the Contract api, which would give a uniform interface for off-chain code such as different node backends (private and public testnets, and emulator) and chain-indexer queries (
cardano-node-clientor Marconi in future). It should also reduce the amount of boilerplate code and provide additional features such as trace logging.
There’s value continuing to test
cardano-cliwith Plutus transactions for specific cli flags and the cli’s error handling with script evaluation. Some examples of tests that should be covered:
Cli flags that require use of Plutus scripts E.g. tx-out-reference-script-file or calculate-plutus-script-cost
Cli behaviour when script evaluation passes. This could be displaying expected fee correctly.
Cli behaviour when script evaluation fails. This can be how different types errors are formatted.
At some point, we may wish to incorporate the cardano-node-emulator as an alternative to
cardano-testnet. This would enable us to run property based tests due to the node being much faster without consensus. With CHaP,
cardano-node-emulatorwould be released as a separate component, so no need to depend on
We reserve the option of including additional packages to test from
Plutus-appsat a later stage.
Pros of building and maintaining our own test framework¶
Plutus tools team will have full ownership of the end-to-end test environment and its priorities for
Plutus scripts can be defined alongside the tests.
cardano-node-testsrequires pre-compiled scripts.
Tests will demonstrate how these Haskell packages can be used together to guide Plutus application development using the node apis. Particularly useful for less experienced Haskell developers.
Possible to define tests once and run at different levels. For example, on private or public testnets and with
Benefits from use of all
Plutusapis. For example, using PlutusTx to produce scripts using a typed interface, and optionally the Contract monad from
Have the opportunity to add more components under test at a later stage, such as Marconi or a PAB.
cardano-cliwould not be a dependency for Plutus test coverage so no risk of being blocked by its stage of development.
Less dependence on repetitive manual approach for regression testing because tests can be planned and implemented in parallel with feature implementation and integration.
Plutus team can implement and review majority of tests in Haskell rather than Python, which is likely to be the team’s preference. Also won’t need to review as many tests in
Less friction caused by cross-team: planning, dependencies and expectations. Plutus team won’t need to wait for node test team to implement the tests. It’s likely that other node/cli features will often be prioritised.
This approach will improve our high-level perspective of each component and help guide UX improvements.
Now that some
plutustests exist in
cardano-node-teststhe process for adding new tests will be relatively straightforward, for some it’s mostly a copy/paste job. This means less work to support some duplicate tests in both frameworks.
Node team are not pressured to focus on Plutus scenarios, they retain control of their priorities.
Cons of building and maintaining our own test framework¶
cardano-node-testsis well established and already has useful features such as: running tests in different eras, transitioning between eras, reporting, and measuring deviations in script cost.
It could be quicker for us to get going to reusing the bash scripts
cardano-node-testshave. See local testnet notes section for other examples of spinning up a local testnet.
We could continue getting
plutusend-to-end test coverage without the need to build our own framework because the node test team will continue to maintain theirs regardless.
Plutus team will still be required to support the node test team with defining and reviewing a subset of Plutus tests in
Node test team may grow, less delays in getting Plutus tests implemented by a Python developer.
The tests using
cardano-clialready provide some assurance that downstream components are working correctly, so there will be some duplication of test coverage by having an additional framework.
Business stakeholders will want to see test results to think about producing and storing a report. It would be to open source this along with the tests, like
At first, tests will be run on a private testnet but we must consider how these tests can also be run on a public testnet. For example, initial wallet balances and utxos will need to be handled dynamically because we’d only have control over these in the private testnet.
cardano-node-clientare still in early stages of production it would make sense not to block creation of
plutus-e2e-tests. We can begin using
cardano-apiand switch over when ready.
End-to-end tests can be slow to execute and as the suite grows we may want to run a subset at more frequent intervals. For example, we run tests for the latest Plutus version nightly but older tests/versions are run weekly, or for tags/release only.
Instead of creating a new repository it is possible the end-to-end tests could live in
Although, because the components under test span other repositories it would be restrictive and
additional work at the time when dependencies are updated in
Plutus-apps, see argument 1.
We could use bash scripts to spin up a local testnet, which is the approach teams such as Djed and
Although, the decision is to use
cardano-testnet, see argument argument 2.
This ADR document should be moved out of
Plutus-apps` and into the new end-to-end test repository
Benchmarking hasn’t been covered above because we already have a team dedicated to testing cardano-node
performance that includes some Plutus scripts. It is an automated approach using
Other places spinning up a local testnet¶