7.1.1. Testing Guidelines
We recommend when creating your definition of done, you declare that every piece of work must contain appropriate tests, mostly notably any bug fixes must contain a regression test. What's considered an appropriate test depends on the trade-offs between different test types.
The testing pyramid should be easily visible in your project, each testing layer involving a different set of technologies and workflows, hence the trade-offs between the various tests becomes even more important to consider.
The testing guideline below describes some of these trade-offs in detail:
There are 3 main ways to write tests in your project:
ELPS [unit] tests, which live in:
phylum/*_test.lisp. This is an appropriate place to unit test ELPS logic.
Go [integration] tests, many of which live in:
oracleserv/X/oracle/X_test.go. These tests are closer to integration tests, and test e2e connectivity of the chaincode (phylum) layer. This is an appropriate place for edge-case testing, and complex logic testing. These tests can run a mock blockchain ("in-memory" mode).
E2E tests using the
martintool, which are stored under the
tests/dir with filenames like
X.martin_collection.yaml. These tests exercise end-to-end functionality of the oracle REST/JSON APIs using the
postmantool. This kind of test is most appropriate in demonstrating a happy path for a story, and not edge-case or unit testing. These tests also form documentation used by the frontend team to see how a new feature works. New tests should live under a directory describing the general entity APIs that are tested, e.g, a test that exercises the documents API should live under
tests/documents. The JIRA ticket number should be included in the test file name.
This section contains instructions for how to run each type of test.
Unit [phylum] tests
To run unit tests, you need to first ensure you're in the phylum directory. You can then run make test to run all the unit tests defined in
To run all the tests in one or more test files, run:
$(make echo:SHIRO_TEST) *_test.lisp [...]
To run specific tests with a regex, use
--run which has 'go test -run' semantics 1. E.g.,
phylum❯ $(make echo:SHIRO_TEST) --run "/regulating" === RUN TestFile_scaffold === RUN TestFile_scaffold/$load --- PASS: TestFile_scaffold (1.75s) --- PASS: TestFile_scaffold/$load (1.75s) === RUN TestFile_assertions === RUN TestFile_assertions/$load --- PASS: TestFile_assertions (0.34s) --- PASS: TestFile_assertions/$load (0.34s) === RUN TestFile_document === RUN TestFile_document/$load --- PASS: TestFile_document (0.37s) --- PASS: TestFile_document/$load (0.36s) === RUN TestFile_mortgage === RUN TestFile_mortgage/$load --- PASS: TestFile_mortgage (0.32s) --- PASS: TestFile_mortgage/$load (0.32s) === RUN TestFile_permission_utils === RUN TestFile_permission_utils/$load === RUN TestFile_permission_utils/perm-required-over-regulating-org? --- PASS: TestFile_permission_utils (0.70s) --- PASS: TestFile_permission_utils/$load (0.35s) --- PASS: TestFile_permission_utils/perm-required-over-regulating-org? (0.35s) === RUN TestFile_utils === RUN TestFile_utils/$load --- PASS: TestFile_utils (0.34s) --- PASS: TestFile_utils/$load (0.34s)
Integration [go] test
Integration tests can be run via the Makefile using make targets Specific test cases can be isolated using the helper script using
-tags all includes all tests) flags, for example:
./scripts/containerize.sh . go test -timeout 999m -parallel 100 -v -run=JIRA1039 -tags case ./...
Running Go tests directly on a Darwin host This is only possible on a Darwin host and only if you have first installed the exact same version of Go as was used to build the plugin. IMPORTANT: If you run tests via CLI, to easily handle multiple versions of go, please use the
goenv tool. E.g.,
brew install goenv --HEAD
Then add following to your shell profile (e.g., ~/.zshrc):
export GOENV_DISABLE_GOPATH=1 eval "$(goenv init -)"
go) should now use the version of go specified in
PROJECT_ROOT/.go-version which should match the native plugin.
./scripts/go-substrate-wrapper.sh can used to prefix a command that will run with the appropriate environment to load the substrate plugin. For example:
./scripts/go-substrate-wrapper.sh go test ./...
NOTE: To be able to run the Go test directly with
go test, you need to have
SUBSTRATE_PLUGIN_FILE env vars set to appropriate values, see
./scripts/go-substrate-wrapper.sh for more details.
Running Go tests from your favorite editor
If your favorite programming editor supports launching go tests, and if your programming editor properly forwards environment variables, it should be possible to launch your favorite programming editor via the substrate wrapper script and have it launch go tests directly:
./scripts/go-substrate-wrapper.sh, your current working directory must be the repository root.
For Goland specific instructions see the Goland IDE setup section in the root README.
End-to-end [martin] tests
For e2e testing, you can launch the oracles and run fabric in several modes, described below. Make sure to set an env var
API_KEY in the same shell you use to bring up the network and to have run
Here are the steps for running the martin tests:
cdinto the root directory of the project.
export API_KEY=abc123The value can be set to anything, it just need to be set before starting the oracle and should also be set when running the e2e tests.
make upfor full version. (See below for explanation of the difference)
make e2e FILE=./tests/document/JIRA-251-get-documents.martin_collection.yamlreplacing FILE with the file you'd like to run.
Debugging E2E tests
You can add additional console.logs in your tests to better see what's being returned however it's typically very useful to proxy all the requests to the oracle through a local proxy and watch the response and requests directly.
You can find instructions for how this can be setup here: Setting up a proxy for martin tests
Oracles in the environment serve the broker and client oracle off of different urls which necessitates
Various methods for starting martin tests
There are a few different ways to run the martin tests, the
make e2e and
make e2e-local targets are shortcuts for 2 common ways of running these tests.
These make targets take an optional FILE argument, which if not provided will mean the targets will run all the tests.
e2e target will run the tests in a docker environment, meaning less setup is necessary to be able to run them;
e2e-local will run the tests directly on your machine, which means certain binaries must be installed. Setting up a proxy for martin tests contains instructions for setting up these binaries (and explanation of why you might want to do that.)
Alternatively, you can run the martin tests more directly. For example the following
will run all of the
martin tests under the
tests/ directory using the local docker configuration. To run a specific test you can pass the file as a 2nd argument:
./tests/run-postman-collections-docker.sh ./tests/Docker.postman_environment.json ./tests/document/JIRA-251-get-documents.martin_collection.yaml
make mem-up vs make up
The oracle can be run locally in memory mode or replicate a full network.
make mem-up will run the oracle in in-memory mode which emulates fabric directly within the oracle process, this is a light-weight and fast way of running tests and does not require launching the entire fabric network.
make up will run a local network which launches a full fabric network locally. This is a heavier method, but more accurate/realistic way of testing.
After running either, you can execute
docker ps to list which services are running. You can also use
docker logs $CONTAINER_NAME to dump the logs for that container. E.g.,
docker logs oracle
Remember that when running the oracle locally, the REST/JSON API is accessible from your localhost on port 8080. You can issue CURL commands to spot test locally, so long as you have set your
COOKIE env variables. For example,
time curl -X GET -H "X-API-Key: $API_KEY" --cookie "$COOKIE" -s http://localhost:8080/v1/luther/system_metrics | jq ''
This section contains testing guidelines that we recommend you follow in your repo.
Unit [phylum] tests
- Critical functions should be unit tested, especially if they're stateless and are easier to setup state for
- Functions that take in large states, such as a case having passed certain verifications and being at a certain stage, are typically not unit tested due to difficulty of building up such state (Integration tests are ideal for these), however there are exceptions when the functionality is critical, e.g. calculations.
- Break up large functions into composition of smaller functions. This will allow smaller function to be unit tested when needed and increase readability and maintainability.
- Migrations are primarily tested via unit tests.
Integration [go] test
- Use t.Run wherever possible to breakup the tests. This will allow others to run segments of a tests rapidly when debugging a broken test.
- Use/create helper functions for setting up state: This will help with readability and maintainability. For example consider that we may change the requirements for common operations, and we don't want to have to update a lot of tests that repeat the same operation.
- If the helper function is generic enough, it should be placed in oracle_test.go
- Give every test a descriptive name. Remember, a test's value is in helping find the relevant unit/line when it fails. By giving them a descriptive name, we can hint at what the purpose of the test is. Avoid names like AC* and etc.
- Add ticket numbers for a test as a comment above the test, not in its name. For very long/old tests, placing the ticket number in the test name gets very confusing and doesn't help much. A descriptive name is much preferred, if a developer would like to find the ticket number they can look at the comment or use git blame
End-to-end [martin] tests
- These are more of smoke tests
- Cover essential/critical paths
- Try to replicate realistic configurations