This document defines the YAML format and structure used for Eth2 testing.
Ethereum 2.0 uses YAML as the format for all cross client tests. This document describes at a high level the general format to which all test files should conform.
The particular formats of specific types of tests (test suites) are defined in separate documents.
generator: a program that outputs one or more test-cases, each organized into a
config > runner > handler > suitehierarchy.
config: tests are grouped by configuration used for spec presets. In addition to the standard configurations,
generalmay be used as a catch-all for tests not restricted to one configuration. (E.g. BLS).
type: the specialization of one single
generator. E.g. epoch processing.
runner: where a generator is a "producer", this is the "consumer".
runnerfocuses on only one
type, and each type has only one
runnermay be too limited sometimes, you may have a set of tests with a specific focus that requires a different format. To facilitate this, you specify a
handler: the runner can deal with the format by using the specified handler.
suite: a directory containing test cases that are coherent. Each
suiteunder the same
handlershares the same format. This is an organizational/cosmetic hierarchy layer.
case: a test case, a directory in a
suite. A case can be anything in general, but its format should be well-defined in the documentation corresponding to the
case part: a test case consists of different files, possibly in different formats, to facilitate the specific test case format better. Optionally, a
meta.yamlis included to declare meta-data for the test, e.g. BLS requirements.
The configuration constant types are:
(genesis data, timestamp) -> epoch number, you end up needing both constants.
Based on these types of changes, we model the config as a list of key value pairs, that only grows with every fork (they may change in development versions of forks, however; git manages this). With this approach, configurations are backwards compatible (older clients ignore unknown variables) and easy to maintain.
Tests should be independent of any sync-data. If one wants to run a test, the input data should be available from the YAML. The aim is to provide clients with a well-defined scope of work to run a particular set of test-suites.
File path structure: tests/<config name>/<fork or phase name>/<test runner name>/<test handler name>/<test suite name>/<test case>/<output part>
Configs are upper level. Some clients want to run minimal first, and useful for sanity checks during development too. As a top level dir, it is not duplicated, and the used config can be copied right into this directory as reference.
<fork or phase name>/
This would be: "phase0", "altair", etc. Each introduces new tests, and modifies any tests that change: some tests of earlier forks repeat with updated state data.
<test runner name>/
The well known bls/shuffling/ssz_static/operations/epoch_processing/etc. Handlers can change the format, but there is a general target to test.
<test handler name>/
Specialization within category. All suites in here will have the same test case format.
handler in a
runner is optional. A
core (or other generic) handler may be used if the
runner does not have different formats.
<test suite name>/
Suites are split up. Suite size (i.e. the amount of tests) does not change the maximum memory requirement, as test cases can be loaded one by one. This also makes filtered sets of tests fast and easy to load.
Cases are split up too. This enables diffing of parts of the test case, tracking changes per part, while still using LFS. Also enables different formats for some parts.
These files allow for custom formats for some parts of the test. E.g. something encoded in SSZ.
Or to avoid large files, the SSZ can be compressed with Snappy.
post.ssz_snappy provides all the information for testing, when decompressed and decoded.
Then the difference between pre and post can be compared to anything that changes the pre state, e.g.
Note that by default, the SSZ data is in the given test case's <fork or phase name> version, e.g., if it's
altair test case, use
altair.BeaconState container to deserialize the given state.
YAML is generally used for test metadata, and for tests that do not use SSZ: e.g. shuffling and BLS tests. In this case, there is no point in adding special SSZ types. And the size and efficiency of YAML is acceptable.
Between all types of tests, a few formats are common:
.yaml: A YAML file containing structured data to describe settings or test contents.
.ssz: A file containing raw SSZ-encoded data. Previously widely used in tests, but replaced with compressed variant.
.ssz, but compressed with Snappy block compression. Snappy block compression is already applied to SSZ in Eth2 gossip, available in client implementations, and thus chosen as compression method.
If present (it is optional), the test is enhanced with extra data to describe usage. Specialized data is described in the documentation of the specific test format.
Common data is documented here:
Some test-case formats share some common key-value pair patterns, and these are documented here:
bls_setting: int -- optional, can have 3 different values: 0: (default, applies if key-value pair is absent). Free to choose either BLS ON or OFF. Tests are generated with valid BLS data in this case, but there is no change of outcome when running the test if BLS is ON or OFF. 1: known as "BLS required" - if the test validity is strictly dependent on BLS being ON 2: known as "BLS ignored" - if the test validity is strictly dependent on BLS being OFF reveal_deadlines_setting: -- optional, can have 2 different values: 0: default, `process_reveal_deadlines` is ON. 1: `process_reveal_deadlines` is OFF.
The runtime-configurables may be different for specific tests. When present, this replaces the default runtime-config that comes with the otherwise compile-time preset settings.
The format matches that of the
Config values that are introduced at a later fork may be omitted from tests of previous forks.
The constants configurations are located in:
<specs repo root>/configs/<config name>.yaml
And copied by CI for testing purposes to:
<tests repo root>/tests/<config name>/<config name>.yaml
<config name> is a directory, which contains exactly all tests that make use of the given config.
The basic pattern for test-suite loading and running is:
operations > deposits). Again, repeat for each if running all.
Step 1 may be a step with compile time selection of a configuration, if desired for optimization. The base requirement is just to use the same set of constants, independent of the loading process.