source code: py/test/test_lists/JSON_TEST_LIST.md
How To Write a ChromeOS Factory Test List (In JSON)¶
The new test lists will be written in JSON format, and will be loaded by
cros.factory.test.test_lists.manager
.
A test list file is a JSON file containing one object, the object can have following fields:
inherit
: a list of strings, e.g.["a.test_list", "b.test_list"]
. Specifies base config files for this config file, e.g.a.test_list.json
andb.test_list.json
. Fieldsconstants
,options
,definitions
will be loaded and merged with current config files. Inheritance order is resolved by C3 linearization.constants
: key value pairs to define some constants that could be used later. Please refer to Evaluation section for usage of constants.options
: test list options, please refer tocros.factory.test.test_lists.test_list.Options
.definitions
: define some reusable test objects.tests
: a list of test objects, which are the top level tests in this test list.__comment
: just a comment. The test list manager will ignore this field.override_args
: a dictionary where keys are test paths you want to override test arguments, values are the test arguments that will be merged with the old one.
Examples¶
You can find examples under ./manager_unittest/
folder,
e.g. a.test_list.json.
Test Objects¶
Each test object represents a cros.factory.test.factory.FactoryTest
object
(with some additional information). Here are some attributes you can set to a
test object.
ID¶
Each test item must have an ID. ID will be used to define the path of a test. path is defined as:
test.path = test.parent.path + '.' + test.ID
For example, the test group SMT
will have test path SMT
, and the test item
ProbeCamera
will have test path SMT.ProbeComponents.ProbeCamera
.
ID
can be auto generated, see Syntactic Sugar for more
details.
label¶
label
is a string that will be shown on UI. label
will be treated as an i18n
string by default.
pytest_name and args¶
Leaf nodes (the test objects have no subtests) of a test list should be a
pytest. A pytest is a test written in python and placed under
py/test/pytests/
in a public or private factory source tree.
Each pytest can define their arguments, the ARGS
variable in the class. And
the args
is used to assign values to these arguments. args
is a dictionary
of key value pairs where keys are mapped to the name of each argument in
ARGS
.
Subtests¶
To create a group of tests, you just need to
{
"label": "Test Group Label",
"subtests": [
"a",
"b",
"c"
]
}
By default, the test group will be tested in a sequence, that is, if any
subtest will be retested, all of the subtests will be retested in order. To
create a group that subtests can be run individually, inherit from
TestGroup
:
{
"label": "Test Group Label",
"inherit": "TestGroup",
"subtests": [
"a",
"b",
"c"
]
}
Locals¶
Each test object can have a locals
attribute, which is a dictionary
of key value pairs. locals
will be available when Goofy is resolving args
that will be passed to pytest.
Allow Unexpected Reboots¶
Set allow_reboot
to true
if you want to allow unexpected shutdown or reboot
when running the test.
More specifically, when goofy starts up, it checks if the last running test is
marked as allow_reboot
or not. If it’s false
, it means an unexpected
shutdown just happened, so it stops all pending tests and waits for further
manual inspection. If it’s true
, it means the shutdown is allowed or expected,
and the test will be re-run.
Please use this option with care. The rule of thumb is to properly shutdown the DUT whenever possible, by using the shutdown test. One usage of this option is at SMT line, when display is not attached to the mlb, and the operator might move the mlb to different test stations to perform the test multiple times.
Parallel Tests¶
To make two or more tests run at the same time, you need to group them under a
test group, and add attribute "parallel": true
to this test group.
Action On Failure¶
The action_on_failure
attribute allows you to decide what the next test should
be when this test fails. There are three possible values:
NEXT
: this is the default value, the test list will continue on the next test item.PARENT
: stop running other tests under the current parent. Let parent decide what the next test should be.STOP
: stop running other tests.
Teardown¶
Sometimes, you want a test be run after some tests are finished, no matter those tests success or not. For example, a test item that uploads log files to the shopfloor server should always be run despite the status of previous tests.
{
"label": "Test Group Label",
"subtests": [
{
"normal tests...",
{
"label": "Tear Down Group",
"teardown": true,
"subtests": [
...
]
}
}
]
}
Tests in teardowns can have their subtests as well. Those tests will become
teardown tests as well. We assume that teardowns will never fail. If a teardown
test fails, Goofy will ignore the failure and continue on the next teardown test.
Therefore, for teardown tests, action_on_failure
will always be set to NEXT
.
Additional Fields¶
Test list manager will process these fields, they are not directly used by
FactoryTest
.
inherit
: a string, the name of the base test object, default toFactoryTest
. The base test object should be defined indefinitions
section. For example, you can define aLEDTest
test object indefinitions
section:{ "definitions": { "LEDTest": { "pytest_name": "led", "args" : { "colors": ["RED", "GREEN"] } } } }
Now you can use “LEDTest” as base test object:
{ "tests": [ { "id": "MyFirstLEDTest", "inherit": "LEDTest" } ] }
If a test object defined in
definitions
section inherits itself, it means that it’s a class defined incros.factory.test.test_lists.test_object
module. For example, the definition ofFactoryTest
is:{ "definitions": { "FactoryTest": { "inherit": "FactoryTest" } } }
Please refer to base test list for more examples.
locals
: in JSON style pytest, attributelocals
of a test object will be passed to its subtests. Andlocals
will be evaluated before being set to theFactoryTest
object. Seelocals.test_list.json
as an example.child_action_on_failure
: default value ofaction_on_failure
of subtests.__comment
: this field will be ignored and discarded by the test list manager, it’s just a comment.override_args
: another way to override arguments of a test specified by its path. Seeoverride_args.test_list.json
as an example.
Expression Evaluation¶
For args
of a test object, if a value is a string and starts with "eval! "
,
the rest of the string will be interpreted as a python expression. The
expression will be evaluated by the python eval
statement. However, for
simplicity, the expression has the following restrictions:
Single expression (not necessary single line, but the parsed result is a single expression)
Not all operators are allowed, currently, the following expressions are not allowed:
Generator (e.g.
(x * x for x in range(10))
)Lambda function (e.g.
lambda x: x * x
)Other expressions that don’t make sense without a context, e.g.
yield
,return
When executing, the namespace will only contains:
built-in functions
constants
andoptions
dut
,station
locals
state_proxy
(created bystate.GetInstance()
)device
(a shortcut forstate_proxy.data_shelf.device
)
The result of the evaluated expression will not be evaluated again even if it
starts with "eval! "
as well.
i18n support¶
You can use "i18n! English string"
to specify an i18n string.
Or, an alternative way to create an i18n string is:
{"en-US": "English string", "zh-CN": "Chinese string"}
.
Therefore, both definitions.Start.label
and definitions.Start.args.prompt
are valid i18n strings.
{
"definitions": {
"Start": {
"pytest_name": "start",
"label": "i18n! Start",
"args": {
"prompt": {"en-US": "English prompt", "zh-CN": "Chinese prompt"}
}
}
}
}
Syntactic Sugar¶
For simplicity, we provide the following syntactic sugar:
Label¶
We support auto generated labels from
pytest_name
. For example, if the pytest name issome_test
, then the label will bei18n! Some Test
by default.Since the label should always be an i18n string, so
"label": "Start"
is equivalent to"label": "i18n! Start"
.
Automatic ID¶
If a test object defined in
definitions
(before merged with base test object) does not containid
field, the key string will be the default ID, for example:{ "definitions": { "LEDTest": { "pytest_name": "led", "args" : { "colors": ["RED", "GREEN"] } } } }
The test object
LEDTest
will have default"id": "LEDTest"
.If a test object defined in
tests
does not specifyid
field, it will inherit from its parent, therefore, in the following snippet,{ "tests": [ { "inherit": "LEDTest" }, { "inherit": "LEDTest" } ] }
The first led test will be:
LEDTest
, the second one will be:LEDTest_2
(_2
is automatically appended to resolve path duplication).If the test object contains only
inherit
field, you can just write a string, so the previous example can be simplified to:{ "tests": [ "LEDTest", "LEDTest" ] }
Generic Test Lists¶
generic_main.test_list.json is an example.
And you should reuse generic_*.test_list.json
to create a test list for your
board.
generic_common.test_list.json
¶
This defines all kinds of tests that might be used in the Chromebook factory. By inheriting this test list, you can use most pytests directly.
generic_<station>.test_list.json
¶
We have defined GRT
, FAT
, RUN_IN
, FFT
, SMT
stations. All of them are
based on generic_common.test_list.json
, with some additional station specific
tests, or test arguments overriding.
Create Your Own Test List¶
Try to reuse generic test lists if possible. In general, you need to define the following files in private overlay:
common.test_list.json
: this inheritsgeneric_common.test_list.json
, overrides some test arguments, and adds some board specific pytests.<station>.test_list.json
: each of them inherits correspondinggeneric_<station>.test_list.json
andcommon.test_list.json
.main.test_list.json
: this inherits<station>.test_list.json
andcommon.test_list.json
(andgeneric_main.test_list.json
if you want to inheritoptions
andconstants
). Definesoverride_args
,constants
,options
in this file, and also thetests
.
In most of the cases, if you are not adding / removing / reordering any tests,
you only need to override constants
, options
and override_args
.
Dump Python Test List in JSON Format¶
On a DUT, you can use factory dump-test-list <test list id> --format json
to
dump a test list in JSON format.
Main Wipe Test Lists¶
This is a special test list which only runs ModelSKU, updates fingerprint firmware and finalization. The test list could be useful if you would like to test different versions of recovery image on a finalized DUT. A typical use case could be:
There’s a finalized DUT, and it is booted from a recovery image. You have done some testing on this recovery image and would like to test another version of the recovery image.
You have a factory shim and would like to install different versions of recovery image from dome.
Upload recovery image, test image and toolkit to dome.
Enter recovery mode and boot into factory shim.
Press E to perform RSU to remove HWWP. The DUT will reboot after RSU unlock.
Enter recovery mode again, and press I to install from dome.
After installation, the DUT will reboot, and you’ll see the Goofy web page.
Run main_wipe test list.
Go to step 1 and repeat the whole process.
IMPORTANT: Please complete the finalization step. If the pre-condition is not met, then you can not ship the device because it violates factory requirements.
Modify the test list accordingly¶
By default, the main_wipe test list inherits from generic_fat and generic_grt
test lists. You should modify the inherited test list accordingly. For example,
inherit from common test list. Moreover, you should also modify the name of
this test list to main_
Why does the test list run ModelSKU
and Fingerprint Test Group
?¶
Finalization will reinitialize FPMCU entropy and this will cause some problems. (see b/194449380 for more info). Therefore, we need to first run ModelSKU to get the device data, then reflash the FPFW, and finally, run finalization. If the DUT you’re using does not have a fingerprint sensor, then you can remove ModelSKU and fingerprint test group from the test list.