Bug 10873 - Randomize order in which tests are run
Randomize order in which tests are run
Status: NEW
Product: Samba 4.1 and newer
Classification: Unclassified
Component: Test infrastructure
unspecified
All All
: P5 normal
: ---
Assigned To: Jelmer Vernooij
Samba QA Contact
:
Depends on: 10939
Blocks: 10872 10874
  Show dependency treegraph
 
Reported: 2014-10-12 21:58 UTC by Jelmer Vernooij
Modified: 2014-11-16 17:23 UTC (History)
2 users (show)

See Also:


Attachments
Subunit output when running with --random on sn-devel-104 (515.69 KB, application/x-xz)
2014-10-13 00:29 UTC, Matthieu Patou
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Jelmer Vernooij 2014-10-12 21:58:13 UTC
Some tests depend on changes made to the environment by previous tests. This is Bad. 

This bug is a prerequisite for running the tests in "make test" in parallel, since that will break the order in which they're run (since subsequent tests might be run by different workers).

In order to prevent this kind of dependency, we should run tests in a random order.
Comment 1 Jelmer Vernooij 2014-10-12 22:04:52 UTC
We already have a --randomize-order flag to selftest to do this. However, we need to fix the existing tests to cope with randomized order before we can turn on this flag everywhere.
Comment 2 Matthieu Patou 2014-10-13 00:29:56 UTC
Created attachment 10347 [details]
Subunit output when running with --random on sn-devel-104
Comment 3 Matthieu Patou 2014-10-13 00:41:01 UTC
The current list of failed tests are:

samba4.drs.replica_sync.python(vampire_dc) 
samba.tests.blackbox.samba_tool_drs 
samba4.blackbox.samba_tool_demote(vampire_dc) 
samba4.drs.replica_sync.python(promoted_dc) 
samba4.blackbox.dbcheck(dc) 
samba4.nbt.dgram(dc) 
samba4.blackbox.dbcheck(vampire_dc) 
samba.tests.messaging 
samba4.blackbox.dbcheck(promoted_dc) 
samba4.ldap_schema.python(dc)