Bug 10873 - Randomize order in which tests are run
Summary: Randomize order in which tests are run
Status: NEW
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: Test infrastructure (show other bugs)
Version: unspecified
Hardware: All All
: P5 normal (vote)
Target Milestone: ---
Assignee: Jelmer Vernooij
QA Contact: Samba QA Contact
URL:
Keywords:
Depends on: 10939
Blocks: 10872 10874
  Show dependency tree / graph
 
Reported: 2014-10-12 21:58 UTC by Jelmer Vernooij
Modified: 2014-11-16 17:23 UTC (History)
2 users (show)

See Also:

Attachments
Subunit output when running with --random on sn-devel-104 (515.69 KB, application/x-xz)
2014-10-13 00:29 UTC, Matthieu Patou 		no flags Details
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Jelmer Vernooij 2014-10-12 21:58:13 UTC 
Some tests depend on changes made to the environment by previous tests. This is Bad. 

This bug is a prerequisite for running the tests in "make test" in parallel, since that will break the order in which they're run (since subsequent tests might be run by different workers).

In order to prevent this kind of dependency, we should run tests in a random order.
Comment 1 Jelmer Vernooij 2014-10-12 22:04:52 UTC 
We already have a --randomize-order flag to selftest to do this. However, we need to fix the existing tests to cope with randomized order before we can turn on this flag everywhere.
Comment 2 Matthieu Patou 2014-10-13 00:29:56 UTC 
Created attachment 10347 [details]
Subunit output when running with --random on sn-devel-104
Comment 3 Matthieu Patou 2014-10-13 00:41:01 UTC 
The current list of failed tests are:

samba4.drs.replica_sync.python(vampire_dc) 
samba.tests.blackbox.samba_tool_drs 
samba4.blackbox.samba_tool_demote(vampire_dc) 
samba4.drs.replica_sync.python(promoted_dc) 
samba4.blackbox.dbcheck(dc) 
samba4.nbt.dgram(dc) 
samba4.blackbox.dbcheck(vampire_dc) 
samba.tests.messaging 
samba4.blackbox.dbcheck(promoted_dc) 
samba4.ldap_schema.python(dc)