We have observed on Debian unstable that Samba has failed to build on Linux SPARC (64-bit) starting with version 4.16.0 while 4.13.14 built without any problems. The failure indicates a header problem with gssapi.h: [3398/6183] Linking bin/default/source3/libtrusts-util-samba4.so 10:27:05 runner ['/usr/bin/gcc', '-Wl,--as-needed', '-Wl,--version-script=/<<PKGBUILDDIR>>/bin/default/source3/trusts_util.vscript', '-shared', '-Wl,-h,libtrusts-util-samba4.so.0', 'source3/libsmb/trusts_util.c.113.o', '-o/<<PKGBUILDDIR>>/bin/default/source3/libtrusts-util-samba4.so', '-Wl,-Bstatic', '-Wl,-Bdynamic', '-L/<<PKGBUILDDIR>>/bin/default/source4/cluster', '-L/<<PKGBUILDDIR>>/bin/default/source4/lib/socket', '-L/<<PKGBUILDDIR>>/bin/default/libcli/nbt', '-L/<<PKGBUILDDIR>>/bin/default/source4/lib/messaging', '-L/<<PKGBUILDDIR>>/bin/default/libcli/dns', '-L/<<PKGBUILDDIR>>/bin/default/source4/libcli/ldap', '-L/<<PKGBUILDDIR>>/bin/default/auth', '-L/<<PKGBUILDDIR>>/bin/default/lib/messaging', '-L/<<PKGBUILDDIR>>/bin/default/libcli/registry', '-L/<<PKGBUILDDIR>>/bin/default/libcli/smb', '-L/<<PKGBUILDDIR>>/bin/default/lib/addns', '-L/<<PKGBUILDDIR>>/bin/default/libcli/cldap', '-L/<<PKGBUILDDIR>>/bin/default/lib/socket', '-L/<<PKGBUILDDIR>>/bin/default/lib/krb5_wrap', '-L/<<PKGBUILDDIR>>/bin/default/source4/auth/kerberos', '-L/<<PKGBUILDDIR>>/bin/default/lib/tdb_wrap', '-L/<<PKGBUILDDIR>>/bin/default/third_party/heimdal_build', '-L/<<PKGBUILDDIR>>/bin/default/source4/lib/events', '-L/<<PKGBUILDDIR>>/bin/default/libcli/ldap', '-L/<<PKGBUILDDIR>>/bin/default/lib/ldb', '-L/<<PKGBUILDDIR>>/bin/default/libcli/security', '-L/<<PKGBUILDDIR>>/bin/default/auth/gensec', '-L/<<PKGBUILDDIR>>/bin/default/libcli/util', '-L/<<PKGBUILDDIR>>/bin/default/lib/param', '-L/<<PKGBUILDDIR>>/bin/default/libcli/named_pipe_auth', '-L/<<PKGBUILDDIR>>/bin/default/source4/librpc', '-L/<<PKGBUILDDIR>>/bin/default/lib', '-L/<<PKGBUILDDIR>>/bin/default/librpc', '-L/<<PKGBUILDDIR>>/bin/default/libcli/auth', '-L/<<PKGBUILDDIR>>/bin/default/lib/util', '-L/<<PKGBUILDDIR>>/bin/default/lib/dbwrap', '-L/<<PKGBUILDDIR>>/bin/default/lib/ldb-samba', '-L/<<PKGBUILDDIR>>/bin/default/libds/common', '-L/<<PKGBUILDDIR>>/bin/default/lib/replace', '-L/<<PKGBUILDDIR>>/bin/default/auth/credentials', '-L/<<PKGBUILDDIR>>/bin/default/source4/dsdb', '-L/<<PKGBUILDDIR>>/bin/default/nsswitch/libwbclient', '-L/<<PKGBUILDDIR>>/bin/default/source3', '-L/usr/local/lib', '-L/usr/local/lib', '-lsamba-passdb', '-lmsrpc3-samba4', '-llibcli-netlogon3-samba4', '-lwbclient', '-lsamdb-common-samba4', '-lsamba-credentials', '-lreplace-samba4', '-lsamba3-util-samba4', '-lsamdb', '-lflag-mapping-samba4', '-lldbsamba-samba4', '-ldbwrap-samba4', '-lsamba-modules-samba4', '-lcliauth-samba4', '-lsmbldap', '-lsamba-util', '-lsecrets3-samba4', '-lsmbldaphelper-samba4', '-lndr-standard', '-lsamba-sockets-samba4', '-lndr-nbt', '-lndr-samba4', '-lnpa-tstream-samba4', '-lsamba-hostconfig', '-lsamba-errors', '-lndr-samba-samba4', '-lndr', '-lgse-samba4', '-ltevent-util', '-lgensec-samba4', '-lutil-tdb-samba4', '-lsamba-security-samba4', '-llibsmb-samba4', '-ldcerpc-samba-samba4', '-ldcerpc-binding', '-lsmbconf', '-lldb', '-lcli-ldap-common-samba4', '-levents-samba4', '-lgssapi-samba4', '-ltdb-wrap-samba4', '-lcom-err-samba4', '-lauthkrb5-samba4', '-lkrb5samba-samba4', '-lsamba-debug-samba4', '-lkrb5-samba4', '-lasn1util-samba4', '-ltime-basic-samba4', '-lgenrand-samba4', '-lsocket-blocking-samba4', '-lutil-setid-samba4', '-lsys-rw-samba4', '-liov-buf-samba4', '-lCHARSET3-samba4', '-linterfaces-samba4', '-lroken-samba4', '-lndr-krb5pac', '-lserver-role-samba4', '-lcli-cldap-samba4', '-laddns-samba4', '-lcli-smb-common-samba4', '-lmessages-util-samba4', '-lutil-reg-samba4', '-lserver-id-db-samba4', '-lsamba-cluster-support-samba4', '-lmessages-dgm-samba4', '-lsmbd-shim-samba4', '-ltalloc-report-printf-samba4', '-lheimbase-samba4', '-lhcrypto-samba4', '-lasn1-samba4', '-lwind-samba4', '-lcommon-auth-samba4', '-lhx509-samba4', '-lcli-ldap-samba4', '-lclidns-samba4', '-lsmb-transport-samba4', '-lmsghdr-samba4', '-lMESSAGING-SEND-samba4', '-lcli-nbt-samba4', '-lnetif-samba4', '-lcluster-samba4', '-ltevent', '-ltalloc', '-lldap', '-ltalloc', '-ltdb', '-ldl', '-lbsd', '-lpthread', '-llber', '-lz', '-lresolv', '-lutil', '-lgnutls', '-licui18n', '-licuuc', '-licudata', '-lsystemd', '-ljansson', '-lcups', '-lcap', '-Wl,-z,relro', '-Wl,-z,now', '-Wl,-z,relro,-z,now', '-Wl,-no-undefined', '-Wl,--export-dynamic'] In file included from ../../third_party/heimdal/kdc/kdc_locl.h:46, from ../../third_party/heimdal/kdc/mssfu.c:34: third_party/heimdal/kdc/kdc-private.h:224:9: error: unknown type name ‘gss_OID_set’ 224 | gss_OID_set */*oidsp*/); | ^~~~~~~~~~~ In file included from ../../third_party/heimdal/kdc/mssfu.c:34: ../../third_party/heimdal/kdc/kdc_locl.h:116:5: error: unknown type name ‘gss_OID_set’ 116 | gss_OID_set gss_mechanisms_allowed; | ^~~~~~~~~~~ The full log can be obtained from here [1]. The issue can be reproduced on HEAD on the GCC porter box gcc202 [2]. I have tried bisecting the problem which lead to the following bisect log: glaubitz@gcc202:~/samba$ git bisect log git bisect start # bad: [5e00c230ec8d37044205a16c3a6ceaa64de382d0] py:gpo: Fix testing of 0x8000 bit git bisect bad 5e00c230ec8d37044205a16c3a6ceaa64de382d0 # good: [3fe82c204f0d88cb6db50b7bd1f798b591a918f8] VERSION: Disable GIT_SNAPSHOT for the 4.13.0 release. git bisect good 3fe82c204f0d88cb6db50b7bd1f798b591a918f8 # good: [8c86998910d0520a09dee729c2892f26dccc4b65] VERSION: Disable GIT_SNAPSHOT for the 4.13.0rc1 release. git bisect good 8c86998910d0520a09dee729c2892f26dccc4b65 # good: [fec996ff277479e5bac2159e2aab78d838d86b4c] samldb: Fix function name typo in error message git bisect good fec996ff277479e5bac2159e2aab78d838d86b4c # good: [ebc9137cee94dee9dcf0e47d5bc0dc83de7aaaa1] tests/krb5: Align PAC buffer checking to more closely match Windows with PacRequestorEnforcement=2 git bisect good ebc9137cee94dee9dcf0e47d5bc0dc83de7aaaa1 # bad: [0f5d7ff1a9fd14fd412b09883d413d1d660fa7be] s4:kdc: redirect pre-authentication failures to an RWDC git bisect bad 0f5d7ff1a9fd14fd412b09883d413d1d660fa7be # good: [358c59f51ab39175ffe72afdfc4c2e0ed23b5929] ctdb-recoverd: No longer take cluster lock during recovery git bisect good 358c59f51ab39175ffe72afdfc4c2e0ed23b5929 # bad: [c266ed40aeb1b1f59a1811cd4511e32e44a4a719] s3/libads: simplify storing existing ads->ldap.ss git bisect bad c266ed40aeb1b1f59a1811cd4511e32e44a4a719 # bad: [49d18f2d6e8872c2b0cbe2bf3324e7057c8438f4] s3:libads: Remove trailing spaces from sasl.c git bisect bad 49d18f2d6e8872c2b0cbe2bf3324e7057c8438f4 # good: [232a1fa46af0b05ba12bf5edc944caeb4f919c38] smbd: Fix a typo git bisect good 232a1fa46af0b05ba12bf5edc944caeb4f919c38 # bad: [c7bd176f4cb5d058337b64819858eca2764bd88e] s4:kdc: Move calls using the samba4 name to be right after each other git bisect bad c7bd176f4cb5d058337b64819858eca2764bd88e # bad: [6e8ac61b36ec74581fde8720107bce8971989015] tests: Update latin1 list and ignored file list for new Heimdal import git bisect bad 6e8ac61b36ec74581fde8720107bce8971989015 # good: [12ca34115eabbb430cd0b01afeaaebfac76174d3] lib: Remove unused asprintf_strupper_m() git bisect good 12ca34115eabbb430cd0b01afeaaebfac76174d3 # good: [5636bfa9a27707895c97a32b0836ab9801456499] netlogon.idl: Add FAST support bits git bisect good 5636bfa9a27707895c97a32b0836ab9801456499 # skip: [40b65c840e03bd5eb7f3b02fe80144650c63c005] s4:heimdal: import lorikeet-heimdal-202201172009 (commit 5a0b45cd723628b3690ea848548b05771c40f14e) git bisect skip 40b65c840e03bd5eb7f3b02fe80144650c63c005 # bad: [b2c96d927a661d5e830b271043a6a2be94d4c04d] s4:heimdal_build: changes required to build after import git bisect bad b2c96d927a661d5e830b271043a6a2be94d4c04d # good: [d2a3016a9c59f93f89cf4bb86d40938d56400453] s4:heimdal_build: include heimdal headers relative to heimdal_build git bisect good d2a3016a9c59f93f89cf4bb86d40938d56400453 # only skipped commits left to test # possible first bad commit: [b2c96d927a661d5e830b271043a6a2be94d4c04d] s4:heimdal_build: changes required to build after import # possible first bad commit: [40b65c840e03bd5eb7f3b02fe80144650c63c005] s4:heimdal: import lorikeet-heimdal-202201172009 (commit 5a0b45cd723628b3690ea848548b05771c40f14e) glaubitz@gcc202:~/samba$ However, that didn't really give me any clue what the problem was since the issue apparently was introduced by the update of the embedded heimdal code. > [1] https://buildd.debian.org/status/fetch.php?pkg=samba&arch=sparc64&ver=2%3A4.16.1%2Bdfsg-6&stamp=1653820067&raw=0 > [2] https://cfarm.tetaneutral.net/machines/list/
The issue is understood to be related to the set reduction used in samba_deps.py being non-determinately ordered between platforms (and indeed at all times unless PYTHONHASHSEED is set). This in turn changes the link and include order for the build. No developer resources have been so far available to address the core issues this has raised in the build system.
The bug title is a symptom (one of many), not a cause. The cause is, like Andrew correctly wrote, is the random order of everything on the compiler/linker command lines. And while this particular symptom is easy to deal with (one can just remove the one of the 3 gssapi.h files and the build will be successful), there are other symptoms which are much more difficult to fix and even to notice. One of possible outcomes of this non-deterministic order is a successful build with resulting executables being crashy in random places. And we've seen this on Debian already - before we know about PYTHONHASHSEED thing. While setting this variable fixed the build for us, it is actually not the right fix, because the order is *still* random (as seen by this report), it is just a bit more deterministic and the amount of randomness is a bit lower than before. It is *still* possible to get broken builds, I think.
Thanks for the detailed explanation to both of you. I guess that reducing the number of parallel jobs during build should help to work around the problem. I will give it a try.
Sadly no, the die is cast at the start of the script. Alternate values for PYTHONHASHSEED might help, but really we need someone to apply some determinism to the set manipulations.
I don't have enough knowlege of python. It should be easy to fix, - by using the current set/hash in parallel with a list/array (whatever these constructs are called as in python), - use the array for all real operations, and use set to check if a given element is already there before adding it to the array.
Adding this reference for an example of how this can - in theory - be worked around: https://lists.samba.org/archive/samba-technical/2022-May/137349.html
and another data point, https://lists.samba.org/archive/samba-technical/2022-November/137788.html . The real fix would be to stop adding all the directories on the way as -I include paths for the compiler.
I think the issue is that Python sets don't have a consistent order which is a known fact. And yes, when you start the interpreter, as Andrew mentioned it "casts the die" right then, which seems to determine the order throughout (until you restart the Python shell). A quick experiment, start with a list in a python repl shell: >>> l = ['a', 'b', 'c'] Then cast it to a set: >>> set(l) {'c', 'a', 'b'} >>> set(l) {'c', 'a', 'b'} >>> set(l) {'c', 'a', 'b'} Now every time you type set(l) you will get back the same result. UNLESS you restart the python repl shell and it "recasts the die" then you will start seeing a different order. >>> l = ['a', 'b', 'c'] >>> set(l) {'b', 'a', 'c'}