We have observed on Debian unstable that Samba has failed to build on Linux SPARC (64-bit) starting with version 4.16.0 while 4.13.14 built without any problems. The failure indicates a header problem with gssapi.h: [3398/6183] Linking bin/default/source3/libtrusts-util-samba4.so 10:27:05 runner ['/usr/bin/gcc', '-Wl,--as-needed', '-Wl,--version-script=/<<PKGBUILDDIR>>/bin/default/source3/trusts_util.vscript', '-shared', '-Wl,-h,libtrusts-util-samba4.so.0', 'source3/libsmb/trusts_util.c.113.o', '-o/<<PKGBUILDDIR>>/bin/default/source3/libtrusts-util-samba4.so', '-Wl,-Bstatic', '-Wl,-Bdynamic', '-L/<<PKGBUILDDIR>>/bin/default/source4/cluster', '-L/<<PKGBUILDDIR>>/bin/default/source4/lib/socket', '-L/<<PKGBUILDDIR>>/bin/default/libcli/nbt', '-L/<<PKGBUILDDIR>>/bin/default/source4/lib/messaging', '-L/<<PKGBUILDDIR>>/bin/default/libcli/dns', '-L/<<PKGBUILDDIR>>/bin/default/source4/libcli/ldap', '-L/<<PKGBUILDDIR>>/bin/default/auth', '-L/<<PKGBUILDDIR>>/bin/default/lib/messaging', '-L/<<PKGBUILDDIR>>/bin/default/libcli/registry', '-L/<<PKGBUILDDIR>>/bin/default/libcli/smb', '-L/<<PKGBUILDDIR>>/bin/default/lib/addns', '-L/<<PKGBUILDDIR>>/bin/default/libcli/cldap', '-L/<<PKGBUILDDIR>>/bin/default/lib/socket', '-L/<<PKGBUILDDIR>>/bin/default/lib/krb5_wrap', '-L/<<PKGBUILDDIR>>/bin/default/source4/auth/kerberos', '-L/<<PKGBUILDDIR>>/bin/default/lib/tdb_wrap', '-L/<<PKGBUILDDIR>>/bin/default/third_party/heimdal_build', '-L/<<PKGBUILDDIR>>/bin/default/source4/lib/events', '-L/<<PKGBUILDDIR>>/bin/default/libcli/ldap', '-L/<<PKGBUILDDIR>>/bin/default/lib/ldb', '-L/<<PKGBUILDDIR>>/bin/default/libcli/security', '-L/<<PKGBUILDDIR>>/bin/default/auth/gensec', '-L/<<PKGBUILDDIR>>/bin/default/libcli/util', '-L/<<PKGBUILDDIR>>/bin/default/lib/param', '-L/<<PKGBUILDDIR>>/bin/default/libcli/named_pipe_auth', '-L/<<PKGBUILDDIR>>/bin/default/source4/librpc', '-L/<<PKGBUILDDIR>>/bin/default/lib', '-L/<<PKGBUILDDIR>>/bin/default/librpc', '-L/<<PKGBUILDDIR>>/bin/default/libcli/auth', '-L/<<PKGBUILDDIR>>/bin/default/lib/util', '-L/<<PKGBUILDDIR>>/bin/default/lib/dbwrap', '-L/<<PKGBUILDDIR>>/bin/default/lib/ldb-samba', '-L/<<PKGBUILDDIR>>/bin/default/libds/common', '-L/<<PKGBUILDDIR>>/bin/default/lib/replace', '-L/<<PKGBUILDDIR>>/bin/default/auth/credentials', '-L/<<PKGBUILDDIR>>/bin/default/source4/dsdb', '-L/<<PKGBUILDDIR>>/bin/default/nsswitch/libwbclient', '-L/<<PKGBUILDDIR>>/bin/default/source3', '-L/usr/local/lib', '-L/usr/local/lib', '-lsamba-passdb', '-lmsrpc3-samba4', '-llibcli-netlogon3-samba4', '-lwbclient', '-lsamdb-common-samba4', '-lsamba-credentials', '-lreplace-samba4', '-lsamba3-util-samba4', '-lsamdb', '-lflag-mapping-samba4', '-lldbsamba-samba4', '-ldbwrap-samba4', '-lsamba-modules-samba4', '-lcliauth-samba4', '-lsmbldap', '-lsamba-util', '-lsecrets3-samba4', '-lsmbldaphelper-samba4', '-lndr-standard', '-lsamba-sockets-samba4', '-lndr-nbt', '-lndr-samba4', '-lnpa-tstream-samba4', '-lsamba-hostconfig', '-lsamba-errors', '-lndr-samba-samba4', '-lndr', '-lgse-samba4', '-ltevent-util', '-lgensec-samba4', '-lutil-tdb-samba4', '-lsamba-security-samba4', '-llibsmb-samba4', '-ldcerpc-samba-samba4', '-ldcerpc-binding', '-lsmbconf', '-lldb', '-lcli-ldap-common-samba4', '-levents-samba4', '-lgssapi-samba4', '-ltdb-wrap-samba4', '-lcom-err-samba4', '-lauthkrb5-samba4', '-lkrb5samba-samba4', '-lsamba-debug-samba4', '-lkrb5-samba4', '-lasn1util-samba4', '-ltime-basic-samba4', '-lgenrand-samba4', '-lsocket-blocking-samba4', '-lutil-setid-samba4', '-lsys-rw-samba4', '-liov-buf-samba4', '-lCHARSET3-samba4', '-linterfaces-samba4', '-lroken-samba4', '-lndr-krb5pac', '-lserver-role-samba4', '-lcli-cldap-samba4', '-laddns-samba4', '-lcli-smb-common-samba4', '-lmessages-util-samba4', '-lutil-reg-samba4', '-lserver-id-db-samba4', '-lsamba-cluster-support-samba4', '-lmessages-dgm-samba4', '-lsmbd-shim-samba4', '-ltalloc-report-printf-samba4', '-lheimbase-samba4', '-lhcrypto-samba4', '-lasn1-samba4', '-lwind-samba4', '-lcommon-auth-samba4', '-lhx509-samba4', '-lcli-ldap-samba4', '-lclidns-samba4', '-lsmb-transport-samba4', '-lmsghdr-samba4', '-lMESSAGING-SEND-samba4', '-lcli-nbt-samba4', '-lnetif-samba4', '-lcluster-samba4', '-ltevent', '-ltalloc', '-lldap', '-ltalloc', '-ltdb', '-ldl', '-lbsd', '-lpthread', '-llber', '-lz', '-lresolv', '-lutil', '-lgnutls', '-licui18n', '-licuuc', '-licudata', '-lsystemd', '-ljansson', '-lcups', '-lcap', '-Wl,-z,relro', '-Wl,-z,now', '-Wl,-z,relro,-z,now', '-Wl,-no-undefined', '-Wl,--export-dynamic'] In file included from ../../third_party/heimdal/kdc/kdc_locl.h:46, from ../../third_party/heimdal/kdc/mssfu.c:34: third_party/heimdal/kdc/kdc-private.h:224:9: error: unknown type name ‘gss_OID_set’ 224 | gss_OID_set */*oidsp*/); | ^~~~~~~~~~~ In file included from ../../third_party/heimdal/kdc/mssfu.c:34: ../../third_party/heimdal/kdc/kdc_locl.h:116:5: error: unknown type name ‘gss_OID_set’ 116 | gss_OID_set gss_mechanisms_allowed; | ^~~~~~~~~~~ The full log can be obtained from here [1]. The issue can be reproduced on HEAD on the GCC porter box gcc202 [2]. I have tried bisecting the problem which lead to the following bisect log: glaubitz@gcc202:~/samba$ git bisect log git bisect start # bad: [5e00c230ec8d37044205a16c3a6ceaa64de382d0] py:gpo: Fix testing of 0x8000 bit git bisect bad 5e00c230ec8d37044205a16c3a6ceaa64de382d0 # good: [3fe82c204f0d88cb6db50b7bd1f798b591a918f8] VERSION: Disable GIT_SNAPSHOT for the 4.13.0 release. git bisect good 3fe82c204f0d88cb6db50b7bd1f798b591a918f8 # good: [8c86998910d0520a09dee729c2892f26dccc4b65] VERSION: Disable GIT_SNAPSHOT for the 4.13.0rc1 release. git bisect good 8c86998910d0520a09dee729c2892f26dccc4b65 # good: [fec996ff277479e5bac2159e2aab78d838d86b4c] samldb: Fix function name typo in error message git bisect good fec996ff277479e5bac2159e2aab78d838d86b4c # good: [ebc9137cee94dee9dcf0e47d5bc0dc83de7aaaa1] tests/krb5: Align PAC buffer checking to more closely match Windows with PacRequestorEnforcement=2 git bisect good ebc9137cee94dee9dcf0e47d5bc0dc83de7aaaa1 # bad: [0f5d7ff1a9fd14fd412b09883d413d1d660fa7be] s4:kdc: redirect pre-authentication failures to an RWDC git bisect bad 0f5d7ff1a9fd14fd412b09883d413d1d660fa7be # good: [358c59f51ab39175ffe72afdfc4c2e0ed23b5929] ctdb-recoverd: No longer take cluster lock during recovery git bisect good 358c59f51ab39175ffe72afdfc4c2e0ed23b5929 # bad: [c266ed40aeb1b1f59a1811cd4511e32e44a4a719] s3/libads: simplify storing existing ads->ldap.ss git bisect bad c266ed40aeb1b1f59a1811cd4511e32e44a4a719 # bad: [49d18f2d6e8872c2b0cbe2bf3324e7057c8438f4] s3:libads: Remove trailing spaces from sasl.c git bisect bad 49d18f2d6e8872c2b0cbe2bf3324e7057c8438f4 # good: [232a1fa46af0b05ba12bf5edc944caeb4f919c38] smbd: Fix a typo git bisect good 232a1fa46af0b05ba12bf5edc944caeb4f919c38 # bad: [c7bd176f4cb5d058337b64819858eca2764bd88e] s4:kdc: Move calls using the samba4 name to be right after each other git bisect bad c7bd176f4cb5d058337b64819858eca2764bd88e # bad: [6e8ac61b36ec74581fde8720107bce8971989015] tests: Update latin1 list and ignored file list for new Heimdal import git bisect bad 6e8ac61b36ec74581fde8720107bce8971989015 # good: [12ca34115eabbb430cd0b01afeaaebfac76174d3] lib: Remove unused asprintf_strupper_m() git bisect good 12ca34115eabbb430cd0b01afeaaebfac76174d3 # good: [5636bfa9a27707895c97a32b0836ab9801456499] netlogon.idl: Add FAST support bits git bisect good 5636bfa9a27707895c97a32b0836ab9801456499 # skip: [40b65c840e03bd5eb7f3b02fe80144650c63c005] s4:heimdal: import lorikeet-heimdal-202201172009 (commit 5a0b45cd723628b3690ea848548b05771c40f14e) git bisect skip 40b65c840e03bd5eb7f3b02fe80144650c63c005 # bad: [b2c96d927a661d5e830b271043a6a2be94d4c04d] s4:heimdal_build: changes required to build after import git bisect bad b2c96d927a661d5e830b271043a6a2be94d4c04d # good: [d2a3016a9c59f93f89cf4bb86d40938d56400453] s4:heimdal_build: include heimdal headers relative to heimdal_build git bisect good d2a3016a9c59f93f89cf4bb86d40938d56400453 # only skipped commits left to test # possible first bad commit: [b2c96d927a661d5e830b271043a6a2be94d4c04d] s4:heimdal_build: changes required to build after import # possible first bad commit: [40b65c840e03bd5eb7f3b02fe80144650c63c005] s4:heimdal: import lorikeet-heimdal-202201172009 (commit 5a0b45cd723628b3690ea848548b05771c40f14e) glaubitz@gcc202:~/samba$ However, that didn't really give me any clue what the problem was since the issue apparently was introduced by the update of the embedded heimdal code. > [1] https://buildd.debian.org/status/fetch.php?pkg=samba&arch=sparc64&ver=2%3A4.16.1%2Bdfsg-6&stamp=1653820067&raw=0 > [2] https://cfarm.tetaneutral.net/machines/list/
The issue is understood to be related to the set reduction used in samba_deps.py being non-determinately ordered between platforms (and indeed at all times unless PYTHONHASHSEED is set). This in turn changes the link and include order for the build. No developer resources have been so far available to address the core issues this has raised in the build system.
The bug title is a symptom (one of many), not a cause. The cause is, like Andrew correctly wrote, is the random order of everything on the compiler/linker command lines. And while this particular symptom is easy to deal with (one can just remove the one of the 3 gssapi.h files and the build will be successful), there are other symptoms which are much more difficult to fix and even to notice. One of possible outcomes of this non-deterministic order is a successful build with resulting executables being crashy in random places. And we've seen this on Debian already - before we know about PYTHONHASHSEED thing. While setting this variable fixed the build for us, it is actually not the right fix, because the order is *still* random (as seen by this report), it is just a bit more deterministic and the amount of randomness is a bit lower than before. It is *still* possible to get broken builds, I think.
Thanks for the detailed explanation to both of you. I guess that reducing the number of parallel jobs during build should help to work around the problem. I will give it a try.
Sadly no, the die is cast at the start of the script. Alternate values for PYTHONHASHSEED might help, but really we need someone to apply some determinism to the set manipulations.
I don't have enough knowlege of python. It should be easy to fix, - by using the current set/hash in parallel with a list/array (whatever these constructs are called as in python), - use the array for all real operations, and use set to check if a given element is already there before adding it to the array.
Adding this reference for an example of how this can - in theory - be worked around: https://lists.samba.org/archive/samba-technical/2022-May/137349.html
and another data point, https://lists.samba.org/archive/samba-technical/2022-November/137788.html . The real fix would be to stop adding all the directories on the way as -I include paths for the compiler.
I think the issue is that Python sets don't have a consistent order which is a known fact. And yes, when you start the interpreter, as Andrew mentioned it "casts the die" right then, which seems to determine the order throughout (until you restart the Python shell). A quick experiment, start with a list in a python repl shell: >>> l = ['a', 'b', 'c'] Then cast it to a set: >>> set(l) {'c', 'a', 'b'} >>> set(l) {'c', 'a', 'b'} >>> set(l) {'c', 'a', 'b'} Now every time you type set(l) you will get back the same result. UNLESS you restart the python repl shell and it "recasts the die" then you will start seeing a different order. >>> l = ['a', 'b', 'c'] >>> set(l) {'b', 'a', 'c'}
The set that Andrew named in 2022 can be pointed at precisely. It is inc_deps in build_includes(), buildtools/wafsamba/samba_deps.py: inc_deps = includes_objects(bld, self, set(), {}) ... for d in inc_deps: # unordered set of names inclist = getattr(t, 'samba_includes_extended', [])[:] if getattr(t, 'local_include', True): inclist.append('.') # own directory goes last includes_objects() returns a set of target names, so the order in which dependencies contribute -I flags is unspecified, and it differs by architecture and python version. PYTHONHASHSEED pins it for one interpreter only. How mixed this currently is, measured on the two Debian buildd logs of 2:4.24.6+dfsg-1 by counting compiler command lines that carry both -I.../heimdal/lib/gssapi and -I.../heimdal/lib: amd64 (build succeeded) 310 with lib/gssapi first, 1426 with lib first sparc64 (build failed) 143 with lib/gssapi first, 610 with lib first So the order is not even consistent within a single build - it varies per target, because each target gets its own set iteration. That is why this reproduces as "some architectures, some versions, some files". A separate instance of the same root cause, with a different symptom, is bug 16229: the heimdal crypto-headers.h in third_party/heimdal/include shadowing Samba's replacement in third_party/heimdal_build, which makes sparc64 fail with #error "need config.h" while compiling source4/kdc/sdb_to_hdb.c. Note that PYTHONHASHSEED=1, which buildtools/wafsamba/wscript:18 already requires, does not pin the order: with that seed set on both machines, hash('HEIMDAL_KRB5') is -5777948353383250625 on amd64 and 7800708043191837418 on a sparc64 T4, so the sets iterate differently and the -I order differs. Pinning the seed cannot fix this. Proposed for this bug: iterate in sorted order, and honour local_include_first for dependencies the way build_includes() already honours it for the target being built, so a dependency's own directory precedes the directories it borrows headers from. That does not by itself make any particular header choice correct - it makes the choice the same everywhere, so a wrong one fails on every architecture instead of one. Measured on amd64 against master, counting the 1025 targets that have both third_party/heimdal_build and third_party/heimdal/include on their include path, and how many search heimdal/include first: seed 1 2 3 4 5 6 7 8 9 10 11 12 baseline 0 0 662 0 891 1 1 0 0 381 628 3 with patch 0 0 0 0 0 0 0 0 0 0 0 0 and PYTHONHASHSEED=5, which fails to build on master, builds to completion with the patch applied. Given how many objects change -I order, this still needs a full autobuild before anyone trusts it. MR: https://gitlab.com/samba-team/samba/-/merge_requests/4685
Just a small clarification. What MR 4685 should do for this bug is make sparc64 resolve that pair the same way amd64 does, and amd64 builds. If the intent is to fix the gssapi.h pair itself rather than just pin the order, the same approach as MR 4684 would apply: whichever directory is meant to shadow the other has to be searched first by construction, not by luck.
An update from my MR !4685, attempting to make dependency include order deterministic. CI wasn't to happy about that because it bruke the samba-fuzz build when 13 headers names resolved to a different file. I changed it to a draft now. The gssapi.h row is this bug: sorted order selects third_party/heimdal/lib/gssapi - the wrapper whose entire body is `#include <gssapi/gssapi.h>` - over the real third_party/heimdal/lib/gssapi/gssapi. So the set ordering Andrew identified in 2022 is the mechanism, but pinning it is not by itself a fix: the build currently depends on 13 duplicate header names resolving a particular way, and any global reordering changes some of them. Whoever takes this will need to decide, per name, which copy is intended ,or remove the ambiguity, the way third_party/heimdal_build/crypto-headers.h is meant to shadow the vendored copy in !4684.