Bug 14164 - ASN1 syntax error - build is using bundled asn1_compile when configured to use system
Summary: ASN1 syntax error - build is using bundled asn1_compile when configured to us...
Status: NEW
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: Build (show other bugs)
Version: 4.11.0
Hardware: All All
: P5 normal (vote)
Target Milestone: ---
Assignee: Samba QA Contact
QA Contact: Samba QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-10-18 17:36 UTC by Neil MacLeod
Modified: 2023-07-07 11:12 UTC (History)
3 users (show)

See Also:


Attachments
Full samba build log (168.91 KB, application/octet-stream)
2019-10-18 17:36 UTC, Neil MacLeod
no flags Details
a proposed fix for master (3.46 KB, patch)
2019-10-19 21:34 UTC, Uri Simchoni
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Neil MacLeod 2019-10-18 17:36:42 UTC
Created attachment 15553 [details]
Full samba build log

Cross-compiling for arm target on x86_64 host (Ubuntu 16.04), with toolchain based on gcc-9.2.0, glibc-2.3.0, heimdal-7.7.0, Python-3.7.5.

Samba 4.9.13 and this toolchain (using Python2 in place of Python3) builds fine when cross-compiling.

However, the latest Samba v4.11-test[1] with patches[2,3] and configuration[4] results in the following ASN1 syntax error failure (full log attached):

[246/412] Processing source4/heimdal/lib/asn1/rfc2459.asn1
/home/ubuntu/projects/LibreELEC.tv/build.LibreELEC-RPi2.arm-9.80-devel/samba-7f5334a92c4a378f88c0ee8c5fde46dd087a9dc0/bin/asn1_compile: 1: /home/ubuntu/projects/LibreELEC.tv/build.LibreELEC-RPi2.arm-9.80-devel/samba-7f5334a92c4a378f88c0ee8c5fde46dd087a9dc0/bin/asn1_compile: Syntax error: word unexpected (expecting ")")

/home/ubuntu/projects/LibreELEC.tv/build.LibreELEC-RPi2.arm-9.80-devel/samba-7f5334a92c4a378f88c0ee8c5fde46dd087a9dc0/bin/asn1_compile: 1: /home/ubuntu/projects/LibreELEC.tv/build.LibreELEC-RPi2.arm-9.80-devel/samba-7f5334a92c4a378f88c0ee8c5fde46dd087a9dc0/bin/asn1_compile: Syntax error: word unexpected (expecting ")")

/home/ubuntu/projects/LibreELEC.tv/build.LibreELEC-RPi2.arm-9.80-devel/samba-7f5334a92c4a378f88c0ee8c5fde46dd087a9dc0/bin/asn1_compile: 1: /home/ubuntu/projects/LibreELEC.tv/build.LibreELEC-RPi2.arm-9.80-devel/samba-7f5334a92c4a378f88c0ee8c5fde46dd087a9dc0/bin/asn1_compile: Syntax error: word unexpected (expecting ")")

/home/ubuntu/projects/LibreELEC.tv/build.LibreELEC-RPi2.arm-9.80-devel/samba-7f5334a92c4a378f88c0ee8c5fde46dd087a9dc0/bin/asn1_compile: 1: /home/ubuntu/projects/LibreELEC.tv/build.LibreELEC-RPi2.arm-9.80-devel/samba-7f5334a92c4a378f88c0ee8c5fde46dd087a9dc0/bin/asn1_compile: Syntax error: word unexpected (expecting ")")

/home/ubuntu/projects/LibreELEC.tv/build.LibreELEC-RPi2.arm-9.80-devel/samba-7f5334a92c4a378f88c0ee8c5fde46dd087a9dc0/bin/asn1_compile: 1: /home/ubuntu/projects/LibreELEC.tv/build.LibreELEC-RPi2.arm-9.80-devel/samba-7f5334a92c4a378f88c0ee8c5fde46dd087a9dc0/bin/asn1_compile: Syntax error: word unexpected (expecting ")")

/home/ubuntu/projects/LibreELEC.tv/build.LibreELEC-RPi2.arm-9.80-devel/samba-7f5334a92c4a378f88c0ee8c5fde46dd087a9dc0/bin/asn1_compile: 1: /home/ubuntu/projects/LibreELEC.tv/build.LibreELEC-RPi2.arm-9.80-devel/samba-7f5334a92c4a378f88c0ee8c5fde46dd087a9dc0/bin/asn1_compile: Syntax error: word unexpected (expecting ")")

Waf: Leaving directory `/home/ubuntu/projects/LibreELEC.tv/build.LibreELEC-RPi2.arm-9.80-devel/samba-7f5334a92c4a378f88c0ee8c5fde46dd087a9dc0/bin/default'
Build failed
 -> task in 'HEIMDAL_KX509_ASN1_ASN1' failed with exit status 2 (run with -v to display more information)
 -> task in 'HEIMDAL_HDB_ASN1_ASN1' failed with exit status 2 (run with -v to display more information)
 -> task in 'HEIMDAL_GSSAPI_ASN1_ASN1' failed with exit status 2 (run with -v to display more information)
 -> task in 'HEIMDAL_SPNEGO_ASN1_ASN1' failed with exit status 2 (run with -v to display more information)
 -> task in 'HEIMDAL_RFC2459_ASN1_ASN1' failed with exit status 2 (run with -v to display more information)
 -> task in 'HEIMDAL_DIGEST_ASN1_ASN1' failed with exit status 2 (run with -v to display more information)


The most obvious issue is that the Samba build is using the *bundled* asn1_compile even though the build is configured to use the system version:

  --bundled-libraries='ALL,!asn1_compile,!compile_et,!zlib' \

I've tried removing the leading "ALL," in case that is confusing Samba but it makes no difference, I have the same failure with only "!asn1_compile,!compile_et".

When building 4.9.13 (same --bundled-libraries option) the following is logged:

Checking for program compile_et                                                                 : /home/neil/projects/scratch/alternates/LibreELEC.tv/build.LibreELEC-RPi2.arm-9.80-devel/toolchain/bin/compile_et
Checking for program asn1_compile                                                               : /home/neil/projects/scratch/alternates/LibreELEC.tv/build.LibreELEC-RPi2.arm-9.80-devel/toolchain/bin/asn1_compile

However I don't see this logged at all when building v4.11-test.

Anyone got any ideas? Is --bundled-libraries being ignored?

1. https://github.com/samba-team/samba/tree/7f5334a92c4a378f88c0ee8c5fde46dd087a9dc0
2. https://bugzilla.samba.org/show_bug.cgi?id=13846#c23
3. https://github.com/buildroot/buildroot/blob/8b11b96f41a6ffa76556c9bf03a863955871ee57/package/samba4/0006-heimdal_build-wscript_build-do-not-add-host-include-.patch
4. https://github.com/LibreELEC/LibreELEC.tv/blob/f35b9f6b8ed4445edff20630cc39b25e47c428fb/packages/network/samba/package.mk
Comment 1 Uri Simchoni 2019-10-18 20:22:25 UTC
That looks like a degradation in 4.11 (unlike some other cross-compile-related issues that came with 4.10.x) - I cross-built Samba 4.10.8 using buildroot.org - worked fine. Then used the buildroot recipe for Samba master - got this asn1_compile error. In my case it just wouldn't run asn1_compile because I was building for AArch64:

bin/sh: /home/vagrant/buildroot/output/build/samba4-25b1e478ec5e8ac40d41013a824311824482fa67/bin/asn1_compile: cannot execute binary file: Exec format error

Will look further into this...
Comment 2 Neil MacLeod 2019-10-18 21:55:23 UTC
Yes! Finally!! :) I can confirm that samba-4.10.9 + buildroot patches[1] (all of them, in place of the patch from [2]) is now building for me.

Previously I tested building samba-4.10.0 with the various patches discussed in these bugs, but had no success - maybe it was because at the time I was using Python-2.7.13. Or the wrong version of the patches. I also didn't have the Heimdal buildroot work-around patch (a ping on my bug would have been nice, to let me know!)

Unfortunately I've now run into #14132[1] so can't actually install anything I've built, but - at least it now builds! :)

Thanks for the pointers Uri. Please ping me if you discover anything about the ASN1 regression and want me to test it. Who knows, maybe 4.11.x installs stuff too. :)

1. https://github.com/buildroot/buildroot/tree/8b11b96f41a6ffa76556c9bf03a863955871ee57/package/samba4
2. https://bugzilla.samba.org/show_bug.cgi?id=13846#c23
3. https://bugzilla.samba.org/show_bug.cgi?id=14132
Comment 3 Neil MacLeod 2019-10-18 22:11:00 UTC
Yes, on further investigation, the asn1_compile binary being used by 4.11.x is the 32-bit arm target, which is of course a bit useless on my x86_64 build host, and probably explains all the syntax errors... not that the build should be creating/using this binary, as it should be using the x86_64 system binary!

$ file samba-4.11.1/bin/default/source4/heimdal_build/asn1_compile
samba-4.11.1/bin/default/source4/heimdal_build/asn1_compile: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-, for GNU/Linux 4.4.0, not stripped
Comment 4 Neil MacLeod 2019-10-19 06:53:35 UTC
Presumably this is only an issue because the 4.11.x build is no longer respecting !asn1_compile (etc.) in --bundled-libraries - somewhere a check has been dropped or skipped as the build has been told to use the system binary, and to not build/use the bundled version (which will never work when cross-compiling - that's why we use !asn1_compile).

If the --bundled-libraries check is fixed so it works as it did in 4.10.9 then this building & executing target binary on host issue should go away automatically.
Comment 5 Uri Simchoni 2019-10-19 20:49:30 UTC
That seems to be a regression of 8061983d4882f3ba3f12da71443b035d7b672eec
Comment 6 Uri Simchoni 2019-10-19 21:34:19 UTC
Created attachment 15558 [details]
a proposed fix for master
Comment 7 Uri Simchoni 2019-10-19 21:41:40 UTC
Neil,

Can you please test the attached fix for Samba 4.11? I was able to get past the asn1_compile hurdle on master branch and my buildroot setup using this patch (didn't get to a full build due to some unrelated header issue, but that may be a master-only degradation).

Thanks,
Uri.
Comment 8 Uri Simchoni 2019-10-19 23:22:17 UTC
Also, if you do get to test this patch on 4.11, can you use the cross-compilation fix from https://bugzilla.samba.org/show_bug.cgi?id=13846#c23 instead of the buildroot cross-compilation fix? That would validate this fix and allow me to push it upstream.

Thanks,
Uri.
Comment 9 Neil MacLeod 2019-10-20 00:04:34 UTC
Hi Uri

Many thanks, and I have good news!

I managed to cross-compile 4.11.1 x86_64->arm/32-bit with the following 4 patches:

1. https://bugzilla.samba.org/show_bug.cgi?id=14164#c6 (ASN1 fix)
2. https://bugzilla.samba.org/show_bug.cgi?id=13846#c23 (cross-answers fix)
3. https://bugzilla.samba.org/show_bug.cgi?id=13856#c11 (buildroot Heimdal cross-compile fix)
4. https://bugzilla.samba.org/show_bug.cgi?id=13856#c13 (fix "link redeclared" issue)

Without the 3rd patch I get the "cross-compile badness" error due to /usr/include/heimdal being included.
Without the 4th patch I get the "link redeclared" issue.

The patch 3 and 4 failures are both bug #13856.

Unfortunately Samba 4.11.1 doesn't actually install anything (bug #14132), but we're getting there... :)

Thanks
Neil
Comment 10 Andrew Bartlett 2019-10-20 00:35:38 UTC
We shouldn't be using a system asn1_compile to build parts of heimdal, the idea was to build those with the host C compiler and then execute those tools as part of the build.

otherwise we end up in unpredictable territory as asn1_compile and compile_et (less likely) could well generate files that rely on aspects of the matching library.  If those get out of sync then there could be trouble.

So the fix is to make use_hostcc work properly. 

Sorry!
Comment 11 Andrew Bartlett 2019-10-20 00:50:43 UTC
(In reply to Andrew Bartlett from comment #10)
I particular it appears that this allows the use of an old, unused upstream compile_et from MIT kerberos to compile these for Heimdal, triggering other workarounds. 

For example the issue raised in 
https://bugzilla.samba.org/show_bug.cgi?id=13856#c13 appears to be from the MIT compile_et removed in this commit:

commit 939bb6c500aba6359f80bd27ba6269c23168fc16
Author: Greg Hudson <ghudson@mit.edu>
Date:   Tue Apr 29 11:03:03 2014 -0400

    Remove compile_et.c
    
    We have used awk scripts in preference to the C version of compile_et
    since 1994 (r4426).  Get rid of the sources for the C version and all
    of the Makefile.in references to them.
Comment 12 Rudi Heitbaum 2023-07-07 11:12:05 UTC
since 4.18
A) the patch no longer applies as the lines from wscript_configure_system_heimdal  have been removed from wscript_configure_system_heimdal
B) we are still carrying the wscript_configure_embedded_heimdal insertions of the patch for the aarch64 and arm cross compile build on x86_64