Bug 13777 - Configure with python2 fails
Summary: Configure with python2 fails
Status: RESOLVED FIXED
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: Build (show other bugs)
Version: 4.8.0rc2
Hardware: All All
: P5 regression (vote)
Target Milestone: 4.10
Assignee: Karolin Seeger
QA Contact: Samba QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-02-06 12:01 UTC by Karolin Seeger
Modified: 2019-03-04 10:23 UTC (History)
3 users (show)

See Also:


Attachments
maybe patch for master (7.57 KB, patch)
2019-02-06 15:32 UTC, Noel Power
no flags Details
master patch (7.57 KB, patch)
2019-02-06 16:44 UTC, Noel Power
no flags Details
4.10 backport (7.68 KB, patch)
2019-02-14 10:17 UTC, Noel Power
metze: review+
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Karolin Seeger 2019-02-06 12:01:20 UTC
Configure with python2 fails:



 Checking getconf LFS_CFLAGS                                  : -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 
Checking getconf large file support flags work               : ok 
u'-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'
flag[:2]=[u'-D']
[u'_LARGEFILE_SOURCE']
Traceback (most recent call last):
  File "/root/build/4.10.0rc2-1/BUILD/samba-4.10.0rc2/third_party/waf/waflib/Scripting.py", line 158, in waf_entry_point
    run_commands()
  File "/root/build/4.10.0rc2-1/BUILD/samba-4.10.0rc2/third_party/waf/waflib/Scripting.py", line 251, in run_commands
    ctx = run_command(cmd_name)
  File "/root/build/4.10.0rc2-1/BUILD/samba-4.10.0rc2/third_party/waf/waflib/Scripting.py", line 235, in run_command
    ctx.execute()
  File "/root/build/4.10.0rc2-1/BUILD/samba-4.10.0rc2/third_party/waf/waflib/Configure.py", line 159, in execute
    super(ConfigurationContext, self).execute()
  File "/root/build/4.10.0rc2-1/BUILD/samba-4.10.0rc2/third_party/waf/waflib/Context.py", line 204, in execute
    self.recurse([os.path.dirname(g_module.root_path)])
  File "/root/build/4.10.0rc2-1/BUILD/samba-4.10.0rc2/third_party/waf/waflib/Context.py", line 286, in recurse
    user_function(self)
  File "/root/build/4.10.0rc2-1/BUILD/samba-4.10.0rc2/wscript", line 142, in configure
    conf.RECURSE('lib/replace')
  File "./buildtools/wafsamba/samba_utils.py", line 33, in fun
    return f(*k, **kw)
  File "./buildtools/wafsamba/samba_utils.py", line 436, in RECURSE
    return ctx.recurse(relpath)
  File "/root/build/4.10.0rc2-1/BUILD/samba-4.10.0rc2/third_party/waf/waflib/Context.py", line 286, in recurse
    user_function(self)
  File "/root/build/4.10.0rc2-1/BUILD/samba-4.10.0rc2/third_party/waf/waflib/Utils.py", line 816, in wrap
    ret = fun(*k)
  File "/root/build/4.10.0rc2-1/BUILD/samba-4.10.0rc2/lib/replace/wscript", line 30, in configure
    conf.RECURSE('buildtools/wafsamba')
  File "./buildtools/wafsamba/samba_utils.py", line 33, in fun
    return f(*k, **kw)
  File "./buildtools/wafsamba/samba_utils.py", line 436, in RECURSE
    return ctx.recurse(relpath)
  File "/root/build/4.10.0rc2-1/BUILD/samba-4.10.0rc2/third_party/waf/waflib/Context.py", line 286, in recurse
    user_function(self)
File "/root/build/4.10.0rc2-1/BUILD/samba-4.10.0rc2/third_party/waf/waflib/Utils.py", line 816, in wrap
    ret = fun(*k)
  File "/root/build/4.10.0rc2-1/BUILD/samba-4.10.0rc2/buildtools/wafsamba/wscript", line 481, in configure
    if not conf.CHECK_LARGEFILE():
  File "./buildtools/wafsamba/samba_utils.py", line 33, in fun
    return f(*k, **kw)
  File "./buildtools/wafsamba/samba_conftests.py", line 103, in CHECK_LARGEFILE
    conf.DEFINE(flag_split[0], '1')
  File "./buildtools/wafsamba/samba_utils.py", line 33, in fun
    return f(*k, **kw)
  File "./buildtools/wafsamba/samba_autoconf.py", line 20, in DEFINE
    conf.define(d, v, quote=quote)
  File "./buildtools/wafsamba/samba_utils.py", line 33, in fun
    return f(*k, **kw)
  File "./buildtools/wafsamba/samba_waf18.py", line 89, in define
    assert key and isinstance(key, str)
AssertionError
Comment 1 Karolin Seeger 2019-02-06 12:04:23 UTC
Re-assiging to Noel.
Comment 2 Stefan Metzmacher 2019-02-06 12:11:51 UTC
I noticed that this is caused by always using .decode('utf8').

git grep decode buildtools/
buildtools/wafsamba/samba_abi.py:    sigs = Utils.cmd_output([abi_gen, libpath]).decode('utf8')
buildtools/wafsamba/samba_conftests.py:        ret = Utils.cmd_output(cmd).decode('utf8')
buildtools/wafsamba/samba_conftests.py:        out = Utils.cmd_output(cmd).decode('utf8').split('\n')
buildtools/wafsamba/samba_cross.py:                ans = (p.returncode, ce_out.decode('utf8'))
buildtools/wafsamba/samba_dist.py:    return Utils.cmd_output(ls_files_cmd, cwd=cwd, env=env).decode('utf8').split('\n')
buildtools/wafsamba/samba_perl.py:            output = output.decode('utf8')
buildtools/wafsamba/samba_version.py:    git = Utils.cmd_output(env.GIT + ' show --pretty=format:"%h%n%ct%n%H%n%cd" --stat HEAD', silent=True, env=environ).decode('utf8')


Looking at python/samba/compat.py I think we need to use something like
get_string(), which avoids decode('utf8') unless we're using python3.

The specific difference is that 'getconf LFS_CFLAGS' returns a non empty
string on e.g. on rhel6-i386.

But the same can happen with any other usage of .decode('utf8')


I noticed this in third_party/waf/waflib/fixpy2.py
code = code.replace(".decode(sys.stdout.encoding or'latin-1',errors='replace')", '')
which seems to remove .decode(sys.stdout.encoding or'latin-1',errors='replace')
with nothing when executing in python2.
Comment 3 Noel Power 2019-02-06 14:20:35 UTC
This looked familiar

https://lists.samba.org/archive/samba-technical/2019-January/thread.html#132069
should fix this (it needs a second reviewer) Can you verify it works
Comment 4 Stefan Metzmacher 2019-02-06 14:27:39 UTC
(In reply to Noel Power from comment #3)

No, that just fix the specific problem not all of them!

We just should not use .decode('utf-8') when using python2.
Comment 5 Noel Power 2019-02-06 14:36:40 UTC
(In reply to Stefan Metzmacher from comment #4)
To be honest I would have liked not to change more than necessary waf (especially for py2)

git grep isinstance buildtools/ | grep str 
buildtools/wafsamba/samba_deps.py:            if not isinstance(s, str):
buildtools/wafsamba/samba_dist.py:    if not isinstance(appname, str) or not appname:
buildtools/wafsamba/samba_perl.py:        if not isinstance(output, str):
buildtools/wafsamba/samba_utils.py:    if isinstance(str, list):
buildtools/wafsamba/samba_utils.py:    if isinstance(varstr, list):
buildtools/wafsamba/samba_utils.py:    if not isinstance(varstr, str):
buildtools/wafsamba/samba_utils.py:        if not isinstance(interp, str):
buildtools/wafsamba/samba_version.py:            if isinstance(value, string_types):
buildtools/wafsamba/samba_waf18.py:        if isinstance(xx, str):
buildtools/wafsamba/samba_waf18.py:   assert key and isinstance(key, str)
buildtools/wafsamba/samba_waf18.py:    assert key and isinstance(key, str)
buildtools/wafsamba/samba_waf18.py:    kw['shell'] = isinstance(cmd, str)
buildtools/wafsamba/wafsamba.py:        shell=isinstance(rule, str),
buildtools/wafsamba/wafsamba.py:    if not isinstance(file, str):

makes me think maybe you are right :/
Comment 6 Noel Power 2019-02-06 15:32:04 UTC
Created attachment 14825 [details]
maybe patch for master

lets see how this goes

https://gitlab.com/samba-team/devel/samba/pipelines/46402178
Comment 7 Noel Power 2019-02-06 16:44:21 UTC
Created attachment 14828 [details]
master patch

maybe
https://gitlab.com/samba-team/devel/samba/pipelines/46409483

will have better success
Comment 8 Stefan Metzmacher 2019-02-06 23:15:33 UTC
(In reply to Noel Power from comment #7)

Thanks Noel!

Karolin, please test that patch. If fixes your problem we should push it to master and backport it to 4.10.
Comment 9 Karolin Seeger 2019-02-08 08:15:54 UTC
(In reply to Stefan Metzmacher from comment #8)
Patch fixes the problem. :-)

Thanks!
Comment 10 Karolin Seeger 2019-02-08 09:21:56 UTC
Pushed to autobuild-master.
Comment 11 Noel Power 2019-02-14 10:17:19 UTC
Created attachment 14846 [details]
4.10 backport
Comment 12 Noel Power 2019-02-14 10:18:37 UTC
Comment on attachment 14846 [details]
4.10 backport

I forgot about this, here is the 4.10 backported fix
Comment 13 Karolin Seeger 2019-02-22 08:33:37 UTC
(In reply to Noel Power from comment #12)
Pushed to autobuild-v4-10-test.
Comment 14 Karolin Seeger 2019-03-04 10:23:55 UTC
(In reply to Karolin Seeger from comment #13)
Pushed to v4-10-test.
Closing out bug report.

Thanks!