Bug 14209 - samba-tool fails with uncaught exception - bad call flags on Python 3.8
Summary: samba-tool fails with uncaught exception - bad call flags on Python 3.8
Status: RESOLVED FIXED
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: Python (show other bugs)
Version: 4.10.10
Hardware: All All
: P5 critical (vote)
Target Milestone: ---
Assignee: Karolin Seeger
QA Contact: Samba QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-11-29 17:24 UTC by Val Kulkov
Modified: 2020-01-14 08:20 UTC (History)
6 users (show)

See Also:


Attachments
patch in the missing METH_VARARGS (1.08 KB, patch)
2019-12-02 22:20 UTC, Douglas Bagnall
no flags Details
patch for 4.10 (1.22 KB, patch)
2019-12-04 21:53 UTC, Douglas Bagnall
abartlet: review+
Details
patch for 4.11 (1.20 KB, patch)
2019-12-04 21:54 UTC, Douglas Bagnall
abartlet: review+
Details
Proposed patch for pygensec.c (887 bytes, patch)
2019-12-09 10:32 UTC, Torsten Fohrer
no flags Details
pygensec.c patch (1.38 KB, patch)
2019-12-15 16:05 UTC, Torsten Fohrer
kseeger: review? (abartlet)
dbagnall: review+
slow: review+
Details
pygensec patch for 4.11 (1.76 KB, patch)
2019-12-19 02:53 UTC, Douglas Bagnall
abartlet: review+
Details
pygensec patch for 4.10 (1.76 KB, patch)
2019-12-19 02:54 UTC, Douglas Bagnall
abartlet: review+
Details
pygensec patch for 4.10 v2 (added BUG: line) (1.82 KB, patch)
2019-12-19 02:59 UTC, Douglas Bagnall
dbagnall: review? (slow)
abartlet: review+
Details
pygensec patch for 4.11 v2 (added BUG: line) (1.82 KB, patch)
2019-12-19 03:02 UTC, Douglas Bagnall
dbagnall: review? (slow)
abartlet: review+
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Val Kulkov 2019-11-29 17:24:34 UTC
samba-tool reports an uncaught exception and quits shortly after start:

[root@nuc2 samba]# samba-tool
ERROR(<class 'SystemError'>): uncaught exception - bad call flags
  File "/usr/bin/samba-tool", line 44, in <module>
    retval = cmd._run("samba-tool", subcommand, *args)
  File "/usr/lib/python3.8/site-packages/samba/netcmd/__init__.py", line 255, in _run
    cmd = self.subcommands[cmd_name]
  File "/usr/lib/python3.8/site-packages/samba/netcmd/main.py", line 35, in __getitem__
    self[attr] = getattr(__import__('samba.netcmd.%s' % package,
  File "/usr/lib/python3.8/site-packages/samba/netcmd/domain.py", line 44, in <module>
    from samba.join import join_RODC, join_DC, join_subdomain
  File "/usr/lib/python3.8/site-packages/samba/join.py", line 24, in <module>
    from samba import gensec, Ldb, drs_utils, arcfour_encrypt, string_to_byte_array


Some additional info:
[root@nuc2 samba]# uname -a
Linux nuc2 5.2.14-arch2-1-ARCH #1 SMP PREEMPT Thu Sep 12 10:42:38 UTC 2019 x86_64 GNU/Linux
[root@nuc2 samba]# pacman -Qs "^(samba|python3)$"
local/python 3.8.0-1
    Next generation of the python high-level scripting language
local/samba 4.10.10-2
    SMB Fileserver and AD Domain server

Someone else reported this issue at the Arch Linux bug tracker: https://bugs.archlinux.org/index.php?do=details&action=details.addvote&task_id=64537
Comment 1 Val Kulkov 2019-11-29 17:30:18 UTC
A somewhat similar issue was observed with libcomps by Redhat users:

https://bugzilla.redhat.com/show_bug.cgi?id=1734777#c3
https://github.com/rpm-software-management/libcomps/pull/50

A quote from the latter link:

In Python 3.7, import didn't check descriptor flags (METH_KEYWORDS):
these flags were only checked when the methods were called.

In Python 3.8, the flags are checked at soon as the module is
imported, which prevents the module to be imported.

As reported in https://bugs.archlinux.org/index.php?do=details&action=details.addvote&task_id=64537 downgrading to Python 3.7 resolves the issue. Therefore it appears likely that checking flags immediately at the module import in Python 3.8 can be the root cause of the issue.
Comment 2 Douglas Bagnall 2019-12-02 22:20:48 UTC
Created attachment 15660 [details]
patch in the missing METH_VARARGS

Thanks Val for the report and your work identifying the problem.

I think this patch should fix it.
Comment 3 Douglas Bagnall 2019-12-02 22:38:10 UTC
I guess this will need backporting to 4.10 and 4.11.
Comment 4 Douglas Bagnall 2019-12-04 21:53:26 UTC
Created attachment 15665 [details]
patch for 4.10
Comment 5 Douglas Bagnall 2019-12-04 21:54:00 UTC
Created attachment 15666 [details]
patch for 4.11
Comment 6 Douglas Bagnall 2019-12-04 21:57:02 UTC
Val Kulkov: are you able to confirm this patch works?

The patch for 4.10 is different from the others due to formatting changes.
Comment 7 Val Kulkov 2019-12-04 22:07:20 UTC
Douglas, until 2019-Dec-09 I will be unable to do anything unfortunately. Even after 2019-12-09 it may take a few days for me to confirm that the patch works. I am going to contact the maintainer of "samba" package on Arch Linux to see if they can do something in the interim.
Comment 8 Andrew Bartlett 2019-12-04 23:24:06 UTC
Assigning to Karolin for the next available releases.
Comment 9 Torsten Fohrer 2019-12-09 10:32:54 UTC
Created attachment 15673 [details]
Proposed patch for pygensec.c

Fix missing method type
Comment 10 Val Kulkov 2019-12-10 18:23:04 UTC
The patch in Comment 2 is not enough; samba-tool fails with:

ERROR(<class 'SystemError'>): uncaught exception - bad call flags
  File "bin/samba-tool", line 44, in <module>
    retval = cmd._run("samba-tool", subcommand, *args)
  File "bin/python/samba/netcmd/__init__.py", line 256, in _run
    cmd = self.subcommands[cmd_name]
  File "bin/python/samba/netcmd/main.py", line 35, in __getitem__
    self[attr] = getattr(__import__('samba.netcmd.%s' % package,
  File "bin/python/samba/netcmd/domain.py", line 44, in <module>
    from samba.join import join_RODC, join_DC, join_subdomain
  File "bin/python/samba/join.py", line 24, in <module>
    from samba import gensec, Ldb, drs_utils, arcfour_encrypt, string_to_byte_array

However, adding the patch in Comment 9 appears to fix the problem, at least in my case. Thanks Douglas and Torsten.
Comment 11 Douglas Bagnall 2019-12-11 05:08:56 UTC
Torsten,

Thank you!

Would you be able to attach a version of your patch with a commit message containing a "Signed-off-by:" line (using roughly `git add -p; git commit -s; git format-patch HEAD^`).

Also please decide whether you need to send in a developers certificate of origin, per https://www.samba.org/samba/devel/copyright-policy.html
Comment 12 Torsten Fohrer 2019-12-15 16:05:04 UTC
Created attachment 15683 [details]
pygensec.c patch

Signed Off/from git master
Comment 13 Ralph Böhme 2019-12-18 08:56:53 UTC
https://bugzilla.samba.org/attachment.cgi?id=15683 lgtm. Pushed to autobuild.
Comment 14 Val Kulkov 2019-12-19 01:23:28 UTC
No backporting to 4.10 and 4.11?
Comment 15 Douglas Bagnall 2019-12-19 02:53:10 UTC
Created attachment 15707 [details]
pygensec patch for 4.11
Comment 16 Douglas Bagnall 2019-12-19 02:54:11 UTC
Created attachment 15709 [details]
pygensec patch for 4.10

pygensec patch for 4.10
Comment 17 Andrew Bartlett 2019-12-19 02:56:14 UTC
G'Day Karolin,

Please push both the original and pygensec patches to 4.10 and 4.11.

Thanks!
Comment 18 Douglas Bagnall 2019-12-19 02:59:00 UTC
Created attachment 15710 [details]
pygensec patch for 4.10 v2 (added BUG: line)
Comment 19 Douglas Bagnall 2019-12-19 03:02:58 UTC
Created attachment 15711 [details]
pygensec patch for 4.11 v2 (added BUG: line)

Val: our process involves getting the patches into master before generating backports so that we can cross-reference with the master commit id.

(Our process also sometimes involves forgetting to do this, so thanks for the reminder).
Comment 20 Andrew Bartlett 2019-12-19 03:06:12 UTC
Comment on attachment 15710 [details]
pygensec patch for 4.10 v2 (added BUG: line)

Thanks for the BUG line.
Comment 21 Karolin Seeger 2019-12-19 07:57:49 UTC
(In reply to Andrew Bartlett from comment #17)
Pushed both patches to autobuild-v4-{11,10}-test.
Comment 22 Karolin Seeger 2020-01-14 08:20:08 UTC
(In reply to Karolin Seeger from comment #21)
Pushed to both branches.
Closing out bug report.

Thanks!