Created attachment 14774 [details] Patch: use a wrapper function to limit the returned value In talloc 21.1.15 and earlier versions, pytalloc_default_cmp and pytalloc_base_default_cmp implement Python comparison functions for talloc objects. These functions return the difference between two pointers. This is wrong for a couple of reasons: - These functions are meant to return only -1, 0 or 1 (see https://docs.python.org/2/c-api/typeobj.html). Recent versions of the Python runtime system enforce this and produce a warning -- "tp_compare didn't return -1, 0 or 1" -- when they're used. - The return value is an int, which is smaller than ptrdiff_t on some platforms. I've attached a patch which fixes this using a wrapper function.
Adam -- I think your patch *was* probably correct, but now that we don't support Python 2, and Python3 only has richcmp, not cmp, I think the correct thing is to remove this altogether. It is only referenced here: #if PY_MAJOR_VERSION >= 3 .tp_richcompare = pytalloc_default_richcmp, #else .tp_compare = pytalloc_default_cmp, #endif };
Just regarding the bug title which mentions Py#'s tp_richcompare: it is not affected, and returns a Python Boolean, like so: switch (op) { case Py_EQ: return PyBool_FromLong(ptr1 == ptr2); case Py_NE: return PyBool_FromLong(ptr1 != ptr2); case Py_LT: return PyBool_FromLong(ptr1 < ptr2); case Py_GT: return PyBool_FromLong(ptr1 > ptr2); case Py_LE: return PyBool_FromLong(ptr1 <= ptr2); case Py_GE: return PyBool_FromLong(ptr1 >= ptr2); }