System:
AIX 7100-04-01-1543
xlccmp.16.1.0             16.1.0.1    C     F    XL C compiler

during configure rpath ability is detected as working:
 Checking linker accepts ['-Wl,-R,.']        : yes
 Checking for rpath library support          : yes

build reports:
 ld: 0706-027 The -R libdir flag is ignored.
and rpath is not set in binaries

ld supports -R only if option -bsvr4 is set, otherwise option -R is ignored

But why does configure wrongly detect the abilities?

The check 'Checking linker accepts ['-Wl,-R,.']' succeeds because ld doesn't gives an error on -R (only warning). That is ok.

The check 'Checking for rpath library support' succeeds because it is broken to my understanding. On any platform I would say.

buildtools/wafsamba/samba_waf18.py:
        lib_node = bld.srcnode.make_node('libdir/liblc1.c')
        lib_node.parent.mkdir()
        lib_node.write('int lib_func(void) { return 42; }\n', 'w')
        main_node = bld.srcnode.make_node('main.c')
        main_node.write('int main(void) {return !(lib_func() == 42);}', 'w')
        linkflags = []
        if version_script:
            script = bld.srcnode.make_node('ldscript')
            script.write('TEST_1.0A2 { global: *; };\n', 'w')
            linkflags.append('-Wl,--version-script=%s' % script.abspath())
        bld(features='c cshlib', source=lib_node, target='lib1', linkflags=linkflags, name='lib1')
        o = bld(features='c cprogram', source=main_node, target='prog1', uselib_local='lib1')
        if rpath:
            o.rpath = [lib_node.parent.abspath()]
        def run_app(self):
             args = conf.SAMBA_CROSS_ARGS(msg=msg)
             env = dict(os.environ)
             env['LD_LIBRARY_PATH'] = self.inputs[0].parent.abspath() + os.pathsep + env.get('LD_LIBRARY_PATH', '')
             self.generator.bld.cmd_and_log([self.inputs[0].abspath()] + args, env=env)
        o.post()
        bld(rule=run_app, source=o.link_task.outputs[0])

it creates:
 libdir/liblc1.c
 main.c
compiles to:
 libdir/liblc1.c.1.o
 main.c.2.o
creates lib:
 liblib1.so
creates binary:
 prog1

And now checks if prog1 get the lib loaded. But the lib is located in the same directory as the binary! So this succeeds without a correct set rpath in the binary!

To fix I use the following patch which creates the lib in libdir/ (o.rpath setting should be made better, I'm not familar with WAF...)

--- buildtools/wafsamba/samba_waf18.py.orig	2019-04-25 13:45:49.000000000 +0200
+++ buildtools/wafsamba/samba_waf18.py	2019-04-25 15:05:52.000000000 +0200
@@ -212,10 +212,10 @@
              script = bld.srcnode.make_node('ldscript')
              script.write('TEST_1.0A2 { global: *; };\n', 'w')
              linkflags.append('-Wl,--version-script=%s' % script.abspath())
-         bld(features='c cshlib', source=lib_node, target='lib1', linkflags=linkflags, name='lib1')
+         bld(features='c cshlib', source=lib_node, target='libdir/lib1', linkflags=linkflags, name='lib1')
          o = bld(features='c cprogram', source=main_node, target='prog1', uselib_local='lib1')
          if rpath:
-             o.rpath = [lib_node.parent.abspath()]
+             o.rpath = [lib_node.parent.parent.abspath() + '/testbuild/default/libdir']
          def run_app(self):
               args = conf.SAMBA_CROSS_ARGS(msg=msg)
               env = dict(os.environ)