In 10.5 Mac OS X added enough utmpx support to make it past the first two preprocessor checks around line 390 of src/smbd/utmp.c and on to the utmpx block which uses updwtmpx().  Unfortunately, as ./configure correctly surmised, 10.5 doesn't have that.  Seems to be a straightforward logic error in the preprocessor guard checks, fixable with the patch below.  (Not sure if this is Odd or not in the comment.)

--- smbd/utmp.c.orig    2008-01-09 20:25:53.000000000 -0500
+++ smbd/utmp.c 2008-01-09 20:30:08.000000000 -0500
@@ -398,6 +398,10 @@
        /* Odd.  Have utmpx.h but no "getutmpx()".  Drop to non-x stuff */
        DEBUG(1,("utmp_update: have utmpx.h but no getutmpx() function\n"));
        utmp_nox_update(u, claim);
+#elif !defined(HAVE_UPDWTMPX)
+    /* Have utmpx.h but no "updwtmpx()".  Drop to non-x stuff */
+    DEBUG(1,("utmp_update: have utmpx.h but no updwtmpx() function\n"));
+    utmp_nox_update(u, claim);
 #else
        pstring uname, wname;
        struct utmpx ux, *uxrc;