Bug 14424 - crypt_r() on CentOS 7 can return with errno to EISDIR or ENOENT due to using NSS
Summary: crypt_r() on CentOS 7 can return with errno to EISDIR or ENOENT due to using NSS
Status: RESOLVED FIXED
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: AD: LDB/DSDB/SAMDB (show other bugs)
Version: 4.12.3
Hardware: All All
: P5 normal (vote)
Target Milestone: ---
Assignee: Karolin Seeger
QA Contact: Samba QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-06-30 09:43 UTC by Andrew Bartlett
Modified: 2020-07-09 07:35 UTC (History)
4 users (show)

See Also:


Attachments
backported patch for 4.11 (30.65 KB, patch)
2020-07-01 23:31 UTC, Andrew Bartlett
dbagnall: review+
abartlet: ci-passed+
Details
backported patch for 4.12 (30.65 KB, patch)
2020-07-01 23:33 UTC, Andrew Bartlett
dbagnall: review+
abartlet: ci-passed+
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Andrew Bartlett 2020-06-30 09:43:32 UTC
Samba's password_hash.c code has:

	/*
	 * Relies on the assertion that cleartext_utf8->data is a zero
	 * terminated UTF-8 string
	 */
	errno = 0;
#ifdef HAVE_CRYPT_R
	hash = crypt_r((char *)io->n.cleartext_utf8->data, cmd, &crypt_data);
#else
	/*
	 * No crypt_r falling back to crypt, which is NOT thread safe
	 * Thread safety MT-Unsafe race:crypt
	 */
	hash = crypt((char *)io->n.cleartext_utf8->data, cmd);
#endif
	/* crypt_r and crypt may return a null pointer upon error depending on
	 * how libcrypt was configured. POSIX specifies returning a null
	 * pointer and setting errno. */
	if (hash == NULL || errno != 0) {

The problem here is that on CentOS 8, where crypt_r() is backed by NSS, the libfreeblpriv3.so tries to read /tmp as a file to update as an RNG cache during nspr init.  This system call fails, causing errno to be modified, and crypt_r() does not reset the errno.

Samba therefore fails to operate on any installation where 

"password hash userPassword schemes = CryptSHA256"

is set.
Comment 1 Kay Winkelhane 2020-06-30 09:50:36 UTC
Retype new SMB password:

Breakpoint 1, __crypt_r (key=0x5555566af560 "Dasistneu123/", salt=0x55555635d3d0 "$6$1TBpsoGFSIJNpZdw", data=0x7ffffffdcf90) at crypt-entry.c:86
86 {
(gdb) s
93 if (strncmp (md5_salt_prefix, salt, sizeof (md5_salt_prefix) - 1) == 0)
(gdb) n
86 {
(gdb) s
93 if (strncmp (md5_salt_prefix, salt, sizeof (md5_salt_prefix) - 1) == 0)
(gdb) n
86 {
(gdb) n
93 if (strncmp (md5_salt_prefix, salt, sizeof (md5_salt_prefix) - 1) == 0)
(gdb) n
106 if (strncmp (sha256_salt_prefix, salt, sizeof (sha256_salt_prefix) - 1) == 0)
(gdb) n
111 if (strncmp (sha512_salt_prefix, salt, sizeof (sha512_salt_prefix) - 1) == 0)
(gdb) n
112 return __sha512_crypt_r (key, salt, (char *) data,
(gdb) s
__sha512_crypt_r (key=key@entry=0x5555566af560 "Dasistneu123/", salt=salt@entry=0x55555635d3d0 "$6$1TBpsoGFSIJNpZdw", buffer=0x7ffffffdcf90 "") at sha512-crypt.c:127
127 if (strncmp (sha512_salt_prefix, salt, sizeof (sha512_salt_prefix) - 1) == 0)
(gdb) n
129 salt += sizeof (sha512_salt_prefix) - 1;
(gdb) n
105 {
(gdb) n
127 if (strncmp (sha512_salt_prefix, salt, sizeof (sha512_salt_prefix) - 1) == 0)
(gdb) n
105 {
(gdb) n
131 if (strncmp (salt, sha512_rounds_prefix, sizeof (sha512_rounds_prefix) - 1)
(gdb) n
120 bool rounds_custom = false;
(gdb) n
127 if (strncmp (sha512_salt_prefix, salt, sizeof (sha512_salt_prefix) - 1) == 0)
(gdb) n
119 size_t rounds = ROUNDS_DEFAULT;
(gdb) n
129 salt += sizeof (sha512_salt_prefix) - 1;
(gdb) n
131 if (strncmp (salt, sha512_rounds_prefix, sizeof (sha512_rounds_prefix) - 1)
(gdb) n
129 salt += sizeof (sha512_salt_prefix) - 1;
(gdb) n
131 if (strncmp (salt, sha512_rounds_prefix, sizeof (sha512_rounds_prefix) - 1)
(gdb) n
145 salt_len = MIN (strcspn (salt, "$"), SALT_LEN_MAX);
(gdb) n
146 key_len = strlen (key);
(gdb) n
148 if ((key - (char *) 0) % _alignof_ (uint64_t) != 0)
(gdb) n
146 key_len = strlen (key);
(gdb) n
148 if ((key - (char *) 0) % _alignof_ (uint64_t) != 0)
(gdb) n
122 char *free_key = NULL;
(gdb) n
121 size_t alloca_used = 0;
(gdb) n
114 char *copied_key = NULL;
(gdb) n
168 if ((salt - (char *) 0) % _alignof_ (uint64_t) != 0)
(gdb) n
115 char *copied_salt = NULL;
(gdb) n
168 if ((salt - (char *) 0) % _alignof_ (uint64_t) != 0)
(gdb) n
170 char *tmp = (char *) alloca (salt_len + _alignof_ (uint64_t));
(gdb) n
171 salt = copied_salt =
(gdb) n
170 char *tmp = (char *) alloca (salt_len + _alignof_ (uint64_t));
(gdb) n
173 - (tmp - (char *) 0) % _alignof_ (uint64_t),
(gdb) n
171 salt = copied_salt =
(gdb) n
180 NSSLOWInitContext *nss_ictx = NSSLOW_Init ();
(gdb) s
181 if (nss_ictx == NULL)
(gdb) n
180 NSSLOWInitContext *nss_ictx = NSSLOW_Init ();
(gdb) n
181 if (nss_ictx == NULL)
(gdb) n
194 sha512_init_ctx (&ctx, nss_ctx);
(gdb) n
197 sha512_process_bytes (key, key_len, &ctx, nss_ctx);
(gdb) n
201 sha512_process_bytes (salt, salt_len, &ctx, nss_ctx);
(gdb) n
206 sha512_init_ctx (&alt_ctx, nss_alt_ctx);
(gdb) n
209 sha512_process_bytes (key, key_len, &alt_ctx, nss_alt_ctx);
(gdb) n
212 sha512_process_bytes (salt, salt_len, &alt_ctx, nss_alt_ctx);
(gdb) n
215 sha512_process_bytes (key, key_len, &alt_ctx, nss_alt_ctx);
(gdb) n
219 sha512_finish_ctx (&alt_ctx, nss_alt_ctx, alt_result);
(gdb) n
222 for (cnt = key_len; cnt > 64; cnt -= 64)
(gdb) n
224 sha512_process_bytes (alt_result, cnt, &ctx, nss_ctx);
(gdb) n
228 for (cnt = key_len; cnt > 0; cnt >>= 1)
(gdb) n
222 for (cnt = key_len; cnt > 64; cnt -= 64)
(gdb) n
229 if ((cnt & 1) != 0)
(gdb) n
230 sha512_process_bytes (alt_result, 64, &ctx, nss_ctx);
(gdb) n
228 for (cnt = key_len; cnt > 0; cnt >>= 1)
(gdb) n
229 if ((cnt & 1) != 0)
(gdb) n
232 sha512_process_bytes (key, key_len, &ctx, nss_ctx);
(gdb) n
228 for (cnt = key_len; cnt > 0; cnt >>= 1)
(gdb) n
229 if ((cnt & 1) != 0)
(gdb) n
230 sha512_process_bytes (alt_result, 64, &ctx, nss_ctx);
(gdb) n
228 for (cnt = key_len; cnt > 0; cnt >>= 1)
(gdb) l
223 sha512_process_bytes (alt_result, 64, &ctx, nss_ctx);
224 sha512_process_bytes (alt_result, cnt, &ctx, nss_ctx);
225
226 /* Take the binary representation of the length of the key and for every
227 1 add the alternate sum, for every 0 the key. */
228 for (cnt = key_len; cnt > 0; cnt >>= 1)
229 if ((cnt & 1) != 0)
230 sha512_process_bytes (alt_result, 64, &ctx, nss_ctx);
231 else
232 sha512_process_bytes (key, key_len, &ctx, nss_ctx);
(gdb) l
233
234 /* Create intermediate result. */
235 sha512_finish_ctx (&ctx, nss_ctx, alt_result);
236
237 /* Start computation of P byte sequence. */
238 sha512_init_ctx (&alt_ctx, nss_alt_ctx);
239
240 /* For every character in the password add the entire password. */
241 for (cnt = 0; cnt < key_len; ++cnt)
242 sha512_process_bytes (key, key_len, &alt_ctx, nss_alt_ctx);
(gdb) l
243
244 /* Finish the digest. */
245 sha512_finish_ctx (&alt_ctx, nss_alt_ctx, temp_result);
246
247 /* Create byte sequence P. */
248 if (__libc_use_alloca (alloca_used + key_len))
249 cp = p_bytes = (char *) alloca (key_len);
250 else
251 {
252 free_pbytes = cp = p_bytes = (char *)malloc (key_len);
(gdb) l
253 if (free_pbytes == NULL)
254 {
255 free (free_key);
256 return NULL;
257 }
258 }
259
260 for (cnt = key_len; cnt >= 64; cnt -= 64)
261 cp = mempcpy (cp, temp_result, 64);
262 memcpy (cp, temp_result, cnt);
(gdb) l
263
264 /* Start computation of S byte sequence. */
265 sha512_init_ctx (&alt_ctx, nss_alt_ctx);
266
267 /* For every character in the password add the entire password. */
268 for (cnt = 0; cnt < 16 + alt_result[0]; ++cnt)
269 sha512_process_bytes (salt, salt_len, &alt_ctx, nss_alt_ctx);
270
271 /* Finish the digest. */
272 sha512_finish_ctx (&alt_ctx, nss_alt_ctx, temp_result);
(gdb) l
273
274 /* Create byte sequence S. */
275 cp = s_bytes = alloca (salt_len);
276 for (cnt = salt_len; cnt >= 64; cnt -= 64)
277 cp = mempcpy (cp, temp_result, 64);
278 memcpy (cp, temp_result, cnt);
279
280 /* Repeatedly run the collected hash value through SHA512 to burn
281 CPU cycles. */
282 for (cnt = 0; cnt < rounds; ++cnt)
(gdb) l
283 {
284 /* New context. */
285 sha512_init_ctx (&ctx, nss_ctx);
286
287 /* Add key or last result. */
288 if ((cnt & 1) != 0)
289 sha512_process_bytes (p_bytes, key_len, &ctx, nss_ctx);
290 else
291 sha512_process_bytes (alt_result, 64, &ctx, nss_ctx);
292
(gdb) l
293 /* Add salt for numbers not divisible by 3. */
294 if (cnt % 3 != 0)
295 sha512_process_bytes (s_bytes, salt_len, &ctx, nss_ctx);
296
297 /* Add key for numbers not divisible by 7. */
298 if (cnt % 7 != 0)
299 sha512_process_bytes (p_bytes, key_len, &ctx, nss_ctx);
300
301 /* Add key or last result. */
302 if ((cnt & 1) != 0)
(gdb) l
303 sha512_process_bytes (alt_result, 64, &ctx, nss_ctx);
304 else
305 sha512_process_bytes (p_bytes, key_len, &ctx, nss_ctx);
306
307 /* Create intermediate result. */
308 sha512_finish_ctx (&ctx, nss_ctx, alt_result);
309 }
310
311 #ifdef USE_NSS
312 /* Free libfreebl3 resources. */
(gdb) l
313 NSSLOW_Shutdown (nss_ictx);
314 #endif
315
316 /* Now we can construct the result string. It consists of three
317 parts. */
318 cp = __stpncpy (buffer, sha512_salt_prefix, MAX (0, buflen));
319 buflen -= sizeof (sha512_salt_prefix) - 1;
320
321 if (rounds_custom)
322 {
(gdb) ll
Undefined command: "ll". Try "help".
(gdb) l
323 int n = snprintf (cp, MAX (0, buflen), "%s%zu$",
324 sha512_rounds_prefix, rounds);
325 cp += n;
326 buflen -= n;
327 }
328
329 cp = __stpncpy (cp, salt, MIN ((size_t) MAX (0, buflen), salt_len));
330 buflen -= MIN ((size_t) MAX (0, buflen), salt_len);
331
332 if (buflen > 0)
(gdb) l
333 {
334 *cp++ = '$';
335 --buflen;
336 }
337
338 void b64_from_24bit (unsigned int b2, unsigned int b1, unsigned int b0,
339 int n)
340 {
341 unsigned int w = (b2 << 16) | (b1 << 8) | b0;
342 while (n-- > 0 && buflen > 0)
(gdb) l
343 {
344 *cp++ = b64t[w & 0x3f];
345 --buflen;
346 w >>= 6;
347 }
348 }
349
350 b64_from_24bit (alt_result[0], alt_result[21], alt_result[42], 4);
351 b64_from_24bit (alt_result[22], alt_result[43], alt_result[1], 4);
352 b64_from_24bit (alt_result[44], alt_result[2], alt_result[23], 4);
(gdb) l
353 b64_from_24bit (alt_result[3], alt_result[24], alt_result[45], 4);
354 b64_from_24bit (alt_result[25], alt_result[46], alt_result[4], 4);
355 b64_from_24bit (alt_result[47], alt_result[5], alt_result[26], 4);
356 b64_from_24bit (alt_result[6], alt_result[27], alt_result[48], 4);
357 b64_from_24bit (alt_result[28], alt_result[49], alt_result[7], 4);
358 b64_from_24bit (alt_result[50], alt_result[8], alt_result[29], 4);
359 b64_from_24bit (alt_result[9], alt_result[30], alt_result[51], 4);
360 b64_from_24bit (alt_result[31], alt_result[52], alt_result[10], 4);
361 b64_from_24bit (alt_result[53], alt_result[11], alt_result[32], 4);
362 b64_from_24bit (alt_result[12], alt_result[33], alt_result[54], 4);
(gdb) l
363 b64_from_24bit (alt_result[34], alt_result[55], alt_result[13], 4);
364 b64_from_24bit (alt_result[56], alt_result[14], alt_result[35], 4);
365 b64_from_24bit (alt_result[15], alt_result[36], alt_result[57], 4);
366 b64_from_24bit (alt_result[37], alt_result[58], alt_result[16], 4);
367 b64_from_24bit (alt_result[59], alt_result[17], alt_result[38], 4);
368 b64_from_24bit (alt_result[18], alt_result[39], alt_result[60], 4);
369 b64_from_24bit (alt_result[40], alt_result[61], alt_result[19], 4);
370 b64_from_24bit (alt_result[62], alt_result[20], alt_result[41], 4);
371 b64_from_24bit (0, 0, alt_result[63], 2);
372
(gdb) l
373 if (buflen <= 0)
374 {
375 __set_errno (ERANGE);
376 buffer = NULL;
377 }
378 else
379 *cp = '\0'; /* Terminate the string. */
380
381 /* Clear the buffer for the intermediate result so that people
382 attaching to processes or reading core dumps cannot get any
(gdb) l
383 information. We do it in this way to clear correct_words[]
384 inside the SHA512 implementation as well. */
385 #ifndef USE_NSS
386 __sha512_init_ctx (&ctx);
387 __sha512_finish_ctx (&ctx, alt_result);
388 memset (&ctx, '\0', sizeof (ctx));
389 memset (&alt_ctx, '\0', sizeof (alt_ctx));
390 #endif
391 memset (temp_result, '\0', sizeof (temp_result));
392 memset (p_bytes, '\0', key_len);
(gdb) l
393 memset (s_bytes, '\0', salt_len);
394 if (copied_key != NULL)
395 memset (copied_key, '\0', key_len);
396 if (copied_salt != NULL)
397 memset (copied_salt, '\0', salt_len);
398
399 free (free_key);
400 free (free_pbytes);
401 return buffer;
402 }
(gdb) break 373
Breakpoint 2 at 0x7fffeb6414ad: file sha512-crypt.c, line 373.
(gdb) c
Continuing.

Breakpoint 2, __sha512_crypt_r (key=<optimized out>, key@entry=0x5555566af560 "Dasistneu123/", salt=<optimized out>, salt@entry=0x55555635d3d0 "$6$1TBpsoGFSIJNpZdw",
buffer=0x7ffffffdcf90 "$6$1TBpsoGFSIJNpZdw$QLtD83pKm421EPQVUm4WV5YD.GOseIMKBaz9WJZ1QnJ3AO0te1vuSchHeVlu8TEahtjwNEvWGFBoP7p28RIOD") at sha512-crypt.c:373
373 if (buflen <= 0)
(gdb) n
371 b64_from_24bit (0, 0, alt_result[63], 2);
(gdb) n
373 if (buflen <= 0)
(gdb) n
379 *cp = '\0'; /* Terminate the string. */
(gdb) n
391 memset (temp_result, '\0', sizeof (temp_result));
(gdb) n
392 memset (p_bytes, '\0', key_len);
(gdb) n
391 memset (temp_result, '\0', sizeof (temp_result));
(gdb) n
392 memset (p_bytes, '\0', key_len);
(gdb) n
393 memset (s_bytes, '\0', salt_len);
(gdb) n
394 if (copied_key != NULL)
(gdb) n
396 if (copied_salt != NULL)
(gdb) n
397 memset (copied_salt, '\0', salt_len);
(gdb) n
399 free (free_key);
(gdb) n
400 free (free_pbytes);
(gdb) n
401 return buffer;
(gdb) p buffer
$1 = 0x7ffffffdcf90 "$6$1TBpsoGFSIJNpZdw$QLtD83pKm421EPQVUm4WV5YD.GOseIMKBaz9WJZ1QnJ3AO0te1vuSchHeVlu8TEahtjwNEvWGFBoP7p28RIOD1"
(gdb) finish
Run till exit from #0 __sha512_crypt_r (key=<optimized out>, key@entry=0x5555566af560 "Dasistneu123/", salt=<optimized out>, salt@entry=0x55555635d3d0 "$6$1TBpsoGFSIJNpZdw",
buffer=0x7ffffffdcf90 "$6$1TBpsoGFSIJNpZdw$QLtD83pKm421EPQVUm4WV5YD.GOseIMKBaz9WJZ1QnJ3AO0te1vuSchHeVlu8TEahtjwNEvWGFBoP7p28RIOD1") at sha512-crypt.c:401
__crypt_r (key=0x5555566af560 "Dasistneu123/", salt=<optimized out>, data=0x7ffffffdcf90) at crypt-entry.c:155
155 }
Value returned is $2 = 0x7ffffffdcf90 "$6$1TBpsoGFSIJNpZdw$QLtD83pKm421EPQVUm4WV5YD.GOseIMKBaz9WJZ1QnJ3AO0te1vuSchHeVlu8TEahtjwNEvWGFBoP7p28RIOD1"
(gdb) finish
Run till exit from #0 __crypt_r (key=0x5555566af560 "Dasistneu123/", salt=<optimized out>, data=0x7ffffffdcf90) at crypt-entry.c:155
0x00007fffd8208e9b in setup_primary_userPassword_hash () from /usr/lib64/samba/ldb/password_hash.so
Value returned is $3 = 0x7ffffffdcf90 "$6$1TBpsoGFSIJNpZdw$QLtD83pKm421EPQVUm4WV5YD.GOseIMKBaz9WJZ1QnJ3AO0te1vuSchHeVlu8TEahtjwNEvWGFBoP7p28RIOD1"
(gdb) finish
Run till exit from #0 0x00007fffd8208e9b in setup_primary_userPassword_hash () from /usr/lib64/samba/ldb/password_hash.so
0x00007fffd820a354 in setup_supplemental_field () from /usr/lib64/samba/ldb/password_hash.so
(gdb) c
Continuing.
{"timestamp": "2020-06-30T04:12:03.175301-0500", "type": "dsdbChange", "dsdbChange": {"version": {"major": 1, "minor": 0}, "statusCode": 1, "status": "Operations error", "operation": "Modify", "remoteAddress": null, "performedAsSystem": false, "userSid": "S-1-5-18", "dn": "CN=kwinkelhane,OU=Employees,OU=People,OU=Users,DC=labsso,DC=indeed,DC=net", "transactionId": "cca62198-c816-41a5-83d7-e4dc6985cbf0", "sessionId": "2874751f-6a38-4255-981a-e80065e338ee", "attributes": {"pwdLastSet": {"actions": [{"action": "replace", "values": [{"value": "132379809580000000"}]}]}, "clearTextPassword": {"actions": [{"action": "replace", "redacted": true}]}, "userAccountControl": {"actions": [{"action": "replace", "values": [{"value": "512"}]}]}}}}
{"timestamp": "2020-06-30T04:12:03.175497-0500", "type": "passwordChange", "passwordChange": {"version": {"major": 1, "minor": 1}, "eventId": 4724, "statusCode": 1, "status": "Operations error", "remoteAddress": null, "userSid": "S-1-5-18", "dn": "CN=kwinkelhane,OU=Employees,OU=People,OU=Users,DC=labsso,DC=indeed,DC=net", "action": "Reset", "transactionId": "cca62198-c816-41a5-83d7-e4dc6985cbf0", "sessionId": "2874751f-6a38-4255-981a-e80065e338ee"}}
{"timestamp": "2020-06-30T04:12:03.175549-0500", "type": "dsdbTransaction", "dsdbTransaction": {"version": {"major": 1, "minor": 0}, "action": "rollback", "transactionId": "cca62198-c816-41a5-83d7-e4dc6985cbf0", "duration": 964393036}}
Failed to modify account record CN=kwinkelhane,OU=Employees,OU=People,OU=Users,DC=labsso,DC=indeed,DC=net to set user attributes: setup_primary_userPassword: generation of a CryptSHA512 password hash failed: (Is a directory)
Failed to modify entry for user kwinkelhane.
[Inferior 1 (process 77194) exited with code 01]
(gdb) r
Starting program: /bin/smbpasswd kwinkelhane
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
New SMB password:
Retype new SMB password:

Breakpoint 1, __crypt_r (key=0x5555566af560 "Dasistneu123/", salt=0x55555635d3d0 "$6$99UAi/H8.lS/29ZP", data=0x7ffffffdcf90) at crypt-entry.c:86
86 {
(gdb) p errno
$4 = 0
(gdb) watch errno
Hardware watchpoint 3: errno
(gdb) finsih
Undefined command: "finsih". Try "help".
(gdb) finish
Run till exit from #0 __crypt_r (key=0x5555566af560 "Dasistneu123/", salt=0x55555635d3d0 "$6$99UAi/H8.lS/29ZP", data=0x7ffffffdcf90) at crypt-entry.c:86
Hardware watchpoint 3: errno

Old value = 0
New value = 2
0x00007fffea2322a0 in __GI___xstat (vers=<optimized out>, name=0x7fff8a2ea3d2 "/etc/utmp", buf=0x7ffffffd85c0) at ../sysdeps/unix/sysv/linux/wordsize-64/xstat.c:36
36 return INLINE_SYSCALL (stat, 2, name, buf);
(gdb) c
Continuing.
Hardware watchpoint 3: errno

Old value = 2
New value = 21
0x00007fffea259b22 in __read_chk (fd=23, buf=0x7ffffffd8650, nbytes=<optimized out>, buflen=<optimized out>) at read_chk.c:33
33 return INLINE_SYSCALL (read, 3, fd, buf, nbytes);
(gdb) up
#1 0x00007fff8a28c223 in RNG_FileUpdate () from /lib64/libfreeblpriv3.so
(gdb) up
#2 0x00007fff8a28c3d1 in RNG_SystemInfoForRNG () from /lib64/libfreeblpriv3.so
(gdb) up
#3 0x00007fff8a29e4b6 in rng_init () from /lib64/libfreeblpriv3.so
(gdb) up
#4 0x00007fffe7819c1a in PR_CallOnce () from /lib64/libnspr4.so
(gdb) c
Continuing.

Breakpoint 2, __sha512_crypt_r (key=<optimized out>, key@entry=0x5555566af560 "Dasistneu123/", salt=<optimized out>, salt@entry=0x55555635d3d0 "$6$99UAi/H8.lS/29ZP",
buffer=0x7ffffffdcf90 "$6$99UAi/H8.lS/29ZP$49HNGOprU7bjqchV09m/7J5cTd8Xy2uZEJ0QjIbpFm7xYkpYSMEUHn0V4bv9./qcVe5B07wBixlfinN5z2Yxe") at sha512-crypt.c:373
373 if (buflen <= 0)
(gdb) finish
Run till exit from #0 __sha512_crypt_r (key=<optimized out>, key@entry=0x5555566af560 "Dasistneu123/", salt=<optimized out>, salt@entry=0x55555635d3d0 "$6$99UAi/H8.lS/29ZP",
buffer=0x7ffffffdcf90 "$6$99UAi/H8.lS/29ZP$49HNGOprU7bjqchV09m/7J5cTd8Xy2uZEJ0QjIbpFm7xYkpYSMEUHn0V4bv9./qcVe5B07wBixlfinN5z2Yxe") at sha512-crypt.c:373
__crypt_r (key=0x5555566af560 "Dasistneu123/", salt=<optimized out>, data=0x7ffffffdcf90) at crypt-entry.c:155
155 }
Value returned is $5 = 0x7ffffffdcf90 "$6$99UAi/H8.lS/29ZP$49HNGOprU7bjqchV09m/7J5cTd8Xy2uZEJ0QjIbpFm7xYkpYSMEUHn0V4bv9./qcVe5B07wBixlfinN5z2Yxe/"
(gdb) p errno
$6 = 21
(gdb)
Comment 2 Andrew Bartlett 2020-06-30 10:03:16 UTC
Finding a line between:

https://manpages.debian.org/unstable/libcrypt-dev/crypt.3.en.html

Upon error, crypt_r, crypt_rn, and crypt_ra write an invalid hashed passphrase to the output field of their data argument, and crypt writes an invalid hash to its static storage area. This string will be shorter than 13 characters, will begin with a ‘*’, and will not compare equal to setting.

and 

https://linux.die.net/man/3/crypt

Return Value

On success, a pointer to the encrypted password is returned. On error, NULL is returned.

Will be quite difficult unless we can (say) get a fix into RHEL8 to set errno to 0 on success or into the NSS layers not to pollute errno somehow.
Comment 3 Andrew Bartlett 2020-06-30 23:16:40 UTC
I can reproduce on CentOS 7 (not 8, that was an error), but I get "No such file or directory" (ENOENT) rather than "Is a directory" (EISDIR).

The difference might be FIPS mode, which I can't easily do in a container.

We know that the EISDIR is part of the same problem because this was seen in an strace where NSPR (for NSS) opens /etc/utmp (which does not exist) for some reason, and this is left in errno.
Comment 5 Andrew Bartlett 2020-07-01 23:31:44 UTC
Created attachment 16104 [details]
backported patch for 4.11
Comment 6 Andrew Bartlett 2020-07-01 23:32:39 UTC
Comment on attachment 16104 [details]
backported patch for 4.11

Patch is under CI here: https://gitlab.com/samba-team/devel/samba/-/pipelines/162221804
Comment 7 Andrew Bartlett 2020-07-01 23:33:17 UTC
(In reply to Andrew Bartlett from comment #6)
Sorry, that is 4.12.  4.11 is under CI here:
https://gitlab.com/samba-team/devel/samba/-/pipelines/162221859
Comment 8 Andrew Bartlett 2020-07-01 23:33:51 UTC
Created attachment 16105 [details]
backported patch for 4.12
Comment 9 Andrew Bartlett 2020-07-01 23:48:45 UTC
Patch series in master ends with cabf873b75b1d4d456190358bc3ed051bca16978.
Comment 10 Andrew Bartlett 2020-07-02 02:04:08 UTC
Assigning to Karolin for 4.11.next and 4.12.next
Comment 11 Karolin Seeger 2020-07-03 09:41:50 UTC
(In reply to Andrew Bartlett from comment #10)
Pushed to autobuild-v4-{12,11}-test.
Comment 12 Karolin Seeger 2020-07-09 07:35:22 UTC
(In reply to Karolin Seeger from comment #11)
Pushed to both branches.
Closing out bug report.

Thanks!