It seems the behaviour to fail the FSCTL_DUP_EXTENTS_TO_FILE if the ioctl() failed was never really implemented. Commit 373d3015bc7c42d8ab28e72cd76806cce1d9c78f probably should have checked VFS_COPY_CHUNK_FL_MUST_CLONE in vfs_btrfs.c, but it seems this was overlooked. After changing the VFS API from copy-chunk to offload read/write the check is still missing from vfs_default offload-write implementation.
Have patch, need bugnumber.