Bug 13266 - Writing of files to Apple's new "AP" controller model SSD's takes 4x longer.
Summary: Writing of files to Apple's new "AP" controller model SSD's takes 4x longer.
Status: NEW
Alias: None
Product: rsync
Classification: Unclassified
Component: core (show other bugs)
Version: 3.1.3
Hardware: x64 Mac OS X
: P5 critical (vote)
Target Milestone: ---
Assignee: Wayne Davison
QA Contact: Rsync QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-02-11 16:53 UTC by Daniel Shepherd
Modified: 2018-05-03 16:53 UTC (History)
0 users

See Also:


Attachments
2017 MacBook Pro made in 2018 showing "AP" model drive. (421.44 KB, image/jpeg)
2018-02-11 16:53 UTC, Daniel Shepherd
no flags Details
Early 2017 MacBook Pro (mine) with "SM" model drive. (383.64 KB, image/jpeg)
2018-02-11 16:53 UTC, Daniel Shepherd
no flags Details
iMac Pro with "AP" drive. (470.24 KB, image/jpeg)
2018-02-11 16:54 UTC, Daniel Shepherd
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Daniel Shepherd 2018-02-11 16:53:11 UTC
Created attachment 13953 [details]
2017 MacBook Pro made in 2018 showing "AP" model drive.

I regularly use rsync to deploy roughly 40gb of data to new Macs that come in. For the past few years this has taken roughly 8-10 minutes to transfer to SSD models.

A few weeks ago this time suddenly jumped to over 40 minutes with no other exchanges to hardware.

I've deeply troubleshooted this performance issue and found only one change has occurred. Originally blaming APFS, i tested with both APFS and HFS read and write sources. It occurred on both an iMac Pro and two MacBook Pros. macOS has been forked for the iMac Pro and I considered that perhaps it's build (which my admin OS was running) had a different SSD kext - however I tried both 10.13.3 17D47 and the iMac Pro build 10.13.3 17D2047 and confirmed the performance issue on both.

We've been recreating this bug for faster testing by copying the Applications folder from the source to the destination (with a destination where the Applications folder doesn't exist, just to get raw copy speed). The folder is 25gb. With Finder, including the lengthy time it takes to pre-scan files, it copies in 6 minutes, with rsync it's nearly 25 minutes, again about 1 minute per gb.

I tested on my 2017 MacBook Pro and observed that both Finder and rsync could copy the data in 6 minutes - same source, destination of my MacBook Pro was APFS formatted too.

I had seen other MacBook Pros and SSD equipped iMacs come in and all copy with the correct speed and deploy within 10 minutes.

Finally upon inspecting what is different about the recent MacBook Pros and the iMac Pro, in the SSD NVME tab they have listed an "AP" model for the SSD (Presumably Apple with their own controller) my 2017 MacBook Pro that doesn't show this poor performance has an "SM" model (Samsung).

The only consistent thing on all the systems showing this 4x slower speed with rsync is that they have AP model SSDs. The MacBook Pro has AP0512J, the iMac Pro has AP1024M.
My MacBook Pro which works fine and was made just after the refresh is SM1024L.

So I assume Apple has quietly changed to the AP drives (and controller?) in recent weeks for the MacBook Pro (some "SM" are still coming through which work fine with rsync) and of course the iMac Pro is shipping with it.

I'll be getting a number of these systems in, so any testing, logs or anything I can do to get to the bottom of this should be no problem - I have as well filed a Rdar with Apple as it could be something they need to fix in their own firmware.

Attached an SSD model number.
Comment 1 Daniel Shepherd 2018-02-11 16:53:54 UTC
Created attachment 13954 [details]
Early 2017 MacBook Pro (mine) with "SM" model drive.
Comment 2 Daniel Shepherd 2018-02-11 16:54:10 UTC
Created attachment 13955 [details]
iMac Pro with "AP" drive.
Comment 3 Daniel Shepherd 2018-02-22 12:05:26 UTC
Have tested this with a 2016 MacBook that also has the "AP" SSD controller and can confirm the same performance issue.

So it's not just new machines, it's any Mac with an "AP" SSD formatted with APFS.
Comment 4 Chris Tipper 2018-02-22 20:17:19 UTC
Model: iMac 4K (2015)
Platform: macOS 10.13.3
Storage: APFS running on APPLE SSD SM0512G 

Symptom: 5% throughput over Gigabit LAN compared to rsync 3.1.2 (4Mb/s vs 100Mb/s)
The target machine is FreeBSD 11.1 running rsync 3.1.3

I don't have any diagnostic but I have downgraded to rsync 3.1.2 from 3.1.3 and seen a 20-fold increase in throughput. It seems unlikely that it is to do with the specific model of SSD as mine is not one of the type listed in the bug report. I suggest looking into APFS. I would also note that I installed rsync from Macports.
https://github.com/macports/macports-ports/tree/master/net/rsync
Comment 5 Chris Tipper 2018-03-12 10:36:34 UTC
I think I ought to report that the problem has resolved itself on my setup, I didn't change anything but after the initial sync it returned to previous throughput. I am using the --whole-files switch if that provides any insight, it seemed to have an initial problem with archives produced on a previous run with rsync 3.1.2. These are virtual machine images.
Comment 6 Daniel Shepherd 2018-03-12 10:58:14 UTC
Respectfully Chris you didn't have this specific bug, as I said it only affects AP controllers. Whatever you were experiencing has nothing to do with this. I've verified the AP controller bug on MacBook, MacBook Pro and iMac Pro.
Comment 7 Chris Tipper 2018-05-03 16:53:43 UTC
Sorry for hijacking your bug but I had an issue and did not want to create a separate report. And you may have verified your bug to your own satisfaction but not provided any diagnostic. So I don't see how it can be addressed in its current state.