I have not seen a progress bar that busy since the day I brought my iPhone 6 back home. Whatever I tried, iTunes simply would not sync everything in my library. In the end, the problem turned out to be as simple—but as deadly—as this:
This is a known-to-Apple issue, and it will be fixed in a forthcoming update. I'm fairly certain it's an iTunes bug, but as Apple didn't clearly state which it was, I'm calling it iTunes/iOS. Either way, until it's fixed, it's a really bad bug.
Here's what happens: If you have duplicates of purchased songs, iTunes simply silently stops syncing when it hits one of those duplicates. From your perspective, it will look like everything is working—iTunes never throws an error, and it proceeds through all six (or seven or whatever) steps of the sync process, as seen in the status window of iTunes.
But behind the scenes, nothing is happening—at least, nothing relative to syncing your files. As seen by my troubles, this can be incredibly frustrating and hard to fix.
Continue reading to see how I was able to finally (with Apple's help) get my devices syncing again—the tricky part is finding all the duplicates, because they're not all obvious. Also note that if you are not having sync issues, I wouldn't worry about duplicates—no need to endanger what's already working well!
If you're affected by this bug, you'll find that your iOS device will get some, but not all, of your content. When you look at the device in iTunes, you'll see something like this:
Those items with the gray dotted circle didn't actually sync—iTunes told the iOS device the files were coming (so they show up in the list), but the file itself didn't transfer. What seems to happen is that iTunes will sync up until it hits a duplicate of a purchased song, then it just stops (without telling you it stopped).
To fix the problem, you have to get rid of the duplicates—note that it's only duplicates of purchased songs that cause the problem. In theory, this should be as simple as selecting your Music library, then using View > Show Duplicates. I did that first, and went through the list of matches to find actual duplicates of purchased songs.
I found a few, and removed them, but sync still wouldn't work. Eventually, iTunes wasn't reporting any duplicates, so I was somewhat stuck. I'd then send my sync log files back to Apple (they had me install a logging profile), and they'd be able to identify the next song that caused the problem. I'd remove that one, and then we'd then repeat this process, one painfully slow song file at a time. Ugh.
Why didn't iTunes see these duplicates? Because they're just different enough to not meet iTunes' criteria, it seems. For example, here's a duplicate that Apple identified which passed iTunes' filters:
As you can see, the artist and time are identical, but the title and album are not. Listening to each track, though, they sounded identical to my ear. When you look at the Get Info window for these songs, you can see some other differences (click to zoom, wide window required!):
Notice that the file sizes are slightly different, and that one item has a purchase date while the other does not. These differences are enough that iTunes doesn't think these files are duplicates, but are close enough to trigger the nasty bug.
So how do you find these devious semi-duplicates, if iTunes won't identify them? I turned to Dupin ($15), Doug Adams' excellent duplicate finder. I tried various combinations of Criteria settings to make sure I was getting all the dupes; this one was particularly effective:
By searching on identical album, artist and track numbers, I could find all those duplicates that had different song titles (the first three songs above, for instance, all have illegitimate dupes). Not everything found would be a true "bad" duplicate, as some CDs are just weird. You can even see that in the screenshot—Counting Crows' Holiday in Spain and Paved Paradise are both track number 13 on Hard Candy.
Instead of using Dupin to remove the dupes (which it can do), I just switched back to iTunes and decided which one to keep and which one to delete.
After I completed this process—I found about 20 such files, all much older purchases—I was able to sync my iPhone and get 100% of my content transferred—hooray!
A very important heads up
When you first connect your iOS device after clearing your duplicates, you may see this dialog:
If you see this dialog, it's really important that you click the Don't Transfer button! If you do transfer the purchases back, you'll be recreating your duplicates. And that would not be a good thing to do. (Ideally, you'll have transferred any other legit purchases prior to clearing out duplicates, otherwise you'll need to redownload them.)
So how'd it happen?
Just where did the duplicates come from? That's a great question, and one I can't completely answer. Every duplicate I found was a very old iTunes purchase—most from 2003, none newer than 2009. I don't use iTunes Match, so it couldn't have been some automatic thing that just happened. And all the duplicates have notable changes—some have a higher bit rate, some are DRM free vs. protected, some are longer in duration, etc.
It appears to me that, somehow at some point in time, I must have told iTunes to upgrade my collection (perhaps related to the removal of DRM, many years ago?). Those new versions should have replaced the old versions, but for whatever reason, they did not.
Years pass, and there's no problem, because this bug is apparently a recent arrival. But once the "bugged" version of iTunes was on my Mac, the duplicate songs became sync killers.
Wrapping it all up
This was a long painful process—it's only been four days since I picked up my iPhone, but solving the sync issue is about all I've been doing since then. I thank Apple for the prompt response to my issue, and for working with me to get everything working again. Everyone was incredibly professional and seemed genuinely interested in solving the problem—even on a Sunday morning.
Apple's "normal" tech support folks were also very good, even if they didn't seem to have the knowledge to solve the problem. They spent quite a bit of time with me, making sure they understood the problem. They checked in with me yesterday, to let me know they were still working on it.
Regardless of who solved it, I'm just very happy it's solved…and hopefully this article will help someone else out of the same situation.