The Robservatory

Robservations on everything…

 

A possible fix broken search in macOS Mail

Over the weekend, I wrote about my totally useless search in Mail. I got so frustrated by my inability to search in Mail that I decided it was time to for a complete rebuild. I exported all my locally-saved mail, deleted my accounts, quit Mail, trashed its prefs and data files, rebooted, then rebuild it mailbox by mailbox, account by account.

I started with my iCloud account, which I barely use for anything—it has a total of seven messages in the inbox (four of which are iTunes Store receipts), and only 121 sent messages. As a test, I searched for Linea, an excellent drawing app that I had recently purchased. No matches.

At that point, I decided to quit Mail and force Spotlight to rebuild its index overnight. In Terminal, sudo mdutilmdutil -E / will do just that (and take many hours). Today, opened Mail, and search was still dead. Argh! (I had also tried this suggested fix, but it made no difference.)

But doing some random testing today, I discovered a fix! It’s a weird fix, but it seems to work:

If I move all the messages from an inbox or local storage folder into a different local storage folder, they’ll be indexed and findable. I can then move them back into the inbox or source folder, and they remain findable.

Even more important, newly-added messages seem to be properly indexed, in both the inboxes and the local storage folders.

This doesn’t make any sense to me, as any one of my recent actions—rebuilding mailbox indexes, reimporting, and redoing the entire Spotlight index—should have been enough to force a rebuild. But for whatever reason, only manually moving the messages seems to force a rebuild.

Now pardon me while I go back to manually dragging a quarter-million email messages around…

19 Comments

Add a Comment
  1. So you need to move all the email to a local folder (thereby deleting it off the IMAP/Exchange/GApps server) and then push it all back up to the server – just to get Mail to see them in the Spotlight index? Yet, if you use a Spotlight search or 3rd party utilities, the emails are indexed in the Spotlight database? Nice going Apple!

    1. Most of my archive is already in local storage, so it’s mainly just moving it around on the hard drive. But yes, the few messages still sitting on servers had to be copied down to the Mac, then copied back to the IMAP server.

      -rob.

  2. Rob, you of course know about the excellent utility MsgFiler, yes? http://msgfiler.com — this will let you semi-automate the ‘manual’ moving of tons of messages… with its ‘Engine’ addon, it’s super fast. I think I’ve experienced a variant of this problem you’ve seen. Somehow, sometimes, a message won’t get ‘seen’ by my filtering rules, and I’ll have to move it *back to the Inbox* (or, as you’ve discovered, any other mailbox) – then re-apply filtering rules, for it to actually get filtered. Seems like this is related. Somehow the database gets out of sync w/ the UI (or something), and manually (i.e., via the UI) moving something around, *re-alerts* the database to… the existence of these messages, in terms of indexing, or in terms of filtering.

    1. Yea, I used to use MsgFiler, but when I was trying to clean up this issue a while back I purged everything out of Mail, just to see if that helped. I should probably add it back :).

      -rob.

    2. Thanks, Ed. Is there a place I can go to get more explicit directions on how to do this with Msgfiler?
      Thanks
      Robert

  3. The same work around also works for Notes. When search doesn’t work, move them all to a new folder, then copy them back to the original folder and trash the folder you just created. Search will work again. Don’t know how this fixes anything but it works, in other apps as well it seems.

    1. There is a relationship, at some level, between Notes and Mail. Remember that in some version of OS X, Notes were integrated (by default) in Mail. Now, when you create a new Mail account, you’re asked if you’d like to use it for Notes, Mail, or both. I imagine they must use the same database on the backend.

      Good to know the same fix works.

      -rob.

      1. Exchange, Google and iCloud all have Notes as a part of the account; back in the day when we ran our own Dovecot imap server, Notes objects were stored in a specific account folder on the server as ordinary mail messages.

        When we transitioned to Exchange we had to download the notes and re-add them into the new account.

        They’re treated as separate things in Outlook or Notes, but notes in an exchange account are just essentially email messages, just like everything else in an Exchange server. I suspect the same holds for Google and iCloud accounts as well.

  4. It worked! Thanks so much for letting us know. I’ve told others, as well. I even have to move my Sent mail to a folder and back in order to search that, too. Sigh. Seriously, thanks a million!! :)

  5. This is crazy, but dam, it works. Thank you for saving me from losing my mind!! My email is really my management system, frail as it is, its the most convenient way to track all my clients and projects. If that doesn’t work, I can’t work. You saved me!

    Gratefully yours, almost-200K-messages-to-move.

    1. Just create a new folder in Mail, in the “On my Mac” section, and move the messages into it. Let them sit for a bit, then move them back.

      -rob.

  6. Hi Rob, Just to clarify, when I do this does it force a reindex of only the messages that I move, or does it force a reindex of all messages? i.e. do I have to do this trick for all the messages in each folder I have separately in order to fix search for all of them?

    Mail search hasn’t worked reliably for years. Yet it’s a basic essential feature. Apple shouldn’t allow Mail to ship like this, let alone for years running without fixing or even acknowledging the problem… I’ve wasted so much time trying to fix it… incredibly frustrating…

    1. Just tried it and I can answer my own question for the benefit of anyone else reading this: every mailbox and sub-mailbox has to be done separately. It only reindexes what you move. This is going to be time consuming seeing as I have a ton of sub-mailboxes, but at least this is a hack that works. After a year fighting with Mail to get it to do basic searches properly this is the only thing I’ve seen that does. I guess that’s progress, though not on Apple’s part.

  7. Thanks a lot! I just ran into this problem and I was desperate. I cannot use my mail without a reliable search capability. The suggested fix worked. It seems that you might not have to move all mail out from/back to a mailbox, only parts of it. At least this worked for me. Why cannot Apple fix such a really bad bug like this one in one year? I was ready to switch to Outlook, which could have been a big job. I have Office 365 and it works well. The only remaining Apple SW products that I still use on my Mac are Mail and Safari, which I am considering replacing because of its terrible Bookmarks that cannot be sorted and searched. This gave me some time to decide if I replace Mail w. Outlook. It is unfortunate that the quality of Apple’s sw became this deplorable.

  8. A massive big thanks. I have just updated to 10.13.2 High Sierra and had exactly the problem you described. I created another mailbox on my hard drive and shifted the messages into it and then back into the Inbox. Worked like a dream and I can suddenly find what was previously invisible.

  9. Thanks Rob – worked for me too. After much pulling of hair this has been only solution that has worked. Well done

  10. In my case, after trying the above and every solution on many forums discussing this, what worked was REINSTALLING MACOS FROM THE RECOVER MODULE. Took about an hour, all software and 350GB+ data intact. It was have saved me and my client many hours of work, disruption and downtime if I had tried that first, instead of: deleting Envelope Index files, Rebuilding mailboxes, deleting all data from ~/Library/Mail + com.apple.mail.plist, reindexing Macintosh HD, Disk Utility First Aid, etc., etc.

Leave a Reply

The Robservatory © 2018 • Privacy Policy Built from the Frontier theme