Of late, my Mojave-running iMac has been having major Spotlight problems: Occasionally I'd find it rebuilding the main index, despite me not having done anything to require such an action. Even worse, though, is that it would stop working entirely in Mail until I rebooted.
My main use of search in Mail is to help our customers find lost license files—I have records that go back to 2010, so I can usually find their license if they did buy from us. With Spotlight broken, I'd have to login to the two different cart providers we've used over the years to find license files. Having a functional Spotlight in Mail was fairly job-critical to me.
Some digging showed that a process named suggestd was repeatedly crashing…
Any macOS engineers out there who might be able to solve a recurring suggestd crash? I've searched, and can't find anything useful. I cleaned up Mail, but that was no help.
Crash line: …NSInternalInconsistencyException', reason: 'Asset identifier storage too small' #HaveLogs
— Rob Griffiths (@rgriff) November 11, 2021
When this happened, it seemed it would often, but not always, kill Spotlight in general and in Mail. After a lot of debugging, I gave up on fixing the suggestd crash—it's stil crashing multiple times a day—and instead, set out to find another way to search my Mail archives without the help of Spotlight.
I wanted to offload all that historical Mail data to some other app whose search feature wouldn't be dependent on a functional Spotlight. And so, the search began.
I considered—and quickly discarded—a couple of possible solutions. FileMaker Pro seemed ideal, but it would mean developing my own front-end, and worse, paying for a Ferrari to solve a Chevrolet-level issue—FileMaker Pro has become very expensive.
I also thought about using a SQLite database and front-end app to manage it, but that'd mean learning a new database system and creating the database. I even considered a web-based UI to MySQL, but again, I'd be spending time designing and creating, which I really didn't want to do.
So I went looking for a Mac app that would simply manage a large collection of text (the emails) with its own indexing feature. And during that search, I rediscovered EagleFiler, a very long-lived1Version 1.0 was launched in 2006 Mac app for managing data, with a particular ability to manage Mail's messages.
After testing it for a bit, I decided to import all my saved email from before 2021 into files and folders in EagleFiler. Import from Mail is ridiculously easy—select a mailbox or a number of messages and press Option-F1 in Mail.
Imported Mail comes with everything, including attachments and even flags. I organized everything by app and within those folders, by year. (I did this in Mail first, then imported the year-specific mailboxes.) There's no real need to do this, as EagleFiler could just as easily search one huge pile of messages, but this helps keep me organized.
At the end of each year, I'll export the current year's messages to EagleFiler and have a mostly-empty Mail app to work with. One limitation of this method is that I still have to use Spotlight to search the current year's Mail…but one happy side effect of offloading all the old messages is that Spotlight in Mail hasn't died since I did so—even though suggestd keeps crashing, Spotlight in Mail has been fine.
As EagleFiler is AppleScriptable, I wrote a Keyboard Maestro macro that pops up a search box, so I can easily enter someone's name or email address from Mail, and have the search run in EagleFiler. The built-in search and smart folders are impressively fast, and the smart folders feature lets you create multiple and/any criteria with any set of conditions, making for some very powerful saved searches.
My use of EagleFiler so far has barely scratched the surface of what it can do, as it can easily capture data from many different apps, and will make a good repository of "stuff" that I can sync with my laptop and have available from anywhere. I've only been using it for a couple of weeks, but I'm guessing I've more than saved its $50 one-time cost in time saved looking up customer data.