Skip to content

On leaving the Spotlight behind

Of late, my Mojave-running iMac has been having major Spotlight problems: Occasionally I'd find it rebuilding the main index, despite me not having done anything to require such an action. Even worse, though, is that it would stop working entirely in Mail until I rebooted.

My main use of search in Mail is to help our customers find lost license files—I have records that go back to 2010, so I can usually find their license if they did buy from us. With Spotlight broken, I'd have to login to the two different cart providers we've used over the years to find license files. Having a functional Spotlight in Mail was fairly job-critical to me.

Some digging showed that a process named suggestd was repeatedly crashing…

When this happened, it seemed it would often, but not always, kill Spotlight in general and in Mail. After a lot of debugging, I gave up on fixing the suggestd crash—it's stil crashing multiple times a day—and instead, set out to find another way to search my Mail archives without the help of Spotlight.

I wanted to offload all that historical Mail data to some other app whose search feature wouldn't be dependent on a functional Spotlight. And so, the search began.

tl;dr summary: I chose EagleFiler, a long-lived Mac app that works wonderfully for this task (and many others). It creates its own indices, so it's not reliant on Spotlight, and it's quite speedy at ingesting large amounts of Mail. An unexpected side benefit is that the database is small enough—and fully self-contained—that it's easy to sync to my laptop—so I now always have my email archives available.

I considered—and quickly discarded—a couple of possible solutions. FileMaker Pro seemed ideal, but it would mean developing my own front-end, and worse, paying for a Ferrari to solve a Chevrolet-level issue—FileMaker Pro has become very expensive.

I also thought about using a SQLite database and front-end app to manage it, but that'd mean learning a new database system and creating the database. I even considered a web-based UI to MySQL, but again, I'd be spending time designing and creating, which I really didn't want to do.

So I went looking for a Mac app that would simply manage a large collection of text (the emails) with its own indexing feature. And during that search, I rediscovered EagleFiler, a very long-lived1Version 1.0 was launched in 2006 Mac app for managing data, with a particular ability to manage Mail's messages.

After testing it for a bit, I decided to import all my saved email from before 2021 into files and folders in EagleFiler. Import from Mail is ridiculously easy—select a mailbox or a number of messages and press Option-F1 in Mail.

Imported Mail comes with everything, including attachments and even flags. I organized everything by app and within those folders, by year. (I did this in Mail first, then imported the year-specific mailboxes.) There's no real need to do this, as EagleFiler could just as easily search one huge pile of messages, but this helps keep me organized.

At the end of each year, I'll export the current year's messages to EagleFiler and have a mostly-empty Mail app to work with. One limitation of this method is that I still have to use Spotlight to search the current year's Mail…but one happy side effect of offloading all the old messages is that Spotlight in Mail hasn't died since I did so—even though suggestd keeps crashing, Spotlight in Mail has been fine.

As EagleFiler is AppleScriptable, I wrote a Keyboard Maestro macro that pops up a search box, so I can easily enter someone's name or email address from Mail, and have the search run in EagleFiler. The built-in search and smart folders are impressively fast, and the smart folders feature lets you create multiple and/any criteria with any set of conditions, making for some very powerful saved searches.

My use of EagleFiler so far has barely scratched the surface of what it can do, as it can easily capture data from many different apps, and will make a good repository of "stuff" that I can sync with my laptop and have available from anywhere. I've only been using it for a couple of weeks, but I'm guessing I've more than saved its $50 one-time cost in time saved looking up customer data.

3 thoughts on “On leaving the Spotlight behind”

  1. I have been plagued by bad Spotlight for years — similar issues in Mail, super slow (10 second) lag to show results when invoked from the keyboard, but much faster when I used the search box in Finder…. For what it’s worth, to my pleasant surprise, I found that Monterey finally fixed these. For that alone, the update was worth it for me.

    1. Yea, I can't upgrade yet—our accounting app is 32-bit only, and there are small handful of other apps I rely on. Virtualization is probably the answer, but our accounting app is activated, and they no longer make it, so I fear trying to reinstall it.

      -rob.

  2. When you do upgrade (and even before), you might try this. Once it's configured a little, it can really make you feel confident about finding stuff in the file system. If you expect a lot of matches, you can get only a list. If you expect fewer, you can get the list and have the matches revealed in Finder. But it's one of the best GUI's for the Find command I've ever seen.

    https://github.com/abra-code/FindApp

Comments are closed.