Skip to content

Unix

Quickly expand compressed files in Finder

In this post, I lamented on the incredible slowdown seen when expanding many compressed files in Finder. To save a trip back to that article, here's the tldr; version:

Expand 24 .gz filesFinder:
12.8 seconds		Terminal:
.013 seconds		Terminal is 984.6x faster than Finder

With a bit of work, though, you can harness the power of Unix in Finder, and get both the GUI and the speed when expanding files. There are many ways to do this, including third-party apps such as The Unarchiver. You could also write an Automator Service to do the work for you. But I chose to write a Keyboard Maestro macro, because I could make the macro a menu bar option that only shows when Finder is frontmost:

As seen, I actually have two expanders, one that expands just the Apple sales reports mentioned in my original post, and the other that will handle any mix of compressed files. (Yes, the 'all files' macro could also handle the Apple sales reports, but because of how it has to run for multiple file types, it's marginally slower than the dedicated macro.)

These macros both work in the same way—they call on Unix apps to do the expansion, bypassing Apple's slow GUI expander (which is called Archive Utility). If you're curious how they work, and/or would like to download them for your own use, keep reading.

[continue reading…]



The Finder’s GUI tax can be very expensive

Once a month, I download roughly 25 gzipped (.gz) files from Apple—these are our Mac App Store sales reports, with one file for each App Store region. I could have Safari expand these files (via the "Open 'safe' files after downloading' item in its preferences), but I prefer to leave that option unchecked. (Why? I often download archives that I want to leave archived, and I like to keep originals of many of the things I download).

If you work with lots of compressed files, you're probably familiar with what happens in Finder (see note) when you go to expand any semi-large number of files: The infamous Dancing Dialog™. It looks something like this…

[Note: Technically, this isn't Finder, it's Archive Utility doing the expansion. But this is the service that Apple provides to expand compressed files, and it's what 99% of macOS users will use. It can be changed via the Get Info dialog, but very few people will take that step. So to most users, it seems it's Finder handling the expansion. For ease of reference, I'm going to say it's Finder doing the expansion.]

Not only is this randomly-resizing dialog box visually annoying, it turns what should be a super-fast process into one that takes a ridiculous amount of time. The end result is that users think they have a slow machine—"it took over 12 seconds to expand 25 tiny little archives!"—when what they really have is a horrendously slow GUI interface to a super fast task.

Just how fast is the task, if the GUI doesn't get in the way? Thanks to the Unix core of macOS, we can answer that question using Terminal, the geeky front-end to the Unix core. The Unix command to expand .gz archives is gzip; so to expand all the .gz files in a folder (and keep the originals), you'd use this command in Terminal:

gzip -d -k *.gz

If you try this, you'll find out it's nearly instantaneous—press Return, and the files are expanded. Unix actually gives us a way to see exactly how fast it is, via the time command:

$ time gzip -d -k *.gz

real	0m0.013s
user	0m0.002s
sys	0m0.005s

This was for a set of 24 .gz archive files (on a solid state drive), and the real line in the output shows exactly how long it took to expand them all: 0.013 seconds. By comparison, I made a screen recording (with an onscreen stopwatch for timing) of Finder expanding the same 24 files; it took 12.8 seconds for all the dialog dancing to end. Think about that…

Expand 24 .gz filesFinder:
12.8 seconds		Terminal:
.013 seconds		Terminal is 984.6x faster than Finder

To put those results another way, if expansion time is linear, gzip could expand 23,631 files in the time Finder takes to expand 24 files. That's nuts!

(You can watch this video for a visual comparison of expanding the same set of files in Finder and Terminal.)

So it's not the computer that's slow, it's the GUI interface to the computer that (in this particular use case) is incredibly, horrendously slow. And there's no need for it—the separate individual progress bars, appearing and vanishing in under a second each, provide no useful feedback to the user. They just slow down the task.

Finder (née Archive Utility) should just execute the task without any visual feedback (though it should pop up a window if there are exceptions). If visual feedback is really required, a window with a single progress bar for the entire task would be OK, but would still slow operations down.

This is a great example of how an everyday task can make you think you have a slow computer, when what you really have is a fast computer with a slow interface element. Given how often we all deal with compressed files, it'd be nice to see Apple clean up this mess. Until they do, however, you can harness the power of Unix—even while in Finder—to speed up the task. Here's one way to do just that.



Total PDF pages in subfolders across folder structure

Last week, I wrote a script that ran through a folder structure and output the page count of every PDF in all folders and sub-folders, and also spit out a grand total.

While this worked well, what I really wanted was a script that just totaled PDF pages by sub-folder, without seeing all the file-by-file detail. After trying to retrofit the first script, I realized that was a waste of time, and started over from scratch.

The resulting script works just as I'd like it to, traversing a folder structure and showing PDF page counts by folder:

$ countpdfbydir
    47: ./_Legal
     2: ./_Medical-Dental
    15: ./_Medical-Dental/Kids
    11: ./_Medical-Dental/Marian
     2: ./_Medical-Dental/Rob
    35: ./_Personal Documents/Kids
    87: ./_Personal Documents/Marian
    28: ./_Personal Documents/Rob
    10: ./_Personal Documents/Rob/Golf
    12: ./_Personal Documents/Rob/Travel
-------------------------------------------------------------------
   249: Total PDF Pages

It took a few revisions, but I like this version; it even does some simplistic padding to keep the figures lined up in the output.

[continue reading…]



Count pages in all PDFs within a folder structure

Please see this newer post, with a new script that provides subtotals by subfolder, which is what I really wanted when I wrote this one.

Recently I've been trying to go paperless (well, mostly paperless) via a Fujitsu ScanSanp ix500. (I'll have more to say about the scanner in a future post).

One way to go paperless is to just go from now forward—start scanning stuff and don't worry about history. I decided that I'd go the other route, and work through our old paper files: some would be scanned and kept, much would just be recycled. The process went really quickly, compared to what I had expected. It helps that the Fujitsu is a wicked-fast document scanner!

But I was curious about how much I was scanning, in terms of total PDF pages—not files, but counting the pages in the files. Spotlight to the rescue; the field kMDItemNumberOfPages returns the number of pages in a document, and it seemed accurate in testing via mdls:

$ mdls /path/to/somefile.pdf | grep kMDItemNumberOfPages
kMDItemNumberOfPages = 4

So I set out to write a script to traverse my "Scans" folder, and return the total number of PDF pages.

[continue reading…]



Encrypt files then backup to a cloud service via script

Most cloud services tell you that their data stores are safe, that your data is encrypted in transit and on their drives, that employees don't have access, etc. For the vast majority of the stuff I store in the cloud, this is more than good enough for me—the data isn't overly sensitive, and if someone were to hack their way in, all they'd get are a bunch of work and personal writing files and some family photos.

For other files—primarily financial and family related—those assurances just aren't enough for me. But I still want the flexibility and security that comes from having a copy of these files in the cloud. So what's a paranoid user to do to take advantage of the cloud, with added security, but with a minimum of hassle?

The solution I came up with involves using local encrypted disk images and a shell script. Using this script (and some means of scheduling it), you can automatically encrypt and back up whatever files you like to a cloud service.

[continue reading…]



Goodbye cron task, hello launchd agent

On April 29th, 2005, Apple launched Mac OS X 10.4, aka Tiger. With Tiger came launchd, a new Unix-side job scheduling tool. launchd was intended to replace cron, the long-established (and quite cryptic) tool for such tasks.

And now, a mere nine-plus years later, I decided it was time to give up cron and move to launchd myself. Mr. Bleeding Edge, that's me! (Note: Unless you enjoy the Unix side of OS X and currently use cron to schedule tasks, this article won't be of much interest to you.)

Why now, after so long as a holdout? Primarily because I kept running into issues with cron tasks that needed to do things as "authorized me," such as mounting an encrypted disk image, or even just mounting a network share. Or my Mac would be asleep for a scheduled cron task, and it therefore wouldn't run. (launchd will queue any missed tasks to execute when the Mac reawakens.) Finally, my cron file was getting huge and unwieldy, and making simple changes was fraught with danger of breaking something.

So I dedicated a portion of a recent weekend to figuring out launchd, and migrating my cron tasks to this brave no-longer-at-all-new world. If you're still hanging on to cron, read on to see what I've learned about launchd—maybe it'll inspire you to move, too (or not).

[continue reading…]



Behind OS X’s modern face lies an aging collection of Unix tools

Note: This article has been heavily modified, as I was a total knucklehead and completely forgot about GPL v3—thanks to Geordie Korper for pointing it out in the comments. Basically, the cause of the aging Unix tools in OS X is GPL v3. I've updated the following table to include the license information. In every case but one (nano), GPL v3 is the license on the non-updated apps. So that's that…I'll leave this up, though, in case anyone's curious about this stuff.

As I covered recently, the version of bash that ships with OS X is quite old, and as such, is vulnerable to the recently-revealed bash security hole.

At some point, Apple will release an OS X update containing a patched version of bash. (Update released.)

So while Apple has patched bash, this version of the shell is simply ancient. Just how old is it? bash 3.2.53(1) is roughly seven years behind the current version, 4.3.25. Seven years is like, well, forever, in Internet time!

With that bash age gap in mind, I took at look at a number of common Unix apps—in both Mavericks and Yosemite—to see which versions were in use. Then I checked the same apps in MacPorts, a tool that makes it simple to install many Unix apps.

(MacPorts doesn't necessarily have the absolute latest version of every Unix app, but they do stay relatively current. For instance, they already have a patched version of bash that fixes the above vulnerability.)

The results were interesting, to say the least—many of the core Unix utilities in OS X are years and multiple versions behind their open source, er, sources. You can thank GPL v3 for that, as noted above (and covered in more detail below). But that still leaves the big question:

Does it matter?

That is, should we care that these tools aren't keeping up with their latest and (presumably) greatest versions? Is it a problem, or merely a statement that what we have works well enough for the majority of users? (For those who don't want to bother reading, my general opinion is no, it doesn't matter.)

[continue reading…]



A [possibly dangerous] fix for the bash security exposure

In case you missed the news, there’s a big security hole in the bash shell. If you’re using an OS X machine and it’s not exposed to the internet (and even then, not running a publicly-facing service that communicates with bash), you shouldn’t have much of anything to worry about. In theory, anyway.

In practice, because I like to live dangerously, I decided I wanted to make my machine safe. Or at least try. What follows is the how-to for how I did it.

Note: Please exercise extreme caution if you're going to follow my how-to! I am replacing a couple of key system-level files with a new, untested-by-Apple version. It's quite possible this may break my system in ways I haven't yet discovered. Also, given that there's probably not any exposure for a typical user not running web-facing apps that interface with bash, this really isn't worth doing for most users.

With that disclaimer out of the way, read on for my how-to…

[continue reading…]



My most-useful and least-used shell script

I have a large number of small shell scripts I've either written or collected over the years. Today I had the opportunity to use my favorite one—which is rare, as I only need it a couple times a year. But when I do need it, it's a wonderful little script.

It's also a very simple-minded script, as it does just one thing: it copies my public IP address to the clipboard and shows it in a pop-up message, as seen at right. OK, so that's two things, but they're very closely related.

Clearly this isn't something I need to do often, but when I do, the script changes this…

Switch to browser, open new tab, load the DynDNS check IP page, drag mouse to select IP address, press Command-C to copy, switch back to destination app, press Command-V to paste

…into this…

Press a key combo, wait about a second, then press Command-V

This is a big timesaver, obviously, and it makes the process about as easy as it could be.

I originally wrote this up for Mac OS X Hints a few years back, but thought I'd post it here (given the changes at Macworld, I'm not sure how long the hints site may be around). I've also modified it a bit, as I no longer use growlnotify for the onscreen display of the copied IP address.

You can read the original how-to at hints, or below, where I've posted the updated version that no longer uses growlnotify.

[continue reading…]