The Robservatory

Robservations on everything…

 

Terminal

Selective pruning of old rsync backups

In yesterday’s post, I described a couple rsync oddities, and how they’d led me to this modified command for pruning old (older than four days) backups:

find /path/to/backups/ -d 1 -type d -Bmin +$((60*4*24)) -maxdepth 1 -exec rm -r {} +

After getting this working, though, I wondered if it’d be possible to keep my backups from the first day of each month, even while clearing out the other dates. After some digging in the rsync man page, and testing in Terminal, it appears it’s possible, with some help from regex.

My backup folders are named with a trailing date and time stamp, like this:

back-2017-05-01_2230
back-2017-05-02_0534
back-2017-05-02_1002

To keep any backups made on the first of any month, for my folder naming schema, the modified find command would look like this:

find /path/to/backups/ -d 1 -type d -Bmin +$((60*4*24)) -maxdepth 1 -not -regex ".*-01_.*" -exec rm -r {} +

The new bits, -not -regex ".*-01_.*" basically say “find only files that do not contain anything surrounding a string that is ‘hyphen 01 underscore.’ And because only backups made on the first of the month will contain that pattern, they’re the only ones that will be left out of the purge.

This may be of interest to maybe two people out there; I’m documenting it so I remember how it works!

How to not accidentally delete all your rsync backups

With my Time Machine-like rsync backups running well, I decided it was time to migrate over the cleanup portion of my old script—namely, the bit that removes older backups. Soon after I added this bit to my new script, though, I had a surprise: All of my backups, save the most recent, vanished.

In investigating why this happened, I stumbled across two rsync/macOS behaviors that I wasn’t aware of…and if you’re using rsync for backup, they may be of interest to you, too.

(more…)

How to install ruby gems in Terminal

In yesterday’s tip, See sensor stats in Terminal, I implied that installation of the iStats ruby gem was a simple one-line command. As a commenter pointed out, that’s only true if you already have the prerequisites installed. The prerequisites in this case are the Xcode command line tools. Thankfully, you can install those without installing the full 5GB Xcode development environment.

(Rather than starting from scratch, I’m just going to borrow this bit from my detailed instructions for installing the transcode-video tools, because the Xcode command line tools are required there, too.)

Here’s how to install the command line tools. Open Terminal, paste the following line, and press Return:

xcode-select --install

When you hit Return, you’ll see a single line in response to your command:

$ xcode-select --install
xcode-select: note: install requested for command line developer tools

At this point, macOS will pop up a dialog, which is somewhat surprising as you’re working in the decidedly non-GUI Terminal:

Do not click Get Xcode, unless you want to wait while 5GB of data downloads and installs on your Mac. Instead, click the Install button, which will display an onscreen license agreement. Click Agree, then let the install finish—it’ll only take a couple of minutes.

If you’re curious as to what just happened, the installer created a folder structure in the top-level Library folder (/Library > Developer > CommandLineTools), and installed a slew of programs in the usr folder within the CommandLineTools folder.

(more…)

See sensor stats in Terminal

Someone—perhaps it was Kirk—pointed me at this nifty Ruby gem to read and display your Mac’s sensors in Terminal: iStats — not to be confused with iStat Menus, a GUI tool that does similar things.

Installation is sinmple, via sudo gem install iStats. After a few minutes, iStats will be ready to use. In its simplest form, call istats by itself with no parameters. Normally I’d list the Terminal output here, but istats (by default, can be disabled) presents informatiomn with neat little inline bar graphs, so here’s a screenshot:

This tool is especially useful on a laptop, as it provides an easy-to-read battery summary.

(more…)

Easy Unix date formatting

I use the date function quite often in scripts, mainly to append date/time stamps to filenames. For example, something like this…

newtime=`date +%Y-%m-%d_%H%M`
cp somefile $newtime-some_other_file

That particular format is the one I use most often, with the full date followed by the hours and minutes in 24 hour format: 2017-04-12_2315, for example. I use this one so that filenames wind up sorted by date order in Finder views.

Once I move beyond that format, though, the vagaries of date string formatting leave me dazed. Enter strftime.net, where you can build any date string you like using a point-and-click editor with real-time previews:

It doesn’t get much easier than that.

Adjusting for the oddities of ctime

In the shell script I use to back up my web sites (I really should update that, they’re much different now), I include a line that trims the backup folder of older compressed backups of the actual WordPress databases. That line used to look like this:

find path/to/sqlfiles/backups -ctime +5 -delete

I thought this should delete all backups in that folder that are at least five days old, via the ctime +5 bit.1Footnote: I know now I should have been using mtime, though it would have had the same issue I had with ctime. But it turns out I thought wrong. The above will delete all files that are at least six days old. Why? I don’t know why it works this way, but it’s mostly explained in the man page for find (my emphasis added):

-ctime n[smhdw]

If no units are specified, this primary evaluates to true if the difference between the time of last change of file status information and the time find was started, rounded up to the next full 24-hour period, is n 24-hour periods.

If units are specified, this primary evaluates to true if the difference between the time of last change of file status information and the time find was started is exactly n units. Please refer to the -atime primary description for information on supported time units.

To make find do what I wanted it to do, I just needed to change +5 to +5d. Simple enough…but while figuring this out, I stumbled across this page, which has an alternative solution with more flexibility:

find path/to/sqlfiles/backups -mmin +$((60*24*5)) -delete

The mmin parameter is much more precise than ctime:

-mmin n

True if the difference between the file last modification time and the time find was started, rounded up to the next full minute, is n minutes.

By using mmin, I can be really precise. As shown, 60*24*5 gets me the same five-day interval as ctime +5d. (And yes, I could have used 7200 instead of 60*24*5, but I find it clearer to leave it in its expanded form.)

But I could instead delete backups that were older than 3.25 days (60*24*3.25 or 5040), or for any other arbitrary time period. I like the flexibility this offers over ctime, so I’ve switched my script over to this form.

Cancel shell script on remote connect failure

I use a shell script to back up this site (and a variant of the same script to back up the Many Tricks site). I’ve been using these scripts for over a decade (wow), and though they’ve evolved, they’re still fundamentally the same. (I did switch from cron to launchd for launching them, however.)

While the script typically runs very nicely, I recently noticed that my last backup was from a few weeks ago—uh oh. It didn’t take long to figure out what had gone wrong: My ISP changed the hostname of the machine my site runs on, and my script uses ssh, scp and rsync, which connect via the hostname. Unfortunately, the failure mode is silence, because the script runs via a scheduled task. The only way I knew it failed was when I went to check the backup folder. Obviously, something more automatic than that would be desirable.

After much web searching, I couldn’t find anything that seeemed like it’d do what I want: An email (and onscreen alert) if my backup failed. I found lots of Unix solutions to send mail using sendmail, but I didn’t really want to enable that on my Mac. So I futzed around and built a simple checker that will mail me when it can’t reach my web host.

(more…)

Color and ‘human readable’ file sizes in Terminal

These are two very old tips, but I’d forgotten about them until recently, when I sent someone a screenshot and they said “Hey, how’d you do that?”

Do what, exactly? This…

The most-obvious thing in that shot is the colored filenames. But notice, too, the file sizes are in a human-readable form. Both of these changes are pretty simple, though you could spend hours playing with colors.

Human-readable output

To get human readable output—not just from ls but also in du, which shows disk space usage—just include an h with the ls command: ls -alh. Instead of raw bytes, the values are converted and marked with trailing B, K, M, etc.

Because I never use ls in its short form, I actually added a line to my .profile (which loads whenever you open a Terminal session) to make this automatic:

alias ls='ls -alh'

You could do the same thing with du, but I rarely use that command, so I didn’t bother.

(more…)

View app-specific log messages in Terminal

It’s not often I get to use a tweet by Many Tricks own Peter Maurer as the inspiration for a tip. But this tip is such a case, as he recently complained about Console and its inability to see old output. A response from @fzwob taught me something I didn’t know:

That command browses the captured macOS log data and pulls out anything that matches the specified process name. This could be useful if you’re having troubles with an app and wonder if anything was logged relative to your troubles. Or you might be asked to send the log data if you’re working with the developer on your issue.

Unfortunately, the quotes and dashes in the command as tweeted have been prettified (by Twitter?); here it is in raw Terminal form, using our own Moom as an example:

log show --predicate 'processImagePath CONTAINS[c] "Moom"'

When you press Return, the command will start digging into the log file, and soon start spewing output—possibly a lot of output—to your screen.

(more…)

Use macOS VMs in VMware Fusion in retina mode

I use VMware Fusion often—I have virtual machines that span Mac OS X 10.6 to macOS 10.12.4 beta. I use the more-recent of these for supporting our customers on older versions of the OS, and keep the really old versions just for nostalgia purposes. (I have a bunch of non-macOS virtual machines, too, but they’re not relevant to this tidbit.)

In all the time I’ve been using Fusion on my retina Macs, though, I’ve never enabled this setting…

…well, I enabled it once, but turned it off, because the end result was too small to see: In Retina mode, every pixel is an actual pixel, not a doubled pixel. On my 27″ iMac, that meant the macOS VM thought it was running at (for example) 2560×1600 instead of a retina resolution of 1280×800. VMware even warns you of this in their Knowledge Base:

Mac OS X running in a virtual machine is limited to an approximate resolution of 2560 x 1600, and treats the display as a standard DPI device. This makes the text and icons to appear small in the OS X interface.

However, today I stumbled across this solution from Patrick Bougie—and it’s brilliant in its simplicity. Patrick’s post has all the details; I’ll reproduce them here in abbreviated form, just in case his page ever vanishes.

(more…)

The Robservatory © 2017 Built from the Frontier theme