The Robservatory

Robservations on everything…

 

Unix

Easy Unix date formatting

I use the date function quite often in scripts, mainly to append date/time stamps to filenames. For example, something like this…

newtime=`date +%Y-%m-%d_%H%M`
cp somefile $newtime-some_other_file

That particular format is the one I use most often, with the full date followed by the hours and minutes in 24 hour format: 2017-04-12_2315, for example. I use this one so that filenames wind up sorted by date order in Finder views.

Once I move beyond that format, though, the vagaries of date string formatting leave me dazed. Enter strftime.net, where you can build any date string you like using a point-and-click editor with real-time previews:

It doesn’t get much easier than that.

Create Time Machine-like backups via rsync

Taking a break from the recent Frankenmac posts, here’s a little trick for creating “Time Machine like” backups of anything you’d care to back up1I don’t know how well this might work for Mac files, as opposed to Unix files. But Mac files can be saved to the real Time Machine.. In my case, it’s the HTML files off of my web sites, both personal and work. I used to simply back these up, but then realized it’d be better to have versions rather than totally overwriting the backup each day (which is what I had been doing).

Once you’ve got it set up and working, you’ll have a folder structure similar to the one at right, with one folder for each backup, and a “current” link that takes you to the newest backup.

I get zero credit for this one; my buddy James explained that he’d been using this method for a year without any troubles, and pointed me to this great guide that explains the process.

I used that guide and added the following to my backup script to create my own customized Time Machine for the files from here, robservatory.com:

/usr/local/bin/rsync -aP \
  --link-dest=/path/to/quasi/TM_backup/current user@host:/path/to/files/on/server/ \
  --exclude "errors.csv" \
  --delete --delete-excluded \
  /path/to/quasi/TM_backup/back-$newtime
rm -f /path/to/quasi/TM_backup/current
ln -s /path/to/quasi/TM_backup/back-$newtime /path/to/quasi/TM_backup/current

And that’s all there is to it. Note that you may need a newer version of rsync than what comes with macOS now (2.6.9)—I use version 3.1.2 from Homebrew, so I can’t say for sure that this script works with the stock version.

I’ve only been using this for a couple weeks, but it’s working well for me so far.

Adjusting for the oddities of ctime

In the shell script I use to back up my web sites (I really should update that, they’re much different now), I include a line that trims the backup folder of older compressed backups of the actual WordPress databases. That line used to look like this:

find path/to/sqlfiles/backups -ctime +5 -delete

I thought this should delete all backups in that folder that are at least five days old, via the ctime +5 bit.1Footnote: I know now I should have been using mtime, though it would have had the same issue I had with ctime. But it turns out I thought wrong. The above will delete all files that are at least six days old. Why? I don’t know why it works this way, but it’s mostly explained in the man page for find (my emphasis added):

-ctime n[smhdw] If no units are specified, this primary evaluates to true if the difference between the time of last change of file status information and the time find was started, rounded up to the next full 24-hour period, is n 24-hour periods. If units are specified, this primary evaluates to true if the difference between the time of last change of file status information and the time find was started is exactly n units. Please refer to the -atime primary description for information on supported time units.

To make find do what I wanted it to do, I just needed to change +5 to +5d. Simple enough…but while figuring this out, I stumbled across this page, which has an alternative solution with more flexibility:

find path/to/sqlfiles/backups -mmin +$((60*24*5)) -delete

The mmin parameter is much more precise than ctime:

-mmin n True if the difference between the file last modification time and the time find was started, rounded up to the next full minute, is n minutes.

By using mmin, I can be really precise. As shown, 60*24*5 gets me the same five-day interval as ctime +5d. (And yes, I could have used 7200 instead of 60*24*5, but I find it clearer to leave it in its expanded form.)

But I could instead delete backups that were older than 3.25 days (60*24*3.25 or 5040), or for any other arbitrary time period. I like the flexibility this offers over ctime, so I’ve switched my script over to this form.

Cancel shell script on remote connect failure

I use a shell script to back up this site (and a variant of the same script to back up the Many Tricks site). I’ve been using these scripts for over a decade (wow), and though they’ve evolved, they’re still fundamentally the same. (I did switch from cron to launchd for launching them, however.)

While the script typically runs very nicely, I recently noticed that my last backup was from a few weeks ago—uh oh. It didn’t take long to figure out what had gone wrong: My ISP changed the hostname of the machine my site runs on, and my script uses ssh, scp and rsync, which connect via the hostname. Unfortunately, the failure mode is silence, because the script runs via a scheduled task. The only way I knew it failed was when I went to check the backup folder. Obviously, something more automatic than that would be desirable.

After much web searching, I couldn’t find anything that seeemed like it’d do what I want: An email (and onscreen alert) if my backup failed. I found lots of Unix solutions to send mail using sendmail, but I didn’t really want to enable that on my Mac. So I futzed around and built a simple checker that will mail me when it can’t reach my web host.

(more…)

Change shell scripts based on where they run

This is one of those “oh duh!” things that I wish I’d realized earlier. I have a few shell scripts that I’d like to keep on the Many Tricks cloud server, as I’d like to use them on multiple Macs.

But depending on which Mac is running the script, I might need to use unique code. The path to my Dropbox folder, for example, is different on my laptop and my iMac. So if I want to reference the path to my Dropbox folder, it needs to be different on each Mac. I couldn’t figure out how to make that happen with just one script, so I’d been using near-identical versions on each Mac.

Then I remembered the hostname command, which returns the name of the machine running the command:

$ hostname
Robs-rMBP.local

And that was the tidbit of “duh!” knowledge I needed. With that, and the case statement, I can make my shell scripts use code based on which machine runs them. For instance, I can set unique paths for the script that grabs the latest versions of our apps from our server:

myhost=`hostname`
case $myhost in
  Robs-iMac.local) theHub=/path/to/apps/on/manytricks/cloud ;
                   theDest=/path/to/local/copy/of/apps ;;
 
  Robs-rMBP.local) theHub=/different/path/to/apps/on/manytricks/cloud ;
                   theDest=/other/path/to/local/copy/of/apps ;;

                *) echo "Sorry, unrecognized Mac." ;
                   exit ;;

  cp $theHub/$appname $theDest/$appname
  etc

Another nice thing about this is the script won’t run on a Mac I haven’t set up yet, thanks to the #) bit. And if I happen to rename one of my Macs, the script will also fail to run, letting me know I need to update the name in the script.

A simple tip, but one I’d managed to overlook for years. Now that I’ve written it up, that shouldn’t happen again.

Install a cloud driver server of your very own

For many years, Peter and I have managed our shared Many Tricks files via Dropbox. To support Dropbox, we purchased an upgraded plan for $99 a year, which came with 1TB of space. We then used the same login to share the Dropbox folder. We didn’t need anywhere near 1TB (we have about 4GB of shared files), but felt it was right to support Dropbox.

While this worked well, and we had no issue paying for it, we had a few concerns—about space, third-party involvement, and something possibly unique to my usage scenario. You can read the details in the remainder of this post, but to make a long story short, I went looking for a replacement. And I found one in Nextcloud. Nextcloud has a commercial product, but it’s open source, so you can also install it on your own server, and via many hosting companies that have it preinstalled.

I was able to install it easily with our hosting provider; I had the basic install up and running in under 30 minutes. There are also native clients for Mac, as well as Windows, iOS, and Android. The Finder view with the Mac ap installed (it’s an official Finder extension) is very similar to Dropbox or OneDrive or any other cloud client with a Mac app:

For us, Nextcloud has every feature we need for sharing our Many Tricks’ files; read on for more detail on why we moved, install and admin, the Mac client, and some closing comments.

(more…)

Color and ‘human readable’ file sizes in Terminal

These are two very old tips, but I’d forgotten about them until recently, when I sent someone a screenshot and they said “Hey, how’d you do that?”

Do what, exactly? This…

The most-obvious thing in that shot is the colored filenames. But notice, too, the file sizes are in a human-readable form. Both of these changes are pretty simple, though you could spend hours playing with colors.

Human-readable output

To get human readable output—not just from ls but also in du, which shows disk space usage—just include an h with the ls command: ls -alh. Instead of raw bytes, the values are converted and marked with trailing B, K, M, etc.

Because I never use ls in its short form, I actually added a line to my .profile (which loads whenever you open a Terminal session) to make this automatic:

alias ls='ls -alh'

You could do the same thing with du, but I rarely use that command, so I didn’t bother.

(more…)

The little I know about regex…and where to learn more

First off, regex is shorthand for a regular expression. And what, exactly, is a regular expression? According to the linked Wikipedia page, a regular expression is…

…in theoretical computer science and formal language theory, a sequence of characters that define a search pattern. Usually this pattern is then used by string searching algorithms for “find” or “find and replace” operations on strings.

That’s a mouthful, but what it means is that you can write some really bizarre looking code that will transform text from one form to another form. And if you know just a bit of regex, and where to go to look up what you don’t know, then you can use regex to do many useful things.

For example, consider this filename on a scanned-to-PDF receipt:

The Party Place [party supplies] - 02-06-2017

Perhaps you’d prefer it if the date came first, in year-month-day order, so that your receipts were ordered by date, like this:

2017-02-06 - The Party Place [party supplies]

Sure, you could manually rename this one file, but what if you have 500 receipts that you need to rename? Enter regular expressions—they’ll let you do this text manipulation, and many more. What follows is a very brief summary of my knowledge of regex, along with pointers to sites where I go when (very often) the problem I need to solve is beyond my regex skill level.

(more…)

Create a workload for your CPUs

Over the weekend, I was testing how some of our apps work when the CPUs are busy. One way to load the CPUs is to rip a Blu-ray disc, but I was looking for a more controllable CPU load.

A quick search through the Mac OS X Hints archive (use this tip to search the site) found the answer from 15 years ago: Just say yes in Terminal to generate sizable CPU loads…

More specifically, use this command in Terminal:

yes > /dev/null &

If there’s an award for strangest Unix utility, yes might just be the winner. All it does is output y (or whatever you list after the y; the man page suggests an expletive) until you kill the task.

The above command sends the output (via the > redirect) to the null device, which discards it. The ampersand sends the job to the background, so you get your Terminal prompt back.

You can run this command multiple times, each loading the CPU even more heavily (the screenshot shows three yes tasks running). Keep an eye on Activity Monitor to see just how much CPU it takes—as shown above, it does a great job at loading the CPU.

You can kill the tasks by issuing the command killall yes in Terminal, or by quitting Terminal—you’ll be told that quitting will terminate the tasks.

Install and configure Apache, PHP, and MySQL on macOS Sierra

Fair warning: Today’s tip is over-the-top geeky.

After many years of not doing anything with web serving on my local Mac, I recently had a need to look at some mySQL/PHP-based web packages. While I could install these on robservatory as tests, I generally like to install things locally first, because I’ll install a handful of packages until I find the one I like. But it’d been so long, I didn’t have anything configured. (Remember when enabling Apache—i.e. web sharing—was a feature of the System Preferences’ Sharing panel? Ah, good times!)

I could have opted to use the built-in Apache, but I wanted something that I could more easily keep up to date, and that, if I chose, would be easy to remove. The good news is that Homebrew has packages available for Apache, PHP, and MySQL. Homebrew installs everything in its own directory tree, and adding and removing packages is simple—exactly the setup I wanted.

So in theory, installation was as easy as three brew tap commands. The reality, though, is that the installation is a bit more complex. OK, it’s a lot more complex. The good news is that, this being the age of the internet, help is but a search away. Or the advice of a friend away, which is what I used in this case.

A friend pointed me to this excellent installion guide that walks through the entire process, including installing two versions of PHP with the ability to switch between them on the fly.

It was this on-the-fly switching, though, that gave me troubles: I couldn’t get a site to load unless I specified “index.php” or “index.html” as part of the URL. (Apache is configured to grab those files automatically.) Solving that one took a bit of time…

(more…)

The Robservatory © 2018 • Privacy Policy Built from the Frontier theme