Skip to content

Cancel shell script on remote connect failure

I use a shell script to back up this site (and a variant of the same script to back up the Many Tricks site). I've been using these scripts for over a decade (wow), and though they've evolved, they're still fundamentally the same. (I did switch from cron to launchd for launching them, however.)

While the script typically runs very nicely, I recently noticed that my last backup was from a few weeks ago—uh oh. It didn't take long to figure out what had gone wrong: My ISP changed the hostname of the machine my site runs on, and my script uses ssh, scp and rsync, which connect via the hostname. Unfortunately, the failure mode is silence, because the script runs via a scheduled task. The only way I knew it failed was when I went to check the backup folder. Obviously, something more automatic than that would be desirable.

After much web searching, I couldn't find anything that seeemed like it'd do what I want: An email (and onscreen alert) if my backup failed. I found lots of Unix solutions to send mail using sendmail, but I didn't really want to enable that on my Mac. So I futzed around and built a simple checker that will mail me when it can't reach my web host.

First off, this solution isn't perfect, because it's a one-time check, right at the front of the script. There are multiple connections further down the script, so it's possible the first check could pass, then the connection fail on one of the later commands. I wasn't overly concerned about this, though, as my experience has been that if the connection can be made, it's solid. It's just something like this, where the host name changes, that causes bad breakage.

I started by creating a file named testconnect on my web host. (This file doesn't need anything in it, so you can create it with touch testconnectM.) At the beginning of my backup script, I try to copy that file to my iMac's /tmp folder with scp. If the copy is successful, the script runs. If it fails, the script emails me, and puts an alert in Notification Center. Here's the final code:

 ### Test server connection ###

rm -f /tmp/testconnect #Make sure it's gone before trying to copy
scp -q $userhost:$remoteSQL/testconnect /tmp/testconnect

if [ ! -e /tmp/testconnect ]
then
    osascript \
     -e "tell application \"Mail\"" \
     -e "  set newMessage to make new outgoing message with properties {visible:false}" \
     -e "  tell newMessage" \
     -e "    set subject to \"Backup failed!\"" \
     -e "    set content to \"Your personal files backup failed - check the script.\"" \
     -e "    make new to recipient at end of to recipients with properties {address:\"[email protected]\"}" \
     -e "  end tell" \
     -e "  send newMessage" \
     -e "end tell"
    /usr/local/bin/terminal-notifier -title "ERROR!" -message "Unable to connect via scp/ssh; ending."
    exit
fi
... backup script continues

The trickiest part of this turned out to be getting my script to send email via the Mail app, instead of Terminal's built-in mail, which only sends to the Terminal environment. The above AppleScript was cobbled together from bits I found around the web, and it seems to work quite well—Mail doesn't come to the foreground, though there's a brief flash as the message appears onscreen for a split second when it sends.

The call to terminal-notifier creates a Notification Center alert (more about terminal-notifier), in case I'm sitting at my Mac when the backup fails.

It's a simple failure check, but in my testing so far, it's working as I'd expect. The first real test will come when my host changes the server names again.

4 thoughts on “Cancel shell script on remote connect failure”

  1. Dave Rowenhorst

    In a bash script, ssh -q will return an error code, so any return other than 0 is an error. No file copy needed:

    ssh -o ConnectTimeout=5 -q $userhost exit
    SERVERTHERE=$?

    if [ $SERVERTHERE -ne 0 ]
    then
    ...
    fi

  2. Dave:

    I swear I had tried that, but maybe I was using it incorrectly somehow. Thanks for the heads-up, modifying my script!

    -rob.

  3. Usually I prefer not to use Apple script in shell scripts. But these are matters of taste.
    Anyway I think it's quite useful to have a working command line mail program. For these very cases. I'm sure you already have seen some solutions using the integrated mail program. The easiest way from my point of view is to configure the already existing postfix program and use terminal mail. 5 steps. You can find also script for an automated configuration here:

    https://github.com/roubles/postfixconf

    There exist other solutions like msmtp or pysmtp which use the keychain to access the smtp relay password. So you don't have a password saved in some config file.
    Anyway I use a dedicated email account for mail notifications.

    Goddert

    1. I've hesitated to enable postfix due to horror stories of configs gone wrong. That link looks promising, though—I'll give it a try. (I resorted to AppleScript only because I didn't want to enable postfix or install a third-party tool ... but if I can get it going with no risk of misconfiguration, that'd be worth doing.)

      thanks;
      -rob.

Comments are closed.