The Robservatory

Robservations on everything…

 

Digging into an unexpected encryption speed boost

What follows is a lengthy dive into a semi-recent massive performance improvement in openssl speeds in macOS. As it's long, here's a tl;dr version:

  • From 10.14.4 to 10.14.5, a change in macOS improved openssl speed benchmark results anywhere from 15x to 30x.
  • In real world use, encryption of a large sample file (570MB) using a very long password happened nearly twice as quickly as it did before the update.
  • The version number for openssl (which is really LibreSSL) is the same (2.6.5) in both 10.14.4 and 10.14.5. I also confirmed that the packages, as loaded on the Apple Open Source site, are identical.
  • The four libraries that openssl links to have the same version numbers in 10.14.4 and 10.14.5.
  • The binaries for openssl and the four linked libraries all use much less disk space in 10.14.5 than they did in 10.14.4. I can't explain this, except that openssl itself is no longer a universal binary.
  • I believe the performance boost is due to macOS enabling Intel's AES-NI, which allow hardware acceleration of some key cryptography tasks. But I can't figure out how this change was made, given the above data.
  • The Apple Open Source site may hold the answers, but that work is beyond my skill level.

Keep reading if you'd like to see how I came to the above summary…

While working on some benchmarks for an upcoming post, I was looking back at my two most-recent benchmarking articles, one about my 2018 MacBook Air and the other about my 2019 27" iMac.

In both of those posts, I ran a simple benchmark test using openssl1Apple actually uses LibreSSL, a functionally equivalent alternative to OpenSSL in Terminal. The benchmark runs a series of cryptographic signing and verifying tasks; here's how my 2018 MacBook Air scored on one particular test—I ran this test in November of 2018, while running macOS 10.14.1:

RSA 2048 sign/sec   =  31.0
RSA 2048 verify/sec = 646.5

As I started working on the new post, I re-ran the tests, just to make sure they hadn't changed much, as my machine was now running macOS Catalina (10.15.2). I wasn't expecting much of a change, so I was clearly surprised—shocked, actually—at the results:

RSA 2048 sign/sec   =    953.4
RSA 2048 verify/sec = 17,176.6

Holy cow! The "sign" speed is up roughly 30x, and the "verify" speed up about 26x. That's insane, and clearly beyond any software tweaks from Mojave to Catalina. I then tested my other Macs—my 2019 27" iMac and 2013 13" MacBook Pro. Both saw similar massive improvements in the benchmark results.

This change greatly intrigued me, and me being me, I dove into quite the rabbit hole to try to figure out when this change happened, was it a real performance change or just a benchmark jump, and if it was real, what change enabled the improvement? Unfortunately I can only fully answer two of those questions as of today.

When did this change?

My iMac, which still runs macOS Mojave (10.14.6) also exhibits the massive speed increase on the openssl speed tests, so I knew the change happened sometime before that. I also used this same Mac for my April 2019 comparison with the 2014 iMac; at that time, the new iMac was running 10.14.4, and the results weren't the fast ones I'm seeing in 10.14.6.

So the change, whatever it was, happened in either 10.14.5 or 10.14.6. To figure out exactly which release it was, I mucked about with virtual machines, then created a new APFS volume on my Catalina-running 2013 MacBook Pro. I installed 10.14.4 there, and got the slow results, as expected. After installing 10.14.5 on a second new APFS volume (that is so much nicer than partitioning!), it was obvious that that's when the change occurred:

Whatever the change was, it was in the 10.14.5 update, which was released on May 13, 2019. Question answered.

Is it a real performance change?

To test whether the benchmark test speed increases translated into the real world, I used openssl to encrypt the same file on both 10.14.4 and 10.14.5 on my 2013 MacBooK Pro. I created a zip archive of the Applications folder (fresh installs, no third party apps); the resulting file was 571MB in size. I then used this command to encrypt it:

time openssl enc -des-ede3 -in ~/Desktop/Archive.zip -out ~/Desktop/encrypted_file

The time command returns the execution time for the following command, so I can see exactly how long it takes. I chose the encryption cipher (des-ede3) basically at random, as I was seeing across-the-board improvements in the benchmark tests. I provided a relatively long password (the entire preamble to the Constitution), and then the machine got to work.

In 10.14.4, encryption took 55.2 seconds; in 10.14.5, it took 29.3 seconds—so nearly twice as fast as before the update. Wow.

So yes, this is a real performance change that has tangible benefits. Question answered.

What was the change?

This is the most interesting question, and unfortunately, it's the one I can't (yet?) answer in full. Actually, that's not entirely true: I think I have a good theory about what changed, but I don't understand how it changed. So what changed?

Based on the sheer scale of the improvements, I believe that openssl in 10.14.5 and later takes advantage of the Intel® Advanced Encryption Standard (AES) New Instructions, or AES-NI for short. This is a set of instructions built into Intel's Core family of CPUs since 2010 that provide hardware acceleration of many cryptography tasks.

Here's a key bit about how AES-NI works:

The new AES-NI instruction set is comprised of six new instructions that perform several compute intensive parts of the AES algorithm. These instructions can execute using significantly less clock cycles than a software solution.

LibreSSL is based on OpenSSL, and OpenSSL has had AES-NI support for a very long time, so it's definitely also supported in LibreSSL. My working theory is that, due to some unknown reason (bug? security concerns? testing time?), AES-NI wasn't enabled in macOS until the 10.14.5 release.

When Apple released this update, it enabled the AES-NI hardware acceleration present in all Intel Core CPUs dating back to 2010, leading to the speed increases I saw on my three Macs—even my old MacBook Pro has an AES-NI capable Core chip in it, so it too saw the performance bump.

But what, exactly, did Apple update to enable AES-NI (if my theory is indeed true)? Here are things I've looked at, either through my own testing or based on suggestions on Twitter…

A newer version of LibreSSL

This was my first thought, but a quick check (via openssl version) reports that LibreSSL is version 2.6.5 on both 10.14.4 and 10.14.5. (macOS Catalina has bumped that; it's using version 2.8.3. The most-current version is 3.0.2.)

But perhaps all is not revealed by version numbers? To confirm there were no real changes, I visited the Apple Open Source site, and looked at the bundled apps in both 10.14.4 and 10.14.5. The openssl package is the same for both, OpenSSL098-76.200.2. And just to be really certain, I downloaded both packages and did a comparison of the files in each of them: There are no differences.

So I think I can rule out a newer version of LibreSSL as the cause of the improvements.

Newer versions of the dynamic libraries that are linked to LibreSSL

Thanks to this tweet from @atlauren, I found the libraries that are linked to openssl by using the Apple Developer Tools otool command; here's the output from the 10.14.4 version:

$ otool -L /usr/bin/openssl
/usr/bin/openssl:
    /usr/lib/libssl.44.dylib (compatibility version 45.0.0, current version 45.1.0)
    /usr/lib/libcrypto.42.dylib (compatibility version 43.0.0, current version 43.0.0)
    /System/Library/PrivateFrameworks/TrustEvaluationAgent.framework/Versions/A/TrustEvaluationAgent (compatibility version 1.0.0, current version 31.200.1)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1)

I then used the same command in 10.14.15; I'd paste those results, too, but they're identical to what you see above. Like LibreSSL, there's no change in the version numbers of any of the linked libraries.

Unversioned changes in the linked libraries

Perhaps Apple made changes in the linked libraries, but (for whatever reason) didn't update their version numbers. Figuring out if this happened is difficult, especially for someone of limited programming skill such as myself. One thing to look at that might provide a clue to unversioned changes is the size of the involved files.

Unfortunately, I think this is a dead end, as shown by this file size comparison of the linked libraries (along with openssl)—all sizes in bytes:

File10.14.410.14.5Delta
openssl1,178,976488,624-690,352
libssl.44.dylib833,152569,984-263,168
libcrypto.42.dylib4,163,8402,826,688-1,337,152
TrustEvaluationAgent57,92057,9200

When I compared the two openssl packages from Apple's open source site, I found they were identical. Yet in 10.14.5, they take up about a third of the drive space than they did before. I cannot explain this, nor the similar reductions in the linked libraries, but it makes it impossible to state that "yes, there are changes, because the file sizes differ."

Update: Thanks to @_wodin for pointing me to the file command. The size change in openssl appears to be due to the removal of its 32-bit half—it's no longer a universal binary. The libraries, however, are, so I don't know why they shrunk.

Some other change, somewhere else in the OS

There's nothing about these improvements in the 10.14.5 release notes, nor have I found anything of interest while searching Apple's developer site. I've also had no luck with web searches, using phrases like mojave openssl speed increases or mojave openssl benchmark results improvement.

Someone with more technical skill than I may be able to figure out what happened by looking at the Apple Open Source site and finding changes in some of the provided open source packages. Beyond what I've done above (running a file merge on two folders, not rocket science), this type of work is beyond my technical skill level.

Wrap up

I've reached the end of my investigative abilities, I think—digging any deeper than this requires skills I don't possess. And at the end of the day, it doesn't really matter at all: The speed increased in 10.14.5, that speed increase is usable in real-world scenarios, and it's effective on machines at least as old as my 2013 MacBook Pro. That's all good news, and there's no need to dig deeper to find out why.

But my curiosity level is still high—I'd just like to know what changes Apple made in 10.14.5 that drove this tremendous performance improvement. If you have any ideas, please feel free to share in the comments or email me directly.

Leave a Reply

Your email address will not be published. Required fields are marked *

The Robservatory © 2020 • Privacy Policy Built from the Frontier theme