macOS bugs are causing kext failures

Back in summer of 2018, customer support at Malwarebytes started seeing people with problems activating the kernel extension (kext) in Malwarebytes for Mac. This opened a can of worms that we’re still struggling with today… as soon as we think the worms are back in the can, we start finding new ones. Unfortunately, these worms all belong to macOS, and are affecting other kexts as well.

A history of previous issues

We ultimately traced the initial problem to an issue with the /Library/StagedExtensions/ folder. This folder is used by macOS in the process of activating a kext. By default, this folder is protected by SIP, so it can only be modified by the system.

Unfortunately, we were seeing that a number of users had a StagedExtensions folder that was lacking the restricted flag. (This flag is an indication that an item is protected by SIP, and can be set via the chflags command, although only from recovery mode.) This interfered with activation of the kext, since macOS no longer considered StagedExtensions to be a secure location at which to stage the kext.

The symptom of the problem was pretty easy to spot, if you knew where to look. If you run the following command (where the -O switch shows the file flags):

ls -alO /Library/StagedExtensions

The output should show this for the StagedExtensions folder itself:

drwxr-xr-x@  4 root  wheel  restricted  128 Nov  6 13:37 .

However, on affected systems, it showed:

drwxr-xr-x@  4 root  wheel  -  128 Nov  6 13:37 .

Notice the lack of a restricted flag on the folder, indicating that it is no longer protected by SIP. Of course, this flag should only be able to be cleared by the OS itself, or by a user in recovery mode. There was no reason for users to be clearing that flag themselves, and many of the users struggled with the process of setting it again, so every indication pointed to a system bug.

This was reported to Apple on August 3, 2018 (radar FB5361175), and was fixed in macOS 10.14.1, which I confirmed, and thus closed the bug report on January 28, 2019.

Later, we started seeing the issue again in macOS 10.15 and 10.15.1, and opened a new bug report on November 6, 2019 (radar FB7430509). This bug ended up being closed by Apple for the reason “Unable to diagnose with current information,” which might have irritated me, but the problems stopped again. It must have been fixed independently of our bug report, I guess…?

The current state of kext issues

Unfortunately, the problem is back yet again. This time, however, the problem does not involve the StagedExtensions folder. This time, the issue involves the folder /private/var/db/KernelExtensionManagement/, which is a newer folder that is used in the process of activating kexts on more recent versions of macOS. I’m unsure when this folder first appeared, or exactly how it is used for kext activation compared to StagedExtensions.

Affected Malwarebytes users see an error message in the app and are unable to turn on the real-time protection (RTP) features. In many cases, these users had RTP turned on previously, but suddenly found that it was off. In other cases, these are new installs where users are unable from the start to enable RTP.

Observations of affected users showed exactly the same problem as with StagedExtensions: the KernelExtensionManagement folder had somehow had the restricted flag cleared, meaning it was no longer protected by SIP. Further investigation, via the kextutil command, revealed more information:

$ kextutil -tn /Library/StagedExtensions/Library/Application\ Support/Malwarebytes/MBAM/Kext/MB_MBAM_Protection.kext 2>&1
Error making temporary directory: 1
Memory allocation failure.
Unable to stage kext (/Library/Application Support/Malwarebytes/MBAM/Kext/MB_MBAM_Protection.kext) to secure location.

On affected systems, we’ll often see this same output for a number of other kexts as well.

We’re only seeing this on 10.14.6 and, to a lesser extent, 10.13.6. We don’t really know if the issue affects Catalina as well, as Malwarebytes no longer uses a kext on Catalina, so we don’t have any data from user reports to go on there.

According to the data we’ve seen so far, all but three cases all involve macOS 10.14.6 (18G6032), which was released on September 24. The other three cases (so far) involved older systems, released this past spring or summer: macOS 10.13.6 (17G13035), macOS 10.13.6 (17G14033), macOS 10.14.6 (18G6020).

We saw a few isolated cases months ago, but have been seeing a more significant increase in incidents, starting very roughly sometime in September. So far, there aren’t enough incidents to really make any strong conclusions, other than that it looks like another macOS bug.

This was reported to Apple on October 19, 2020 (radar FB8812838), but there has not been a response yet.

A solution to the problem

The best solution would be for Apple to solve this issue. I do believe that will happen, as we’re not the only ones seeing (and, hopefully, reporting) this issue. There’s a thread on Apple’s developer forums from other people seeing the same problem.

https://developer.apple.com/forums/thread/657789

If you are seeing this issue, or are managing machines with this issue, there is a somewhat easy fix, though it will be quite cumbersome for IT admins trying to fix problems remotely during COVID.

1 – Reboot the affected machine in recovery mode (hold command-R at startup)

2 – Enter the Terminal (from the Utilities menu)

3 – In the Terminal, enter the following command:

chflags restricted /V*/*/private/var/db/KernelExtensionManagement

4 – Reboot the machine normally

Once you have done these steps, you should be back in business.

If you’re a developer of a kext and have customers affected by this issue, please report it to Apple in as much detail as possible. The more details they have, the faster they’ll be able to find the bug and fix it.

How I became a Mac security researcher

Over the years, I’ve been attacked and criticized many times over my views on security. At times, it’s been completely justified, while other times, it stems from not knowing the things that I know.

Thus, spurred on by events that are ultimately unimportant, for the first time publicly, I’ve decided to tell the entire story of how I got into security, how I ended up at an antivirus company, and how and why my views have changed. This is the story of someone who went from a rabid “Macs don’t get viruses” fanboy to a professional malware researcher, and why exactly such a strange turn of events occurred. With a smattering of stories about the history of Mac malware thrown in. 🙂

“Macs don’t get viruses”

In January of 2006, I joined Apple’s discussion forums, termed “Apple Support Communities,” or ASC for short. Around this same time, I was attempting a poorly-thought-out transition to teaching high school science, which I would soon learn I was not cut out for. A few years later, I failed at it and quit before I could finish paying off the debt from getting my Master’s degree. My self-esteem was at an all-time low, and I vented steam at people on the forums. It was not a time I’m particularly proud of.

At that time, I firmly believed that the Mac’s malware problems from the early days were a thing of the past. Before OS X, Apple’s “Classic” system (as we call it now) had all manner of malware. In fact, the first widespread computer virus – the Elk Cloner virus, which appeared in 1982 – actually affected the Apple II computer, a precursor of the Macintosh, and not a PC running DOS or Windows.

Disinfectant logo

The early Macintosh was no stranger to threats, and antivirus software was soon to be considered important. John Norstad’s Disinfectant, introduced in 1989, was considered by many to be the best, though other early pioneers in the antivirus industry were also offering some of the first antivirus software on the Mac, such as McAfee’s VirusScan.

Working in the campus computer lab as a college student, I encountered my first piece of malware, a virus known as WDEF, which spread from disk to disk automatically, infecting any floppies inserted and spreading to the next Mac that used it. (The virus infected the “desktop file” present on every disk on the Classic Mac system.) Despite this, my interest in malware would not truly start until nearly two decades later.

Once Apple transitioned to the Unix-based OS X, all the old malware was suddenly obsolete. OS X was a drastically different system, viruses written for Classic systems no longer worked. On ASC, I routinely told people that Macs didn’t get viruses, thanks to my preconceived notions and lack of knowledge to the contrary, and, frankly, acted like a bit of a jackass.

Information tips the balance

At some point, someone – and I couldn’t tell you for the life of me who it was, or precisely when this happened, at this point – challenged me with some concrete information. I set out on a frenzy of research to refute his (her?) arguments. But my determination to prove that I was correct took a turn that would change my life: I learned that I was actually wrong!

Needless to say, this was a humbling experience. More importantly, though, I felt like I was glimpsing the periphery of a hidden world. I decided I wanted to know more.

Over the next few years, I believe that I dug up information on every piece of OS X malware that ever became publicly known. I knew the symptoms, the dangers, and how to remove every obscure piece of malware that nobody ever saw anymore. I started monitoring for and documenting the emergence of every new piece of malware. In 2011, I created The Safe Mac to serve as a repository for this knowledge and a way of educating people about the potential dangers.

The Safe Mac logo

I was still, at that time, a bit negative on the benefits of antivirus software. I was not nearly as rabid about it as I once was, of course, as I realized that it could provide a valuable service to some people, though I felt I personally did not need it, and that others who shared a similarly technical mindset didn’t need it either.

Adware Removal Guide

I started noticing an increase in a particular kind of malware that become known as adware, because its goal was to steal from advertisers, search engines, and other affiliate programs in general. In 2013, I knew I was onto something big when an adware company I had blogged about threatened to sue me if I didn’t remove all content about them from my website.

Genieo legal threat
Threat posted publicly in the comments on The Safe Mac

This was a particularly scary event, as this legal threat could jeopardize my family’s financial future. I ended up refusing their demands. They backed down, I breathed a sigh of relief, and I continued with my (unpaid) work.

In hindsight, this outcome probably could have been predicted due to the fact that Genieo rather amateurishly posted the threat publicly, for anyone to read. Still, this was an eye-opening event that underscored to me the idea that “adware is malware with a legal team.”

Since the adware problem had been on the rise, I was answering large numbers of questions on ASC about adware removal. In late 2013, I decided to create a set of pages on The Safe Mac that I called the Adware Removal Guide. These pages provided information to help the user figure out what adware they were infected with, and how to remove those infections.

Heading into 2014, I had learned that people had great difficulty following manual removal instructions. I do not mean to imply that they were stupid, as some would uncharitably say; they were simply unsure of themselves, or over-sure of themselves, or perhaps just not particularly careful readers. Whatever the case, I went through many iterations of my instructions, and through trial-and-error learned a lot about how to write those instructions clearly.

Creation of TSMART

TSMART code

In an ironic twist, it was the actions of that same adware company I had butted heads with that spurred me to try something different. Genieo’s adware was installing files in such a way as to render the system incapable of starting up if one file were removed without removing another. Countless people ran afoul of this issue after failing to follow my removal instructions to the letter, and some of them became very irate with me over the issue, as if it were my fault.

This, coupled with a timely suggestion from someone (who, I can no longer recall) that a shell script might be handy to automate removal, sparked an idea. Soon, the abysmally named TSM Adware Removal Tool (abbreviated TSMART) was born. This tool was written in AppleScript, compiled into an application to make it easy to run, and it automated removal of everything described in my removal instructions.

It wasn’t long before the adware makers escalated with techniques like randomizing file names, making my script more difficult to maintain. Every change required a complete update of the script, and although I built functionality to alert the user when an update was available, it just became too cumbersome.

I did a thing…

AdwareMedic logo

Thus was born the idea for AdwareMedic: an application that supported flexible identification logic coded into rules, which could be updated independently of the application itself. It certainly wasn’t my intention to build an antivirus program – indeed, AdwareMedic was specifically meant to target adware, not malware – but evolution sometimes results in two different paths to the same end.

The first version of AdwareMedic was written in a very short time – a weekend for the first prototype, and about a month for something that could be released, if memory serves. It was done as a fun project that could help some people. Next thing I knew, it had become a huge success. I didn’t have a formal way to track its usage, but I firmly believe that by early 2015 there were hundreds of thousands, maybe even millions, of Macs running AdwareMedic. I began to hear that Apple Geniuses were recommending it to people!

AdwareMedic became, I believe, the first real challenge to adware companies on the Mac platform. So much so that I saw adware responding and changing behaviors based on AdwareMedic detections. At one point, one piece of adware, called VSearch (also known as Pirrit), began fighting back by modifying the content of the AdwareMedic website when viewed from an infected machine… in the initial stages, by redirecting the Download button to the MacKeeper website.

I fought back, of course, with scripts designed to detect those modifications and show the user information about how to combat the issue. The VSearch folks escalated each time, eventually going so far as to block access to the AdwareMedic site by replacing the site’s content in the infected browser with a fake “server not found” page.

This behavior directly resulted in Apple adding rules for VSearch to the XProtect antimalware feature in macOS.

rule VSearchA 
 {
     meta:
         description = "OSX.VSearch.A" 
     condition:
         Macho and
         filesize <= 2000000 and
         ( hash.sha1 ( 0 , filesize ) =="6c6acb179b232c0f1a6bb27699809320cc2c1529" or
         hash.sha1 ( 0 , filesize ) =="cebb19fee8fd72c0975ea9a19feea3b5ce555f94" or
         hash.sha1 ( 0 , filesize ) =="1503f1d7d275e976cd94cfd72929e0409e0cf76a" or
         hash.sha1 ( 0 , filesize ) =="c50adfa949a70b33d77050d7f0e2f86bccbc25cf" or
         hash.sha1 ( 0 , filesize ) =="40346b3946d7824d38f5ba71181f5c06805200af" ) 
 }

Do you have time for a quick chat?

Next thing I knew, I got an e-mail from Malwarebytes CEO Marcin Kleczynski. I didn’t know much about them, but what I read was intriguing, and Marcin was very straightforward.

When I flew out to California for a meeting, I learned that Malwarebytes started out in an almost identical fashion. Marcin started out on forums, after having infected the family computer. He built up a foundation of knowledge, with the help of some friends made on forums, and built a product. In a million years, I couldn’t have imagined a better fit.

Once I was on board at Malwarebytes, a whole world of possibilities opened up. With AdwareMedic, I had never dared to remove certain things. My experience with Genieo made me gunshy. I simply couldn’t afford to risk a lawsuit, and yet adding and later walking back a detection would be nearly as bad, as it would show that I could be bullied into submission.

Malwarebytes does not tolerate PUPs (potentially unwanted programs), and isn’t afraid to go to court when threatened by PUP vendors. With Malwarebytes at my back, I was finally able to start doing things that AdwareMedic users had been requesting for a long time – like removal of MacKeeper!

The end…?

Thus ends the story of how I went from rabid Mac malware denier to a creator and proponent of security software. If you were to ask me to sum up with one single thing that pulled me in this direction, that thing would be knowledge. It’s easy to have strong opinions in a position of ignorance. It’s also easy to keep those opinions if you never challenge your own preconceptions. But it’s very hard to maintain an opinion in the face of contrary information, and you’ll never discover that information if you don’t seek it out.

Of course, this is only the end of the story of how I got here. I’ll have more stories to tell from the trenches in the future, as I fully intend to keep fighting the scumbags that are intent on turning my favorite platform into a cesspool of malware and crapware.

My years of research and experience in the trenches taught me an important lesson: people need help. It’s not their fault that they’re getting infected. The creators of malware, adware, and other unwanted programs are extremely good at tricking people, and they’re not going to stop. We’re slowly heading towards a world where Mac malware becomes as sophisticated as what’s seen on Windows.

Even now, not all threats involve tricking the user… sometimes, malware spreads through stealthy techniques that infect even the most savvy among the tech community. As an example, read the story of how Panic, Inc got hacked.

My request to you, dear reader, is this: spread the word. Mac malware does exist. Mac adware is just a millimeter shy of malware, and you don’t want that infection any more than you want malware. In fact, I’d willingly infect myself with certain pieces of malware before I would do the same with most of the adware out there!

Am I saying you have to force your friends and family install antivirus software? No. I believe it is a useful tool for many people, but I also admit it’s not a silver bullet. I make antivirus software, and I’m telling you my software is not, and never will be, perfect! Anyone telling you differently about their own software is lying to you.

Whatever you do, if you are able, help those you know to stay safe online, in whatever way suits them best. Just keep in mind that what suits someone best is not always what suits you best… that’s a lesson I’ve learned the hard way, repeatedly.