Back in summer of 2018, customer support at Malwarebytes started seeing people with problems activating the kernel extension (kext) in Malwarebytes for Mac. This opened a can of worms that we’re still struggling with today… as soon as we think the worms are back in the can, we start finding new ones. Unfortunately, these worms all belong to macOS, and are affecting other kexts as well.
A history of previous issues
We ultimately traced the initial problem to an issue with the /Library/StagedExtensions/ folder. This folder is used by macOS in the process of activating a kext. By default, this folder is protected by SIP, so it can only be modified by the system.
Unfortunately, we were seeing that a number of users had a StagedExtensions folder that was lacking the restricted flag. (This flag is an indication that an item is protected by SIP, and can be set via the chflags command, although only from recovery mode.) This interfered with activation of the kext, since macOS no longer considered StagedExtensions to be a secure location at which to stage the kext.
The symptom of the problem was pretty easy to spot, if you knew where to look. If you run the following command (where the -O switch shows the file flags):
ls -alO /Library/StagedExtensionsThe output should show this for the StagedExtensions folder itself:
drwxr-xr-x@ 4 root wheel restricted 128 Nov 6 13:37 .
However, on affected systems, it showed:
drwxr-xr-x@ 4 root wheel - 128 Nov 6 13:37 .
Notice the lack of a restricted flag on the folder, indicating that it is no longer protected by SIP. Of course, this flag should only be able to be cleared by the OS itself, or by a user in recovery mode. There was no reason for users to be clearing that flag themselves, and many of the users struggled with the process of setting it again, so every indication pointed to a system bug.
This was reported to Apple on August 3, 2018 (radar FB5361175), and was fixed in macOS 10.14.1, which I confirmed, and thus closed the bug report on January 28, 2019.
Later, we started seeing the issue again in macOS 10.15 and 10.15.1, and opened a new bug report on November 6, 2019 (radar FB7430509). This bug ended up being closed by Apple for the reason “Unable to diagnose with current information,” which might have irritated me, but the problems stopped again. It must have been fixed independently of our bug report, I guess…?
The current state of kext issues
Unfortunately, the problem is back yet again. This time, however, the problem does not involve the StagedExtensions folder. This time, the issue involves the folder /private/var/db/KernelExtensionManagement/, which is a newer folder that is used in the process of activating kexts on more recent versions of macOS. I’m unsure when this folder first appeared, or exactly how it is used for kext activation compared to StagedExtensions.
Affected Malwarebytes users see an error message in the app and are unable to turn on the real-time protection (RTP) features. In many cases, these users had RTP turned on previously, but suddenly found that it was off. In other cases, these are new installs where users are unable from the start to enable RTP.
Observations of affected users showed exactly the same problem as with StagedExtensions: the KernelExtensionManagement folder had somehow had the restricted flag cleared, meaning it was no longer protected by SIP. Further investigation, via the kextutil command, revealed more information:
$ kextutil -tn /Library/StagedExtensions/Library/Application\ Support/Malwarebytes/MBAM/Kext/MB_MBAM_Protection.kext 2>&1 Error making temporary directory: 1 Memory allocation failure. Unable to stage kext (/Library/Application Support/Malwarebytes/MBAM/Kext/MB_MBAM_Protection.kext) to secure location.
On affected systems, we’ll often see this same output for a number of other kexts as well.
We’re only seeing this on 10.14.6 and, to a lesser extent, 10.13.6. We don’t really know if the issue affects Catalina as well, as Malwarebytes no longer uses a kext on Catalina, so we don’t have any data from user reports to go on there.
According to the data we’ve seen so far, all but three cases all involve macOS 10.14.6 (18G6032), which was released on September 24. The other three cases (so far) involved older systems, released this past spring or summer: macOS 10.13.6 (17G13035), macOS 10.13.6 (17G14033), macOS 10.14.6 (18G6020).
We saw a few isolated cases months ago, but have been seeing a more significant increase in incidents, starting very roughly sometime in September. So far, there aren’t enough incidents to really make any strong conclusions, other than that it looks like another macOS bug.
This was reported to Apple on October 19, 2020 (radar FB8812838), but there has not been a response yet.
A solution to the problem
The best solution would be for Apple to solve this issue. I do believe that will happen, as we’re not the only ones seeing (and, hopefully, reporting) this issue. There’s a thread on Apple’s developer forums from other people seeing the same problem.
https://developer.apple.com/forums/thread/657789
If you are seeing this issue, or are managing machines with this issue, there is a somewhat easy fix, though it will be quite cumbersome for IT admins trying to fix problems remotely during COVID.
1 – Reboot the affected machine in recovery mode (hold command-R at startup)
2 – Enter the Terminal (from the Utilities menu)
3 – In the Terminal, enter the following command:
chflags restricted /V*/*/private/var/db/KernelExtensionManagement
4 – Reboot the machine normally
Once you have done these steps, you should be back in business.
If you’re a developer of a kext and have customers affected by this issue, please report it to Apple in as much detail as possible. The more details they have, the faster they’ll be able to find the bug and fix it.