Why 99% Hit Defer Every Time with Robert Hammen, SAP, ex-SpaceX

Robert Hammen [00:00:00]:

Change is the core of it. If you don't like change and you're working in it, you're in the wrong field.

 

Arek Dreyer [00:00:12]:

Today's guest is Robert Hammen, Principal Mac consultant at SAP and former IT systems engineer at SpaceX. Robert spent his career managing fleets of devices at scale, bridging the gap between IT operations and engineering, and championing smart, secure approaches to patching. In this episode, we'll dig into one of his core beliefs, that patching doesn't have to be painful if you strike the right balance between enforcement and end user flexibility. Robert, welcome to Patch Me If You Can™.

 

Robert Hammen [00:00:44]:

Thanks, thanks. Glad to be here, Arek.

 

Arek Dreyer [00:00:45]:

Well, let's start with patching. Why do you think it's still such a challenge in enterprise it?

 

Robert Hammen [00:00:52]:

Well, I think in a lot of cases the tools aren't exactly great. A lot of the built in ones, there are certainly third party tools that a lot of people can do for various, you know, for patching and software management. And I think this has been a focus of the industry for some period of time. It just seems like things are starting to get a little bit better. I think a lot of people also think that patching is simple. You just enable automatic updates and then away you go. And the reality is if you don't validate that the updates are being applied, you're going to run into problems where things happen. And this happened to me too.

 

Robert Hammen [00:01:21]:

So users also do fun things like move or rename applications. You know, if I had a dollar for every copy of Google Chrome 2 app that I found on the system, I'd probably be able to buy a nice car. But in general, I think that people don't really think about the patching. And it's both users and administrators, they don't realize how important it actually is. Especially if you work in a very high security environment where you're always in danger of being hacked. You want to make sure that you get those patches applied on a very timely basis.

 

Arek Dreyer [00:01:51]:

Was there a moment that really cemented your belief that building a patching strategy that includes both deferrals and enforcement is super important?

 

Robert Hammen [00:02:02]:

Yeah, in my previous role I had people who were on console when things were being sent to space. So naturally you don't want to just push an update to them because that's kind of bad. Although that did actually happen once when the user ran out of deferrals. But that's why I'm a big believer in deferrals. You know, you let the user know. I mean, and I had scripting logic built so that, you know, I would look for the process running, you know, in the background. If the process wasn't running, I would just go ahead and push the update. If it was running, I would prompt the user and say, hey, you know, there's an update available for Chrome, you've got X number of deferrals, what do you want to do? And 99 times out of 100 people would hit the defer button.

 

Robert Hammen [00:02:40]:

It seems to be a cultural thing here. And I was having a conversation with another friend of mine who's from the UK and he made the comment that he's now working in the US and he made the comment that working in the uk people grumbled about patching but understood it. But here in the US people tend to push back more. I don't know if it's hustle culture or what, or just cultural differences upon it, but you know, it's definitely a challenge. And I think a bigger thing too is you have to explain to users why this is important. Like, hey, there's a zero day vulnerability that affects, you know, Google Chrome, for example, or Mac OS and we really need people to update ASAP because it puts our organization at risk. And you know, a patch to Spotify is probably not as important as a patch to Chrome or the os. So you have to take those things into consideration as well.

 

Arek Dreyer [00:03:24]:

It's really interesting that you mentioned that for Chrome you were looking for the process name, not the app name. So even if the user's like, hey, I don't want to be bothered by these deferrals and I just want to use Chrome and you know, I'm not doing anything really important, so no one's going to target me, no one's going to hack me. And they changed the name. Even if they change the name or copy it to their desktop and run from there, you're looking for the application name so that you can see, hey, this is running, it's vulnerable, it needs to be patched.

 

Robert Hammen [00:03:57]:

Although depending on how you're collecting the information, you may end up prompting the user to keep reinstalling the app. If there's multiple copies of it on the system, the old one is in a different location or with and it's. And so it's not being replaced by or overridden by an upgrade, that's really.

 

Arek Dreyer [00:04:16]:

Important to watch out for. You mentioned communicating with users and saying, hey, this is really important. How have you established kind of baseline communications with users so that the first time they hear from you isn't necessarily, oh my gosh, Our hair's on fire. Here's what I need you to do so that it's not a surprise, it's not an out of band communication. Yeah.

 

Robert Hammen [00:04:42]:

And you know, it depends on the significance of the issue. I mean there have been times when it's been like, okay, we have to email everybody and tell them to update, you know, because there's this really bad bug or in other cases it may be something where you have like a Slack channel or a teams channel or some method of communication where you can, you know, one to many, where users can go look for the information. Why am I getting prompted to upgrade Chrome again when I just upgraded it two days ago? Oh, there's a new CVE and I oftentimes would link to, you know, NIST or other, you know, bleeping computer type blogs showing, you know, what was going on and was being exploited in the wild and why it was important.

 

Arek Dreyer [00:05:15]:

Yeah. So it's, it's by providing some context, but not too much context, you establish that trust with users.

 

Robert Hammen [00:05:27]:

Right. And that's, that's the key is finding that balance. A lot of times it'll be like the executive summary. And you know, Apple Intelligence is really good at like helping write that stuff. That's one of the things that actually works well. I've used that when writing documentation and things to make things more concise or make them clear. Sometimes it mangles it, but it can be helpful.

 

Arek Dreyer [00:05:46]:

So once you've, you mentioned the script and if the script that you wrote that if the app was not running it would just update it so you weren't bothering users with information they didn't need, it would just update exactly once they're, if they did need to be interrupted, tell me about how much time they had between the time that like, hey, there's an update and the time that, okay, now you've had enough time, we really need to do this.

 

Robert Hammen [00:06:13]:

Sure, sure. And we, we generally tried to be reasonable and set the deadline for. I think you got five deferrals so you were prompted once a day. So if I get an updated, prompted at 2pm to update Chrome on a Tuesday, I'm not going to see another prompt to upgrade it on 2pm on Wednesday. And of course, you know, there's still the NFC people being in the middle of something and there's a defer, defer, defer. And then when they hit that last time, it's like, oh, you're getting up, you're getting quit and updated and there's some pushback. Hey, I lost Data. It's like we gave you all these chances and there was some education that we had to tell people that look, if you want to do this on the off hour, you can use a self service type tool or a managed software center or what have you to go in and install the latest version from there that will update you and like oh, okay.

 

Robert Hammen [00:06:58]:

So they kind of give them the power to go update on their schedule.

 

Arek Dreyer [00:07:01]:

So kind of building that trust without compromising your compliance.

 

Robert Hammen [00:07:06]:

Right.

 

Arek Dreyer [00:07:07]:

Once you had this system in place for patching and making sure that patches were installed reliably, what kind of work did that unlock for you and your team?

 

Robert Hammen [00:07:16]:

So it really helped us once we had this in process to start doing a lot more with automation and different tooling. There's various tools out there that can download packages and upload them to your MDM or there's add on tools for a lot of MDMs or built in tools to various MDMs for this. And so we got to the point because some things, obviously new OS is a matter of things that everybody has to test, right? So you don't want to say hey Mac OS 26 is out, everybody upgrade. No, you need to make sure you've tested it and it's qualified. But for some other things that get updated frequently like Chrome or Office apps, we really don't have the luxury of taking time to test. Sometimes that's problematic for people. The most recent Office version had a lot of issues, but anyway for a lot of these things, you know, we had an automated pipeline where you know, the new versions were downloaded, added to the MDM and then like made available, you know, in the software installation tools that users could upgrade or you know, if they would remove and reinstall, they would get the latest version. So I think automation is key if you can and that's the key to having especially as you scale into a larger and larger environment.

 

Robert Hammen [00:08:29]:

If you can do, you know, if you can detect problems on an endpoint and fix them without the user knowing, you are way ahead. You know, if you have to ask the user to do something now you're again you're bothering them and they're in the middle of doing their job and you know, they're not IT people, they don't want to do it work. So it kind of relates to common, common remediations. Like a lot of things that are common as a file vault keys get not escrowed or invalid or bootstrap tokens for secure token access are not escrowed near mdm. And there's tooling like Escrow Buddy and Bootstrap Buddy, which I've implemented in multiple environments now that just make the process of detecting and remediating I don't have to think about it. I'll just periodically go look and go, oh, I see we fixed this on 63 machines. Another one that a lot of admins don't really Understand is the ROSADA 2 emulation environment for Apple Silicon Macs. A lot of orgs will install Rosetta 2 automatically because they don't know what software the user might need, so they'll put it on the system.

 

Robert Hammen [00:09:27]:

So you think at that point you're done, right? Except there's a nice little bug in macOS that's been going on for years where sometimes after a software update Rosetta gets uninstalled. And the way you find that out is something doesn't work like oh, I can't print, oh, I don't have Rosetta. When these drivers need Rosetta or my Software for managing 802 likes Ethernet access isn't, isn't Apple Silicon native, but a bigger one still. And I've run into this all the time now is the way the behavior of the macOS installer. If a installation package in macOS doesn't specifically declare that it supports Apple Silicon, the installer app attempts to use intel to evaluate it. And if Rosette is not installed, the package installation fails. So you've got a good package on a system that should be able to run it and it fails because Rosetta is not there. So you have to build in this logic to detect if Rosetta's there's and remediate it.

 

Robert Hammen [00:10:16]:

So that's the kind of thing that, you know, you think setting and forgetting is easy, but you got to follow up and actually see all the edge cases where things go off the rails.

 

Arek Dreyer [00:10:25]:

So having kind of your baseline patching systems in place frees up time for you to work on these automations to cover the corner cases and to really build those automations out and test them and make sure that they're, they're, they're working for you. Do you, do you have anything about rolling out patches to kind of a test group or anything like that?

 

Robert Hammen [00:10:49]:

Yeah, and a lot of it depends too. If we're, if we're, you know, especially where I work now, where there's tens of thousands of Macs, you know, we don't just blast a package out to all those machines at once. Well, a lot of times we'll stage this where we'll do like deployment Groups. So we'll, we'll scope it to all devices that are eligible for it, but that will exclude, we'll have specific groups that we set up statically that, you know, deployment group one that will add those as exclusions. And we'll like remove one one day so it deploys to like 20% of the fleet and then remove one the next day if there's no problems. So we kind of expand it out. You know, it's really contingent on the number of devices. Like when you start hitting more than 10,000 devices, you really should have a strategy for that.

 

Arek Dreyer [00:11:25]:

Is 10,000 kind of the tipping point for you in your experience?

 

Robert Hammen [00:11:30]:

I think so, yeah. Once you start getting into the big numbers like that, then yes, the last thing you want to do is accidentally blast out a bad update to, you know, 10,000 plus systems. Because that could be a lot of manual remediation work to get that sorted or a lot of help desk tickets.

 

Arek Dreyer [00:11:46]:

Which is what you're trying to avoid in the first place.

 

Robert Hammen [00:11:48]:

Exactly.

 

Arek Dreyer [00:11:49]:

So earlier you mentioned using Apple Intelligence to help you write your documentation. I know that you're also concerned with vendor documentation. If you want to share some thoughts about the state of vendor documentation here in 2025.

 

Robert Hammen [00:12:07]:

Yeah, that's kind of a challenge really because the number of MDMs out there is growing and now with the new features in macOS 26 where people can migrate MDMs, I fully expect more and more people. You know, it's been a monoculture in a lot of ways and I suspect it's going to get a lot more diverse with a lot of providers out there. So the problem you have is that you have a lot of companies that produce documentation. They'll do it for one MDM because it's the one that is the market leader. But then you have somebody trying to use a different MDM and they have to like try to interpret the instructions. So I'm of the opinion that, you know, anytime, especially when you're dealing with like security software that needs a configuration profile, sometimes they'll just throw up a mobile config on their website or in their support page and say, here, download, install this. And it's like, I'm not going to install this on a machine, you know, I'm not just going to willfully download it and upload it because I don't know what's in this, especially if it's signed, you know, without going through it, unsigning it and looking at the contents. But I'd like, I will, I would like instructions because maybe I don't want to deploy a single profile or maybe I can't deploy a single profile to deploy, you know, for this.

 

Robert Hammen [00:13:12]:

Because an example would be if the product has a Chrome extension, you have a different profile as Chrome Extensions. You can't have two profiles that force install Chrome extensions, they have to be combined into one. So I need the instructions on how to what their extension is to add to my existing Chrome profile rather than have a kind of like monolithic profile that has all these settings for this app.

 

Arek Dreyer [00:13:31]:

So you'd like to see the recipe for the profile and not necessarily just the profile.

 

Robert Hammen [00:13:36]:

Yeah, I think I'd like both. But you know, and then when it comes to that and when it comes to documentation, the worst offender is the one that makes the platform because they don't do the greatest job with their documentation. It's gotten better, I will say that. But like, you know, change logs for Apple use, Apple products and enterprise networks don't exist. And I have harped on this product, this issue for years, you know, because Apple's constantly changing this. And this is the thing that breaks in a lot of corporate environments where there are firewalls and things that are, where traffic is inspected or whatnot. And also things like release notes. There have been times where you know how Apple builds release notes for the OS is different.

 

Robert Hammen [00:14:10]:

They will, some things they'll flag. These are developer related issues, some are consumer related and some are enterprise. And on multiple occasions I've seen things like oh, fixed an issue with OneDrive causing problems, wasn't on the enterprise release notes should have been, you know, things like that. So and in general I think Apple, you know, the Apple does have a lot of extensive documentation and I know you've spent a lot of time digging through it but on things like MDM and ddm but they don't give you enough examples. And I know, I know there have been many times when you know, there were some enrollment changes in Mac OS15.4 and basically the Apple documentation on the impact was relatively non existent and we were like these candidates out. It's like okay, I'm digging in and here's what I'm finding and posting on Mac Admin Slack and saying hey, if you haven't paid attention there's a lot of changes coming and this is going to interfere with your workflows and you're going to have to update your documentation on enrollment because screens are changing and.

 

Arek Dreyer [00:15:05]:

They'Re bugs and so you'd rather have kind of a centralized source of information as opposed to having to go out and look at different types of release notes and Slack, stuff like that.

 

Robert Hammen [00:15:18]:

Yeah, I mean, yeah, generally when I'm looking for information on stuff, it's like, yes, I'll start with Apple, and you know, I'll start looking and then also on Slack and I'll look at Rich Carton's blog and, you know, and so on. And eventually I'll. And, you know, and eventually I'll find enough material and I may end up, you know, rewriting it or linking to it or, you know, putting it on internal documentation pages.

 

Arek Dreyer [00:15:40]:

Sure.

 

Robert Hammen [00:15:41]:

So it's always a. It's always a challenge. And again, more examples is good. The problem I have with some vendors is they write documentation like it costs them a million dollars a word. Verbosity is great in some respects, but when you have a technical audience, you really need details and examples.

 

Arek Dreyer [00:15:56]:

So why do you think documentation hasn't kept pace with the complexity of our modern IT environment?

 

Robert Hammen [00:16:03]:

Well, I think it's through the rate of change, you know, things we've gotten. The cycle now we've been at for what, 14 years where we get a new OS every year, a new Mac OS every year, and iOS has been that way since it came out. So, you know, it seems like we barely get through, you know, an OS release and fixing the bugs that OS released, and all of a sudden here's the next new one coming out. And so, you know, it's like a treadmill from maybe a, you know, like a hamster wheel.

 

Arek Dreyer [00:16:29]:

I mean, on the one hand, I definitely appreciate the velocity and there's stuff that I can't wait until it gets here. And on the other hand, I'm like, oh, hold on, slam the brakes, because I need to get ready for it. So we're in a funny position as.

 

Robert Hammen [00:16:43]:

Practitioners, and this is definitely the time of year when we are trying to keep the current environment running and fixing things while also seeing if the next environment that's coming out is going to break stuff.

 

Arek Dreyer [00:16:54]:

So if you had to give IT teams one mindset shift to adopt today, based on what we've talked about, based on what you've learned, what would it be?

 

Robert Hammen [00:17:04]:

Well, besides the necessity of patching, I think that you have to kind of aligns with the last comment we had about change. Change is the core of it. If you don't like change and you're working in it, you're in the wrong field. And the other thing, I think especially this is an ongoing thing. Companies need to periodically evaluate their decisions they've made about how they've implemented technology or what technology they're using. You know, companies that get in the mindset, well, we've always done it this way and don't look to see if there's a better way to do it or to change things, whether it be things like, you know, naming conventions or whether it's the approach you take in deploying or updating software, you know, you periodically need to say, is there something better out there for us to do? So.

 

Arek Dreyer [00:17:47]:

Yeah, and sometimes the technology forces that change. Right. Like back before iPhone was embraced in big heavy use by. In enterprise, everything was Active Directory bound to Active Directory. And you know, when old triangle. Yeah, yeah. And when you thought about, well, okay, well, we can't do this unless you bind to Active Directory. Well, with the iPhone, you can't bind to Active Directory.

 

Arek Dreyer [00:18:13]:

So that, that was a forcing function for it to kind of reevaluate how they're doing, network access control and so many other different things. And so sometimes that that change is forced upon us. But it sounds like you're advocating for organizations to constantly reevaluate that before the change is forced on them.

 

Robert Hammen [00:18:34]:

Right. And a good time to do that is when the new OS is released. Because again, if you're dependent upon ad binding and ad binding is going to go away, which it seems like that it's still there in macOS 26, but I wouldn't take bets on 27, you know, that you definitely need to have a plan what you're going to do for that. Another example I can give you is I remember during the pandemic and having just started a new job, like less than two months before everything happened, and all of a sudden everyone is remote and dealing with mobile accounts and file vaults and password expert. And then because the computers weren't people were not in the office ever and rebooting their Macs, the password for FileVault never got updated. And so people would reboot their Macs and get locked out of them and not know how to get into it. And then there'd be all kinds of support cases and phone calls. And you know, that's the kind of thing that you sometimes you have to, you know, move on the fly to address or not have it.

 

Robert Hammen [00:19:29]:

Not having a video conferencing solution and suddenly everybody being remote, that's kind of a problem. You got to roll something out really quickly, you know, sign a deal with a vendor and go and get it deployed to the fleet.

 

Arek Dreyer [00:19:39]:

So funny you mentioned filevault because like with iPhone and iPad, you, you don't have to worry about that.

 

Robert Hammen [00:19:46]:

Right, exactly, exactly.

 

Arek Dreyer [00:19:48]:

And so I can envision a future where Mac doesn't have that issue. Mac doesn't have FileVault, but it's as secure as iPhone and iPad and I can see movement in that direction. I mean I can see when the Apple Silicon Mac first came out and Apple's like, well, hey, macOS is running and it's on the network, but you can't actually do anything. That made me really excited, like, well, what can we do? Nothing now. But as the oss progress, we're seeing that there actually is things that you will be able to do with the Mac that has restarted it at that pre file vault login window. Right now with Mac OS 26, you're going to be able to SSH unlock it with ssh. Yeah, that is fascinating. You can't send MDM commands, you can't send MDM profiles and queries right now with Mac OS 26.

 

Arek Dreyer [00:20:48]:

But I am so excited for the future of that, that, that state and what we may be able to do with that, you know, platform SSO at FileVault.

 

Robert Hammen [00:20:59]:

That's a big one.

 

Arek Dreyer [00:21:00]:

Super, super exciting. Can't wait to see it in production.

 

Robert Hammen [00:21:06]:

And all the guest mode features too in the new os. So again, I think we're getting to a point where you're probably right, where we're not going to have a choice about whether it's encrypted device, just encrypted and we're kind of removing all the limitations. Why you wouldn't want to encrypt the device. Like I had caching servers in my previous role and I certainly did not want them encrypted because I didn't want somebody to have to take a crash card into a data center and plug in just to unlock the disk if it was out or the machine kernel panicked or whatever.

 

Arek Dreyer [00:21:32]:

Right.

 

Robert Hammen [00:21:32]:

And so, you know, now that we have a solution for that and with.

 

Arek Dreyer [00:21:37]:

So when you were, when you were running those caching servers, did you have a carve out in your internal compliance so that, hey, these are computers but they don't have full disk encryption. Like how did you deal with that?

 

Robert Hammen [00:21:50]:

Yeah, yeah, I definitely had to, you know, you know, obviously Information Security and Assurance, you know, came up and says, hey, we got these Max and ioncrypt is like, yep, those are caching servers. They live in a data center and they need to be excluded. Okay. It's all it took is one line justification.

 

Arek Dreyer [00:22:05]:

So you just documented it, right?

 

Robert Hammen [00:22:07]:

Documented. And that's the big thing too. I think too many people take these tools like Nessus and so on, the vulnerability scanners as guidance, and they're not, they're tools to tell you what, how a device is configured, what may be installed and what made and what services are active or not. You know, a lot of times they'll do things like flag older Perl or, you know, binaries or something like that. You know, on things that are SIP protected, it can't be rooted in macOS. And people say, you've got to uninstall this. And I'm like, I'll break the OS if I do. So this is not a good idea.

 

Robert Hammen [00:22:38]:

You know, so there's a lot of, there's a lot of that. And again, I think people, a lot of people want to hit the easy button, you know, they want to say, okay, this tells me that I got to fix this, so I'll fix this. You actually have to look and see, you know, what's the most important thing. Obviously if you use Chrome as your default browser across the. Org and there's Chrome vulnerabilities being spoiled in the wild, that's your important thing. So is the os. You know, like I said, you know, Pearl built in, the OS is way lower on the, you know, in your, in your threat model assessment of what vulnerabilities could affect your system. And it is important, I think, for people in it to also, to build those bridges to the people in information security and endpoint, you know, security tooling and compliance to try to, you know, say, hey, look, I get, you know, I'm trying to make, I'm.

 

Robert Hammen [00:23:21]:

Everything I'm doing is trying to make your life easier.

 

Arek Dreyer [00:23:24]:

Right, right. Well, speaking of the easy button, if you could instantly patch something in your world, what would it be?

 

Robert Hammen [00:23:33]:

Well, I probably would. And again, I'm going to limit this to tech because I don't want to get political. But in general, I think that if I could make something easier, it'd be being able to more easily update the os. And we're getting better, certainly with declarative device management. That's the biggest pain point for a lot of organizations is getting people to update to the latest os. Besides ddm, there's tools like Nudge and Super that people use and stuff like that. One that actually works really well from my perspective, is the whole Microsoft device compliance piece, because basically if you don't have a compliant device, you lose access to your email and teams. So you basically can't do your job.

 

Robert Hammen [00:24:12]:

So that kind of is the carrot and stick approach against the stick to upgrade your os. And again working in an environment with tens of thousands of max, the compliance ratio is really high, higher than any other place I've worked. So that actually does work in real life but that queries of course to have a Microsoft backend again. I think there's a lot of push by Apple to get this to happen and we used to be able to defer software updates forever and now we have maximum of 90 days and so on. So that's another mindset change that organizations need to have is like you're going to have to support the new os, especially if you're planning on buying new hardware that requires it.

 

Arek Dreyer [00:24:49]:

I can foresee that wish from you coming true in the future. Well thank you Robert for joining us on this episode of Patch Me If You Can™. If you liked this episode, hit follow and share it with someone who's ready to lead it and security from the front. We'll see you next time.