79: Sociotechnical Exploitation with Bruce Schneier

This episode has been automatically transcribed by AI, please excuse any typos or grammatical errors.

CHRIS: Bruce Schneier is a renowned American cryptographer computer security expert and author who has dedicated his career to advancing the field of digital security and privacy. His latest book, titled A Hacker’s Mind how the Powerful Bends Society’s Rules and how to Bend Them Back, focuses on how using a hacker’s mindset can change how you think about your life and the world. The long awaited return to Barcode. Bruce, welcome. How are you, my friend?

BRUCE: Thanks for having me back.

CHRIS: Absolutely, man. So first off, Bruce, I want to ask you what inspired you to write this book and focus on this topic specifically?

BRUCE: It’s what I’ve been thinking about. The notion of hacking extends far beyond computers. The idea of subverting rules, of finding loopholes, of figuring out tricks to skirt the law and those aren’t new concepts, but they’re actually hacking and they share a lot in common with computer hacking and the mindset and frame we have to think about. The computer world is more general and there’s a benefit to policy of thinking of these things in terms of hacks, both attack and defense.

CHRIS: Yeah, absolutely. Extends way beyond technology and that’s what I love about this particular book.

CHRIS: And at the beginning of this book you really talk about the importance of now. Why now is important to recognize different types of hacking and then also defend against the malicious formats of hacking. Why do you feel that the importance is more time critical in today’s world?

BRUCE: A couple of things I think we are getting really good at hacking. The rich and powerful are really good at finding tax loopholes or getting their legislative priorities passed or figuring out holes in our regulatory compliance structure. That it’s just getting to be that there’s two sets of law, one for those who figure out how to evade it and the other for everybody else. And that’s just bad for society.

BRUCE: So I think that’s important now. I think the risks are more catastrophic now that these are global systems and whether it’s the Internet or climate or AI or bio, that the risks of this hacking is becoming greater and we need to think about that. I think there is now the chance that AI will start doing hacking, not that they will find, I don’t want to say this, that hacking is a creative process looking at a set of rules that figure out what the loopholes are and that’s a very human process.

BRUCE: But I think it’s going to become an AI process in the next few years and that’s going to change speed, scale and scope. And I talk in my book about that. So that’s going to make things different. So all of that lines up to make this critically important, to say it more generally, it’s kind of like our systems of law and governance no longer work for us as a species. We’re just too powerful as a species for our laws. So we need some other ways of thinking about it. And that’s really what I tried to do in the book.

CHRIS: Yeah, I do want to talk about AI in a moment, but you mentioned systems specifically and this could be AI and beyond, right? What do you feel are the fundamental principles about systems that allow them to be hacked?

BRUCE: So a system is really a set of rules. It’s a set of things that can and can’t be done sometimes they are physical rules. Like you can imagine a mechanical system obeys a bunch of physical rules about how it works, but it can also be a system of laws. The tax code is a system. Our systems of finance and these are sociotechical systems and they are all similar in that they are controlled by the rules that govern them.

BRUCE: And I think that all sets of rules are either inconsistent or incomplete. They have things that the designers haven’t thought of. They have mistakes or circumstance changes and they’re now using a different situation than they were designed for. And so given that all systems are hackable, there’s no such thing as a hackproof system. Just like there’s no such thing as computer code that doesn’t have bugs. We just don’t know how to do that.

BRUCE: It is just so complex and so interrelated, so many interactions that these bugs, these vulnerabilities are inevitable. These hacks are inevitable. So much of our life is governed by these complex sociotechnical systems, our systems of governance, our systems of commerce, our systems of interaction and they’re all hackable.

CHRIS: Do you feel like AI moving forward will help strengthen that at all? Or do you feel like that just becomes another victim in that cycle?

BRUCE: I think it’s both. That’s what makes it interesting. It’s certainly another system. Like we have already seen lots of hacks against AI. It’s called adversarial machine learning in the, in the community. And it’s different ways to manipulate and abuse an AI system. And lots of systems are vulnerable to that. And that’s going to be true in the foreseeable future. I mentioned previously the idea of an AI finding hacks and other systems.

BRUCE: So you feed an AI the entire nation’s tax codes, or maybe the world’s tax codes and you say find me the loopholes. Then you figure out will it tell you to register your company in Delaware and register your ship in Panama and what will it tell you? That we don’t know. So there’s that. I think there’s going to be this usage of AI to find vulnerabilities and systems to find hacks in these socio technical systems.

BRUCE: But those same AI vulnerability finding code can be used for the defense. So we can imagine the software world that a company, Microsoft, anybody, would feed its code to an AI saying find the bugs, find the vulnerabilities and then they get patched before the code is released. So suddenly we are writing more secure, better, more liable code because the AI is finding and fixing the bugs before the release date.

BRUCE: And you can imagine the same thing used against the tax code. So someone proposes a new tax law and then the candidate or a watchdog group or the press runs this loophole, finding AI against this new tax proposal in the rest of the tax law and finds the vulnerabilities, finds the loopholes. It doesn’t mean they get fixed, right? Our legislative process is more complicated than that, but it doesn’t mean they get discovered and they become part of the debate.

BRUCE: So AI in all cases, this is just an example of that benefits both sides, attack and defense. And who comes up on top at the end? I don’t think we know yet. I think that the defense prevails in the long run. Simply because if the defendant AI finds a vulnerability, it gets fixed, but in the meantime, the attacker finds a vulnerability at all the existing things and then can exploit them.

CHRIS: Yeah. So in terms of what we know as the endless arms race, right, with AI technology becoming more mainstream now with tools like ChatGPT, right, do you think that the accessibility of these tools will ultimately increase AI based attacks in the future? And what does that mean for defenders as this technology continues to evolve?

BRUCE: A couple of things I want to tease out of what you asked ChatGPT… these are the AIs that we and the public allowed to play with, and I say aloud very specifically, these are designed by big corporations for profit, and they are letting us play with them because that’s how they’re training them, right? They’re getting better product because of all of our free labor. That’s the exception. Most AIs are owned by these big corporations for their own benefit.

BRUCE: And that’s why I think in this arms race, the advantage goes to the already powerful. Now, that tax break, finding AI, you don’t get to run that. Goldman Sachs gets to run that. It’s going to sit in their basement, it’s going to find tax loopholes, they’re going to sell to their wealthy clients. That’s not going to be something that we are allowed to use unless things change. So the AIs are being built by these powerful companies because you need a lot of power to build an AI. You need computing power, you need training data, you need researchers.

BRUCE: This is not something that we as hobbyists can do. So I think democratization of AI is extraordinarily important for society, for what you said, give people an idea of what’s possible, but also to allow the underdogs access to the tools. So you think of ChatGPT. Sure, my students can use it to complete their essays and cheat and plagiarize, and I’m probably not going to know and kind of a waste of their education.

BRUCE: That’s too bad. But I really worry about it being used as a persuasive tool not to write op eds, anybody can write op eds, but to mimic millions of people on the Internet, on Twitter, on Facebook, on other social media and comments columns and newspapers and act like regular people with a political bias. Now, I don’t care if one person does that, but by the millions, by the billions, that overwhelms political discourse.

BRUCE: You go online, you read what you think is a robust political debate, and it’s bots arguing with other bots. And unfortunately, the bots that win those arguments are going to be the better funded bots because they’ll be the better at it. So this is tied up a lot with our current inequality, and the fear is that it magnifies that. The other thing that you kind of touched on is the need for agility. You find a tax loophole, it takes about two years to patch it.

BRUCE: And that might be okay if you find one. But if we find 1000, that’s not going to fly. We need some way to patch our systems faster. Microsoft finds a bug, they patch it within weeks. Apple the same thing. There’s a bug in the tax code. It’s years if it’s ever patched, and that cycle needs to be shortened. We need more agility in these socioeconomic health systems in the same way we have them right now in our computer systems.

CHRIS: I completely agree with that. In your book, you spend a great deal of time discussing how law and policy can be hacked. How do you think AI and digital simulation technology will change hacking policy and legal code? And then what do you feel is the timeline that we could expect to see these changes occur?

BRUCE: It’s this idea that AI will find vulnerabilities timeline. I don’t know. If I was Goldman Sachs, I’d be working on it right now. And it’s not going to be that the AI finds a loophole and just hands it to you. It’ll be what we know works best, that it’s AIS and people working in tandem. So the AI finds some candidates. The people say, you know, that’s not great. Here’s a tweet, go back and think about it some more. And the AI does, and it goes back and forth. And eventually there’s something that the humans say, this is a good idea. Here’s how to make it a great idea. And then it’s released.

BRUCE: So I don’t know. Timeline. I think it’s sooner than we think. I think finding loopholes in the tax code or loopholes in financial regulations is not more than a couple of years off.

CHRIS: Interesting. And what are your thoughts around the integration of technology with policy? So you think of smart contracts, blockchain, law, bots, and how do you expect that to change things?

BRUCE: Oh God, I will hope they go away. Smart contracts are stupid. Blockchain is idiotic. None of that actually does anything useful. We already have contracts that trigger automatically, so I don’t think it changes anything. I mean, it could be the dumpster fire. It is, but hopefully the blooms off the rose, it’ll go away. We don’t have to worry about it ever again.

CHRIS: You provide a lot of great information in this book with a lot of takeaways, but I’m curious to know if there were any takeaways for you. You know, did anything either surprise you or was there anything that you learned from personally going through this process?

BRUCE: I was surprised how much hacking reinforces existing inequalities because we think of hackers as teenagers in hoodies, countercultural, the little guy fighting the man. And that’s not really the way it works, that most hacks of the tax code are by the rich. If I found a tax loophole, one, I can’t make much money from it, and two, the IRS is going to close it. Wealthy billionaire finds a tax code. One, they make more money, and two, they will lobby so that Congress Intrins it into law.

BRUCE: And then when I think about it further, I realize, you know, the NSA are actually the world’s best hackers, that in fact, hacking does magnify existing power. And I was surprised that I mean, in retrospect, I shouldn’t have been, but I really came into it with much more traditional countercultural mindset than I came out of it.

CHRIS: So the comment that you just made about the NSA being the world’s best hackers, can you just expand on that for me?

BRUCE: So we learned this from Snowden. We saw glimpses in the NSA’s hacking capabilities, and it turns out they have a lot of money, but they have budget, they have expertise, they can hire people. And it’s a career path. It’s not just a hobby. It’s not just something you do in your spare time. You are a hacker for the government as your job, and that’s your expertise. And you can have conferences on it, and you can get better at it, and you get rewarded for it, and it’s legal. You don’t go to jail for making a mistake, and you have the budget and demand of the US. Government behind you.

BRUCE: And that just makes you better at it than if you were some teenager in your room trying to hack a system because it’s fun, or even if you’re a career criminal.

CHRIS: For those that are looking to get into hacking, I mean, number one, obviously pick up this book, but any other resources that you can think of that maybe extend beyond the scope of what we typically hear as hacking from the technical angle?

BRUCE: I think I’m the one who took this hacking idea into the vernacular. But in the notion of loopholes is not new. A hack is a loophole. So there’s lots of people who write about loopholes and how they’re exploited in the computer field. I always recommend Ross Anderson’s book Security Engineering as a great resource for anybody in this field. Adam Schostek wrote a new book called Threats, which is very good and talks about some of this. And then his previous books on threat modeling are really useful. So that’s where I’d start me poke around.

BRUCE: I maintain a blog. I write every day, so there’s always a lot of stuff there. Everything I do is on Schneider.com and look around. I think there’s a lot more in common with the non computer world, which has different languages. So if you think about it in terms of loopholes, then I think you find a lot more in the political science and the legal literature and the policy literature about them and how they’re exploited and who exploits them and what the damages to trust in society are. And I think that’s really important that a lot of these things damage our trust, and we are less compliant when we don’t trust the system.

BRUCE: When you learn that somebody rich is getting away with paying almost no tax because of a loophole. You are more likely to cheat on your own taxes. We know that from studies. So when there are these systems that allow the powerful to get out of following the rules, we are less likely to follow the rules. And that’s just bad for society.

CHRIS: Yeah. And what I learned from your book is to think outside of the box. Think along the lines of really any system, like you said, not just a computer system. And I think that anyone, no matter what degree of security, education you have, where you are in your career professionally, I think anyone could look at this book and make that connection. And I think that was really something special and something that you rarely hear or get a perspective on from someone that’s been in the industry as long as you have.

BRUCE: Yeah. Thank you. I really like the examples. The most fun about that book are the examples from sports, from religion, from casino games. It was really fun to write because there are so many examples of hacking all through the world, all through history.

CHRIS: Oh, yeah, that section of the book was extremely fun to read through, and some of those stories took me down a rabbit hole.

BRUCE: It was just as much fun to write. So thank you.

CHRIS: Okay, so where can we get this book? Obviously it’s on Amazon, but tell us where we can get it and also where our listeners can continue to connect with you online.

BRUCE: Well, so the book is available wherever fine books are sold. You know that answer? Everybody has it online. I saw it in an airport last week, which is super exciting. Nice, right? If you think about airports, they’re not known for their wide book selection. So being one of the books at the Roanoke Airport made me very happy. Everything I do is on Schneider.com. So you see my blog, my essays, my other books.

BRUCE: That’s sort of where I am.

CHRIS: Got you.

BRUCE: I’m not a social media person. Makes me a freak, but highly productive.

CHRIS: Nah, it’s better you stay away from it. I wish I could do it.

BRUCE: That stuff will rot your brain.

CHRIS: It does rot your brain. Excellent, Bruce. Well, listen, like I said, I do have a copy of the book, and I encourage everyone else listening to go get a copy of it ASAP.

CHRIS: It’s definitely a must read. Bruce, thanks again. Always a pleasure to speak with you.

BRUCE: Thanks for having me. That was really good.

CHRIS: Take care.