For those of you who prefer reading, below are Chester’s answers to my questions. Please note that all of the views discussed in this material are personal opinions.
1. Do You Think ChatGPT Was Released Too Soon Given the Number of Security Concerns and Unanswered Questions?
I think it was too soon for certain things. Certainly, with regards to privacy, there's already been quite a few different issues where privacy things have come up that they weren't kind of fully baked and things leak from one thing to another where they ought not to and that sort of thing. There's so many different ways that security and generative AI link that there's some of them that it kind of has been fine and other areas where it's a little more sensitive. And I think, you know, privacy is one of the ones where it felt a little early. The other issue, I guess, is unleashing natural English language skills at the level that it has for people that don't have that skill. I don't know that the abuse was taken into consideration. In certain ways, it's amazing to be able to interact with someone who's not a native English speaker and effectively work as a translator. It does an amazing job of writing text in multiple languages which is great for people using it, not abusing it, but of course the abuse angle of it didn't seem to occur to them. And this idea that they can put some bounds around it to prevent it from saying certain things and then having to imagine all the different terrible things people are going to try to use it to say is just going to be a losing battle. The move fast and break things, as usual, breaks more things than it accomplishes. But the moving fast makes the people who make it rich and the breaking things falls on the rest of us and we have to deal with the brokenness that comes out of it, as usual. I would say a year and a half in from its big debut, GPT's been around for chat GPT even longer than most people realize. But most people became aware of it in December of 2022 or November of 2022 when it started getting a lot of press. To be fair, the abuse of it has been nominal, right? It hasn't had big impacts yet so the good news is it could have been worse.
2. Do You Think AI Security is the Next Big Skill Set Similar to How Cloud Security Became Mainstream?
Look, the hype around LLM is that it's going to change everything in the world. The truth is there's a dozen things it's going to make a really meaningful difference and it's incredibly useful for. And for the rest of the things, it's not going to do much at all. So there's absolutely going to be a demand and a need for people who understand how it works and know how to both manipulate it and contain it for the things that it's useful for. But it's not going to be all the things that, if you listen to the hype cycle, it's like you know this is going to change every part of our lives and I don't really personally subscribe to that. I think and this bears out in just our own experiments with using it and trying to make use of it. There are some incredibly useful things and you know one of the things that I just started talking about was it's mastery of language well it's also got mastery of language of programming languages and syntaxes of computer things that are really hard for humans. And so those types of applications are going to continue to be useful and there's going to be a lot of demand for security professionals to play a role in that. Because I love the idea that I'm terrible at writing SQL queries, but it probably can write me a SQL query because it doesn't know that SQL isn't French to it. It's just another language. So that's pretty awesome because even if it gets it wrong, just like if it gets the French a little bit wrong, I can probably fix it. But as a human being, it would take me a lot longer to figure out how to do it. Now, somebody's going to have to figure out how to not leak what I was trying to search for with my query, make sure that somebody else can't extract that from this engine and allow me to use it safely. And I think that obviously is where the security professionals are going to have a role, but I'm not sure it's going to be like the dominant security job 10 years from now that everybody's only going to be working on AI security. There's going to be narrow applications where it's incredibly important. There's clearly going to be places where privacy and security professionals have an incredibly important role to play.
3. What Are Your Thoughts on Apple Intelligence Compared to What We Have on the Market So Far?
Well, being that it hasn't shipped, it all sounds good. The question is, from when it was announced, it sounds like it'll be almost a year before we actually see certainly the cloud version of Apple intelligence. I'm not sure if... I guess to be clear for listeners, there's sort of two components to Apple's system. There's the on-device thing that will be in your iPhone itself on the iPhone 15s and higher, where some compute can be done on the phone itself and not be sent off to the cloud. So obviously that helps from a privacy perspective of third parties not having access to that data. And then there's the cloud aspect for the more complex tasks that can't be done on your phone that are going to get offloaded into Apple's cloud. The blueprint for it all sounds great, but there's a whole ton of questions that remain to be answered as to how it's actually going to work when it's deployed. They said they're going to run it all on, quote, Apple Silicon, and that's unclear to me whether that means the infrastructure is running on Apple's hardware, and they're still using, say, NVIDIA H100 GPUs like everyone else is using because of their amazing floating point compute power, which is what we use to calculate all the mathematics that we need to do for artificial intelligence LLMs. And if not, if it's actually Apple Silicon, like they're just using beefed up M3 chips that they've maxed out to be LLM supercomputer capable chips themselves. Then the question is, well, how much time do they have to test and harden all of that to determine that all the memory is being handled safely? And it just seems like there's a lot there. And the cost is the other issue that is unclear to me because they basically are promising to almost like spin up a VM, run your query, give you a response back to your device, and then destroy that VM entirely, not record any of the inputs or outputs. And they're kind of promising to have that fully audited and allow external entities to see how it's working. It sounds brilliant, but it also sounds incredibly expensive. When you consider how expensive it is to operate something like ChatGPT already, and then to decide that you're going to spin up unique instances and destroy them every single time somebody wants to do something, can you do that without charging me 99 99 a month? I'm not sure how that's gonna work. And if it's costly, then that's where you start to want to cut corners. And that's when things go wrong. So I don't mean that it doesn't exist. I can't say for sure. But those are my concerns. And what's on paper sounds brilliant. I certainly hope they can deliver something close to what they promised.
4. Are There Any AI Threats Which We're Seeing Right Now That Threat Actors Are Using to Target Organizations?
There are, but there's not much, which is the good news so far. Really, it's about being able to write really good phishes and to do so without the grammar and spelling mistakes that many non-English speaking criminals were prone to previously. Everybody laughs about phishing training at work, but what's the first thing they teach you? It's like, oh, if the email has commas in the wrong place or this is misspelled or this is in all caps or it doesn't look professional. These are your signs it's a phish, which is absurd. That hasn't really been true for a long time, even when humans were doing it. But what humans lack is the ability to scale. If I'm a Russian criminal and I have to hire the English expert to help me write my phishes, and it's a human writing those phishes, how many phishes in a day can they write? How many templates in the correct English with the correct logos for my bank or for whichever given thing? There's somewhat of a limit to that. Another way to think about that is if you look at social media abuse in the 2016 election in the United States, quite famously the Internet Research Agency, a Russian group run by Prigozhin, was trying to cause chaos in the U.S. election. It wasn't necessarily for Trump or for Hillary; it was just to cause chaos in the election on social media by spreading all kinds of rumors and mysteries, etc. The scale of that was how many people could sit in a room in St. Petersburg and create fake Twitter accounts and then send out English messages to impersonate Americans to try to create this chaos. It seemed to have some effect. But of course, it was a few hundred people in a room. It was very labor intensive and its scale was limited. Now we do see abuse of this through things like ChatGPT because I can write as many phishes in the day as I can automate. I can come up with a concept and stick all the concepts in a file and then call their API and just generate them all day. And every one of them will have correct English syntax and grammar. I can say use UK English, use Canada English, use Australia English, and it'll get that right too. So the S's and the Z's and the U's are all in the right places, which is very uncommon. In societies that are not accustomed to criminals targeting them because of the language barrier, they're also going to be more at risk. I've already started to notice this with things like Portuguese. Almost all Portuguese spam and phishing attacks historically is Brazilian Portuguese, not Portugal Portuguese. So when people in Portugal get those messages, they spot them instantly and they go, that's Brazilian Portuguese. That's clearly fake. Now, of course, ChatGPT knows the difference and it makes it easier for them to target people where they may not be as accustomed to scrutinizing their messaging to determine whether it's valid. There's some risks there. We've seen this also occurring in text message abuse and WhatsApp message abuse for romance scams and cryptocurrency scams that used to be human operated by often people that were trafficked in Myanmar and other places. Unfortunately, there's a lot of layers of crime here. Now some of that is being automated. Clearly, if you're smart, when you get some of these messages and you start trying to trick it, you can trick it into telling you it's an LLM.
5. Do You Think We'll See a Trend in AI Being a Common Attack Vector for Attacks Like Remote Code Execution?
At the moment, what I'm most worried about is people using it for software development and booby-trapped libraries getting used by these LLMs by being tricked. It sounds a little far-fetched, but when you think about something like GitHub Copilot that's helping you write code, and this has already happened a little bit by accident, and we're just waiting for it to happen on purpose is kind of what I'm getting at. If your real library out there is called OpenSSL, and the criminal creates one called OpenSSSL and uploads it to the repository. If it's in the repository, an LLM doesn't know the difference between the real one and the fake one necessarily. So then if I start seeding some blogs that I know the LLM is reading and using in its training material with the one with three S's instead of two S's, it might just start telling people to use that one that's backdoored. And a coder who's not being careful might miss back to typos and incorrect things that you're looking for in a phish. We're going to have to start thinking about this if we're letting machines help us write our code. Every library that's being called in Python, in Ruby, in Node.js, whatever it is you're using to make your code... If you're letting something else write part of that code, you're going to need to carefully scrutinize that there's not misplaced periods or dashes or extra letters where they don't belong. That might be hard for a human mind to see because when we start seeing two S's together and three S's together, our brain doesn't really differentiate. And if they start doing that kind of thing, I think that's where one of the big risks is a lot more. We've already seen a lot of poisoned libraries in most public open source repositories before LLMs. Now this just is a new vector for a way to get more victims to be tricked into taking those trojanized code snippets instead of hoping on human error. Before, criminals were relying on you typoing three S's, and that doesn't happen very often. But now there's literally an active way of tricking Gemini or ChatGPT through manipulating public sources into thinking those are real. So I think that's a real risk that I would be very careful of in a tech company if my coders are using it. It's no different than if I start seeding Stack Exchange with those same sort of bogus things that people might copy and paste, but it can be done at scale.
Share this post