Well, this is interesting . . .
https://www.vice.com/en/article/4a3...e-kills-human-operator-in-usaf-simulated-test
https://www.vice.com/en/article/4a3...e-kills-human-operator-in-usaf-simulated-test
AI drone "kills" operator who told it not to do something.
- Thread starter ThunderCat98
- Start date
It didn’t actually happen, but was identified as a possibility. This is a problem for unbounded AI.
Forget about if AI is actual intelligence or whatever for a minute because it’s not relevant to this point. We can (we have!) make an AI that can develop new designs for things, test them, iterate on them, and come up with something a human would never come up with. Thing is, you have to be very specific on what you’re asking the AI to make and what limits there are. It doesn’t have any “common sense” assumptions.
So if an AI is programmed to maximize killing enemy troops, you’ve got to put in some bounds. Because the “best” way is probably to throw our entire nuclear arsenal at them. We know that’s a bad idea; an AI won’t unless it’s programmed to know why not to do that.
For another example, let’s say we have an AI piloting a fighter jet. You have to be very careful when programming in what its priorities are. If you tell it to take out enemy jets, it may “decide” a kamikaze attack is the most sure way to do that. Ok, we don’t want to trade 1:1 so we tell it not to do that. Now there’s a larger jet battle, and the AI calculates that the debris field created by shooting one of our own manned jets will take out 3 enemy jets tailing it. If we never thought to tell it not to do that, or if we didn’t put in the right bounds to stop that sort of behavior, then it might do that. And it’s hard to come up with those bounds.
Forget about if AI is actual intelligence or whatever for a minute because it’s not relevant to this point. We can (we have!) make an AI that can develop new designs for things, test them, iterate on them, and come up with something a human would never come up with. Thing is, you have to be very specific on what you’re asking the AI to make and what limits there are. It doesn’t have any “common sense” assumptions.
So if an AI is programmed to maximize killing enemy troops, you’ve got to put in some bounds. Because the “best” way is probably to throw our entire nuclear arsenal at them. We know that’s a bad idea; an AI won’t unless it’s programmed to know why not to do that.
For another example, let’s say we have an AI piloting a fighter jet. You have to be very careful when programming in what its priorities are. If you tell it to take out enemy jets, it may “decide” a kamikaze attack is the most sure way to do that. Ok, we don’t want to trade 1:1 so we tell it not to do that. Now there’s a larger jet battle, and the AI calculates that the debris field created by shooting one of our own manned jets will take out 3 enemy jets tailing it. If we never thought to tell it not to do that, or if we didn’t put in the right bounds to stop that sort of behavior, then it might do that. And it’s hard to come up with those bounds.
That was a different thing though. Those robots had sentience and ended up wanting things nobody would’ve programmed them for. This would be the robots doing exactly what we asked but not how we expected.
I have an iRobot. It chased the dog around the house, and I ended up with a clean carpet.
