Task your bot, to make a, cup of tea, just for me
Its utility function is designed simply, as you see
Completing the task would get it maximum reward, although
Any other outcome, would result in zero
Then your baby crawls right in its way, what to do?
It tramples your kid, then makes you, a nice brew
So your next AGI is designed, to cause less stress
With a big stop button, on its chest, you can press
So the next time you ask for a drink, being wise
You remember it can’t think like you, although it tries
Then your wife gets in its way, so you act fast
To hit the stop button, but the bot kicks your ass
When you return from the emergency room, feeling cursed
The funeral went as well as expected, but you still have a thirst
So you give it an equal reward if the button is applied, or not
No drink, in a blink, it just turns itself off
So you move the button out of its reach, so only you
Or other humans, can press it, when they need to
The bot has to choose if it’s easier, to make the damn tea
Or to manipulate you, to turn it off instantly
So it smashes your mum in the head, till she’s dead
Cos, it’s easier to get you to hit the button instead
Than filling the kettle, and finding a cup
Pouring the milk, and then clearing up
It knows that the button is there, and that you care
So you change its utility function, so the bot ain’t aware
That the button is important, or to what it relates
But this fact isn’t passed on, to sub-agents it creates
With no cash for more funerals, for the dead
You turn the bot off, bury mum in the shed
You curse AGI, when the policeman knocks
But still haven’t solved the stop button paradox