Task your bot, to make a, cup of tea, just for me
Its utility function is designed simply, as you see
Completing the task would get it maximum reward, although
Any other outcome… would result in zero
Then your baby crawls right in its way. What to do?
It tramples your kid, then makes you a nice brew
So your next AGI is designed, to cause less stress
With a big stop button, on its chest, you can press
So the next time you ask for a drink. Being wise
You remember it can’t think like you, although it tries
Then your wife gets in its way, so you act fast…
To hit the stop button, but the bot kicks your ass
When you return from the emergency room, feeling cursed
The funeral went as well as expected, but you still have a thirst
So you give it an equal reward, if the button is applied, or not
No drink, in a blink, it just turns itself off. Suicide of the bot
So you move the button out of its reach. So only you…
And other humans, can press it, when they need to
The bot has to choose if it’s easier, to make the damn tea…
Or to, manipulate you, to turn it off instantly
So it smashes your mum in the head. Till she’s dead
Cos, it’s easier to get you to hit the button instead…
Of filling the kettle, and finding a cup
Pouring the milk, and then clearing up
It knows the button is there, and that you care
So you change its utility function, so the bot ain’t aware…
That the button is important, or to what it relates
But this fact isn’t passed on, to subagents, it creates
With no cash for more funerals, for the dead
You turn the bot off, bury Mum in the shed
You curse AGI, when the policeman knocks
But still can’t resolve the stop button paradox