"When we are in the supermarket and" joke

When we are in the supermarket and someone rams our ankle with a shopping cart then apologizes for doing so, why do we say' Its all right'? It isn't all right, so why don't we say,' That hurt, you stupid idiot'?