You're punching with the wrong part of your hand.
I'm guessing it would hurt the one doing the punching.

Mark

I was thinking it might damage the phone as well.

Only if you're holding it wrong The 'phone is held in the palm of your hand, fingers through the holes, so no contact between the knuckles and the brass. In theory, at least...