It's a bit like a burglar saying because you'd forgotten to lock your door, you'd effectively given him permission to be in your house.
That just doesn't wash with me. No permission was given and the intent was there on his part.

If a "burglar" enters your house with no intention of stealing anything or harming anyone, then it's not burglary - it's simple trespass - a civil tort, not a criminal offence.