Although this is true, seeing someone walk up to your door and open with a key, less people will notice and say something. I think when they walk up carrying a crowbar someone is likely to notice and say something and get a description. Insurance is going to cover the door but when the criminal claims you gave him the key the insurance may not, and they will claim you gave it to them.
Otherwise why bother to lock the doors at all.
Think about things before you do them make life easier not harder.