Whoever turns away from us might not offend us in doing so...

Whoever turns away from us might not offend us in doing so perhaps, but he certainly offends our followers.