Thank you for pointing that out! You are right, of course. And I indeed have never taken him seriously in the movie (just thought how good an actor the actor is), and in the books the Dursley-chapters were always ones to read pretty fast to get to the action / my beloved characters...
You are especially right about that there is nothing one could come up with to defend him...