I always found it a sign of a weak author or lousy planning to have to rely on the 'it's a book it doesn't have to make sense' excuse. A story should make sense at least by it's own rules many Sci-fi or fantasy series have a psuedo science to explain what they do and how they do it, it may not be realistic but it makes sense in that own universe. Ignoring that was one the biggest flaws with the Dursley's.
JKR put a lot of effort into establishing Harry's home life being pretty lousy, but the why or how it could be that way was rather empty. Dumbledore merely knowing about how badly Harry was treated would not explain how they got away with it, why did no one ever notice Harry's poor health or clothes in comparision to dudley? Especialy in a little closed comunity that seemed to thrive on being noisy and gossip? The nurse at hogwarts also seemed to never notice Harry's poor health or eating habbits either. In my mind that means Dumbledore or some one else had to take an active hand in covering up evidence and ignoring complaints for them, just like Dumbles did for Snape.
Not to mention acidental magic, canon says Harry teleported, super jumped or something to the school roof just being slightly scared from Dudley and his gang, why didn't he ever reach that level of fear for magic to work from Death eaters or Dementors? Dudley pushing harry down at the Zoo got him angry enough to 'punish' Dudley with some magic, yet we lead to believe Harry been abused for years? Either his magic would been going off all the time or it would been beaten out of him to even think of rebelling at such a small slight. Aunt Marge insulding his mother wasn't much better, puts off the whole 'age or wand' theory into acidental magic stopping once reach hogwarts. So why does he over react yet again in such a magical way at a few bad words when he's lived his whole life hearing how stupid his parents were from the Dursleys, Malfoy, Snape etc?
Like i siad before the Harry Potter series are more like a selection of books, each written by a different author then a continous series by the same person, unless they have alltimers or something.