Then there's also the fact that most Wizards in the HP series are nowhere near the peak of physical capability. In fact, if anything, they are probably far below the average even when compared to the real world.
Even not having read The Destroyer series I'm pretty sure that Harry would be nowhere near the peak of physical prowess. He seemed more like a professional athlete amongst a crowd of skinny (maybe rich) boys. Not a physical (demi-)god amongst a crowd of skinny boys.