Some interesting potentials here, but it does need some tweaking to be more HP-centric. Some of the suggestions made above would work, as would, say, Harry being thrown out of the house and chased by Dudley and his 'associates'. Either that or it needs to be shown that the wards Dumb les put up were nowhere near as grand and powerful as he claimed.
Author's response
don't be impatient mate. its all explained eventually.