You need a convincing reason why Tom wouldn't notify the ministry, which would eventually bring in Dumbledore and Harry would be returned 'for his own good'.
Maybe Tom does notify some magical equivalent of Child Protective Services, which puts him into magical foster care. There would be no strong reason to notify Dumbledore of such a thing, no matter his positions, since this would be a standard thing to do.. Dumbledore doesn't notice until the blood protections fail, and then it's a done deal.
Author's response
you probably never got a chance to read the first draft I made of this story, but I put in that when Tom found Harry he had a Healer look at him and discovered he was suffering from massive malnutrition and was sickeningly underweight. Besides, a bit of background for Tom I intend to work in at a later date is that he himself was part of the magical equivalent of Child Protection Services and thus his decision to look after Harry himself. Harry will be adopted by another family some time soon and you will be surprised by who it is...