Seems like a good story. As for the Harem thing it really depends on what you want to do. If your going for smut with little to no plot then your idea's of girls for the Harem/throw away is fine but if you going for a real story you might want to rethink it. Such as Ginny and Lavander would offer very little in a real story to be in the Harem while Hermione as a vast supply of information that can be utilized.
In the end it's your choice but as of right now you got those groups set up for a plotless smut in my mind.