I have an idea for you. Just re-read the stories that have been downrated, and if it matches what it has been downrated on, then that's probably what it is. If it's not then it's probably just a/some trolls.
Author's response
I will do. I just looked back Someone has rated them so they go back to zero o.O
Hopefully they won't come back :')