The first people to walk away from your life are the ones that were mad because you didn't trust them enough to open up. 

Once you trust them, they promise you that they will never leave you, so you open up and they leave. End of story. Or shall I say, Game over.