Who says they can't? Nothing says you can't go on whatever team you want. Purple have more than enough. They can at least let go of a few. And at least you.

[Because lets face it, they would have a fight on their hands if they tried to take Prompto away from his side.]