Another big thing is that recruiting students will have its own requirements. They showed Lorenz and Byleth needed to be good in certain stats for him to come over.
Given the time limit of the school year, the player will unlikely be able to cover all the skills to recruit every member outside of their chosen house. Plus, as some noted, some characters like Dedue would likely be too committed to their lords to change.