I'd say Blue and Silver left the most impact. Before anyone screams NOSTALGIA!!111, it's more that they follow a different trend than the other rivals who are more on the friendly side rather than being a flat-out asshat (Blue) or a douche (Silver). Plus, Silver actually learns to be a kinder person as time goes on. He's still super serious, but he's no longer insulting you.
Otherwise, I like Hugh because music. The other rivals are laaaaaame.