Mmmm, maybe parents should interfer here, I agree. Maybe they could deliver 'your message' as theirs.
If this doesn't works, then you must talk with him, trying to convince him that he's freqquently arriving at moments your family is not comfortable at.
The thing is, it will be a bit awkward, so it can prove to be difficult to handle.