There are work arounds to every scenario you've pointed out by being a little creative. But at the end of the day HB/Swinger players aren't obligated to pick a target N1 (unlike Lover.) If the role isn't something you want to play, blacklist it or don't pick a target N1.