The Jester doesn't need to visit anyone. He exercises his ability at his own risk. If there's a drawback as you describe Lobster, then it's his own fault.