Re: Is there a setting to change for this?

I don't think there are any settings you can change to prevent this. If this happens often, you could try joining a public lobby and then inviting your friends from there. That way you could play together without being in a party.