Whilst it is frustrating I imagine the reasons people quit are quite legit and not just a simply hissy fit or attempt to save their K/D like in Multiplayer.
It's most probable that people leave either because people aren't playing the game properly (going for challenges I expect is the main cause) or they have commitments/interruptions.
Always exceptions to the rule but I don't think it's worth getting too bothered by.