Exactly, however i have not gone out and done it but i have say been 15 mins grabbing food or whatever and just finished a long game and came upstairs to 15+ downs so i would be pissed.
If you have a whole game inactive it should kick you out to main menu.
Or just make it so a 4 player game needs 4 players to ready up to start