It appears that there's a bug currently where if you try to join a friend's private game, it kicks you and says you've been banned from the game session. This doesn't happen if it's a custom game.

The fix Bloodline and I found is that you can start the game as a Custom Game, get your friends in, hit Escape to bring up the menu and click Hide Game (far right side of the screen). This converts your Custom Game to a Private Game, and everyone is still in. If you need to add more friends later, you can hit Escape and click Show Game, get them in, then hit Hide Game again so you don't have random puggies joining up.