Somebody's probably done it before but here goes
1. Get to Oliver
2. Notice that Louis is now in your party and get suicidal tendencies
3. realise you can't kill yourself
4. get killed by Oliver fast enough so Louis can't revive you
5. activate mistle and disable Louis from your party
6. come back to Oliver
and congrats, you're about to fight him alone.
If you want to take no damage then to make it easy, get queenslayer thorn, study it's drain attack range and bait out three overhead slam attack in second phase, in the first one just stay close to his ass and slap it after he performs an attack(lost broadsword recommended)
he has no AOE attacks and he's slow so it should be very easy.