Being outnumbered is not good

[Rias]

If you are under attack by multiple enemies at the same time, your defenses will degrade. So be careful when fighting against groups of foes - if you're all alone with nobody else around to take some of the pressure off you, your defenses will diminish!

On the positive side, getting a group of your own together to gang up on the bad guys will be effective in your favor.