Is it awkward because of them or others? Is there obvious and noticeable tension between them (they argue, talk about each other to guests, etc) to the point where the mood of the party is ruined? Or are other people just uncomfortable due to your in-laws' relationship?

If they are creating visible tension, it would be acceptable to sit them down individually and explain to them the problem. Ask them to put aside any differences for the sake of the host and guests who are trying to enjoy the party. Let them know that if either of them causes a problem, then they will be asked to leave. They need to understand that being invited to an event is more of a priviledge than a right (family or not) and they should be more respectful. 

However, if they come to the party, are respectful and don't mind each other and people are just uneasy due to their situation, it may be hard or unfair to not invite both or one of them.

I have some people in my family that do not get along and they are given a fair warning before being invited as to who else is coming and to keep it fun (no arguing, screaming, cursing, drama) or leave... if they decide to attend a party. Usually they understand and if they attend, they keep their distance from the person they don't get along with.   

If you really cannot invite both, it is best to give them a fair explanation and see if they are willing to take turns attending events. Maybe just giving them the truth will open their eyes and encourage them to manage their emotions when together.

Tell both of them separately the truth, that they can't both be together at the same party. Let them know that only one of them at a time will be invited to a party, and that you'll trade off invites between the two of them.