You don't

I usually hate this type of answer but unfortunately this sort of thing is rarely done well at tables and should be avoided. Particularly in a first session this will be considered railroading.

No matter how strong the initimidation is the players will still think they have a chance to win and are extremely unlikely to give in. This results in three options:

1. TPK, the players will refuse to do what is asked and would rather fight to the death.


2. Your players micraculously win and destroy your carefully planned campaign.


3. The PCs run away. This might be a welcome outcome for you but it is still unlikely they will do what you want them too.

... the villain will force the party to do his bidding ...

This is almost a definition for railroading. You shouldn't be forcing your party to do anything. You can present the villian as evil and describe his actions but you should never dictate how your PCs respond to it. For experience it is extremely unlikely this scenario will go the way you want without railroading.

Some Alternatives

Straight up forcing your players to do something through intimidation is a bad idea. But there are other ways that you could achieve a similar result.

• 
  Blackmail them. Hold a town to ransom, capture the PCs little sister, something suitably evil and compel the PCs that way. This may still not get the result you want but will portray the villain as suitably evil.


• 
  Trick them. Have the villain pose as the quest giver, give them vague or inaccurate reasons for what he wants done. Maybe the people they are sent to kill "killed my son", which is of course a lie. A villain lying to the party is totally ok and something I have done in the past. This is the most likely approach to get the players to commit terrible acts, make them think they weren't terrible while they were doing them. Warning; some players may not takes this well. You will need to judge based on your own group of players.


• 
  Run it as a cutscene. Before the game starts you describe the situation, the players are in the midst of the preparation to commit this terrible act. Describe how the powerful enemy inspired terror in them and they choose to do his bidding. Most importantly they regain control of their character before they commit the act. They can then choose to not do it if they want to. Never force a PC to do something they don't want to, if you do you are denying them their agency.


• 
  Have it part of the campaign setting. Estabilish the dominance of the villain as part of your campaign setting, the PCs are already living under the thumb of the bad guy. Have it part of the characters backstory how they came to be under his control. _{Thanks to Nitsua60 in chat for the idea}
  

I'd also suggest you watch this video from Matt Coville on Running the Bad Guys. He introduces a villain called Karalel the Vile and its a great example of how to introduce a powerful and evil enemy. You could also check out the episode of dice camera action where Chris Perkins first introduces Strahd (I'm not sure which).

Session 0 and Campaign Expectation

It is possible that all the advice I have given is totally inaccurate, your players may enjoy this kind of play. If so they wouldn't be like any of the players I have ever met but that doesn't mean it is wrong.

If this is something you believe your players will be onboard with you need to set it up in the campaign expectation during session 0. Explicitly explain that you would like to be able to force the characters to do thing through intimidation. If they aren't onboard then you shouldn't do it. I expect this will be the case, if I'm wrong however, I wish you luck.

“I’d rather die!”

Player’s choose their character’s response to situations and the above is a perfectly sensible choice. Your only sensible response is the kill the PCs, complement the players on their moral integrity and start planning your new campaign. Are you cool with that?

I’m thinking of Darth Vader and I cannot think of any time the heroes voluntarily helped him. I remember Leia being tortured, I remember Han futility firing his pistol at Vader until forcefully (ha) disarmed and Luke choosing to plunge to his death (as he thought) because, you know, heroes.

It’s not the mechanics of D&D that work against you here - it’s the expectations.

“Sign me up!”

There’s the other type of player who when tempted by the dark side will grab it with both hands. You know, like Darth Vader.

If they go this way then you need to be prepared to find out just how depraved your friends can be which can be confronting for everyone.

Not railroading means letting the story go where the player’s want

Which might not be where you want or expect. Or are even comfortable with.

Don’t use mechanics to communicate

As I used to say to my children: “Use your words.” Talk to the player’s about the themes you want to explore and how you want to explore them. Listen to what they want.

Let the player’s make the running and have the villain enjoy it

Let the villain make it clear that every achievement the player’s have made has only served the villain’s agenda.

They saved the town? That allowed the villain to capture the princess. They saved the princess? That allowed the villain to steal the Sword of Ultimate Awesomeness. They recovered the sword? That allowed the villain to unleash the etc. etc.

Fish with a worm and a hook rather than with a stick of dynamite

Is there a way for me to accomplish this without railroading them into
the choice?

No. What you described is a railroad. If your players are OK with a railroad game, then press on. If not, stop now and change your approach. Put the hard choices in front of them and see how they respond. Then riff off of their response.

Or is 5e the wrong system to do this?

A lot of systems are the wrong system for this.

You can try to influence them to go in a particular direction, but if they resist, you need to be flexible enough to see where those decisions by the players takes the game. You will most likely find some great new things happening based on how they try to get out of this mess / difficult choice.

(This approach served me well in numerous games and in numerous campaigns as a DM).

Let their efforts to evade or overcome this compulsion become the new "plot" such as it is. And rather than compulsion, you can set it up that the players owe the villain a favor; when they discover what the payback is, their ingenuity will be challenged in how to get out of or around an odious assignment.

Fun all around.

Is the plot or the story more important to you, as the DM?

You should answer that question before the next session. The story that comes from your game, your campaign, is what the players did, and what decisions they made, when confronted with hard choices, dangers, challenges, etc, and also the consequences of their decisions.

Have you and your players discussed the chance that this campaign may end in grand and glorious failure? In other words, is the story about what they did, even if in the end their greatest efforts weren't enough? (Feanor's story was like that).

That's a story too: defying the BBEG. Some of the most memorable D&D campaigns I was in had catastrophe as the end play. The trick to making this successful and fun is in making this about the choices the players make, and maybe a few unlucky rolls of the dice, rather than about your domination of their characters. You own all of the cards in the deck: what's the point in dealing them a losing hand?

Mike Mearls has this to say about how fickle the dice(Fate) can be:

the dice will be cruel to you, but you will soldier on (PHB, p. 4)

I want to intimidate them into submission, simply by showing how great the difference in strength between them and this villain, but not kill them.

Multiple options for this one:

• Let the villain use magic, either himself or one of his minions. There are enough spells which can incapacitate or otherwise hinder the characters without the risk of killing them. This will show the power difference without killing the characters, but it is not very intimidating though. The villain may also use magic to force characters to do his bidding, for example Geas, but this definitely counts as railroading.




• Give the players a NPC ally which the villain can slaughter to show his strength. This character can also be used to explain the players to submit to the villain, so they can strike once they are powerful enough to beat the villain.




• Let the villain save the players from an impossible encounter. First let the players struggle around until they are on their last legs, and then let the villain jump in to single handily beat the encounter. This shows the strength of the villain, as well as establishing the villain as the dominant party as the party owes the villain for saving them, making it easier to submit the characters.




• Hint that having the characters doing the villains bidding may be preferable to the alternative. The village is getting massacred one way or another, but the minions of the villain plan to put every inhabitant to the sword. If the characters do it instead, the death toll can be kept to a minimum. This incentives the characters to submit as they still retain some autonomy and are still able to make some moral decisions.

Tell them they tried that already

For example, you could say something like:

"Okay, so as the adventure begins, you're lying on the ground, waking up from unconsciousness. You have one hit point. Oh geez, what happened? You gradually remember: there was this big guy in black spiky armor, and he wanted you do to something, and you decided to fight him instead.

"The guy in the black spiky armor is standing over there. He says: 'I was careful not to kill any of you, that time. I won't be so generous again. Now then: do you want to do this quest? Or would you actually rather die?"

Use a magic curse effect

D&D does not naturally contain death-curse spells (the nearest in 5e is geas), but you can houserule one in. The villain captures the main characters and puts a curse on them, such that the consequences for disobedience get gradually worse until they have to obey or die. This question: I need a "Suicide Squad" control for an adventure party has some more suggestions along these lines.

The accepted practice of a fictional villain aiming to terrify rather than kill (probably a real villain, too) is to fire something very deadly over the party's heads, or in some other way flashily miss them by inches, perhaps vaporising a nearby statue or, if you really want the party to hate them, a nearby minion. This could go on for quite a while if you really want to make it a full-length fight. If they want the party to do something for them, they WILL be deliberately aiming to miss.

See also the beginning of "The Colour of Magic", and any other Discworld novel in which Lord Vetinari of Ankh-Morpork wants somebody to do something they don't care for. A favourite negotiating tactic of his is to negotiate with the other party standing on a fifth-floor windowsill.

But as other people have said, that won't get you all the way. You can trust unwilling henchmen only while you're standing over them with your Staff of Fireball Everything. So even though you technically can, making it just a straight fight won't get you all the way. Even making it a straight fight ending in them being overpowered, knocked out and put on the fifth-floor windowsill might not work, unless it's with the connivance of your players; there's always the possibility that they might win by some freak accident, and they're going to be annoyed that you deliberately set a fight that was meant to be unwinnable whatever they did. So you have to find other ways that bad guys force people to do things.

I can't help thinking of this (from Girl Genius, a merry tale of evil scientists in which this kind of thing happens all the time): http://www.girlgeniusonline.com/comic.php?date=20100329

"So if the Baron's son if such a villain, shouldn't we be taking him out, rather than rescuing him?"

"Ordinarily, yes, but there are complicated circumstances. I have a sort of agreement with the Baron - "

"Oh, I see. You've got one of those 'splodey collars too, huh?"

Actually, a D&D villain probably could do that. I seem to remember that an Artificer can put a single-use spell into an item rather than casting it there and then, and that there's a spell called "condition" that will make a spell go off only when a certain thing happens, like, for instance, if you're not back within three days. A creative villain, or a villain with a creative mage in his basement, could guarantee to kill the party in any number of amusing ways.

Finally, though, I'd like to agree heartily with everyone who's suggested telling the players that the villain forcing them to cooperate is the basis of your plot. My experience of role-playing games is only from forum ones, but in those, you'd be surprised how often players are very happy to connive with the GM to bring about a fun scenario, or how annoying they often find it when they suspect the GM has something particular in mind but they don't know what and can only thrash about until they hit on something that will advance the plot!

Not easily possible

I'd outright echo the "don't do it" sentiment other answers put up. But I would also say you can get the same effect but with an alternative means. Let's examine the situation at first

1. You want the players to face the villain


2. The villain prevails against them


3. The player characters join forces with the villain in light of this (even reluctantly)

The biggest problem is the first thing. As a general rule of thumb in any system anything with stats is beatable. As a corollary facing such a thing means it should be possible to beat it - maybe it's extremely unlikely (facing level 30 enemy with level 1 heroes) but you should just avoid the situation.

Instead, just skip 1. and maybe re-work the whole premise.

Let the villain win before the game even started

If the villain is so powerful, he doesn't need to prove it. He has already won. Think of it this way - would the king personally go fight some upstart peasants that think he sucks? A king wouldn't even bother, if he even notices them at all. In a fantasy setting the king would have really formidable stats while the peasants would be starting adventurers so the difference can really be level 30 to level 1. In addition the king would be controlling large forces.

My suggestion is to put the villain in control of something. Could be a village, could be several cities, could be a guild, could be some other association. I'd go with town but the concept can be easily swapped and scaled depending on needs. The villain has some Goal he wants to achieve and delegates pats of the plan to the town. Say, he needs the fresh blood of intelligent creatures or whatever. Something grisly but not completely immoral at first. The town would then contract the characters to s̶l̶a̶u̶g̶h̶t̶e̶r̶ "clear out" a den of goblins. Once the characters have proven themselves, they would get more tasks that ultimately serve the villain's plan. Further tasks can be more evil, so as to not start in the deep black, morality-wise. As the characters progress, they would get more power (levels) and maybe even intelligence on the villain until they are ready to face him.

This can be scaled as needed - the villain might control the town from the shadows, perhaps because he's holding the mayor's child hostage to make him do his bidding. Or the villain might control some organization that contracts the players. In this case, the villain would be low key and more mysterious. Or the villain could be an evil king or wizard - a visible figure who controls the town by threat of force. The villain's relation with the player characters can also be adjusted as needed - maybe the characters know that the villain is behind this and they are working for him by proxy but this is the only way to stop him. Or perhaps they don't know about the villain at all but learn about the nefarious schemes through the course of their journeys.

At any rate, the most important thing this provides is a justification to why the players work for the villain and don't just face him. A secretive villain likely wouldn't be anywhere to be found and the players need to work for him to even get a crack at him. A visible villain is likely too powerful to just face - the king can just dispatch the army and wipe the town, a wizard can flick his wrist and incinerate the entire population. In either case they heroes can't just "show up" either but may need to work up to the privilege to be let in the court/wizard tower and/or to rouse up the population for a rebellion.

If you need a show of force from the villain, just have him destroy a village in a "cutscene". The players might be there - this might even be their village and this is the opening scene. Or it could be just news they hear of a nearby town just gone overnight. They can be told explicitly who it was that did this act of destruction or they could work to find out, depending on if you want the villain to be mysterious or not but in either case there is no need for a showdown. Going with the Darth Vader analogy, this would be like him ordering the destruction of Alderaan. Darth Vader just did it to prove his power - not his own fighting prowess but the resources he commands.

The party can still be a credible threat to a single high-level character

You're right to be wary of the players attacking your villain 5 against 1, because they actually have a decent chance of winning. As an example from a campaign I play in, our party of 5 adventurers at 4th level recently ambushed and killed an enemy wizard who was at least 15th level (capable of casting 8th level spells) but who I suspect was actually 20th level.

Hence, you can't expect your players to just give up, because that runs contrary to how the game is expected to work. The expectation is that you will throw challenges at the players that they are capable of overcoming, and there really isn't a good way for you to signal that you are breaking that rule temporarily for the sake of exposition, short of just telling them out of character.

Use an illusory double to monologue at the players

Instead, as others have suggested, have your players arrive on the scene (or wake up from being knocked out) to find that they're too late and the carnage has already happened. Then have the villain appear right in front of them and start delivering whatever monologue you have prepared. If the players try to attack the villain (which they will almost certainly do immediately), they find that it is merely an illusion, created by a spell like mislead (5th level) or project image (7th level). You can then finish your monologue, perhaps adding something about how if the players fell for such an obvious trick, they have no hope of defeating the villain.

Tell the players that they recognize signs of high-level spells

As a bonus, you can tell the player with the best Arcana skill that they recognize that the villain's illusion must be quite a high-level spell if it allows the caster to see and speak through it. In addition, you can describe the surrounding carnage in ways that imply that specific high-level spells were used: a pile of dust where a person was obviously standing might imply a disintegrate spell, an entire crowd of people withered to dried husks indicating circle of death, multiple charred bodies in multiple rooms implying the use of multiple fireball spells, etc. You can be as subtle or as obvious with your hints as you need to be to drive home the fact that the villain is a high-level spellcaster, way out of their league. You could even have the villain's illusory double walk over and say something like "Oh yeah, here's that guy I disintegrated. Good times."

For a non-caster villain, use overwhelming numbers

Obviously the entire answer up to now assumes that your villain is a high-level spellcaster, which might not be true. If you've got a non-caster villain, they are unlikely to be able to go 1 against 5 reliably, so they will likely to employ another strategy: lots and lots of troops. While your players might be tempted to try their luck against a single enemy no matter how high level, they probably won't like their odds against a squad of several dozen armored soldiers surrounding them with weapons drawn. And if they do decide to fight, the villain can order the soldiers to use non-lethal melee attacks to subdue the players and start again. Even if the players try their luck once, they probably won't try again after waking up an hour later with 1 hit point each.

Be prepared to make good on your threats

Whatever you do, if your villain threatens to wipe out an entire village, then you'd better be prepared to actually do it if the players refuse to do their bidding. Importantly, you need to consider whether wiping an entire village off the map is something that fits the intended tone of the campaign. If it doesn't match the tone that you're going for or the tone your players are expecting, then you can't have your villain make that threat credibly, and you may have to scrap this entire idea.

The Villain need leverage to make characters to obey him.

"I beat you unconscious, and I will beat you to death if you do not follow my orders" is kind of week as leverage. It works only as long as the characters have to accompany the Villain and he is not distracted. Otherwise the characters need to only get out of his reach in order to defy him and then hope that he has more pressing matters than pursuing them personally.

You have to come up with better leverage if you want to characters commit atrocities in service of the villain.

*The Villain may threaten something that the characters hold dear and can not defend from him. They can escape but they can not take their community with them.

* The Villain may make them accomplices by performing not-so-evil acts in his service, and then get them to play along to cover up the past crimes. The characters often have "flexible" morals and while mas murder is no-no, stealing, burglary and casual violence may be OK for neutral and "good" characters if the price or incentive is right.

As for mechanical side - it is not that important.

First - if the Villain hurting characters physically he should have an option to knock them unconscious with the final blow - so he can have overwhelming stats and still avoid killing the party.

Second - a serious Villain should have minions. Numbers alone should be enough to persuade characters that they not have any interest in starting a fight.

Have the players come up with their own reasons at character creation

You are the GM. PCs cannot appear in your game without your permission. You will find all kinds of advice for dealing with broken mechanics and/or troublesome spells to the effect of "don't let players play characters built with $BROKEN_THING".

The same thing applies to character backstories. I've been in a game where a questionnaire was distributed asking for background details. You can have your players do the heavy lifting for you. There are two variations here.

1. Straight up tell them "all PCs in this campaign are under this guy's thumb at the start. The reason why is up to you, but it needs to be something terrible enough that your character will go along with almost anything."




2. Give vague hints about threats to come, and then have them fill in the same background information. This will still let players know something is coming, but not exactly what.

The players will be invested in their PC's reasons for feeling like they have no choice but to go along, because they came up with the reasons themselves.

Be prepared for some players to bow out

The issue underlying the other answers is that not everyone is going to be comfortable being evil. The hardest failed save to recover from is a failed Will save versus the desire to stop playing. This is simply a topic that you should expect might result in pushback due to extreme content.

This is the biggest reason you should be up front with what kind of campaign it's going to be. You called it the 'new' campaign; have you RPed with this group before? Do you want to continue doing so? If your idea for a new campaign is going to make some folks so unwilling or uncomfortable that they say 'no thanks', you should find that out before launching into something that - if it goes badly - could generate enough ill will to detonate your group.

Citing the ever-smart, endlessly clever and infinitely attractive AngryGM:

If you determine the action is not possible (...), THE ACTION FAILS

If you, as the GM, decide that the villain can't be defeated, the players shouldn't even be able to fight him properly. This should not be a combat encounter if there should be no possibility for them to win. Like other answers stated, this is indeed a little railroading, but it's not hard to pull off:

PC: I want to attack the villain.

GM: You swing your sword at him, but he flicks it away easily, countering with a hit himself. You realize there is no way for you to best him. Take X damage.

Something along the lines of this will probably tell your players that they are not supposed to defy him and that they need to get stronger to face him. And by doing that you don't need to show them any of his stats or take the risk of accidentally killing them off.

Even if the players say something along the line of "I'd rather die", the villain doesn't want them to die, he wants them to work for him, so he won't kill them. Just describe how he beat them to a pulp and tells them straight to the face that they got no other choice. This would also make him easy to resent for the players so they start planning their revenge with more hate against the guy.

This is not a general answer, but a specific one. After reading the question, I started thinking about how I would go about doing this if I really felt it was necessary, and I came up with a way to allow the players a choice, but also make it clear that they're not going to get their way, in an engaging and fun manner: a Groundhog Day scenario.

The party encounters the villain, he attempts to intimidate them, they refuse to bow, so he kills them all in some creative manner.

Then, he brings them back to life, with their memories of what happened intact.

The villain repeats the process, killing the party in various different creative ways, each time returning them to life, and each time offering them further inducements and threats.

As the party is trapped in this endless loop, they are able to make choices and decisions of their own, but any time they stray from his desired outcome, they die in a horrible/creative/hilarious fashion, the scenario resets and they are back where they started.

As they proceed through different versions of the scenario, they have the opportunity to interact with the villain, and possibly learn more about him and his motivations. Perhaps each time they get a little further along, until they finally figure out exactly what he wants them to do, and how they might be able to cooperate without giving up.

This would likely be a one-time scenario, not reusable, but it would provide some great opportunities for role playing, and allow some storytelling as well as some dark humor if that's your flavor.

The villain is able to kill the entire party easily, and then bring them back again as many times as he wants, so that makes his greater level of power clear (Could be accomplished by use of a powerful artifact, which the players end up needing to destroy later on.)

It's still railroading, but it's a form of railroading where the players retain some choice and might actually enjoy the process.

This is railroading, because you are removing the effects of the player's choices, or "agency" as it's often called. If you go down this path,¹ then you should accept what it is you are doing, and the potential consequences that may result, such as unhappy players, and possibly leaving players. It could be an interesting start to a campaign, and an interesting campaign theme (nature of culpability and redemption), but it seriously something you need to get buy-in from yours players for before you start playing.

That said, in 5e there are three "good" (i.e. supported by rules, which can make it more palatable)² ways to do this that I've found, via spells (in descending order of utility): Dominate Person, Geas, and Power Word Pain.

Note a few things:

1. These are from WOTC-published material (the first two are from the PHB, the last is from Xanathar's Guide). They are also high level spells, which you may want to upcast higher: depending on your players' knowledge that may be enough to "frighten" them.




2. Dominate Person: By RAW this allows you use your action to control a person directly for a turn, should they fail their saving throw. They do not get a new saving throw unless they take damage. The effect lasts for 1min/10min/1hour/8hours when using a level 5/6/7/8 spell slot to cast the spell.




3. Geas: After a minute casting type, you lay out commands. The effected creature knows it takes a massive amount of damage (5d10 psychic) each day if it fails to follow your instructions.




4. Power Word Pain: While this has potentially infinite duration (it lasts until a saving throw, made at disadvantage at the end of each turn, is succeed), it explicitly states it causes pain and imposes a series of debilitating mechanical conditions. Simple coercive force.




5. The main limitation on all of these Enchantment spells is that they are single target. You may give this powerful "Big Bad" NPC an ability (if more of a spell-caster) or an (Aberrant? Fey?) artifact (if more of a warrior, in which case it may also grant them the ability to use these spells) that allows them to use Enchantment spells on all targets in an area (which by itself is enough to make a knowledgeable player view them as a locality-level threat). Note that in 5e, unlike some prior editions, NPCs are explicitly not built using PC rules. As a result, they can have skills or abilities not available to PCs, or to PCs of their type.

¹ Generally, I like others here, don't like or recommend removing players agency, i.e. railroading, in no small part because it breaks the default social contract, the the DM/GM/ST, etc controls the world, and the players control their respective characters. Different social contracts are available, but they should be explicitly agreed to.

If it must be done, like another answer suggested, do it during "the backstory" or "session zero". Failing that, if you must do it, I would recommend getting it over with at the beginning of the first session, if and only if "getting over it" is a major theme of the campaign, though honestly this is the type of thing that I really thing you need buy-in from your players on before you start playing.

² Using "in-game" rules, especially officially published ones over home-brew, can be more palatable to certain players. They typically have saving throws, are at least somewhat balanced (especially for the level they are expected to be encountered at), and eventually have the possibility of being used by the players. There are many effects that can compromise a player's agency (a notably acceptable example, being grappled), but because they are game rules, they are generally not considered rail-roading.

Don't be the railroader. Use partecipationism.

You can have an invincible big bad that your players cannot beat, and force the story on them, sure. Is this going to backfire? Oh, sure it will, as you can see in most answers here.

Most characters, and most players, will see the evil guy trying to force their will on them and be all "like hell I'll do what he asks!". They will get killed, they will run, they will rebel and they will fight, which of course is why you're trying to find a balance between "strong enough to survive" and not killing your players with the reaction.

Have your players accept your idea first, out of game. Have the pitch for your game talk about it. "We're going to focus on characters who will unwittingly work fot the evil overlord because they don't think they have any other choice, but really want to overthrow him when the occasion comes."

This is a cool story of redemption, chances are your players will be ok in helping you make it work (or they won't like the idea, which basically means you just saved yourself a big headache fot that's what you would have got with a surprise plot and their reaction.)

Have the villain be the ruler of the location they are in.

Make them the king of the area they are in, and their foes plausible enemies.

"I King Wrath thank you for your service thus far. I have more need of you. There is an orc village nearby that has been raiding my caravans. As a show of force, destroy them for me my knights, down to the last child."

Have them have an obviously overwhelming personal entourage. High level wizards, magical monsters, several knights, and explicitly say they look like an overwhelming force.

Then they can decide what to do with said village. They may side with them. Evil people tend to have plausible reasons for why their course of action is good.

First ask yourself, Why am I doing this?

1. I need the players to be forced to do some tasks for the villain, because that is how the story is set up.

If that's the case, tell them that's what you want them to do.

Tell them you want to set up a major plot hook and you need the players to be reluctantly forced into working for or with the villain, but they will get an opportunity to escape, rebel, save the prisoner later. Then do it as a quick cut scene. Many players will have no problem getting the players to agree as long as they know it is a set piece and its short. There is nothing wrong with admitting you don't have a different setup or that you want to try something different.

Alternatively, give the villain hostages and make it really clear they hostages die if the PC's don't cooperate, win or lose. Players are surprisingly less likely to sacrifice NPCs in a such a situation. Its one thing to sacrifice yourself its another to sacrifice someone else.

2. I need to show my villain is powerful, so defeating them will carry weight.

Then make them powerful, give them spells or have spellcasters working for them, have them cast sleep on the party or trap the party and just leave them, clearly not seeing the party as worth their time. If they fight in melee the villain can decide to leave them unconscious but alive from a melee strike.

Alternatively give them a different kind of power, make them a powerful aristocrat or foreign dignitary, the players can fight them, but make it clear they will then be fighting an entire army or even their allies.

Lastly villains have contingency plans, if they lose they may teleport, or the players could discover they are simulacrum "how strong is the real guy?", or that was just the villains underling the real villain looks like X. each of these is is fairly easy to leave as an option, then you only use them if you need them.

Play a game system that is better suited for the game you want to play.

There are a number of games like Exalted or the Chronicles of Darkness that have fleshed-out social systems that allow NPCs to compel PC behaviour (and vice-versa) to some degree, upon successful rolls. Dungeons and Dragons does not have these sorts of systems, so it is less optimal for these sorts of things.