In Macbeth, why might Shakespeare have chosen to make Macduff rather than Malcolm the agent of nemesis?

This is a great questions. Usually a nemesis is considered
a "bad guy" but actually, Malcolm is acting as  the moral agent as well as a nemesis in
this play.  First let us look at some definitions of
nemesis.

Nemesis is defined in several way, generally we
know it as some type of unbeatable opponent.  However, another definition is a reference
to Greek Myth showing Nemesis as the Greek goddess of retributive
justice.

Even though modern movies show a nemesis as losing
to hero, such as Joker to Batman, in earlier literature, the nemesis usually ended up
victorious. 

Malcolm, as we can tell from early in the play
in Act I, is a young man.  He is being named Prince of Cumberland, so he is likely only
in his early teens.  This makes him a bit young to take on the role of nemesis, or to
adminster justice to Macbeth.  He also does not fulfill the prophecies of the
witches.

Macduff however, is roughly the same age and
position as Macbeth (before he became King, of course).  His entire home has been
slaughtered, and most importantly, he was not born of woman.  By convincing Malcolm to
return to Scotland, he takes on a greater, if temporary role, as mentor to a future
King.