Dungeons & Dragons 5th edition began a new golden age of D&D because of one, simple reason: approachability. Where previous editions involved adding together multiple modifiers (up to eighteen) and so many rulebooks the Session 0 often involved asking around how many books could be used, D&D 5th was… simple. Proficiency bonus plus attribute, maybe plus a bonus from an item. That’s it.
Contrast this with Pathfinder 2nd edition. That book is dense. Beautifully, beautifully dense. Racial abilities, class specialisations, endless tweaks to perfectly your vision of a character, with rules backing each choice up to create a wonderfully crunchy system that is truly delightful.
Unless your players have dyscalculia, ADHD, or just had a new baby on top of a toddler and they need a dead-simple build to pilot during their four hours out of the house they get every two weeks.
Thus, the question asked during campaign inception: which system? D&D or Pathfinder? Mutants & Masterminds vs. Sentinels? Blades in the Dark or World of Darkness? When do you need complexity, and when do you need abstraction?
Complex Characters
Some games really like Man vs. Self conflicts. Vampires who need fresh blood but try to resist, Chthonian horrors slowly draining sanity, or complex combat rules involving long-term injury all require rules more complex than mere hit points. Different but related are rules covering disability, neurodiversity, mental illness; if these are important to the narrative of your game (or your players), they deserve to be represented within the world, which means they should have rules.
The greater the diversity of character creation, the more complex the rules need to be to account for those differences. If you want to have character specialisations in different abilities, and for those abilities to work differently depending on the character, you need a decent chunk of complex rules to account for the differences and ensure balance across your PCs.
Character sheets are a portrait painted in math. If a character’s portrait is incomplete without certain rules, they should added, or a new subject for the portrait is needed.
Complex Conflicts
This depends on the campaign, but the more complex you want a conflict to be, the more rules you’ll need.
Should breaking into a safe be handled with the same rules as gaining information from an unreliable source? What about combat? Should magic exist, and if so do the rules work differently from swinging a sword?
Rules can make a conflict feel more real, by creating meaningful choices, or can abstract a conflict away into something easy. I haven’t met anyone (yet) who wants their climactic air-ship battle to be hand-waved with a nominal die roll and a description of their ultimate victory. In general, the higher the stakes of the conflict, the more rules support it should have so the terms of victory and defeat are known in advance.
During campaign inception, the type of conflict driving the plot should be relatively obvious. Make sure your rules support the conflict meaningfully.
Complex Results
This is one I don’t see people discuss very often, but is an interesting question: After the die is cast and the math is done, how many potential outcomes do you want to exist? Most likely at least one form of success and failure, but what about partial success? Can players sacrifice resources to increase a near-miss to success? What if they accept a complication?
I’ve seen groups talk about whether fudging a die roll to manoeuvre the campaign away from catastrophic failure like the loss of a beloved PC, but few ask for rules to describe when this is allowed. By actively thinking about definitions of failure, success, and everything in between, a group can decide to formalise these decisions (which can be satisfying) or communally decide on social conventions to do the same (which can be faster).
No matter what, if you want to have a difference between success and failure, exceptional triumph or near defeat, rules will help illustrate your game.
Long-term Games
The longer the game, the more situations will come up, thus the more rules you’re probably going to need for the game to stay satisfying.
Game balance is hard to get right in any system, and character progression means not only does a game need to be balanced at the start, but stay relatively balanced as play continues and characters gain experience, levell up, get new powers, and generally become more Badass. Keeping characters roughly equal in power is… complicated, particularly if you want the characters to be different.
Not to mention, their opposition should also become more powerful, and most likely more interesting and complex as PCs get a larger toolbox to solve problems. All this throughout the arc of each character’s progression, for any of the myriad of character options.
Conclusion
Rules complexity depends on the length of the game and the complexity desired within the game. Long-running, well-established systems tend to handle character progression balance better than others, if only because of the sheer number of playtest hours. (Not unlike software.) Larger rulebooks are more likely to have all the rules you’ll need, but can create more friction during play while players remember and reference the rules.
At their most fundamental, rules represent the model and physics of the world within your game. Gritty, realistic, and conflict-driven games need more rules. Otherwise, use the lightest system possible to reduce friction and maximise time having fun rather than looking at a rulebook.