In this post I will outline three (or two) different ways in which rules can be good. For the purpose of this post, let rules be good if and only if they produce good play, and let good play be defined by the people playing.
I am also assuming that freeform play, defined as play in which resolution is handled by social negotiation or fiat and where there are few explicit rules, is not utterly broken and can actually work.
Fading to the background
As was assumed above, freeform can be good play. Rules that fade to the background make freeform easier and channel it to specific fictional style. Take, for example, a game that has attributes (strength, speed, …). Creating characters gives players a sense of how capable their characters are when compared to each other, and also possibly to the rest of the world. Note that this also happens to other numbers that represent the character.
It has been my experience that given a set of rules that tend to fade, the resolution system is first used frequenly to resolve anything and everything, but it is used less and less as the game proceeds, because people already know what is going to happen. Your character has three times bashed a door down and charged in, so we already know your character is capable of bashing doors down, and there’s no need to roll anymore (assuming a door that is not stronger than those bashed before).
Rules, even though they fade to the background, also shape the fictional world. Take a random fantasy rpg and add magic rules stolen from D&D or Ars Magica, and you will have vastly different sorts of mages in the setting.
I have heard that BRP (basic roleplay, used in Runequest, Stormbringer and Call of Cthulhu) and Unisystem are this kind of games.
Creating fun play
Some rules have fairly discrete mini-games. The traditional example is combat in modern D&D (3rd and 4th editions). It is clear that such mini-games don’t fade into background; rather, they are entertaining in and of themselves (given players who like such mini-games, of course). Another feature of them is that there is no need to ever use them; you can play D&D 3rd and never touch the combat rules. It would be something of a waste in that most of the game would be unused, but that doesn’t really matter, as long as the play is good.
In summary: Good rules can create new kinds of good play, or make an existing activity interesting in a new way.
An extreme case of the above is rules that don’t work properly unless they are embraced. Almost all board games are in this category. Burning Wheel is close, because the character development and artha mechanics require actively using and remembering the rules.
I’d be as bold as to say that many games influenced by rpg theory from the Forge are close to this extreme case. It might even be possibly to characterise them by this classification scheme, though there is bound to be loose ends.
From design perspective it is useful to have a baseline; what am I improving? (My default baseline is freeform play.) Minigames and games that simply don’t work unless used properly may benefit from another point of comparison, or may be considered without much context, which is not advisable to any game that relies heavily on people already knowing a particular activity or mode of play.
Continuum, not absolutes
As is generally true, this classification creates a continuum, not two (or three) pair-wise distinct sets.