Start by saving the rule as a manual rule and reading the players it put in each group. Most of the time the logic isn't doing what you meant, and the player list shows you where.
A manual rule is a snapshot. It lists exactly which players matched your logic, one group at a time. If the wrong players are there, or the right ones are missing, or there are far more groups than you expected, the rule is built wrong. Fix it, check again, then save it as an auto rule so it recalculates every build.
Three things account for most of what goes wrong.
Group By is set wrong. Group By controls how many groups get made. Say you want at least one player under 5% owned in every lineup. If you group that by player, you get one group per qualifying player, each saying "use at least one of this player," which tells the builder to use every one of them in every lineup (which is impossible, of course). Turn Group By off and you get one group of every qualifying player with a single "use at least one," which is what you meant.
The conditional is backwards. The primary group is the trigger. "If a primary player is used, then apply this to the secondary players." If the trigger is in the secondary slot, the rule fires in the wrong direction.
The rules conflict. If your build produces very few lineups or fails, the rules can't all be satisfied at once. Remove them and add back one at a time.
If the rule is saved as manual when you wanted auto, it's also stale: a manual rule keeps the players who matched when you saved it, even after projections change.