When it has no mechanical weight.
Alignment can be a touchy issue for gamers. Some people insist that it’s a great way to inform roleplaying, a starting point for your character. Others think it’s an unrealistic straight jacket, that nobody’s always good all of the time or wholly and irredeemably evil. It could be either of these things but most of the time it seems like an afterthought, a vestigial rule that lacks meaning for modern gamers.
Why does it lack meaning? Are the concepts of good and evil no longer relevant in a society of moral grays? Is it unrealistic to think that a character can be a shining beacon of his or her beliefs, an example to others? I kind of doubt that this is the case; I think these ideas are just as relevant now as they always were, and may actually be appealing to many because of their simplicity, their black and white nature. The problem, I think, is with implementation.
Let’s look at the current incarnation of D&D as an example. Alignment exists but it has very little effect on the game. Yes, your cleric has to be of an alignment compatible with his or her god, but what does being Good or Lawful Good or whatever actually mean? It can inform roleplay in the right hands but there’s no incentive to let it do so. I think a lot of people probably forget that it’s even on their character sheet; I certainly had to remind the paladin in my group that he’d find torture repugnant on more than one occasion.
Past editions gave it some weight: your alignment could be detected, certain weapons would respond well (or not so well) to certain alignments, and violating your alignment came with some sort of penalty. But even this isn’t a great way to incentivize alignment. It’s like working at a job you don’t care about: the threat of being fired will keep you working just well enough to not get fired, but you’re unlikely to do anything beyond that.
The solution (my solution at least) is incentive. Reward. I talk about incentives an awful lot for a reason: mechanical incentives are how you encourage the behavior you want in your games. If you want the Lawful Good paladin to uphold the law and oppose evil, you need to provide an incentive to do so that’s more attractive than the alternative. This could be as simple as providing bonuses to skill checks, attack rolls, damage, and so forth whenever the spirit of the alignment is being actively pursued. It could involve bonus XP. It could involve earning some sort of points or tokens that can be cashed in for benefits later. The exact incentive doesn’t really matter.
The important thing is that, if alignment is an important part of your game and the characters within your game, there needs to be a reason to pay attention to it. Punishment will help but will only get you so far; players will work a lot harder for rewards, things that make them better.
Alignment without mechanical weight, without incentive, fails to fulfil its purpose in the game. It might as well not even be there.