The blurry lines between formal and informal forms in Japanese

I've got a question about Japanese that I'd like to address to the experts (Tae Kim, I hope you're listening!).

Most Japanese grammars present the formal desu/-masu forms and the regular forms as a dichotomy: you pick one and use it 100% of the time with a certain person or you pick the other and use it 100% of the time with that person. The thing is that a lot of native Japanese speakers mix up the two in actual usage with the same person. My question is what rules do these follow? When can you throw in a few informal forms in otherwise formal speech and vice versa? There seem to be a variety of conditions for this, but I've never heard anyone try to explain it.

Read more...Sometimes the reason for such mixing is clear. For instance, friends who otherwise never use the formal forms with each other will sometimes use them jokingly, creating an effect of feigned formality that can often be used for sarcasm. Parallels to this usage can be found in all the languages I've learned; think of using "Would you be so kind as to pass the bread?" to your boyfriend of girlfriend. You're either joking or annoyed.

But there are other times when it's not so clear.

For instance, my Japanese father-in-law typically only uses the informal forms with me. But he'll sometimes throw in the desu/-masu forms as well. It's probably more than 95% informal forms. He seems to use the formal forms when he wants to emphasize a point, and typically it's either a desu yo or a -te masu. Sometimes it doesn't seem like it has anything to do with making a point. For instance, if I say something like, "Can I borrow this?", an informal way to say "Sure" would be "Ii yo", but he might say "Ii desu yo". While that sounds more formal, it certainly doesn't feel more formal, and he'll whip right back into the informal forms on a dime after that. My mother-in-law and him also often will toss in formal forms when they are speaking to each other in roughly the same manner.

Another instance I encountered was on the train a few days ago. I was sitting down and two guys got on and were standing in front of me talking. Based on how old they looked and what they were saying, I got the impression that they were college kids, one younger than the other. The younger one was generally using the formal forms while the older one was generally using the informal forms. That is typical enough. However, while the older one didn't seem to include any formal forms, the younger one would on occasion include some informal forms.

It's clear that the use of these forms is not a complete bifurcation. There are shades of gray, in which you can throw some informal forms in with formal forms and vice versa, and I'd love to know what the rules behind this are.