Usage of would rather

It's a great question!

Would rather + infinitive verb

If you are talking about your own preference in a specific situation (i.e., what you want to do), or asking someone about his or her preference in a specific situation (i.e., what he or she wants to do), use would rather with an infinitive verb.

I'd rather stay in tonight.

Would you rather stay in or go out tonight?

Would rather + past tense verb

Use would rather with a past tense verb only when stating your preference about what you want someone else to do in a specific situation, or asking someone about his or her preference about what he or she wants someone else to do in a specific situation.

I'd rather he stayed in tonight.

I'd rather you didn't.

Would you rather I stayed in or went out tonight?

Please also note that, although we use a past tense verb, this construction always has a present or future meaning, not past.

It is possible to use a present tense (I'd rather he stays in tonight) or present subjunctive (I'd rather he go home now) but those tenses for this very specific what-you-want-someone-else-to-do meaning are rare compared to would rather + past tense verb.

Because using the present subjunctive tense is uncommon but not considered ungrammatical, I'd rather he be angry than grumpy is okay but I'd rather he was angry than grumpy is more commonly used.

One last point:  I endlessly repeated the phrase in a specific situation above because, like would prefer, would rather is only used to talk about preferences in a specific situation/context.  The main verb prefer is correct if you are talking about your general preferences.  Compare:

Tonight, I would rather stay in.

Tonight, I would prefer to stay in.

I prefer fish to pork.

I usually prefer to go out on Friday nights, but tonight I'd rather stay in.

I hope this helps.

You are right.  Would rather phrase is always followed by past tense verbs.

I will give you two more examples.

I'd rather you went accompanied than alone.

I'd rather you didn't come home too late.