Yes, there is no way that paying the annual fee on a CC dropped your score. It isn't on your credit reports, and if something isn't on your credit reports, it doesn't go into your scores.
My guess is that the reported balance went up on an existing account, or an extra account reported a balance or something. And it's possible that if that $20 fee showed up on your credit report (as if it were a charge or something), that might have done it, especially since you haven't used the card in a while. It wouldn't be because it was the annual fee or that you then paid it; it would be because it was a new balance on a card previously reporting $0.
You can go nuts quickly, and many of us have, sorting out cause and effect with score changes, but safe to say, paying your fee isn't one of them.
Here are two sources you might want to start with:
Understanding Your FICO ® ScoreCredit Scoring 101 (at least the first post.)
* Credit is a wonderful servant, but a terrible master. * Who's the boss --you or your credit?
FICO's: EQ 781 - TU 793 - EX 779 (from PSECU) - Done credit hunting; having fun with credit gardening. - EQ 590 on 5/14/2007