Why not ask them to waive the annual fee since you're such a loyal customer? Or better yet, ask for a product change to Blue Cash or Clear with no annual fee to replace the card you have? Keep your $50 AND your history.
I agree. Call them when the annual fee is due and ask them to waive it. Or call them and convert it to one of their cards that does not have an annual fee. Or if they will not let you convert, login to their website so they know who you are, go to apply for cards, select one that appeals to you that does not have an annual fee, apply for it. It will arrive with the opening date of the old card. I think throwing away the age would be foolish when you can keep it for free. Keep your $50 and your history.
If you close the account, you won't give up the history for 10 years. Upon close, it will remain on CR and be calculated in your AAoA for 10 years, then it will drop. By that time all of your existing accounts will be 10 years older (meaning your average 16 year accounts will be 26).
myFICO.com "hints" shows that the "average FICO high achiever" has an oldest account of 19 years and AAoA of 6 to 12 years.
If Amex has continued to charge you an AF and you have paid it, then the account should be open and available and they should not deny use, unless you try something big, but a tank of gas or lunch shouldn't be a problem.
As others said, you could ask to convert to a no AF account. You could ask for a waiver of AF, but I doubt they will do this every year and you would have to call in each time.
In the end, if your AAoA is 16 years, you probably have some 20 plus year accounts too. I HIGHLY doubt that if you cancel the account your FICO will be affected, or at least not much and not for long.
My advice is always to act with an end result in mind. If you don't like paying the $50 and your desire is to stop paying, then cancelling or converting achieve the same result. Converting, though maintains your oldest credit reference which can never be a bad thing to have.
Went ahead and applied online for Blue Sky and just received notification by phone that it was approved. And it will carry the 1967 date. Will having 2 cards with 41 year history improve FICO (currently 787,807,800)?