Do you use the card at all? If you don't then you don't have much bargaining leverage.
Do you have a balance on the other card? If so, you might consider asking them to end the fee permanently so that you can transfer your balance from the other card over. They either will or won't.
If they won't, then the question really is... is a good score worth $48 per year to you? A 3-way report each year costs that... and a drop of just a few points could cost you more than that per year in interest.