The only way this could hurt your score right now is if you currently carry balances on your other credit cards. If you do, then removing this credit limit from the mix will raise your utilization percentage. If you do not carry credit card balances, your score should not be affected.
The account will remain on your reports up to 10 years. It will continue to add to your average age of accounts all that time. We are always told never to cancel a credit card, but it can be done selectively as we outgrow credit products we don't need or want. No need to pay an annual fee if it doesn't benefit you.
I just did the same thing. I had the card for two years and barely used it. It didn't have rewards or anything like that.
I called to see if they would waive the fee in my case it's $45 and they wouldn't do it.
They did offer me six months at a lower APR but I don't carry a balance on the card so I said no and told them to close it.