I did this with WF financial Visa - it's a subprime card (was 23.24 APR). I called and threatened to CXL the card. I asked them to waive the $30 AF. They put me on hold and when the CSR came back she said the fee would be waived. PLUS, I didn't ask for it, but the APR was lowered to 12.25 APR variable (which, thanks to Uncle Ben Bernake is now 11%).
So call ahead first and see if the "account retention" department can do something nice for you!