I'd be more worried about saving hundreds (probably thousands if you have $38k loan) of dollars in interest rates than bumping my FICO score. From what I can tell in your message, you'll be saving money in the long run, money that you can use to pay other debts if you have any and with time, your scores will go up.