More importantly, what is the rate you got from HSBC? It is lower right?
As long as the rate is lower, and you are saving money, you should definately do it! Saving money is more important than score fluctuations. The overall impact on your score after refinancing will be minimal. Your score will probably go down in the short term as you are closing a current account and opening a new one. They will bounce back though.