It is tough to say if it matters or not. All banks have different UW requirements and different levels of customer service. Your broker is going to want to give you the best chance to get approved with the least effort possible. It is unlikely a broker would pick a bank that they didn't think it would be easy to push your file through with.
After closing you will see differences in customer service and in services like free autodraft payments. However typically you give permission for your loan to be sold so you don't have any control of who holds it after closing anyway.
