Collections and charge-offs can tank your FICO score...at first. However, after they age a few years, their effect depends on you. If you accumulate positive credit history--accounts in good standing--this tends to dilute negative accounts. I have four such baddies on my CR, yet if not for my high utilization--something I really need to work on--my FICOs would probably be near or even slightly above 700. In a year or so, the collections will still be there, but so will two years of unblemished credit history with about five revolving accounts, and so my scores will be in the mid-700s.
And yes, losing credit history and taking a length of credit history hit can be an unintended but very real consequence of disputing accounts.