It could be hurting or helping your score. If this TL is older than your AAoA, especially by a good number of years, then it's age is probably helping more than the CO reference is hurting. If younger, then it is hurting. The damage is based on the other baddies reporting outside of this and how this pd CO is reporting (lates, etc.). However, you still have the CA reporting and the damage from that is quite a bit. If that is removed, the points gained could vary. It would depend on a number of variables like the assigned date, DOFD, DOLA, how it is reporting, the number of CAs remaining and the number of other baddies remaining. Most people see anywhere from 20-100 points when their last CA (and last baddie) is removed. DW just had her last CA removed (w/ one pd CO remaining) and her score only increased 3 pts. YMMV.
YMMV based on the lender. Some will take issue and some won't.