It is possible that these lates are hurting your score but I just had an account deleted from Experian with lates and NO score increase whatsoever. I have found a bigger score jump with CO coming off and with "improved" accounts. I had one on Equifax that was a derog but since it was paid in full they changed it to paid as agreed and closed. Equifax went up 9 points. Still waiting for Experian to update the same account but so far I have NO LUV from Experian.
I believe that the "condition" of the account helps/hurts the score more. (derog, included in wage earner plan, paid as agreed, etc)