"is it just a function of time and getting rid of lates? "
Generally speaking, yes ozgirl. Unless of course you can write goodwill letters, and have the lates removed. Case in point: I had a total of 34 lates(all 30 days) on my CR's back in August. Now the lates stand at 27. So 7 have fallen off, by pure aging off the reports. And I saw a slight increase in the scores.
