Sometimes when you dispute things it makes FICO count the lates as newer than they really are, which is one reason you should not dispute if the information is accurate.
I say give it time you will get the points back, but do let the CRA's know that the account is closed - maybe that will help with some points, by showing a lesser amount of opened accounts. YMMV.
