topic Re: CFPB released a report comparing FAKOs, FICOs, and VantageScore in Understanding FICO® Scoring

CFPB released a report comparing FAKOs, FICOs, and VantageScore

cashnocredit — Sun, 30 Sep 2012 04:38:58 GMT

For the geeks amongst us, the CFPB published the first seriouis comparison of FAKOs and FICOs.

http://files.consumerfinance.gov/f/201209_Analysis_Differences_Consumer_Credit.pdf

There's a lot of good info in this report for those interested in FAKO/FICO correlations as well as FICO to FICO correlations. They studied a really large sample of credit scores.

Generally, FAKO's correlation coef was approx .92 with FICOs. Interestingly, VantageScore (v 1.0) correlated somewhat less even though it is a validated credit score with published statistics. This is somewhat surprising since FAKOs are generally "educational scores" not used by lenders and they haven't gone through the processes needed to be accepted by lenders.

Perhaps one of the more useful tidbits is that CC industry specific FICOs correlate at .99 to generic FICOs so you can pretty much bet that the generic FICOs availalble here are useful approximations for CC specific ones.

One caveat. There was no attempt to determine how well any of the scores actually predicted default risk, their purported goal. One can't tell much about the efficacy of the various scores from the study.

Also, the study normalized all scores into a rank ordering percentile so as to compare apples to apples since the scores have different ranges and distributions. Most have a long tail on the high risk side. VantageScore being the exception which is fairly evenly distributed.

Another interesting thing is that the correlation is higher for low risk scores so FAKOs, to the extent they can be used as rough proxies for FICOs, are better at the higher score levels than they are for folks with impaired credit.

Anecdotally, I've had score variations between FICO and Score Plus, which was studied, that was about 4 standard deviations outside of their published distributions, which reminds me that YMMV too!

Re: CFPB released a report comparing FAKOs, FICOs, and VantageScore

Revelate — Mon, 01 Oct 2012 16:51:39 GMT

Interesting that FICO listed the inquiry penalty as being 5 points for individually counted inquiries (vis a vis non-mortgage / SL / auto within the 30 day window on the FICO '04 model). I wonder if that's the same for all versions, namely bankcard as compared to classic/standard?

Re: CFPB released a report comparing FAKOs, FICOs, and VantageScore

pipeguy — Mon, 01 Oct 2012 18:36:55 GMT

Thanks for the link - VERY interesting study!

I like this quote which shows how FICO might be the "standard" but even using FICO, there is no set standard:

FICO’s most current model is FICO 08, but commercial users still use earlier versions of FICO products. Additionally, FICO’s generic scoring models – the most common FICO scores that are developed to predict performance on all types of credit - vary across the nationwide CRAs because the FICO scoring models are designed specifically for each CRA and reflect differences in how they organize and present credit report data.

FICO offers industry-specific models for credit cards, mortgages, auto loans, and telecommunication services. FICO models typically generate credit scores in the range between 300 and 850. FICO also builds custom models that are designed for specific companies’ credit underwriting needs.

Re: CFPB released a report comparing FAKOs, FICOs, and VantageScore

cashnocredit — Mon, 01 Oct 2012 18:38:09 GMT

I think they meant the 5 points as an average but I'm pretty sure it has to be similar on bankcards v generic. They did state that bankcard specific FICO scores correlated at .99 with generic FICOs. So if they differ it cannot be very much.

Re: CFPB released a report comparing FAKOs, FICOs, and VantageScore

Revelate — Mon, 01 Oct 2012 19:25:42 GMT

cashnocredit wrote:
I think they meant the 5 points as an average but I'm pretty sure it has to be similar on bankcards v generic. They did state that bankcard specific FICO scores correlated at .99 with generic FICOs. So if they differ it cannot be very much.

You're probably correct on both counts, though to state the correlation factor is 99%, it would've been handy to give us some details as to the average report data as well as we know that not all credit reports are remotely close to equal even if they have the same score under some given version of the model.

I didn't look through the report closely enough, but on their initial data set they didn't suggest whom the bankcard-enhanced score was provided from. Also while mortgage is intrinsically tied to FICO '04, which was likely the overarching theme of the study (their breakpoints had mortgage industry written all over it), I do wonder if the FICO '08 model which has the second (I think, at least subsequent) revision of the bankcard enhanced score, might be a larger disparity.

Re: CFPB released a report comparing FAKOs, FICOs, and VantageScore

cashnocredit — Mon, 01 Oct 2012 19:55:06 GMT

Revelate wrote:
cashnocredit wrote:
I think they meant the 5 points as an average but I'm pretty sure it has to be similar on bankcards v generic. They did state that bankcard specific FICO scores correlated at .99 with generic FICOs. So if they differ it cannot be very much.
You're probably correct on both counts, though to state the correlation factor is 99%, it would've been handy to give us some details as to the average report data as well as we know that not all credit reports are remotely close to equal even if they have the same score under some given version of the model.

I didn't look through the report closely enough, but on their initial data set they didn't suggest whom the bankcard-enhanced score was provided from. Also while mortgage is intrinsically tied to FICO '04, which was likely the overarching theme of the study (their breakpoints had mortgage industry written all over it), I do wonder if the FICO '08 model which has the second (I think, at least subsequent) revision of the bankcard enhanced score, might be a larger disparity.

Yes, there are interesting questions one could have of the more recent vintage scores. The housing crisis changed the time honored pattern of paying mortgages before anything else. That FICO has strategic default risk scores that predict those mortgages at high risk of default while the consumer continues to pay their CC suggests a recent divergence between generic FICO scores and CC specific ones assuming the models have caught up.

Re: CFPB released a report comparing FAKOs, FICOs, and VantageScore

GregB — Mon, 01 Oct 2012 21:39:19 GMT

I find it difficult to believe that Classic FICO and Industry Specific FICO correlate at .99.

All the Industry Specific FICO run the Classic FICO Scorecards and then add two additional Scorecards can affect that score by 50 points in either direction. The stated goal of each industry is to give a score that is equivalent risk as the same Classic FICO but limited to that industry. In other words, a CC Industry FICO of 700 is the same risk for a CC as another person that had a Classic FICO of 700.

Re: CFPB released a report comparing FAKOs, FICOs, and VantageScore

cashnocredit — Mon, 01 Oct 2012 22:05:53 GMT

GregB wrote:
I find it difficult to believe that Classic FICO and Industry Specific FICO correlate at .99.

All the Industry Specific FICO run the Classic FICO Scorecards and then add two additional Scorecards can affect that score by 50 points in either direction. The stated goal of each industry is to give a score that is equivalent risk as the same Classic FICO but limited to that industry. In other words, a CC Industry FICO of 700 is the same risk for a CC as another person that had a Classic FICO of 700.

From the report:

"CFPB analyzed credit scores from 200,000 credit files from each of the three major nationwide CRAs: TransUnion, Equifax, and Experian."

"FICO Auto scores had a correlation of 0.99, 0.95, and 0.98 for bureaus 1, 2 and 3 respectively with a generic FICO score. FICO Bank Card scores had a correlation of 0.99, 0.99, and 0.99 for bureaus 1, 2, and 3 respectively with generic FICO scores."

This is by far the largest sample setI've seen so I'm inclined to believe it.

Re: CFPB released a report comparing FAKOs, FICOs, and VantageScore

Revelate — Tue, 02 Oct 2012 03:24:47 GMT

I don't know; the auto-enhanced option being 0.99, 0.95, 0.99 respectively just appears to be laughably off unless out of 200k consumers, 99% of them had an auto-loan on their record. I can't imagine that's the case. The "first-time-buyer" penalty is so well known, I just find it shockingly remote that the correlation is that high.

Either everyone is using FICO '08 versions for auto and possibly bankcard enhanced industry specific options that the FICO '04 ones in the study are irrelevant, or something just doesn't seem right with their analysis on that honestly. Granted anecdotal evidence isn't proof of anything, but at least on the auto forums it comes up over and over and over again to the point it's simply accepted as near absolute truth. In credit card approvals we also see a wide variation from the best FICO '04 baseline we have available, namely myFICO's EQ Beaction 5.0 (also provided by DCU), and the longer term members who have access to their EX V2 score from PSECU, also seems to disagree with their results as GregB suggests.

Re: CFPB released a report comparing FAKOs, FICOs, and VantageScore

cashnocredit — Tue, 02 Oct 2012 14:30:12 GMT

Revelate wrote:
I don't know; the auto-enhanced option being 0.99, 0.95, 0.99 respectively just appears to be laughably off unless out of 200k consumers, 99% of them had an auto-loan on their record. I can't imagine that's the case. The "first-time-buyer" penalty is so well known, I just find it shockingly remote that the correlation is that high.

Either everyone is using FICO '08 versions for auto and possibly bankcard enhanced industry specific options that the FICO '04 ones in the study are irrelevant, or something just doesn't seem right with their analysis on that honestly. Granted anecdotal evidence isn't proof of anything, but at least on the auto forums it comes up over and over and over again to the point it's simply accepted as near absolute truth. In credit card approvals we also see a wide variation from the best FICO '04 baseline we have available, namely myFICO's EQ Beaction 5.0 (also provided by DCU), and the longer term members who have access to their EX V2 score from PSECU, also seems to disagree with their results as GregB suggests.

To start with, only one of the three CRAs correlate at .99, The other two are .98 as well as .95.

But yes, overall it seems surprisingly high. In my own case the scores I had just a year ago were way outside that expected from the published correlations.I had a Plus score that was well above the 90 percentile while EQ FICO was at 40 percentile. This is an extreme rarity according to the report's data.

There's another factor at work here. .99 or .98 let alone .95 correlation does not mean the scores are all that close. It allows for a significant number to be quite different from the decile they are compared to. If you look at FAKO/FICO comparisons that correlate at .93 there is more than a 50% chance the scores will be in different deciles and that can map to material FICO score deltas. Especially at the lower end. .99 could still be correct if the differences are skewed towards a small group, the very young just starting out in both credit and first auto. That combo of short CC history and no auto loan would inidcate beginning driver and that's where the risk differential is.

Re: CFPB released a report comparing FAKOs, FICOs, and VantageScore

llecs — Tue, 02 Oct 2012 14:33:57 GMT

Disclaimer: I didn't read the entire PDF and missed the methodology probably, but I'm curious how they obtained the data. If a lender pulled a FICO, they wouldn't both pull an industry-specific score and a Classic FICO at the same time. There's no need to do that and it costs them money....unless the study paid for these 200k pulls. Without reading more, and on the surface, I have some red flags.

Re: CFPB released a report comparing FAKOs, FICOs, and VantageScore

cashnocredit — Tue, 02 Oct 2012 16:44:16 GMT

llecs wrote:
Disclaimer: I didn't read the entire PDF and missed the methodology probably, but I'm curious how they obtained the data. If a lender pulled a FICO, they wouldn't both pull an industry-specific score and a Classic FICO at the same time. There's no need to do that and it costs them money....unless the study paid for these 200k pulls. Without reading more, and on the surface, I have some red flags.

They actually did pull all the scores from a CRA at the same time. So they have 200,000 identity filtered CRs with VantageScore, FAKO, FICO, and two industry specific FICOs pulled on each consumer report at the same time.

However, these 200,000 are random samples of each CRA's entired database. They are different sets of individuals so there is no way to compare score consistency on the same individual across CRAs nor was that the studies goal.

The study was mandated by Congress and targeted only to scoring variation from one algorithm to another on the same consumer report.

Re: CFPB released a report comparing FAKOs, FICOs, and VantageScore

cashnocredit — Tue, 02 Oct 2012 18:51:15 GMT

As it is clear that "correlation" behaviors aren't generally known, Just a note of clarification.

Correlation values of .99 do not mean that 99% of the scores match or that two scores will only be, on average 1% off. It's a statistical measure and one that leaves a lot of room for variation in FICO scores. It's also kind of weird in that as it increased it tends to overstate what one would expect.

For instance if a correlation of .92 produces a typical variation of 40 points (using FICO scaling and taking a SWAG) around the 620 mark, a correlation of .98 would only reduce the expected variation to 20 points. Alternately, if 1 out of 4 people had no history of autoloans and took a 40 point hit as a result but otherwise matched the generic FICO that would also produce a correlation of .98