Re: Scorecard reassignment and wonky score changes... - Page 2 - myFICO® Forums

Anonymous · ‎11-25-2020

@Anonymous wrote:
Maybe AoOA of 3 years being a segmentation factor isn't the best idea then?

They must have found a good set of predictor variables for the <3years group that weren't so great in the >3years group. They didn't make it up arbitrarily.

They can only predict defaults within a narrow window of time. It seems like 2 years is the maxmium look ahead that's reliable in credit risk modeling. I see that number a lot in FICO whitepapers, and they use that timeframe for model development profile data.

Because FICO says they need 6 months of data to generate a score in the first place, I think recovery from those hard drops when switching scorecards will be mostly complete within 6 months of history on the new scorecard. I'm definitely going to find out firsthand over the next 6 months.

Curious_George2 · ‎11-25-2020

@Anonymous wrote:
It's not a penalty due to age alone, though. Penalties are only higher for certain characteristics - inquiries, new accounts, utilization, etc. - because the percentage of bads in that scorecard is higher than other scorecards. More bad profile targets means more of a chance to have something in common with some number of them, which will lower our scores.

These scorecards (bins) are set primarily from the already observed profile data over 2 years, and guided by data scientists to find the best predictors (Information Values and Weight of Evidence). It's not "People new to credit are inherently a higher risk", it's "The data we have already collected says that people new to credit are a higher risk". They can already see the performance over time for a person that started fresh and ended up defaulting within 2 years.
...

Thanks for your insightful (as always) comments. You make excellent points. I was still feeling a little unclear on why FICO uses scorecards and your reference to whitepapers inspired me to go looking for one. I found a whitepaper that makes a variety of arguments in favor of using scorecards, including "palatability" and "interpretability" of the inner workings of the scoring model by the human users. That makes sense to me, since each model is a product that FICO has to persuade lenders to buy. That would be harder if it were built entirely out of incomprehensibly complex mathematical formulas. Scorecard-based algorithms are much more PowerPoint friendly.

The whitepaper also says that the selection of the specific boundaries between bins (apparently they're called "knots") is as much art as science, and it is sometimes based on what the developer or user feels is a "nice" number. Interesting stuff!

Anonymous · ‎11-25-2020

@Curious_George2 wrote:
@Anonymous wrote:
It's not a penalty due to age alone, though. Penalties are only higher for certain characteristics - inquiries, new accounts, utilization, etc. - because the percentage of bads in that scorecard is higher than other scorecards. More bad profile targets means more of a chance to have something in common with some number of them, which will lower our scores.

These scorecards (bins) are set primarily from the already observed profile data over 2 years, and guided by data scientists to find the best predictors (Information Values and Weight of Evidence). It's not "People new to credit are inherently a higher risk", it's "The data we have already collected says that people new to credit are a higher risk". They can already see the performance over time for a person that started fresh and ended up defaulting within 2 years.
...
Thanks for your insightful (as always) comments. You make excellent points. I was still feeling a little unclear on why FICO uses scorecards and your reference to whitepapers inspired me to go looking for one. I found a whitepaper that makes a variety of arguments in favor of using scorecards, including "palatability" and "interpretability" of the inner workings of the scoring model by the human users. That makes sense to me, since each model is a product that FICO has to persuade lenders to buy. That would be harder if it were built entirely out of incomprehensibly complex mathematical formulas. Scorecard-based algorithms are much more PowerPoint friendly.

The whitepaper also says that the selection of the specific boundaries between bins (apparently they're called "knots") is as much art as science, and it is sometimes based on what the developer or user feels is a "nice" number. Interesting stuff!

Oh cool, you read the FICO Model Builder paper. That's a great general overview.

Here's some more information from FICO about domain expertise: Superior Scorecards at Warp Speed: Algorithmic Learning Meets Domain Expertise

Anonymous · ‎11-29-2020

@Anonymous wrote:
Looking to have a discussion on this topic, as I know there will be differing opinions. We know that score ranges for different scorecards can overlap in spots. As a result, someone that has a positive change happen to their file could result see a score drop. I understand that scorecards are based on groups of people with similar profile data and that we really shouldn't get hung up on any one individual example. For me though personally, I don't necessarily agree with how that works. If a Fico score is a representation of risk / likelihood of default, it doesn't make sense that a positive profile change would result in a lower score.

For example, someone reaches an age of accounts scorecard reassignment that results in a score drop. Looking at that individual alone, there's no way that they became more of a risk (lower score now) due to positively crossing an age of accounts threshold. Now that person lost (say) 20-30 points which could be the difference between X rate and Y rate on a new mortgage. That just seems counterproductive to me.

Anyway, I get it that it "is what it is" and how the system works when looking at groups of individuals and that it isn't going to be perfect at all times. Just curious to hear thoughts from others on this topic, whether you think it matters and/or if you believe this issue should be addressed. If so, what would you propose... such as no overlapping score cards, for example?

@Anonymous I have not yet read the responses to the thread so I preface my post with that.

in theory, I agree with you. It does create an issue that the score is not linear and is instead generated by placing profiles in groups/scorecards, according to logistic regression analysis. This is one of the imperfections because, as you pointed out, becoming a day older does not necessarily translate to an increase in risk.

I don't know how this could be fixed, as I'm not privy to how exactly the algorithm is designed and generated, but I would point out that crossing that age threshold does not always result in a decrease. It can result in an increase, depending on the individual profile, as you so eloquently stated.

The simple fact is the characteristics/metrics are weighted differently per scorecard and so by moving to a different group/scorecard , it causes a reassessment that can result in an increase or decrease, tho typically the example you provided results in a decrease.

I think this is an example of what was misguidedly called in one article instability, which can correct itself with a little time, a few to several months. So I think the general answer is that while the characteristic change may cause an abrupt score change, with a little bit of time, it should correct itself.

Anonymous · ‎11-29-2020

Now that I’ve read the replies, I have a couple comments. First, regarding the segmentation point of 3 years for age of oldest account, keep in mind on the old versions that was 2 years.

Apparently due to analysis, that was moved to 3 years with version 8. Additionally, we must keep in mind that the objective of the algorithm is to predict the likelihood of a 90 day default within a 24 month period. That could be significant as well.

And Cassie‘s comments about it correcting itself within six months are illustrative of what I was speaking of regarding the temporary instability as it has been described that corrects itself in a short period of time.

Remedios · ‎11-30-2020

@Anonymous please help me find one example of score increase when first account reached 3 years of age because I cannot (one where there is no doubt that's what caused increase)

Anonymous · ‎11-30-2020

@RemediosOK let me do some searching and maybe even make a post, I will try to find one.

@Remedios Edit: I am hoping that Cassie is one due to the inquiry count and low utilization; it also may play a part that she had a loan first and therefore her score has not went as high because the age of her revolving account is not as high. Plus the new accounts are penalized less on a mature card, so that argues in her favor too.

Remedios · ‎11-30-2020

Thank you

Remedios · ‎11-30-2020

@Anonymous spare yourself from searching, you won't find an example of a gain.

As far as @Anonymous , I'd guess she's going into 740something to 750 on TU and EQ.

EX is already kinda leveled, so I wouldn't expect too much of a change there

Or, maybe she will stay roughly where she is. My scores were higher than Cassie's when I got sent to big girl scorecard, so I guess I had to "lose" more

EQ was my highest (788), and that's where the real drop happened, about 25 points, bringing it in line with the other two who were around 765, and they dropped to 751 (identical drop, data not identical due to number of HPs).

I had multiple accounts under a year, and it didn't make it any better. Before I could have up to 50% of accounts report a balance with miniscule drops, after reassignment, penalties for additional cards reporting balances were much higher.

That's just my experience, but @Anonymous profile and mine are nothing alike, I had over five stupid accounts before Fico was generated (no kidding), I've never experienced HP points losses like Cassie, and I cannot remember a life without new account penalty.

I should be free on January 1st. I want all the Ficos! All of them.

SouthJamaica · ‎11-30-2020

@Remedios wrote:
@Anonymous spare yourself from searching, you won't find an example of a gain.

As far as @Anonymous , I'd guess she's going into 740something to 750 on TU and EQ.
EX is already kinda leveled, so I wouldn't expect too much of a change there
Or, maybe she will stay roughly where she is. My scores were higher than Cassie's when I got sent to big girl scorecard, so I guess I had to "lose" more
EQ was my highest (788), and that's where the real drop happened, about 25 points, bringing it in line with the other two who were around 765, and they dropped to 751 (identical drop, data not identical due to number of HPs).

I had multiple accounts under a year, and it didn't make it any better. Before I could have up to 50% of accounts report a balance with miniscule drops, after reassignment, penalties for additional cards reporting balances were much higher.
That's just my experience, but @Anonymous profile and mine are nothing alike, I had over five stupid accounts before Fico was generated (no kidding), I've never experienced HP points losses like Cassie, and I cannot remember a life without new account penalty.
I should be free on January 1st. I want all the Ficos! All of them.

When I tested 3 years AAoA I had a "gut feeling" that I'd gained 5 or 10 points out of it, but there was too much going on in my profile to isolate anything. https://ficoforums.myfico.com/t5/Understanding-FICO-Scoring/Is-3-years-a-threshold-in-AAoA/m-p/5135502

But Revelate felt that he was getting zero points out of it.

Most likely that was before I was getting the daily update at experian.com, which has helped my ability to isolate single factors.