topic Re: Scorecard reassignment and wonky score changes? in Understanding FICO® Scoring

Scorecard reassignment and wonky score changes?

Anonymous — Wed, 25 Nov 2020 15:12:25 GMT

Looking to have a discussion on this topic, as I know there will be differing opinions. We know that score ranges for different scorecards can overlap in spots. As a result, someone that has a positive change happen to their file could result see a score drop. I understand that scorecards are based on groups of people with similar profile data and that we really shouldn't get hung up on any one individual example. For me though personally, I don't necessarily agree with how that works. If a Fico score is a representation of risk / likelihood of default, it doesn't make sense that a positive profile change would result in a lower score.

For example, someone reaches an age of accounts scorecard reassignment that results in a score drop. Looking at that individual alone, there's no way that they became more of a risk (lower score now) due to positively crossing an age of accounts threshold. Now that person lost (say) 20-30 points which could be the difference between X rate and Y rate on a new mortgage. That just seems counterproductive to me.

Anyway, I get it that it "is what it is" and how the system works when looking at groups of individuals and that it isn't going to be perfect at all times. Just curious to hear thoughts from others on this topic, whether you think it matters and/or if you believe this issue should be addressed. If so, what would you propose... such as no overlapping score cards, for example?

Re: Scorecard reassignment and wonky score changes?

SouthJamaica — Thu, 26 Nov 2020 04:32:56 GMT

@Anonymous wrote:
Looking to have a discussion on this topic, as I know there will be differing opinions. We know that score ranges for different scorecards can overlap in spots. As a result, someone that has a positive change happen to their file could result see a score drop. I understand that scorecards are based on groups of people with similar profile data and that we really shouldn't get hung up on any one individual example. For me though personally, I don't necessarily agree with how that works. If a Fico score is a representation of risk / likelihood of default, it doesn't make sense that a positive profile change would result in a lower score.

For example, someone reaches an age of accounts scorecard reassignment that results in a score drop. Looking at that individual alone, there's no way that they became more of a risk (lower score now) due to positively crossing an age of accounts threshold. Now that person lost (say) 20-30 points which could be the difference between X rate and Y rate on a new mortgage. That just seems counterproductive to me.

Anyway, I get it that it "is what it is" and how the system works when looking at groups of individuals and that it isn't going to be perfect at all times. Just curious to hear thoughts from others on this topic, whether you think it matters and/or if you believe this issue should be addressed. If so, what would you propose... such as no overlapping score cards, for example?

1. I do agree that it would be inappropriate for someone's score to drop due to a positive change in their profile, just because the positive change resulted in a scorecard reassignment.

2. This has never happened to me, and I have been on different scorecards.

3. Personally, I think the frequency of this occurring is greatly overrated by the cognoscenti among us. In my opinion it's not a big deal only because it doesn't really happen much, if ever.

Re: Scorecard reassignment and wonky score changes?

Anonymous — Wed, 25 Nov 2020 16:39:59 GMT

These scorecard changes with score drops seem to happen quite frequently to those with relatively young credit profiles compared to most of the members on this forum. Here's a whole thread of them: Oldest account hits 3 yrs scores drops up to 22 points

I'd say the first 5 years at least are pretty sensitive to these changes.

Here's my own report of it happening with the EX 2 and TU 4 models: AoOA 2yr 0mo / AoYA 1yr 0mo

On EX 2 (-22 points) I had an interesting change in reason statements that gives us a clue to what is going on inside the algorithm:

NOV	1yr 11mo	11mo	Short credit history, Too many accounts with balances, Seeking credit, Recently opened too many new credit accounts	733
DEC	2yr 0mo	1yr 0mo	Short credit history, Short revolving history, High revolving balances, Seeking credit	711

Account section was reporting exactly the same on those 2 3Bs - exact balances on same cards. I did this on purpose to isolate the aging changes.

I believe that 'Short revolving history' reappearing so prominently - after having disappeared at the AoYRA/AoYA 6mo mark - is indicating that I don't have enough history on the new scorecard.

We had something similar to this in machine analytics when certain equipment was moved to a different environment elsewhere in a factory complex: not enough data yet in the new environment, so certain predictive variables would be weighted differently for a while, even though that machine was showing a low probability of failure before the move.

This binning (scorecarding) of the entire dataset is a necessary part of predictive analysis. The entire profile dataset that FICO used for FICO 8 is from 2006-2008 - it's only 2 years wide. They set the scorecards from that based on math and not entirely on someone's opinion. People with 10+ years of credit history obviously behaved quite differently from people just starting out.

Re: Scorecard reassignment and wonky score changes?

Anonymous — Wed, 25 Nov 2020 17:32:53 GMT

Right, but do you think it makes sense from a risk standpoint that someone on May 31 has a score of X then on June 1 their score drops to X-25 simply because an age of accounts metric increased by 1 month? It sounds like you've experienced this yourself, so I welcome your opinion on the subject.

SJ is probably right that in the grand scheme of things this sort of thing happens very rarely.

Re: Scorecard reassignment and wonky score changes?

Curious_George2 — Wed, 25 Nov 2020 18:08:30 GMT

@Anonymous:

Isn't the fundamental problem the existence of the scorecards (bins)? Seems like anytime one uses bins, there will be some number of sub-optimal risk calculations at the edges, when an individual moves from one bin to the next. Getting rid of the bins would eliminate that, but you say binning is necessary.

The factory example makes sense, but can you explain why binning is necessary in the consumer credit scoring context? Shouldn't the vast number of datapoints, even in a sample spanning just a few years, make it possible to dispense with bins? Whereas you had no information about machines like yours performing in that particular environment, it seems like FICO must have data about huge numbers of consumers with essentially identical profiles making any particular change (e.g. hitting 2 yrs AoOA).

Re: Scorecard reassignment and wonky score changes?

Anonymous — Wed, 25 Nov 2020 18:09:48 GMT

@Anonymous wrote:
Right, but do you think it makes sense from a risk standpoint that someone on May 31 has a score of X then on June 1 their score drops to X-25 simply because an age of accounts metric increased by 1 month? It sounds like you've experienced this yourself, so I welcome your opinion on the subject.

Yes, it absolutely makes sense when considering the various scorecard groupings. It's an objective analysis; subjectively it's not going to make sense to a lot of people.

In 5 days I should see my scores drop significantly due to that AoOA 3yr 0mo scorecard change, but in reality I'm no more of a credit risk than I was last month, and I'm sure that applies to most of the people who see those score changes. This is a subjective opinion, though.

There is certainly a push in the credit risk assessment space to be more subjective in evaluating a person's profile, using very personal information like near real-time bank account data, behavioral data gathered from use of mobile phones, and several other types of privacy-invasive alternative data. There likely wouldn't be any sudden change in scores for those models like there can be with the static datasets FICO has been using.

I'd rather just take this hit on December 1st than avoid it by being scored on what I bought recently or what time of night I picked up my phone to text someone.

SJ is probably right that in the grand scheme of things this sort of thing happens very rarely.

It happens to every single person that is just starting out with credit. I'm guessing these dramatic score shifts with no changes in profile data no longer occur after 5 years. The population of people closely monitoring their young credit profiles is relatively small on this forum, and even on Reddit forums where the demographics skew much more toward college students.

Re: Scorecard reassignment and wonky score changes?

Anonymous — Wed, 25 Nov 2020 18:50:09 GMT

@Curious_George2 wrote:
@Anonymous:

Isn't the fundamental problem the existence of the scorecards (bins)? Seems like anytime one uses bins, there will be some number of sub-optimal risk calculations at the edges, when an individual moves from one bin to the next. Getting rid of the bins would eliminate that, but you say binning is necessary.

The factory example makes sense, but can you explain why binning is necessary in the consumer credit scoring context? Shouldn't the vast number of datapoints, even in a sample spanning just a few years, make it possible to dispense with bins? Whereas you had no information about machines like yours performing in that particular environment, it seems like FICO must have data about huge numbers of consumers with essentially identical profiles making any particular change (e.g. hitting 2 yrs AoOA).

The binning is necessary because the number of 'bads' is going to be significantly different for those just starting out with credit compared with a group that's been managing credit for a much longer time. Like a new robot design performing in a factory for the first time compared to the old proven design (that probably used a lot more energy or took longer, but was incredibly reliable).

Some things just don't change with time, like a relatively high proportion of college kids getting their first credit card or auto loan and defaulting within a year or two. That's why a static profile dataset from 2006-2008 isn't such a bad thing for FICO 8.

But sometimes data from a certain timeframe can include too many in the bad category, which is why they update models from time to time, as shown here:

They cited changes in consumer behavior including:

Mortgage crisis and Great Recession
Decrease in consumers with new account openings
Consumers seeking credit less often

Re: Scorecard reassignment and wonky score changes?

Curious_George2 — Wed, 25 Nov 2020 19:28:58 GMT

But isn't that an answer to a different question? I see your example of risky new-to-credit borrowers as supporting the existence of a penalty for low AoOA. I don't see how that necessitates binning. The risk probably changes a little each month, rather than changing a lot at exactly, say, 24 months. So wouldn't the lenders (FICO's real customers) get a more accurate picture of the risk if the scoring system spread the removal of that particular penalty more smoothly over time?

Re: Scorecard reassignment and wonky score changes?

Anonymous — Wed, 25 Nov 2020 20:47:30 GMT

@Curious_George2 wrote:
But isn't that an answer to a different question? I see your example of risky new-to-credit borrowers as supporting the existence of a penalty for low AoOA. I don't see how that necessitates binning. The risk probably changes a little each month, rather than changing a lot at exactly, say, 24 months. So wouldn't the lenders (FICO's real customers) get a more accurate picture of the risk if the scoring system spread the removal of that particular penalty more smoothly over time?

It's not a penalty due to age alone, though. Penalties are only higher for certain characteristics - inquiries, new accounts, utilization, etc. - because the percentage of bads in that scorecard is higher than other scorecards. More bad profile targets means more of a chance to have something in common with some number of them, which will lower our scores.

These scorecards (bins) are set primarily from the already observed profile data over 2 years, and guided by data scientists to find the best predictors (Information Values and Weight of Evidence). It's not "People new to credit are inherently a higher risk", it's "The data we have already collected says that people new to credit are a higher risk". They can already see the performance over time for a person that started fresh and ended up defaulting within 2 years.

There was something special about that AoOA 3yr 0mo value in the dataset after which the percentage of bads was lower. Then there will be another dividing line where the percentage is even lower. Penalty weighting will be set accordingly. Some penalties are going to be high no matter what, since they are shared by most bads across all scorecards - lates, very high utilization, high number of inquiries.

(FICO says: "Statistically, people with six inquiries or more on their credit reports can be up to eight times more likely to declare bankruptcy than people with no inquiries on their reports." )

I know it doesn't seem logical to lose so many points merely from crossing a scorecard boundary. At least this way, they can compare apples to apples. And it's not going to happen that often anyway.

This form of logistic regression has been accurate enough for decades for most major lenders. Their relatively low net charge-off rate shows that. Of course, they would always welcome something that could lower that even more - like FICO 10 with trended data.

Re: Scorecard reassignment and wonky score changes?

Anonymous — Wed, 25 Nov 2020 21:50:54 GMT

Maybe AoOA of 3 years being a segmentation factor isn't the best idea then?

Re: Scorecard reassignment and wonky score changes?

Anonymous — Thu, 26 Nov 2020 01:16:02 GMT

@Anonymous wrote:
Maybe AoOA of 3 years being a segmentation factor isn't the best idea then?

They must have found a good set of predictor variables for the <3years group that weren't so great in the >3years group. They didn't make it up arbitrarily.

They can only predict defaults within a narrow window of time. It seems like 2 years is the maxmium look ahead that's reliable in credit risk modeling. I see that number a lot in FICO whitepapers, and they use that timeframe for model development profile data.

Because FICO says they need 6 months of data to generate a score in the first place, I think recovery from those hard drops when switching scorecards will be mostly complete within 6 months of history on the new scorecard. I'm definitely going to find out firsthand over the next 6 months.

Re: Scorecard reassignment and wonky score changes?

Curious_George2 — Thu, 26 Nov 2020 06:02:12 GMT

@Anonymous wrote:
It's not a penalty due to age alone, though. Penalties are only higher for certain characteristics - inquiries, new accounts, utilization, etc. - because the percentage of bads in that scorecard is higher than other scorecards. More bad profile targets means more of a chance to have something in common with some number of them, which will lower our scores.

These scorecards (bins) are set primarily from the already observed profile data over 2 years, and guided by data scientists to find the best predictors (Information Values and Weight of Evidence). It's not "People new to credit are inherently a higher risk", it's "The data we have already collected says that people new to credit are a higher risk". They can already see the performance over time for a person that started fresh and ended up defaulting within 2 years.
...

Thanks for your insightful (as always) comments. You make excellent points. I was still feeling a little unclear on why FICO uses scorecards and your reference to whitepapers inspired me to go looking for one. I found a whitepaper that makes a variety of arguments in favor of using scorecards, including "palatability" and "interpretability" of the inner workings of the scoring model by the human users. That makes sense to me, since each model is a product that FICO has to persuade lenders to buy. That would be harder if it were built entirely out of incomprehensibly complex mathematical formulas. Scorecard-based algorithms are much more PowerPoint friendly.

The whitepaper also says that the selection of the specific boundaries between bins (apparently they're called "knots") is as much art as science, and it is sometimes based on what the developer or user feels is a "nice" number. Interesting stuff!

Re: Scorecard reassignment and wonky score changes?

Anonymous — Thu, 26 Nov 2020 06:20:49 GMT

@Curious_George2 wrote:
@Anonymous wrote:
It's not a penalty due to age alone, though. Penalties are only higher for certain characteristics - inquiries, new accounts, utilization, etc. - because the percentage of bads in that scorecard is higher than other scorecards. More bad profile targets means more of a chance to have something in common with some number of them, which will lower our scores.

These scorecards (bins) are set primarily from the already observed profile data over 2 years, and guided by data scientists to find the best predictors (Information Values and Weight of Evidence). It's not "People new to credit are inherently a higher risk", it's "The data we have already collected says that people new to credit are a higher risk". They can already see the performance over time for a person that started fresh and ended up defaulting within 2 years.
...
Thanks for your insightful (as always) comments. You make excellent points. I was still feeling a little unclear on why FICO uses scorecards and your reference to whitepapers inspired me to go looking for one. I found a whitepaper that makes a variety of arguments in favor of using scorecards, including "palatability" and "interpretability" of the inner workings of the scoring model by the human users. That makes sense to me, since each model is a product that FICO has to persuade lenders to buy. That would be harder if it were built entirely out of incomprehensibly complex mathematical formulas. Scorecard-based algorithms are much more PowerPoint friendly.

The whitepaper also says that the selection of the specific boundaries between bins (apparently they're called "knots") is as much art as science, and it is sometimes based on what the developer or user feels is a "nice" number. Interesting stuff!

Oh cool, you read the FICO Model Builder paper. That's a great general overview.

Here's some more information from FICO about domain expertise: Superior Scorecards at Warp Speed: Algorithmic Learning Meets Domain Expertise

Re: Scorecard reassignment and wonky score changes?

Anonymous — Mon, 30 Nov 2020 06:51:09 GMT

@Anonymous wrote:
Looking to have a discussion on this topic, as I know there will be differing opinions. We know that score ranges for different scorecards can overlap in spots. As a result, someone that has a positive change happen to their file could result see a score drop. I understand that scorecards are based on groups of people with similar profile data and that we really shouldn't get hung up on any one individual example. For me though personally, I don't necessarily agree with how that works. If a Fico score is a representation of risk / likelihood of default, it doesn't make sense that a positive profile change would result in a lower score.

For example, someone reaches an age of accounts scorecard reassignment that results in a score drop. Looking at that individual alone, there's no way that they became more of a risk (lower score now) due to positively crossing an age of accounts threshold. Now that person lost (say) 20-30 points which could be the difference between X rate and Y rate on a new mortgage. That just seems counterproductive to me.

Anyway, I get it that it "is what it is" and how the system works when looking at groups of individuals and that it isn't going to be perfect at all times. Just curious to hear thoughts from others on this topic, whether you think it matters and/or if you believe this issue should be addressed. If so, what would you propose... such as no overlapping score cards, for example?

@Anonymous I have not yet read the responses to the thread so I preface my post with that.

in theory, I agree with you. It does create an issue that the score is not linear and is instead generated by placing profiles in groups/scorecards, according to logistic regression analysis. This is one of the imperfections because, as you pointed out, becoming a day older does not necessarily translate to an increase in risk.

I don't know how this could be fixed, as I'm not privy to how exactly the algorithm is designed and generated, but I would point out that crossing that age threshold does not always result in a decrease. It can result in an increase, depending on the individual profile, as you so eloquently stated.

The simple fact is the characteristics/metrics are weighted differently per scorecard and so by moving to a different group/scorecard , it causes a reassessment that can result in an increase or decrease, tho typically the example you provided results in a decrease.

I think this is an example of what was misguidedly called in one article instability, which can correct itself with a little time, a few to several months. So I think the general answer is that while the characteristic change may cause an abrupt score change, with a little bit of time, it should correct itself.

Re: Scorecard reassignment and wonky score changes?

Anonymous — Mon, 30 Nov 2020 07:11:20 GMT

Now that I’ve read the replies, I have a couple comments. First, regarding the segmentation point of 3 years for age of oldest account, keep in mind on the old versions that was 2 years.

Apparently due to analysis, that was moved to 3 years with version 8. Additionally, we must keep in mind that the objective of the algorithm is to predict the likelihood of a 90 day default within a 24 month period. That could be significant as well.

And Cassie‘s comments about it correcting itself within six months are illustrative of what I was speaking of regarding the temporary instability as it has been described that corrects itself in a short period of time.

Re: Scorecard reassignment and wonky score changes?

Remedios — Mon, 30 Nov 2020 14:49:11 GMT

@Anonymous please help me find one example of score increase when first account reached 3 years of age because I cannot (one where there is no doubt that's what caused increase)

Re: Scorecard reassignment and wonky score changes?

Anonymous — Mon, 30 Nov 2020 18:53:26 GMT

@RemediosOK let me do some searching and maybe even make a post, I will try to find one.

@Remedios Edit: I am hoping that Cassie is one due to the inquiry count and low utilization; it also may play a part that she had a loan first and therefore her score has not went as high because the age of her revolving account is not as high. Plus the new accounts are penalized less on a mature card, so that argues in her favor too.

Re: Scorecard reassignment and wonky score changes?

Remedios — Mon, 30 Nov 2020 18:52:22 GMT

Thank you

Re: Scorecard reassignment and wonky score changes?

Remedios — Mon, 30 Nov 2020 19:36:05 GMT

@Anonymous spare yourself from searching, you won't find an example of a gain.

As far as @Anonymous , I'd guess she's going into 740something to 750 on TU and EQ.

EX is already kinda leveled, so I wouldn't expect too much of a change there

Or, maybe she will stay roughly where she is. My scores were higher than Cassie's when I got sent to big girl scorecard, so I guess I had to "lose" more

EQ was my highest (788), and that's where the real drop happened, about 25 points, bringing it in line with the other two who were around 765, and they dropped to 751 (identical drop, data not identical due to number of HPs).

I had multiple accounts under a year, and it didn't make it any better. Before I could have up to 50% of accounts report a balance with miniscule drops, after reassignment, penalties for additional cards reporting balances were much higher.

That's just my experience, but @Anonymous profile and mine are nothing alike, I had over five stupid accounts before Fico was generated (no kidding), I've never experienced HP points losses like Cassie, and I cannot remember a life without new account penalty.

I should be free on January 1st. I want all the Ficos! All of them.

Re: Scorecard reassignment and wonky score changes?

SouthJamaica — Mon, 30 Nov 2020 22:56:00 GMT

@Remedios wrote:
@Anonymous spare yourself from searching, you won't find an example of a gain.

As far as @Anonymous , I'd guess she's going into 740something to 750 on TU and EQ.
EX is already kinda leveled, so I wouldn't expect too much of a change there
Or, maybe she will stay roughly where she is. My scores were higher than Cassie's when I got sent to big girl scorecard, so I guess I had to "lose" more
EQ was my highest (788), and that's where the real drop happened, about 25 points, bringing it in line with the other two who were around 765, and they dropped to 751 (identical drop, data not identical due to number of HPs).

I had multiple accounts under a year, and it didn't make it any better. Before I could have up to 50% of accounts report a balance with miniscule drops, after reassignment, penalties for additional cards reporting balances were much higher.
That's just my experience, but @Anonymous profile and mine are nothing alike, I had over five stupid accounts before Fico was generated (no kidding), I've never experienced HP points losses like Cassie, and I cannot remember a life without new account penalty.
I should be free on January 1st. I want all the Ficos! All of them.

When I tested 3 years AAoA I had a "gut feeling" that I'd gained 5 or 10 points out of it, but there was too much going on in my profile to isolate anything. https://ficoforums.myfico.com/t5/Understanding-FICO-Scoring/Is-3-years-a-threshold-in-AAoA/m-p/5135502

But Revelate felt that he was getting zero points out of it.

Most likely that was before I was getting the daily update at experian.com, which has helped my ability to isolate single factors.