The Order of Score Factors can be Unreliable

[Curious_George2]

I was afraid you would say that this was the source for your archived data. Ouch. 

I agree that the Scores presentation seems accurate, as it always matches Experian's CMS. I don't know if this will be helpful, but I think I have figured out that the screwy order of the factors on the Reports is not exactly random, but the ordering is the result of some questionable design choices colliding with random coincidences. I don't think I will explain it well, but I'll do my best.

Let's start with the "Compare" view of the factors, which shows all factors for all three bureaus for a related trio of scores (e.g. the 9s, the AU8s, Mortgage). Here's mine for Mortgage scores:

EQ is always shown on the left (i.e. first), TU is in the middle (second) and EX is on the right (third). Notice how the green checkmarks are arranged in a waterfall pattern, flowing from the top-left corner to the bottom-right. The MF web people chose to list the EQ factors first, in descending order of significance.

Next, any of the EQ factors that are also among the top four for TU or EX are given checkmarks in those respective columns. Let's call these the EQ/TU shared factors and the EQ/EX shared factors, respectively. 

Next, any factors in TU's top four that weren't already represented among the EQ/TU shared factors are listed, perhaps in descending order of significance, and given checkmarks in the TU column. Following the pattern, any TU/EX shared factors among these get checkmarks in the EX column. 

Finally, any factors in the EX top four that haven't already been represented on EQ or TU are listed, perhaps in descending order of significance, and given checkmarks.

Ok, fine, no big deal. It's just a comparison chart, right? Nope. It's more than that. It appears to me that in every case, the order in which a bureau's factors appear on this comparison chart is preserved as the order for their display on that bureau's own Factors page. That's a convoluted sentence, and you almost have to click around and see it for yourself to understand what I mean. It might be easier to see the pattern of you open two windows side-by-side, one showing the Compare view, the other showing just one bureau (EX is the most dramatic). Like this:

I think the upshot is that EQ factors are displayed in a consistent, meaningful order. TU's have some amount of disorder on Reports because of the randomness of some factors being EQ/TU shared factors and some not. EX on Reports has the most disorder because it is victimized by two layers of randomness: EQ/EX shared factors and TU/EX shared factors, and whichever factors happen to be EX-only get tacked on at the end.

[Revelate]

---

@Anonymous wrote:

 I'm not going for it. There's too much official fico documentation that says it's listed in the order of precedence. 

I will grant that version 8 codes have always been poor, but they should not be coming up out of order. If official documentation is worthless, then everything else we've learned, we may as well throw out as well.

I think just like you get an uninformed CSR every now and then, well I think you got someone uninformed here. I mean look what happened when you brought it to the supervisors attention, they immediately knew it was wrong. 

@Revelate MF is apparently saying code order doesn't matter. Any suggestions?

---

Glitch as others have stated.  That looks like a software bug to me honestly and MF service should be better than that.  Agreed that there's overwhelming documentation from FICO and their official disclosures that state it's in order.  I've never been 100% happy with the FICO Consumer presentation TBH, but this is a mistake on any level and I do not accept that the comment that just because they're the same ones in different order, means they're both equally as accurate.  They are not.

[Anonymous]

---

@Curious_George2 wrote:

I was afraid you would say that this was the source for your archived data. Ouch. 

It's not all bad. Even if the reason statements weren't in the correct order they did shift each month in a logical and consistent way.

Such as 'Short revolving history' shifting down a spot every 3 months until I switched scorecards on some AoOA boundary, like 2yrs and 3yrs. Then it would shoot back to #2 and continue the same downward trend.

Ok, fine, no big deal. It's just a comparison chart, right? Nope. It's more than that. It appears to me that in every case, the order in which a bureau's factors appear on this comparison chart is preserved as the order for their display on that bureau's own Factors page. That's a convoluted sentence, and you almost have to click around and see it for yourself to understand what I mean.

It looks like a multiset union gone wrong to me:  EQ ⊎ TU ⊎ EX

Your deduction seems reasonable to me.

Since it's so, so easy to get all this right in design/programming the first time, it really makes me wonder if all these various CMS front-end errors are intentional.

I just pulled a myEquifax.com free report and they consistently get various aging values wrong due to using the exact account opening dates without resetting them to the 1st before using the inbuilt date-diff functions of the language they're using.

[Curious_George2]

Do you have an aged Experian CMS subscription that can serve as a secondary source for old factor orders?

It's really unfortunate that the accurate depiction MF provides in the Scores tab is the ephemeral one, and the persistent one in Reports is corrupted.

I haven't seen any evidence that the EQ factors are out of order here, so one-third of your data is still OK.

To @Revelate and @Anonymous and any other skeptics, I would say: you don't have to take my word for any of this, and it's not a question of whether one accepts the statements of the customer support person. You can check this yourself. Look at the order of EX factors on any MF 3B Report you have available. Are they listed in the same order on the Compare view and the Experian-only view? I bet they are, 100% of the time. Then, if possible, compare that factor order against Experian's CMS for the same date. If they're the same on the F8 score, check the other scores. I expect you won't have to check many to find a discrepancy, though perhaps cleaner profiles with fewer overall factors invoked will mask the problem to some degree.

[Anonymous]

---

@Curious_George2 wrote:

Do you have an aged Experian CMS subscription that can serve as a secondary source for old factor orders?

Yes, all the way back to 12/23/2018 when I got a second credit card 3 days after getting the first one I ever had.

But Experian's FICO 8 score factors are even worse than unreliable. lol It would usually only show 2 or 3 even in the first month of my entire revolving credit history.

But the all important EX 2 factors should be fine.

It's really unfortunate that the accurate depiction MF provides in the Scores tab is the ephemeral one, and the persistent one in Reports is corrupted.

I haven't seen any evidence that the EQ factors are out of order here, so one-third of your data is still OK.

To @Revelate and @Anonymous and any other skeptics, I would say: you don't have to take my word for any of this, and it's not a question of whether one accepts the statements of the customer support person. You can check this yourself. Look at the order of EX factors on any MF 3B Report you have available. Are they listed in the same order on the Compare view and the Experian-only view? I bet they are, 100% of the time. Then, if possible, compare that factor order against Experian's CMS for the same date. If they're the same on the F8 score, check the other scores. I expect you won't have to check many to find a discrepancy, though perhaps cleaner profiles with fewer overall factors invoked will mask the problem to some degree.

---

I remember Birdman7 and I discussed these differences in score factors between myFICO's 3B report and Experian's CMS. We both saw it at one point.

It turns out Experian was getting it right and the so-called Golden Source of Credit Score Information was getting it wrong, at least in the 3B section of the myFICO website.

I have noticed that FICO 9 Classic and Industry score factors were always in alignment across all 3 CRAs. This makes sense since one of the selling points of that model was closer alignment on the numbers with the same profile data.

They sure have made it difficult to pinpoint the exact effect of various characteristic/attribute changes. Many times that fuzzy logic is good enough, as it is for all credit scoring models, since they can't predict future default with 100% certainty either.

[Revelate]

FICO 8 Classic reason codes suck.  Period.

I'm just going to leave it at that, frankly I think the various CMS's do a disservice to the American consumer with how blase they are about it.

[Anonymous]

@Curious_George2 oh I'm not a skeptic about it occurring, I'm simply saying it's unacceptable. Fico documentation plainly establishes the order is significant. Therefore any comments to the contrary I am in disagreement with.

I think you have figured out the ordering. And I don't doubt that they're out of order and I don't doubt that you figured out the way they are displaying them. But the simple fact is, it is inaccurate information. The order should be correct. Especially with the money that we pay for this this isn't a cheap service it's a premium service.

And if it's going to be out of order, then there should be a disclaimer stating so. It's unacceptable for them to provide it out of order and it's unacceptable for them to act like it's not a big deal when it is. Jmho. 

[Anonymous]

---

@Curious_George2 wrote:

Do you have an aged Experian CMS subscription that can serve as a secondary source for old factor orders?

It's really unfortunate that the accurate depiction MF provides in the Scores tab is the ephemeral one, and the persistent one in Reports is corrupted.

I haven't seen any evidence that the EQ factors are out of order here, so one-third of your data is still OK.

To @Revelate and @Anonymous and any other skeptics, I would say: you don't have to take my word for any of this, and it's not a question of whether one accepts the statements of the customer support person. You can check this yourself. Look at the order of EX factors on any MF 3B Report you have available. Are they listed in the same order on the Compare view and the Experian-only view? I bet they are, 100% of the time. Then, if possible, compare that factor order against Experian's CMS for the same date. If they're the same on the F8 score, check the other scores. I expect you won't have to check many to find a discrepancy, though perhaps cleaner profiles with fewer overall factors invoked will mask the problem to some degree.

---

@Curious_George2 I did not realize any part of the report was ephemeral. I'm disappointed to learn that, are there any other parts of the report that are ephemeral?

[Curious_George2]

---

@Anonymous wrote:

@Curious_George2 oh I'm not a skeptic about it occurring, I'm simply saying it's unacceptable. Fico documentation plainly establishes the order is significant. Therefore any comments to the contrary I am in disagreement with.

I think you have figured out the ordering. And I don't doubt that they're out of order and I don't doubt that you figured out the way they are displaying them. But the simple fact is, it is inaccurate information. The order should be correct. Especially with the money that we pay for this this isn't a cheap service it's a premium service.

And if it's going to be out of order, then there should be a disclaimer stating so. It's unacceptable for them to provide it out of order and it's unacceptable for them to act like it's not a big deal when it is. Jmho. 

---

Oh. Gotcha. In that case, we're on the same page. 

If you can give me a few of the best (i.e. clear and official) statements of the rule that order is supposed to be meaningful, I can write back to the customer support person to initiate a complaint that they are failing to follow their own rules. Or do you think posting it in the Tech Support forum here would be the better way to proceed? Could do both, I guess, but I don't want to get banned for being a troublemaker. 

[Curious_George2]

---

@Anonymous wrote:

---

@Curious_George2 wrote:

Do you have an aged Experian CMS subscription that can serve as a secondary source for old factor orders?

It's really unfortunate that the accurate depiction MF provides in the Scores tab is the ephemeral one, and the persistent one in Reports is corrupted.

I haven't seen any evidence that the EQ factors are out of order here, so one-third of your data is still OK.

To @Revelate and @Anonymous and any other skeptics, I would say: you don't have to take my word for any of this, and it's not a question of whether one accepts the statements of the customer support person. You can check this yourself. Look at the order of EX factors on any MF 3B Report you have available. Are they listed in the same order on the Compare view and the Experian-only view? I bet they are, 100% of the time. Then, if possible, compare that factor order against Experian's CMS for the same date. If they're the same on the F8 score, check the other scores. I expect you won't have to check many to find a discrepancy, though perhaps cleaner profiles with fewer overall factors invoked will mask the problem to some degree.

---

@Curious_George2 I did not realize any part of the report was ephemeral. I'm disappointed to learn that, are there any other parts of the report that are ephemeral?

---

I'm certainly not the most experienced MF Premium subscriber out there, but my impression from clicking around is that Scores only shows the one most recent score available. I don't see a way to change the date there. 

The only similar thing I can think of is when there's an Alert that also announces a changed F8 score, the Scores tab on mobile (not desktop) will show all of the top-four negative factors for that bureau's F8 score, but only until the next Alert for that bureau. If you're attentive, you can screenshot them before they vanish.