Only works on reported balances, if those $500 in payments haven't reported yet then you'd get that notification.
As for the 25 point drop, hard to say without seeing the before and after file: number of revolvers (or tradelines, maybe) with balances might produce a ding but typically wouldn't be that large alone.
I may have exaggerated a bit and meant this as tongue-in-cheek, but this seems like what I'm seeing lately 
@vertekal wrote:Me: *Pays $500 on various credit cards*
MyFico: ........
Me: *Uses credit card to buy a soda from a machine for $1.50*
MyFico: The balance on all of your credit accounts has increased by $1.50. Score has decreased 25 points.
$1.50 for a can of soda?!?! File a dispute! 
@vertekalI may have exaggerated a bit and meant this as tongue-in-cheek, but this seems like what I'm seeing lately
You're making the common mistake that many make on this forum every day of thinking that an alert you receive from MF is related to the score change you're seeing. Many times they are competely unrelated and a different event completely was the cause of the score change.
Examples:
1 - Rupert has a 700 credit score and has only 1 major negative item on his CR that's just under 7 years old. He applies for a credit card which results in an inquiry, to which he receives an alert. At the time he's provided with a new credit score. His score is now 778! He very wrongly assumes that the inquiry resulted in a 78 point gain. Upon inspection of his CR however, he sees that the 7 year old lone baddie had fallen off.
2 - Cornelius has a 700 credit score with his oldest account being 15 years in age. He closed this account nearly 10 years ago. He pays down the balance on one of his credit cards a few hundred dollars and receives an alert of the new (lower) reported balance and his credit score provided at the time is now 682. Upon inspection of his credit report, he sees that the 15 year old account has dropped off and that his now oldest account is only 8 years in age. The reason for the score drop here was the AoOA reduction, not the balance paydown of course.
The bottom line here is that any time you see a score shift, be it up or down, it's important to look at your CR before/after to determine the change(s). Very often something else has changed that wasn't indicated by an alert.
OP your Siggy is three years old. What are your current scores?
