The Saturday Fraud Strategist
False Positives Masterclass Part 3: How to reduce FPs inside your system
9 min
Okay, so here’s the thing about reducing false positives. Most teams want to jump straight into tactics. Tune the rule. Adjust the threshold. Add an exemption. Move the weird edge cases into manual review. Fine. All of that might be useful. But honestly, if that is where you start, you are probably guessing.
And guessing in fraud prevention is not exactly my favorite operating model. Not because it never works. Sometimes it does. Which is almost worse, because then everyone gets confident. Not a good look.
In this episode, I continue the False Positives Masterclass by moving from measurement and bucketing into the part everyone actually wants to get to: fixing the parts of the system that are misbehaving. But the point is not just to reduce false positives. The point is to reduce false positives without creating a new fraud problem you only discover three weeks later when the losses mature and everyone starts quietly looking at the dashboard like it personally betrayed them.
This episode is about discipline. It is about manual review, fraud rules, fraud model precision, fraud model recall, shadow mode testing, data quality issues, and the uncomfortable but necessary question every fraud team eventually has to ask: is this rule actually helping, or have we just been emotionally attached to it since that one fraud spike in 2022?
What you’ll hear in this episode:
- Why reducing false positives should start with manual review, not instinct
- How to decide whether a fraud rule should be removed, downgraded, or improved
- Why fraud model precision and fraud model recall matter when rules catch fraud but hurt good users
- How to build exclusions without accidentally creating a back door for fraudsters
- Why shadow mode testing and challenger rules are essential before release
- How data quality issues can make otherwise reasonable fraud prevention logic misbehave
- Why fraud operations teams need to be pragmatic, not elegant, when the data is broken
You should listen to this episode if you:
- Own fraud rules, fraud detection rules, models, AI agents, or review flows
- Are trying to reduce false positives without increasing fraud losses
- Have a high false positive rate but are not sure which part of the system is causing it
- Need a more structured way to review manual review samples and top offenders
- Are dealing with corrupted data, noisy signals, or flows where fraud prevention logic keeps misfiring
Episode notes & key takeaways
Reducing false positives starts with looking, not guessing
The first uncomfortable point in this episode is also the simplest one: you cannot reason your way through a false positive problem from a dashboard alone. You have to look. Manually. At real events. I know, very glamorous. Everyone got into fraud operations so they could inspect 100 blocked cases and ask themselves why a rule exists. Honestly, though, this is where the useful work starts.
If a rule, model, or AI agent is producing a high number of false positives, the first step is to manually inspect a sample of what it blocked. In most cases, 50 to 100 events is enough to get directional clarity. You are not trying to create a dissertation. You are trying to answer practical questions:
- Is this rule actually as inaccurate as we think?
- Does this rule still deserve to exist?
- Should it still make automated decisions?
- Could it move into manual review instead?
- What would reduce false positives without materially increasing fraud losses?
The key is that every answer depends on your business. There is no universal “good enough” threshold. Some teams have manual review capacity. Some do not have manual review at all. Some rules are noisy but still necessary because the volume is too large to send to analysts. Annoying? Yes. But also reality.
Three paths for every misbehaving fraud rule
Once you review the evidence, most misbehaving fraud rules or models fall into one of three categories. This is where the work gets cleaner.
The first category is rules that should not exist at all. These are the low-hanging fruit. Usually they were created during a crisis, caught something once, and then stayed in production forever because nobody wanted to touch them. Very human. Very common. Also, not great. If the fraud coverage is negligible and the false positives are significant, removing the rule may be the safest and cleanest fix.
The second category is rules that should exist, but should not make automated decisions anymore. These are mid-accuracy rules. They catch meaningful fraud, but they do not do it reliably enough to justify auto-decline. If your fraud operations team has capacity, moving that logic into case management can preserve fraud coverage while reducing false positives. You put a human in the loop, and that human can spot pattern changes in real time.
The tradeoff is cost. Manual review is useful, but it is not free. You reduce false positives, but you increase operational workload. So you’ve got to ask yourself whether the compromise is actually worth it.
The third category is the most common and most interesting: rules that should exist, but need to be improved. These rules often have high recall and low precision. In plain English, they catch fraud, but they also hit too many good users. So the answer is not to delete them. The answer is to refine them.
False positive reduction is fraud detection in reverse
One of the more useful ways to think about reducing false positives is that it is basically fraud detection in reverse.
When you build a fraud rule, you usually start with confirmed fraud. You review the fraud cases, identify the pattern they share, compare that pattern against the good population, and then build logic that catches fraud without catching too many legitimate events.
To reduce false positives, you do the same thing with the opposite goal. You start with confirmed false positives, identify the patterns they share, compare those patterns against the fraud cases, and then build exclusions that separate good users from fraud without releasing too much risk.
This is where teams sometimes get into trouble. It is not enough to say, “A lot of false positives look like X.” Okay, useful. But do fraud cases also look like X? If yes, your exclusion is not an exclusion. It is a fraud invitation with better formatting.
That last step matters. A lot.
How to build exclusions without creating fraud back doors
The example in this episode is intentionally simple: decline if IP country does not match account country. It is a basic mismatch rule, and yes, real-world rules are often more complex. But not always as much as we like to pretend.
Suppose you manually review 100 events blocked by this rule. Thirty are fraud. Seventy are legitimate. Of those 70 legitimate events, 20 involve U.S. users whose IP addresses appear in Canada. That is a meaningful false positive pattern.
Now the next question is the important one: do your fraud cases also show that U.S.-account, Canadian-IP pattern? If not, you may have a clean separation. Maybe you have commuters. Maybe VPNs route through Canadian endpoints. Maybe your user base lives near the border. Whatever the cause, the false positive pattern is real, and the fraud overlap is low.
That gives you the basis for a safer exclusion:Decline if IP country does not match account country, unless IP country is Canada and account country is the U.S.
Or, depending on your business footprint and available features, maybe you broaden that into neighboring-country logic. The point is not the exact rule. The point is the method. The exclusion comes from the false positive population and is validated against the fraud population.
That is how you reduce false positives without accidentally creating a new entry point for fraudsters. Am I being too optimistic that everyone does this step? Probably.
Test every fraud logic change before release
Any time you change fraud prevention logic, even if it is “just” an exclusion, you are changing your risk posture. Treat it like a new rule release.
Shadow mode testing and challenger rules are your best friends here. Keep the current rule active, but run the refined version in parallel. Then measure the difference.
You want to know:
- How many additional events would the new version allow?
- How many of those events eventually mature into fraud?
- How many false positives would the change prevent?
- Does the new logic behave consistently over time?
If the results hold, you can deploy with more confidence. If they do not, adjust. It is not glamorous. It is not a big dramatic launch moment. It is just better fraud operations.
The monitoring window should depend on how quickly fraud matures in your environment. If you have time, wait long enough to compare fraud outcomes in the challenger version against the original. If you are under pressure, manually review random samples. Not perfect, but better than flying blind.
Data quality issues can make good logic look bad
Sometimes the problem is not the rule. It is the data feeding the rule.
This is frustrating because everyone wants the fix to be in the fraud logic. Change the rule, lower the score, add the exception, ship it. But if the underlying field is corrupted, missing, inconsistent, or broken in a specific flow, the rule may be behaving exactly as designed. The input is just bad.
Okay. Annoying, but useful to know.
If the data issue is under your control, the best long-term answer is to fix the data quality issue itself. That might mean repairing an API payload, updating an SDK version, fixing an internal integration, or working with product and engineering to close the gap. It may take days. It may take weeks. But once the data is fixed, the fraud rule may start behaving correctly again.
Also, you may fix other problems at the same time. Data issues rarely ruin just one thing. They usually wander around the system quietly making several things worse. Very efficient, in the worst possible way.
When the data cannot be fixed quickly, adjust the logic
Not every data quality issue is fixable in the near term. Sometimes it comes from an external partner. Sometimes it comes from a platform you do not control. Sometimes it comes from a system you technically do control, but the fix is sitting somewhere behind 14 roadmap priorities and one team that keeps saying “next quarter.” Not a good look, but familiar.
If you cannot fix the data soon, adjust the fraud logic for that flow. That might mean excluding the problematic flow from the rule, lowering the rule’s weight, changing the rule’s impact, or moving those cases into manual review.
This is where fraud prevention needs to be pragmatic rather than elegant. A clean system is nice. A working system is better. Sometimes the right answer is not the most theoretically satisfying one. Sometimes it is the one that keeps good users moving while still containing fraud risk.
Final takeaway:
Reducing false positives is not about making your system softer. It is about making it more precise.
That means reviewing the actual events, separating rules that should be removed from rules that should be downgraded or improved, building exclusions based on evidence, validating those exclusions against fraud cases, testing changes in shadow mode, and fixing data quality issues when the logic is not really the problem.
Anyway, the slightly uncomfortable takeaway is this: if you are reducing false positives without looking at the underlying cases, you are not tuning. You are guessing.
And maybe you get lucky. Maybe.
But in fraud prevention, luck is not really a control. It is more of a temporary condition.
Connect with Chen Zamir | LinkedIn
Host of The Saturday Fraud Strategist
Helping fintechs build smarter fraud defenses
Co-author of “The Fraud Fighter’s AI Playbook”
Not ready to stop the conversation about my, and hopefully your, favorite subject? Subscribe to The Saturday Fraud Strategist newsletter.
Episode transcript
Chen Zamir
00:03
When people talk about reducing false positives, they usually jump straight to tactics: tuning rules, adjusting thresholds, adding exemptions, redirecting edge cases to manual review, and so on. All of that is important, but it shouldn't be where you start. In the previous parts of this masterclass, we've covered the two pieces of work that most teams skip: how to measure your false positives, and how to map them into buckets by actor, by solution, by flow, by data quality, and by what is or isn't under your control. That's the groundwork. Now you're ready to tackle the main course, fixing the parts of the system that are misbehaving. But this is where we should approach the work with discipline. Otherwise, you'll end up making changes that don't matter, overlook changes that would have mattered, or worst of all, open the door for additional fraud without realizing it. So let's get down to it.
Chen Zamir
01:07
One of the best ways to ground yourself before making any change is to go back to manual review. Pick a rule, a model, or an AI agent that creates a high number of false positives, and manually inspect a sample of the events it blocked. 50 to 100 should be enough in most cases. What you're trying to answer are often very basic questions. Is this rule actually as inaccurate as we think? Do we still want this rule to exist at all? If it exists, should it still make automated decisions, or should it fit into manual review instead? And if it should remain automated, what would it take to reduce its false positives without materially increasing our fraud losses? Each of these questions is business specific. There is no universal threshold for good enough. Sometimes your fraud ops team can take on the additional workload. Sometimes you have no manual reviews at all as part of your process. And sometimes a rule needs to remain automated, even if it's noisy, because you simply cannot afford to manually review that volume. But you need clarity before you change anything. The worst case is thinking a rule is fine because it looks logical, only to discover under review that it's blocking overwhelmingly legitimate traffic.
Chen Zamir
02:43
You can't reason your way through these situations, you have to look. But most importantly, this isn't about validating that you're fixing a real problem. It's about discovering how to do it.
Chen Zamir
03:02
Once you've reviewed the evidence, almost every misbehaving solution falls into one of three categories. The first is that the rule shouldn't exist at all. This happens more often than teams admit. Many times it would catch only a small population, would not block much fraud, but would generate a disproportionate amount of false positives. Usually it exists only because someone added it during a crisis, and no one ever bothered to revisit it since then. Removing such rules feels uncomfortable the first few times you do it, but if the fraud they catch is negligible and the false positives are significant, turning them off is one of the cleanest, safest ways to improve your system. So this category has your low-hanging fruits. The second category is of rules that should exist, but no longer make automated decisions. This is common with mid accuracy logic that still captures meaningful fraud, but not reliably enough to block without human review. If you have manual review capacity, and if this particular logic tends to produce events your analysts are comfortable judging, then flagging it to case management instead of auto decline can be the perfect compromise. You preserve fraud coverage, reduce false positives, and put a human in the loop that can observe pattern changes in real time. The downside, of course, is that it's resource intense. You have fewer false positives, but at higher operational costs. And to be honest, I try as much as possible not to resort to it. The third category is of rules that should exist, but they need to be improved first.
Chen Zamir
05:06
This is the most interesting scenario, and it's also the most common one. In this case, the logic captures fraud effectively, meaning the recall is high, but at the cost of impacting many good users, meaning the precision is low. So you need to refine it. The question is how?
Chen Zamir
05:31
The process for improving a false positive heavy rule is exactly the same as when you design a new fraud rule, just in reverse. Think about it. When you design a fraud rule, you review a set of confirmed fraud cases, identify the pattern they share, compare that fraud pattern to the general good population, and build logic that captures the fraud without catching too many good events. Now, to reduce false positives, you do exactly the same thing, but with the opposite goal in mind. You review a set of confirmed false positive cases, identify the repeating patterns, compare those patterns to the fraud cases to make sure you know how to separate them, and finally build exclusions that capture that separation without releasing too much fraud. And this last step is crucial. It's not enough to say a lot of false positives look like X. You need to prove that your fraud cases don't also look the same. Otherwise, your exclusion will undermine your fraud detection. But here's the thing: if you've worked through the steps I outlined without skipping any, you already did this manual review while validating the false positive metrics. Let's take an example. Suppose you have a very standard mismatch rule: decline if IP country mismatch account country. Obviously, given a simplified scenario, but keep in mind that in reality, many rules are not that much more sophisticated than that. Now imagine you manually review a sample of 100 events this rule blocked. After tagging them, you notice 30 were fraud, 70 were legitimate, and out of those 70 legitimate events, 20 involved US users whose IP addresses appeared in Canada. 20 out of 70 is a meaningful pattern. Maybe your business has a large US-Canadian commuting demographic. Maybe VPNs for Canadian endpoints are common. Maybe part of your user base works close to the border. For whatever reason, this specific mismatch, US account and Canadian IP, seems to produce a lot of false positives. If in your fraud cases you see little to no fraud coming from the same US-Canada pattern,
Chen Zamir
07:59
Then you have a clean separation. That means you can safely introduce an exclusion. Decline if IP country mismatch account country and not IP country equals Canada and account country
Chen Zamir
08:18
Equals US. Or better yet, broaden it slightly to include general neighboring country logic, depending on your business footprint and available features. The point is that the exclusion is derived from reviewing your false positive population, and is validated against your fraud population. This is how you refine rules without accidentally creating backdoors for fraudsters to exploit.
Chen Zamir
09:10
Anytime you change a fraud logic, even if it's just an exclusion on an existing rule, you're changing your posture. That's the same as releasing a new rule completely from scratch. So before you release the new version, test it. Shadow mode and challenger rules are your best friends here.
Chen Zamir
09:36
Keep the existing rule active, but run your refined rule in parallel. Measure the differences. How many additional events would it have allowed to go through? How many of those events matured into fraud? How many false positives would it have prevented? If the results hold over time, you can deploy with confidence. If not, adjust. How much time should you monitor these changes? Ideally, enough to gain a good measure of how fast fraud matures in the new version, and whether that's in line or not with the original version. And if you're really pressed for time, you may want to consider manually reviewing random samples.
Chen Zamir
10:25
As we explored in part two of the masterclass, sometimes the root cause of false positives is not faulty logic, but corrupted data. Best case, you've already identified it in your groundwork analysis. Worst case, you went through all of the motions just to figure it out on your detailed manual review. When this is the identified root cause, you have two options, depending on whether the data issue is fixable in the near term. Option one is to fix the data quality issue itself. If the corrupted field is under your control, be it your SDK, your internal integration, or your infrastructure, then the best long-term solution is to repair the data. That might mean fixing an API payload, adjusting an SDK version, or working with a product team to plug a hole. It may take a few days or a few weeks, depending on your organization. But once the data is fixed, the rule will often start behaving correctly again. Also, it's likely that you also solved a whole bunch of other issues that were plaguing other solutions at the same time. Your second option is to adjust the fraud logic for that flow. Not all data issues are internally fixable. Sometimes they originate from external parties that you just cannot control. Or, in the most frustrating cases, you do control these issues, but there's no fix in sight. If you cannot fix the underlying data soon, then you may need to adjust your fraud logic specifically for that flow. You have a couple of choices here. Either exclude the problematic flow from the rule altogether, lower the rule's weight or change its impact, or move those cases to manual review.
Chen Zamir
12:19
Whatever you do, just keep in mind that fraud prevention should be more pragmatic than elegant. And sometimes it's not about being right, it's about being smart.
View full transcript
