AI vs Traditional Fraud Detection Systems: Which Works Better, With a Worked Example

Key takeaways

• Rules are simple, explainable and good for known patterns; models score many weak signals.
• In a synthetic test, a model beat a fixed rule at the same review budget, but real data differs.
• Fraud is rare, so accuracy misleads; use precision and recall.
• Most real systems combine rules, models and human review.

What are the two ways of spotting fraud?

Traditional systems use rules written by people: "block a card used in two countries within an hour", "hold any transfer above a set amount", "flag a new payee plus a large amount". They are simple to explain and audit, and they work well against fraud patterns you already know.

AI and machine learning systems learn from historical transactions labelled as fraud or genuine, and score each new transaction by how fraud-like it looks. They can combine many weak signals (amount, device, time, location, speed, past behaviour) that no single rule captures.

Side-by-side comparison

A worked comparison on synthetic data

To make the difference concrete, here is a small experiment. The data is synthetic: I generated 20,000 transactions with a fraud pattern built in, so the model has something real to learn. This shows how the two approaches behave, not how they perform in a real bank. Real data is messier, and fraud rates and patterns differ.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import precision_score, recall_score

rng = np.random.default_rng(42)
n = 20000
amount = np.exp(rng.normal(7.5, 1.0, n)) # rupees
night = rng.random(n) < 0.15 # late-night transaction
new_dev = rng.random(n) < 0.08 # first time this device is seen
foreign = rng.random(n) < 0.05 # card used abroad
velocity = rng.poisson(1.2, n) # transactions in last hour

logit = -6.8 + 1.0*np.log(amount/1800) + 2.2*new_dev + 1.8*foreign + 0.5*velocity + 1.2*night
fraud = rng.random(n) < 1/(1+np.exp(-logit))
print("fraud cases:", fraud.sum(), "of", n)

X = np.c_[np.log(amount), night, new_dev, foreign, velocity]
Xtr, Xte, ytr, yte, atr, ate, ntr, nte, ftr, fte = train_test_split(
 X, fraud, amount, new_dev, foreign, test_size=0.3, random_state=1)

rule = (ate > 20000) | (nte & fte) # fixed rule
model = LogisticRegression(class_weight="balanced", max_iter=1000).fit(Xtr, ytr)
score = model.predict_proba(Xte)[:, 1]
model_flag = score >= np.sort(score)[-rule.sum()] # flag the same number as the rule

for name, flag in [("rule", rule), ("model", model_flag)]:
 print(name, "flagged", flag.sum(),
 "precision", round(precision_score(yte, flag), 2),
 "recall", round(recall_score(yte, flag), 2))
print("overlap", int((rule & model_flag).sum()))

The code needs Python with NumPy and scikit-learn (see the Python documentation if you are new to running scripts). Output when run with the fixed seed in the code:

fraud cases: 218 of 20000
rule flagged 76 precision 0.13 recall 0.14
model flagged 76 precision 0.21 recall 0.23
overlap 27

Here the analyst team can review 76 transactions. The fixed rule (amount above Rs 20,000, or a new device used abroad) catches 14 percent of the fraud in the test set. The model, limited to the same 76 reviews, catches 23 percent and has better precision. Only 27 of the 76 flagged cases overlap, so the two methods find different fraud, which is a good argument for running both.

Things this example does not show: how the model degrades as fraudsters change tactics, how you would handle label delays (fraud is often confirmed weeks later), and fairness checks.

Why do most real systems use both?

• Rules for known, hard limits and compliance. Sanctions screening, regulatory limits and obvious patterns suit rules.
• Models for scoring risk. Use the score to decide whether to approve, step up authentication, or send to review.
• Humans for the grey zone. Analysts review high-risk cases and their decisions become new labels for the next model.

What are the common problems with AI fraud detection?

• Class imbalance. Fraud is rare, so accuracy is a misleading measure. A model that says "genuine" every time is over 98 percent accurate in the example above and catches nothing. Use precision, recall and cost of errors.
• Concept drift. Fraudsters adapt. Monitor performance and retrain.
• Label quality. Unreported or late-confirmed fraud poisons training data.
• Bias. A model can learn patterns that unfairly target groups. Test for it and keep human appeal routes.
• Adversarial behaviour. Fraudsters probe systems to learn what gets through.
• Customer friction. Every false positive annoys a real customer, so measure it.

How to choose

1. Small business with few transactions: start with a handful of well-chosen rules from your payment provider.
2. Growing volume and enough labelled fraud: add a model for scoring, and keep the rules.
3. Regulated lender or payments firm: plan governance, explanation and testing from the start.

In India, banks and payment firms work under Reserve Bank of India requirements on fraud risk management, and the details change, so check current RBI directions for your sector.

Next steps

To learn the machine learning behind this, see the Machine Learning course. Related reading: AI vs human analysts in fraud detection and can AI eliminate financial crime.