How I Built a Self-Learning Fraud Detection System for Shopify

Fraud is one of those problems that every Shopify merchant knows about but most handle poorly. You either rely on Shopify's built-in fraud analysis — which is decent for obvious cases but misses sophisticated fraud — or you bolt on a third-party tool that flags so many legitimate orders that your team spends hours every day manually reviewing them. Neither approach scales. I got tired of both, so I built my own.

What I wanted was simple in concept: scan every order that comes in, analyze it with AI, give it a fraud grade, and automatically handle the risky ones — quarantine, customer outreach, and cancellation if needed. No manual review queue for my team to babysit. And critically, I wanted a system that gets better over time as it processes more data. After a few months of building and iterating, that's exactly what I have running in production.

The problem with existing fraud tools

I've used most of the major fraud platforms over the years. They all share the same fundamental tension: set the sensitivity too high and you flag legitimate customers, creating friction and lost sales. Set it too low and fraudulent orders slip through, costing you chargebacks, lost inventory, and fees.

Shopify's native fraud analysis gives you a risk level — low, medium, or high — based on signals like AVS match, IP geolocation, and order velocity. It's a good starting point, but it doesn't learn from your specific business patterns. A first-time customer placing a large order from a new IP address might be perfectly legitimate for my store — we sell high-end products, so large orders from new customers are normal. But the generic fraud model doesn't know that.

Third-party tools like NoFraud, Signifyd, and Riskified are more sophisticated, but they're expensive, they operate as black boxes, and their accuracy on niche or high-AOV stores is often mediocre. I was paying for a service that still required my team to review a stack of flagged orders every morning. The tool was supposed to save time, but it was just shifting the work from "check for fraud" to "check the fraud tool's homework."

What I really needed was a system that understood my business, combined signals from multiple approaches, and handled the entire workflow automatically — not just flagging, but acting.

Architecture: webhook to database to AI to action

The system has four components, and they're intentionally simple. Complexity is the enemy of reliability when you're processing every order that hits your store.

Order webhook. When a new order is created in Shopify, a webhook fires and sends the full order payload to a serverless function. This function extracts the relevant data points: customer info, shipping and billing addresses, IP address, order total, line items, payment method details, discount codes used, and the customer's order history.

Database write. Every order gets written to a Supabase database with all extracted fields, plus computed fields like distance between billing and shipping addresses (using Haversine), whether the email domain is disposable, whether the customer has ordered before, and the time between account creation and first order. This database becomes the foundation for both real-time analysis and historical pattern recognition.

AI analysis and grading. This is the core of the system. The order data is sent to an AI model along with a carefully engineered prompt that includes the grading rubric, recent fraud patterns from the database, and the customer's full history. The AI returns a structured fraud grade — a letter grade from A (clearly legitimate) through F (almost certainly fraudulent) — along with a confidence score and a written explanation of its reasoning.

Automated action. Based on the grade, the system takes action. A and B grades pass through with no intervention. C grades get logged for periodic review but aren't held. D grades trigger quarantine — the order is tagged, fulfillment is held, and the customer receives an outreach email asking them to verify their identity. F grades are quarantined immediately with a shorter response window. If a quarantined customer doesn't respond within the configured timeframe, the order is automatically canceled and refunded.

The grading system: combining signals that matter

The grading system is where this tool differentiates itself from off-the-shelf solutions. Instead of relying on a single fraud model, I combine signals from multiple approaches into a composite grade. Each signal contributes a weighted score, and the weights adjust based on historical accuracy.

Address signals. Does the billing address match the shipping address? If not, how far apart are they? Is the shipping address a known freight forwarder? Has this address appeared on previous fraudulent orders? Is it a residential or commercial address?

Customer behavior signals. Is this a new customer or a returning one? If returning, what's their order history? Did they create their account moments before placing the order? Are they using a disposable email provider? How does their order value compare to their typical purchase?

Payment signals. Did the AVS check pass? What about CVV verification? Is the payment method consistent with the customer's previous orders? Are they using a prepaid card or a virtual card number?

Order pattern signals. Is this order unusually large? Does it contain only high-resale-value items? Was a discount code used, and if so, was it a publicly leaked code? Does the combination of items make sense for a real customer, or does it look like someone filling a cart with whatever they can flip?

Velocity signals. How many orders has this IP address placed in the last 24 hours? How many different customer accounts are associated with this billing address? Are there multiple failed payment attempts preceding this successful one?

The AI model evaluates all of these signals together, weighted by their historical reliability for my specific store. A signal that's highly predictive for my business might be irrelevant for another — and the system knows the difference because it's learned from my data.

The quarantine flow

Quarantine is where the system goes from "fraud detection" to "fraud resolution." When an order receives a D or F grade, here's what happens automatically:

Step 1: Hold fulfillment. The order is tagged in Shopify and fulfillment is paused. No product ships until the order is cleared. This happens within seconds of the order being placed.

Step 2: Customer outreach. The customer receives a professional email explaining that we need to verify some details before their order can ship. The tone is careful — it doesn't accuse anyone of fraud. It simply asks the customer to confirm their shipping address and respond to the email. For D-grade orders, the customer gets 48 hours to respond. For F-grade orders, 24 hours.

Step 3: Wait and monitor. The system monitors for a response. If the customer replies and confirms their details, the order is moved to manual review for a quick human check before being released. Legitimate customers respond quickly and without issue — I've found that the vast majority of D-grade customers are real buyers who had some anomalous signal (new address, large order, etc.).

Step 4: Auto-cancel if no response. If the customer doesn't respond within the configured window, the order is automatically canceled and a refund is issued. The customer receives a cancellation notice. The order data is flagged in the database as "canceled — no verification response," which feeds back into the learning system.

This flow catches fraudulent orders before they ship, while giving legitimate customers a simple path to verify themselves. It's not perfect — nothing is — but it's dramatically better than either ignoring fraud or manually reviewing every flagged order.

Self-learning: how the system gets smarter

This is the part I'm most proud of. The system doesn't just process orders — it learns from the outcomes.

Every order that flows through the system eventually gets a resolution: it either ships successfully and no chargeback occurs (legitimate), or it results in a chargeback, a cancellation due to no verification response, or a manual fraud flag by my team. These outcomes are stored alongside the original order data and the fraud grade the system assigned.

Periodically, I run a feedback loop that analyzes the system's accuracy. It looks at every order graded in the past 90 days and compares the predicted grade against the actual outcome. Did a B-grade order turn into a chargeback? That's a miss — the system should have flagged it. Did a D-grade order turn out to be completely legitimate? That's a false positive — the system was too aggressive.

These accuracy metrics feed back into the model in two ways. First, the signal weights are adjusted. If shipping-billing address distance turns out to be a weak predictor for my store (because many of my customers legitimately ship gifts to different addresses), its weight decreases. If disposable email domains are highly correlated with chargebacks, that weight increases. Second, the AI prompt is updated with recent examples of both misses and false positives, giving it better calibration for borderline cases.

The result is a system that was reasonably accurate when I first deployed it and has become significantly more accurate over time. In the first month, the false positive rate was around 12% — about 1 in 8 quarantined orders turned out to be legitimate. After six months of learning, that rate dropped to under 3%. The miss rate — fraudulent orders that weren't caught — has also improved, though that's harder to measure precisely since some fraud isn't discovered until weeks later.

Results and impact

The numbers tell the story. Since deploying this system, chargebacks at my store have dropped by over 80%. The handful that still occur are sophisticated social engineering attacks that no automated system would catch — situations where a real person uses their real identity to commit fraud. Those require human judgment, and they always will.

More importantly, the system handles the entire workflow without my team touching it. We used to spend 30–45 minutes every morning reviewing flagged orders. Now we spend zero. The system either passes the order, gets verification from the customer, or cancels it — all automatically. My team gets a daily summary in Slack showing what was processed, what was quarantined, and what was canceled. They only intervene if something unusual shows up in the summary.

The cost savings go beyond chargebacks. Every prevented fraudulent order saves the product cost, shipping cost, chargeback fee, and the time your team spends on the dispute process. For a store doing significant volume, those savings add up fast.

If you're running a Shopify store and fraud is a persistent problem — or if you're paying for a fraud tool that still requires constant manual review — a custom solution like this is worth considering. The architecture isn't complicated, the AI capabilities are there, and the self-learning aspect means the system only gets better the longer it runs. Off-the-shelf tools treat every store the same. Your fraud patterns are unique to your business, and your fraud detection should be too.