How to build a machine learning fraud model is one of the most frequent questions we get as we work with our customers. Because it’s such an enormous topic—and can have a big impact on your fraud prevention efforts—we wanted to speak with someone who’s doing machine learning right.
Dustin MacDonald, the senior director of fraud prevention for Momentum Travel, was generous enough to join us for a webinar about why he and his team decided to build a machine learning model, how they implemented it, and how they’ve iterated on it.
Preventing fraud at Momentum Travel
Momentum Travel is an online travel agency headquartered out of Montreal, Canada. They have two sites, justfly.com (in the U.S.), and flighthub.com (in Canada). The fraud team is based in Bellevue, WA, with team members in Montreal and Gurgaon, India.
Dustin and his team have many travel agency-specific challenges. All goods are completely digital, and travel (and travel purchases) happen round the clock. In addition, the airline agency model is particularly at risk to fraud because the margins on ticket sales are small, yet in the case of a chargeback the agency is on the hook for the entire price of the ticket.
The last challenge is that Momentum Travel is a growing company, which means Dustin needed to build a fraud program that can scale with the company. He thought machine learning might hold the answer to increasing the accuracy of their fraud prevention efforts, as well as helping the team scale as the company grows.
Building and testing the model
Momentum Travel’s data scientist, Yiwei Cai, joined the webinar to talk about the basics of building and testing a machine learning model.
Yiwei went into detail on how to ensure data integrity, how to cross validate the training set, and how to choose the metrics that match your business’ KPI goals. Because it’s so important to understand how the model is behaving before you launch it into production, he also spoke at length about how to test the model offline to ensure that it’s working the way it should once it goes online.
In fraud, one of the big challenges in training a machine learning model is that your target — fraudsters — is hopefully a very small part of your sample data. Yiwei also discussed the different techniques to sample existing data in order to get the clean training data set you need to fine-tune the model.
Launching the model
After building and training the model, Dustin and his team started A/B testing. They began by having a small percentage of the bookings be evaluated by the model, then monitored that on a daily basis against results from the legacy review system. This allowed them to assess the performance of the machine learning model and see where it conflicted with the legacy model.
This also helped them fine-tune the model. For example, Dustin and his team quickly realized that the model wasn’t taking into account the value of the ticket. From a business standpoint, they need to be most careful with the highest-value tickets. This led them to put more weight on the value of the ticket and still manually review those transactions.
So far, the machine learning model at Momentum Travel has achieved great results:
Decrease in manual review rate: The biggest result Dustin and his team have seen since launching the model is a decrease in the manual review rate. That’s great news for their goal of scalability.
More accurate auto-rejection: They’re also seeing fewer false positives in the number of transactions that are auto rejected, which keeps their customers happy.
Better fraud chargeback control: The chargeback rates with their legacy system were already pretty good, but the machine learning model has improved that number, saving them money.
Better analyst QA response: One of the more interesting ways that Dustin’s team has used the data is to surface manual review errors. If the model was showing a low probability of fraud but the agent still cancelled the booking, they were able to get feedback to the operations team to understand why they were being cancelled incorrectly.
Move from quantity to quality: Through reducing the manual review rate, Dustin’s team is able to focus more on accuracy and making the right tags on the data, rather than being overwhelmed by the review queue.
A high-performing machine learning model is about continuous improvement, and Momentum Travel is focusing on feeding clean data back into the system in order to get even more lift through retraining. They’re also looking at ways to create new variables (such as lead time + dollar amount), as well as sourcing data from third parties like Ekata in order to improve the model.
As Dustin and his team dig more into the machine learning model, they’re also hoping that it can help them surface fraud trends and other anomalies in the data in order to get even more ahead of fraudsters.
Watch the webinar to hear more about these topics in detail, as well as to learn how Ekata data and Confidence Score can work with your machine learning model.