Why "Change" Pushes Model Performance Improvements

At Ekata, we are constantly looking for ways to improve and innovate on our solutions, infrastructure, and data science. Over the past several years, we’ve continued to push the boundaries of identity verification data and, more importantly, machine learning to build scale, stability, and flexibility in our fraud prevention solutions. Machine learning has become pervasive in applications and technology that we use throughout our everyday lives. However, it’s also a practice not fully understood at the deepest level for a large portion of us. Many of us know enough to be dangerous in either its use or assumptions of how it should work, but don’t truly understand the intricacies of it. With 20 years of expertise in data sourcing and data science, our team of product managers, engineers, and data scientists are always teaching and training our internal and external audiences on the truth and power of machine learning and more specifically, our models. They continuously investigate and assess how to push our models and their performance for the benefit of the entire customer base. To help provide some clarity, we wanted to share with you some of the most helpful insights to understanding models but more importantly, the Ekata scoring models that feed the Confidence Score and our newly released Pro Insight solution.
1. Models are living, breathing, and learning.
Models are meant to adapt; they are not meant to be static. That means by nature, there is inevitably “change.” They are only as smart as the data that feeds them and, of course, the smart folks building the algorithms and turning the dials, but without breadth and depth of quality data, it’s garbage in, garbage out— no matter who your data scientists are. Luckily for Ekata, quality identity data and top-notch talent are standard practice. We have access to millions of transactions to evaluate our models and always look at ways data and science can move the needle.
2. The value of a truth / training set
A key set of data inputs absolutely vital to the models’ success is the actual “truth or training set”. What does that mean? Simply put, the actual outcomes of transactions are key to helping the model differentiate between good transactions versus the fraudulent ones. Outcome data however, is not real-time, so usual feedback loops take 30-90 days post the transaction before we see them. The larger and more diverse the amount of outcome data we have the better the model can get at predicting fraud. So as fraud trends change, so can the model to best predict them.
3. One global model
As an identity verification company, we are not in the business of building individual models for each customer. We have a domestic and international model built for our global solutions to provide lift to the entire customer base across Pro Insight and our APIs. This means we measure performance in overall improvements in AUC, score distribution, and our newly released positive and negative insights found in our new Pro Insight solution. While we fine tune the model to some extent based on the industry of our customer, we do not have separate models for individual customers. Our customers benefit from the whole of the conglomerate when using our models.
4. Patterns and predictions
When the models train and learn, they see patterns that help them bucket what is most likely fraudulent versus what is good, and then everything in between. We assess this performance by looking at overall score distribution and trying to fit it into a uniform distribution. We are often asked how much this distribution shifts as the model gets smarter. The answer to this question is it depends. If you have a new model, as it ingests more data, especially outcome data, and continues to get smarter, you can see larger shifts in the score distribution. This is great because the model is getting better at predicting more fraudulent actors; therefore, it will start to catch more fraud and bucket those in the higher score ranges. When you have a more mature model, the shifts are far more subtle, because the model is incrementally improving AUC and already bucketing the fraudulent transactions and the good transactions at a highly accurate rate. However, the shift in score distribution is also a positive here because the model is always getting smarter and therefore catching more fraud!
5. Our score in your rules
We have thousands of customers that rely on our solutions to fight fraud and verify good transactions. So, what’s the impact of leveraging machine learning in a workflow that often incorporates a variety of data for intricate rule writing? Machine learning via the way of a score, like our Confidence Score, allows businesses to have a comprehensive assessment of a transaction by leveraging the millions of transactional patterns across a network and the power of Identity Check APIs 70+ data signals in a single, actionable score. But it is also only a part of a holistic fraud prevention strategy. Therefore, it’s important you review and analyze the performance of your overall rules, combination of rules, and score incorporation when predicting fraud, especially as fraud changes, and models learn and therefore adapt too. Sometimes this means you may see an uptick in your manual review queues, which ultimately could mean your manual review team and KPIs feel the additional weight, which may initially give you a negative gut reaction. But what it really means is your scores, combinations of rules, and overall fraud strategy is actually catching MORE fraud.
6. Our score in your models
Increasingly, our customers are using our score in their machine learning fraud models used to reduce chargebacks and identify good customers. Because fraud patterns change, its a best practice to constantly retrain these models and keep them relevant and effective. With this process, these models adapt to improvements in all our attributes including the Confidence score.
Want to learn more about our modelling and the Confidence Score, contact one of our data experts.

Start a Free Trial

See how Ekata can reduce fraud risk for your business, contact us for a Demo.