I think the fraud rate was like 5% of the overall revenue.
You have somebody, you know, the CEO's friend gets blocked
or uh or doesn't get a discount and then you get
a very direct feedback that rules or not work.
Let's agree that you will take this loss on yourself
then and then there immediately start the stop.
It's like okay explain to me again what will happen.
Processers always manage to surprise you
and come up with something different.
So it's better to allow for some fraud to come in and then
Welcome to another episode of the Saturday Fraud Strategist.
With me today, Jaanus Uudmäe from Bolt. Jaanus, welcome to the show.
That's awesome. Pleasure to have you on the show.
Jaanus, I've seen your post popping up on my LinkedIn feed
for probably the better part of two or three years now.
And you always have very interesting takes on fraud and also
specifically on managing fraud at Bolt, which, if you guys are not
familiar with Bolt, is on-demand marketplace, headquartered in Estonia.
But Before we get there, Jaanus, maybe you can tell us a bit
about yourself, about your career, what brought you to
this to this role that you have right now at at Bolt.
Sure. Like first of all, thank you for following me on LinkedIn.
So that's always nice to hear that actually the
stories that I write resonate with people. So that's good.
I got started in this fraud space already a long time
ago, like 20 years ago, when in 2007 I joined Skype
In Estonia as a software developer. So straight out of college.
I just graduated from University of Arkansas in the United States.
And I came back to Estonia, I was looking for a job.
And a good friend of mine, my old classmate, he was working
at Skype, said it's a pretty cool place to join.
And by accident, I ended up in the fraud
team, not knowing anything about fraud.
Like most people in this space, you don't really study anti-fraud
or fraud fighting in the university, but yeah, I ended
up as a developer of fraud tools in Skype.
Very cool experience, work there.
For several years doing the internal tools
building and things related to that.
And in the process, I got to know more about fraud
and all the interesting and cool schemes.
And over time, as the team grew, I became more involved on the
product side of fraud and became the product manager there.
And ended up spending 10 years in Skype.
And then Skype was acquired by Microsoft at Microsoft as well.
And then in 2017, I believe, yes, this company called Taxify,
which later got renamed to Bolt, approached me and and
said that hey, we kind of have a fraud problem as well
popping up that would you be interested in joining?
And I thought that yeah, it's interesting
company, so so why not give it a try?
Kind of restart of my career because I went from product
management to kind of doing hands-on analytics. At the time.
So we had just two people working on
the fraud department at that point.
So this is how I ended up at Bolt.
And over time here, the team has grown.
So right now I have 14 people reporting to me.
And Bolt operates in like 50 plus countries.
We have many different business lines.
So we have the kind of taxi business, food delivery business,
micromobility, which is scooters and e-bikes, then car rentals.
And a business product so in different geographical
regions, different product lines, mostly in Europe
and Africa, but also Asia, South America, Canada.
So we see a variety of different customers and different fosters.
So life is very interesting here at Vault.
Yeah, yeah, yeah, yeah, I bet.
It it was interesting because when when we had our first call, and I
found out that he worked at Skype, I figured that at a certain point
we were kind of like colleagues because I was working from 2009 at
PayPal, and weirdly enough, if you know the history, PayPal and Skype
separately were bought by eBay around about the same time.
Which is very weird to think about it nowadays that these
two companies ever lived under the same roof.
But yeah, that's super cool.
Bolt, I would say, I mean, for our American listeners, you cannot
really compare it to any kind of single company in the US, right?
Like in my mind it was always kind of like Uber,
but it's not really like Uber, right? Or not only.
It is. Yeah, well it is it yeah, well it's like Uber but better.
Right. That's what we would like to think.
But in many ways, actually, we are very similar to Uber
and we operate in many of the same markets, right?
We were not in the US, but Europe, Africa, we're on the same
markets, especially on the ride hailing side, and then Uber
has it's Uber Eats, and we have our own Bolt food.
So a lot of them business lines are very
similar, the products are similar.
The services that we offer are similar and speaking
to the Uber's fraud people, obviously fraudsters
and the fraud schemes are very similar as well.
So so let's talk a bit about that because I think that
marketplaces are, you know, it's a known issue.
And you know, when you look at the retailer side of marketplaces,
obviously your attack or sorry, the surface of attack is
is wider and there are all sorts of different schemes and
let's say fraud patterns that can cause you damage.
I do think that it actually gets worse when you're dealing with
kind of like the on-demand economy economy marketplaces.
So tell us a bit about that.
Like I'm guessing that you're seeing like extremely weird stuff when
it comes to fraud and to a lot of extent your definition of fraud.
Is probably quite different than how a lot of fraud fighters
who are listening to us right now would think about it.
I think pretty much every company has its
own challenges when it comes to fraud.
It so much depends on what you're selling,
what your what your services are.
But for us, I think we see every every
piece of it pretty much like that.
The standard fraud that most companies that sell anything
online deal with is this issue with chargebacks.
So basically stolen credit cards that somebody
uses to purchase services on your system.
So this is a very standard thing.
This is what kind of industry standard,
what the Visa and MasterCard measure.
We gotta make sure that we are below those
fraud thresholds that they offer.
So this is kind of yeah when you speak to other companies,
other fraud fighters, everybody sees this part that how
do you deal with those people who'd come with stolen
cards and then try to purchase your services.
But being on the marketplace we also have Decide that we don't
really control the services and the and and what's offered.
So we can have issues with drivers that are colluding with the
riders, and we can have issues with potentially restaurants.
So this bec creates a lot of complexity.
And in food delivery, especially, that you have this three
sided marketplace that you have the person ordering the food
that we call the eater, and then you have the restaurant
preparing the food, and you have the courier in play.
So if and when something goes wrong.
And there is some accusation of fraud, it's sometimes
very hard to figure out actually what's going on.
Is somebody colluding with somebody,
or is it just a misunderstanding?
So so we have those issues, and then also a lot of our attention
nowadays goes to things which are not easily that measurable.
So, for example, I think what what we call campaign
fraud, or some companies call bonus uh
Is obviously, you want to get new customers to the platform.
So we offer quite good discounts for new customers.
Obviously, some people like fraudsters they think that, well, why
would why would I ever pay a full price for a service when if I
create new accounts, I can just continue getting stuff at the discount?
And so we're trying to prevent that.
And that's very hard to fight against because you don't have a
good way of understanding like was it now a good customer that
used a discount and later just churned for whatever reason?
Or was it the fraudster who who didn't
ever intend to pay the full price?
So this is this is one of the big challenges
for us in an ongoing fight.
Yeah, I think especially the like the three party model is very
interesting because just thinking about collusion from a single vector,
which is like buyer and seller, you suddenly have three.
You have buyer and seller, and you have seller
and driver, and you have driver and buyer.
And I'm just thinking how much just adding this that this third
party adds to the surface of attack and what kind of very
weird things probably you've seen along the years.
And and I'm guessing that this is a lot
of the times this is the challenge.
Just the just managing the complexity and thinking about
the like the fraud stories themselves and trying
to stay ahead of the fraud stories especially.
And I'm guessing that this is also an issue that Bolt
has, what at least five different product lines,
and you keep shipping more and new features.
So also the product landscape keeps changing under your feet.
Correct, yeah. So it never becomes boring, let's put it this way.
Yeah, with all the new features and new services.
And just to be clear, like most of our customers are good customers.
So nothing wrong with them.
So just that because we have so many of them, we see all
sorts of weird patterns that sometimes are not even fraud,
they're just issues with the platform, and then trying
to distinguish between an issue and somebody intentionally
trying to abuse your platform, that's difficult.
Like let's take a simple example of refund abuse.
So, what what I mean when I say refund abuse is
that somebody intentionally lies to you about quality
of services in order to get a refund.
Just kind of initially pays for the service, but then comes back and
says that they're not happy to to get the money back effectively.
And it's this is very common in the delivery business because with
food delivery it's very hard for us to prove what was the quality
of the food, or did it arrive in in a nice way, or did the courier
like mess it up and and shake the pizza in the package?
And was it all a big big just chunk, right?
So so there when we see a customer complaint coming
in, like we have a lot of data about the customers,
about the restaurants, the courier.
We can see the route that they drove and all that all the times and
everything, but we're not there to assess the quality of the food.
So to understand whether somebody is telling the truth or not
about some missing item, it's it's almost impossible.
Like we don't have any hard evidence.
So we have to look at other data to kind of understand
like, could there be that somebody's lying?
Like we can see that does the customer have an
issue, like history of complaining a lot.
Like every time they order, they they always
say that something is missing.
That's obviously a clear sign that they're not a good customer.
Or does the restaurant have a history
of not putting the items in the bag?
Or does the courier always do something weird?
So we have those data points.
But it's not always so clear-cut because even for
good customers, like stuff like this happens.
Good restaurant as well, sometimes they forgot to add an item into
the into the bag or they put a wrong wrong item in there.
So you can't immediately say that based on
one case like that somebody's at fault.
I'm guessing that it's also
Yeah, I'm guessing it's also very difficult when it's the first
transaction of a new buyer and you both don't have the history and also
you don't know you know, because is it a good customer that is now in
a real danger to churn because they had such a shitty experience?
Or is it a fraudster that is clearly defrauding?
The first time customers, those are the hardest because
if you already have history about the customer, it's
quite straightforward in most of the cases.
But the new ones, that's the trickiest.
Because in the fraud team we don't want to you
know, prevent hundred percent of the fraud.
It's we know it's not possible.
What we want is we want to grow the business but in
a sustainable way so we don't lose too much money.
But yeah, the first time customers it's always the trickiest
because if they have a bad experience, that can
mean that we lose them as a customer forever.
So we want to make sure that we take good care of them, but at the
same time we want to make sure that those are not fraudsters.
So the way we approach it is that we look at the data.
So if we on the fraud side can somehow prove
that this customer is actually a fraudster.
And the way we do it is that we can look at data items that we see,
like for example, if they use the same email address then a previous
customer who has a history of asking for a refund, we can quite
safely say that, okay, those two accounts are linked.
This is the same person just trying to abuse our refund policy.
Or if you have some other data items which are hint strongly
suggesting that this is the same customer, like IP addresses or
or some device information, then we can say safely say that yeah,
this is not a real customer or this is not the customer that
we want, and they will they don't deserve a discount.
But on if we can't prove it, then we just default to the
standard store policy that Bolt has, like whatever it is
in a particular market, what's the standard refund refund
amount and and how many times you can get a refund.
So this is the way we work it. But it's it's always very tricky.
And we work together with the business teams as well
to kind of show that, okay, this is what we see.
Like this this is what a normal customer behaves, and
we can show that if a customer, you know, three times
in a row, they always complain about something.
It's a sign that they're a bad they have bad intentions, that
that they never just do a normal order and stop complaining.
Do you do you also w when you say that you're working about that
you're working on it with the business team, what what does that
actually entail when when you're outline your outlining your
fraud policy and how you treat first time customers?
So by business team I might mean mostly our customer service
department who is in charge of like handling those flows and who
owns the policy of like what's the okay if you complain that your
pizza is gold, like what's the refund amount that you should get?
So they own the defaults and then we work with them to show
the data to also see that we don't just want to block everybody
from getting a refund, but with the data we show that
like we think that this is reasonable, explain those
fraud scenarios at high level to them as well.
And it's like a test for us.
If we can prove to them that that makes sense, then it's good.
Then we we can implement the policy.
That does not mean that we we need to run every rule change by them.
It's more at the high level that this is this
is the type of fraud that we want to tackle.
These are the kind of the data points, and this is this is the
amount that we can kind of cover or protect for the company.
'Cause it also means that like if if that money goes out
to the fraudsters as a refund, that's potentially a refund
amount that some legitimate customer won't get.
Yeah, yeah, yeah. So but I think that's the interesting part, right?
That you're working against a budget, but like a lost
budget, and you're trying to make sure that you're
spending this lost budget on the right folks. That makes sense.
Tell me, I mean, this sounds obviously like a very hard problem.
Uh and it sounds like like I wonder without having the evidence, and
as you said, like it's very hard to know for sure what transpired.
And it's definitely hard to know for sure if you decide
not to give a refund or if you decide even not
to accept a transaction in the first place.
How do you how do you actually manage to automate and build
rules or even models around these kind of problems?
So I'll start with a simple case, like if we're
talking about chargebacks, stolen cards.
Uh for that, it's actually possible
to measure it in a reasonable way.
So there we we can run an A/B test to say that you know there are
certain transactions that our system flags as high risk that we
would normally stop or where we would do a 3DS challenge.
So we can take a small percentage of those transactions
and let them go on without any friction.
And then we can measure what happens.
So if that transaction results in a chargeback, then
we know that our rule that triggered was correct.
Because indeed there was a fraudulent signal in the end.
So our rule was correct, and we can be happy about
it that our system is working as normal.
But if it turns out that we flag it as high risk, but if we allow
the transaction to happen and there were then nothing bad happened,
it was a good case, then we know that it's a false positive.
And that's that's the F-word that in the fraud team everybody
is very concerned about false positives, right?
But but that works really well for for labeled data.
So in in in this in the case of chargebacks, the data is labeled.
You know what is good, what is bad, and you can target it.
But now let's say, yeah, that that's the that's the easy part.
But even there, the it's it's it's not super easy.
There are complications there as well, but it's relatively easy.
But then if you're talking about, let's say, this campaign
campaign abuse and like which customers should get
the campaign codes and which which accounts are basically
created in order to just milk that money.
It's a much harder problem because nobody comes back to you and
says that, you know, I was able to trick you and and get this
discount for for a hundred different accounts that I created.
So so the data is not labeled.
So it is our job in the fraud team to somehow put rules in place to
label it so we can see that somehow say that these are this is the
money that went to fraudsters and this is for good customers.
The simplistic, you know, approach that we can take
and where we can look at the total opportunities.
Like let's look at all the customers where as a first time
buyer they get some discount, let's say 30% off.
And then after that discount period, did they
make any transactions at normal rate.
And you can say that if they made any transactions with a full
price, you can say that these are actually good customers.
That's one way of doing it.
And if they stop making any transactions,
does it mean that they're fraudsters?
No, maybe there's just customers who who wanted to try the product,
but for some reason didn't want to pay the full price for it.
That doesn't mean that they're fraudsters,
it's just customers who try it. It's it's quite normal. Right.
So so it's up up to us to kind of label the data
and say, like, at which point do we draw the line
and say that this is fraud, this is not.
And to make it even more interesting, that we have seen fraudsters
who are kind of call them opportunistic fraudsters.
So what that means is that you kinda you behave as a normal customer
under normal circumstances, but then let's say your friend tells you
that, there is a way to get some money back from Bolt or a little
trick to to get a discount, which is not completely okay by the
rules, but you know, this is how you do it, then they will try it.
So this customer they will try to cheat the system a bit.
So if we allow them to do it, they will continue doing that.
And we as a company lose money.
But if our rules step in and stop that behavior, like stop
that refund or don't give them the discount code, then this
this customer see that the hack doesn't work basically.
So I will just go back to using the product as a good customer.
So does it mean that this people are now fraudsters or not.
They're more like on this gray area.
So building any A/B tests is very tricky for them because
they can behave as fraudsters or as good customers.
So when you're trying to measure it in an A/B test like are
they good or bad, it completely like messes up your data.
Yeah, and even if they cause you loss on a single
transaction, it might be that they are still net
positive when you look at their lifetime value.
But I I want to ask you a different question because what you
describe is I think definitely the right approach, which means
in the end, you need to manufacture your own labels.
There is no way around it if you want to to do automation.
But the risk in doing that is that it's very easy to be tempted, let's
say it like that, or to fall into the trap of recursive logic.
Meaning I label trans accounts by saying, Yeah, they didn't transact
for a full price after their discount period, and then and
this creates my labels, and then I write a rule that you know,
like uses somehow the same logic to make decisions.
And if my rule logic and my labor logic is the same.
I mean, obviously I will have good results, but it
can also be that I was wrong in my assumption.
And even though my manufactured numbers look
good, I create a lot of false positives.
How do you make sure that you don't you know,
like fall into these kind of like biases?
Well, we do have guardrails for that.
Like we measure, for example, customer service
tickets related to that issue.
So obviously customers get good customers get very angry when they
don't get the refund when they rightfully deserve one right.
So we this is kinda proxy that we see that if the reef if the
customer service ticket amount goes up, that maybe something is wrong.
It might be that the fraudsters are actually
being very vocal about it as well.
But at least it's it's a sign that hey,
we should look more deeper into it.
Like maybe there is something in it.
And then, like the in the worst case or in the most interesting
case, it's you have somebody, you know, the CEO's friend gets
blocked or or doesn't get a discount, and then you get a very
direct feedback that, hey, your rules are good, rules are not
working, and then the then you need to explain yourself.
So that's a proxy way, right?
But it it is very tricky, especially
when you create your own labels.
Like how do you know that it's it's yeah, like you said,
it's a trap that you can say that I'm right because
I blocked it and I blocked it because I'm right.
And they automatically say that that this is the way.
But for that we we we are playing it actually
quite conservatively in our company.
I think that usually when we don't give a discount or say
that somebody is behaving as a abuser or fraudster.
I want it to be in a very clear way that I can explain
it to anybody in the company or even in my family.
Like these are these are very s that's very strong evidence
that this is not a good customer with not good intentions.
But yeah, even then, mistakes do happen.
And we're always balancing this side that you can
see that maybe we're blocking too much in some
markets, then we scale back the rules a bit.
And there is yeah, there is no right or wrong answer.
There there is no like state where you can say now the rules are
ready, they're perfect, you you cannot make them better.
And it's always a fluctuation that we see that you
can optimize the rules first to stop the losses.
Usually that's how we do it.
Like when you're bleeding a lot of money,
you need to stop the bleeding.
And then once the once the situation has stabilized, then you
start looking at okay, where are my false positives here?
How can I make the rules more accurate to
allow more good customers to go through?
And And yeah, but but it it's a proxy way of CS, just anecdotal
evidence from from friends and family and just common sense.
And and these proxy labels or proxy actually counter labels, do you
measure them against your rules or also against your label sources?
I'm guessing like the labeling itself is not done manually.
I'm guessing that you have kind of like batch processes that
just go through whatever, through the entire data every night
and and kind of like mark what you think is is bad.
Do you measure like both only the rules, only the labels?
How does it how does it happen?
We at like automatically we measure it
at the whole set, at the high level.
Not not at the rule level, but yeah especially with those labels
that we create ourselves, then it's it's more trickier.
But then you see that it's the health of the system overall.
If everything is fine, no reason to dig any deeper.
But if you see that something is trending in a weird way,
then you can manually kind of go deeper and see like
well, well, which rules are triggering the most.
Is it because of fraud? Is it because of maybe some data issue?
Because yeah, like a rule triggering more than usual doesn't mean
that you have all sudden all of a sudden false positives.
It might be that there is pressure from fraudsters
and you actually want this rule to trigger.
So you need to look more into it to to get to get the data.
And the problem is also that with those labels that the time
it takes for the label to appear, we call it maturing, it
can be quite long because you make a decision today about
one transaction, but in order to assess whether that customer
is a good or a bad customer, you need to wait for the
next transaction, next order, and maybe a few more.
And that can take in some cases weeks before you have sufficient
data because some of our customers, you know, order multiple
times per day, then you have quick labels.
But a lot of them it's just you make an order a couple of times a
month and it's then very like you you're waiting during that period
and you don't know whether you make made the right or wrong decision.
So there is also this inertia in changing the rules.
That what you change today has an effect
that you can observe maybe next month.
Hey folks, I want to take a quick break
to speak about today's sponsor, me.
If you're finding this useful, do me a small favor, like and subscribe.
It really helps with the algorithm.
And if you're not already on my newsletter, the Saturday
fraud strategist, you should definitely check it out.
Every Saturday, I break down fraud trends, real cases, and strategies
from my own experience of building and operating fraud prevention systems.
Whether you're new to the space or a season practitioner,
I'm sure you'll find it interesting. Check the link in the description.
Takes 2 seconds. You'll thank yourself next Saturday.
I I wonder with the risk of, you know, like going on a tangent
with AI, here I said the word, with AI, there like I see two
I don't know if to call them risks here or enablers.
Let's call it in enabling factors.
One, we know that today to send, you know, a picture of a rotten
burger or an upside down pizza is very, very easy. Yeah.
It's like one prompt away from you.
And secondly, like literally in the last like month or
two, we have all of these agent assistants, right?
The instincts, the muses.
And now you also don't need to like you know
interact with customer support yourself.
You can just you know like manufacture with a prompt, your image, tell
your assistant through WhatsApp to get a refund, and that's it.
You're pretty much done with it.
I wonder like how have you seen AI shape the kind of like
first party fraud scene, both on the opportunistic side,
if you can say, and also on the on the fraud side.
Uh it's definitely yeah, like altering the pictures or
coming up with with pictures is definitely a thing.
But I would say that this is not so prominent.
At least I have not seen it yet.
Or maybe maybe they're so good that I've not seen it.
But in some cases for us, like for fraudsters, they
know the right kind of things to go after as well.
So if you say that you have a missing item, you don't
even need a fake photo because there is no Like we
don't ask for a photo if the item is missing.
You take a picture of nothing.
So you don't even need to generate the picture.
So that's that's the easier way. You don't burn any tokens.
Like you can write maybe a really good, sad story
about like describing your problem as well.
But to counter that on our side, we we have also like machine
learning models looking at what what are you writing about like
like what's the actual issue that you're describing and then we
can assess from fraud side that does it make sense or not.
And then we can look at the data.
Like you can fake a picture, but it's usually it
is harder to to fake all the other elements that
we see in the background about the order.
So we have to rely on that.
Do you think though it helps more the opportunistic I I don't want to
call them fraudsters, but kind of like the the legitimate accounts that
opportunistically commit refund fraud because it's not only relatively
easy in their mind, but also it's a bit more far removed from them.
It's like it's not them doing that, right? It's their agent doing that.
I don't know if you've seen this effect.
It could, I I've not seen like usually these agents and these more
sophisticated schemes, these are for professional fraudsters.
The opportunistic people are more like,
hey, I heard my friend did it.
So he or she got a discount, so why am I the stupid
one, you know, paying the full price always?
Like let me do it as well.
But usually then people try to do it manually.
And and yeah, they don't go to agents at at this stage yet.
So maybe some do, but in the end for us the methodology
of detecting it is still the same.
Like and and to making sure that we we kinda manage the risk there.
Got it. Very interesting.
I I wonder okay, so we talked a bit about the labeling part and
how you how you can make decisions in this kind of like uncertain
environment where it's plagued with first party fraud and
also like three parties within this business model.
I wonder how like how do you actually measure your effectiveness here?
Because, you know, with chargebacks, again, it's very easy.
I I get chargebacks, I write a rule, I get less chargebacks.
And here, you know, I I don't know how
much like I I would say differently.
It probably you probably always have refunds.
And I'm I would assume that, you know, like certain attacks
don't really change your refund rate tremendously.
I might be wrong here, but it's probably very hard for you to A get
a sense of how how many refunds you actually saved on one hand.
And on the other hand, I don't know if you if it's that easy to
understand like what do the false positive cost you at the same time.
Exactly. Yeah, yeah, you're absolutely right.
It is a difficult problem.
And talking to my industry peers as well, like
there is there is no easy way to do it.
Like yeah, like one way indeed, like to measure how much we have
potentially saved, that's quite straightforward because we can
say that you know, we need we deny the refund to X amount of
cases and the refund value would have been you multiply it whatever
whatever the amount is, like you get to a number.
So this is potentially what you have saved for the company.
But what you don't know indeed there is that the false positive
aspect of how how many of those were now good customers that
got really pissed off at you because they they got a bad
service and then you denied them a refund potentially
and you lose their business going forward, right?
So this is the tricky part.
And to measure that, like we discussed, it's to
measure that accurately is is quite difficult.
It's just because you always have refunds and you're
gonna always have people who who can lie about
it depending on their mood or whatnot.
So it is a constant constant struggle, let's put it this way,
that we kind of know, yeah, like how much we can prevent.
And and then you rely on the stories of what somebody picks up
manually that they I see a weird pattern of somebody requesting a
lot of refunds, and for some reason our fraud rules don't pick it up.
So we can look into that, like what's the pattern, and you
make an assessment: does it look like fraud or not?
But yeah, the false positives are even trickier because a lot of
the customers, good customers, don't come and complain to you.
So you don't even have this proxy of CS.
And it therefore it's also very hard to set targets.
Because w w what is your and how do you how do you kind
of measure that your team is doing a good job or not?
Because there are external factors, like the amount of refunds
depends on quality of the restaurants, the quality of the couriers,
maybe even the overall economic situation when the economy is
not doing so well, people are more tempted, good people are
more tempted to ask for a refund to game the system a bit.
So there are external factors playing as well.
So we kind of try to establish some sort of range where
we say that okay, we we think that our refund fraud is at
this level based on the data and what we can prove.
So let's try to kind of make sure that we
we roughly save that amount of money.
And if it goes up, then we need to explain
it or goes down, like what are we doing?
And the false positives as well, like the the proxy metrics or the
cardrails with CS tickets and and those cases are are there to keep
us honest and and to keep us from making too many mistakes.
But it's very hard to say like where is the sweet spot.
How do you know that now the team is performing really well
and are we doing better than last year, for example?
So I I think I I mean I I would kind of expect this answer, but then
my my next assumption would be that actually the biggest challenge
is facing this kind of like downward pressure from leadership.
Because if you cannot really say, we're doing great, leave
us alone, and cousin of the CEO now got declined or banned
or whatever, and there aren't really set goals.
It's very easy for the business to kind of
like say, look, we need to grow faster.
We like we just entered this market.
And you know, and and our numbers like are
way worse than our other other markets.
And and you, you know, you can be like,
Yeah, because the market is way worse. What can I do?
And fraudsters are early adopters. How do you how do you manage?
Because I think this is like the biggest challenge.
How do you manage your own business leadership around this uncertainty
and around this inability to kind of like, you know, like
give them a confident measure of how well you're doing.
Mm-hmm. Very good question.
Like first of all, I would I I think that our
leadership is quite good and and and smart.
So they understand the challenge.
And I'm not saying it only because they pay my
salary, but because I truly see that as well.
That they they do understand the struggle.
Like there is it's possible to explain it
to them, so that's the starting point.
And then the second what we have done is that we don't overreact
or over control it when we enter a new market.
So usually we go in with just two new products, new new areas with
just a kind of reasonable set of controls that we know that need to
be there for sure, like chargeback controls and things like this.
So kind of go in quite light on the rules and
then observe what the market actually does.
It's it's good for two things.
Like first it enables expansion because that's like when you
go to a new market that's the key that we we want to get
reasonable market share we want to grow as fast as possible
and you can say that the fraud cost is just the cost
that you are paying as part of the expansion cost.
That we're willing to take a bit more losses.
But then once you have the losses then it's much easier for for me
to go and show that hey based on the data we can see that this is
this is going on like guys you can see I can pull up some concrete
example accounts, like this is the type of fraud that's happening.
Like obviously, everybody understands that we we want to stop it.
So then it's much easier for us to kind of prove that
we need these rules and we will add some friction,
we will do the things that we need to do.
And smart people will also understand that ultimately.
Unfortunately, that there will always be false positives as well.
So so so then it, yeah, exactly.
So I I remember back in the day when our Bolt food delivery
business was just starting and we we didn't have really
fraud for a long time, like chargebacks in that.
And we had just expanded to Latvia and in Latvia we operated
for a couple of months without any issues as well.
And then we got hit by fraud.
And and at one point I think the fraud rate was
like five percent of the overall revenue.
And that was with like reasonable fraud controls.
It was it's not that we were running without any controls.
We had quite okayish rules in place, but
it was five percent of the revenue. So that's like wow.
Very easy to explain to the local teams and to the management
that what like why we need these these rules, these these
endpoints, this data, like it was a no-brainer.
But but yeah, before that, like even when you look back, like what
rules could we have built to prevent this five percent loss?
No matter how hard you think or how how many tokens you burn
nowadays with AI to try to come up with the rules for the
schemes, like fraudsters always managed to surprise you.
And come up with something different.
So it's better to allow for some fraud to come in and
then actually deal with the actual problem, not just
just some hypothetical thing and be wrong.
I I I love it because I think it touches two two points.
One, which is I I like see as a recurrent theme in how you guys
are working, is really to to to like plainly budget it.
And I and I and I liked how you positioned going into a new market,
you need to kind of like budget fraud as part of your expansion.
I was always thinking about it, you you basically you buy data.
I I think your expansion expansion tag price
sorry price tag is is is even better.
Secondly, I think like playing it cool, and just giving the business
what it wants and let them from time to time to actually fall into fraud.
Many times, even though it sounds very how should I say it, very
Machiavellistic, but many times it is actually much easier.
To manage both the business and fraud, then going out very, very with
a very low risk appetite, with very high constraint, and with this kind
of like, you know, this this fear that you might get a lot of fraud,
that actually many times get the opposite, the opposite result.
I I really love how you how you frame this,
especially around market expansion. That was very interesting.
That that it that is true indeed.
And it of course we're we're being reasonable.
Like when we see that there is a product that like is has so many
loopholes, like then we strongly advise the business like guys,
like look, like guys and girls, like this doesn't make sense.
Like we know from experience that this is a bad
idea, we should put some controls in place.
Like we do strongly warn them.
Very rarely do we like put the foot down and say like no way we're
going with this, like what we say that okay, if we're going, like
let's agree that you will take this loss on yourself, then.
And then they're immediately start that the stop is
like, okay, explain to me again what will happen.
Like before that, they're not even listening that much,
but then it's like, okay, now the loss is on you.
Then they will listen and and usually they're
yeah, they they will understand.
But we have had some cases like last year as well, where
we indeed launched a product knowingly that that it it had
some major issues and we were lacking fraud controls and
we did get hit hard, harder than we even expected.
And it was it was a good lesson for everybody in the company that
hey, well these guys kinda warned us and and these things actually
do happen that there there are frauds out there that will take
advantage of the of the weaknesses that you built into your products.
Absolutely. What can I say? I've been there.
I want to switch gear a bit.
And you know, we we talked a lot about fraud and how
we deal with fraud, especially first party fraud.
I I wonder, you know, like going a bit more to
the side of defenses and how you guys work.
I think we we talked very little about it when we had our prep call.
But it it does sound like your team A are relatively technical.
I mean, in the end, you guys do very little manual review, right?
Most of what you do is automation with
rules, with models, with whatever.
I wonder how how much you guys use AI.
And and also in what ways you used AI.
What's working, what's not so much working.
Yeah, maybe before even talking about AI, a quick overview
of our kind of fraud defenses or how it's built up built
up is that yeah, like you said, it's the team is technical
and we don't spend a lot of time on manual reviews.
And main reason being that our average
transaction value is relatively low.
Like a ride can cost in an African market, just a few Euros.
Maybe even in UK we're talking about 20, 30 euros.
So to review one case at a time, it doesn't really make sense.
There are certain aspects where we must do human reviews, like when
we suspect that the driver is actually involved in the scheme, then
by regulations that sort of decision needs to be done by a human.
Or when we're talking about do we pay out some big amount
to a driver or restaurant, then human is involved.
But mostly the rules are based on data and analytics, that
like plainly, like if there is a new scheme, especially what
we have is a fraud analyst going to the database, pulling
some data, trying to find the patterns and doing this
sort of data extraction and data analysis, right?
This is what we have done from day one since I joined Bolt.
But on top of that, what we have built as a next layer as well is
the kind of, I would call it the traditional machine learning.
So, machine learning models that learn from labeled data,
that you have scoring each new transaction, each new
activity, and then making the decision of like do
we allow it, do we step up the challenge or not?
So, this is the kind of the automation and and machine learning.
And it works really well in known for known fraud patterns.
So, this machine learning is it's very accurate,
false positives are low, it's super good.
So, the way we have it right now is that machine learning
covers most of the core fraud, and then the human rules
are there to address new and emerging trends that
the machine learning doesn't yet know about.
And it's doesn't have the training data.
So the human can build a rule, deploy it really fast in a matter
of you know hours if needed, make changes to stop the bleeding.
So and then eventually machine learning will also like
the model will be retrained, it will know about it and
then hopefully we can get rid of the manual rule.
So this is the traditional way.
And now with AI, like where where does AI come in?
AI definitely helps us in this analysis part that the human does.
Because as I explained, we have so many different countries,
so many product lines, so many features.
Like there is even in when you think about it in our ride
hailing product, when you when you ride like sure.
In the beginning of the ride you get an estimated price.
We can book that amount from your credit card, but
then stuff can start happening while you're ride.
You can have a destination change event.
You can have a paid wait time fee.
You can have just a route can change.
And at the end of the trip you can decide to leave a tip and then
it's up to us to decide like do we allow you to leave a tip?
Do we allow you to add a destination change waypoint?
Which changes the price drastically,
like what do we do in those steps?
And at the database level, all of those data points are
stored in various tables and the architecture keeps changing
because you know the the feature teams are optimizing
for it, building like re-engineering the work.
So to know all of this context is quite difficult.
Even for a seasoned analyst who has worked on that field
for a long time, there is always gotchas that oh, but for
this payment method, the data is stored a little bit differently,
and what does this thing actually mean?
So for us, this conversational analytics
is really a game changer there.
That now we're storing all of this knowledge that used
to be in some analyst head who was looking at it.
We're storing it in context files.
So all of this works that hey, if you have a destination change
initiated by the driver, then you should look at this data.
And what does this GPS point actually mean?
Is it the GPS point of the that you get
from the rider app or the driver app?
And all of these little details get that can make a difference.
And when you're doing the analysis as a human, you might
get it wrong or you might have a wrong assumption.
But working with AI, then you store this information gets
stored in a context file, and next time somebody runs an
analysis, hopefully it pulls the right context, it it makes
the right decision of how to interpret this data.
And then as an analyst, the beauty is that when you discover that,
actually the assumption was wrong that I I know better, that
actually you should also incorporate this information.
Then you can just tell AI that hey, this is what
you do, this is this is the stuff that you're missing,
and it will go and update the context.
So we're kind of building this knowledge base, this living
documentation and kind of making it better and better.
So it's kind of our collective brain at this point of
how we understand the complex of data structures.
Yeah, I mean that that's super fascinating because I I was sure
that you were going to go on a whole different direction.
Because usually when you're speaking about AI in the context of data
analysis, you're speaking about AI doing the data analysis for you.
Which is in a technical team, A, you would say,
okay, how much time does it really save you?
And B, you would also say, Okay, there there are enough guardrails
because it's a technical team and they know SQL and they understand the
quirks and they understand like where the AI can can go wrong and so on.
But what I got from you is something else completely AI
is actually how you use it is to create this like context
layer, which might be missing for the analyst.
We and I find this statement very interesting because usually
we say that it is AI that is missing context.
That you know, if we give AI a data set and it only has this
data set and it has no business context, then you know it
can make decisions that might look good on the data, but
might not have like the the business sense behind them.
And I really love that you expose that that has nothing to do with AI.
That is just you know poor data analysis in general.
And and looking just at the data without bringing it in
context is is is just you know poor poor analytics.
And it's very interesting that the way that
you approach like solving that is with AI.
That's the first time I ever heard about the idea.
How how how I'm I'm wondering how do you see the effect of it?
Like, like is it helpful? Is it successful?
How do you know it's successful?
Yeah, good question. Like, how do you measure the effect of it?
Like subjectively, or what you can kind of measure is that how long
it even takes somebody to write a certain script, or how much of
their work day do they spend sitting behind a screen typing in SQL
versus actually looking at the numbers and making a business decision?
So right now, assuming that the concept context
is relatively well described like this writing
the SQL becomes quite trivial and automated.
So that's already a big win.
And it's for me personally, it's a huge win because when I started
as a data analyst in this team, then yeah, I I knew the in
and ins and outs of all the tables that were available.
But as a team lead nowadays I have no idea.
Like I just know the main tables, but
I don't need to know them anymore.
I can just go and ask and get quite good
answers to my questions from AI.
So it has enabled me to get stuff done on my own.
And I'm I don't have to spend time and effort on like writing
the correct SQL and debugging some SQL issues.
Like that's done by AI, and I can focus already
on the outcomes, on the numbers, on the next next
part validating them, making sense of them.
So that AI has definitely enabled it.
And it all the dashboards that my team is building now.
For ourselves as well, that we're moving away from this traditional
BI-based dashboards and we're actually generating dashboards which
are very specific to our product and features that enable us
to look into okay, in this market, what are the trends?
How many of the payments missing authorizations or we failed
to capture, and how many of them are charge chargebacks?
And you can easily then ask the next question.
And hey, I've very helpful in like discovering
And it's what's also awesome is it allows us to
quickly kind of expand outside of our sandbox.
Because usually in our team and most fraud
teams, kind of you optimize for fraud.
So you optimize fraud losses and false positives.
Because that that's what you know.
That you can only as a human focus on that much.
But now it's very easy for us to add to the context,
for example, hey, what about payment costs?
Because we operate with many payment services providers.
And now it becomes a very interesting exercise that with some
providers you have certain features available on the payment
side, which enables you to tackle fraud better.
So you can get the fraud rates down, but at the same time, just the
payment processing cost is a bit higher on those providers.
And you can with AI it's quite trivial to
add that into the equation as well.
You don't have to know the all the details about it.
The payments team helps us to build that context, but we can
put it together so we can see that maybe what we're building
it helps to save on the fraud side, but actually on the
payment side the cost goes up, so it's a net zero.
So why why bother doing it?
Or vice versa, that without sacrificing on the fraud
rates, maybe we can change something and help
to bring the payment cost down even further.
So I think that this AI is not taking our job, but it's enabling
us to be much more powerful and have a bigger effect for the
company as a whole, because we can just look at so many things
at the same time and and to make the right decision.
So, okay, so you guys do use AI to also write the queries
themselves and also at the context layer, especially outside
of your own team's context, which is interesting.
How do you how do you see? I have an opinion about it.
How do you think about non-technical team
members that don't have SQL knowledge?
Or are not expert on your data scheme.
And having them work with AI for data analysis,
where do you draw the line?
So that needs to be managed carefully.
That's it's a tricky part because I can see that
like the fraud conversational analytics context
that we have built is already very good.
It keeps improving every day, but still it it makes mistakes.
And sometimes the mistakes can have can result
in really weird out outcomes in the analysis.
So So you still need that human filter in the loop for kind
of making this drastic or very deep drawing deep conclusions
which will which will change the logic completely.
Because usually when something looks really promising,
you're missing something, or like a how can
you like cut the fraud by fifty percent?
It's it's probably unrealistic that you're
missing some vital piece of information.
So a person without any SQL knowledge, or worst of all no domain
knowledge can take ask the AI some question and get results that
look very good, but they have no way of validating them.
So, therefore, yeah, having this ability to at least read SQL and to
understand that ask, okay, what the SQL or how do you get the data?
And then look at SQL to see that does it make sense at all?
And then knowing your fraud domain and business
domain overall as well, it helps you to validate
because hallucinations do happen with AI.
And sometimes it it's it's also very good at giving you
the data in a way that sounds super convincing.
That is like very well presented, uses very fancy
words, backed up with statistics and data science,
and it's very can be very misleading. It's it sounds very confident.
And then at the same time, as soon as you push back, you're like,
yes, you're absolutely right, it's the totally opposite.
So the human at least today is needed.
But at the same time, I can see that the co as the context
gets better that we're building and putting the guard rails
in place, the answers are more and more correct.
But yeah, like I would not take the human
out of the loop completely yet.
That's interesting. Yeah.
Yeah, I think I think it's not like where I'm going with that
is not whether we should or can take the human out of the
loop, but really is the fact that AI can write SQL, does
that mean that we don't need these skills anymore?
We don't need like the analytical skills and AI can just take that.
And you know we can give it to CS reps
and they can run the the SQL queries.
Or do you still need to have all of this knowledge and technically
understand what's going on and also the mindset to sanity check
and to challenge results that maybe look too promising?
Which is kind of like, you know, if it doesn't happen to you
every day, what are you doing as a fraud fighter, right?
Or or as a fraud analyst?
AI's job is not necessarily to come in and give you an ability
that you didn't have before, but just to maybe just efficiency,
just to save time, specifically in this use case.
I'm not saying all use cases.
I think that yeah, out of those, the analytical mindset,
and the especially the fraud kind of skeptical
analytical mindset is the key part of it.
Because if I can, I don't know, draw some parallels
that in early days of computing, you know, the
programming happened at with ones and zeros.
Very like down to the hardware, and then it was with assembler
language also very close to the computer hardware.
Like you really could control every registry
and everything that was happening. It was super slow.
And then you move to the higher level languages like
C and C++ and and SQL itself, like that you're writing
queries, it's also a higher-level language.
You kind of describe what you want.
Underneath, you still have you know bits
and pieces moving on on a file system.
So maybe this conversational analytics is just another layer
that you're you're building on top of the SQL that that
maybe writing SQL is kind of becoming a more niche thing
and it's not absolutely necessary for an analyst.
I think today it's still very good just to
validate what you have, sanity check-in.
But but I I can see from my speaking to my team as well.
Like people are actually saying that they are losing this ability to
write the SQL fast that since most of the s things are written by
AI, that it does a good enough, very good job of writing the core of
it that you don't need to go in and write it yourself manually.
It becomes more of a hobby than that, I'm today I
feel special, I'm gonna write something by hand.
But analytical mindset is something that that adds the most value.
Being able to ask the right question.
And that's the difference between, anyway, a regular
analyst or business analyst and a fraud analyst.
Like you can take somebody who has really awesome SQL and
Python skills and data visualization skills, but has
no idea of fraud, and you put them in a fraud team.
Initially, it will take them quite some time
to come up with something meaningful.
Even if you give them the task that they build with
this, do this analysis, they are lacking the context
and ability to kind of address the fraud.
So it takes some time to get them up to speed.
So yeah, the the analytical mindset is still very needed.
This critical approach, challenging in the
right places, and in our team as well.
What what's super important is that sometimes your analysis,
you do everything right, you come up with very good ideas.
But in order to implement them, it's not just that
you can write the fraud rule and implement it.
You need sometimes you need the business to change something
in their interface, something in the flow of somebody signing
up to be a business user with us or a driver.
And for that, you need to go talk to that team, need to
present the data, convince them, have a conversation.
Maybe they will push back really hard with valid points, then you
need to go back and and a little bit adjust your system as well.
So this analyst a lot of times are acting
as product managers as well.
That they're they're talking to other teams.
And this is not just purely analytics and the AI agent or
the conversational analytics agent today cannot do that because
on the other side you all you have also humans.
And I think, you know, in the end, the in fraud analytics, I think
the difference between that and like classical analytics is that you're
not only like crunching data and presenting graphs, right?
You're releasing products, you're releasing models, you're releasing
rules, like the solutions that you that you release, these are products.
So I definitely agree with the product manager kind
of like half-hat that that you need to wear.
And I would wonder, can you actually develop this mindset, this DNA,
without doing it, you know, to a degree with your own hands?
And we still don't know, right?
It will be a few years until the first data analyst that
would grow grow up and be trained from scratch with
AI, and and we'll see if it's possible or not.
But I think today we don't know. Awesome.
Jaanus, thank you very much for this. I I learned a lot.
I want to kind of like go over a couple of the takeaways
that I noted down as we were speaking.
Because I think some of these things are super or can be super, super
helpful for teams that encounter similar similar challenges.
And I think a lot of the teams out there are encountering
similar challenges, are encountering similar challenges.
I think one thing that we talked about is how you deal
with on one hand making uncertain decisions that are
very hard to argue for, even after the fact.
And doing that especially around, let's call
it business-sensitive populations.
And I think we talked about two kinds.
The first is very new customers or first-time customers,
where you know, any any friction that you bring into
the system can cost you a lot of lifetime revenue.
And the second type of population that we talked about is a bit more
kind of like high level, is basically going into a new market.
And I think what I got from you is is that your team's approach
to it, and and I fully agree with it, is actually two pronged.
On one hand, you have a bit of a higher bar, a higher
standard for actioning and for introducing friction.
You talked about it several times when you talked about in the
sense of proving that this account is a fraudulent account
through history if you have it, through linking if you have
it, through what whatever kind of like heuristics that are
kind of like very, very safe and very high confident.
But on the other hand, also to have some sort of budget that is
negotiated, and maybe negotiated is a big word here, but negotiate
between the fraud team and let's say the commercial team.
The commercial team can be CS, the commercial team can
be product, the commercial team can be, you know, the
business unit of the new market you expand to.
Where you're trying to get a feel, or actually not a feel, but an
agreement of how much we're willing to lose here for this white
glove white glove treatment that we're giving this population.
And really to kind of like define it as part of the budget,
whether this is like customer acquisition cost and
part of that is acquiring fraudulent customers.
Or whether this is part of your acquisition, sorry, not acquisition,
your expansion, your market expansion cost.
And having higher losses in a new market,
that's that's part of expansion.
So I think in general, switching from like a KPI slash goal-oriented
measurement to a budget, an agreed budget, and starting to optimize in
around optimizing your decisions around these kind of like budgets.
This is definitely, I think, the right approach. I really love it.
And I really like how you kind of like how this was really like
the red line through a lot of the topics that we talked about.
That was really, really good.
And I think this is like proper, proper lesson that
a lot of teams can can and should imitate. We also talked a bit, yeah.
Maybe to clarify that well, we don't have
like an official budget line for it.
So so it's not like officially part of the budget that yeah,
we'll go there and this is how much we spend on fraud.
But that's just kind of mindset that we have that kind
of unwritten agreement with the business teams as well,
that especially new mark new markets or new feature
launches initially will have higher fraud costs.
Yes. Yes, no, no, definitely, definitely.
That that we will deal with a problem.
Yeah, so we we're not stopping the business, we we let the
business get started and then we optimize it later.
So it's to kind of treat it treat it as a budget
line without actually that being in the budget.
Yeah, or at least maybe hidden in the marketing cost or something
like that, because you don't show your board. Yeah, anyways.
Another thing that we talked about, which I really liked,
it was in the context of first party fraud.
But I I can guess that it can also apply to completely other vectors.
By the way, we didn't talk about it today.
I'm sure it's relevant also to both, but scams.
Also an attack vector where labels are very you know uncertain
and there's like underreporting and how you basically need
as the fraud team to manufacture your own labels.
And how in turn that can also lead to very
adverse effects if you don't control it.
If you go into this kind of like recursive logic of as you
as you put it, you were right because you blocked, and
you blocked because you're right, you were right.
And I think here like I really like the approach that you gave around
measuring not only your solutions, but also the labels themselves and
making sure that on the base layer you're making the right decisions.
Because in the end, if you haven't automated process that labels
accounts for whatever reason, that is also a solution.
Even if it doesn't act on on accounts or on transactions.
So I think that that was also like a very
interesting piece of of of of advice.
And lastly, we talked about AI and specifically
how you use AI for data analysis.
I think I like like two things that you said.
So yes, you're you're using it as an SQL monkey to save you time and to
to make sure that your SQL is kind of like efficient and and you don't
like you know forget the the right commas in the right places.
But I think two things that you mentioned that were very interesting.
One is to actually use AI to manage the context layer and how
that not only helps AI itself to produce better results, but it
helps the analysts, the humans to produce better results.
Especially when you're working in a high change,
high complexity environment like yours.
Where the product always changes and because of that
the data scheme always changes and there are you
know there are multiple products and and so on.
I think that bit was very, very interesting.
And I think on top of that, and I'm guessing that is kind of like
you know like a a subpart of of of of that point is that this
also helps the fraud team to gain context from other teams.
In your case, the payment team.
And I think I need to do like a whole entire podcast on that and how
like payment teams and fraud teams need to kind of like cooperate
and how to how to optimize both of these verticals together.
Because you gave the the right example.
It might be that you you know you choose to go to a different vendor
because you will have better results, but then that's some of
it because of different interchange costs or whatever that
is completely invisible to fraud is is the same or high.
So I think also like getting the context of outside teams
and understanding what's your real incremental value.
Not how good this rule is, but really how does it affect the entire
system and how does it affect the like bottom line profit margin?
And using AI for that is, I think, really clever.
I think both of these things are the first time that I heard about at
least in how you use AI to to to to kind of like increase your not
efficiency, but really productivity and the value that you bring.
That was super interesting.
So, Jaanus, thank you very much for all of these insights.
It was a pleasure. Very interesting discussion, yes.
Well, I I was assuming with a business like Bolt that will
have a lot of interesting topics to to talk about.
And yeah, I I I I was proven right once again.
It's always it's always a good feeling.
No, Jaanus, thank you very much.
It was a pleasure having you on the show. Super interesting stuff.
And for all of you guys, see you next Saturday.