In fast-paced areas like digital wallet lending, moving from people checking things to using automated ranking systems creates a “governance gap.”
When an AI makes a label, like an income label, and that goes into the process of selecting who gets credit, the main question is not just “is it correct?” but rather: Does this AI input really change who gets credit?
If changing the AI’s signal changes who is ranked higher among potential borrowers, then it has a big effect.
Materiality means we need strict control, no matter if the AI is making the final “Yes/No” decision.
The Framework for Measuring Influence
To figure out if an AI input needs strong controls, we must understand how much it affects the decision process.
For example, imagine a model that uses how people behave, like buying expensive items such as women’s cocktail rings or regular luxury items like peridot earrings, to guess if someone can pay back money, along with an AI-generated income label.
Here are four ways to measure how much influence the AI has:
1.
Formula Extraction & Decision Mapping
The most straightforward way is to look at the “Scorecard.”
If the ranking is set in code or through settings, you can see directly how much the AI signal affects the final ranking.
Score = 0.35 (Transactions) + 0.25 (Repayment) + 0.15 (AI_Income_Label) + 0.25 (KYC)
A clear decision map shows that the AI label has a 15% impact and is done on purpose.
2.
The Ablation (Removal) Test
This is a simulation done offline where you re-calculate the list of people being targeted as if the AI input wasn’t there.
The Threshold: If removing the “Low/Medium/High” income label causes 6–8% of people who were previously approved to fall below the cutoff, then the AI signal is important.
The Insight: This tells you exactly who is being included or excluded because of the AI’s judgment.
3.
Sensitivity Grids
Governance teams should test different scenarios by changing how much the AI’s signal affects the decision in 5% steps.
By looking at how the top 1,000 most important users change when the AI weight goes from 5% to 15%, you can find the point where the AI starts to have more influence than other signals like actual spending habits.
4.
Controlled A/B Experiments
The best way to see the real-world impact is through live tests.
You can split users into two groups: one group uses the AI-enhanced system, and the other uses the standard system (like one that depends on spending history and buying items like peridot earrings). You can then check the difference in how many people get approved and how many default on their loans.
Determining the Governance Tier
If reducing the AI’s influence changes the list of people being targeted, then the impact is clear.
This “shuffling effect” triggers the need for accountability. Before deciding how risky the AI is or how much it can be used, governance should:
– Quantify the Delta: How many users are affected?
– Analyze the Bias: Does the change affect certain groups more than others?
– Define Safeguards: If the influence is high (e.g., over 10% change), then add checks by people or bias correction systems.
Governance Note: An AI signal that isn’t neutral can be a problem and carry responsibility.
Measuring influence is the first step in controlling that responsibility.
https://zenyalegal.com/insights/labelling-the-synthetic
AI Bias & Accountability | First Principles 002: Quantifying Materiality in Automated Ranking