This document explains the confidence metrics displayed in the Alvie platform for Alvie attribution results. These metrics help you assess the relative quality and reliability indicators for attribution results across marketing channels.
Important Note: All attribution models involve inherent uncertainty and rely on assumptions. Additionally, the quality of attribution results depends heavily on the quality of data going in - accurate tracking, consistent data collection, and reliable measurement are essential foundations for meaningful results. These confidence metrics indicate relative reliability within the model, but should always be considered alongside other business data and judgment when making marketing decisions.
Overview
The Alvie dashboard shows 5 confidence metrics for each channel:
Channel Confidence - Overall reliability score for the channel
Quality of Fit - How precisely we can measure the channel's impact
Data Sufficiency - Whether there's enough quality data for this channel
Activity Variation - Whether the channel has enough activity variation to identify its effect
Stability Score - How consistent the results are across different model runs
All metrics are scored from Low to High where higher scores indicate better quality.
What To Do About Low Scores
Low Metric | What It Means | What You Can Do |
Data Sufficiency | Not enough activity data | Ensure channel runs more consistently; avoid long gaps in channel activity |
Activity Variation | Too flat or too sparse | For paid channels: vary spending levels; try different activity patterns instead of always-on at the same level |
Quality of Fit | Model struggles with this channel | Check data quality; ensure accurate tracking; may improve as overall model improves |
Stability Score | Results fluctuate too much | Investigate data quality issues; ensure consistent tracking; check for external factors causing instability |
Note on Channel Structure: How you define and structure your channels can significantly impact confidence metrics. Having too many narrowly-defined channels can lead to fragmented data with low data sufficiency scores for individual channels. Conversely, consolidating too many activities into a single broad channel can make it difficult for the model to separate distinct signals and effects. Finding the right balance is key - you have control over how channels are defined in your setup, and thoughtful channel grouping can improve both attribution clarity and confidence scores.
Detailed Metric Descriptions
1. Channel Confidence
What You See in Alvie:
Main confidence slider from "Low" to "High"
What It Means:
This is the overall reliability indicator for a channel's attribution results. It combines multiple quality factors into one score that helps you assess the relative strength of the model's attribution results for this channel. Channels with higher channel confidence scores have stronger supporting evidence. However, these metrics show model-based reliability - always combine them with your own knowledge of campaign performance, market conditions, and business context when making decisions.
How It's Calculated:
Channel confidence weighs four quality factors:
Quality of Fit: 30%
Activity Variation: 25%
Data Sufficiency: 25%
Stability Score: 20% (when available)
If Stability Score is N/A (insufficient historical data), weights auto-redistribute across the other three metrics: Quality of Fit (37.5%), Activity Variation (31.25%), Data Sufficiency (31.25%).
How to Read the Score:
High: Stronger supporting evidence across quality factors; attribution results have relatively lower uncertainty
Medium: Moderate supporting evidence; attribution results should be supplemented with additional validation
Low: Weaker supporting evidence; attribution results have higher uncertainty and require careful validation
2. Quality of Fit
What You See in Alvie:
Labeled indicator showing Low/Medium/High
What It Means:
Quality of fit indicates the reliability of this channel's attribution results. It combines two factors: (1) how narrow the uncertainty range is around the channel's ROAS (Return On Ad Spend) / ROS (Return On Session) value, and (2) how well the overall model performs in predicting the model target (e.g., conversions, revenue).
Understanding Uncertainty Ranges:
Instead of giving just a single ROAS number (e.g., "exactly 2.5"), the model provides a range of likely values (e.g., "between 2.0 and 3.0"). This reflects the inherent uncertainty in attribution. Think of it like a weather forecast - saying "21-24°C" is less certain than "22-23°C". Narrower ranges mean the model is more confident about where the true value lies. This approach gives you a more honest picture of what the data can reliably tell us.
A higher quality of fit indicates that the attribution result has relatively narrower uncertainty ranges (more precision) and the model shows better overall performance. A lower score indicates wider uncertainty ranges around the channel's attributed impact.
How to Read the Score:
High: Relatively narrower uncertainty range; lower uncertainty
Medium: Moderate uncertainty - wider uncertainty range
Low: Wider uncertainty range; higher uncertainty
3. Data Sufficiency
What You See in Alvie:
Labeled indicator showing Low/Medium/High
What It Means:
Data sufficiency indicates whether there's sufficient quality data to support more reliable estimation of this channel's effect. The model evaluates three aspects of your data:
How many days the channel was active - More active days provide more data to learn from
How consistently the channel appeared - Channels active every week are better than channels with long gaps
Data quality and reliability - Predictable patterns without extreme outlier spikes that might indicate data quality issues
What Causes Low Scores:
Channel only active for a few days/weeks
Long gaps with no activity
Erratic data with extreme outlier spikes (may indicate data quality issues)
How to Read the Score:
High: More consistent data with regular activity over time
Medium: Adequate data but with some gaps or irregularities
Low: Limited data - sparse or highly irregular activity
4. Activity Variation
What You See in Alvie:
Labeled indicator showing Low/Medium/High
What It Means:
Activity variation measures whether the channel's activity patterns have enough variety for the model to identify its effect. To understand a channel's impact, we need to see what happens when activity goes up, down, or stops completely. Flat, constant activity makes it nearly impossible to separate the channel's effect from other factors.
Note: For paid channels this uses spend data; when spend isn't available, it uses sessions data.
The model evaluates three aspects of activity variation:
Day-to-day volatility - How much activity changes from one day to the next (more variation is better)
Relative spread in activity levels - Whether activity levels vary significantly over time (not just flat)
Activity consistency - Whether the channel shows regular activity rather than being mostly inactive
What Causes Low Scores:
Always the same activity level every day (flat line)
Almost always zero activity (too sparse)
What Leads To High Scores:
Changing activity levels over time
Day-to-day variation in activity amounts
Mix of high-activity and low-activity days
Regular activity patterns rather than constant inactivity
How to Read the Score:
High: More informative variation - clearer activity patterns that support effect identification
Medium: Some variation present but could be improved with more activity changes
Low: Limited variation - mostly flat, constant, or very sparse
5. Stability Score
What You See in Alvie:
Labeled indicator showing Low/Medium/High (or N/A for new channels)
What It Means:
Stability score shows how consistent the channel's ROAS/ROS attribution results are across different weekly model runs over the past 12 weeks. More stable channels show similar results week after week. Less stable channels have results that vary significantly, which might signal data quality issues or structural challenges in isolating the channel's effect.
The score looks at whether the channel's ROAS/ROS stays relatively steady or fluctuates dramatically between model runs.
Why It Matters:
If a channel's ROAS attribution result is 3.0 one week and 1.5 the next week (with no real business change), this suggests higher uncertainty. More stable patterns indicate the model produces more consistent attribution results over time, though stability alone doesn't guarantee accuracy.
When You'll See "N/A":
Stability requires historical data from at least 3 recent model runs. New channels or channels that haven't been modeled before will show "N/A" until enough history builds up.
How to Read the Score:
High: More consistent results across model runs
Medium: Moderate consistency with some fluctuation between runs
Low: Less consistent - notable fluctuations between runs
N/A: Insufficient historical data (fewer than 3 recent model runs)