78% Wanted This AI Feature. Almost Nobody Used It - Supriya Subudhi, Bumble

Highlights

• 78% of surveyed Bumble users stated they wanted AI feedback on their dating profiles, but only 10 to 15% actually used the feature upon launch.

• Bumble's initial generative AI model was 10 times more expensive than a simpler approach, which made it financially unsustainable at scale.

• Generative AI operational costs increase directly with user success and frequency, unlike traditional machine learning where incremental usage costs remain low.

• AI safety guardrails prevented the feedback tool from telling users to change uncontrollable traits like skin color and height.

• Varying global regulations restricted the AI photo feedback feature to a U.S. launch, preventing an immediate worldwide rollout.

Timeline

1. 00:00 - 05:18

   Initial Research Disconnect and Adoption Realities

   Customer interviews and discovery phases often highlight a strong demand for AI solutions, but expressed interest does not reliably predict actual behavior. When Bumble shipped an AI photo feedback tool as a standalone feature, engagement dropped significantly below survey projections. Embedding the tool directly into existing user actions, such as uploading a picture or troubleshooting a lack of matches, successfully drove adoption by providing contextual awareness.

   • Surveys showed that 78% of users wanted guidance on their dating profiles.
   • An A/B test revealed actual feature usage reached only 10 to 15%.
   • Standalone AI features failed to capture user attention without native workflow integration.
2. 05:19 - 07:45

   Evaluating Average Wins and Guardrails

   Aggregate positive metrics can obscure critical edge cases where generative AI fails or gives redundant advice. Analyzing long-tail data and segment performance across genders and languages revealed unexpected model behaviors. Establishing strict guardrails ensured the AI avoided commenting on uncontrollable physical traits while maintaining accuracy across diverse user demographics.

   • Top-line positive metrics masked flaws like the AI telling users to smile when they were already smiling.
   • Error scope differs vastly between entertainment products like Spotify AI DJ and financial or personal advice apps.
   • Robust evaluation setups tracked real user outputs rather than relying on aggregate green-versus-red dashboards.
3. 07:46 - 11:48

   Balancing Green Dashboards and Rising API Bills

   High accuracy and quality requirements initially pushed product teams to select the most advanced and expensive language models. However, unlike traditional machine learning algorithms with high upfront training costs and low incremental service fees, generative AI incurs continuous per-request expenses that scale with user activity. Organizations must evaluate whether short-term revenue gains or long-term brand equity justifies these ongoing operational costs.

   • Initial reliance on the highest-performing LLM proved 10x more expensive than simpler architectures.
   • Generative AI operational costs scale directly with user growth, contrasting with traditional machine learning compute models.
   • Switching to a cheaper model reduced expenses but simultaneously degraded accuracy and feature impact.
4. 11:49 - 14:36

   Regulatory Compliance and Global Rollouts

   Divergent international legal frameworks create substantial hurdles for deploying generative AI applications worldwide. Projects often launch in a single country like the United States due to immediate compliance viability, forcing teams to halt broader deployment. Product managers and engineers must integrate legal assessments early in the development lifecycle to navigate complex regulatory requirements effectively.

   • AI regulations vary globally, preventing universal feature releases like Meta's Muse AI availability limits.
   • Legal and financial risks require early cross-functional consideration of compliance constraints.
   • Product teams must choose between restricting launches to single compliant regions or building region-specific versions.