---
agent_capabilities:
- Structured Data Analyzer
availability: IDEA
description: Compares performance of control vs. variant groups in an A/B test — using
  metrics like conversion rate — and calculates lift, helping teams make data-backed
  product and design decisions.
name: A/B Test Result Summary
purple_chat_link: https://marketplace.moveworks.com/purple-chat?conversation=%7B%22messages%22%3A%5B%7B%22parts%22%3A%5B%7B%22richText%22%3A%22%3Cp+xmlns%3D%5C%22http%3A%2F%2Fwww.w3.org%2F1999%2Fxhtml%5C%22%3EWhat+were+the+results+of+the+A%2FB+test+on+the+homepage+redesign%3F+Did+the+new+version+lead+to+more+sign-ups%3F%3C%2Fp%3E%22%7D%5D%2C%22role%22%3A%22user%22%7D%2C%7B%22parts%22%3A%5B%7B%22richText%22%3A%22%3Cp%3EI+found+a+few+A%2FB+tests+related+to+%27homepage+redesign%27.+Which+one+are+you+referring+to%3F%3C%2Fp%3E%3Cp%3E1.+Homepage+Redesign+Q2+2024%3C%2Fp%3E%3Cp%3E2.+Homepage+CTA+Button+Color+Test%3C%2Fp%3E%22%7D%5D%2C%22role%22%3A%22assistant%22%7D%2C%7B%22parts%22%3A%5B%7B%22richText%22%3A%22%3Cp%3EFirst+one%3C%2Fp%3E%22%7D%5D%2C%22role%22%3A%22user%22%7D%2C%7B%22parts%22%3A%5B%7B%22reasoningSteps%22%3A%5B%7B%22richText%22%3A%22%3Cp+xmlns%3D%5C%22http%3A%2F%2Fwww.w3.org%2F1999%2Fxhtml%5C%22%3EFound+feature+flag+%3Cb%3Ehomepage-redesign-q2-2024%3C%2Fb%3E+in+%3Cb%3ELaunchDarkly%3C%2Fb%3E%3C%2Fp%3E%22%2C%22status%22%3A%22success%22%7D%2C%7B%22richText%22%3A%22%3Cp+xmlns%3D%5C%22http%3A%2F%2Fwww.w3.org%2F1999%2Fxhtml%5C%22%3EFetching+user+cohorts+and+sign-up+events+from+%3Cb%3EPendo%3C%2Fb%3E%3C%2Fp%3E%22%2C%22status%22%3A%22success%22%7D%2C%7B%22richText%22%3A%22%3Cp+xmlns%3D%5C%22http%3A%2F%2Fwww.w3.org%2F1999%2Fxhtml%5C%22%3EGenerating+Python+script+to+compare+conversion+rates%3C%2Fp%3E%22%2C%22status%22%3A%22success%22%7D%2C%7B%22richText%22%3A%22%3Cp+xmlns%3D%5C%22http%3A%2F%2Fwww.w3.org%2F1999%2Fxhtml%5C%22%3EAnalyzing+results...%3C%2Fp%3E%22%2C%22status%22%3A%22pending%22%7D%5D%7D%2C%7B%22apiBlock%22%3A%7B%22code%22%3A%22%7B%5Cn++%5C%22script%5C%22%3A+%5C%22import+pandas+as+pd%5C%5Cnfrom+scipy+import+stats%5C%5Cn%5C%5Cn%23+Step+1%3A+Get+data+from+previous+actions%5C%5Cn%23+This+data+is+passed+from+the+Pendo+and+LaunchDarkly+API+calls%5C%5Cnlaunchdarkly_flag_data+%3D+actions.launchdarkly_1.get_feature_flag_details.response%5C%5Cnpendo_events_data+%3D+actions.pendo_1.get_events.response%5C%5Cn%5C%5Cn%23+Step+2%3A+Create+DataFrames+from+the+retrieved+data%5C%5Cncontrol_users+%3D+pd.DataFrame%28launchdarkly_flag_data.body.users.control_group%29%5C%5Cnvariant_users+%3D+pd.DataFrame%28launchdarkly_flag_data.body.users.variant_group%29%5C%5Cnsignup_events+%3D+pd.DataFrame%28pendo_events_data.body.events.signups%29%5C%5Cn%5C%5Cn%23+Step+3%3A+Calculate+conversion+rates%5C%5Cncontrol_conversions+%3D+len%28pd.merge%28control_users%2C+signup_events%2C+on%3D%27user_id%27%29%29%5C%5Cnvariant_conversions+%3D+len%28pd.merge%28variant_users%2C+signup_events%2C+on%3D%27user_id%27%29%29%5C%5Cn%5C%5Cncontrol_total+%3D+len%28control_users%29%5C%5Cnvariant_total+%3D+len%28variant_users%29%5C%5Cn%5C%5Cncontrol_rate+%3D+control_conversions+%2F+control_total%5C%5Cnvariant_rate+%3D+variant_conversions+%2F+variant_total%5C%5Cn%5C%5Cn%23+Step+4%3A+Calculate+lift+and+confidence%5C%5Cnlift+%3D+%28%28variant_rate+-+control_rate%29+%2F+control_rate%29%5C%5Cn%5C%5Cnconversions+%3D+%5Bcontrol_conversions%2C+variant_conversions%5D%5C%5Cntotals+%3D+%5Bcontrol_total%2C+variant_total%5D%5C%5Cn_chi2%2C+p_value%2C+_dof%2C+_expected+%3D+stats.chi2_contingency%28%5Bconversions%2C+totals%5D%29%5C%5Cnconfidence+%3D+1+-+p_value%5C%5Cn%5C%5Cn%23+Step+5%3A+Return+results+as+a+JSON+object%5C%5Cnresults+%3D+%7B%5C%5Cn++%27control_rate%27%3A+f%27%7Bcontrol_rate%3A.4f%7D%27%2C%5C%5Cn++%27variant_rate%27%3A+f%27%7Bvariant_rate%3A.4f%7D%27%2C%5C%5Cn++%27lift%27%3A+f%27%7Blift%3A.3f%7D%27%2C%5C%5Cn++%27confidence%27%3A+f%27%7Bconfidence%3A.2f%7D%27%5C%5Cn%7D%5C%5Cn%5C%5Cnprint%28results%29%5C%22%5Cn%7D%22%2C%22caption%22%3A%22This+script+was+generated+to+analyze+the+A%2FB+test+results+by+comparing+user+cohorts+from+Pendo+against+the+feature+flag+data+from+LaunchDarkly.%22%2C%22connectorName%22%3A%22python%22%2C%22title%22%3A%22Generated+Python+Script%22%7D%7D%2C%7B%22richText%22%3A%22%3Cp+xmlns%3D%5C%22http%3A%2F%2Fwww.w3.org%2F1999%2Fxhtml%5C%22%3EThe+new+homepage+design+resulted+in+a+%3Cb%3E15.7%25+lift%3C%2Fb%3E+in+user+sign-ups.+Here+are+the+details+of+the+A%2FB+test%3A%3C%2Fp%3E%3Cul+xmlns%3D%5C%22http%3A%2F%2Fwww.w3.org%2F1999%2Fxhtml%5C%22%3E%3Cli%3E%3Cb%3EControl+Conversion+Rate%3A%3C%2Fb%3E+5.20%25%3C%2Fli%3E%3Cli%3E%3Cb%3EVariant+Conversion+Rate%3A%3C%2Fb%3E+6.02%25%3C%2Fli%3E%3Cli%3E%3Cb%3ELift%3A%3C%2Fb%3E+%2B15.7%25%3C%2Fli%3E%3Cli%3E%3Cb%3EConfidence%3A%3C%2Fb%3E+98%25%3C%2Fli%3E%3C%2Ful%3E%22%7D%2C%7B%22citations%22%3A%5B%7B%22citationTitle%22%3A%22homepage-redesign-q2-2024%22%2C%22connectorName%22%3A%22launchdarkly%22%7D%5D%7D%5D%2C%22role%22%3A%22assistant%22%2C%22showFeedbackTray%22%3Atrue%7D%5D%7D
solution_tags:
- Marketing
systems:
- pendo
- launchdarkly
---