VividKit Crystal clear AI coding
VI
Choose Your Setup Get Started with AgentKit Continue with ClaudeKit Migrate from ClaudeKit to AgentKit Clean cutover from ClaudeKit What Is AgentKit? How AgentKit Works AgentKit Kits, Skills, Agents & Hooks AgentKit Runtime Support AgentKit Desktop App AgentKit Configuration AgentKit CLI Quick Reference AgentKit Skills Cheatsheet AgentKit Workflows AgentKit Statusline Update AgentKit CLI & Kits AgentKit Helper Clean reinstall of AgentKit AgentKit Troubleshooting What is ClaudeKit? ClaudeKit CLI Guide Command Finder Commands Cheat Sheet Workflow Recipes CLI Quick Reference Claude Mechanics Permission Modes ClaudeKit Hooks Coexistence ClaudeKit x Codex CK → Other coding tools How ClaudeKit Works Inside ClaudeKit Finding Your Unknowns Session Recovery Fix from Logs UI/UX Guide Accounts Switcher IDE Config Remote Control Codex App System Diagramming UI Review Gate Promotions Support VividKit

Quick Ref / Command

Marketing Kit

/ak:ab-test-setup

A/B Test Setup

Plan statistically valid experiments with a clear hypothesis, one isolated variable, pre-committed sample size, business-tied metrics, guarded rollout, and reusable learnings.

01

Assess context

02

Frame hypothesis

03

Choose test type

04

Size the test

Rule 01

Start with a data-backed hypothesis, not curiosity-driven tinkering.

Rule 02

Test one meaningful variable so the result stays interpretable.

Rule 03

Pre-commit sample size, duration, primary metric, and guardrails before launch.

Rule 04

Judge both statistical and practical significance before recommending rollout.

Execution Map

Core lanes from input to output, aligned with the command's real execution path.

01

Start

Input, scope, route

  1. 1 Assess context Clarify what should improve, the proposed change, why it matters, baseline conversion, traffic, history, timeline, tools, and implementation complexity.
  2. 2 Frame hypothesis Use the documented Because-we-believe-will-cause-for-we'll-know framework so the experiment predicts an outcome for a defined audience.
  3. 3 Choose test type Pick A/B, A/B/n, multivariate, split URL, or feature-flag rollout based on traffic, implementation risk, and how many variables must be isolated.
02

Work

Agent / skill execution

  1. 4 Size the test Calculate sample size from baseline conversion rate, MDE, significance level, power, number of variants, traffic, and required business cycles.
  2. 5 Select metrics Define one primary metric tied to the hypothesis, secondary metrics for explanation, and guardrail metrics such as revenue, retention, satisfaction, refunds, or activation.
  3. 6 Design variants Document the unchanged control and one meaningful variant; keep the changed variable specific enough that a winner explains what worked.
03

Verify

Gate, review, validation

  1. 7 Plan implementation Choose client-side, server-side, split URL, or percentage rollout; assign traffic evenly or conservatively while preserving repeat visitor consistency.
  2. 8 Pre-launch QA Check hypothesis, metric definitions, sample size, duration, variant rendering, tracking, all variant QA, and stakeholder notice before launch.
  3. 9 Run cleanly Monitor technical issues and segment quality, document outside events, but do not change variants, add new traffic sources, or stop early.
04

Close

Report, handoff, artifact

  1. 10 Analyze and learn Confirm sample size, significance, effect size, secondary support, guardrail impact, and segments; document decision, action, learnings, and next test.

Sample Prompt

Concrete invocations for each flag, subcommand, mode, or route available in the live workflow.

Pricing-page experiment Recommended 
/ak:ab-test-setup pricing page
Use when:
You need a plan for a pricing-page change.
Expected:
A hypothesis, metrics, sample-size needs, variants, rollout method, and analysis plan.
Feature experiment 
/ak:ab-test-setup signup flow
Use when:
A product or funnel feature needs controlled validation.
Expected:
A test design that names implementation approach, traffic split, guardrails, and success criteria.
Checkout CTA test 
/ak:ab-test-setup checkout CTA color
Use when:
A single checkout CTA change needs a statistically valid experiment plan.
Expected:
Hypothesis, one isolated variable, sample size, primary metric, guardrails, and rollout plan.

Handled Scope

  • Hypothesis design
  • Sample-size planning
  • Metric and guardrail selection
  • Variant documentation
  • Experiment analysis

Artifacts Produced

Experiment plan

Pattern

A/B Test: [Name] with hypothesis, design, variants, metrics, implementation, and analysis plan.

The skill's output format is a test plan first, followed by results summary and recommendations when the experiment completes.

Next