Methodology

Summary

What this methodology optimizes for:

  • Consistency month-to-month
  • High-quality evidence
  • Reliable insight extraction

Scope

What apps we cover

Our library focuses on apps where the product is the experience. Self-sufficient mobile products where growth is driven primarily by:

  • onboarding and activation
  • monetization, offers and paywalls

As of now, we intentionally avoid categories where performance is heavily influenced by external advantages (and therefore are harder to compare or reuse as references), such as:

  • content (news/media/streaming)
  • fintech
  • e-commerce
  • marketplaces

This keeps our insights relevant for teams building high-quality mobile apps without relying on external distribution or content moats.

App analysis methodology

1) Device & environment setup

We record every app with a standardized approach to maximize comparability across apps and over time:

  • App Store region: United States (US)
  • Device language: English (EN-US)
  • Device size class: ~6.1" iPhone (baseline range: iPhone XR → iPhone 16)
  • Text size & display scaling: Default system settings

2) Cadence

We run the full methodology on a monthly cadence. The goal is to get a consistent monthly snapshot that makes app evolution easier to track over time.

3) What is captured

We capture three parts of the experience.

A. First-session flow (end-to-end). From first open → onboarding/setup → until the user lands on the main screen where they are free to explore.

B. Monetization placements reachable in the first session

Every paywall that is reachable from:

  • the first-session flow, and/or
  • the main screen / primary navigation (e.g., tabs)

For each paywall we record:

  • All plan options shown (tiers, durations, trial vs no-trial, etc.)
  • Checkout flow (App Store purchase UI and product names)
  • Web purchase flow, only if it’s offered inside the app

C. Main app exploration

A structured walkthrough of the core product screens: main tabs, key sections, screens that define the value delivery.

4) Authenticated flows

If an app requires an account to access, we capture post-login flows as well. In those cases, the monthly snapshot includes:

  • the first-session experience (what a new user sees),
  • and the authenticated experience (what the product actually is after login)

5) Capture rules (consistency & watchability)

To ensure consistency, during each recorded run:

  • We spend at least 2 seconds on every screen
  • We wait for all animations and transitions to finish before moving forward

This ensures recordings are readable, reviewable, and comparable.

6) Experiment detection

To improve our chances of detecting experiments and segmented experiences, each monthly snapshot is recorded 3 times per app under 3 constraints:

  • Each run starts from a clean install (uninstall → reinstall before every run)
  • All 3 runs are done back-to-back to reduce time-based drift
  • If signup is required, each run uses a different email

7) In-app permissions & privacy

We keep permission handling consistent to avoid branching caused by “maybe allowed / maybe not” decisions:

  • We allow all product permissions (e.g., notifications, location, contacts, etc.) to access the intended experience
  • We do not allow advertising tracking (we never grant permission that would enable IDFA-based tracking)

This preserves privacy constraints while still reflecting the “full product” experience.