Pre-registration — PROOF Series № 1

Status

Version 1.0, committed 2026-05-14, before data collection. This page summarizes the registered design; the complete pre-registration (full prompt set, covariate operationalization, significance threshold, and power analysis) is available to peer reviewers on request.

Pre-registration means the methodology is fixed and made public before the data is in. It is the discipline that separates empirical research from post-hoc storytelling: it removes the temptation to fit the analysis to the result. This page states the design we have committed to for PROOF Series № 1.

Hypothesis

California law firms whose public-facing content materially complies with Cal Bar Rule 7.1 are cited more frequently in AI-search answers than firms with documented compliance gaps — controlling for domain authority, content velocity, practice-area mix, and firm size.

This is a hypothesis grounded in pilot observation and the general direction of RLHF reward modeling — not a finding. The study tests it and publishes whether the result is positive, null, or negative.

Sample frame

200 California law firms: 100 classified as materially compliant and 100 with documented compliance gaps, stratified by practice area and firm size. The inclusion and exclusion criteria and the sampling procedure are fixed in the full pre-registration.

Prompt set

50 buyer-intent prompts distributed across five practice areas, with high-intent, sub-intent, language-specific, and informational variants. The full prompt list is frozen before data collection so it cannot be tuned to results.

Outcome measures

Primary outcome: citation rate — the share of relevant prompts on which a firm is cited — measured across ChatGPT, Perplexity, Google AI Overviews, and Claude over the collection window. Secondary outcomes (engine-level variation and longitudinal stability) are specified in the full document.

Analysis plan and falsification

The compliance–citation relationship is estimated with the firm-level covariates named in the hypothesis entered as controls. The model specification and the pre-committed significance threshold are fixed before data collection.

The hypothesis is not supported if the compliance term is non-significant or negative under that pre-committed analysis. The null result publishes either way.

Timeline

The study runs on a committed schedule:

2026-05-14 — Pre-registration v1.0 and coding rubric v1.0 published.
2026-05-15 to 06-30 — Sample-frame construction, coder training, inter-rater reliability check.
2026-07-01 to 08-29 — 60 consecutive days of data collection across all four engines.
2026-08-30 to 09-14 — Analysis, robustness checks, writing.
2026-09-15 (target) — Preprint published; submission to a peer-reviewed venue.

Ethics and conflict of interest

The principal investigator (Shawn Lai) founded Legal Torch, which would commercially benefit from a positive result. This conflict is disclosed up front. Mitigations: the analysis plan is fixed before data is in; compliance coding is performed by coders blind to the hypothesis; the commitment to publish the null result is on the record; and the raw dataset and analysis code are published with the paper to enable independent replication.

Full document

The complete pre-registration — full prompt list, covariate operationalization, significance threshold, and power analysis — is available to peer reviewers, journalists, and replicators on request at info@legaltorch.ai.