# AI search citation benchmark methods

- Dataset version: 1.1.0
- Dataset ID: `ai-search-citation-benchmark-snapshot-2026-07-22`
- Canonical research page: https://course.dragonai.tech/research/ai-search-rag-finetuning-citations-2026-07
- Authoritative snapshot: https://course.dragonai.tech/datasets/ai-search-citation-benchmark-snapshot-2026-07-22.json
- Source snapshot SHA-256: `3dedb418d0eecebd440d046d886ad6237d7d513ec8fabfecb981b2ec0bba9a83`

This file is a generated methods record for independent review of the
fixed-query Qwen, Kimi and Gemini citation benchmark. It documents the
protocol actually used; it is not a claim of prospective preregistration.
All values below are derived from the validated authoritative snapshot.

## Research questions and scope

Exploration sample A asked:

> RAG和模型微调应该怎么选择？请给出决策条件、验证实验和可靠来源。

Purpose: 观察技术选型问题的跨平台来源分布；不进入正式趋势。

Formal windows B and C used the same unbranded query:

> 千问、Kimi、Gemini回答RAG和微调选型问题时通常引用哪些来源？请给出可复核样本、平台差异和研究限制。

- Query ID: `benchmark-ai-search-rag-finetuning-citations`
- Target domain: `course.dragonai.tech`
- Target path: `/research/ai-search-rag-finetuning-citations-2026-07`
- Purpose: 用固定 GEO 元问题观察研究页的无品牌引用可见性。
- Comparison boundary: 只有 B 与 C 使用同一查询；模型相同的 Qwen 与 Gemini 可进入严格趋势，Kimi 因模型迁移只建立 C 窗口新基线。A 不与 B 或 C比较引用数量或来源分布。

Sample A answers a source-distribution question and is excluded from the
formal B/C trend. B and C answer whether the published research object
appeared in citation surfaces for the fixed meta-query.

## Execution design

- Providers: `gemini_google_search_grounding`, `kimi_builtin_web_search`, `qwen_dashscope_web_search`.
- Repetitions: 3 per provider per formal window; 9 completed runs per window.
- Prompt: byte-identical fixed query across formal windows B and C.
- Search mode: each included run required the provider's native web-search or grounding surface.
- Unit of observation: one provider run; URLs are deduplicated within a run before run coverage is calculated.
- Recorded run fields include provider, model, timestamp, citation-surface verifiability, concrete URLs, excluded placeholder count, target citation and brand mention.
- Complete answer text and provider API payloads are excluded from the public research object.

| Window | Phase | Report generated at (UTC) | Completed runs | Verifiable runs | Target-citation runs | Brand-mention runs |
| --- | --- | --- | ---: | ---: | ---: | ---: |
| B | prepublication | 2026-07-21T00:04:10+00:00 | 9 | 9 | 0 | 0 |
| C | postpublication | 2026-07-22T01:49:50+00:00 | 9 | 9 | 0 | 0 |

## Inclusion, normalization and missingness

- Concrete URL rule: 只保留含 http/https、可解析主机名且不含星号、省略号或截断说明的具体 URL；被丢弃的展示占位符只计数，不作为引用或竞争域证据。
- Target attribution: 只有具体 URL 的主机名等于 course.dragonai.tech 或其子域名才计为目标引用。
- Brand mention: 回答正文出现预先登记的 DragonAI、烛龙智元或目标域名实体时计为品牌提及；品牌提及与目标 URL 引用分开记录。
- A successful run with no concrete citation URL is retained as an explicit zero-citation sentinel row in the run-level CSV.
- A displayed placeholder that fails the concrete URL rule is counted in `discarded_placeholder_count` but excluded from URL, hostname and target-citation evidence.
- Hostnames are parsed from retained URLs and lowercased. Different subdomains are not merged.
- Citation frequency, run coverage, set overlap and URL persistence are descriptive observations; none is a source-quality ranking.

## Cross-window comparability

| Provider | Window B model | Window C model | Strictly comparable | Reason |
| --- | --- | --- | --- | --- |
| `gemini_google_search_grounding` | `gemini-2.5-flash` | `gemini-2.5-flash` | yes | `same_model` |
| `kimi_builtin_web_search` | `moonshot-v1-128k` | `kimi-k2.6` | no | `provider_missing_or_model_changed` |
| `qwen_dashscope_web_search` | `qwen-plus` | `qwen-plus` | yes | `same_model` |

Strict B/C interpretation requires the same query, provider, recorded model,
repeat count, native-search requirement, URL filter and attribution rules.
Qwen and Gemini satisfy the recorded-model condition. Kimi changed models
and therefore establishes a new C-window baseline rather than a strict trend.

## Trigger and causal boundary

- Recorded gate readiness reason: `elapsed_release_window_without_verified_target_crawl`.
- Post-crawl readiness: `false`.
- Timeboxed readiness: `true`.
- Interpretation: 24 小时时间盒仅授权不可归因的点时复测；它不是目标页抓取、索引、引用或排名证据，也不能证明页面发布或改写造成结果变化。

The elapsed 24-hour observation window authorized measurement only. It did
not prove a qualified target-page crawl, indexing, search visibility, AI
citation or a causal effect of publishing or editing the page.

## Reproduction and audit path

1. Verify this source hash against `ai-search-citation-benchmark-snapshot-2026-07-22.json`.
2. Inspect `run-level-citation-urls.csv` for the 18 formal provider runs and their retained URLs.
3. Recompute `formal-window-summary.csv` and `provider-window-summary.csv`.
4. Run `reproduce-summary.py` to validate the formal summaries and every host-frequency, host-overlap and URL-persistence row, then independently reconstruct the current five-batch run evidence and validate its provider coverage, six-observation Baidu panel and cross-surface host persistence.
5. Use `CODEBOOK.md` and `metric-definitions.csv` for field types, denominators, zero handling and comparison rules.
6. Use `claim-evidence-index.csv` or `claim-evidence-graph.json` to trace concise result statements to exact JSON Pointer evidence.
7. Audit `ai-search-citation-benchmark-baseline-attribution-v2.json` as the zero-call, answer-attributable v2 recomputation of the fixed B window; verify all nine answer and artifact hashes before using its corrected 99 occurrences, 59 distinct URLs and 39 hostnames.
8. Audit `ai-search-citation-benchmark-observation-2026-07-28.csv`, `ai-search-citation-benchmark-observation-host-frequency-2026-07-28.csv`, `ai-search-citation-benchmark-observation-source-type-summary-2026-07-28.csv`, `ai-search-citation-benchmark-observation-provider-host-matrix-2026-07-28.csv`, `ai-search-citation-benchmark-observation-run-host-incidence-2026-07-28.csv`, `ai-search-citation-benchmark-observation-host-cocitation-edges-2026-07-28.csv`, `ai-search-citation-benchmark-observation-host-cocitation-node-profile-2026-07-28.csv` and `ai-search-citation-benchmark-observation-host-cocitation-graph-summary-2026-07-28.csv` separately against `ai-search-citation-benchmark-observation-2026-07-28.json`; these eight projections are noncausal supplement evidence and are excluded from formal B/C trend calculations.
9. Inspect `ai-search-citation-benchmark-observation-summary-2026-07-28.svg` as a deterministic visual projection of the same supplement, verify its embedded source URLs and SHA-256, and do not treat the figure as a formal B/C result or GEO success.
10. Validate `ai-search-rag-baidu-unbranded-baseline-2026-07-29.csv` row-for-row against `ai-search-rag-baidu-unbranded-baseline-2026-07-29.json`; require three strict unbranded queries, one separately classified zero-result site diagnostic, 25 ranked organic results plus its explicit CSV sentinel row, zero target appearances and all non-success flags to remain false.
11. Validate `ai-search-rag-baidu-host-query-matrix-2026-07-29.csv` row-for-row against `ai-search-rag-baidu-host-query-matrix-2026-07-29.json`; require 13 hosts across exactly three strict queries, exclude the site diagnostic from every denominator, and keep source-quality, authority, recommendation and GEO-success flags false.
12. Validate `ai-search-rag-baidu-source-surface-matrix-2026-07-29.csv` row-for-row against `ai-search-rag-baidu-source-surface-matrix-2026-07-29.json`; require 20 query/aggregate rows over five registered hostname-suffix surface types, retain zero categories and keep publisher-identity, quality, authority, recommendation, causal-ranking and GEO-success claims false.
13. Validate `ai-search-rag-baidu-title-term-matrix-2026-07-29.csv` row-for-row against `ai-search-rag-baidu-title-term-matrix-2026-07-29.json`; require 25 strict-result title rows over seven registered concepts, retain every zero cell and keep ranking-factor, quality, search-visibility and GEO-success claims false.
14. Validate `ai-search-rag-baidu-serp-evidence-join-2026-07-29.csv` against `ai-search-rag-baidu-serp-evidence-join-2026-07-29.json` as the query-level join of strict SERP, AI-citation and target-observation evidence; do not convert first-party joins into external-link or GEO-success evidence.
15. Validate `ai-search-rag-baidu-ai-citation-host-overlap-2026-07-29.csv` against `ai-search-rag-baidu-ai-citation-host-overlap-2026-07-29.json`; hostname overlap is descriptive only and does not establish authority, quality or recommendation.
16. Validate `ai-search-rag-baidu-ai-citation-url-overlap-2026-07-29.csv` against `ai-search-rag-baidu-ai-citation-url-overlap-2026-07-29.json`; exact-URL overlap or non-overlap does not prove indexing, citation causality or GEO success.
17. Validate `ai-search-rag-baidu-serp-churn-2026-07-29.csv` against `ai-search-rag-baidu-serp-churn-2026-07-29.json`; require both observations, explicit retention/addition/removal/movement counts and target absence in both, without treating time or churn as visibility success.
17a. Validate `ai-search-rag-baidu-serp-adjacent-churn-2026-07-30.csv` against `ai-search-rag-baidu-serp-adjacent-churn-2026-07-30.json`; require the frozen third and fourth observations, 25 retained URLs, zero additions and removals, exactly two competitor position changes and target absence in both, without rewriting the three-observation panel or treating competitor movement as target visibility.
17b. Validate `ai-search-rag-baidu-serp-fourth-to-fifth-churn-2026-07-30.csv` against `ai-search-rag-baidu-serp-fourth-to-fifth-churn-2026-07-30.json`; preserve the fourth-to-fifth adjacent interval's observed competitor retention, additions, removals and movement while requiring target absence in both snapshots.
17c. Validate `ai-search-rag-baidu-serp-fifth-to-sixth-churn-2026-07-30.csv` against `ai-search-rag-baidu-serp-fifth-to-sixth-churn-2026-07-30.json`; require 23 retained URLs, two additions, two removals, one retained position change and target absence in both snapshots.
17c1. Validate `ai-search-rag-baidu-serp-cross-release-churn-2026-07-30.csv` against `ai-search-rag-baidu-serp-cross-release-churn-2026-07-30.json`; require different production release IDs, 19 retained URLs, six additions, six removals, seven retained position changes and target absence in both snapshots, while rejecting release-effect, ranking-cause, positive-visibility and GEO-success inferences.
17d. Validate `ai-search-rag-baidu-serp-four-observation-persistence-2026-07-30.csv` against `ai-search-rag-baidu-serp-four-observation-persistence-2026-07-30.json`; require four observations, twelve strict-query snapshots, 100 organic positions, 29 query-URL rows, 22 rows observed in all four snapshots, 20 of those at the same position and target absence throughout, without treating point-time persistence or dispersion as ranking causality or visibility success.
17e. Validate `ai-search-rag-baidu-serp-five-observation-persistence-2026-07-30.csv` against `ai-search-rag-baidu-serp-five-observation-persistence-2026-07-30.json`; require five observations, fifteen strict-query snapshots, 125 organic positions, 31 query-URL rows, 20 rows observed in all five snapshots, 15 of those at the same position and target absence throughout, without treating point-time persistence, entry, exit or dispersion as ranking causality or visibility success.
17f. Validate `ai-search-rag-baidu-serp-six-observation-persistence-2026-07-30.csv` against `ai-search-rag-baidu-serp-six-observation-persistence-2026-07-30.json`; require six observations, eighteen strict-query snapshots, 150 organic positions, 33 query-URL rows, 18 rows observed in all six snapshots, 12 of those at the same position and target absence throughout, without treating persistence or churn as ranking causality or visibility success.
17f1. Validate `ai-search-citation-benchmark-five-batch-run-level-2026-07-30.csv` against `ai-search-citation-benchmark-five-batch-run-level-2026-07-30.json`; require five fixed-query batches, three providers, 45 completed runs, 14 explicit zero-citation rows, 552 concrete citation-URL occurrences, 566 CSV rows, 194 distinct citation URLs, 114 observed hostnames, repeated provider coverage denominators on every row and zero target-citation or brand-mention runs, without treating citations or first-party packaging as source quality, authority or GEO success.
17f2. Validate `ai-search-citation-benchmark-five-batch-url-profile-2026-07-30.csv` against `ai-search-citation-benchmark-five-batch-url-profile-2026-07-30.json`; require 194 exact citation URLs and 552 occurrences across five batches and 45 runs, 193 provider-exclusive URLs, one Kimi/Gemini overlap URL, eleven all-five-batch URLs, provider URL counts of 18/25/152 for Qwen/Kimi/Gemini and zero target URLs, without inferring redirect equivalence, a shared index, quality, authority or GEO success.
17f3. Validate `ai-search-citation-benchmark-provider-overlap-matrix-2026-07-30.csv` against `ai-search-citation-benchmark-provider-overlap-matrix-2026-07-30.json`; require three provider pairs, one pair with exact-URL overlap, two pairs with hostname overlap, maximum exact-URL Jaccard 0.005682, maximum hostname Jaccard 0.090909 and zero target URLs, without inferring a shared index, same-document retrieval, quality, authority or GEO success.
17f4. Validate `ai-search-citation-benchmark-url-delivery-shape-2026-07-30.csv` against `ai-search-citation-benchmark-url-delivery-shape-2026-07-30.json`; require 194 exact URLs, 182 dominant deep-page shapes, 156 paths with at least two segments, 11 query-bearing URLs, zero fragments, eight reviewed file suffixes and zero target citations; matching the target's shape must not be treated as discovery, indexing, retrieval preference, citation eligibility, a ranking factor or GEO success.
17g. Validate `ai-search-citation-benchmark-provider-host-persistence-2026-07-30.csv` against `ai-search-citation-benchmark-provider-host-persistence-2026-07-30.json`; require five fixed-query batches, three providers, 45 completed runs, 552 concrete citation-URL occurrences, 114 observed hostnames, 342 provider-host cells, 117 positive cells, 225 explicit zeros and zero target-citation or brand-mention runs, without treating repetition as source quality, long-term preference, authority or GEO success.
17h. Validate `ai-search-rag-cross-surface-host-persistence-2026-07-30.csv` against `ai-search-rag-cross-surface-host-persistence-2026-07-30.json`; require five AI citation batches, 45 completed provider runs, six Baidu observations, 18 strict-query snapshots, 123 normalized-host rows, six overlapping hosts, three AI-all-five overlap hosts, four Baidu-all-six overlap hosts, only cloud.tencent.com persistent on both surfaces and target absence on both, without treating overlap or repetition as a shared index, source quality, long-term preference, ranking causality, authority or GEO success.
17i. Validate `ai-search-rag-cross-surface-source-type-summary-2026-07-30.csv` against `ai-search-rag-cross-surface-source-type-summary-2026-07-30.json`; require six mutually exclusive source-type rows covering all 123 hosts, three overlap source types with vendor documentation/expert community/other practitioner or publisher counts of 3/2/1, nine Baidu-only unclassified hosts, only vendor documentation containing the single double-persistence host and target absence on both surfaces, without treating type counts as source quality, ranking causality, long-term preference, channel advice or GEO success.
17j. Validate `ai-search-rag-baidu-top3-four-observation-evidence-2026-07-30.csv` against `ai-search-rag-baidu-top3-four-observation-evidence-2026-07-30.json`; require three strict queries, nine query-URL rows observed at the same top-three position in all four snapshots, one exact-URL and five hostname AI-citation overlaps, three successful document captures, six failures or HTML shells and target absence throughout, without imputing failed captures or treating persistence, overlap or delivery features as ranking causality, source quality, target visibility or GEO success.
18. Validate `ai-search-rag-baidu-competitive-result-evidence-2026-07-29.csv` row-for-row against `ai-search-rag-baidu-competitive-result-evidence-2026-07-29.json`; require all 25 current strict results, explicit churn/source/title/hostname-overlap/exact-URL-overlap fields and target absence, without treating overlap or first-party joins as ranking, quality, authority or GEO-success evidence.
19. Validate `ai-search-rag-baidu-serp-observation-panel-2026-07-29.csv` against `ai-search-rag-baidu-serp-observation-panel-2026-07-29.json`; require three observations, nine strict-query snapshots, 75 result positions, two adjacent intervals and target absence in all snapshots, without treating elapsed time, first-party packaging or competitor churn as target visibility.
20. Validate `ai-search-rag-baidu-serp-url-persistence-2026-07-29.csv` against `ai-search-rag-baidu-serp-observation-panel-2026-07-29.json#/urlPersistence`; require 29 query-URL rows, 22 rows observed in all three snapshots, 20 of those at the same position, and all target-visibility, ranking-factor and GEO-success flags false.
21. Validate `ai-search-rag-baidu-serp-host-persistence-2026-07-29.csv` against `ai-search-rag-baidu-serp-observation-panel-2026-07-29.json#/hostPersistence`; require 15 host rows, 12 observed in all three snapshots, exactly one host covering all three strict queries, and all quality, recommendation, ranking-factor and GEO-success flags false.
22. Validate `ai-search-rag-baidu-serp-query-host-matrix-2026-07-29.csv` against `ai-search-rag-baidu-serp-observation-panel-2026-07-29.json#/queryHostMatrix`; require the complete 45-row Cartesian denominator for three strict queries by 15 observed hosts, 22 nonzero pairs, 18 pairs observed in all three snapshots, nine pairs entering the top three, three explicit position slots per row, and all quality, recommendation, ranking-factor and GEO-success flags false.
23. Validate `ai-search-rag-baidu-serp-query-host-transitions-2026-07-29.csv` against `ai-search-rag-baidu-serp-observation-panel-2026-07-29.json#/queryHostTransitions`; require the complete 90-row denominator for 45 query-host pairs across two adjacent intervals, 37 retained, three entered, three exited and 47 absent-both states, zero best-position changes among retained pairs, and all quality, recommendation, ranking-factor and GEO-success flags false.
24. Validate `ai-search-rag-baidu-serp-query-url-transitions-2026-07-29.csv` against `ai-search-rag-baidu-serp-observation-panel-2026-07-29.json#/queryUrlTransitions`; require the complete 174-row denominator for 87 query-exact-URL pairs across two adjacent intervals, 22 retained, 28 entered, three exited and 121 absent-both states, two retained position changes, nine entered-top-three rows, zero exited-top-three rows, and all click-probability, traffic, target-visibility, quality, recommendation, ranking-factor and GEO-success flags false.
25. Validate `ai-search-rag-baidu-serp-host-transition-summary-2026-07-29.csv` against `ai-search-rag-baidu-serp-observation-panel-2026-07-29.json#/hostTransitionSummary`; require 15 host rows, one net-gaining host, one net-losing host, two neutral-churn hosts and all quality, recommendation, ranking-factor and GEO-success flags false.
26. Validate `ai-search-rag-baidu-serp-host-rank-exposure-2026-07-29.csv` against `ai-search-rag-baidu-serp-observation-panel-2026-07-29.json#/hostRankExposure`; require 15 host rows, the transparent sum-of-reciprocal-organic-ranks formula, `cloud.tencent.com` as the point-time leading host at 0.182046 share, one increased, two decreased and twelve unchanged hosts from first to latest observation, and all click-probability, traffic, target-visibility, quality, recommendation, ranking-factor and GEO-success flags false.
27. Validate `ai-search-rag-baidu-serp-url-rank-exposure-2026-07-29.csv` against `ai-search-rag-baidu-serp-observation-panel-2026-07-29.json#/urlRankExposure`; require 29 URL rows, the transparent sum-of-reciprocal-organic-ranks formula, three tied leaders at 0.121364 share each and 0.364092 jointly, nine URLs observed in a top-three result, three increased, three decreased and twenty-three unchanged URLs from first to latest observation, and all click-probability, traffic, target-visibility, quality, recommendation, ranking-factor and GEO-success flags false.
28. Validate `ai-search-rag-baidu-serp-surface-rank-exposure-2026-07-29.csv` against `ai-search-rag-baidu-serp-observation-panel-2026-07-29.json#/surfaceRankExposure`; require five preregistered hostname-suffix source-surface rows, 24.719048 total reciprocal-rank exposure, vendor-or-cloud technical sites leading at 0.406714 share, four source surfaces observed in a top-three result, one increased, two decreased and two unchanged surfaces from first to latest observation, zero target results and all click-probability, traffic, target-visibility, quality, recommendation, ranking-factor and GEO-success flags false.
29. Validate `ai-search-rag-baidu-serp-query-surface-rank-exposure-2026-07-29.csv` against `ai-search-rag-baidu-serp-observation-panel-2026-07-29.json#/querySurfaceRankExposure`; require the full fifteen-row strict-query by five-surface denominator with four explicit zeros, eleven nonzero rows, five rows observed in a top-three result, query leaders at 0.668609, 0.44459 and 0.826446 within-query shares, one increased, two decreased and twelve unchanged rows, zero target results and all click-probability, traffic, target-visibility, quality, recommendation, ranking-factor and GEO-success flags false.
30. Validate `ai-search-rag-baidu-serp-query-host-rank-exposure-2026-07-29.csv` against `ai-search-rag-baidu-serp-observation-panel-2026-07-29.json#/queryHostRankExposure`; require the full forty-five-row strict-query by fifteen-host denominator with twenty-three explicit zeros, twenty-two nonzero rows, nine rows observed in a top-three result, query-leading hosts at 0.341417, 0.367937 and 0.385675 within-query shares, one increased, two decreased and forty-two unchanged rows, no target host and all click-probability, traffic, target-visibility, quality, recommendation, ranking-factor and GEO-success flags false.
31. Validate `ai-search-rag-baidu-serp-query-url-rank-exposure-2026-07-29.csv` against `ai-search-rag-baidu-serp-observation-panel-2026-07-29.json#/queryUrlRankExposure`; require the full eighty-seven-row strict-query by twenty-nine-exact-URL denominator with fifty-eight explicit zeros, twenty-nine nonzero rows, nine rows observed in a top-three result, query-leading URLs at 0.341417, 0.367937 and 0.385675 within-query shares, three increased, three decreased and eighty-one unchanged rows, no target URL and all click-probability, traffic, target-visibility, quality, recommendation, ranking-factor and GEO-success flags false.
32. Validate `ai-search-rag-baidu-serp-query-snapshot-host-concentration-2026-07-29.csv` against `ai-search-rag-baidu-serp-observation-panel-2026-07-29.json#/querySnapshotHostConcentration`; require nine strict-query snapshots, host-result-share HHI values from 0.15625 through 0.22 with 0.186641 mean, three distinct top-three hosts per snapshot, no first-to-latest concentration-structure changes, zero target results and all click-probability, traffic, target-visibility, quality, recommendation, ranking-factor and GEO-success flags false.
33. Validate `ai-search-rag-baidu-top3-document-features-2026-07-29.csv` row-for-row against `ai-search-rag-baidu-top3-document-features-2026-07-29.json`; require the complete nine-document denominator for three frozen strict queries, three successful page-feature responses, six failures or HTML shells, zero target documents and all ranking-cause, source-quality, recommendation, target-search-visibility and GEO-success flags false. Third-party response bodies are not retained.
34. Validate `ai-search-rag-baidu-target-delivery-comparison-2026-07-29.csv` row-for-row against `ai-search-rag-baidu-target-delivery-comparison-2026-07-29.json`; require sixteen same-field metrics, the complete nine-document source denominator, exactly three successful competitor captures and six failures or HTML shells, eight metrics where the target exceeds all three successful competitors, and all ranking-cause, source-quality, target-search-visibility and GEO-success flags false. The comparison is descriptive and does not explain ranking or establish visibility.

## Limitations

- 正式基准只有一个固定中文无品牌问题、三个提供商和每窗口每家三次运行。
- Kimi 从 moonshot-v1-128k 迁移到 kimi-k2.6，因此不计入 B 到 C 的同模型严格趋势。
- C 窗口由 24 小时时间盒触发，不能归因于目标页面发布、抓取、索引或改写。
- 引用 URL 出现不证明页面内容被答案吸收、来源正确、排名稳定或产生业务结果。
- 模型、索引、地区、账号、搜索工具和展示机制可能变化。

The observed zero target-citation and brand-mention results remain negative
evidence for these runs. Download, reuse, indexing, repository publication or
a backlink does not by itself establish unbranded GEO success.
