Atlas evaluation report

atlas-eval-full Evidence Report

Deterministic retrieval evidence; Luna agent evidence has not been collected.

GATED FAILNeeds work

11d93a5bac6datlas-eval-full72Aug 3, 2026 19:37 UTCrepo-local artifact

Retrieval is correct but slow: 72/72 pass, p95 latency is past the warn ceiling.

Pass gate: 72/72 (100%) (good). Rank: Recall@5 97% (good), MRR 0.82 (good), Recall@1 61% (good). Latency: p95 retrieval latency 1214ms (bad). Safety: Abstain accuracy 100% (good), Forbidden-path accuracy 100% (good). Regression vs baseline: p95 retrieval latency moved by +931ms.

Primary signalp95 retrieval latency

1214msMeasured deterministic result

Executive Summary

Retrieval pass rate

100.0%+0.0ppPass

Recall@5

97.2%+33.1ppPass

Retrieval p95

1214ms+931msFail

Luna paired benchmark

Not collectedRun locally before a releaseInfo

Quality Gates

Only deterministic thresholds gate CI. Local Luna measurements are reported separately.

9 / 9 passed

Status	Layer	Metric	Actual	Required	Delta	Type
✓	Retrieval	Pass rate	100.0%	≥ 98.0%	+0.0pp	Required
✓	Retrieval	Path recall	100.0%	≥ 90.0%	+0.0pp	Required
✓	Retrieval	Term recall	100.0%	≥ 90.0%	+0.0pp	Required
✓	Retrieval	Non-empty context	100.0%	≥ 90.0%	+0.0pp	Required
✓	Retrieval	Recall@5	97.2%	≥ 50.0%	+33.1pp	Required
✓	Retrieval	MRR	0.82	≥ 0.25	+0.52	Required
✓	Retrieval	Abstain accuracy	100.0%	≥ 95.0%	+0.0pp	Required
✓	Retrieval	Forbidden-path accuracy	100.0%	≥ 100.0%	+0.0pp	Required
✓	Retrieval	p95 latency	1214ms	≤ 1500ms	+931ms	Required

Threshold values come from this report’s machine-readable deterministic result.

Cross-Layer Summary

Treatment answer completion and canonical grounding require a local Luna run.

Not collected

No result is shown until a real local Luna snapshot is available.

Failure Analysis

Observed deterministic failures; agent failures are not collected.

0 retrieval failures

No deterministic retrieval failures in this run.

Retrieval Performance

Actual rank, latency, and safety measurements. Only evaluated ranks are plotted.

72/72 deterministic cases passed

Recall@161%+50.5pp

Recall@388%+58.6pp

Recall@597%+33.1pp

MRR0.82+0.52

p95 latency1214ms+931ms

Forbidden-path accuracy100%Safety check

Rank quality

Recall at evaluated ranks

This runBaseline

Latency

Distribution

≤250ms

00%

251-500ms

34%

501-1000ms

5069%

>1000ms

1926%

Quality Gates

Only deterministic thresholds gate CI. Local Luna measurements are reported separately.

9 / 9 passed

Status	Layer	Metric	Actual	Required	Delta	Type
✓	Retrieval	Pass rate	100.0%	≥ 98.0%	+0.0pp	Required
✓	Retrieval	Path recall	100.0%	≥ 90.0%	+0.0pp	Required
✓	Retrieval	Term recall	100.0%	≥ 90.0%	+0.0pp	Required
✓	Retrieval	Non-empty context	100.0%	≥ 90.0%	+0.0pp	Required
✓	Retrieval	Recall@5	97.2%	≥ 50.0%	+33.1pp	Required
✓	Retrieval	MRR	0.82	≥ 0.25	+0.52	Required
✓	Retrieval	Abstain accuracy	100.0%	≥ 95.0%	+0.0pp	Required
✓	Retrieval	Forbidden-path accuracy	100.0%	≥ 100.0%	+0.0pp	Required
✓	Retrieval	p95 latency	1214ms	≤ 1500ms	+931ms	Required

Threshold values come from this report’s machine-readable deterministic result.

72 / 72 casesPage 1

No cases match these filters.

1 of 1

Selected case

maintainer-build-artifact

maintainer-workflow · maintainer

Pass

Query

How does a maintainer build and publish .moxel/atlas artifacts?

Expected behavior

Retrieve the expected source paths and terms while satisfying any negative, diagnostic, confidence, and hit-count expectations.

Evaluation layers

Deterministic result

PassPassed

Recall@5100%

MRR1.00

Latency1136ms

Retrieval evidence

Ranked Sources

3 paths

Rank	Path
1	`docs/ingestion-build-flow.md`
2	`README.md`
3	`docs/runtime-surfaces.md`

Observed result

The case passed deterministic retrieval expectations.

Risk area: workflow-retrieval
Feature: artifact-build
Priority: p0
Ranked hits: 15

atlas-eval-full Evidence Report

Retrieval is correct but slow: 72/72 pass, p95 latency is past the warn ceiling.

Executive Summary

Quality Gates

Cross-Layer Summary

Failure Analysis

Retrieval Performance

Recall at evaluated ranks

Distribution

Quality Gates

Quality Gates

Cross-Layer Summary

Cross-Layer Summary

Retrieval

Retrieval Performance

Recall at evaluated ranks

Distribution

Answer Quality

Luna + Atlas MCP

Failure Analysis

Actual failing cases

Trends & Baselines

Methodology

What this page measures

Reproducibility

Case Explorer

maintainer-build-artifact

Deterministic result

Ranked Sources

Coverage Analysis

Ranking worklist

atlas-eval-full · Evidence Report

Retrieval is correct but slow: 72/72 pass, p95 latency is past the warn ceiling.

Executive Summary

Quality Gates

Cross-Layer Summary

Failure Analysis

Retrieval Performance

Recall at evaluated ranks

Distribution

Quality Gates

Cross-Layer Summary

Retrieval Performance

Recall at evaluated ranks

Distribution

Actual failing cases

What this page measures

maintainer-build-artifact

Deterministic result

Ranked Sources

Quality by capability

Ranking worklist

atlas-eval-full Evidence Report