Atlas evaluation report

atlas-eval-full Evidence Report

Deterministic retrieval evidence; Luna agent evidence has not been collected.

GATED WARNReview advised

2984abb2dd54atlas-eval-full72Aug 23, 2026 16:26 UTCrepo-local artifact

Retrieval is passing with measurable rank headroom: 72/72 pass, ranking metrics in the warn band.

Pass gate: 72/72 (100%) (good). Rank: Recall@5 98% (good), MRR 0.81 (good), Recall@1 58% (warn). Latency: p95 retrieval latency 832ms (warn). Safety: Abstain accuracy 100% (good), Forbidden-path accuracy 100% (good). Regression vs baseline: p95 retrieval latency moved by +549ms.

Primary signalRecall@1

58%Measured deterministic result

Executive Summary

Retrieval pass rate

100.0%+0.0ppPass

Recall@5

97.9%+33.8ppPass

Retrieval p95

832ms+549msWarn

Luna paired benchmark

Not collectedRun locally before a releaseInfo

Quality Gates

Only deterministic thresholds gate CI. Local Luna measurements are reported separately.

9 / 9 passed

Status	Layer	Metric	Actual	Required	Delta	Type
✓	Retrieval	Pass rate	100.0%	≥ 98.0%	+0.0pp	Required
✓	Retrieval	Path recall	100.0%	≥ 90.0%	+0.0pp	Required
✓	Retrieval	Term recall	100.0%	≥ 90.0%	+0.0pp	Required
✓	Retrieval	Non-empty context	100.0%	≥ 90.0%	+0.0pp	Required
✓	Retrieval	Recall@5	97.9%	≥ 50.0%	+33.8pp	Required
✓	Retrieval	MRR	0.81	≥ 0.25	+0.51	Required
✓	Retrieval	Abstain accuracy	100.0%	≥ 95.0%	+0.0pp	Required
✓	Retrieval	Forbidden-path accuracy	100.0%	≥ 100.0%	+0.0pp	Required
✓	Retrieval	p95 latency	832ms	≤ 1500ms	+549ms	Required

Threshold values come from this report’s machine-readable deterministic result.

Cross-Layer Summary

Treatment answer completion and canonical grounding require a local Luna run.

Not collected

No result is shown until a real local Luna snapshot is available.

Failure Analysis

Observed deterministic failures; agent failures are not collected.

0 retrieval failures

No deterministic retrieval failures in this run.

Retrieval Performance

Actual rank, latency, and safety measurements. Only evaluated ranks are plotted.

72/72 deterministic cases passed

Recall@158%+47.7pp

Recall@390%+60.0pp

Recall@598%+33.8pp

MRR0.81+0.51

p95 latency832ms+549ms

Forbidden-path accuracy100%Safety check

Rank quality

Recall at evaluated ranks

This runBaseline

Latency

Distribution

≤250ms

00%

251-500ms

913%

501-1000ms

6388%

>1000ms

00%

Quality Gates

Only deterministic thresholds gate CI. Local Luna measurements are reported separately.

9 / 9 passed

Status	Layer	Metric	Actual	Required	Delta	Type
✓	Retrieval	Pass rate	100.0%	≥ 98.0%	+0.0pp	Required
✓	Retrieval	Path recall	100.0%	≥ 90.0%	+0.0pp	Required
✓	Retrieval	Term recall	100.0%	≥ 90.0%	+0.0pp	Required
✓	Retrieval	Non-empty context	100.0%	≥ 90.0%	+0.0pp	Required
✓	Retrieval	Recall@5	97.9%	≥ 50.0%	+33.8pp	Required
✓	Retrieval	MRR	0.81	≥ 0.25	+0.51	Required
✓	Retrieval	Abstain accuracy	100.0%	≥ 95.0%	+0.0pp	Required
✓	Retrieval	Forbidden-path accuracy	100.0%	≥ 100.0%	+0.0pp	Required
✓	Retrieval	p95 latency	832ms	≤ 1500ms	+549ms	Required

Threshold values come from this report’s machine-readable deterministic result.

72 / 72 casesPage 1

No cases match these filters.

1 of 1

Selected case

maintainer-build-artifact

maintainer-workflow · maintainer

Pass

Query

How does a maintainer build and publish .moxel/atlas artifacts?

Expected behavior

Retrieve the expected source paths and terms while satisfying any negative, diagnostic, confidence, and hit-count expectations.

Evaluation layers

Deterministic result

PassPassed

Recall@5100%

MRR1.00

Latency469ms

Retrieval evidence

Ranked Sources

3 paths

Rank	Path
1	`docs/ingestion-build-flow.md`
2	`README.md`
3	`docs/runtime-surfaces.md`

Observed result

The case passed deterministic retrieval expectations.

Risk area: workflow-retrieval
Feature: artifact-build
Priority: p0
Ranked hits: 12

atlas-eval-full Evidence Report

Retrieval is passing with measurable rank headroom: 72/72 pass, ranking metrics in the warn band.

Executive Summary

Quality Gates

Cross-Layer Summary

Failure Analysis

Retrieval Performance

Recall at evaluated ranks

Distribution

Quality Gates

Quality Gates

Cross-Layer Summary

Cross-Layer Summary

Retrieval

Retrieval Performance

Recall at evaluated ranks

Distribution

Answer Quality

Luna + Atlas MCP

Failure Analysis

Actual failing cases

Trends & Baselines

Methodology

What this page measures

Reproducibility

Case Explorer

maintainer-build-artifact

Deterministic result

Ranked Sources

Coverage Analysis

Ranking worklist

atlas-eval-full · Evidence Report

Retrieval is passing with measurable rank headroom: 72/72 pass, ranking metrics in the warn band.

Executive Summary

Quality Gates

Cross-Layer Summary

Failure Analysis

Retrieval Performance

Recall at evaluated ranks

Distribution

Quality Gates

Cross-Layer Summary

Retrieval Performance

Recall at evaluated ranks

Distribution

Actual failing cases

What this page measures

maintainer-build-artifact

Deterministic result

Ranked Sources

Quality by capability

Ranking worklist

atlas-eval-full Evidence Report