Requirements Traceability & Gate Decision - Validation Checklist

Workflow: testarch-trace Purpose: Ensure complete traceability matrix with actionable gap analysis AND make deployment readiness decision (PASS/CONCERNS/FAIL/WAIVED)

This checklist covers two sequential phases:

PHASE 1: Requirements Traceability (always executed)
PHASE 2: Quality Gate Decision (decision fields emitted only when allow_gate: true and the collection is gate-eligible)

PHASE 1: REQUIREMENTS TRACEABILITY

Prerequisites Validation

A coverage oracle is available or inferred (formal requirements, spec, resolvable external pointer, or synthetic journeys)
Test suite exists (or gaps are acknowledged and documented)
If tests are missing, recommend *atdd (trace does not run it automatically)
Test directory path is correct (test_dir variable)
Story file is accessible (if using BMad mode)
Knowledge base is loaded (test-priorities, traceability, risk-governance)

Context Loading

Story file read successfully (if applicable)
Oracle items extracted or inferred correctly
Story ID identified (e.g., 1.3)
test-design.md loaded (if available)
tech-spec.md loaded (if available)
PRD.md loaded (if available)
Relevant knowledge fragments loaded from tea-index.csv

Test Discovery and Cataloging

Tests auto-discovered using multiple strategies (test IDs, describe blocks, file paths)
Tests categorized by level (E2E, API, Component, Unit)
Test metadata extracted:
- Test IDs (e.g., 1.3-E2E-001)
- Describe/context blocks
- It blocks (individual test cases)
- Given-When-Then structure (if BDD)
- Priority markers (P0/P1/P2/P3)
All relevant test files found (no tests missed due to naming conventions)

Criteria-to-Test Mapping

Each oracle item mapped to tests (or marked as NONE)
Explicit references found (test IDs, describe blocks mentioning criterion)
Test level documented (E2E, API, Component, Unit)
Given-When-Then narrative verified for alignment
Traceability matrix table generated:
- Criterion ID
- Description
- Test ID
- Test File
- Test Level
- Coverage Status

Coverage Classification

Coverage status classified for each criterion:
- FULL - All scenarios validated at appropriate level(s)
- PARTIAL - Some coverage but missing edge cases or levels
- NONE - No test coverage at any level
- UNIT-ONLY - Only unit tests (missing integration/E2E validation)
- INTEGRATION-ONLY - Only API/Component tests (missing unit confidence)
Classification justifications provided
Edge cases considered in FULL vs PARTIAL determination

Duplicate Coverage Detection

Duplicate coverage checked across test levels
Acceptable overlap identified (defense in depth for critical paths)
Unacceptable duplication flagged (same validation at multiple levels)
Recommendations provided for consolidation
Selective testing principles applied

Gap Analysis

Coverage gaps identified:
- Criteria with NONE status
- Criteria with PARTIAL status
- Criteria with UNIT-ONLY status
- Criteria with INTEGRATION-ONLY status
Coverage heuristics gaps identified:
- Endpoints referenced in requirements/specs but not covered by API tests
- Auth/authz criteria missing denied/invalid path tests
- Criteria with happy-path-only coverage (missing error scenarios)
- Inferred UI journeys missing E2E/component coverage
- Inferred UI journeys missing loading/empty/error/permission state coverage
Gaps prioritized by risk level using test-priorities framework:
- CRITICAL - P0 criteria without FULL coverage (BLOCKER)
- HIGH - P1 criteria without FULL coverage (PR blocker)
- MEDIUM - P2 criteria without FULL coverage (nightly gap)
- LOW - P3 criteria without FULL coverage (acceptable)
Specific test recommendations provided for each gap:
- Suggested test level (E2E, API, Component, Unit)
- Test description (Given-When-Then)
- Recommended test ID (e.g., 1.3-E2E-004)
- Explanation of why test is needed

Coverage Metrics

Overall coverage percentage calculated (FULL coverage / total criteria)
P0 coverage percentage calculated
P1 coverage percentage calculated
P2 coverage percentage calculated (if applicable)
Coverage by level calculated:
- E2E coverage %
- API coverage %
- Component coverage %
- Unit coverage %

Test Quality Verification

For each mapped test, verify:

Explicit assertions are present (not hidden in helpers)
Test follows Given-When-Then structure
No hard waits or sleeps (deterministic waiting only)
Self-cleaning (test cleans up its data)
File size < 300 lines
Test duration < 90 seconds

Quality issues flagged:

BLOCKER issues identified (missing assertions, hard waits, flaky patterns)
WARNING issues identified (large files, slow tests, unclear structure)
INFO issues identified (style inconsistencies, missing documentation)

Knowledge fragments referenced:

test-quality.md for Definition of Done
fixture-architecture.md for self-cleaning patterns
network-first.md for Playwright best practices
data-factories.md for test data patterns

Phase 1 Deliverables Generated

Traceability Matrix Markdown

File created at {test_artifacts}/traceability-matrix.md
Template from trace-template.md used
Full mapping table included
Coverage status section included
Gap analysis section included
Quality assessment section included
Recommendations section included

Machine-Readable JSON Output

e2e-trace-summary.json written to {e2e_trace_summary_output}
JSON is valid and parseable
schema_version field present
repo, collection_mode, collection_status, inventory_basis, and source_sha fields populated
gate_basis populated (priority_thresholds when gate-eligible, none otherwise)
snapshot_at replaces the old generated_at timestamp field
Oracle metadata populated (resolution_mode, confidence, sources, external_pointer_status, synthetic)
target.type and target.id identify the evaluated story / epic / release / hotfix
gate_status populated only when allow_gate: true and collection_status is COLLECTED
coverage.inventory includes covered, total, and pct
coverage.priority_breakdown includes P0–P3 and coverage.by_level includes e2e/api/component/unit/other
tests counts are deduplicated from unique discovered tests (no per-requirement double counting)
risk_summary counts match Phase 1 gap analysis
heuristics fields populated (endpoint_gaps, auth_negative_path_status, error_path_status)
UI heuristic fields populated when using a source-derived oracle (ui_journey_status, ui_state_status)
gate_criteria thresholds and actuals match gate decision
blockers array present (may be empty)
recommendations array present (may be empty)
links.trace_report_path points to traceability-matrix.md
links.trace_report_url, links.artifact_url, and links.journey_evidence_url fields present (may be empty)
gate-decision.json written to {gate_decision_output} when gate-eligible
gate-decision.json contains evaluated_at, gate_basis, gate_status, rationale, and per-criterion status fields

Updated Story File (if enabled)

“Traceability” section added to story markdown
Link to traceability matrix included
Coverage summary included

Phase 1 Quality Assurance

Accuracy Checks

All oracle items accounted for (none skipped)
Test IDs correctly formatted (e.g., 1.3-E2E-001)
File paths are correct and accessible
Coverage percentages calculated correctly
No false positives (tests incorrectly mapped to criteria)
No false negatives (existing tests missed in mapping)

Completeness Checks

All test levels considered (E2E, API, Component, Unit)
All priorities considered (P0, P1, P2, P3)
All coverage statuses used appropriately (FULL, PARTIAL, NONE, UNIT-ONLY, INTEGRATION-ONLY)
All gaps have recommendations
All quality issues have severity and remediation guidance

Actionability Checks

Recommendations are specific (not generic)
Test IDs suggested for new tests
Given-When-Then provided for recommended tests
Impact explained for each gap
Priorities clear (CRITICAL, HIGH, MEDIUM, LOW)

Phase 1 Documentation

Traceability matrix is readable and well-formatted
Tables render correctly in markdown
Code blocks have proper syntax highlighting
Links are valid and accessible
Recommendations are clear and prioritized

PHASE 2: QUALITY GATE DECISION

Note: Phase 2 always emits e2e-trace-summary.json; gate decision fields are populated only when allow_gate: true and collection_status resolves to COLLECTED.

Prerequisites

Evidence Gathering

Test execution results obtained (CI/CD pipeline, test framework reports)
Story/epic/release file identified and read
Test design document discovered or explicitly provided (if available)
Traceability matrix discovered or explicitly provided (available from Phase 1)
NFR evidence audit discovered or explicitly provided (if available)
Code coverage report discovered or explicitly provided (if available)
Burn-in results discovered or explicitly provided (if available)

Evidence Validation

Evidence freshness validated (warn if >7 days old, recommend re-running workflows)
All required assessments available or user acknowledged gaps
Test results are complete (not partial or interrupted runs)
Test results match current codebase (not from outdated branch)

Knowledge Base Loading

risk-governance.md loaded successfully
probability-impact.md loaded successfully
test-quality.md loaded successfully
test-priorities.md loaded successfully
ci-burn-in.md loaded (if burn-in results available)

Process Steps

Step 1: Context Loading

Gate type identified (story/epic/release/hotfix)
Target ID extracted (story_id, epic_num, or release_version)
Decision thresholds loaded from workflow variables
Risk tolerance configuration loaded
Waiver policy loaded

Step 2: Evidence Parsing

Test Results:

Total test count extracted
Passed test count extracted
Failed test count extracted
Skipped test count extracted
Test duration extracted
P0 test pass rate calculated
P1 test pass rate calculated
Overall test pass rate calculated

Quality Assessments:

P0/P1/P2/P3 scenarios extracted from test-design.md (if available)
Risk scores extracted from test-design.md (if available)
Coverage percentages extracted from traceability-matrix.md (available from Phase 1)
Coverage gaps extracted from traceability-matrix.md (available from Phase 1)
NFR status extracted from nfr-assessment.md (if available)
Security issues count extracted from nfr-assessment.md (if available)

Code Coverage:

Line coverage percentage extracted (if available)
Branch coverage percentage extracted (if available)
Function coverage percentage extracted (if available)
Critical path coverage validated (if available)

Burn-in Results:

Burn-in iterations count extracted (if available)
Flaky tests count extracted (if available)
Stability score calculated (if available)

Step 3: Decision Rules Application

P0 Criteria Evaluation:

P0 test pass rate evaluated (must be 100%)
P0 oracle-item coverage evaluated (must be 100%)
Security issues count evaluated (must be 0)
Critical NFR failures evaluated (must be 0)
Flaky tests evaluated (must be 0 if burn-in enabled)
P0 decision recorded: PASS or FAIL

P1 Criteria Evaluation:

P1 test pass rate evaluated (threshold: min_p1_pass_rate)
P1 oracle-item coverage evaluated (PASS >=90%, CONCERNS 80-89%, FAIL <80%)
Overall test pass rate evaluated (threshold: min_overall_pass_rate)
Overall oracle coverage evaluated (threshold: >=80%)
Code coverage considered if available (informational unless explicitly required by policy)
P1 decision recorded: PASS or CONCERNS

P2/P3 Criteria Evaluation:

P2 failures tracked (informational, don’t block if allow_p2_failures: true)
P3 failures tracked (informational, don’t block if allow_p3_failures: true)
Residual risks documented

Final Decision:

Decision determined: PASS / CONCERNS / FAIL / WAIVED
Decision rationale documented
Decision is deterministic (follows rules, not arbitrary)

Step 4: Documentation

Gate Decision Document Created:

Story/epic/release info section complete (ID, title, description, links)
Decision clearly stated (PASS / CONCERNS / FAIL / WAIVED)
Decision date recorded
Evaluator recorded (user or agent name)

Evidence Summary Documented:

Test results summary complete (total, passed, failed, pass rates)
Coverage summary complete (P0/P1 criteria, code coverage)
NFR validation summary complete (security, performance, reliability, maintainability)
Flakiness summary complete (burn-in iterations, flaky test count)

Rationale Documented:

Decision rationale clearly explained
Key evidence highlighted
Assumptions and caveats noted (if any)

Residual Risks Documented (if CONCERNS or WAIVED):

Unresolved P1/P2 issues listed
Probability × impact estimated for each risk
Mitigations or workarounds described

Waivers Documented (if WAIVED):

Waiver reason documented (business justification)
Waiver approver documented (name, role)
Waiver expiry date documented
Remediation plan documented (fix in next release, due date)
Monitoring plan documented

Critical Issues Documented (if FAIL or CONCERNS):

Top 5-10 critical issues listed
Priority assigned to each issue (P0/P1/P2)
Owner assigned to each issue
Due date assigned to each issue

Recommendations Documented:

Next steps clearly stated for decision type
Deployment recommendation provided
Monitoring recommendations provided (if applicable)
Remediation recommendations provided (if applicable)

Step 5: Status Updates and Notifications

Gate YAML Created:

Gate YAML snippet generated with decision and criteria
Evidence references included in YAML
Next steps included in YAML
YAML file saved to output folder

Stakeholder Notification Generated:

Notification subject line created
Notification body created with summary
Recipients identified (PM, SM, DEV lead, stakeholders)
Notification ready for delivery (if notify_stakeholders: true)

Outputs Saved:

Gate decision document saved to {outputFile}
e2e-trace-summary.json saved to {e2e_trace_summary_output} (always)
gate-decision.json saved to {gate_decision_output} (when gate-eligible)
All outputs are valid and readable

Phase 2 Output Validation

Gate Decision Document

Completeness:

All required sections present (info, decision, evidence, rationale, next steps)
No placeholder text or TODOs left in document
All evidence references are accurate and complete
All links to artifacts are valid

Accuracy:

Decision matches applied criteria rules
Test results match CI/CD pipeline output
Coverage percentages match reports
NFR status matches assessment document
No contradictions or inconsistencies

Clarity:

Decision rationale is clear and unambiguous
Technical jargon is explained or avoided
Stakeholders can understand next steps
Recommendations are actionable

Gate YAML

Format:

YAML is valid (no syntax errors)
All required fields present (target, decision, date, evaluator, criteria, evidence)
Field values are correct data types (numbers, strings, dates)

Content:

Criteria values match decision document
Evidence references are accurate
Next steps align with decision type

Phase 2 Quality Checks

Decision Integrity

Decision is deterministic (follows rules, not arbitrary)
P0 failures result in FAIL decision (unless waived)
Security issues result in FAIL decision (unless waived - but should never be waived)
Waivers have business justification and approver (if WAIVED)
Residual risks are documented (if CONCERNS or WAIVED)

Evidence-Based

Decision is based on actual test results (not guesses)
All claims are supported by evidence
No assumptions without documentation
Evidence sources are cited (CI run IDs, report URLs)

Transparency

Decision rationale is transparent and auditable
Criteria evaluation is documented step-by-step
Any deviations from standard process are explained
Waiver justifications are clear (if applicable)

Consistency

Decision aligns with risk-governance knowledge fragment
Priority framework (P0/P1/P2/P3) applied consistently
Terminology consistent with test-quality knowledge fragment
Decision matrix followed correctly

Phase 2 Integration Points

CI/CD Pipeline

Gate YAML is CI/CD-compatible
YAML can be parsed by pipeline automation
Decision can be used to block/allow deployments
Evidence references are accessible to pipeline

Stakeholders

Notification message is clear and actionable
Decision is explained in non-technical terms
Next steps are specific and time-bound
Recipients are appropriate for decision type

Phase 2 Compliance and Audit

Audit Trail

Decision date and time recorded
Evaluator identified (user or agent)
All evidence sources cited
Decision criteria documented
Rationale clearly explained

Traceability

Gate decision traceable to story/epic/release
Evidence traceable to specific test runs
Assessments traceable to workflows that created them
Waiver traceable to approver (if applicable)

Compliance

Security requirements validated (no unresolved vulnerabilities)
Quality standards met or waived with justification
Regulatory requirements addressed (if applicable)
Documentation sufficient for external audit

Phase 2 Edge Cases and Exceptions

Missing Evidence

If test-design.md missing, decision still possible with test results + trace
If traceability-matrix.md missing, decision still possible with test results (but Phase 1 should provide it)
If nfr-assessment.md missing, NFR validation marked as NOT ASSESSED
If code coverage missing, coverage criterion marked as NOT ASSESSED
User acknowledged gaps in evidence or provided alternative proof

Stale Evidence

Evidence freshness checked (if validate_evidence_freshness: true)
Warnings issued for assessments >7 days old
User acknowledged stale evidence or re-ran workflows
Decision document notes any stale evidence used

Conflicting Evidence

Conflicts between test results and assessments resolved
Most recent/authoritative source identified
Conflict resolution documented in decision rationale
User consulted if conflict cannot be resolved

Waiver Scenarios

Waiver only used for FAIL decision (not PASS or CONCERNS)
Waiver has business justification (not technical convenience)
Waiver has named approver with authority (VP/CTO/PO)
Waiver has expiry date (does NOT apply to future releases)
Waiver has remediation plan with concrete due date
Security vulnerabilities are NOT waived (enforced)

FINAL VALIDATION (Both Phases)

Non-Prescriptive Validation

Traceability format adapted to team needs (not rigid template)
Examples are minimal and focused on patterns
Teams can extend with custom classifications
Integration with external systems supported (JIRA, Azure DevOps)
Compliance requirements considered (if applicable)

Documentation and Communication

All documents are readable and well-formatted
Tables render correctly in markdown
Code blocks have proper syntax highlighting
Links are valid and accessible
Recommendations are clear and prioritized
Gate decision is prominent and unambiguous (Phase 2)

Final Validation

Phase 1 (Traceability):

All prerequisites met
All oracle items mapped or gaps documented
P0 coverage is 100% OR documented as BLOCKER
Gap analysis is complete and prioritized
Test quality issues identified and flagged
Deliverables generated and saved

Phase 2 (Gate Decision):

All quality evidence gathered
Decision criteria applied correctly
Decision rationale documented
e2e-trace-summary.json written and valid JSON
gate-decision.json written when gate-eligible
Status file updated (if enabled)
Stakeholders notified (if enabled)

Workflow Complete:

Phase 1 completed successfully
Phase 2 completed successfully (if enabled)
All outputs validated and saved
Ready to proceed based on gate decision

Sign-Off

Phase 1 - Traceability Status:

✅ PASS - All quality gates met, no critical gaps
⚠️ WARN - P1 gaps exist, address before PR merge
❌ FAIL - P0 gaps exist, BLOCKER for release

Phase 2 - Gate Decision Status (if enabled):

✅ PASS - Deploy to production
⚠️ CONCERNS - Deploy with monitoring
❌ FAIL - Block deployment, fix issues
🔓 WAIVED - Deploy with business approval and remediation plan

Next Actions:

If PASS (both phases): Proceed to deployment
If WARN/CONCERNS: Address gaps/issues, proceed with monitoring
If FAIL (either phase): Run *atdd for missing tests, fix issues, re-run *trace
If WAIVED: Deploy with approved waiver, schedule remediation

Notes

Record any issues, deviations, or important observations during workflow execution:

Phase 1 Issues: [Note any traceability mapping challenges, missing tests, quality concerns]
Phase 2 Issues: [Note any missing, stale, or conflicting evidence]
Decision Rationale: [Document any nuanced reasoning or edge cases]
Waiver Details: [Document waiver negotiations or approvals]
Follow-up Actions: [List any actions required after gate decision]

checklist.md 23KB 原始文件 Blame 歷史記錄