Nevar pievienot vairāk kā 25 tēmas Tēmai ir jāsākas ar burtu vai ciparu, tā var saturēt domu zīmes ('-') un var būt līdz 35 simboliem gara.

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670
  1. # Non-Functional Requirements (NFR) Criteria
  2. ## Principle
  3. Non-functional requirements (security, performance, reliability, maintainability) are **validated through automated tests**, not checklists. NFR assessment uses objective pass/fail criteria tied to measurable thresholds. Ambiguous requirements default to CONCERNS until clarified.
  4. ## Rationale
  5. **The Problem**: Teams ship features that "work" functionally but fail under load, expose security vulnerabilities, or lack error recovery. NFRs are treated as optional "nice-to-haves" instead of release blockers.
  6. **The Solution**: Define explicit NFR criteria with automated validation. Security tests verify auth/authz and secret handling. Performance tests enforce SLO/SLA thresholds with profiling evidence. Reliability tests validate error handling, retries, and health checks. Maintainability is measured by test coverage, code duplication, and observability.
  7. **Why This Matters**:
  8. - Prevents production incidents (security breaches, performance degradation, cascading failures)
  9. - Provides objective release criteria (no subjective "feels fast enough")
  10. - Automates compliance validation (audit trail for regulated environments)
  11. - Forces clarity on ambiguous requirements (default to CONCERNS)
  12. ## Pattern Examples
  13. ### Example 1: Security NFR Validation (Auth, Secrets, OWASP)
  14. **Context**: Automated security tests enforcing authentication, authorization, and secret handling
  15. **Implementation**:
  16. ```typescript
  17. // tests/nfr/security.spec.ts
  18. import { test, expect } from '@playwright/test';
  19. test.describe('Security NFR: Authentication & Authorization', () => {
  20. test('unauthenticated users cannot access protected routes', async ({ page }) => {
  21. // Attempt to access dashboard without auth
  22. await page.goto('/dashboard');
  23. // Should redirect to login (not expose data)
  24. await expect(page).toHaveURL(/\/login/);
  25. await expect(page.getByText('Please sign in')).toBeVisible();
  26. // Verify no sensitive data leaked in response
  27. const pageContent = await page.content();
  28. expect(pageContent).not.toContain('user_id');
  29. expect(pageContent).not.toContain('api_key');
  30. });
  31. test('JWT tokens expire after 15 minutes', async ({ page, request }) => {
  32. // Login and capture token
  33. await page.goto('/login');
  34. await page.getByLabel('Email').fill('test@example.com');
  35. await page.getByLabel('Password').fill('ValidPass123!');
  36. await page.getByRole('button', { name: 'Sign In' }).click();
  37. const token = await page.evaluate(() => localStorage.getItem('auth_token'));
  38. expect(token).toBeTruthy();
  39. // Wait 16 minutes (use mock clock in real tests)
  40. await page.clock.fastForward('00:16:00');
  41. // Token should be expired, API call should fail
  42. const response = await request.get('/api/user/profile', {
  43. headers: { Authorization: `Bearer ${token}` },
  44. });
  45. expect(response.status()).toBe(401);
  46. const body = await response.json();
  47. expect(body.error).toContain('expired');
  48. });
  49. test('passwords are never logged or exposed in errors', async ({ page }) => {
  50. // Trigger login error
  51. await page.goto('/login');
  52. await page.getByLabel('Email').fill('test@example.com');
  53. await page.getByLabel('Password').fill('WrongPassword123!');
  54. // Monitor console for password leaks
  55. const consoleLogs: string[] = [];
  56. page.on('console', (msg) => consoleLogs.push(msg.text()));
  57. await page.getByRole('button', { name: 'Sign In' }).click();
  58. // Error shown to user (generic message)
  59. await expect(page.getByText('Invalid credentials')).toBeVisible();
  60. // Verify password NEVER appears in console, DOM, or network
  61. const pageContent = await page.content();
  62. expect(pageContent).not.toContain('WrongPassword123!');
  63. expect(consoleLogs.join('\n')).not.toContain('WrongPassword123!');
  64. });
  65. test('RBAC: users can only access resources they own', async ({ page, request }) => {
  66. // Login as User A
  67. const userAToken = await login(request, 'userA@example.com', 'password');
  68. // Try to access User B's order
  69. const response = await request.get('/api/orders/user-b-order-id', {
  70. headers: { Authorization: `Bearer ${userAToken}` },
  71. });
  72. expect(response.status()).toBe(403); // Forbidden
  73. const body = await response.json();
  74. expect(body.error).toContain('insufficient permissions');
  75. });
  76. test('SQL injection attempts are blocked', async ({ page }) => {
  77. await page.goto('/search');
  78. // Attempt SQL injection
  79. await page.getByPlaceholder('Search products').fill("'; DROP TABLE users; --");
  80. await page.getByRole('button', { name: 'Search' }).click();
  81. // Should return empty results, NOT crash or expose error
  82. await expect(page.getByText('No results found')).toBeVisible();
  83. // Verify app still works (table not dropped)
  84. await page.goto('/dashboard');
  85. await expect(page.getByText('Welcome')).toBeVisible();
  86. });
  87. test('XSS attempts are sanitized', async ({ page }) => {
  88. await page.goto('/profile/edit');
  89. // Attempt XSS injection
  90. const xssPayload = '<script>alert("XSS")</script>';
  91. await page.getByLabel('Bio').fill(xssPayload);
  92. await page.getByRole('button', { name: 'Save' }).click();
  93. // Reload and verify XSS is escaped (not executed)
  94. await page.reload();
  95. const bio = await page.getByTestId('user-bio').textContent();
  96. // Text should be escaped, script should NOT execute
  97. expect(bio).toContain('&lt;script&gt;');
  98. expect(bio).not.toContain('<script>');
  99. });
  100. });
  101. // Helper
  102. async function login(request: any, email: string, password: string): Promise<string> {
  103. const response = await request.post('/api/auth/login', {
  104. data: { email, password },
  105. });
  106. const body = await response.json();
  107. return body.token;
  108. }
  109. ```
  110. **Key Points**:
  111. - Authentication: Unauthenticated access redirected (not exposed)
  112. - Authorization: RBAC enforced (403 for insufficient permissions)
  113. - Token expiry: JWT expires after 15 minutes (automated validation)
  114. - Secret handling: Passwords never logged or exposed in errors
  115. - OWASP Top 10: SQL injection and XSS blocked (input sanitization)
  116. **Security NFR Criteria**:
  117. - ✅ PASS: All 6 tests green (auth, authz, token expiry, secret handling, SQL injection, XSS)
  118. - ⚠️ CONCERNS: 1-2 tests failing with mitigation plan and owner assigned
  119. - ❌ FAIL: Critical exposure (unauthenticated access, password leak, SQL injection succeeds)
  120. ---
  121. ### Example 2: Performance NFR Validation (k6 Load Testing for SLO/SLA)
  122. **Context**: Use k6 for load testing, stress testing, and SLO/SLA enforcement (NOT Playwright)
  123. **Implementation**:
  124. ```javascript
  125. // tests/nfr/performance.k6.js
  126. import http from 'k6/http';
  127. import { check, sleep } from 'k6';
  128. import { Rate, Trend } from 'k6/metrics';
  129. // Custom metrics
  130. const errorRate = new Rate('errors');
  131. const apiDuration = new Trend('api_duration');
  132. // Performance thresholds (SLO/SLA)
  133. export const options = {
  134. stages: [
  135. { duration: '1m', target: 50 }, // Ramp up to 50 users
  136. { duration: '3m', target: 50 }, // Stay at 50 users for 3 minutes
  137. { duration: '1m', target: 100 }, // Spike to 100 users
  138. { duration: '3m', target: 100 }, // Stay at 100 users
  139. { duration: '1m', target: 0 }, // Ramp down
  140. ],
  141. thresholds: {
  142. // SLO: 95% of requests must complete in <500ms
  143. http_req_duration: ['p(95)<500'],
  144. // SLO: Error rate must be <1%
  145. errors: ['rate<0.01'],
  146. // SLA: API endpoints must respond in <1s (99th percentile)
  147. api_duration: ['p(99)<1000'],
  148. },
  149. };
  150. export default function () {
  151. // Test 1: Homepage load performance
  152. const homepageResponse = http.get(`${__ENV.BASE_URL}/`);
  153. check(homepageResponse, {
  154. 'homepage status is 200': (r) => r.status === 200,
  155. 'homepage loads in <2s': (r) => r.timings.duration < 2000,
  156. });
  157. errorRate.add(homepageResponse.status !== 200);
  158. // Test 2: API endpoint performance
  159. const apiResponse = http.get(`${__ENV.BASE_URL}/api/products?limit=10`, {
  160. headers: { Authorization: `Bearer ${__ENV.API_TOKEN}` },
  161. });
  162. check(apiResponse, {
  163. 'API status is 200': (r) => r.status === 200,
  164. 'API responds in <500ms': (r) => r.timings.duration < 500,
  165. });
  166. apiDuration.add(apiResponse.timings.duration);
  167. errorRate.add(apiResponse.status !== 200);
  168. // Test 3: Search endpoint under load
  169. const searchResponse = http.get(`${__ENV.BASE_URL}/api/search?q=laptop&limit=100`);
  170. check(searchResponse, {
  171. 'search status is 200': (r) => r.status === 200,
  172. 'search responds in <1s': (r) => r.timings.duration < 1000,
  173. 'search returns results': (r) => JSON.parse(r.body).results.length > 0,
  174. });
  175. errorRate.add(searchResponse.status !== 200);
  176. sleep(1); // Realistic user think time
  177. }
  178. // Threshold validation (run after test)
  179. export function handleSummary(data) {
  180. const p95Duration = data.metrics.http_req_duration.values['p(95)'];
  181. const p99ApiDuration = data.metrics.api_duration.values['p(99)'];
  182. const errorRateValue = data.metrics.errors.values.rate;
  183. console.log(`P95 request duration: ${p95Duration.toFixed(2)}ms`);
  184. console.log(`P99 API duration: ${p99ApiDuration.toFixed(2)}ms`);
  185. console.log(`Error rate: ${(errorRateValue * 100).toFixed(2)}%`);
  186. return {
  187. 'summary.json': JSON.stringify(data),
  188. stdout: `
  189. Performance NFR Results:
  190. - P95 request duration: ${p95Duration < 500 ? '✅ PASS' : '❌ FAIL'} (${p95Duration.toFixed(2)}ms / 500ms threshold)
  191. - P99 API duration: ${p99ApiDuration < 1000 ? '✅ PASS' : '❌ FAIL'} (${p99ApiDuration.toFixed(2)}ms / 1000ms threshold)
  192. - Error rate: ${errorRateValue < 0.01 ? '✅ PASS' : '❌ FAIL'} (${(errorRateValue * 100).toFixed(2)}% / 1% threshold)
  193. `,
  194. };
  195. }
  196. ```
  197. **Run k6 tests:**
  198. ```bash
  199. # Local smoke test (10 VUs, 30s)
  200. k6 run --vus 10 --duration 30s tests/nfr/performance.k6.js
  201. # Full load test (stages defined in script)
  202. k6 run tests/nfr/performance.k6.js
  203. # CI integration with thresholds
  204. k6 run --out json=performance-results.json tests/nfr/performance.k6.js
  205. ```
  206. **Key Points**:
  207. - **k6 is the right tool** for load testing (NOT Playwright)
  208. - SLO/SLA thresholds enforced automatically (`p(95)<500`, `rate<0.01`)
  209. - Realistic load simulation (ramp up, sustained load, spike testing)
  210. - Comprehensive metrics (p50, p95, p99, error rate, throughput)
  211. - CI-friendly (JSON output, exit codes based on thresholds)
  212. **Performance NFR Criteria**:
  213. - ✅ PASS: All SLO/SLA targets met with k6 profiling evidence (p95 < 500ms, error rate < 1%)
  214. - ⚠️ CONCERNS: Trending toward limits (e.g., p95 = 480ms approaching 500ms) or missing baselines
  215. - ❌ FAIL: SLO/SLA breached (e.g., p95 > 500ms) or error rate > 1%
  216. **Performance Testing Levels (from Test Architect course):**
  217. - **Load testing**: System behavior under expected load
  218. - **Stress testing**: System behavior under extreme load (breaking point)
  219. - **Spike testing**: Sudden load increases (traffic spikes)
  220. - **Endurance/Soak testing**: System behavior under sustained load (memory leaks, resource exhaustion)
  221. - **Benchmarking**: Baseline measurements for comparison
  222. **Note**: Playwright can validate **perceived performance** (Core Web Vitals via Lighthouse), but k6 validates **system performance** (throughput, latency, resource limits under load)
  223. ---
  224. ### Example 3: Reliability NFR Validation (Playwright for UI Resilience)
  225. **Context**: Automated reliability tests validating graceful degradation and recovery paths
  226. **Implementation**:
  227. ```typescript
  228. // tests/nfr/reliability.spec.ts
  229. import { test, expect } from '@playwright/test';
  230. test.describe('Reliability NFR: Error Handling & Recovery', () => {
  231. test('app remains functional when API returns 500 error', async ({ page, context }) => {
  232. // Mock API failure
  233. await context.route('**/api/products', (route) => {
  234. route.fulfill({ status: 500, body: JSON.stringify({ error: 'Internal Server Error' }) });
  235. });
  236. await page.goto('/products');
  237. // User sees error message (not blank page or crash)
  238. await expect(page.getByText('Unable to load products. Please try again.')).toBeVisible();
  239. await expect(page.getByRole('button', { name: 'Retry' })).toBeVisible();
  240. // App navigation still works (graceful degradation)
  241. await page.getByRole('link', { name: 'Home' }).click();
  242. await expect(page).toHaveURL('/');
  243. });
  244. test('API client retries on transient failures (3 attempts)', async ({ page, context }) => {
  245. let attemptCount = 0;
  246. await context.route('**/api/checkout', (route) => {
  247. attemptCount++;
  248. // Fail first 2 attempts, succeed on 3rd
  249. if (attemptCount < 3) {
  250. route.fulfill({ status: 503, body: JSON.stringify({ error: 'Service Unavailable' }) });
  251. } else {
  252. route.fulfill({ status: 200, body: JSON.stringify({ orderId: '12345' }) });
  253. }
  254. });
  255. await page.goto('/checkout');
  256. await page.getByRole('button', { name: 'Place Order' }).click();
  257. // Should succeed after 3 attempts
  258. await expect(page.getByText('Order placed successfully')).toBeVisible();
  259. expect(attemptCount).toBe(3);
  260. });
  261. test('app handles network disconnection gracefully', async ({ page, context }) => {
  262. await page.goto('/dashboard');
  263. // Simulate offline mode
  264. await context.setOffline(true);
  265. // Trigger action requiring network
  266. await page.getByRole('button', { name: 'Refresh Data' }).click();
  267. // User sees offline indicator (not crash)
  268. await expect(page.getByText('You are offline. Changes will sync when reconnected.')).toBeVisible();
  269. // Reconnect
  270. await context.setOffline(false);
  271. await page.getByRole('button', { name: 'Refresh Data' }).click();
  272. // Data loads successfully
  273. await expect(page.getByText('Data updated')).toBeVisible();
  274. });
  275. test('health check endpoint returns service status', async ({ request }) => {
  276. const response = await request.get('/api/health');
  277. expect(response.status()).toBe(200);
  278. const health = await response.json();
  279. expect(health).toHaveProperty('status', 'healthy');
  280. expect(health).toHaveProperty('timestamp');
  281. expect(health).toHaveProperty('services');
  282. // Verify critical services are monitored
  283. expect(health.services).toHaveProperty('database');
  284. expect(health.services).toHaveProperty('cache');
  285. expect(health.services).toHaveProperty('queue');
  286. // All services should be UP
  287. expect(health.services.database.status).toBe('UP');
  288. expect(health.services.cache.status).toBe('UP');
  289. expect(health.services.queue.status).toBe('UP');
  290. });
  291. test('circuit breaker opens after 5 consecutive failures', async ({ page, context }) => {
  292. let failureCount = 0;
  293. await context.route('**/api/recommendations', (route) => {
  294. failureCount++;
  295. route.fulfill({ status: 500, body: JSON.stringify({ error: 'Service Error' }) });
  296. });
  297. await page.goto('/product/123');
  298. // Wait for circuit breaker to open (fallback UI appears)
  299. await expect(page.getByText('Recommendations temporarily unavailable')).toBeVisible({ timeout: 10000 });
  300. // Verify circuit breaker stopped making requests after threshold (should be ≤5)
  301. expect(failureCount).toBeLessThanOrEqual(5);
  302. });
  303. test('rate limiting gracefully handles 429 responses', async ({ page, context }) => {
  304. let requestCount = 0;
  305. await context.route('**/api/search', (route) => {
  306. requestCount++;
  307. if (requestCount > 10) {
  308. // Rate limit exceeded
  309. route.fulfill({
  310. status: 429,
  311. headers: { 'Retry-After': '5' },
  312. body: JSON.stringify({ error: 'Rate limit exceeded' }),
  313. });
  314. } else {
  315. route.fulfill({ status: 200, body: JSON.stringify({ results: [] }) });
  316. }
  317. });
  318. await page.goto('/search');
  319. // Make 15 search requests rapidly
  320. for (let i = 0; i < 15; i++) {
  321. await page.getByPlaceholder('Search').fill(`query-${i}`);
  322. await page.getByRole('button', { name: 'Search' }).click();
  323. }
  324. // User sees rate limit message (not crash)
  325. await expect(page.getByText('Too many requests. Please wait a moment.')).toBeVisible();
  326. });
  327. });
  328. ```
  329. **Key Points**:
  330. - Error handling: Graceful degradation (500 error → user-friendly message + retry button)
  331. - Retries: 3 attempts on transient failures (503 → eventual success)
  332. - Offline handling: Network disconnection detected (sync when reconnected)
  333. - Health checks: `/api/health` monitors database, cache, queue
  334. - Circuit breaker: Opens after 5 failures (fallback UI, stop retries)
  335. - Rate limiting: 429 response handled (Retry-After header respected)
  336. **Reliability NFR Criteria**:
  337. - ✅ PASS: Error handling, retries, health checks verified (all 6 tests green)
  338. - ⚠️ CONCERNS: Partial coverage (e.g., missing circuit breaker) or no telemetry
  339. - ❌ FAIL: No recovery path (500 error crashes app) or unresolved crash scenarios
  340. ---
  341. ### Example 4: Maintainability NFR Validation (CI Tools, Not Playwright)
  342. **Context**: Use proper CI tools for code quality validation (coverage, duplication, vulnerabilities)
  343. **Implementation**:
  344. ```yaml
  345. # .github/workflows/nfr-maintainability.yml
  346. name: NFR - Maintainability
  347. on: [push, pull_request]
  348. jobs:
  349. test-coverage:
  350. runs-on: ubuntu-latest
  351. steps:
  352. - uses: actions/checkout@v4
  353. - uses: actions/setup-node@v4
  354. - name: Install dependencies
  355. run: npm ci
  356. - name: Run tests with coverage
  357. run: npm run test:coverage
  358. - name: Check coverage threshold (80% minimum)
  359. run: |
  360. COVERAGE=$(jq '.total.lines.pct' coverage/coverage-summary.json)
  361. echo "Coverage: $COVERAGE%"
  362. if (( $(echo "$COVERAGE < 80" | bc -l) )); then
  363. echo "❌ FAIL: Coverage $COVERAGE% below 80% threshold"
  364. exit 1
  365. else
  366. echo "✅ PASS: Coverage $COVERAGE% meets 80% threshold"
  367. fi
  368. code-duplication:
  369. runs-on: ubuntu-latest
  370. steps:
  371. - uses: actions/checkout@v4
  372. - uses: actions/setup-node@v4
  373. - name: Check code duplication (<5% allowed)
  374. run: |
  375. npx jscpd src/ --threshold 5 --format json --output duplication.json
  376. DUPLICATION=$(jq '.statistics.total.percentage' duplication.json)
  377. echo "Duplication: $DUPLICATION%"
  378. if (( $(echo "$DUPLICATION >= 5" | bc -l) )); then
  379. echo "❌ FAIL: Duplication $DUPLICATION% exceeds 5% threshold"
  380. exit 1
  381. else
  382. echo "✅ PASS: Duplication $DUPLICATION% below 5% threshold"
  383. fi
  384. vulnerability-scan:
  385. runs-on: ubuntu-latest
  386. steps:
  387. - uses: actions/checkout@v4
  388. - uses: actions/setup-node@v4
  389. - name: Install dependencies
  390. run: npm ci
  391. - name: Run npm audit (no critical/high vulnerabilities)
  392. run: |
  393. npm audit --json > audit.json || true
  394. CRITICAL=$(jq '.metadata.vulnerabilities.critical' audit.json)
  395. HIGH=$(jq '.metadata.vulnerabilities.high' audit.json)
  396. echo "Critical: $CRITICAL, High: $HIGH"
  397. if [ "$CRITICAL" -gt 0 ] || [ "$HIGH" -gt 0 ]; then
  398. echo "❌ FAIL: Found $CRITICAL critical and $HIGH high vulnerabilities"
  399. npm audit
  400. exit 1
  401. else
  402. echo "✅ PASS: No critical/high vulnerabilities"
  403. fi
  404. ```
  405. **Playwright Tests for Observability (E2E Validation):**
  406. ```typescript
  407. // tests/nfr/observability.spec.ts
  408. import { test, expect } from '@playwright/test';
  409. test.describe('Maintainability NFR: Observability Validation', () => {
  410. test('critical errors are reported to monitoring service', async ({ page, context }) => {
  411. const sentryEvents: any[] = [];
  412. // Mock Sentry SDK to verify error tracking
  413. await context.addInitScript(() => {
  414. (window as any).Sentry = {
  415. captureException: (error: Error) => {
  416. console.log('SENTRY_CAPTURE:', JSON.stringify({ message: error.message, stack: error.stack }));
  417. },
  418. };
  419. });
  420. page.on('console', (msg) => {
  421. if (msg.text().includes('SENTRY_CAPTURE:')) {
  422. sentryEvents.push(JSON.parse(msg.text().replace('SENTRY_CAPTURE:', '')));
  423. }
  424. });
  425. // Trigger error by mocking API failure
  426. await context.route('**/api/products', (route) => {
  427. route.fulfill({ status: 500, body: JSON.stringify({ error: 'Database Error' }) });
  428. });
  429. await page.goto('/products');
  430. // Wait for error UI and Sentry capture
  431. await expect(page.getByText('Unable to load products')).toBeVisible();
  432. // Verify error was captured by monitoring
  433. expect(sentryEvents.length).toBeGreaterThan(0);
  434. expect(sentryEvents[0]).toHaveProperty('message');
  435. expect(sentryEvents[0]).toHaveProperty('stack');
  436. });
  437. test('API response times are tracked in telemetry', async ({ request }) => {
  438. const response = await request.get('/api/products?limit=10');
  439. expect(response.ok()).toBeTruthy();
  440. // Verify Server-Timing header for APM (Application Performance Monitoring)
  441. const serverTiming = response.headers()['server-timing'];
  442. expect(serverTiming).toBeTruthy();
  443. expect(serverTiming).toContain('db'); // Database query time
  444. expect(serverTiming).toContain('total'); // Total processing time
  445. });
  446. test('structured logging present in application', async ({ request }) => {
  447. // Make API call that generates logs
  448. const response = await request.post('/api/orders', {
  449. data: { productId: '123', quantity: 2 },
  450. });
  451. expect(response.ok()).toBeTruthy();
  452. // Note: In real scenarios, validate logs in monitoring system (Datadog, CloudWatch)
  453. // This test validates the logging contract exists (Server-Timing, trace IDs in headers)
  454. const traceId = response.headers()['x-trace-id'];
  455. expect(traceId).toBeTruthy(); // Confirms structured logging with correlation IDs
  456. });
  457. });
  458. ```
  459. **Key Points**:
  460. - **Coverage/duplication**: CI jobs (GitHub Actions), not Playwright tests
  461. - **Vulnerability scanning**: npm audit in CI, not Playwright tests
  462. - **Observability**: Playwright validates error tracking (Sentry) and telemetry headers
  463. - **Structured logging**: Validate logging contract (trace IDs, Server-Timing headers)
  464. - **Separation of concerns**: Build-time checks (coverage, audit) vs runtime checks (error tracking, telemetry)
  465. **Maintainability NFR Criteria**:
  466. - ✅ PASS: Clean code (80%+ coverage from CI, <5% duplication from CI), observability validated in E2E, no critical vulnerabilities from npm audit
  467. - ⚠️ CONCERNS: Duplication >5%, coverage 60-79%, or unclear ownership
  468. - ❌ FAIL: Absent tests (<60%), tangled implementations (>10% duplication), or no observability
  469. ---
  470. ## NFR Assessment Checklist
  471. Before release gate:
  472. - [ ] **Security** (Playwright E2E + Security Tools):
  473. - [ ] Auth/authz tests green (unauthenticated redirect, RBAC enforced)
  474. - [ ] Secrets never logged or exposed in errors
  475. - [ ] OWASP Top 10 validated (SQL injection blocked, XSS sanitized)
  476. - [ ] Security audit completed (vulnerability scan, penetration test if applicable)
  477. - [ ] **Performance** (k6 Load Testing):
  478. - [ ] SLO/SLA targets met with k6 evidence (p95 <500ms, error rate <1%)
  479. - [ ] Load testing completed (expected load)
  480. - [ ] Stress testing completed (breaking point identified)
  481. - [ ] Spike testing completed (handles traffic spikes)
  482. - [ ] Endurance testing completed (no memory leaks under sustained load)
  483. - [ ] **Reliability** (Playwright E2E + API Tests):
  484. - [ ] Error handling graceful (500 → user-friendly message + retry)
  485. - [ ] Retries implemented (3 attempts on transient failures)
  486. - [ ] Health checks monitored (/api/health endpoint)
  487. - [ ] Circuit breaker tested (opens after failure threshold)
  488. - [ ] Offline handling validated (network disconnection graceful)
  489. - [ ] **Maintainability** (CI Tools):
  490. - [ ] Test coverage ≥80% (from CI coverage report)
  491. - [ ] Code duplication <5% (from jscpd CI job)
  492. - [ ] No critical/high vulnerabilities (from npm audit CI job)
  493. - [ ] Structured logging validated (Playwright validates telemetry headers)
  494. - [ ] Error tracking configured (Sentry/monitoring integration validated)
  495. - [ ] **Ambiguous requirements**: Default to CONCERNS (force team to clarify thresholds and evidence)
  496. - [ ] **NFR criteria documented**: Measurable thresholds defined (not subjective "fast enough")
  497. - [ ] **Automated validation**: NFR tests run in CI pipeline (not manual checklists)
  498. - [ ] **Tool selection**: Right tool for each NFR (k6 for performance, Playwright for security/reliability E2E, CI tools for maintainability)
  499. ## NFR Gate Decision Matrix
  500. | Category | PASS Criteria | CONCERNS Criteria | FAIL Criteria |
  501. | ------------------- | -------------------------------------------- | -------------------------------------------- | ---------------------------------------------- |
  502. | **Security** | Auth/authz, secret handling, OWASP verified | Minor gaps with clear owners | Critical exposure or missing controls |
  503. | **Performance** | Metrics meet SLO/SLA with profiling evidence | Trending toward limits or missing baselines | SLO/SLA breached or resource leaks detected |
  504. | **Reliability** | Error handling, retries, health checks OK | Partial coverage or missing telemetry | No recovery path or unresolved crash scenarios |
  505. | **Maintainability** | Clean code, tests, docs shipped together | Duplication, low coverage, unclear ownership | Absent tests, tangled code, no observability |
  506. **Default**: If targets or evidence are undefined → **CONCERNS** (force team to clarify before sign-off)
  507. ## Integration Points
  508. - **Used in workflows**: `*nfr-assess` (automated NFR validation), `*trace` (gate decision Phase 2), `*test-design` (NFR risk assessment via Utility Tree)
  509. - **Related fragments**: `risk-governance.md` (NFR risk scoring), `probability-impact.md` (NFR impact assessment), `test-quality.md` (maintainability standards), `test-levels-framework.md` (system-level testing for NFRs)
  510. - **Tools by NFR Category**:
  511. - **Security**: Playwright (E2E auth/authz), OWASP ZAP, Burp Suite, npm audit, Snyk
  512. - **Performance**: k6 (load/stress/spike/endurance), Lighthouse (Core Web Vitals), Artillery
  513. - **Reliability**: Playwright (E2E error handling), API tests (retries, health checks), Chaos Engineering tools
  514. - **Maintainability**: GitHub Actions (coverage, duplication, audit), jscpd, Playwright (observability validation)
  515. _Source: Test Architect course (NFR testing approaches, Utility Tree, Quality Scenarios), ISO/IEC 25010 Software Quality Characteristics, OWASP Top 10, k6 documentation, SRE practices_