# Performance Testing Plan - CBTAPPS Production Readiness

**Task:** ENI-63 - Production Readiness Performance Validation
**Date:** April 8, 2026
**Objective:** Validate production performance capacity and ensure scalability requirements are met

## Executive Summary

This document outlines the comprehensive performance testing strategy for CBTAPPS, a multi-tenant Laravel-based Computer Based Test application. The testing builds upon previous performance optimizations (ENI-46) including Redis-based sessions/cache, database query optimization, and Redis-based token deduction.

## Success Criteria

- **Response Time:** p95 < 200ms under 2x expected load
- **Concurrency:** Support 1000+ concurrent users without degradation
- **Database:** Queries perform efficiently under load
- **Infrastructure:** Auto-scaling functions correctly
- **Deployment:** CI/CD completes in <10 minutes

## Test Environment

### Production Environment
- **URL:** https://cbtapps.com
- **Database:** PostgreSQL (production)
- **Cache/Session:** Redis (Unix socket: `/home/cbtappsc/redis/redis.sock`)
- **Storage:** Google Cloud Storage
- **Multi-tenant:** Subdomain-based isolation

### Current Optimizations (from ENI-46)
- Redis-based sessions and cache
- Database indexes on exam-related tables
- Redis-based token deduction (eliminates lock contention)

## Critical Endpoints to Test

### 1. Student CBT Endpoints (High Priority)
| Endpoint | Method | Description | Expected p95 |
|----------|--------|-------------|--------------|
| `/siswa/cbt` | GET | Dashboard with exam list | <150ms |
| `/siswa/cbt/{id}/start` | POST | Start exam (token deduction) | <200ms |
| `/siswa/cbt/{attemptId}` | GET | Show exam interface | <150ms |
| `/siswa/cbt/save-answer` | POST | Save answer during exam | <100ms |
| `/siswa/cbt/{attemptId}/submit` | POST | Submit exam | <200ms |

### 2. Authentication Endpoints (Medium Priority)
| Endpoint | Method | Description | Expected p95 |
|----------|--------|-------------|--------------|
| `/login` | POST | User login | <200ms |
| `/logout` | POST | User logout | <100ms |

### 3. Admin Endpoints (Medium Priority)
| Endpoint | Method | Description | Expected p95 |
|----------|--------|-------------|--------------|
| `/admin/dashboard` | GET | Admin dashboard | <200ms |
| `/admin/exams` | GET | Exam list | <150ms |
| `/admin/students` | GET | Student list | <150ms |

### 4. Static Assets (Low Priority)
| Endpoint | Method | Description | Expected p95 |
|----------|--------|-------------|--------------|
| `/manifest.json` | GET | PWA manifest | <50ms |
| `/og-image` | GET | Open Graph image | <100ms |

## Test Scenarios

### Scenario 1: Baseline Performance (Single User)
**Objective:** Establish baseline response times

**Steps:**
1. Run tests with 1 user
2. Execute each critical endpoint 100 times
3. Measure p50, p95, p99 response times
4. Document baseline metrics

**Success Criteria:**
- All endpoints meet expected p95 times
- No errors or timeouts

### Scenario 2: Concurrent User Load Test
**Objective:** Validate system handles 1000+ concurrent users

**Test Configuration:**
- **Users:** 1000 virtual users
- **Ramp-up:** 100 users/second over 10 seconds
- **Duration:** 5 minutes sustained load
- **Distribution:**
  - 60% reading exam questions
  - 20% saving answers
  - 10% starting exams
  - 10% submitting exams

**Success Criteria:**
- p95 response time <200ms
- Error rate <0.1%
- No database connection pool exhaustion
- Redis connection stable

### Scenario 3: Peak Load Test (2x Expected Load)
**Objective:** Validate system handles 2x expected load (2000 concurrent users)

**Test Configuration:**
- **Users:** 2000 virtual users
- **Ramp-up:** 200 users/second over 10 seconds
- **Duration:** 10 minutes sustained load

**Success Criteria:**
- p95 response time <200ms (primary requirement)
- System remains stable
- No data corruption
- Graceful degradation (if any)

### Scenario 4: Stress Test (Break Point)
**Objective:** Find system breaking point

**Test Configuration:**
- **Users:** Start at 2000, increase by 500 every 2 minutes
- **Duration:** Until failure or 5000 users

**Success Criteria:**
- Document breaking point
- Identify failure modes
- Provide recommendations

### Scenario 5: Token Deduction Concurrency Test
**Objective:** Validate Redis-based token deduction under high concurrency

**Test Configuration:**
- **Users:** 500 users starting exams simultaneously
- **Ramp-up:** Instant (all at once)
- **Duration:** 1 minute

**Success Criteria:**
- All token deductions succeed
- No database lock contention
- Token balances remain consistent
- No duplicate charges

### Scenario 6: Database Performance Under Load
**Objective:** Validate database query performance

**Metrics to Monitor:**
- Slow query log
- Connection pool usage
- Query execution times
- Index usage statistics

**Success Criteria:**
- No queries >100ms under load
- Connection pool not exhausted
- Index usage >95%

### Scenario 7: Redis Performance Under Load
**Objective:** Validate Redis cache/session performance

**Metrics to Monitor:**
- Redis connection count
- Command execution times
- Memory usage
- Eviction rate (if any)

**Success Criteria:**
- Redis operations <5ms
- No connection timeouts
- Memory usage stable

## Testing Tools

### 1. k6 (Load Testing)
**Why k6?**
- Modern, scriptable load testing tool
- Good JavaScript API
- Excellent metrics and reporting
- Cloud execution support

**Installation:**
```bash
brew install k6  # macOS
# Or download from https://k6.io/
```

### 2. Laravel Debug Bar (Development)
- Query analysis
- Request performance
- Memory usage

### 3. Redis CLI
```bash
redis-cli -s /home/cbtappsc/redis/redis.sock
INFO stats
SLOWLOG GET 10
```

### 4. PostgreSQL Query Analysis
```sql
-- Enable slow query log
ALTER SYSTEM SET log_min_duration_statement = 100;
-- Check query performance
SELECT * FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 20;
```

## Test Data Setup

### Seed Data Requirements
```bash
# Create test data
php artisan db:seed --class=PerformanceTestDataSeeder

# Required data:
- 1000+ students
- 50+ exams
- 5000+ questions
- 100+ exam packages
- Sufficient token balances
```

## Monitoring During Tests

### Key Metrics to Collect

1. **Application Metrics**
   - Response times (p50, p95, p99)
   - Request rate (requests/second)
   - Error rate
   - Throughput

2. **Database Metrics**
   - Connection pool usage
   - Query execution times
   - Slow query count
   - Lock wait time

3. **Redis Metrics**
   - Connection count
   - Command execution time
   - Memory usage
   - Hit/miss ratio

4. **System Metrics**
   - CPU usage
   - Memory usage
   - Disk I/O
   - Network I/O

## Test Execution Schedule

### Phase 1: Development Testing (Day 1-2)
- [ ] Create test scripts
- [ ] Set up test data
- [ ] Run baseline tests
- [ ] Fix any critical issues

### Phase 2: Staging Testing (Day 3-4)
- [ ] Run all scenarios on staging
- [ ] Validate monitoring setup
- [ ] Document baseline metrics
- [ ] Optimize if needed

### Phase 3: Production Testing (Day 5-6)
- [ ] Run tests during off-peak hours
- [ ] Monitor production metrics
- [ ] Validate auto-scaling
- [ ] Document results

### Phase 4: Analysis & Reporting (Day 7)
- [ ] Analyze all test results
- [ ] Create performance report
- [ ] Provide recommendations
- [ ] Update documentation

## Risk Mitigation

### Backup & Rollback Plan
1. Database backup before tests
2. Redis snapshot before tests
3. Application rollback plan
4. Monitoring alerts configured

### Test Isolation
- Use dedicated test database if possible
- Separate Redis instance for testing
- Test during low-traffic periods
- Clear cache before each test run

## Deliverables

1. **Performance Testing Scripts** (`tests/performance/`)
   - k6 test scripts
   - Database validation scripts
   - Redis validation scripts

2. **Performance Report** (`PERFORMANCE_TEST_RESULTS.md`)
   - Baseline metrics
   - Load test results
   - Bottleneck analysis
   - Recommendations

3. **Monitoring Setup**
   - Dashboards configured
   - Alerts configured
   - Runbooks updated

4. **CI/CD Integration**
   - Automated performance tests
   - Performance regression detection

## Success Metrics Summary

| Metric | Target | Measured |
|--------|--------|----------|
| p95 Response Time (baseline) | <200ms | ___ |
| p95 Response Time (1000 users) | <200ms | ___ |
| p95 Response Time (2000 users) | <200ms | ___ |
| Concurrent Users Supported | 1000+ | ___ |
| Error Rate | <0.1% | ___ |
| Token Deduction (500 concurrent) | 100% success | ___ |
| Database Queries (p95) | <100ms | ___ |
| Redis Operations (p95) | <5ms | ___ |

## Next Steps

1. ✅ Create performance testing plan (this document)
2. ⏳ Set up testing infrastructure
3. ⏳ Create test scripts
4. ⏳ Execute tests
5. ⏳ Analyze results
6. ⏳ Document findings and recommendations

---

**Document Owner:** Senior Golang Engineer (ENI-63)
**Last Updated:** April 8, 2026
**Status:** In Progress