# Phase 3: Image Processing Pipeline - Implementation Guide

## Overview
This guide documents the implementation of an asynchronous image processing pipeline using Laravel queues. Images uploaded by clients are processed in the background to resize, compress, and convert them to WebP format.

## What Was Implemented

### 1. Image Processing Job

#### `app/Jobs/ProcessUploadedImage.php`
Background job for processing uploaded images:

**Job Properties:**
- **Max Attempts:** 3 retries
- **Backoff Strategy:** 1min → 5min → 10min
- **Timeout:** 5 minutes per attempt
- **Queue:** `image-processing`

**Processing Steps:**
1. **Download** original image from GCS temp location
2. **Process** image:
   - Resize to max width 1280px (maintain aspect ratio)
   - Compress to 75% quality
   - Convert to WebP format
3. **Upload** processed image to final location
4. **Delete** original temp file from GCS
5. **Log** completion for monitoring

**Error Handling:**
- Automatic retries with exponential backoff
- Comprehensive logging for all operations
- Failed jobs logged with full stack traces
- Non-critical operations (like delete) don't fail the job

**Methods:**
- `handle()` - Main job execution logic
- `downloadFromGcs()` - Download original file
- `processImage()` - Resize, compress, convert to WebP
- `uploadToGcs()` - Upload processed file
- `deleteFromGcs()` - Clean up temp file
- `getFinalPath()` - Generate final storage path
- `failed()` - Handle permanent job failures

### 2. Updated Upload Controller

#### Modified `app/Http/Controllers/UploadController.php`
Updated `confirmUpload()` method to dispatch image processing job:

```php
if ($type === 'image') {
    // Dispatch image processing job to queue
    ProcessUploadedImage::dispatch($fileId, $path, $tenantId);
}
```

**Behavior:**
- Images: Processing job dispatched to `image-processing` queue
- Audio: Stored as-is, no processing needed
- Both types return immediate confirmation to client

## Queue Configuration

### Setup Redis Queue Driver

The queue configuration is already set up in `config/queue.php`. To use Redis:

#### 1. Update `.env` File
```env
QUEUE_CONNECTION=redis
REDIS_CLIENT=phpredis
REDIS_HOST=127.0.0.1
REDIS_PASSWORD=null
REDIS_PORT=6379
```

#### 2. Verify Redis Configuration
Check `config/queue.php`:
```php
'redis' => [
    'driver' => 'redis',
    'connection' => env('REDIS_QUEUE_CONNECTION', 'default'),
    'queue' => env('REDIS_QUEUE', 'default'),
    'retry_after' => (int) env('REDIS_QUEUE_RETRY_AFTER', 90),
    'block_for' => null,
    'after_commit' => false,
],
```

### Queue Database Tables

Laravel needs database tables for queue management. Run migrations:

```bash
php artisan queue:table
php artisan migrate
```

This creates:
- `jobs` - Queue jobs table
- `job_batches` - Job batching table
- `failed_jobs` - Failed jobs table

## Running Queue Workers

### Development Environment

#### Option 1: Using Laravel Sail
```bash
# Start queue worker in background
./vendor/bin/sail artisan queue:work redis --queue=image-processing --tries=3
```

#### Option 2: Direct PHP
```bash
php artisan queue:work redis --queue=image-processing --tries=3
```

#### Option 3: With Logging
```bash
php artisan queue:work redis \
  --queue=image-processing \
  --tries=3 \
  --timeout=300 \
  --sleep=3 \
  --max-jobs=1000
```

**Parameters:**
- `--queue=image-processing` - Process only image-processing queue
- `--tries=3` - Max retry attempts
- `--timeout=300` - Max execution time per job (5 minutes)
- `--sleep=3` - Wait 3 seconds between jobs
- `--max-jobs=1000` - Process 1000 jobs before restarting

### Production Environment

#### Using Supervisor
Recommended for production to keep queue workers running:

**Install Supervisor:**
```bash
# Ubuntu/Debian
sudo apt-get install supervisor

# CentOS/RHEL
sudo yum install supervisor
```

**Create Supervisor Config:**
Create `/etc/supervisor/conf.d/cbt-queue-worker.conf`:

```ini
[program:cbt-queue-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/cbtQ/artisan queue:work redis --queue=image-processing --sleep=3 --tries=3 --timeout=300
autostart=true
autorestart=true
user=www-data
numprocs=2
redirect_stderr=true
stdout_logfile=/var/www/cbtQ/storage/logs/queue-worker.log
stopwaitsecs=3600
```

**Start Supervisor:**
```bash
sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl start cbt-queue-worker:*
```

**Monitor Workers:**
```bash
sudo supervisorctl status cbt-queue-worker:*
tail -f /var/www/cbtQ/storage/logs/queue-worker.log
```

#### Multiple Queue Workers
For high-traffic systems, run multiple workers:

```ini
numprocs=4
```

This will run 4 parallel workers processing the `image-processing` queue.

## Image Processing Flow

### Complete Upload & Processing Pipeline

```
┌─────────────┐
│   Client    │
└──────┬──────┘
       │
       │ 1. Get Upload URL
       ▼
┌─────────────┐
│ Laravel API │
│ /upload/url │
└──────┬──────┘
       │
       │ 2. Signed URL
       ▼
┌─────────────┐
│   Client    │
│  Uploads   │
└──────┬──────┘
       │
       │ 3. PUT to GCS
       ▼
┌─────────────┐
│ GCS Storage │
│  temp/...   │
└──────┬──────┘
       │
       │ 4. Confirm Upload
       ▼
┌─────────────┐
│ Laravel API │
│/upload/conf │
└──────┬──────┘
       │
       │ 5. Dispatch Job
       ▼
┌─────────────┐
│ Redis Queue │
└──────┬──────┘
       │
       │ 6. Process Job
       ▼
┌─────────────┐
│ Queue Worker│
└──────┬──────┘
       │
       │ 7. Download
       ▼
┌─────────────┐
│ GCS Storage │
│ temp/...    │
└──────┬──────┘
       │
       │ 8. Process
       ▼
┌─────────────┐
│  Image      │
│Processing   │
└──────┬──────┘
       │
       │ 9. Upload
       ▼
┌─────────────┐
│ GCS Storage │
│processed/..  │
└──────┬──────┘
       │
       │ 10. Delete Temp
       ▼
┌─────────────┐
│ GCS Storage │
│  (clean)    │
└─────────────┘
```

## File Path Structure

### Temp Storage (Before Processing)
```
temp/
  └── {tenant_id}/
      └── images/
          └── {file_id}.{ext}  # Original uploaded file
```

### Final Storage (After Processing)
```
processed/
  └── {tenant_id}/
      └── images/
          └── {file_id}.webp  # Processed file
```

## Image Processing Specifications

### Resize
- **Max Width:** 1280 pixels
- **Aspect Ratio:** Maintained
- **Upsizing:** Disabled (won't enlarge small images)

### Compression
- **Format:** WebP
- **Quality:** 75% (good balance of quality/size)
- **Compatibility:** All modern browsers support WebP

### Benefits
- **File Size:** Typically 25-35% smaller than JPEG
- **Quality:** Visually identical to original
- **Performance:** Faster load times
- **Bandwidth:** Reduced storage and transfer costs

## Monitoring & Logging

### Job Monitoring

#### View Queue Status
```bash
php artisan queue:monitor redis
```

#### View Failed Jobs
```bash
php artisan queue:failed
```

#### Retry Failed Jobs
```bash
# Retry all failed jobs
php artisan queue:retry all

# Retry specific failed job
php artisan queue:retry {job-id}
```

#### Clear Failed Jobs
```bash
php artisan queue:flush
```

### Logging

All image processing events are logged to `storage/logs/laravel.log`:

**Log Levels:**
- `info()` - Job started, completed successfully
- `warning()` - Non-critical failures (e.g., delete failure)
- `error()` - Processing failures with stack traces

**Log Format:**
```json
{
  "file_id": "550e8400-e29b-41d4-a716-446655440000",
  "temp_path": "temp/123/images/550e8400-e29b-41d4-a716-446655440000.jpg",
  "tenant_id": 123,
  "final_path": "processed/123/images/550e8400-e29b-41d4-a716-446655440000.webp"
}
```

## Performance Optimization

### Queue Configuration

#### High Traffic
```ini
numprocs=4  # 4 parallel workers
```

#### Low Traffic
```ini
numprocs=1  # 1 worker
```

#### Resource Constraints
Reduce timeout if processing is fast:
```bash
--timeout=180  # 3 minutes instead of 5
```

### Batch Processing
For bulk uploads, consider batching jobs:

```php
use Illuminate\Bus\Batch;
use Illuminate\Support\Facades\Bus;

$batch = Bus::batch([
    new ProcessUploadedImage($fileId1, $path1, $tenantId),
    new ProcessUploadedImage($fileId2, $path2, $tenantId),
    new ProcessUploadedImage($fileId3, $path3, $tenantId),
])->dispatch();
```

## Troubleshooting

### Issue: Queue worker not processing jobs
**Solutions:**
1. Check Redis is running: `redis-cli ping`
2. Verify queue connection: `php artisan queue:work --verbose`
3. Check `.env` has `QUEUE_CONNECTION=redis`
4. Ensure Redis server is accessible

### Issue: Jobs failing with timeout
**Solutions:**
1. Increase timeout: `--timeout=600` (10 minutes)
2. Check GCS connection speed
3. Verify image sizes are reasonable

### Issue: Failed jobs accumulating
**Solutions:**
1. Check logs for common errors
2. Verify GCS credentials are correct
3. Ensure sufficient disk space for temporary processing
4. Check Intervention Image installation

### Issue: Images not processing
**Solutions:**
1. Verify queue worker is running
2. Check `image-processing` queue is being processed
3. Confirm Redis has jobs: `redis-cli LRANGE queues:default 0 -1`

## Testing

### Manual Job Dispatch
```php
use App\Jobs\ProcessUploadedImage;

$fileId = 'test-file-id';
$tempPath = 'temp/123/images/test.jpg';
$tenantId = 123;

ProcessUploadedImage::dispatch($fileId, $tempPath, $tenantId);
```

### Test with Real Upload
1. Upload an image using Phase 2 API
2. Check queue: `php artisan queue:work --once`
3. Verify GCS has processed file
4. Check logs for success message

### Performance Testing
```bash
# Upload 10 images
# Monitor queue: watch -n 1 'php artisan queue:monitor redis'
# Check processing time in logs
```

## Security Considerations

### Isolation
- ✅ Jobs process files only from their tenant
- ✅ No cross-tenant access
- ✅ Each tenant has separate directory structure

### Validation
- ✅ Files already validated before upload (Phase 2)
- ✅ No malicious files reach processing pipeline
- ✅ Content type enforced by GCS signed URLs

### Cleanup
- ✅ Temp files automatically deleted after processing
- ✅ No leftover files in temp directory
- ✅ Failed jobs logged but don't leave temp files

## Integration with Phases

### Phase 1: GCS Migration
- ✅ Uses GCS for storage
- ✅ Compatible with migrated path format
- ✅ Private bucket access

### Phase 2: Signed URL Upload
- ✅ Receives uploaded files from temp directory
- ✅ Processes images after confirmation
- ✅ Audio files already in final location

### Phase 4: Secure Access
- ✅ Processed images in final location
- ✅ Ready for secure delivery
- ✅ Supports watermarking

## Best Practices

### Development
- ✅ Use `queue:work --once` for testing
- ✅ Enable verbose mode for debugging
- ✅ Monitor logs during development
- ✅ Test with various image sizes

### Production
- ✅ Use Supervisor for workers
- ✅ Monitor queue length
- ✅ Set up alerts for failed jobs
- ✅ Regular cleanup of failed jobs
- ✅ Monitor Redis memory usage

### Maintenance
- ✅ Regularly check `failed_jobs` table
- ✅ Monitor GCS storage costs
- ✅ Review processing logs weekly
- ✅ Update Intervention Image package regularly

## Next Steps

After completing Phase 3:
1. ✅ Queue infrastructure is set up
2. ✅ Images are processed asynchronously
3. ✅ Optimized WebP files generated
4. ✅ Temp files cleaned up automatically

Proceed to **Phase 4: Secure Access + Anti-Cheating**
- Secure file delivery with signed URLs
- Implement image watermarking
- Add anti-cheating detection
- Monitor suspicious activities