# Redis Monitoring & Auto-Recovery Solution

## Problem
Web application becomes inaccessible when Redis server fails, causing continuous redirect loops.

## Solution Overview
This solution provides multiple layers of protection:

### 1. **Failover Cache** ✅ (Already Applied)
- **Primary**: Redis cache
- **Fallback**: Database cache
- **Last Resort**: In-memory cache
- When Redis fails, application automatically switches to database cache

### 2. **Enhanced Redis Configuration** ✅ (Already Applied)
- Increased memory from 512MB to 1GB
- Added password authentication support
- Auto-restart policy: `always` (was `unless-stopped`)
- Memory limits and resource reservations
- Data persistence with AOF enabled

### 3. **Auto-Recovery Monitoring** ✅ (Created)
- Monitors Redis health every 5 minutes
- Auto-restart if Redis is not responding
- Restart cooldown to prevent restart loops
- Comprehensive logging

## Setup Instructions

### Step 1: Apply Configuration Changes
```bash
# Rebuild and restart containers with new configuration
cd /path/to/cbtQ
docker-compose -f docker-compose.prod.yml up -d --build

# Verify Redis is running
docker exec cbt-redis redis-cli ping
```

### Step 2: Setup Monitoring (Optional but Recommended)
```bash
# Make setup script executable and run it
chmod +x docker/redis-setup.sh
sudo docker/redis-setup.sh
```

### Step 3: Monitor Logs
```bash
# View Redis monitoring logs
tail -f /var/log/redis-monitor.log

# View Docker logs
docker logs cbt-redis --tail 100 -f
```

## Environment Variables

Add these to your `.env.prod` file:

```bash
# Redis Configuration
REDIS_PASSWORD=your_secure_password_here  # Optional but recommended
REDIS_PORT=6379
REDIS_MAX_CONNECTIONS=100
REDIS_TIMEOUT=5.0
REDIS_READ_TIMEOUT=5.0

# Cache Configuration
CACHE_STORE=failover  # Uses failover cache (Redis → Database → Array)
```

## How It Works

### Normal Operation
```
Request → Laravel → Check Cache → Redis → ✅ Cache Hit → Response
```

### When Redis Fails
```
Request → Laravel → Check Cache → Redis ❌ → Try Database Cache → ✅ Cache Hit → Response
```

### Auto-Recovery
```
1. Monitor detects Redis is down (every 5 minutes)
2. Attempt to restart Redis container
3. Verify Redis is responding
4. Log the result
5. Application automatically switches back to Redis when available
```

## Troubleshooting

### Redis Not Responding
```bash
# Check Redis status
docker ps | grep redis

# Check Redis logs
docker logs cbt-redis --tail 50

# Restart Redis manually
docker restart cbt-redis

# Test connection
docker exec cbt-redis redis-cli ping
```

### Application Still Redirecting
```bash
# Clear Laravel cache
php artisan cache:clear
php artisan config:clear
php artisan route:clear

# Check cache configuration
php artisan tinker
>>> config('cache.default')
// Should return: "failover"
```

### Monitoring Not Working
```bash
# Check if cron job is installed
crontab -l | grep redis-monitor

# Check logs
sudo tail -f /var/log/redis-monitor.log

# Manually test monitoring script
sudo docker/redis-monitor.sh
```

## Prevention Tips

1. **Regular Monitoring**: Check logs weekly
2. **Resource Limits**: Monitor Redis memory usage
3. **Backup**: Regular backups of Redis data
4. **Updates**: Keep Redis image updated
5. **Testing**: Test failover mechanism regularly

## Performance Impact

- **Failover Cache**: Minimal overhead (only when Redis fails)
- **Monitoring**: Negligible (runs every 5 minutes)
- **Memory**: Increased to 1GB (better for high traffic)

## Next Steps

1. Apply configuration changes
2. Setup monitoring script
3. Test failover mechanism
4. Monitor logs for first week
5. Consider Redis clustering for high availability

## Support

If issues persist:
1. Check logs: `/var/log/redis-monitor.log`
2. Check Docker logs: `docker logs cbt-redis`
3. Verify environment variables
4. Test failover: `docker stop cbt-redis` (then start after testing)

## Files Modified/Created

1. `config/cache.php` - Added failover cache configuration
2. `docker-compose.prod.yml` - Enhanced Redis configuration
3. `docker/redis-monitor.sh` - Monitoring and auto-recovery script
4. `docker/redis-setup.sh` - Setup script for monitoring