Infrastructure Upgrades

Instructions

Upgrade the ITI 9-container Docker stack safely. Every upgrade follows the same pattern: backup, verify baseline, upgrade, verify, and optionally rollback.

Pre-upgrade checklist (MANDATORY before any upgrade):

1. Run backup: bash backup.sh
2. Verify backup files are non-zero in Archives/
3. Capture resource baseline: docker stats --no-stream > Archives/docker-stats-pre-upgrade-$(date +%Y%m%d).log
4. Run smoke tests: pytest test_docker_health.py test_n8n_webhooks.py -m smoke -v
5. Record current image versions: docker compose images
6. Ensure no critical workflows are currently executing (check n8n Executions UI)

Routine image updates (monthly):

cd ITI/infrastructure/n8n-dify
docker compose pull
docker compose up -d

Then run the full post-upgrade verification below.

n8n version upgrades:

• n8n uses n8nio/n8n:latest — a docker compose pull upgrades automatically
• Encryption key: N8N_ENCRYPTION_KEY in .env must never change across upgrades; it encrypts stored credentials. Losing this key makes all n8n credentials unrecoverable.
• n8n runs internal DB migrations on startup — watch docker compose logs n8n for migration output
• After upgrade: verify all workflows are active (GET /api/v1/workflows?active=true)
• If webhooks stop responding: the upgrade may have deactivated workflows; re-publish each one

Dify version upgrades:

• dify-api, dify-worker, and dify-web must always be the same version
• dify-sandbox and dify-plugin-daemon are pinned to specific versions in docker-compose.yml — do not blindly update these
• Upgrade procedure:

1. Check Dify release notes for breaking changes
2. Update image tags in docker-compose.yml if pinning (or pull :latest)
3. docker compose up -d — Dify runs its own DB migrations
4. Verify Dify UI loads at http://localhost:3000
5. Verify API proxy at http://localhost:3001/console/api/setup
6. Test KB retrieval via API

PostgreSQL major version upgrades (e.g., pg16 → pg17): This is the highest-risk upgrade. The procedure:

1. Full backup of all three databases: n8n, dify, dify_plugin
2. docker compose down
3. Update image in docker-compose.yml: pgvector/pgvector:pg17
4. Remove the old volume: docker volume rm n8n-dify_postgres_data (data is in the SQL dump)
5. docker compose up -d postgres — wait for healthy
6. Restore databases:

   docker exec -i iti-postgres psql -U postgres < Archives/n8n-db-YYYYMMDD.sql
   docker exec -i iti-postgres psql -U postgres < Archives/dify-db-YYYYMMDD.sql

1. Verify pgvector extension: docker exec iti-postgres psql -U postgres -d dify -c "SELECT extversion FROM pg_extension WHERE extname='vector'"
2. docker compose up -d — bring up remaining services
3. Run full test suite

Redis upgrades:

• Redis 7-alpine is a minor-version-pinned image; patch updates are automatic on pull
• For major version upgrades: Redis is backward-compatible with AOF files across minor versions
• After upgrade: verify redis-cli ping returns PONG and check redis-cli INFO server for version

Post-upgrade verification (MANDATORY after any upgrade):

1. docker compose ps — all 9 containers healthy
2. Run smoke tests: pytest test_docker_health.py test_n8n_webhooks.py -m smoke -v
3. Run integration tests: pytest test_n8n_workflows.py test_postgres_pgvector.py -v
4. Verify n8n workflows are active: count via API should match pre-upgrade
5. Verify Dify KBs are accessible: list datasets via API
6. Check n8n Executions for any new errors post-upgrade
7. Compare resource usage to pre-upgrade baseline

Rollback procedures:

Per-service rollback:

# Revert a single service to previous image
docker compose stop <service>
# Edit docker-compose.yml to pin the previous version
docker compose up -d <service>

Full stack rollback:

docker compose down
# Restore database from pre-upgrade backup
docker exec -i iti-postgres psql -U postgres -d n8n < Archives/n8n-db-YYYYMMDD.sql
docker exec -i iti-postgres psql -U postgres -d dify < Archives/dify-db-YYYYMMDD.sql
# Revert image tags in docker-compose.yml
docker compose up -d

Version pinning strategy:

• Pin in production: pgvector/pgvector:pg16, redis:7-alpine, dify-sandbox:0.2.14, dify-plugin-daemon:0.5.3-local
• Use :latest for active development only: n8nio/n8n:latest, langgenius/dify-api:latest
• When a :latest upgrade causes issues, pin to the last known-good version from docker compose images output
• Document pinned versions in a comment in docker-compose.yml