Skip to content

TaskiQ Container Deployment Guide

This guide explains how to run the new TaskiQ-based microservices that have replaced the legacy Celery infrastructure.

Overview

The TaskiQ architecture consists of three main microservices:

  • Tracker Fetcher Service - Handles Apple FindMy API calls and location report fetching
  • Unified Geofence Service - Handles geofencing detection and geocoding operations
  • Notification Service - Handles database notifications and monitoring

Each service can run in multiple modes: worker (background processing), api (HTTP endpoints), or once (single execution).

Quick Start

Development Environment

Start all TaskiQ services for development:

cd services/docker
docker compose --profile dev up -d

This starts:

  • tracker-fetcher-dev - Background worker for tracker fetching
  • unified-geofence-dev - Background worker for geofencing/geocoding
  • notification-service-dev - Database notification listener

Development with API Access

To also enable direct API access to services:

cd services/docker
docker compose --profile dev --profile dev-api up -d

This additionally starts:

  • tracker-fetcher-api-dev - HTTP API on port 8001
  • unified-geofence-api-dev - HTTP API on port 8003

Production Environment

For production deployment:

cd services/docker
docker compose --profile services up -d

Service Profiles

Development Profiles

  • dev - Core TaskiQ worker services with live code reloading
  • dev-api - Optional HTTP API endpoints for direct service access

Production Profiles

  • services - Production TaskiQ services with scaling support
  • monitoring - Optional monitoring tools (if needed)

Service Details

Tracker Fetcher Service

Purpose: Fetches location reports from Apple FindMy API

Development Container: tracker-fetcher-dev

Production Container: tracker-fetcher

Key Environment Variables:

ANISETTE_SERVER=http://anisette:6969  # Required
FETCH_INTERVAL=300                    # 5 minutes between batches
MAX_KEYS_PER_BATCH=64                # Apple API batch size

API Endpoints (when using dev-api profile):

  • GET /health - Service health check
  • POST /fetch/batch - Trigger batch fetch
  • POST /fetch/tracker/{id} - Fetch specific tracker
  • GET /metrics - Service metrics

Unified Geofence Service

Purpose: Processes location reports for geofencing and geocoding

Development Container: unified-geofence-dev

Production Container: unified-geofence

Key Environment Variables:

GEOFENCE_PROCESSING_INTERVAL=300      # 5 minutes between cycles
GEOCODING_BATCH_SIZE=50               # Locations per geocoding batch
GEOCODING_MAX_API_CALLS=50            # API rate limiting

API Endpoints (when using dev-api profile):

  • GET /health - Service health check
  • POST /process/location-report/{id} - Process single report
  • POST /process/pending - Process pending reports
  • GET /metrics - Service metrics

Notification Service

Purpose: Handles database notifications and pipeline monitoring

Development Container: notification-service-dev

Production Container: notification-service

Modes:

  • listener - Database notification listener (default)
  • hybrid - Polling-based monitor
  • single - One-time processing
  • health - Health check only
  • cleanup - Clean old records

Manual Service Operations

Running Individual Services

You can run services manually for testing or debugging:

# Tracker Fetcher - single batch
docker compose run --rm tracker-fetcher-dev python -m services.tracker_fetcher.taskiq_main once

# Unified Geofence - single cycle
docker compose run --rm unified-geofence-dev python -m services.unified_geofence_service.taskiq_main once

# Notification Service - health check
docker compose run --rm notification-service-dev python -m services.notification_service.taskiq_main --mode health

Service Health Checks

Check service health:

# Via API (if dev-api profile is running)
curl http://localhost:8001/health  # Tracker Fetcher
curl http://localhost:8003/health  # Unified Geofence

# Via direct container execution
docker compose exec tracker-fetcher-dev python -m services.tracker_fetcher.taskiq_main once
docker compose exec unified-geofence-dev python -m services.unified_geofence_service.taskiq_main once

Service Metrics

Get service metrics:

# Via API (if dev-api profile is running)
curl http://localhost:8001/metrics  # Tracker Fetcher
curl http://localhost:8003/metrics  # Unified Geofence

# Via logs
docker compose logs tracker-fetcher-dev
docker compose logs unified-geofence-dev
docker compose logs notification-service-dev

Configuration

Required Environment Variables

Create a .env file in the services/docker directory:

# Database
POSTGRES_SERVER=db
POSTGRES_PORT=5432
POSTGRES_DB=tracker
POSTGRES_USER=tracker
POSTGRES_PASSWORD=your_password

# Redis (multi-database setup)
REDIS_HOST=dragonfly
REDIS_PORT=6379
REDIS_USERNAME=
REDIS_PASSWORD=your_redis_password

# Apple FindMy
ANISETTE_SERVER=http://anisette:6969

# Service Configuration
FETCH_INTERVAL=300
MAX_KEYS_PER_BATCH=64
GEOFENCE_PROCESSING_INTERVAL=300
GEOCODING_BATCH_SIZE=50
LOG_LEVEL=DEBUG

Production Scaling

Scale production services:

# Scale tracker fetcher to 2 replicas
TRACKER_FETCHER_REPLICAS=2 docker compose --profile services up -d

# Scale unified geofence to 3 replicas
UNIFIED_GEOFENCE_REPLICAS=3 docker compose --profile services up -d

Monitoring and Troubleshooting

Service Logs

View service logs:

# All services
docker compose logs -f

# Specific service
docker compose logs -f tracker-fetcher-dev
docker compose logs -f unified-geofence-dev
docker compose logs -f notification-service-dev

Health Monitoring

The services include comprehensive health monitoring:

  • Database connectivity - Automatic connection testing
  • Redis connectivity - Multi-database health checks
  • Processing metrics - Batch success rates, processing times
  • Business metrics - Ungeocoded reports, missing geofence events

Common Issues

Service won't start:

  1. Check environment variables are set correctly
  2. Verify database and Redis connectivity
  3. Check Docker network configuration

No processing activity:

  1. Verify trackers exist in database with valid production runs
  2. Check last_processed_at timestamps aren't too recent
  3. Verify Apple FindMy account configuration

High error rates:

  1. Check Apple FindMy API connectivity
  2. Verify Anisette server is running
  3. Review geocoding API limits and quotas

Migration from Celery

What Changed

  • No more Celery workers - Replaced with TaskiQ microservices
  • No more Celery Beat - Replaced with database-driven scheduling
  • No more Flower - Replaced with service-specific health endpoints
  • No more Redis queues - Replaced with database-driven processing

Key Differences

  • Database-driven: Processing is based on database queries, not Redis queues
  • Fair processing: Uses last_processed_at timestamps for fair distribution
  • Distributed locking: Redis-based locks prevent duplicate processing
  • Horizontal scaling: Multiple containers can run independently

Rollback Plan

If issues arise, you can temporarily revert to a previous version by:

  1. Stopping TaskiQ services: docker compose --profile dev down
  2. Checking out previous git commit
  3. Starting legacy services with appropriate profiles

Performance Tuning

Batch Sizes

Adjust batch sizes based on your load:

# For high-volume environments
MAX_KEYS_PER_BATCH=64
GEOCODING_BATCH_SIZE=100

# For low-volume environments
MAX_KEYS_PER_BATCH=32
GEOCODING_BATCH_SIZE=25

Processing Intervals

Adjust processing intervals:

# More frequent processing (higher load)
FETCH_INTERVAL=180                    # 3 minutes
GEOFENCE_PROCESSING_INTERVAL=180     # 3 minutes

# Less frequent processing (lower load)
FETCH_INTERVAL=600                    # 10 minutes
GEOFENCE_PROCESSING_INTERVAL=600     # 10 minutes

Resource Limits

Production services include resource limits:

deploy:
  resources:
    limits:
      cpus: "1.0"
      memory: 1G
    reservations:
      cpus: "0.5"
      memory: 512M

Adjust these based on your infrastructure capacity.

API Reference

Tracker Fetcher API

When running with dev-api profile, the Tracker Fetcher API is available on port 8001:

Health Check:

GET /health

Trigger Batch Fetch:

POST /fetch/batch?max_batch_size=32

Fetch Specific Tracker:

POST /fetch/tracker/123?days=7&force=true

Get Metrics:

GET /metrics

Unified Geofence API

When running with dev-api profile, the Unified Geofence API is available on port 8003:

Health Check:

GET /health

Process Location Report:

POST /process/location-report/456

Process Pending Reports:

POST /process/pending?limit=100

Cleanup Old Events:

POST /cleanup/geofence-events?retention_days=90

Get Status:

GET /status

Best Practices

Development

  1. Use dev profile for local development with live reloading
  2. Enable API profiles when you need direct service access
  3. Monitor logs regularly during development
  4. Test with single-run modes before deploying workers

Production

  1. Use services profile for production deployment
  2. Set appropriate resource limits based on your infrastructure
  3. Configure monitoring and alerting for service health
  4. Scale services based on processing load
  5. Regular health checks via API endpoints or direct execution

Security

  1. Use environment variables for all sensitive configuration
  2. Secure Redis with authentication and network isolation
  3. Limit API access to trusted networks only
  4. Regular credential rotation for external services

This completes the TaskiQ deployment guide. The new architecture provides better reliability, scalability, and maintainability compared to the legacy Celery system.