Skip to content

Redis Lock Inspection

This runbook explains how to inspect the tracker fetcher lock keys stored in Redis.

Use it when the tracker fetcher logs show No trackers to process even though rows exist in the database. In that case, the usual cause is that another worker still holds the distributed tracker locks.

What This Does

The tracker fetcher uses Redis keys named like:

tracker_lock:<tracker_id>

The lock value stores the worker container ID and acquisition timestamp:

<container_id>:<unix_timestamp>

How To Run

Open an ECS Exec shell in the same network path as the tracker services.

The easiest way is to use the dedicated shell helper, which starts a temporary task and opens the shell for you:

AWS_PROFILE=glimpse-staging ./scripts/open_tracker_ecs_shell.sh

When the script prints:

Opening ECS Exec session...

you are already in the right shell.

Inside the ECS shell, inspect one or more locks:

python scripts/inspect_tracker_locks.py --tracker-id 2059 --tracker-id 2397

To scan the current lock keys:

python scripts/inspect_tracker_locks.py --all --limit 20

To delete specific lock keys and let the worker reacquire them:

python scripts/inspect_tracker_locks.py --delete --tracker-id 2059 --tracker-id 2397

If you really need to clear every matched tracker_lock:* key, require an explicit --force:

python scripts/inspect_tracker_locks.py --delete --all --force

Example Output

{
  "count": 2,
  "keys": [
    {
      "acquired_at": 1714201262,
      "exists": true,
      "key": "tracker_lock:2059",
      "owner": "ip-10-40-0-174.eu-west-2.compute.internal",
      "raw_value": "ip-10-40-0-174.eu-west-2.compute.internal:1714201262",
      "ttl": 1791
    }
  ]
}

How To Interpret The Result

  • exists: true and a positive ttl means the lock is currently held.
  • ttl: -2 means the key does not exist.
  • ttl: -1 means the key exists but does not expire, which should not happen for these locks.
  • owner tells you which container acquired the lock.
  • acquired_at is the Unix timestamp when the lock was created.
  • Deleting the keys is safer than flushing Redis, because it only removes tracker locks.

What To Check Next

  • If the same container still exists, let it finish or investigate why it is stuck.
  • If the owner container is gone but the lock still has a long TTL, wait for expiry or investigate the cleanup path.
  • If no locks exist and the worker still logs No trackers to process, re-check the eligibility query in the database.