Skip to content

TODO

Frontend

Dashboard

  • On the map I'd like to be able to right click and have an option to create a location (storage or delivery) at the point selected.
  • The location editor now opens with the clicked coordinates prefilled.
  • Clicking an existing zone now opens the popup with an Edit button that launches the location editor.
  • Adding a location that overlaps an existing geofence in the same campaign is prevented, but the message "Unable to save location." is too vague to help the user understand why. Fixed 2026-11-09: the modal now names the overlapping locations from the API's 409 response (describeLocationSaveError).
  • When deleting a location, do we consider cascading it to remove history records where a tracker entered the zone?

Campaign Management

  • The Open locations button at the top of the page is unnecessary. Removed from the admin app's Campaign Management page (tracker-admin-svelte/.../clients/+page.svelte) -- it duplicated the "Open locations" icon already on each campaign/child-run card.

Unified CRUD

  • On the campaign management page the interface does not work well on mobile.
  • I think the mobile display should stack as rows not columns, so each selection fills beneath the next.
  • On selecting a client the mobile interface should jump to brand, on selecting a brand jump to campaign. Fixed all three together: the Clients/Brands/Campaigns/Child runs columns were a fixed-height side-by-side row, which just doesn't fit four columns on a phone. Built a shared EntityColumnGroup component (both apps' campaign management pages now use it, replacing four hand-written EntityColumns each) that keeps the side-by-side layout at the lg breakpoint and up, and below it stacks the columns as full-width rows, each with its own capped height and internal scroll. Selecting an item smooth-scrolls the next column into view (mobile only -- on desktop everything's already visible side by side).

Tracker Cards

  • The tooltips on mobile flow off screen to the left for cards that are in the left column. Can the tooltips be aligned to the right for left hand cards? Fixed by flipping the status tooltip outward on compact card grids, while leaving the wider layouts unchanged.
  • Trying to scroll in mobile causes the drag/drop trigger and the wheel appears. Is there anything to be done to make scrolling better? Fixed by allowing vertical touch panning on move-enabled cards/rows and only starting the radial drag wheel from mouse input.

Admin Panel

Users

  • The user invite gets sent out with the link to the admin panel.
  • The invite should have the frontend link, unless they are an admin, then it should have both. Fixed: the invite link was derived from whichever app's origin submitted the create-user request (always the admin panel, since that's the only place POST /users is called from), not the invited user's own role. Now role-based — ADMIN_URL/FRONTEND_URL are both real settings, and the email includes the admin panel link only when "admin" is in the new user's roles.
  • The user cards should behave like tracker cards. The edit/delete only appears on hover. Fixed in the shared EntityCard component (also used by clients and campaign management), so all three pick up the same hover-reveal behaviour as TrackerCard.
  • User cards could take up less space. Two columns, unless mobile? Added an opt-in columns prop to EntityColumn (default stays 1, so clients/campaign management are unaffected) and set it to 2 on the Users page; falls back to a single column below the sm breakpoint.
  • Cards could show the clients they have access to instead of just "1 client" Now lists the actual client names (comma-separated, capped at 3 with "+N more"), resolved from the clients already loaded for the editor modal.
  • Add a filter to show users with specific client access. Added a "Client access" dropdown next to the Role filter. Along the way, found that User.client_list/User.roles used the generic sqlalchemy.ARRAY type instead of the Postgres-specific one, so .contains() raised NotImplementedError at query time -- this was already broken for the pre-existing Role filter too, just never exercised by a test. Fixed the column type (no migration needed -- the actual DB column was always a real Postgres array; this was a Python-level type annotation bug). Admins match every client filter since they implicitly have access to all clients.
  • Check that no password reset email is sent unless the user is using local auth. Already the case: request_password_reset, the only sender, returns early for users with an auth_provider.

Authentication

better-auth

  • Did we over engineer our own auth? Would be better using https://better-auth.com/docs/concepts/oauth Discussed and rejected. The current JWT + OIDC provider-registry setup (app/core/oidc.py) is a fairly standard size for what it does, not over-engineered. better-auth is TypeScript-only (expects a Node/Bun/Deno backend with Drizzle/Prisma/Kysely) -- this app's auth lives in the Python/FastAPI backend, so adopting it would mean standing up a separate Node auth microservice or rewriting auth in TypeScript, not a drop-in swap.
  • Would it give us scope for other auths facebook/github? Not a reason to switch -- GitHub/Facebook fit the existing _PROVIDERS-dict pattern already used for Google/Microsoft (with per-provider quirks, e.g. GitHub isn't strictly OIDC-compliant so it needs a token-exchange/userinfo step instead of JWKS validation). A scoped addition to the current architecture, not a reason to migrate.

Backend API

Alembic

  • Can we merge the alembic migrations so we don't have branches? Linearized both branch points (alembic heads was already a single head -- the "branches" were two already-resolved fork/merge pairs inside the history, not a live problem). Checked every environment's current stamped revision first (local dev and staging both at head; production intentionally 11 migrations behind, on 46408209ddfd, well before either fork) to confirm nothing was stamped at a revision being removed. Reordered the two branch pairs by creation date, repointed down_revisions to chain them sequentially instead of merging, and deleted the now-unneeded pure no-op merge revision (ab12cd34ef56; the other merge point, e3f1a2b4c5d6, had real schema changes so it was kept, just with a single down_revision instead of a tuple). Verified by building a completely fresh database through the whole chain from scratch -- all 36 revisions apply cleanly in the new linear order, ending at the same head (d0e1f2a3b4c5) as before.
  • Once we get staging, and production to the current head we should flatten the migrations using alembic revision --autogenerate -m "Schema Baseline" Done 2026-11-09, with every environment at 916c6ddda1e8. Autogenerate wasn't usable: the old migrations created functions, triggers, an enum and a hypertable that it can't see. Also, the old chain could no longer build a new database, because 0e7784d6e686 needed location_reports to be a hypertable, which only scripts/create_timescaledb_views.sql creates. The 40 migrations are replaced by one baseline that runs alembic/baseline/916c6ddda1e8_schema.sql, a filtered pg_dump -s of a database at head. It keeps the revision ID 916c6ddda1e8, so existing databases are already at head and need no alembic stamp. Checked by building a new database through scripts/run_migrations.sh and diffing its public schema against a migrated one: they differ only in TimescaleDB internals.
  • scripts/create_timescaledb_views.sql recreates match_geofences() on every deploy. Nothing calls it, and it reads the legacy location_history view. Drop it from the script and the databases. Removed from the script; migration a7c3e9d2f1b4 drops it.
  • The baseline keeps three no-op "reminder" functions from old migrations (location_aggregation_fix_reminder, location_reports_join_fix_reminder, timescaledb_features_reminder). Drop them in a new migration. Dropped by migration a7c3e9d2f1b4.
  • scripts/apply_timescaledb_features.sh exports everything in .env, which overrides the container's POSTGRES_SERVER=db with localhost, so scripts/run_migrations.sh fails at that step inside the dev container. Environment variables that are already set should take precedence. Fixed: it now skips any .env variable that is already set.

Notifications

  • Allow a user to set alert/notifications of tracker devices. Get notified by email on status changes.
  • Extend notification/alerts for when a tracker has "gone dark".

Schema Improvement Notes

  • Several geography-backed tables currently expose both legacy and modern GIST index names on the same column. If backward compatibility is no longer required, consider keeping one canonical index per table. Done in migration b8d4f0e3a2c5. The duplicates came from GeoAlchemy2, which indexes Geography columns automatically, plus a second index the models declared. Delivery and storage locations keep GeoAlchemy2's idx_*_location. location_reports keeps neither: nothing filters it by location, so on the hypertable they were pure write cost.

Support Tooling

  • Built an MCP-based read-only diagnostic server so an AI agent can help triage "tracker hasn't reported" / "not in the expected zone" support cases. See docs/development/mcp-diagnostic-server-design.md for the on-demand ECS + SSM tunnel access pattern and tool surface. Live in local dev and staging; prod still needs its IAM policy attached and secret/role bootstrapped (see that doc's Next Steps).

Infrastructure

  • Move the stack from amd64 to arm64 (Graviton) for cost/perf, planned for the next OS upgrade. See docs/development/arm64-migration-assessment.md for what's actually involved - the Fargate/container side is a fairly clean win, but the self-managed EC2 Postgres/TimescaleDB host (infra/modules/database) is a separate, higher-risk migration that shouldn't be coupled to it.
  • Delete the legacy tracker ALBs glimpse-tracker-api (Apr 2025) and tracker-admin-panel (May 2025) in the default VPC (vpc-a258d4ca), with their target groups. Both tracker hostnames already resolve to tracker-restapi-production-alb. Check Cloudflare for any other record pointing at them first. Saves ~$45/mo and 6 public IPv4s (from the 2026-10 VPC cost review). Done 2026-10-22: no Cloudflare/Route 53 records, 0 targets, no requests in 14 days; ALBs and their 3 target groups deleted.
  • Remove leftovers from the shared Timescale cluster / pgbackrest era (prod has its own DB host since then):
  • stale postgres_server / postgres_read_port in infra/envs/prod/terraform.tfvars (and the matching outputs);
  • aws_vpc_security_group_egress_rule.ecs_to_existing_timescale (ECS -> 172.31.0.0/16:5432);
  • tracker_restore_host pgbackrest variables, the pgbackrest cipher secret lookup, and the SSH rules to/from the repository VPC. Also removed pgbackrest from the host's bootstrap template and the unused pgbackrest inputs of infra/modules/database. Removed from code 2026-11-09; takes effect at the next prod apply.
  • the tracker/production/pgbackrest-cipher-pass secret in Secrets Manager (created out of band, so not in Terraform). It is the key to the old pgBackRest backups, so in this order: apply v2.4.1 to prod (the deployed state still references it); have the tracker stanza removed from the shared repo host; delete the secret with --recovery-window-in-days 30; then purge pgBackRest from the prod DB host (apt-get purge pgbackrest, remove /etc/pgbackrest, which holds a plain-text copy of the passphrase);
  • outside this repo (other owner): the old cluster SG sg-070206889a0cf3e86 still allows 10.40.16.0/24 on 5432/5433, and the tracker stanza still sits on the shared pgbackrest repo host;
  • then the tracker_vpc_peering module, once nothing else uses the peering. Nothing in this repo does any more, but check first that nothing outside it does (for example the Icinga agent on the DB host).
  • Delete the manual data-volume snapshots (2026-05-21, 2026-08-07 "pre-v2-frontend", 2026-09-07 "pre-import", 2026-10-22 "pre-backup-work", 2026-10-22 "pre-tsdb-upgrade") once DLM has 14+ days of history. Done by hand 2026-11-09.
  • Prod TimescaleDB job 1007 move_old_data_to_archive has failed every run (2,303, "cache lookup failed for function 0"); it targets the long-gone location_history table. Delete the job and the leftover functions (move_old_data_to_archive, migrate_nearest_city_data, update_nearest_city, audit_tracker_status_change). Migration c9e5a1f4b3d6 does both at the next deploy.
  • Drop the ad-hoc prod tables (bacardi, bayer, devices, macs_to_remove, tracker_import_staging, trackers1000, trackers_correct) by hand, after checking that nobody needs their data. Deliberately not a migration. Done by hand 2026-11-09.
  • Staging: refresh_location_history() is owned by postgres, so the bootstrap's CREATE OR REPLACE fails on every deploy (harmless). Run ALTER FUNCTION refresh_location_history() OWNER TO tracker once. Done 2026-11-09; the owner is now tracker.
  • Staging's TimescaleDB extension is 2.27.0 with the 2.28.2 package installed; run ALTER EXTENSION timescaledb UPDATE (fresh psql -X session) so staging matches prod's 2.28.2. Done 2026-11-09; staging now runs 2.28.2.
  • Move prod onto infra/modules/database (like staging) so staging tests what prod runs. The data volume must become a standalone aws_ebs_volume via state moves; adopt the existing backup bucket (names already match).
  • Decision (2026-10 VPC cost review, item 6): keep the tracker ALB separate rather than folding it into backend-api-ingestion. Folding would make it cross-VPC over the peering, lose tracker's own WAF ACL and split ownership, all to save ~$25/mo.

Geofence Status Pipeline

  • Investigate why some trackers sit inside a delivery/storage geofence but current_status never updates from IN_TRANSIT. Surfaced by the find_status_zone_mismatches MCP tool (see docs/development/mcp-diagnostic-server-design.md) - confirmed 4 real cases in dev data, all production run 108, all ~12-32m inside the "A&S Packing" storage zone. Likely candidate: services/unified_geofence_service/taskiq_service.py's _determine_status_with_hysteresis/_handle_no_status_change path. Once staging/prod MCP access is live, run the same tool there first to see how widespread this is before deciding on a fix.
  • Prod: ~95% of geofence event inserts failed with fk_geofence_events_location_report_composite violations since 2026-08-07 (events fell from ~28k/day to ~300/day). Fixed 2026-10-22. Cause: TimescaleDB < 2.21.1 bug (timescaledb#8336) - FK checks against a hypertable fail once Postgres switches the cached RI plan to a generic plan, i.e. from a connection's 6th execution, so pooled connections failed almost every insert. Prod ran 2.18.2; dev (2.22) and staging (2.27) were unaffected. Confirmed on prod (inserts 1-5 OK, 6+ fail in one session); mitigated with plan_cache_mode = force_custom_plan, then fixed by upgrading prod to TimescaleDB 2.28.2 (PostgreSQL 16.15) and removing the mitigation.
  • Decide whether to regenerate the transit/dwell geofence events lost between 2026-08-07 and 2026-10-22 from the stored location reports.
  • Geofence event failures are silent: _create_geofence_event_only() logs and returns False, and batches report errors=0. Surface them as batch errors so a regression like the one above is visible. Fixed 2026-11-09: both write paths roll back and re-raise, so the batch records the error.
  • Dev databases built from setup scripts lack fk_geofence_events_location_report_composite (added by hand on 2026-10-22), which is why dev never showed the FK failures. Make dev setup run the full migrations so dev matches staging/prod. Resolved by the migration flatten: dev setup runs alembic upgrade head, and the baseline includes the FK.
  • GeofenceEvent.timestamp is a naive DateTime in the model (app/models/geofence_event.py:57) while the column is timestamptz. Fixed 2026-11-09, along with created_at: both are now DateTime(timezone=True).
  • The unified-geofence worker sometimes logs a burst of psycopg.errors.ConnectionTimeout exactly 10 seconds after a new task starts (connect_timeout: 10 in services/shared/database.py), from process_pending_location_reports. Seen on staging at the 2026-11-05 and 2026-11-09 16:00 deploys, but not at the 15:25 one. No further errors follow it, but it is not yet confirmed that the worker processes reports normally afterwards. Likely the first connection racing the task's network setup (ENI attach or security group) or the database host during the rollout. Check whether prod shows it too, then consider retrying the first connection with backoff at worker start.

Test Suite

  • Fix pre-existing test failures surfaced while verifying the tracker_status_service removal (2026-07-23). Fixed 2026-11-09:
  • test_unified_geofence_processing.py and test_unified_geofence_service.py tested UnifiedGeofenceService, deleted on 2025-11-27. Removed. Their status-determination cases (delivery, storage, delivery-over-storage priority, in transit) had no other coverage, so they were ported to TestStatusDetermination in test_unified_geofence_taskiq_service.py.
  • test_cascade_deletion_maintains_integrity passes again.
  • test_api_router_inclusion: since FastAPI 0.139, app.routes holds included-router entries without .path. It now reads the paths from the OpenAPI schema.
  • test_settings_cors_origins_default: the expected list lacked the Svelte dev ports and the staging hostnames.
  • TestDeliveryLocationCoordinates: DeliveryLocation.get_coordinates calls query().filter().first(), but the mock only stubbed query().first().
  • test_location_report_processing_workflow failed depending on row order: a tracker leaked into the dev database by an earlier run shared its fixed hashed key. It now uses a unique key per run.
  • Tests run against the dev database (POSTGRES_DB, default tracker), isolated only by a rolled-back transaction. Committed test data leaks into dev, and dev data can collide with tests. Give the tests their own database, built from the Alembic baseline.
  • trackers.hashed_advertisement_key is not unique, and _get_tracker_by_hashed_key takes .first() with no ordering. If a key is shared (for example by a reused tracker), reports can be attributed to an arbitrary tracker. Count duplicate keys in prod, then decide between a unique constraint and a deterministic choice. Done 2026-11-09: prod, staging and local had no duplicates, so migration d2f6b8a4c1e7 adds uq_trackers_hashed_advertisement_key, and POST /trackers returns 409 for an existing key. The database now holds one tracker per key even when two physical devices share it. The column previously had no index, so per-report lookups and the fetcher's joins also stop scanning trackers.

Documentation

Usage Guide

  • The user guide needs to be updated to cover the new design and move to the unified crud in Campaign Management Added a "Managing Clients, Brands, and Campaigns" section covering the drill-down interaction, hover-to-reveal edit/delete, mobile behavior, child runs, and per-run tracker management. Cross-linked from "Setup — Required Order".

Remove Stale Documents

  • We need to do a review of documentation and remove cruft relating to the previous React version and any other outdated materials. Rewrote the Admin Panel (8 files) and Front End (7 files) sections wholesale -- both described the deleted React/react-admin apps as current, with dead paths and code samples. Fixed scattered stale mentions in docs/index.md, reference/system-architecture.md, and several guides/services docs. Left legitimate historical/migration docs (the Svelte migration assessments, the cutover TODO) untouched -- they're supposed to talk about the old React app for comparison.
  • The documentation requires a restructure as the top level menu contains too many items and it looks disorganised. Root cause: mkdocs-awesome-pages-plugin defaults to reading .pages files, but every nav file in this repo is named .nav.yml and the plugin was never told to look for that instead -- so none of the curated navigation was ever actually being read, and mkdocs was silently falling back to a flat alphabetical dump of all ~90 pages. Fixed the plugin config (one line), fixed the "*" wildcard token (should be "..." for this plugin version), fixed a broken nested-path reference and a dead redoc link, and moved/removed the handful of stray top-level directories that had no real content (an empty dir, a confusingly-duplicate database/, asset-only dirs). Verified by building the full site and confirming the rendered nav matches the curated structure.

General Behaviour

  • When a tracker changes status to delivered consider not tracking any further activity.
  • We could remove it from the queue so it no longer asks Apple for FindMy reports
  • or do we keep tracking but never change status from delivered?