On the map I'd like to be able to right click and have an option to create a location (storage or delivery) at the point selected.
The location editor now opens with the clicked coordinates prefilled.
Clicking an existing zone now opens the popup with an Edit button that launches the location editor.
Adding a location that overlaps an existing geofence in the same campaign is prevented, but the message "Unable to save location." is too vague to help the user understand why.
Fixed 2026-11-09: the modal now names the overlapping locations from the API's 409 response (describeLocationSaveError).
When deleting a location, do we consider cascading it to remove history records where a tracker entered the zone?
Campaign Management
The Open locations button at the top of the page is unnecessary.
Removed from the admin app's Campaign Management page
(tracker-admin-svelte/.../clients/+page.svelte) -- it duplicated the
"Open locations" icon already on each campaign/child-run card.
Unified CRUD
On the campaign management page the interface does not work well on
mobile.
I think the mobile display should stack as rows not columns, so each
selection fills beneath the next.
On selecting a client the mobile interface should jump to brand, on
selecting a brand jump to campaign. Fixed all three together: the
Clients/Brands/Campaigns/Child runs columns were a fixed-height
side-by-side row, which just doesn't fit four columns on a phone.
Built a shared EntityColumnGroup component (both apps' campaign
management pages now use it, replacing four hand-written
EntityColumns each) that keeps the side-by-side layout at the lg
breakpoint and up, and below it stacks the columns as full-width rows,
each with its own capped height and internal scroll. Selecting an item
smooth-scrolls the next column into view (mobile only -- on desktop
everything's already visible side by side).
Tracker Cards
The tooltips on mobile flow off screen to the left for cards that are in the left column. Can the tooltips be aligned to the right for left hand cards?
Fixed by flipping the status tooltip outward on compact card grids, while leaving the wider layouts unchanged.
Trying to scroll in mobile causes the drag/drop trigger and the wheel appears. Is there anything to be done to make scrolling better?
Fixed by allowing vertical touch panning on move-enabled cards/rows and only starting the radial drag wheel from mouse input.
Admin Panel
Users
The user invite gets sent out with the link to the admin panel.
The invite should have the frontend link, unless they are an admin,
then it should have both. Fixed: the invite link was derived from
whichever app's origin submitted the create-user request (always
the admin panel, since that's the only place POST /users is
called from), not the invited user's own role. Now role-based —
ADMIN_URL/FRONTEND_URL are both real settings, and the email
includes the admin panel link only when "admin" is in the new
user's roles.
The user cards should behave like tracker cards. The edit/delete only
appears on hover. Fixed in the shared EntityCard component (also used
by clients and campaign management), so all three pick up the same
hover-reveal behaviour as TrackerCard.
User cards could take up less space. Two columns, unless mobile?
Added an opt-in columns prop to EntityColumn (default stays 1, so
clients/campaign management are unaffected) and set it to 2 on the
Users page; falls back to a single column below the sm breakpoint.
Cards could show the clients they have access to instead of just "1 client"
Now lists the actual client names (comma-separated, capped at 3 with
"+N more"), resolved from the clients already loaded for the editor
modal.
Add a filter to show users with specific client access. Added a
"Client access" dropdown next to the Role filter. Along the way,
found that User.client_list/User.roles used the generic
sqlalchemy.ARRAY type instead of the Postgres-specific one, so
.contains() raised NotImplementedError at query time -- this was
already broken for the pre-existing Role filter too, just never
exercised by a test. Fixed the column type (no migration needed --
the actual DB column was always a real Postgres array; this was a
Python-level type annotation bug). Admins match every client filter
since they implicitly have access to all clients.
Check that no password reset email is sent unless the user is using local auth.
Already the case: request_password_reset, the only sender, returns early for users with an auth_provider.
Authentication
better-auth
Did we over engineer our own auth? Would be better using https://better-auth.com/docs/concepts/oauth
Discussed and rejected. The current JWT + OIDC provider-registry setup
(app/core/oidc.py) is a fairly standard size for what it does, not
over-engineered. better-auth is TypeScript-only (expects a Node/Bun/Deno
backend with Drizzle/Prisma/Kysely) -- this app's auth lives in the
Python/FastAPI backend, so adopting it would mean standing up a
separate Node auth microservice or rewriting auth in TypeScript, not a
drop-in swap.
Would it give us scope for other auths facebook/github?
Not a reason to switch -- GitHub/Facebook fit the existing
_PROVIDERS-dict pattern already used for Google/Microsoft (with
per-provider quirks, e.g. GitHub isn't strictly OIDC-compliant so it
needs a token-exchange/userinfo step instead of JWKS validation). A
scoped addition to the current architecture, not a reason to migrate.
Backend API
Alembic
Can we merge the alembic migrations so we don't have branches?
Linearized both branch points (alembic heads was already a single
head -- the "branches" were two already-resolved fork/merge pairs
inside the history, not a live problem). Checked every environment's
current stamped revision first (local dev and staging both at head;
production intentionally 11 migrations behind, on 46408209ddfd,
well before either fork) to confirm nothing was stamped at a revision
being removed. Reordered the two branch pairs by creation date,
repointed down_revisions to chain them sequentially instead of
merging, and deleted the now-unneeded pure no-op merge revision
(ab12cd34ef56; the other merge point, e3f1a2b4c5d6, had real
schema changes so it was kept, just with a single down_revision
instead of a tuple). Verified by building a completely fresh database
through the whole chain from scratch -- all 36 revisions apply
cleanly in the new linear order, ending at the same head
(d0e1f2a3b4c5) as before.
Once we get staging, and production to the current head we should flatten the migrations using alembic revision --autogenerate -m "Schema Baseline"
Done 2026-11-09, with every environment at 916c6ddda1e8. Autogenerate
wasn't usable: the old migrations created functions, triggers, an enum
and a hypertable that it can't see. Also, the old chain could no longer
build a new database, because 0e7784d6e686 needed location_reports
to be a hypertable, which only scripts/create_timescaledb_views.sql
creates. The 40 migrations are replaced by one baseline that runs
alembic/baseline/916c6ddda1e8_schema.sql, a filtered pg_dump -s of
a database at head. It keeps the revision ID 916c6ddda1e8, so existing
databases are already at head and need no alembic stamp. Checked by
building a new database through scripts/run_migrations.sh and diffing
its public schema against a migrated one: they differ only in
TimescaleDB internals.
scripts/create_timescaledb_views.sql recreates match_geofences() on
every deploy. Nothing calls it, and it reads the legacy location_history
view. Drop it from the script and the databases.
Removed from the script; migration a7c3e9d2f1b4 drops it.
The baseline keeps three no-op "reminder" functions from old migrations
(location_aggregation_fix_reminder, location_reports_join_fix_reminder,
timescaledb_features_reminder). Drop them in a new migration.
Dropped by migration a7c3e9d2f1b4.
scripts/apply_timescaledb_features.sh exports everything in .env,
which overrides the container's POSTGRES_SERVER=db with localhost, so
scripts/run_migrations.sh fails at that step inside the dev container.
Environment variables that are already set should take precedence.
Fixed: it now skips any .env variable that is already set.
Notifications
Allow a user to set alert/notifications of tracker devices. Get notified by email on status changes.
Extend notification/alerts for when a tracker has "gone dark".
Schema Improvement Notes
Several geography-backed tables currently expose both legacy and modern GIST index names on the same column. If backward compatibility is no longer required, consider keeping one canonical index per table.
Done in migration b8d4f0e3a2c5. The duplicates came from GeoAlchemy2,
which indexes Geography columns automatically, plus a second index the
models declared. Delivery and storage locations keep GeoAlchemy2's
idx_*_location. location_reports keeps neither: nothing filters it by
location, so on the hypertable they were pure write cost.
Support Tooling
Built an MCP-based read-only diagnostic server so an AI agent can help
triage "tracker hasn't reported" / "not in the expected zone" support
cases. See
docs/development/mcp-diagnostic-server-design.md
for the on-demand ECS + SSM tunnel access pattern and tool surface. Live
in local dev and staging; prod still needs its IAM policy attached and
secret/role bootstrapped (see that doc's Next Steps).
Infrastructure
Move the stack from amd64 to arm64 (Graviton) for cost/perf, planned for
the next OS upgrade.
See docs/development/arm64-migration-assessment.md
for what's actually involved - the Fargate/container side is a fairly
clean win, but the self-managed EC2 Postgres/TimescaleDB host
(infra/modules/database) is a separate, higher-risk migration that
shouldn't be coupled to it.
Delete the legacy tracker ALBs glimpse-tracker-api (Apr 2025) and
tracker-admin-panel (May 2025) in the default VPC (vpc-a258d4ca),
with their target groups. Both tracker hostnames already resolve to
tracker-restapi-production-alb. Check Cloudflare for any other record
pointing at them first. Saves ~$45/mo and 6 public IPv4s (from the
2026-10 VPC cost review). Done 2026-10-22: no Cloudflare/Route 53 records,
0 targets, no requests in 14 days; ALBs and their 3 target groups deleted.
Remove leftovers from the shared Timescale cluster / pgbackrest era
(prod has its own DB host since then):
stale postgres_server / postgres_read_port in
infra/envs/prod/terraform.tfvars (and the matching outputs);
tracker_restore_host pgbackrest variables, the pgbackrest cipher
secret lookup, and the SSH rules to/from the repository VPC. Also
removed pgbackrest from the host's bootstrap template and the unused
pgbackrest inputs of infra/modules/database. Removed from code
2026-11-09; takes effect at the next prod apply.
the tracker/production/pgbackrest-cipher-pass secret in Secrets
Manager (created out of band, so not in Terraform). It is the key to
the old pgBackRest backups, so in this order: apply v2.4.1 to prod
(the deployed state still references it); have the tracker stanza
removed from the shared repo host; delete the secret with
--recovery-window-in-days 30; then purge pgBackRest from the prod DB
host (apt-get purge pgbackrest, remove /etc/pgbackrest, which holds
a plain-text copy of the passphrase);
outside this repo (other owner): the old cluster SG sg-070206889a0cf3e86
still allows 10.40.16.0/24 on 5432/5433, and the tracker stanza still
sits on the shared pgbackrest repo host;
then the tracker_vpc_peering module, once nothing else uses the
peering. Nothing in this repo does any more, but check first that
nothing outside it does (for example the Icinga agent on the DB host).
Delete the manual data-volume snapshots (2026-05-21, 2026-08-07
"pre-v2-frontend", 2026-09-07 "pre-import", 2026-10-22
"pre-backup-work", 2026-10-22 "pre-tsdb-upgrade") once DLM has 14+
days of history.
Done by hand 2026-11-09.
Prod TimescaleDB job 1007 move_old_data_to_archive has failed every
run (2,303, "cache lookup failed for function 0"); it targets the
long-gone location_history table. Delete the job and the leftover
functions (move_old_data_to_archive, migrate_nearest_city_data,
update_nearest_city, audit_tracker_status_change).
Migration c9e5a1f4b3d6 does both at the next deploy.
Drop the ad-hoc prod tables (bacardi, bayer, devices,
macs_to_remove, tracker_import_staging, trackers1000,
trackers_correct) by hand, after checking that nobody needs their
data. Deliberately not a migration. Done by hand 2026-11-09.
Staging: refresh_location_history() is owned by postgres, so the
bootstrap's CREATE OR REPLACE fails on every deploy (harmless). Run
ALTER FUNCTION refresh_location_history() OWNER TO tracker once.
Done 2026-11-09; the owner is now tracker.
Staging's TimescaleDB extension is 2.27.0 with the 2.28.2 package
installed; run ALTER EXTENSION timescaledb UPDATE (fresh psql -X
session) so staging matches prod's 2.28.2. Done 2026-11-09; staging
now runs 2.28.2.
Move prod onto infra/modules/database (like staging) so staging tests
what prod runs. The data volume must become a standalone
aws_ebs_volume via state moves; adopt the existing backup bucket
(names already match).
Decision (2026-10 VPC cost review, item 6): keep the tracker ALB separate
rather than folding it into backend-api-ingestion. Folding would make it
cross-VPC over the peering, lose tracker's own WAF ACL and split ownership,
all to save ~$25/mo.
Geofence Status Pipeline
Investigate why some trackers sit inside a delivery/storage geofence
but current_status never updates from IN_TRANSIT. Surfaced by the
find_status_zone_mismatches MCP tool (see
docs/development/mcp-diagnostic-server-design.md) - confirmed 4 real cases in dev data, all production run 108, all
~12-32m inside the "A&S Packing" storage zone. Likely candidate:
services/unified_geofence_service/taskiq_service.py's
_determine_status_with_hysteresis/_handle_no_status_change path.
Once staging/prod MCP access is live, run the same tool there first to
see how widespread this is before deciding on a fix.
Prod: ~95% of geofence event inserts failed with
fk_geofence_events_location_report_composite violations since
2026-08-07 (events fell from ~28k/day to ~300/day). Fixed
2026-10-22. Cause: TimescaleDB < 2.21.1 bug (timescaledb#8336) - FK
checks against a hypertable fail once Postgres switches the cached RI
plan to a generic plan, i.e. from a connection's 6th execution, so
pooled connections failed almost every insert. Prod ran 2.18.2; dev
(2.22) and staging (2.27) were unaffected. Confirmed on prod (inserts
1-5 OK, 6+ fail in one session); mitigated with
plan_cache_mode = force_custom_plan, then fixed by upgrading prod to
TimescaleDB 2.28.2 (PostgreSQL 16.15) and removing the mitigation.
Decide whether to regenerate the transit/dwell geofence events lost
between 2026-08-07 and 2026-10-22 from the stored location reports.
Geofence event failures are silent: _create_geofence_event_only()
logs and returns False, and batches report errors=0. Surface them as
batch errors so a regression like the one above is visible.
Fixed 2026-11-09: both write paths roll back and re-raise, so the batch records the error.
Dev databases built from setup scripts lack
fk_geofence_events_location_report_composite (added by hand on
2026-10-22), which is why dev never showed the FK failures. Make dev
setup run the full migrations so dev matches staging/prod.
Resolved by the migration flatten: dev setup runs alembic upgrade head, and the baseline includes the FK.
GeofenceEvent.timestamp is a naive DateTime in the model
(app/models/geofence_event.py:57) while the column is timestamptz.
Fixed 2026-11-09, along with created_at: both are now DateTime(timezone=True).
The unified-geofence worker sometimes logs a burst of
psycopg.errors.ConnectionTimeout exactly 10 seconds after a new task
starts (connect_timeout: 10 in services/shared/database.py), from
process_pending_location_reports. Seen on staging at the 2026-11-05
and 2026-11-09 16:00 deploys, but not at the 15:25 one. No further
errors follow it, but it is not yet confirmed that the worker processes
reports normally afterwards. Likely the first connection racing the task's network
setup (ENI attach or security group) or the database host during the
rollout. Check whether prod shows it too, then consider retrying the
first connection with backoff at worker start.
Test Suite
Fix pre-existing test failures surfaced while verifying the
tracker_status_service removal (2026-07-23). Fixed 2026-11-09:
test_unified_geofence_processing.py and
test_unified_geofence_service.py tested UnifiedGeofenceService,
deleted on 2025-11-27. Removed. Their status-determination cases
(delivery, storage, delivery-over-storage priority, in transit) had no
other coverage, so they were ported to TestStatusDetermination in
test_unified_geofence_taskiq_service.py.
test_api_router_inclusion: since FastAPI 0.139, app.routes holds
included-router entries without .path. It now reads the paths from
the OpenAPI schema.
test_settings_cors_origins_default: the expected list lacked the
Svelte dev ports and the staging hostnames.
TestDeliveryLocationCoordinates: DeliveryLocation.get_coordinates
calls query().filter().first(), but the mock only stubbed
query().first().
test_location_report_processing_workflow failed depending on row
order: a tracker leaked into the dev database by an earlier run shared
its fixed hashed key. It now uses a unique key per run.
Tests run against the dev database (POSTGRES_DB, default tracker),
isolated only by a rolled-back transaction. Committed test data leaks
into dev, and dev data can collide with tests. Give the tests their own
database, built from the Alembic baseline.
trackers.hashed_advertisement_key is not unique, and
_get_tracker_by_hashed_key takes .first() with no ordering. If a
key is shared (for example by a reused tracker), reports can be
attributed to an arbitrary tracker. Count duplicate keys in prod, then
decide between a unique constraint and a deterministic choice.
Done 2026-11-09: prod, staging and local had no duplicates, so migration
d2f6b8a4c1e7 adds uq_trackers_hashed_advertisement_key, and
POST /trackers returns 409 for an existing key. The database now holds
one tracker per key even when two physical devices share it. The
column previously had no index, so per-report lookups and the
fetcher's joins also stop scanning trackers.
Documentation
We need to look at replacing MkDocs if we want to keep Material.
See docs/development/docs-platform-evaluation.md
for the current analysis of MkDocs 2.0, Zensical, plugin compatibility,
and Cloudflare hosting impact.
Usage Guide
The user guide needs to be updated to cover the new design and move to the unified crud in Campaign Management
Added a "Managing Clients, Brands, and Campaigns" section covering the
drill-down interaction, hover-to-reveal edit/delete, mobile behavior,
child runs, and per-run tracker management. Cross-linked from
"Setup — Required Order".
Remove Stale Documents
We need to do a review of documentation and remove cruft relating to the previous React version and any other outdated materials.
Rewrote the Admin Panel (8 files) and Front End (7 files) sections
wholesale -- both described the deleted React/react-admin apps as
current, with dead paths and code samples. Fixed scattered stale
mentions in docs/index.md, reference/system-architecture.md, and
several guides/services docs. Left legitimate historical/migration
docs (the Svelte migration assessments, the cutover TODO) untouched --
they're supposed to talk about the old React app for comparison.
The documentation requires a restructure as the top level menu contains too many items and it looks disorganised.
Root cause: mkdocs-awesome-pages-plugin defaults to reading
.pages files, but every nav file in this repo is named
.nav.yml and the plugin was never told to look for that instead --
so none of the curated navigation was ever actually being read, and
mkdocs was silently falling back to a flat alphabetical dump of all
~90 pages. Fixed the plugin config (one line), fixed the "*"
wildcard token (should be "..." for this plugin version), fixed a
broken nested-path reference and a dead redoc link, and moved/removed
the handful of stray top-level directories that had no real content
(an empty dir, a confusingly-duplicate database/, asset-only dirs).
Verified by building the full site and confirming the rendered nav
matches the curated structure.
General Behaviour
When a tracker changes status to delivered consider not tracking any further activity.
We could remove it from the queue so it no longer asks Apple for FindMy reports
or do we keep tracking but never change status from delivered?