HTTPS Behind a Load Balancer

Overview

This guide explains how to ensure proper HTTPS handling when running the application behind a load balancer (such as AWS ELB) that terminates SSL.

Problem

When using a load balancer that terminates SSL, the load balancer handles the HTTPS connection and then forwards the request to the container as HTTP. This can cause issues where:

1. The application doesn't know that the original request was HTTPS
2. When generating URLs or redirects, the application uses HTTP instead of HTTPS
3. API requests may be redirected from HTTPS to HTTP, causing security warnings in browsers
4. Mixed content warnings may appear when a page loaded over HTTPS tries to load resources over HTTP

The specific symptoms observed were:

* Connection to host tracker.glimpse.technology left intact
* Clear auth, redirects to port from 443 to 80
* Issue another request to this URL: 'http://tracker.glimpse.technology/api/v1/trackers/?limit=2000&skip=0&include_latest_location=true&sort_field=last_report_received&sort_order=desc'
* Host tracker.glimpse.technology:80 was resolved.

And:

Mixed Content: The page at 'https://tracker-admin.glimpse.technology/clients' was loaded over HTTPS, but requested an insecure XMLHttpRequest endpoint 'http://tracker-admin.glimpse.technology/api/v1/clients/?skip=0&limit=10&sort_field=name&sort_order=asc'. This request has been blocked; the content must be served over HTTPS.

Solution

The solution is to ensure that:

1. The X-Forwarded-Proto header is properly set to "https" for all API requests
2. The Content-Security-Policy header is configured to allow connections only to HTTPS resources
3. The nginx proxy is configured to use the correct API URL (which may be different in development and production)
4. The Strict-Transport-Security header is set to enforce HTTPS
5. IPv6 connectivity issues are addressed
6. 307 redirects are properly handled

Changes Made

1. Modified the Nginx configuration to use environment variables for the API URL:

# Security headers
add_header X-Frame-Options "SAMEORIGIN";
add_header X-XSS-Protection "1; mode=block";
add_header X-Content-Type-Options "nosniff";
add_header Referrer-Policy "strict-origin-when-cross-origin";
add_header Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; img-src 'self' data: https:; font-src 'self' data:; connect-src 'self' https://$host https://*.glimpse.technology https://api.example.com;";
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

# Global resolver for DNS resolution
resolver 1.1.1.1 ipv6=off;
resolver_timeout 5s;

# Special case for auth login endpoint
location = /api/v1/auth/login/json {
    proxy_pass __API_URL__/api/v1/auth/login/json;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host tracker.glimpse.technology;
    proxy_cache_bypass $http_upgrade;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto "https";
    proxy_ssl_server_name on;
    proxy_ssl_protocols TLSv1.2 TLSv1.3;
}

# Special case for users/me endpoint
location = /api/v1/users/me {
    proxy_pass __API_URL__/api/v1/users/me/;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host tracker.glimpse.technology;
    proxy_cache_bypass $http_upgrade;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto "https";
    proxy_ssl_server_name on;
    proxy_ssl_protocols TLSv1.2 TLSv1.3;
}

# Handle API requests
location /api/ {
    # Proxy to the API server (set via environment variable)
    proxy_pass __API_URL__/api/;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';

    # Set the Host header based on the API URL
    proxy_set_header Host tracker.glimpse.technology;

    proxy_cache_bypass $http_upgrade;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto "https";

    # SSL configuration for proxy (if using HTTPS)
    proxy_ssl_server_name on;
    proxy_ssl_protocols TLSv1.2 TLSv1.3;

    # Follow redirects for trailing slashes
    proxy_redirect off;
    proxy_intercept_errors on;
    error_page 301 302 307 = @handle_redirect;
}

# Handle redirects
location @handle_redirect {
    set $saved_redirect_location $upstream_http_location;
    proxy_pass $saved_redirect_location;
}

1. Updated the Dockerfile to use environment variables for the API URL:

# Create a script to replace environment variables in nginx config
RUN echo '#!/bin/sh' > /docker-entrypoint.sh && \
  echo 'API_URL=${API_URL:-http://api:8000}' >> /docker-entrypoint.sh && \
  echo 'echo "Using API URL: $API_URL"' >> /docker-entrypoint.sh && \
  echo 'sed -i "s|__API_URL__|$API_URL|g" /etc/nginx/conf.d/default.conf' >> /docker-entrypoint.sh && \
  echo 'nginx -g "daemon off;"' >> /docker-entrypoint.sh && \
  chmod +x /docker-entrypoint.sh

# Set the entrypoint to our script
ENTRYPOINT ["/docker-entrypoint.sh"]

# Override the CMD since we're running nginx directly in the entrypoint
CMD [""]

1. Updated the docker-compose files to set the API URL:

admin:
  build:
    context: ./tracker-admin
    dockerfile: Dockerfile
  ports:
    - 8080:80
  environment:
    - API_URL=${API_URL:-https://tracker.glimpse.technology} # Default to production URL, can be overridden
  depends_on:
    - api
  networks:
    - tracker-network
  restart: unless-stopped

For development environments, we use a different default:

admin-dev:
  build:
    context: ./tracker-admin
    dockerfile: build/dev/Dockerfile
  ports:
    - 3000:3000
  volumes:
    - ./tracker-admin:/app
  env_file:
    - ./tracker-admin/.env
  environment:
    - API_URL=${API_URL:-http://dev:8000} # Default to development URL, can be overridden
  depends_on:
    - dev
  networks:
    - tracker-network
  restart: unless-stopped
  user: root

Why This Works

The updated configuration ensures that:

1. The API knows that the original request was HTTPS (via the X-Forwarded-Proto header)
2. When generating URLs or redirects, the API will use HTTPS instead of HTTP
3. API requests will maintain their HTTPS protocol, preventing security warnings in browsers
4. The Content-Security-Policy header prevents loading resources over HTTP
5. The Strict-Transport-Security header tells browsers to always use HTTPS for future requests
6. The proxy_pass directive uses HTTPS to connect to the API server
7. The Host header is set to the correct API domain

Additional Considerations

• This solution assumes that all production traffic should be using HTTPS
• The load balancer should be configured to forward the original protocol in the X-Forwarded-Proto header
• For local development, this change should not affect functionality as the API will still work with HTTP
• The Content-Security-Policy header includes https://$host which allows connections to the same host over HTTPS
• The Content-Security-Policy header includes https://*.glimpse.technology which allows connections to all subdomains over HTTPS

Testing

To verify that this solution works:

1. Deploy the updated containers to production
2. Access the application via HTTPS
3. Monitor network requests in the browser's developer tools to ensure all API requests use HTTPS
4. Verify that no redirects from HTTPS to HTTP occur
5. Check for any mixed content warnings in the browser's console

Troubleshooting

If you encounter mixed content warnings:

1. Check the browser's console to see which resources are being loaded over HTTP
2. Verify that the Content-Security-Policy header is correctly configured
3. Ensure that the nginx proxy is correctly configured to use HTTPS when connecting to the API server
4. Clear the browser's cache to ensure it's not using cached HTTP resources
5. Check for IPv6 connectivity issues in the nginx logs
6. Ensure that the nginx proxy is correctly handling 307 redirects

IPv6 Connectivity Issues

If you see errors like the following in your nginx logs:

connect() to [2606:4700:3030::6815:4907]:443 failed (101: Network unreachable) while connecting to upstream

This indicates that nginx is trying to connect to the API server using IPv6, but the connection is failing. To fix this, add a global resolver directive at the server level in your nginx configuration to disable IPv6:

# Global resolver for DNS resolution
resolver 1.1.1.1 ipv6=off;
resolver_timeout 5s;

Using Cloudflare's public DNS server (1.1.1.1) with IPv6 explicitly disabled can provide more reliable DNS resolution than the default Docker DNS (127.0.0.11). The shorter timeout (5s) helps prevent hanging connections.

It's important to place this directive at the server level, not within a location block, to ensure it applies globally to all DNS resolutions performed by nginx.

Handling 307 Redirects

If you see 307 Temporary Redirect responses in your logs, it means that the API server is redirecting the request to a different URL. This often happens when the URL doesn't have the correct trailing slash. There are two approaches to handling this:

1. Special Case Handling for Problematic Endpoints

The first approach is to handle specific endpoints that are known to cause issues:

# Special case for auth login endpoint
location = /api/v1/auth/login/json {
    proxy_pass __API_URL__/api/v1/auth/login/json;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host tracker.glimpse.technology;
    proxy_cache_bypass $http_upgrade;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto "https";
    proxy_ssl_server_name on;
    proxy_ssl_protocols TLSv1.2 TLSv1.3;
}

# Special case for users/me endpoint
location = /api/v1/users/me {
    proxy_pass __API_URL__/api/v1/users/me/;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host tracker.glimpse.technology;
    proxy_cache_bypass $http_upgrade;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto "https";
    proxy_ssl_server_name on;
    proxy_ssl_protocols TLSv1.2 TLSv1.3;
}

This approach is more efficient as it prevents the redirect from happening in the first place. Note that some endpoints like /api/v1/users/me need special handling with exact matching using the = operator.

2. Follow Redirects Automatically

The second approach is to intercept and follow redirects automatically:

# Follow redirects for trailing slashes
proxy_redirect off;
proxy_intercept_errors on;
error_page 301 302 307 = @handle_redirect;

# Handle redirects
location @handle_redirect {
    set $saved_redirect_location $upstream_http_location;
    proxy_pass $saved_redirect_location;
}

This will intercept the redirect response and follow it automatically, without requiring the client to make a second request.

Using both approaches together provides the most robust solution, as it handles both known endpoint patterns and any unexpected redirects.

References

• Nginx Proxy Module Documentation
• AWS ELB and X-Forwarded Headers
• Content Security Policy
• HTTP Strict Transport Security