Run behind a reverse proxy

Use this guide when LinkMesh is exposed via a public hostname with a Let’s Encrypt certificate and collectors (plus the optional agent) connect through that same hostname over the internet. If everything can reach the LinkMesh VM directly (VPC, VPN, bastion, or flat network), the direct VM install is simpler — skip this page.

LinkMesh serves everything on one HTTP port (:8080), and you terminate TLS in front of it:

• The web UI and REST API.
• The agent control channel — a WebSocket at /v1/agent that the optional linkmesh-agent connects to (Bearer-token auth; no client certificate).
• OpAMP at /v1/opamp (otelcol-contrib collectors) and the remotecfg endpoint that Grafana Alloy collectors poll.
• OTLP ingest at /v1/metrics.

There is no separate gRPC port — the agent channel is a WebSocket on this same vhost.

The one snag operators hit: /v1/agent and /v1/opamp upgrade to WebSocket, so the nginx server block must forward the Upgrade/Connection headers and must not negotiate HTTP/2 on that listener (HTTP/2 has no Connection: Upgrade semantics, so an http2 listener returns 400 Bad Request for every handshake). The config below handles it.

Tell LinkMesh its public URL

sudo nano /etc/linkmesh/config.yaml
sudo systemctl restart linkmesh-server

externalUrl: "https://linkmesh.example.com"
http:
  trustedProxies: ["127.0.0.1/32"]

• externalUrl is the server’s public base URL — set it to the proxy’s hostname, not the backend’s. Behind a proxy it is the value the whole install hands out: collector self-telemetry endpoints, enrollment scripts, invite and password-reset links. What it is and what breaks without it is documented once, in the configuration reference; the walkthrough is Quickstart step 2.

• http.trustedProxies lists the addresses your proxy connects to LinkMesh from. Only those peers may set X-Forwarded-* headers. For nginx on the same host that is 127.0.0.1/32. If the proxy is not listed, the audit log and the login throttle record the proxy’s IP for every request instead of the user’s. Keep the list as narrow as you can, because any listed peer can claim to be any client. Get this right and set externalUrl explicitly: X-Forwarded-Host from an untrusted source is discarded, so a proxy missing from this list combined with an unset externalUrl leaves the server unable to work out its own address at all.

  The client IP is the X-Forwarded-For entry appended by the last listed proxy. A value the client sent itself is never believed, and neither is True-Client-IP. If requests pass through several proxies (a CDN or load balancer in front of nginx), list every one of them. Otherwise the outermost unlisted proxy is recorded instead of the user.

  On Kubernetes, list the address range your ingress controller’s pods run in (with the Helm chart: --set 'http.trustedProxies={10.48.0.0/16}'). The controller has to see the real client too. On a cloud LoadBalancer that usually means externalTrafficPolicy: Local on the controller’s Service.

Install nginx + issue a Let’s Encrypt certificate

1. Install nginx and certbot:

   sudo apt-get update
   sudo apt-get install -y nginx certbot

2. Point DNS at the proxy and open the firewall. Add an A/AAAA record for linkmesh.example.com → the proxy’s public IP, and allow inbound 80 and 443. Port 80 is only needed for the ACME challenge and the HTTP→HTTPS redirect.

3. Issue the certificate with certbot’s standalone server (it binds port 80, so stop nginx for the ~10 seconds it takes):

   sudo systemctl stop nginx
   sudo certbot certonly --standalone -d linkmesh.example.com \
     --agree-tos -m [email protected] --non-interactive \
     --deploy-hook "systemctl reload nginx"
   sudo systemctl start nginx

   This writes fullchain.pem + privkey.pem to /etc/letsencrypt/live/linkmesh.example.com/ — the paths the config below references. Certbot’s timer auto-renews and the --deploy-hook reloads nginx.

The TLS certificate must cover the exact hostname agents and collectors connect to — it’s the only certificate in play (there is no separate mTLS/gRPC cert).

nginx config

nginx terminates TLS at :443 and proxies everything to LinkMesh on :8080. The WebSocket control channels (/v1/agent, /v1/opamp) get the upgrade headers and long idle timeouts so the persistent connections aren’t reaped between heartbeats.

# /etc/nginx/sites-enabled/linkmesh.conf

# ── WebSocket upgrade map (http {} top level) ────────────────
map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    # No `http2` — WebSocket Upgrade is HTTP/1.1 only; HTTP/2 returns 400 for
    # every agent / OpAMP handshake.
    listen 443 ssl;
    server_name linkmesh.example.com;

    ssl_certificate     /etc/letsencrypt/live/linkmesh.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/linkmesh.example.com/privkey.pem;

    # Agent + OpAMP WebSocket control channels — preserve Upgrade headers,
    # long idle timeouts for the persistent connections.
    location ~ ^/v1/(agent|opamp) {
        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade    $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host       $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_read_timeout 3600s;
        proxy_send_timeout 3600s;
    }

    # Everything else — REST API, web UI, remotecfg, OTLP ingest.
    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host              $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host  $host;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_http_version 1.1;
        proxy_set_header Connection        "";
    }
}

# ── HTTP → HTTPS redirect ────────────────────────────────────
server {
    listen 80;
    server_name linkmesh.example.com;
    return 301 https://$host$request_uri;
}

A WAF may block binary OTLP

If a WAF / ModSecurity sits in front of nginx, it can block binary protobuf POSTs to /v1/metrics (OTLP) with a 403 before they reach LinkMesh. Exempt the OTLP path if you push metrics through the proxy.

Then validate and reload:

1. Validate, then reload — nginx -t catches a bad cert path or syntax error before you take the proxy down:

   sudo nginx -t && sudo systemctl reload nginx

2. Confirm the surfaces answer from a host that can reach the proxy:

   # Web UI / REST → 200/302
   curl -sI https://linkmesh.example.com/ | head -1
   
   # Agent WebSocket — must return 401 (auth required), NOT 400/301.
   # 400 means nginx is negotiating HTTP/2 or stripping Upgrade headers;
   # 301 means the request hit :80 instead of :443.
   curl -skI -o /dev/null -w '%{http_code}\n' \
     -H 'Upgrade: websocket' -H 'Connection: Upgrade' \
     -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' \
     -H 'Sec-WebSocket-Version: 13' \
     https://linkmesh.example.com/v1/agent

Use https:// / wss://, not http:// / ws://

When you point an agent or an OpAMP-managed collector at this proxy, the URL must use TLS (note the two ss in https/wss). Plain ws:///http:// hits nginx on :80, gets a 301 to HTTPS, and the client gives up with a bad handshake because it doesn’t follow redirects. The install page emits the correct URL automatically; if you hand-edit a config, double-check the scheme.

Related

• Firewall & connectivity — every port and direction involved, and how to verify each one
• How the edge talks to the server — what the proxy is carrying, and why the WebSocket upgrade matters
• Configuration — the server keys this guide sets, including the public URL
• Install on Kubernetes — ingress handles the same job there