This post explains the xDS APIs, how Aggregated Discovery Service (ADS) orders updates, the difference between state-of-the-world and incremental (delta) xDS, how ACK and NACK work, and what a snapshot cache does. At the end we show where Elchi fits and which of these choices it makes.

Static bootstrap vs dynamic configuration​

Every Envoy starts from a bootstrap file. A bootstrap can hold the whole configuration under static_resources, which works until you need to change something: then you edit the file and restart or hot restart the process. The alternative is dynamic_resources, where the bootstrap only says where the management server is and which resource types to fetch from it. From then on, the management server owns the configuration and Envoy keeps a stream open to receive changes.

That management server is the control plane. It is not part of the request path; if it goes away, Envoy keeps serving with the last configuration it accepted.

The xDS APIs​

Each xDS API carries one resource type. The ones every Envoy operator meets are:

Two more appear in larger setups. ECDS (Extension Config Discovery Service) delivers the configuration of a single filter as a TypedExtensionConfig, so a filter inside a listener can be updated without resending the whole listener. VHDS (Virtual Host Discovery Service) delivers virtual hosts on demand instead of as part of a full route configuration.

The resources reference each other by name. A listener's HTTP connection manager names a route configuration, a route names a cluster, an EDS cluster names its endpoint assignment, and a TLS context names a secret. That reference graph is why the order of updates matters.

The authoritative description of the protocol is the xDS REST and gRPC protocol page in the Envoy documentation.

Separate streams vs ADS​

Envoy can fetch each resource type on its own gRPC stream, possibly from different management servers. That is simple but gives no ordering guarantee between types. If a new route arrives before the cluster it points at, requests that hit that route fail until the cluster shows up.

ADS (Aggregated Discovery Service) puts all resource types on a single gRPC stream to a single management server. Because one server sequences everything on one stream, it can apply the ordering the Envoy documentation recommends to avoid dropping traffic during a change:

1. CDS updates first.
2. EDS updates for those clusters after CDS.
3. LDS updates after the clusters and endpoints they need.
4. RDS updates for new listeners after LDS.
5. Only then remove clusters and endpoints that nothing references any more.

This is "make before break": a resource is only referenced once what it depends on is in place, and nothing is removed while it is still referenced.

State of the world vs incremental (delta) xDS​

xDS exists in two flavours, and each works with separate streams or ADS, which gives four variants.

State of the world (SotW) uses DiscoveryRequest and DiscoveryResponse. For listeners and clusters, every response must carry the complete set of resources the client needs, even the ones that did not change. If a listener is missing from an LDS response, Envoy removes it. With hundreds of listeners or thousands of clusters, one small change means resending all of them.

Incremental, or delta, xDS uses DeltaDiscoveryRequest and DeltaDiscoveryResponse. The client subscribes and unsubscribes to resource names (resource_names_subscribe, resource_names_unsubscribe), the server sends only resources that were added or changed, and it lists removed ones explicitly in removed_resources. Each resource carries its own version. On reconnect, the client reports the versions it already has in initial_resource_versions, so the server does not have to resend unchanged resources.

The choice is made in the bootstrap: api_type: GRPC for SotW and api_type: DELTA_GRPC for delta, both under ads_config when ADS is used.

ACK and NACK​

xDS is not fire-and-forget. Every response from the server carries a nonce, and Envoy answers each one with a new request on the same stream:

• ACK: Envoy applied the update. In SotW, the request carries the version_info of the accepted response and its response_nonce.
• NACK: Envoy rejected the update, for example because a listener failed validation. The request carries the response_nonce of the rejected response, an error_detail with the reason, and, in SotW, the version_info of the last version it did accept. Envoy keeps running the previous configuration.

Delta requests carry no version_info at all; ACK and NACK are told apart by whether error_detail is set, and the nonce says which response it answers.

For an operator this is the most useful signal a control plane has. A NACK is the only place where "the proxy refused this configuration, and here is why" shows up. A control plane that does not record NACKs leaves you comparing what you pushed with what /config_dump shows.

Snapshot caches and go-control-plane​

Most control planes are not written from scratch. envoyproxy/go-control-plane is the reference Go implementation: it ships the generated Go types for the Envoy API, a gRPC server for every xDS variant, and caches that the server reads from.

The most common cache is the snapshot cache. The control plane builds a snapshot, a consistent set of listeners, routes, clusters, endpoints, secrets and other resources for one node, and stores it under a key derived from the Envoy node (the NodeHash). The server streams that snapshot to every Envoy that maps to the key, answers ACKs and NACKs, and sends a new update whenever the control plane sets a new snapshot. The cache is created with an ads flag; when it is set, the cache waits until a request has named all the resources it wants before it responds, which suits ADS clients.

That leaves the hard part to you: turning your own data model into correct Envoy resources, deciding which node gets which snapshot, and keeping references between resources consistent.

Where Elchi fits​

Elchi is a control plane for Envoy fleets with a UI and an API in front of it. Here is how it answers each of the choices above, with the code behind it in the elchi-backend and elchi-client repositories.

ADS over delta xDS. The bootstrap Elchi generates for a deployed listener fetches LDS and CDS through ADS and sets api_type: DELTA_GRPC with transport API v3. The other resource types follow from there: route configurations, endpoints, secrets and HTTP filter configurations are referenced from listeners and clusters and fetched over the same aggregated stream. HTTP filters inside the HTTP connection manager are stored as ECDS config_discovery references, not inline configuration, so a filter can change without rewriting the listener. The control plane also serves VHDS.

A snapshot cache keyed by node ID. The control-plane process builds a go-control-plane snapshot cache with ADS enabled and keys it by the Envoy node ID. Every deployed listener has its own node ID in the form listener::project::ip, so each listener gets its own snapshot. The xDS server listens on port 18000 behind the platform's front door. See Architecture for the full wiring.

NACKs are recorded, not just logged. When a delta request carries error_detail, the control plane stores the error per proxy together with the revision it rejected, and the UI shows it on the service. An ACK records which revision the proxy now runs. One detail we document because it surprises people: the delta server does not resend a resource that was NACKed until its content changes, so the fix is to correct the resource and publish again.

One control plane per Envoy version. Envoy's API changes between releases: fields are added, deprecated and removed. A control plane compiled against one release's protobuf definitions can produce configuration that an older proxy does not understand. Elchi builds against versioned-go-control-plane, our fork of go-control-plane with one module version per Envoy release, and the release pipeline produces one control-plane binary per supported Envoy version. Each Envoy sends its node ID and an envoy-version header on its ADS stream. The registry, which runs as an ext_proc filter on the platform's front door, reads both and routes the stream to the control plane built for that version. This is what lets several Envoy versions run side by side in one project; Run several Envoy versions side by side explains the setup and the listener upgrade.

What you edit. Each resource form in the UI is generated from the protobuf schema of the Envoy version it targets, and Elchi assembles the references (ECDS pointers, SDS secrets, EDS assignments) for you. You never write the snapshot by hand. You can still check what a node is served in the snapshot dump.

Next steps​

• Read the Architecture page for every process and connection in an Elchi installation.
• Follow Manage Envoy without YAML to build a listener, route and cluster from the UI.
• See Versions & Upgrades and Run several Envoy versions side by side for version routing.
• Learn more about Elchi Core, or book a demo.

This post walks through the Envoy side of the wiring, the design choices that make ext_proc workable for a WAF, and the rollout sequence that keeps the CRS from blocking legitimate traffic on day one. The examples use Shield, Elchi's ext_proc sidecar, but the Envoy configuration works for any ext_proc service.

How ext_proc works​

envoy.filters.http.ext_proc opens a bidirectional gRPC stream per HTTP transaction to an external service. Envoy sends messages for request headers, request body, response headers and response body, as selected by the filter's processing_mode. The service answers each message: continue, mutate, or return an immediate response (a 403, for a WAF).

Three properties matter for a WAF:

• Body handling is configurable per phase, and when the filter sets allow_mode_override, the external service can change it per request with mode_override. You don't have to buffer every body because some routes need inspection.
• Failure behavior is explicit. failure_mode_allow decides what happens when the service cannot be reached.
• The transport can be a Unix domain socket, which keeps the hop on the host.

The Envoy side​

The external service is a gRPC server, so its cluster must use HTTP/2. A Unix domain socket is the preferred transport:

clusters:
  - name: elchi_shield
    type: STATIC
    connect_timeout: 0.25s
    typed_extension_protocol_options:
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        explicit_http_config:
          http2_protocol_options: {}
    load_assignment:
      cluster_name: elchi_shield
      endpoints:
        - lb_endpoints:
            - endpoint:
                address:
                  pipe:
                    path: /run/elchi-shield/extproc.sock

The filter goes in the HTTP filter chain before the router:

http_filters:
  - name: envoy.filters.http.ext_proc
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor
      failure_mode_allow: true
      allow_mode_override: true
      request_attributes: ["xds.node.id"]
      grpc_service:
        envoy_grpc: { cluster_name: elchi_shield }
        timeout: 0.2s
      processing_mode:
        request_header_mode: SEND
        response_header_mode: SKIP
        request_body_mode: NONE
        response_body_mode: NONE
  - name: envoy.filters.http.router
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

Three lines deserve attention.

request_body_mode: NONE looks wrong for a WAF that needs bodies. It is deliberate. Envoy sends headers only by default, and Shield upgrades the body mode to BUFFERED through mode_override only for the routes whose policy inspects bodies. Routes that need no body inspection never pay for buffering.

allow_mode_override: true is what lets Envoy accept that per-request upgrade. Without it, Envoy ignores mode_override and the processor never sees a body.

failure_mode_allow: true is the availability backstop outside the WAF. If the sidecar process is down or the gRPC call fails, traffic continues uninspected instead of failing. Set it to false only if you accept an outage when the sidecar is down. This is a separate decision from how the WAF treats its own internal errors, covered below.

One more listener setting is not optional: the WAF needs a client IP it can trust.

# HttpConnectionManager
use_remote_address: true
xff_num_trusted_hops: 0

Shield reads the client IP from the right-hand end of X-Forwarded-For. Without use_remote_address, that entry is whatever the client sent, and every IP-based rule (reputation lists, GeoIP, per-IP rate limits) acts on an attacker-controlled value. Full reference: Wiring Shield into Envoy.

Running the CRS inside the processor​

Shield embeds Coraza (the OWASP Go WAF library, which runs SecLang rules) with the CRS compiled into the binary. There are no rule files to ship to the edge. A policy turns it on per host and route:

apiVersion: sentinel.elchi.io/v1
kind: SecurityPolicy
metadata:
  name: api-coraza
spec:
  defaults:
    mode: block
    fail_mode: fail_open
    inspect_request_body: true
    max_request_body_bytes: 1048576
  domains:
    - hosts: ["app.example.com"]
      routes:
        - match:
            path_prefix: "/api/"
          policy:
            mode: detect
            engines:
              coraza:
                include_owasp: true

The CRS uses anomaly scoring: each matching rule adds to a per-transaction score, and the blocking rule fires when the score crosses a threshold. Shield exposes the CRS's usual controls as policy fields: paranoia_level (1 to 4, default 1), detection_paranoia_level (run stricter rules in detection only), inbound_anomaly_threshold (default 5) and outbound_anomaly_threshold (default 4), plus exclude_rule_ids for tuning out noisy rules by ID.

The directive bundle is assembled in a fixed order: the recommended Coraza config, SecRuleEngine On, the CRS setup, your tuning, the CRS rules, your own SecLang directives, and finally the rule exclusions. The WAF is compiled once and swapped in atomically, so a config reload never races a request that is being inspected.

One trap catches almost everyone who puts the CRS in front of an API. On its own, the CRS allows only GET HEAD POST OPTIONS. Rule 911100 is critical (+5), which is enough to block at the default threshold, so every PUT, PATCH and DELETE would get a 403. Shield always writes an explicit method list into the CRS settings for that reason, defaulting to GET HEAD POST OPTIONS PUT PATCH DELETE. If you wire the CRS up yourself, set tx.allowed_methods before your first deploy.

Engine reference: Coraza WAF (OWASP CRS).

Header phase vs body phase​

The CRS is a body-phase engine. Shield runs each request as an ordered pipeline: policy resolution, cheap pre-checks (method, host, headers, content type, source IP) and header-phase engines such as JWT or rate limiting, and only then, if the policy needs it, the body gate and the body-phase engines. A policy whose engines don't need the body never buffers it just to run the WAF. Response inspection is a separate pipeline that reuses the request's policy, and it is where the CRS's outbound rules run. See How Shield Works.

Rolling out without breaking traffic​

The CRS will produce false positives on real APIs, so plan the rollout around that. Shield policies have four modes:

detect and shadow never modify traffic, and they record the full trail of findings rather than stopping at the first, which is what you need for tuning. The recommended sequence is detect, then shadow, then block, per route if needed: establish a baseline, exclude the rule IDs that fire on legitimate traffic, rehearse enforcement, then enforce.

Keep the two failure settings apart:

• fail_mode: fail_open | fail_close (policy) governs engine errors and timeouts inside the WAF. It never lets a confirmed finding through.
• failure_mode_allow (Envoy) governs the sidecar being unreachable.

Details: Modes & Fail Postures.

Getting policies to every edge​

A WAF on one host is a demo. On a fleet, the question is how policy changes reach every sidecar safely. Shield's answer is that its configuration is files in a watched directory. It hot-reloads on change, without a restart, and an invalid config never replaces the last good one.

In Elchi, the controller merges a project's policies into one bundle and the edge agent applies it in phases: it stages each file, validates the staged set with the local elchi-shield validate binary (the same version that will run it), and only then renames the files into place. The agent then confirms that Shield's active config version changed. If Shield rejected the config, the deploy reports Shield's file-and-field error instead of a false success. See Deploying Policies to Edges.

The client installer puts Shield on each edge by default, next to Envoy: The Bundled Shield Sidecar.

Verifying it​

With the policy in detect, a SQL-injection probe still gets the upstream's response and shows up in Security Events as a would-block, with the CRS rule that fired. Move the route to block, repeat the request, and it returns 403 Forbidden, while a clean request still gets 200. The end-to-end walkthrough, including discovering the API first and drafting the policy from its observed traffic, is in Secure an API with Shield.

Takeaways​

• ext_proc gives you a WAF without a custom Envoy build. Use a Unix domain socket, send headers only, and let the processor upgrade the body mode per route (allow_mode_override).
• Make the client IP trustworthy (use_remote_address) before any IP-based rule means anything.
• Allow your API's methods in the CRS before the first deploy.
• Roll out detect, shadow, block, and keep sidecar-down behavior separate from engine-error behavior.
• Treat WAF config as files that are validated before they go live, on every edge.

Next steps​

• Follow the how-to: Add a WAF (OWASP Core Rule Set) to Envoy with ext_proc.
• Read the Shield docs: Shield overview.
• Product overview: Elchi API Security.
• Want to see it against your own APIs? Book a demo.

This post covers what that control plane has to do, why hand-written xDS breaks down, and how we approached each part in Elchi. The problems apply to any Envoy deployment, whichever control plane you end up with.

Envoy's configuration is an API, not a file​

Every Envoy object (listener, route configuration, cluster, endpoint assignment, secret) is a protobuf message. The xDS protocols (LDS, RDS, CDS, EDS and SDS, aggregated over ADS) stream those messages to a proxy, and Envoy applies them without a restart. The static bootstrap only has to contain enough for Envoy to find its control plane.

The resources form a reference graph:

Listener ─▶ filter chain (HTTP connection manager) ─▶ route config ─▶ virtual host ─▶ route ─▶ cluster ─▶ endpoints
   └─ transport socket ─▶ secret (TLS)                                              cluster ─▶ transport socket ─▶ secret

A typo in a cluster name inside a route does not fail when the config loads. The route points at nothing, and requests get a 503 at runtime. Multiply that by every reference in a non-trivial config and the main cost of hand-written xDS becomes clear: the references are the hard part, not the fields.

What a control plane has to do​

If you build your own on top of a library such as go-control-plane, you take on a list of responsibilities. These are the ones that tend to cause trouble:

1. Model resources and their references. Know that deleting a cluster breaks three routes before it happens.
2. Validate before pushing. Protobuf type checking is not enough. Envoy's API carries validation rules (required fields, ranges, enum constraints) that should be enforced before anything leaves the control plane.
3. Stage changes and ship them together. Operators want to change several resources and ship them as one unit, not one push per field edit.
4. Build and serve snapshots per node. Each proxy needs a consistent set of resources, keyed by its node identity.
5. Handle more than one Envoy version. The API changes between releases. During an upgrade window part of the fleet runs the old version and part runs the new one.
6. Show what actually shipped. "I published it" and "the proxy is running it" are different statements.

None of this is exotic, but together it is a product. Elchi is our attempt to build that product so platform teams don't have to.

Typed resources instead of YAML​

In Elchi, everything pushed to Envoy is modeled as a resource: listeners, clusters, routes, endpoints, secrets and the bootstrap. HTTP, network, listener and UDP filters and reusable extension configurations are managed alongside them, versioned per Envoy release.

Every resource is created for one Envoy version, and its editor form is generated from that version's protobuf definitions. That has two practical effects. When a new Envoy release adds a field, the field appears in the editor for that version without hand-built UI work. And the form offers exactly the fields that version supports, so you don't write @type URLs or nest messages by hand.

Details: Resources & the Config Model.

Two-step validation​

Every change is checked twice:

1. In the editor: types generated from the Envoy API catch shape and type errors as you edit.
2. On the server: the controller runs the Envoy API's validation rules before the resource is persisted.

The second step is the one that protects the fleet. A resource that fails it is not stored and never reaches a snapshot, so a proxy is not the first place the error shows up.

Validation has a limit worth stating. A value can be valid against the schema and still be refused by a filter at runtime. The verification step further down is how you catch that case.

Draft, save, publish​

Edits are kept as drafts, so you can stage changes across several resources (say, a new route plus the cluster it targets) and publish them together. On publish, the controller validates the changes, persists them, and pushes a new snapshot to the control plane, which streams it to Envoy over ADS. Envoy applies it in place.

Config changes are also recorded in an audit trail, so "who changed this route, and when" has an answer. See Audit & Syslog Forwarding.

Seeing the reference graph​

The reference problem from the start of this post gets two views:

• A dependency graph for any resource shows what it links to (clusters, routes, secrets, filters). It is the quickest way to spot an orphaned or broken reference before you publish. See Dependency Graph.
• A route map shows how requests flow through a listener's routes to upstreams. See Route Map.

Verifying what Envoy received​

Authoring tells you what you intended. The Snapshot Dump reads back the xDS snapshot the control plane is serving a specific node, with one tab per resource type and the full JSON. If a resource is not in the dump, the node is not running it, whatever the editor shows. A stale snapshot can be cleared for one node, which forces a rebuild and a re-push.

When a change "isn't taking effect", this check splits the problem in two: either the config never reached the node, or it did and the cause is elsewhere. See Snapshot Dump.

Several Envoy versions at once​

Elchi runs a control-plane instance for each Envoy version you deploy. When a proxy connects, a registry matches it to the control plane that speaks the version it reports, so a mixed fleet is a normal state, not an emergency.

Moving listeners to a newer version is a tracked background job. You select the listeners, pick a target version and can run it as a dry run first. The controller:

1. runs a per-listener dependency analysis to find everything that has to move together,
2. recreates the dependencies, then the listeners, in topological order,
3. regenerates snapshots and bootstrap configs for the target version.

Managed listeners switch to the new version through an Envoy hot restart, with no traffic interruption. Once nothing is left on the old version, a cleanup removes that version's leftover resources. See Versions & Upgrades.

Where the proxies run​

A central platform does not help if getting Envoy onto hosts is still manual. Each edge host runs a small agent, elchi-client. It has no inbound listener: it dials out to the controller, registers the node, runs the lifecycle commands the controller sends, and ships logs. Envoy's admin interface stays on loopback.

On the central side, one public TLS listener on :443 carries the UI, the REST API, the agents' command stream, xDS and access logs. An ext_proc filter in front inspects each stream and routes it to the right internal service, so the internal ports never have to be exposed. The full wire map is in Architecture.

A minimal end-to-end path​

The shortest route from nothing to traffic is on Kubernetes:

helm repo add elchi https://charts.elchi.io
helm repo update

helm install elchi elchi/elchi-stack \
  --set-string global.mainAddress=192.168.1.10 \
  --namespace elchi-stack \
  --create-namespace

192.168.1.10 stands for the IP of one of your nodes or your load balancer. Then install the agent on one Linux host and build the four resources every HTTP path needs: endpoints, cluster, route configuration and listener. Publish them and send a request. The full walkthrough, including the client install, is in From Install to First Listener.

Docker Swarm, single-host and bare-metal (systemd) installs bring up the same control plane.

When you don't need this​

If you run a handful of proxies whose config changes rarely, a static bootstrap in version control is a fine choice and much simpler. A control plane earns its keep when the number of proxies, the rate of change, or the number of Envoy versions you run at once outgrows reviewing YAML diffs by hand. That is roughly when the six responsibilities above stop being theoretical.

Summary​

Hand-written xDS fails on references, validation and versions, not on syntax. A control plane that models resources as typed objects per Envoy version, validates them twice, publishes them as a unit and lets you read back what each node received removes most of those failure modes before a proxy ever sees them.

Without a license, Elchi runs a free tier, Elchi Core with a single managed client, which is enough to try the workflow end to end (Licensing).

Next steps​

• Follow the how-to: Manage Envoy proxies without hand-writing xDS or YAML.
• Running more than one Envoy version? See Run multiple Envoy versions.
• Read the product overview: Elchi Core.
• Want to see it on your own topology? Book a demo.

What Kubernetes and a service mesh normally give you for free is everything around the binary: a bootstrap for each proxy, a connection to a control plane, a supervisor that restarts it, a way to upgrade it, and a path for its logs and metrics. This post lists what you have to provide yourself on a VM, and then shows how the Elchi edge agent, elchi-client, provides each piece.

What a VM-based Envoy needs​

A bootstrap per proxy​

Every Envoy process starts from a bootstrap file. At minimum it sets a node identity (node.id, node.cluster), the admin interface, and either the full configuration under static_resources or a dynamic_resources section that points at a control plane. If you run several Envoy processes on one host, each needs its own bootstrap, its own admin port and its own --base-id, the value Envoy uses to keep the shared memory of separate processes apart.

A connection to a control plane​

With a static bootstrap, every change means editing a file on every host and restarting. With dynamic_resources and ADS, Envoy keeps a gRPC stream open to a management server and applies listener, route, cluster, endpoint and secret changes live. Our post What is an Envoy control plane? explains the xDS side.

Process supervision​

On a VM, something has to start Envoy at boot, restart it when it crashes, limit its resources and stop it cleanly. On Linux that is systemd. You also need the unit to fail early on a broken bootstrap instead of crash-looping; envoy --mode validate checks a bootstrap without starting the proxy.

Binary distribution and upgrades​

Envoy releases often. You need a way to get a given version onto each host, verify it, keep the previous version around, and move a running proxy to the new binary. A plain systemctl restart closes the listening sockets and the connections on them. Envoy's answer is hot restart: a new process is started with --restart-epoch one higher, takes over the listening sockets from the old process over a Unix domain socket, and the old process drains its connections for --drain-time-s and is shut down after --parent-shutdown-time-s. Hot restart only works between two binaries that report the same --hot-restart-version, and Envoy itself does not supervise this; the upstream project ships a small Python wrapper, hot-restarter.py, that forks the new epoch when it receives SIGHUP. See the hot restart documentation.

Logs and metrics​

Envoy writes access logs, exposes statistics on its admin interface and can push both through stats sinks and the access log service (ALS). On a VM there is no sidecar collector by default, so you choose where each signal goes and wire it into every bootstrap.

How Elchi does it​

Elchi splits this work between the central platform and an agent on each host. The control plane builds configuration and serves it over xDS; elchi-client runs on every edge VM, registers with the platform, and does the host-side work.

Registration​

elchi-client has no inbound listener. It dials out to the platform's controller over gRPC (through the platform's single front door on port 443) and registers with a token of the form <uuid>--<projectId>, which ties the host to one project. On every registration it also reports the listeners it already runs, so the control plane can reconcile what it expects with what is on the box. Installation is one command; see Client installation.

Envoy binaries​

The agent downloads Envoy binaries from the release archive (https://archive.elchi.io/index.json by default, configurable for a mirror), verifies each download's checksum, and stores every version under its own directory in /var/lib/elchi/envoys/. Several versions can be present at once, and the client's Apps tab in the UI shows which ones are downloaded and which can be installed. See Clients.

One systemd unit per listener​

When you deploy a listener from the UI, the control plane generates its bootstrap. That bootstrap sets the node ID (listener::project::ip), fetches LDS and CDS over ADS with delta xDS, and sends the Envoy version as gRPC metadata so the platform routes the stream to the control plane built for that version.

The agent writes the bootstrap to /var/lib/elchi/bootstraps/ and runs the listener as its own systemd unit, <listener>-<adminPort>.service. One Envoy process per listener means a bad change, a crash or an upgrade affects one listener, not every listener on the host. Each unit:

• validates the bootstrap with the same binary (--mode validate) before starting;
• uses the admin port as --base-id, so the processes on one host never share hot-restart memory;
• runs Envoy through a hot-restart wrapper, with ExecReload sending SIGHUP;
• restarts 3 seconds after Envoy exits, and stops trying after 10 starts in 5 minutes;
• adds the listener's address to the host only once Envoy reports it is live and ready, and removes it when the unit stops, so a crashed or held listener stops attracting traffic;
• caps memory with MemoryHigh and the Envoy overload manager that the agent adds to the bootstrap, rather than a hard kill.

Deployed listeners appear in the UI under Services, with their status per client.

Zero-downtime version upgrades​

Upgrading a listener to a newer Envoy version is a job in the UI or API. The control plane copies the listener's dependencies into the target version and moves the listener and its bootstrap; the agent then switches the running process.

The agent does not point the unit at a versioned binary path. The unit runs a per-listener symlink in /var/lib/elchi/envoys/current/, and the wrapper re-runs that same path on SIGHUP. To upgrade, the agent compares the --hot-restart-version of the running binary with that of the target binary. When they match, it repoints the symlink and reloads the unit: Envoy's hot restart starts the next epoch, which takes over the listening sockets while the old process drains. When they do not match, the agent falls back to a full restart and says so in the job result.

In our end-to-end tests, the listener upgrade runs against a control plane serving at least two Envoy versions, with a client on a separate VM. The recorded run moved a listener from Envoy 1.38.3 to 1.39.0 under a steady load of 20 requests per second: 1,499 requests were sent and 0 were dropped. The service kept the same systemd main PID, the restart epoch went up by one, and the agent reported "hot restarted (epoch 0 -> 1), zero connection loss". A later run of the same test group, upgrading from 1.39.1 to 1.39.3, again lost no request. The test judges a dropped request with Envoy's own request counters, read before and after the load window, not only with the load generator's view.

The conditions matter. The result applies when the two binaries are hot-restart compatible, the target version is already on the agent, and every client of the service is connected; otherwise the job fails or the agent falls back to a full restart, which drops the open connections. The step-by-step guide is Run several Envoy versions side by side, and the details are in Versions & Upgrades.

Logs and metrics​

Envoy statistics go through a stats sink to an OpenTelemetry Collector and on to VictoriaMetrics, with dashboards in the UI and Grafana. Access logs can stream over ALS to elchi-collector for API discovery. Service logs are searchable in the UI's log viewer, and the agent can forward them to Syslog or Elastic/Logstash. See Metrics & Logs.

High availability without Kubernetes​

A single VM is a single point of failure. Elchi offers two ways to remove it.

With BGP, several edge hosts can announce the same listener address, and each unit's address handling means a host whose listener is down stops announcing it.

Without BGP, use an HA pair: two Elchi OS appliances that serve the same services on the same virtual IPs. Both members receive the same xDS configuration and run the same Envoy listeners; only the active member holds the VIPs. The boxes elect the active member themselves with a heartbeat on UDP 8650 and probes of the peer's data plane, so a control-plane outage never moves a VIP. On failover, the standby adds the VIP and sends gratuitous ARP. An Envoy hot restart or an xDS update does not cause a failover. HA pairs need Elchi OS on both members; they are not available for a client installed on a general-purpose Linux host. See HA Pairs Overview for the failover rules and the measured takeover times.

Next steps​

• Install the agent: Elchi Client installation.
• Deploy and upgrade listeners: Run several Envoy versions side by side and Manage Envoy without YAML.
• Remove the single box: HA Pairs Overview and Elchi OS.
• Learn more about Elchi Core, or book a demo.