A Hashicorp Boundary Installation Checklist
If you’ve come here after searching for an error message, see Errors That Might Have Led You Here.
I’ve been playing around with the excellent Boundary (0.5.0) from Hashicorp recently, which you can use as a software defined perimeter (SDP) within a zero-trust architecture (ZTA).
It’s early days for Boundary, the supporting docs and community resources. After reading through the docs & tutorials available on the official site, I thought an installation / troubleshooting / debug checklist was in order. As a distributed security system, there is a lot of configuration to get right and several possible failure modes if you mess up.
Clearly this isn’t going to be exhaustive, but hopefully it helps someone!
Nodes
| Purpose | Software |
|---|---|
| Client | Boundary binary |
| Client | Boundary desktop app |
| Controller | Boundary binary |
| Worker | Boundary binary |
| Load balancer for controller(s) | Web server |
| Database | Postgres |
Network
Check these using netcat from *nix systems & telnet on Windows. You will need to start listener on the receiving end using nc -lvp {port}, if you haven’t started the Boundary services yet.
| Path | Port |
|---|---|
| Client -> LB | 443 |
| LB -> Controller | 9200 |
| Controller <-> Worker | 9201 |
| Client -> Worker | 9202 |
| Worker -> Target | Target app port |
| Controller -> Postgres | 5432 |
OS & Files
Assuming Boundary is being run via Systemd
| Node | Aspect | Check | Remediation |
|---|---|---|---|
| Controller & Worker | boundary user1 | grep boundary /etc/password | See below |
| Controller & Worker | boundary group2 | grep boundary /etc/group | useradd --system --no-create-home --shell /usr/sbin/nologin --user-group boundary |
| Controller & Worker | Allow infinite memory locking (stops secrets leaking via memory paging) | Check Boundary service file for LimitMEMLOCK=infinity Capabilities=CAP_IPC_LOCK+ep CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK | Add missing lines, daemon-reload and restart service. |
| Controller & Worker | PKI file ownership3 | Check boundary:boundary ownership on certificate cert & key | chown boundary:boundary {the files} |
Database
| Aspect | Check | Remediation |
|---|---|---|
boundary user is super user4 | From psql shell, \du | As postgres user in bash, createuser -P --interactive boundary |
boundary database exists5 | From psql shell, \l | As postgres user in bash, createdb boundary |
| Allow external password authentication5, 6, 7 | psql -U boundary -h 127.0.0.1 -d boundary | In pg_hba.conf add host all all 0.0.0.0/0 md5 |
| Postgres listening on external interface5, 7 | ss -plnt should show 0.0.0.0:5432 | In postgresql.conf amend listen_addresses = '*' |
Environment Variables
These are optional, but might make things easier in some cases.
| Name | Should be |
|---|---|
| BOUNDARY_ADDR | {Controller address}:{API port, usually 9200} |
| BOUNDARY_CACERT | Path to a PEM-encoded CA cert file used to verify the Boundary server SSL certificate. |
| BOUNDARY_CAPATH | Path to a directory of PEM-encoded CA cert files to verify the Boundary server SSL certificate. |
| BOUNDARY_CLIENT_CERT | Path to the certificate for Boundary communication |
| BOUNDARY_CLIENT_KEY | Path to the private key for Boundary communication |
| BOUNDARY_CLIENT_TIMEOUT | Timeout in seconds (default 60) |
| BOUNDARY_TLS_INSECURE | Enables or disables SSL verification (boolean) |
| BOUNDARY_TLS_SERVER_NAME | Used to set the SNI host when connecting via TLS |
| BOUNDARY_MAX_RETRIES | Number of retries for the HTTP client |
| BOUNDARY_TOKEN | Add this token to the Boundary client |
| BOUNDARY_RATE_LIMIT | Rate limiting for the client, unsure what value would be used |
| BOUNDARY_SRV_LOOKUP | Enables the client to lookup the host through DNS SRV lookup (boolean) |
Errors That Might Have Led You Here
Check for the reference number in the rest of this post
boundary-worker.service: Failed at step USER spawning /usr/local/bin/boundary: No such processstatus=216/GROUPboundary-controller.service: Failed at step GROUP spawning /usr/local/bin/boundary: No such processstatus=217/USEREnter passphrase for /etc/boundary/pki/key.pem: Error initializing listener of type tcp: error loading TLS cert: EOFstatus=3/NOTIMPLEMENTED- Exceptions arising from
boundary database init, in particular relating topgcrypto - Unable to connect to
boundarydatabase duringboundary database init Peer authentication failedwhen trying to authenticate asboundary, check this withpsqlnotboundary database init- Unable to connect to
5432on Postgres from Controller. Confirmed issue ifnc -v {postgres IP} 5432from Controller does not work