pd dest http turns a pipe into a webhook fan-out: records flowing into the source pipe get POSTed (or GET/PUT/DELETE/PATCH) to a URL you own, with optional retries and a configurable rate cap.
TL;DR
How it works
A dest http pipe holds three things: a source pipe (where records come from), a URL + method (where they go), and a delivery state (running / paused).
When you create a destination it’s paused by default. You need to:
- Connect a source with
--source <name>.
- Verify DNS ownership of the destination host with
pd verify — unless the host is auto-verified (see below).
- Start it with
pd resume.
Auto-verified hosts
Destinations on hosts where ownership is implied (managed services, webhook-testing tools) skip the DNS check. The list:
- Managed services:
*.snowflakecomputing.com, *.amazonaws.com, *.azure-api.net, *.azurewebsites.net, bigquery.cloud.google.com, *.cloudfunctions.net, *.databricks.com, customer.io.
- Webhook-testing tools (rate-limited to protect shared infrastructure):
*.ngrok.io, *.ngrok-free.app, *.ngrok.app, webhook.site, requestbin.com, pipedream.net, hookbin.com, beeceptor.com.
For these, skip step 2 and go straight to pd resume.
Flags:
| Flag | Default | Purpose |
|---|
--source | | Source pipe to read from. Required on create. |
--url | | Target URL. Required on create. |
--method | | HTTP verb: GET, POST, PUT, DELETE, PATCH. Required on create. |
--rate | 10 | Max requests per second. |
--header K=V | | Request header. Repeatable. Content-Type: application/json is added if you don’t set one. |
--retry | false | Retry on non-2xx responses with exponential backoff. |
--start-id | 0 | First record ID to deliver. Lets you skip backlog on first start. |
For domains you control, pd verify prints a TXT record like _pipedata-challenge.<host> and polls DNS until it sees the value. DNS changes can take minutes to propagate. Auto-verified hosts skip this step.
Retries and undelivered
With --retry, failed deliveries (non-2xx, timeouts, connection errors) are retried with exponential backoff capped at 15 minutes between attempts. After 100 attempts (~22h) a record is marked undelivered and skipped — delivery moves on so the pipe doesn’t head-of-line block.
pd ls shows pending fails and persistent undelivered counts. See Manage pipes.
Common patterns
Forward a pipe to a webhook with retries
Send to an API that needs an auth header
Sensitive headers (Authorization, Cookie, anything starting with X-) are masked in pd ls and pd dest http summary output.
Slow down a chatty destination
Skip backlog on first start
See also
