Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.pipedata.io/llms.txt

Use this file to discover all available pages before exploring further.

pd dest http turns a pipe into a webhook fan-out: records flowing into the source pipe get POSTed (or GET/PUT/DELETE/PATCH) to a URL you own, with optional retries and a configurable rate cap.

​
TL;DR

$ pd dest http alerts \
  --source logs \
  --method POST \
  --url https://example.com/hook \
  --retry
$ pd verify  alerts   # add the printed DNS TXT record, then press Enter
$ pd resume  alerts   # start delivering

​
How it works

A dest http pipe holds three things: a source pipe (where records come from), a URL + method (where they go), and a delivery state (running / paused). When you create a destination it’s paused by default. You need to:
  1. Connect a source with --source <name>.
  2. Verify DNS ownership of the destination host with pd verify — unless the host is auto-verified (see below).
  3. Start it with pd resume.

​
Auto-verified hosts

Destinations on hosts where ownership is implied (managed services, webhook-testing tools) skip the DNS check. The list:
  • Managed services: *.snowflakecomputing.com, *.amazonaws.com, *.azure-api.net, *.azurewebsites.net, bigquery.cloud.google.com, *.cloudfunctions.net, *.databricks.com, customer.io.
  • Webhook-testing tools (rate-limited to protect shared infrastructure): *.ngrok.io, *.ngrok-free.app, *.ngrok.app, webhook.site, requestbin.com, pipedream.net, hookbin.com, beeceptor.com.
For these, skip step 2 and go straight to pd resume. Flags:
FlagDefaultPurpose
--sourceSource pipe to read from. Required on create.
--urlTarget URL. Required on create.
--methodHTTP verb: GET, POST, PUT, DELETE, PATCH. Required on create.
--rate10Max requests per second.
--header K=VRequest header. Repeatable. Content-Type: application/json is added if you don’t set one.
--retryfalseRetry on non-2xx responses with exponential backoff.
--start-id0First record ID to deliver. Lets you skip backlog on first start.
For domains you control, pd verify prints a TXT record like _pipedata-challenge.<host> and polls DNS until it sees the value. DNS changes can take minutes to propagate. Auto-verified hosts skip this step.

​
Retries and undelivered

With --retry, failed deliveries (non-2xx, timeouts, connection errors) are retried with exponential backoff capped at 15 minutes between attempts. After 100 attempts (~22h) a record is marked undelivered and skipped — delivery moves on so the pipe doesn’t head-of-line block. pd ls shows pending fails and persistent undelivered counts. See Manage pipes.

​
Common patterns

Forward a pipe to a webhook with retries 
$ pd source http events
$ pd dest http events-webhook \
  --source events \
  --url https://example.com/hook \
  --method POST \
  --retry
$ pd verify events-webhook
$ pd resume events-webhook
Send to an API that needs an auth header 
$ pd dest http forward-paid \
  --source payments \
  --url https://api.example.com/v1/paid \
  --method POST \
  --header "Authorization=Bearer $TOKEN" \
  --header "X-Source=pipedata"
Sensitive headers (Authorization, Cookie, anything starting with X-) are masked in pd ls and pd dest http summary output. Slow down a chatty destination 
$ pd dest http careful \
  --source logs \
  --method POST \
  --url https://api.example.com/in \
  --rate 2
Skip backlog on first start 
$ pd dest http live \
  --source events \
  --method POST \
  --url https://example.com/hook \
  --start-id 1000000

​
See also