Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.pipedata.io/llms.txt

Use this file to discover all available pages before exploring further.

Two ways to get records into a pipe: stream NDJSON from a shell with pd push, or expose an HTTP endpoint with pd source http.

​
TL;DR

# From stdin
cat data.jsonl | pd push orders

# Over HTTP
pd source http orders
# → prints an ingest URL; POST NDJSON or JSON to it with the workspace API key

​
From stdin: pd push

pd push <name> reads NDJSON from stdin and writes it to the named pipe. The pipe is created on first use. 
$ echo '{"event":"signup","user":1}' | pd push events
$ kubectl logs -f my-svc             | pd push logs
$ cat backfill.jsonl                 | pd push orders
Each line of stdin must be a single JSON value. Status output goes to stderr, so redirects stay clean.
pd push is streaming — the writer doesn’t wait for a reader. Push as fast as you like; readers catch up at their own pace.

​
Over HTTP: pd source http

pd source http <name> creates (or looks up) an HTTP ingestion pipe and prints its ingest URL. 
$ pd source http webhooks
created source:http "webhooks"

  POST records to:
    https://ingress.eu-central-1.pipedata.io/123/data/456

  authenticate with your workspace API key:
    Authorization: Bearer <api-key>
Send one record per request, or batch many as NDJSON: 
# Single record
curl -X POST "$INGEST_URL" \
  -H "Authorization: Bearer $PD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"event":"signup","user":1}'

# Batch (NDJSON, one record per line)
curl -X POST "$INGEST_URL" \
  -H "Authorization: Bearer $PD_API_KEY" \
  -H "Content-Type: application/x-ndjson" \
  --data-binary @batch.ndjson
A 2xx response means the records were accepted and persisted.

​
Common patterns

Backfill a pipe from a file 
$ cat history.jsonl | pd push orders
Convert CSV on the fly 
$ awk -F, 'NR>1 {printf "{\"email\":\"%s\"}\n",$2}' users.csv \
  | pd push emails
Mirror an HTTP webhook into a pipe 
$ pd source http shopify-webhooks
# point Shopify at the printed URL with your API key in the Authorization header

​
See also