Send data - Pipedata

Two ways to get records into a pipe: stream NDJSON from a shell with pd push, or expose an HTTP endpoint with pd source http.

TL;DR

# From stdin
cat data.jsonl | pd push orders

# Over HTTP
pd source http orders
# → prints an ingest URL; POST NDJSON or JSON to it with the workspace API key

From stdin: `pd push`

pd push <name> reads NDJSON from stdin and writes it to the named pipe. The pipe is created on first use.

$ echo '{"event":"signup","user":1}' | pd push events
$ kubectl logs -f my-svc             | pd push logs
$ cat backfill.jsonl                 | pd push orders

Each line of stdin must be a single JSON value. Status output goes to stderr, so redirects stay clean.

pd push is streaming — the writer doesn’t wait for a reader. Push as fast as you like; readers catch up at their own pace.

Over HTTP: `pd source http`

pd source http <name> creates (or looks up) an HTTP ingestion pipe and prints its ingest URL.

$ pd source http webhooks
created source:http "webhooks"

  POST records to:
    https://ingress.eu-central-1.pipedata.io/123/data/456

  authenticate with your workspace API key:
    Authorization: Bearer <api-key>

Send one record per request, or batch many as NDJSON:

# Single record
curl -X POST "$INGEST_URL" \
  -H "Authorization: Bearer $PD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"event":"signup","user":1}'

# Batch (NDJSON, one record per line)
curl -X POST "$INGEST_URL" \
  -H "Authorization: Bearer $PD_API_KEY" \
  -H "Content-Type: application/x-ndjson" \
  --data-binary @batch.ndjson

A 2xx response means the records were accepted and persisted.

Common patterns

Backfill a pipe from a file

$ cat history.jsonl | pd push orders

Convert CSV on the fly

$ awk -F, 'NR>1 {printf "{\"email\":\"%s\"}\n",$2}' users.csv \
  | pd push emails

Mirror an HTTP webhook into a pipe

$ pd source http shopify-webhooks
# point Shopify at the printed URL with your API key in the Authorization header

​TL;DR

​From stdin: pd push

​Over HTTP: pd source http

​Common patterns

​See also

TL;DR

From stdin: `pd push`

Over HTTP: `pd source http`

Common patterns

See also