Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.pipedata.io/llms.txt

Use this file to discover all available pages before exploring further.

pd pull <name> writes NDJSON to stdout, one record per line. Pipe it into jq, redirect to a file, or tail with --follow.

​
TL;DR

$ pd pull orders                       # read from where this machine left off
$ pd pull orders --start-id 0          # replay from the top
$ pd pull orders --follow              # tail forever, like tail -f
$ pd pull orders | jq 'select(.amount > 1000)'

​
How it works

Each machine keeps a local cursor per pipe under ~/.pd/. The cursor tracks the last record ID this machine read. The next pd pull resumes from cursor + 1.
  • --start-id N — start reading from record ID N. Does not move the cursor backwards; use it for one-off replays.
  • --follow — when the pipe is caught up, keep the connection open and stream new records as they arrive. Stop with Ctrl-C.
Status output goes to stderr, so pd pull > out.jsonl always produces clean NDJSON in out.jsonl.

​
Common patterns

Filter to interesting records 
$ pd pull payments | jq 'select(.amount > 1000)' > big-payments.jsonl
Tail logs from a server to your laptop 
# server
$ kubectl logs -f api | pd push api-logs

# laptop
$ pd pull api-logs --follow | jq 'select(.level=="error")'
Replay from a known starting point 
$ pd pull canary-traffic --start-id 0          # from the beginning
$ pd pull canary-traffic --start-id 50000      # from record 50000
Hand a dataset to a teammate 
# you
$ cat dump.jsonl | pd push debug-2026-05

# them
$ pd pull debug-2026-05 > dump.jsonl

​
See also

  • Send data — write records into a pipe.
  • Manage pipes — pd ls shows the backlog waiting on the reader.
  • pd pull in the command reference.