feat: support reading from stdin in datafusion-cli

[huan233usc]

Which issue does this PR close?

• Closes support for reading data from stdin for datafusion-cli #9430.

Rationale for this change

Users frequently want to pipe data into the CLI, e.g. cat data.csv | datafusion-cli, but pointing LOCATION at /dev/stdin did not work:

• CSV failed with Illegal seek (a pipe is not seekable).
• Parquet failed with file size of 0 is less than footer (a pipe reports size 0).
• JSON silently returned 0 rows.

This PR makes reading from standard input work for CSV, JSON, and Parquet.

What changes are included in this PR?

stdin is exposed as a stdin:// object store, dispatched alongside the other schemes (s3, gs, http, ...) in get_object_store — conceptually similar to DuckDB's PipeFileSystem.

• rewrite_stdin_location maps the well-known stdin pseudo-paths (/dev/stdin, /dev/fd/0, /proc/self/fd/0) to a canonical stdin:///stdin.<ext> URL, so they flow through the normal object-store/listing code path. The extension matches the declared STORED AS format because the listing layer filters candidate files by extension.
• The stdin:// store reads all of standard input into an in-memory object store. Buffering up front is required because a pipe is not seekable and Parquet stores its metadata at the end of the file.

Known scope/limitations (left as potential follow-ups):

• Only CREATE EXTERNAL TABLE is supported (not dynamic SELECT * FROM '/dev/stdin').
• Input is fully buffered in memory, so it must fit in memory.
• stdin can only be consumed once per session.
• Unix-only (/dev/stdin does not exist on Windows); writing to /dev/stdout is out of scope.

Are these changes tested?

Yes:

• Unit tests in object_storage.rs cover rewrite_stdin_location and end-to-end reads for CSV, JSON, and Parquet via the in-memory store.
• A #[cfg(unix)] integration test in cli_integration.rs drives the real binary through an actual pipe, exercising the real stdin read.
• Manually verified all three formats via real pipes, and confirmed normal local-file reads are unaffected.

Are there any user-facing changes?

Yes — reading from stdin via LOCATION '/dev/stdin' is now supported. Documented in docs/source/user-guide/cli/datasources.md (new "Reading from standard input" section). No breaking changes.

[kosiew]

@huan233usc
Thanks for working on this. I found one blocking issue around /dev/stdin registration and object store replacement that can cause previously created tables to silently change behavior within the same session.

[kosiew]

I think there's a correctness issue here. object_store re-reads process stdin every time a stdin:// store is registered.

After the first CREATE EXTERNAL TABLE ... LOCATION '/dev/stdin', the stdin contents have already been consumed and buffered. If a second stdin-backed table is created in the same session, this path reads EOF and replaces the registered stdin:// store with an empty one. As a result, the first table silently changes from containing data to appearing empty.

Repro:

CREATE EXTERNAL TABLE t ... LOCATION '/dev/stdin';
SELECT count(*) FROM t; -- 2

CREATE EXTERNAL TABLE t2 ... LOCATION '/dev/stdin';
SELECT count(*) FROM t; -- 0

Could we enforce the one-shot stdin invariant when registering the store? Two approaches that seem reasonable are:

• Reject subsequent /dev/stdin registrations with a clear error before replacing the existing store.
• Cache and reuse the first buffered store for the same stdin:// URL.

The important part is avoiding replacement of the backing object after tables have already been registered.

[huan233usc]

Thanks for the feedback. Yes this is a real correctness bug.

I went with the "reuse the buffered store" option. stdin is read once on the first stdin:// table; subsequent stdin:// tables in the same session reuse the already-registered store instead of re-reading (now-EOF) stdin and overwriting it.

I added an unit test to cover the scenario below

$ printf 'a,b\n1,foo\n2,bar\n' | datafusion-cli -q --command "
CREATE EXTERNAL TABLE t STORED AS CSV LOCATION '/dev/stdin' OPTIONS ('format.has_header' 'true');
SELECT count() FROM t; -- 2
CREATE EXTERNAL TABLE t2 STORED AS CSV LOCATION '/dev/stdin' OPTIONS ('format.has_header' 'true');
SELECT count() FROM t; -- 2 (was 0 before the fix)
SELECT count(*) FROM t2; -- 2
"
PTAL. Thanks!

[kosiew]

@huan233usc
Thanks for addressing the original CSV/CSV repro. Reusing the existing stdin:// store and adding coverage for two CSV /dev/stdin tables in a single CLI session fixes the reported case.

That said, I think there's still a gap around the broader one-shot stdin invariant. The current implementation appears to work when subsequent registrations resolve to the same rewritten object path, but it can still break when later /dev/stdin tables use a different STORED AS format.

[kosiew]

Nice catch reusing the existing registered store, but I think there's still a corner case here.

rewrite_location produces format-specific paths such as stdin:///stdin.csv, stdin:///stdin.json, and stdin:///stdin.parquet. The object-store registry lookup is keyed by scheme/authority rather than path, so after the first table registers stdin:///stdin.csv, a later /dev/stdin table rewritten to stdin:///stdin.json will hit this branch and reuse the existing store.

The problem is that the store only contains /stdin.csv. The second table then attempts to read /stdin.json from that store and fails (or appears missing) rather than reusing the buffered stdin bytes or producing a clear error.

So this fixes the reported CSV/CSV case, but not the broader invariant that subsequent /dev/stdin registrations should remain safe and predictable. Could we either make the stored object path canonical across formats, populate the newly requested path from the buffered stdin data, or explicitly reject a second /dev/stdin registration when the rewritten path differs and return a clear error?

[huan233usc]

Thanks @kosiew — went with explicit rejection. The buffered object's name already records what format stdin was read as, so a second /dev/stdin table with a different STORED AS now fails with:

stdin was already read as 'stdin.csv' by an earlier statement; all tables backed by stdin in a session must declare the same STORED AS format

We could revisit sharing the bytes across formats later if there's a real use case, but it can fail silently today (piped NDJSON still "parses" as single-column CSV), so erroring felt safer.

Seperately, I also fixed a footgun I hit while testing: if stdin is already carrying the SQL (REPL, or piping a script without -c/-f), the stdin read used to eat the rest of the script or hang the shell. Now it errors and points to -c/-f. Tests added for both. PTAL!