Merge scan options into FileIO at TableScan.__init__ time

[rccreager]

Closes #3166

Rationale for this change

The Table.scan(..) method accepts a parameter options: Properties; this parameter is not actually threaded into the FileIO instance used in the DataScan class which scans the data. Instead, the internal callers were using a FileIO instance with only the basic catalog-level parameters set and were not respecting the user-specified options.

This is resolved by merging scan options into the FileIO at TableScan.__init__ time, using the existing load_file_io method. This means that if the user provides new parameters to scan, a new FileIO instance must be created with the merged properties. FileIO properties are immutable after construction, and the cached filesystem objects are built from those properties at first access, so a new instance is required.

If the options are empty or if the key:value pairs are all present in the existing FileIO options, you can use the existing FileIO instance.

I wanted to fix the problem at initialization, rather than at the various callsites, because self.io is also used in scan_plan_helper() for manifest reads, not just in to_arrow() for data reads. Fixing at init time covers both.

Are these changes tested?

I added tests to confirm that the init method has the expected behavior.

Are there any user-facing changes?

Yes. Scan-level options passed to Table.scan(options=...) are now respected by the underlying FileIO during data and manifest reads. Previously these options were silently ignored.

[ndrluis]

I have mixed feelings about recreating FileIO here. Even if this works in practice, creating a new IO instance during scan(options=...) makes the scan behavior broader and can also make debugging more complex, because the IO actually used by the scan may no longer be the same one that was originally attached to the table/catalog. More generally, I would expect this kind of IO configuration to already be part of the catalog/table IO instance, rather than being reapplied by rebuilding FileIO inside the scan path.

I also tried to find a similar pattern in the Java implementation and could not find one. There, scan options are added to scan context, while scan planning still uses table.io():

• BaseScan.io(): https://github.com/apache/iceberg/blob/ccb8bc435062171e64bc8b7e5f56e6aed9c5b934/core/src/main/java/org/apache/iceberg/BaseScan.java#L101-L118
• TableScanContext.withOption(...): https://github.com/apache/iceberg/blob/ccb8bc435062171e64bc8b7e5f56e6aed9c5b934/core/src/main/java/org/apache/iceberg/TableScanContext.java#L154-L156
• DataTableScan using table().io(): https://github.com/apache/iceberg/blob/ccb8bc435062171e64bc8b7e5f56e6aed9c5b934/core/src/main/java/org/apache/iceberg/DataTableScan.java#L64-L74

From what I could find, scan options there are mainly used for scan planning overrides like split sizing / lookback / open-file-cost, not for reconfiguring FileIO:

• BaseScan split planning overrides: https://github.com/apache/iceberg/blob/ccb8bc435062171e64bc8b7e5f56e6aed9c5b934/core/src/main/java/org/apache/iceberg/BaseScan.java#L222-L248
• BaseMetadataTableScan split-size override: https://github.com/apache/iceberg/blob/ccb8bc435062171e64bc8b7e5f56e6aed9c5b934/core/src/main/java/org/apache/iceberg/BaseMetadataTableScan.java#L64-L68