chore: add internal markdown link check

[Geethapranay1]

Which issue does this PR close?

• Closes chore: Create CI action that validates links in md files #21747

Rationale for this change

datafusion did not have a CI check for broken links in markdown content, docs workflows build and deploy docs, and dev checks formatting and spelling, but none of them validate link targets.
This pr adds a dedicated link check for internal markdown links so broken references fail early in PRs.
I kept the scope internal-only to avoid flaky CI failures from external websites and rate limits.
Rust doc comments remain covered by the existing rustdoc CI job.

What changes are included in this PR?

• Added a new Dev workflow job, Check Markdown Links, in dev.yml.
• Added LYCHEE_VERSION pin in tool_versions.sh.
• Added markdown_link_check.sh to run lychee on the selected markdown paths.
• Added lychee.toml with internal-link policy and exclusions.
• Added check markdown links to required status checks in .asf.yaml.
• Updated contributor testing docs with the new local command and scope note.
• Fixed internal markdown links that failed under the new check in:
  • roadmap.md
  • 49.0.0.md
  • overview.md
  • dataframe.md
  • format_options.md

Are these changes tested?

Yes,

• python3 ci/scripts/check_asf_yaml_status_checks.py passed.
• bash -n ci/scripts/markdown_link_check.sh passed.
• bash ci/scripts/markdown_link_check.sh passed with 0 errors.
• cargo fmt --all --check passed.

OK: All 5 required_status_checks match existing GitHub Actions jobs.
🔍 12824 Total (in 0s) ✅ 490 OK 🚫 0 Errors 👻 12334 Excluded

Are there any user-facing changes?

No,

There is one contributor-facing CI change: PRs now fail when internal markdown links break in the checked markdown files.

[Geethapranay1]

@comphead PTAL

[comphead]

this is very nice

[comphead]

for local we need to document mapfile is needed

[comphead]

and also mapfile is not available on macOS

[comphead]

and cargo install lychee

[comphead]

Thanks @Geethapranay1 for the PR

I tried to check it locally but with macOS I'm missing some bash commands. Would you help to attach to the PR how bad link would look like.

I'm assuming it should be very clear to the user what link needs to be fixed

[Geethapranay1]

Thanks @comphead for the detailed review and for testing this on macOS.

I will:

• update the local script to avoid macOS-missing bash commands (mapfile),
• document local prerequisites including cargo install lychee,
• add an example of how a broken link error looks in output so it is clear what to fix.

[Geethapranay1]

@comphead PTAL, i have changed according to the review

[comphead]

oh, lychee doesn't refer to a specific line in the md file?

[comphead]

Thanks @Geethapranay1 this pr makes a lot of sense