Skip to content

clear escape delimiter buffer before peek in isEscapeDelimiter#608

Open
rootvector2 wants to merge 1 commit into
apache:masterfrom
rootvector2:escape-delimiter-buffer-clear
Open

clear escape delimiter buffer before peek in isEscapeDelimiter#608
rootvector2 wants to merge 1 commit into
apache:masterfrom
rootvector2:escape-delimiter-buffer-clear

Conversation

@rootvector2

Copy link
Copy Markdown

isEscapeDelimiter() peeks the next characters into the reused escapeDelimiterBuf look-ahead without clearing it first, so a truncated escaped multi-character delimiter at EOF is completed from the stale bytes of an earlier full escaped delimiter and the lexer appends a delimiter the input never contained. clearing the buffer before the peek mirrors the delimiterBuf reset added in nextToken for the plain multi-character delimiter path.

found while reading the partial-delimiter-at-EOF fix (CSV-324): the escaped-delimiter sibling was left unguarded.

for delimiter [|] and escape !, parsing x![!|!]y![!| returns x[|]y[|] before the change and the correct x[|]y![!| after. regression tests added in LexerTest and CSVParserTest (both fail without the one-line change).

  • Read the contribution guidelines for this project.
  • Read the ASF Generative Tooling Guidance if you use Artificial Intelligence (AI).
  • I used AI to create any part of, or all of, this pull request. Which AI tool was used to create this pull request, and to what extent did it contribute?
  • Run a successful build using the default Maven goal with mvn; that's mvn on the command line by itself.
  • Write unit tests that match behavioral changes, where the tests fail if the changes to the runtime are not applied. This may not always be possible, but it is a best practice.
  • Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
  • Each commit in the pull request should have a meaningful subject line and body. Note that a maintainer may squash commits during the merge process.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant