gudusoft.gsqlparser.pp2.token

Class Pp2TokenStreamBuilder

Object

gudusoft.gsqlparser.pp2.token.Pp2TokenStreamBuilder

public final class Pp2TokenStreamBuilder
extends Object

Lossless adapter from TSourceTokenList to Pp2TokenStream.

Implements the Delphi initTokenArray semantics from gsp_vcl/pp/sqlion.pas: walk the token list, fold every ttwhitespace / ttreturn token into the precedingBlanks and precedingLinebreaks counts of the next non-whitespace token. Comments (ttsimplecomment, ttbracketedcomment) are not folded — they are first-class tokens in pp2 so downstream stages can preserve, reanchor, or reflow them per CommentPolicy.

Logical-line semantics

precedingLinebreaks is a count of logical new lines, not raw characters. "\r\n" counts as one linebreak (not two), as does a lone "\n" or a lone "\r". Downstream layout rules in S25/S28 treat each logical linebreak as a new visual line; counting CRLF as two would surface as a spurious blank line on Windows-encoded scripts.

Byte-exact recovery of mixed line endings is not the spine's job — the SourceSpanLedger (S8) records every byte of the original input, including the precise CR/LF/CRLF sequence, so the region assembler (S15) can restore them when emitting output.

Reconstruction property (CRLF-normalized)

After normalizing "\r\n" to "\n" in the original input, summing across the produced stream:

   Σ ( precedingLinebreaks + precedingBlanks + token.text.length() )
     + trailingBlanks + trailingLinebreaks
   == normalized input length
 

The trailing trivia is returned alongside the stream so the source-span ledger (S8) and the region assembler (S15) can place it back into the output.

Counting rules

• "\r\n" consumed together → 1 linebreak.
• Lone "\n" → 1 linebreak.
• Lone "\r" → 1 linebreak.
• Any other character (space, tab, form-feed, vertical tab, NBSP) → 1 blank. countWhitespace() only sees lexer-classified whitespace tokens, so non-whitespace UTF-8 characters never reach it.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Pp2TokenStreamBuilder.BuildResult Result of a build: the stream of solid + comment tokens, plus any trailing trivia characters that appeared after the last such token.

Constructor Summary

Constructors

Constructor and Description
Pp2TokenStreamBuilder()

Method Summary

Modifier and Type	Method and Description
Pp2TokenStreamBuilder.BuildResult	build(TSourceTokenList source) Build a stream from the given source token list.
static int[]	countWhitespace(String text) Classify each character in a whitespace-token's text as blank vs linebreak.
static boolean	isFoldable(ETokenType type) Is this token type rolled into preceding-whitespace counters rather than emitted as its own Pp2Token? Only true whitespace tokens — comments are first-class in pp2.

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Pp2TokenStreamBuilder

public Pp2TokenStreamBuilder()

Method Detail

build

public Pp2TokenStreamBuilder.BuildResult build(TSourceTokenList source)

Build a stream from the given source token list.

Throws:

NullPointerException - if source is null or contains a null element

isFoldable

public static boolean isFoldable(ETokenType type)

Is this token type rolled into preceding-whitespace counters rather than emitted as its own Pp2Token? Only true whitespace tokens — comments are first-class in pp2.

countWhitespace

public static int[] countWhitespace(String text)

Classify each character in a whitespace-token's text as blank vs linebreak. '\n' and '\r' are linebreaks; everything else (space, tab, form-feed, vertical tab, NBSP, ...) is a blank.

Returns:

int[2] with [0]=blanks, [1]=linebreaks