public final class Pp2TokenStream extends Object implements Iterable<Pp2Token>
TSourceTokenList as a sequence of
Pp2Tokens.
The constructor copies token references from the input
TSourceTokenList into a private ArrayList — the source
list is never mutated, and the resulting stream cannot be structurally
mutated (no add/remove/clear). Per-token role bookkeeping on individual
Pp2Tokens remains writable because pp2 stages annotate roles
incrementally (see TokenRole).
Two factory paths are supported:
fromSourceTokenList(TSourceTokenList) — basic wrap with
zero precedingBlanks/precedingLinebreaks. Used by
early Phase-1 callers and tests.ofTokens(List) — internal entry point used by
Pp2TokenStreamBuilder (S7), which folds whitespace into the
next solid token's preceding-whitespace counts before constructing
the stream.Probe A7 (slice S1) established that TGSqlParser reuses the
same TSourceTokenList reference across calls. This stream's
copy-on-construct contract is the reason that finding is not a footgun:
the next re-tokenization cannot invalidate an already-built stream's
structure.
size() and the per-index Pp2Token wrapper
references are stable. Re-tokenizing the source parser does not
grow, shrink, or re-shuffle the stream, and never replaces a
wrapper reference at a given index. toReadOnlyList()
returns this stable structure.Pp2Token.getText()
delegates live to the wrapped TSourceToken. If the parser
re-tokenizes a different SQL and re-uses the underlying
TSourceToken fields, the live text returned by an existing
wrapper changes. Neither toReadOnlyList() nor any other
method in this class snapshots text. Callers that must outlive a
parser re-tokenization MUST snapshot text into their own
List<String> before the second tokenize call (see
Pp2TokenStreamTest#textSnapshotSurvivesParserReuse).| Modifier and Type | Method and Description |
|---|---|
static Pp2TokenStream |
fromSourceTokenList(TSourceTokenList source)
Build a stream by wrapping each
TSourceToken in the input
list as a Pp2Token with zero preceding whitespace. |
Pp2Token |
get(int index) |
boolean |
isEmpty() |
Iterator<Pp2Token> |
iterator() |
static Pp2TokenStream |
ofTokens(List<Pp2Token> tokens)
Build a stream from a previously-prepared list of
Pp2Tokens. |
int |
size() |
List<Pp2Token> |
toReadOnlyList()
Unmodifiable view of the underlying token list.
|
String |
toString() |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, waitforEach, spliteratorpublic static Pp2TokenStream ofTokens(List<Pp2Token> tokens)
Pp2Tokens.
The supplied list is copied; subsequent caller mutations do not leak.NullPointerException - if tokens or any element is nullpublic static Pp2TokenStream fromSourceTokenList(TSourceTokenList source)
TSourceToken in the input
list as a Pp2Token with zero preceding whitespace.
Smarter folding of whitespace into precedingBlanks/
precedingLinebreaks is the job of
Pp2TokenStreamBuilder (S7); this factory is the minimal
Phase-1 wrap.
Contract: the input TSourceTokenList is not mutated. The
stream holds the same TSourceToken references as the input,
verifiable in tests by reference + text equality before vs after
construction.
A null element in the source list is treated as a parser-side bug
and triggers a NullPointerException — silently skipping would
shift the per-index mapping and hide corruption.
NullPointerException - if source is null or contains a
null elementpublic int size()
public boolean isEmpty()
public Pp2Token get(int index)
IndexOutOfBoundsException - if index is out of rangepublic List<Pp2Token> toReadOnlyList()