gudusoft.gsqlparser.parser.powerquery

Class PowerQueryTokenizer

Object

gudusoft.gsqlparser.parser.powerquery.PowerQueryTokenizer

public class PowerQueryTokenizer
extends Object

Hand-written Power Query M-Language tokenizer.

Reads raw M text and produces a TSourceTokenList compatible with the rest of the GSP machinery. Emits:

• IDENT (TBaseType.ident) — simple identifiers
• QIDENT (PowerQueryTokenCodes.QIDENT) — #"..."
• SCONST (TBaseType.sconst) — string literals, raw form preserved in token text
• ICONST / FCONST — integer / decimal numbers
• M keywords (PowerQueryTokenCodes.RW_LET …)
• Single-char punct (ASCII code as token code)
• FAT_ARROW (=>), DOUBLE_DOT (..), ELLIPSIS (...)
• _COMMENT (//) and block-comment tokens — treated as whitespace by downstream, but preserved so the token list can be inspected for bug reports

The tokenizer is deliberately simple and forgiving: unknown characters become single-character tokens rather than errors. The document parser decides what to do with them.

Constructor Summary

Constructors

Constructor and Description
PowerQueryTokenizer(String text)

Method Summary

Modifier and Type	Method and Description
static PowerQueryTokenizer	from(BufferedReader reader) Convenience constructor that slurps a BufferedReader.
void	tokenizeInto(TSourceTokenList out) Tokenize text and append every produced token to out.

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PowerQueryTokenizer

public PowerQueryTokenizer(String text)

Method Detail

from

public static PowerQueryTokenizer from(BufferedReader reader)
                                throws IOException

Convenience constructor that slurps a BufferedReader.

Throws:

IOException

tokenizeInto

public void tokenizeInto(TSourceTokenList out)

Tokenize text and append every produced token to out.