TCustomLexer (General SQL Parser Java Library Core 4.1.0.5 API)

Object
- gudusoft.gsqlparser.TCustomLexer

Direct Known Subclasses:

TLexerAccess, TLexerAnsi, TLexerAthena, TLexerBigquery, TLexerClickhouse, TLexerCouchbase, TLexerDatabricks, TLexerDax, TLexerDb2, TLexerDoris, TLexerFlink, TLexerGaussDB, TLexerGreenplum, TLexerHana, TLexerHive, TLexerImpala, TLexerInformix, TLexerMdx, TLexerMssql, TLexerMysql, TLexerNetezza, TLexerOdbc, TLexerOpenedge, TLexerOracle, TLexerPostgresql, TLexerPresto, TLexerRedshift, TLexerSnowflake, TLexerSoql, TLexerSparksql, TLexerSqlite, TLexerStarrocks, TLexerSybase, TLexerTeradata, TLexerVertica
```
public class TCustomLexer
extends Object
```
Base lexer of all databases - Core tokenization engine for SQL parsing. The lexer reads SQL text character by character and produces tokens that represent the syntactic units of SQL. This process involves several key components and stages:
1. Input Management and Buffering
- yyinput (BufferedReader): Primary input source for SQL text
- yyline (char[]): Current line buffer read from input via readln()
- buf (char[]): Reversed line buffer for character-by-character processing
- bufptr: Current position in buf, decrements as characters are consumed
2. Token Text Formation Process
```
 SQL Input → readln() → yyline[] → reversed into buf[] → get_char() → yytextbuf[]
                                                                        ↓
                                                                yylex() processing
                                                                        ↓
                                                                 yylvalstr (String)
                                                                        ↓
                                                            TSourceToken.astext
 
```
Key Variables in Token Text Storage:
- yytextbuf (char[]): Accumulator buffer for current token being formed
- yytextlen: Current length of text in yytextbuf
- yytextbufsize: Allocated size of yytextbuf (dynamically grows)
- yylvalstr (String): Final token text string created from yytextbuf
- literalbuf (StringBuilder): Special buffer for string literals and complex tokens
3. Position Tracking System
The lexer maintains precise position information for every token:
- yylineno: Current line number (1-based)
- yycolno: Current column number (0-based)
- offset: Absolute character offset from start of input
- yylineno_p, yycolno_p, offset_p: Previous position values for token start
4. Token Creation Workflow
1. Characters are read via get_char() from buf[] into yytextbuf[]
2. yylex() identifies token boundaries and type
3. Token text is extracted: yylvalstr = new String(yytextbuf, 0, yytextlen)
4. yylexwrap() creates TSourceToken with:
  - astext = yylvalstr (full token text copy)
  - lineNo = yylineno_p (start line)
  - columnNo = yycolno_p (start column)
  - offset = offset_p (absolute position)
5. Memory Management and Text Copying
Current Implementation (Eager Loading):
- Every token immediately copies its text from yytextbuf to TSourceToken.astext
- Original SQL text in yyline is discarded after processing each line
- No direct link maintained between token and original input position
6. Tracing Back to Original Position
Currently Possible:
- Token stores lineNo, columnNo, and offset
- These can theoretically locate position in original input
Current Limitations:
- Original input text is not retained after line processing
- yyline buffer is overwritten for each new line
- No mechanism to retrieve original text from position alone
Author:

Gudu Software

Field Summary

Fields
Modifier and Type	Field and Description
`static int`	`bconst`
`static int`	`COLUMN0_COUNT`
`static int`	`COLUMN1_FIRST_X`
`static int`	`COLUMN2_FIRST_Y`
`static int`	`COLUMN3_LAST_X`
`static int`	`COLUMN4_LAST_Y`
`static int`	`COLUMN5_FIRST_POS`
`static int`	`COLUMN6_LAST_POS`
`String`	`defaultDelimiterStr`
`char`	`delimiterchar`
`String`	`dolqstart`
`boolean`	`insideSingleQuoteStr`
`boolean`	`insqlpluscmd`
`static int`	`keyword_type_column`
`static int`	`keyword_type_identifier`
`static int`	`keyword_type_keyword`
`static int`	`keyword_type_reserved`
`static int`	`MAX_TOKEN_COLUMN_SIZE`
`static int`	`MAX_TOKEN_SIZE`
`boolean`	`stringLiteralStartWithUnicodeSingleQuote`
`String`	`tmpDelimiter`
`long[][]`	`TOKEN_TABLE`
`static int`	`UNICODE_ENCODE_ID`
`static int`	`xconst`
`BufferedReader`	`yyinput`

Constructor Summary

Constructors
Constructor and Description

TCustomLexer()

Constructors
Constructor and Description
`TCustomLexer()`

Method Summary

All Methods Static Methods Instance Methods Concrete Methods Deprecated Methods
Modifier and Type	Method and Description
`static EKeywordType`	`getKeywordType(String keyword, HashMap<String,Integer> keywordValueList, HashMap<Integer,Integer> keywordTypeList)` Deprecated. , please use keywordChecker.isKeyword() instead. because there are so many non-reserved keywords in some databases, it's not suitable to put those non-reserved keywords in lexer and parser.
`int`	`getkeywordvalue(String keyword)`
`String`	`getSqlCharset()`
`String`	`getStringByCode(int tokenCode)`
`boolean`	`isAtBeginOfLine()`
`protected boolean`	`isKeyword(int tokenCode)` Check if token code represents a keyword
`int`	`iskeyword(String str)`
`protected boolean`	`isSingleCharOperator(int tokenCode)` Check if token code represents a single character operator
`void`	`reset()`
`void`	`resetTokenTable()` Reset TOKEN_TABLE by only clearing entries that were used (incremental clear).
`void`	`setSqlCharset(String sqlCharset)`
`void`	`setTokenTableValue(TSourceToken token)`
`int`	`yylexwrap(TSourceToken psourcetoken)`

Methods inherited from class Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAX_TOKEN_SIZE
```
public static int MAX_TOKEN_SIZE
```

MAX_TOKEN_COLUMN_SIZE

public static int MAX_TOKEN_COLUMN_SIZE

COLUMN0_COUNT
```
public static int COLUMN0_COUNT
```

COLUMN1_FIRST_X
```
public static int COLUMN1_FIRST_X
```

COLUMN2_FIRST_Y
```
public static int COLUMN2_FIRST_Y
```

COLUMN3_LAST_X
```
public static int COLUMN3_LAST_X
```

COLUMN4_LAST_Y
```
public static int COLUMN4_LAST_Y
```

COLUMN5_FIRST_POS
```
public static int COLUMN5_FIRST_POS
```

COLUMN6_LAST_POS
```
public static int COLUMN6_LAST_POS
```

TOKEN_TABLE
```
public long[][] TOKEN_TABLE
```

yyinput
```
public BufferedReader yyinput
```

dolqstart
```
public String dolqstart
```

insqlpluscmd
```
public boolean insqlpluscmd
```

keyword_type_reserved

public static int keyword_type_reserved

keyword_type_keyword
```
public static int keyword_type_keyword
```

keyword_type_identifier

public static int keyword_type_identifier

keyword_type_column
```
public static int keyword_type_column
```

delimiterchar
```
public char delimiterchar
```

defaultDelimiterStr
```
public String defaultDelimiterStr
```

tmpDelimiter
```
public String tmpDelimiter
```

bconst
```
public static final int bconst
```
See Also:

Constant Field Values

xconst
```
public static final int xconst
```
See Also:

Constant Field Values

UNICODE_ENCODE_ID
```
public static final int UNICODE_ENCODE_ID
```
See Also:

Constant Field Values

insideSingleQuoteStr
```
public boolean insideSingleQuoteStr
```

stringLiteralStartWithUnicodeSingleQuote

public boolean stringLiteralStartWithUnicodeSingleQuote

Constructor Detail
- TCustomLexer
```
public TCustomLexer()
```

Method Detail

resetTokenTable
```
public void resetTokenTable()
```
Reset TOKEN_TABLE by only clearing entries that were used (incremental clear). This is O(usedTokenCount) instead of O(MAX_TOKEN_SIZE * MAX_TOKEN_COLUMN_SIZE). For typical SQL with ~100 distinct token types, this saves clearing ~20,000 entries.

setTokenTableValue

public void setTokenTableValue(TSourceToken token)

setSqlCharset

public void setSqlCharset(String sqlCharset)

getSqlCharset
```
public String getSqlCharset()
```

isSingleCharOperator
```
protected boolean isSingleCharOperator(int tokenCode)
```
Check if token code represents a single character operator

isKeyword
```
protected boolean isKeyword(int tokenCode)
```
Check if token code represents a keyword

iskeyword
```
public int iskeyword(String str)
```

isAtBeginOfLine
```
public boolean isAtBeginOfLine()
```

getStringByCode

public String getStringByCode(int tokenCode)

getkeywordvalue

public int getkeywordvalue(String keyword)

getKeywordType

public static EKeywordType getKeywordType(String keyword,
                                          HashMap<String,Integer> keywordValueList,
                                          HashMap<Integer,Integer> keywordTypeList)

Deprecated. , please use keywordChecker.isKeyword() instead. because there are so many non-reserved keywords in some databases, it's not suitable to put those non-reserved keywords in lexer and parser.

Parameters:: keyword -; keywordValueList -; keywordTypeList -
Returns:

reset
```
public void reset()
```

yylexwrap

public int yylexwrap(TSourceToken psourcetoken)

Class TCustomLexer

1. Input Management and Buffering

2. Token Text Formation Process

Key Variables in Token Text Storage:

3. Position Tracking System

4. Token Creation Workflow

5. Memory Management and Text Copying

6. Tracing Back to Original Position

Field Summary

Constructor Summary

Method Summary

Methods inherited from class Object

Field Detail

MAX_TOKEN_SIZE

MAX_TOKEN_COLUMN_SIZE

COLUMN0_COUNT

COLUMN1_FIRST_X

COLUMN2_FIRST_Y

COLUMN3_LAST_X

COLUMN4_LAST_Y

COLUMN5_FIRST_POS

COLUMN6_LAST_POS

TOKEN_TABLE

yyinput

dolqstart

insqlpluscmd

keyword_type_reserved

keyword_type_keyword

keyword_type_identifier

keyword_type_column

delimiterchar

defaultDelimiterStr

tmpDelimiter

bconst

xconst

UNICODE_ENCODE_ID

insideSingleQuoteStr

stringLiteralStartWithUnicodeSingleQuote

Constructor Detail

TCustomLexer

Method Detail

resetTokenTable

setTokenTableValue

setSqlCharset

getSqlCharset

isSingleCharOperator

isKeyword

iskeyword

isAtBeginOfLine

getStringByCode

getkeywordvalue

getKeywordType

reset

yylexwrap