TSourceToken (General SQL Parser Java Library Core 3.0.9.1 API)

Object
- gudusoft.gsqlparser.TSourceToken

All Implemented Interfaces:

Cloneable
```
public class TSourceToken
extends Object
implements Cloneable
```
Represents a source token which is the basic syntactic unit of SQL. A token can be a key word, an identifier, a quoted identifier, a literal (or constant), or a special character symbol. Tokens are normally separated by whitespace (space, tab, newline), but need not be if there is no ambiguity (which is generally only the case if a special character is adjacent to some other token type).
The parse tree node consists of source tokens.

A list of source token will be available after parse or tokenize the input SQL.
```
 

   TGSqlParser sqlparser = new TGSqlParser(EDbVendor.dbvoracle);
   sqlparser.sqltext = "select col from t";
   int ret = sqlparser.parse();
   if (ret == 0){
    for(int i=0;i<sqlparser.sourcetokenlist.size();i++){
      TSourceToken st =  sqlparser.sourcetokenlist.get(i);
      System.out.println(st.tokentype.toString()+" "+st.toString());
    }
   }else{
     System.out.println(sqlparser.getErrormessage());
   }

 
 
```
Get a list of source tokens after call the method TGSqlParser.parse() or just call TGSqlParser.tokenizeSqltext() if you only need to access tokens of input SQL without generating the full parse tree.

tokencode is the unique id represents the type of token, some typical tokens are: whitespace, return, keyword, identifier. This value is mainly used by the parser internally.
tokentype uniquely identify the token type in a more meaningful way. It's more easier to use this field in your program than tokencode.

Field Summary

Fields
Modifier and Type	Field and Description
`String`	`astext` Text representation for this token.
`long`	`columnNo` the column number of the first character in this token
`TSourceTokenList`	`container` Container for this token which is a list of source token, this is the reference to `TGSqlParser.sourcetokenlist`
`String`	`dolqstart` Start part of Dollar-quoted String Constants of PostgreSQL.
`long`	`lineNo` the line number of the first character in this token
`ESqlClause`	`location`
`long`	`offset` Token's offset from the beginning of the input query.
`int`	`posinlist` Index of this token in the `container`, start from 0
`int`	`prevTokenCode`
`TCustomSqlStatement`	`stmt` SQL statement that owns this token.
`int`	`tag` Space to save a value for temporary use
`int`	`tokencode` Unique id of this token used by parser internally.
`ETokenStatus`	`tokenstatus` Maintenance the status of this token during lex and parsing.
`ETokenType`	`tokentype` Uniquely identify the token type in a more meaningful way.

Constructor Summary

Constructors
Constructor and Description

TSourceToken()
Class constructor

TSourceToken(String s)
Class constructor, set a string value.

Constructors
Constructor and Description
`TSourceToken()` Class constructor
`TSourceToken(String s)` Class constructor, set a string value.

Method Summary

All Methods Static Methods Instance Methods Concrete Methods Deprecated Methods
Modifier and Type	Method and Description
`void`	`appendText(String text)`
`void`	`appendText(TSourceToken st)`
`TSourceToken`	`clone()`
`static void`	`concatInChain(TSourceToken st1, TSourceToken st2)`
`EDbObjectType`	`getDbObjectType()` Token in a `TObjectName` has the same database object type as the objectName.
`int`	`getDbObjType()` Token in a `TObjectName` has the same database object type as the objectName.
`EDbVendor`	`getDbvendor()` The database vendor which the SQL script includes this token will run against
`TSourceToken`	`getLinkToken()` Gets the linked token.
`TSourceToken`	`getNextTokenInChain()`
`Stack<TParseTreeNode>`	`getNodesEndWithThisToken()` A list of nodes whose end token is this token.
`Stack<TParseTreeNode>`	`getNodesStartFromThisToken()` A list of node whose start token is this token
`TSourceToken`	`getPrevTokenInChain()`
`String`	`getQuotedString()`
`int`	`getQuoteSymbolLength()`
`TSourceToken`	`getReplaceToken()` Used in sql formatter package only
`String`	`getTextWithoutQuoted()` String literal in SQL usually inside ``.
`TSourceTokenList`	`getTokensAfter()` Used in sql formatter package only.
`TSourceTokenList`	`getTokensBefore()` Used in sql formatter package only
`ETokenStatus`	`getTokenstatus()`
`void`	`insertANewTokenAfterMe(TSourceToken newToken)`
`void`	`insertANewTokenBeforeMe(TSourceToken newToken)`
`void`	`insertText(TSourceToken st)`
`boolean`	`isChangedInAsCanonical()`
`boolean`	`isFirstTokenOfLine()` Check to see if this token is the first token in a line of the input SQL
`boolean`	`isLastTokenOfLine()` Check to see if this token is the last token of in a line in the input SQL
`boolean`	`isnonsolidtoken()` Is this token a solid token or not.
`static boolean`	`isnonsolidtoken(ETokenType tokentype)` Space, return, comments are treated as non-solid token by default
`boolean`	`issolidtoken()` Is this token a non-solid token or not.
`TSourceToken`	`nextSolidToken()` The next token whose `tokentype` is not ttreturn,ttwhitespace,ttsimplecomment and ttbracketedcomment.
`TSourceToken`	`nextSolidToken(boolean treatCommentAsSolidToken)` The next token whose `tokentype` is not ttreturn,ttwhitespace,ttsimplecomment and ttbracketedcomment.
`TSourceToken`	`nextSolidToken(int pstep)`
`TSourceToken`	`nextToken()`
`TSourceToken`	`prevSolidToken()` The previous token whose `tokentype` is not ttreturn,ttwhitespace,ttsimplecomment and ttbracketedcomment.
`TSourceToken`	`prevSolidToken(boolean treatCommentAsSolidToken)` The previous token whose `tokentype` is not ttreturn,ttwhitespace,ttsimplecomment and ttbracketedcomment
`void`	`removeFromChain()`
`void`	`restoreText()`
`TSourceToken`	`searchToken(int targetTokenCode, int range)`
`TSourceToken`	`searchToken(int targetTokenCode, int range, int stopTokenCode, boolean stopAtSemiColon)` Search a token before or after this token in the same source token list.
`TSourceToken`	`searchToken(String targetTokenText, int range)`
`TSourceToken`	`searchToken(String targetTokenText, int range, int stopTokenCode, boolean stopAtSemiColon)` Search a token before or after this token in the same source token list.
`TSourceToken`	`searchTokenAfterObjectName()` Search the first non-solid token after the next objectName.
`TSourceToken`	`searchTokenAtTheEndOfSameLine()`
`void`	`setDbObjectType(EDbObjectType dbObjectType)` Set the database object type of this token
`void`	`setDbObjType(int dbObjType)` Deprecated. use `setDbObjectType(gudusoft.gsqlparser.EDbObjectType)` instead.
`void`	`setDbvendor(EDbVendor dbvendor)` The database vendor which the SQL script includes this token will run against
`void`	`setLinkToken(TSourceToken linkToken)` Create a link between two tokens.
`void`	`setNextTokenInChain(TSourceToken nextTokenInChain)`
`void`	`setPrevTokenInChain(TSourceToken prevTokenInChain)`
`void`	`setReplaceToken(TSourceToken replaceToken)` Used in sql formatter package only
`void`	`setString(String str)` set new string of this token
`void`	`setTextWithBackup(String newText)`
`void`	`setTokenstatus(ETokenStatus tokenstatus)`
`String`	`toScript()` The string text of this token
`String`	`toString()` The original string text for this token.
`String`	`toStringDebug()` String text with the debug information such as coordinate, token code, token type
`String`	`toUnQuotedString()` Deprecated. since 2.5.3.4
`void`	`updateNodeEndWithThisToken()`
`void`	`updateNodeStartWithThisToken()`

Methods inherited from class Object
equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Field Detail
  - location
```
public ESqlClause location
```
  - prevTokenCode
```
public int prevTokenCode
```
  - tokencode
```
public int tokencode
```
    Unique id of this token used by parser internally. check available value start from TBaseType.cmtslashstar
  - lineNo
```
public long lineNo
```
    the line number of the first character in this token
  - columnNo
```
public long columnNo
```
    the column number of the first character in this token
  - offset
```
public long offset
```
    Token's offset from the beginning of the input query.
```
     
    public void testOffset(){
        TGSqlParser sqlparser = new TGSqlParser(EDbVendor.dbvoracle);
        sqlparser.sqltext = "select f from t\n" +
                "where f>1\n";
        assertTrue(sqlparser.parse() == 0);
        for (int i=0;i<sqlparser.sourcetokenlist.size();i++){
            TSourceToken st = sqlparser.sourcetokenlist.get(i);
            String textFromOffset = sqlparser.sqltext.toString().substring((int)st.offset,(int)st.offset+st.toString().length());
            assertTrue(st.toString().equalsIgnoreCase(textFromOffset));
        }
    }
     
 
```
  - tokentype
```
public ETokenType tokentype
```
    Uniquely identify the token type in a more meaningful way. It's more easier to use this field in your program than tokencode. check available value in ETokenType
  - container
```
public TSourceTokenList container
```
    Container for this token which is a list of source token, this is the reference to TGSqlParser.sourcetokenlist
  - posinlist
```
public int posinlist
```
    Index of this token in the container, start from 0
```
     
    public void testPosinList(){
        TGSqlParser sqlparser = new TGSqlParser(EDbVendor.dbvoracle);
        sqlparser.sqltext = "select f from t\n" +
                "where f>1\n";
        assertTrue(sqlparser.parse() == 0);
        for (int i=0;i<sqlparser.sourcetokenlist.size();i++){
            assertTrue(i == sqlparser.sourcetokenlist.get(i).posinlist);
        }
    }
     
 
```
  - astext
```
public String astext
```
    Text representation for this token.
  - tokenstatus
```
public ETokenStatus tokenstatus
```
    Maintenance the status of this token during lex and parsing. Used by the parser internally.
  - dolqstart
```
public String dolqstart
```
    Start part of Dollar-quoted String Constants of PostgreSQL. While the standard syntax for specifying string constants is usually convenient, it can be difficult to understand when the desired string contains many single quotes or backslashes, since each of those must be doubled. To allow more readable queries in such situations, PostgreSQL provides another way, called "dollar quoting", to write string constants. A dollar-quoted string constant consists of a dollar sign ($), an optional "tag" of zero or more characters, another dollar sign, an arbitrary sequence of characters that makes up the string content, a dollar sign, the same tag that began this dollar quote, and a dollar sign.
    For example, here are two different ways to specify the string "Dianne's horse" using dollar quoting:
```
     $$Dianne's horse$$
     $SomeTag$Dianne's horse$SomeTag$
 
```
    This field will return $$ and $SomeTag$ accordingly.
  - stmt
```
public TCustomSqlStatement stmt
```
    SQL statement that owns this token.
  - tag
```
public int tag
```
    Space to save a value for temporary use
- Constructor Detail
  - TSourceToken
```
public TSourceToken()
```
    Class constructor
  - TSourceToken
```
public TSourceToken(String s)
```
    Class constructor, set a string value.
    
    Parameters:
    
    s - the string value this toke represent for.
- Method Detail
  - clone
```
public TSourceToken clone()
```
    Overrides:
    
    clone in class Object
  - concatInChain
```
public static void concatInChain(TSourceToken st1,
                                 TSourceToken st2)
```
  - insertANewTokenAfterMe
```
public void insertANewTokenAfterMe(TSourceToken newToken)
```
  - insertANewTokenBeforeMe
```
public void insertANewTokenBeforeMe(TSourceToken newToken)
```
  - updateNodeEndWithThisToken
```
public void updateNodeEndWithThisToken()
```
  - updateNodeStartWithThisToken
```
public void updateNodeStartWithThisToken()
```
  - removeFromChain
```
public void removeFromChain()
```
  - setPrevTokenInChain
```
public void setPrevTokenInChain(TSourceToken prevTokenInChain)
```
  - setNextTokenInChain
```
public void setNextTokenInChain(TSourceToken nextTokenInChain)
```
  - getPrevTokenInChain
```
public TSourceToken getPrevTokenInChain()
```
  - getNextTokenInChain
```
public TSourceToken getNextTokenInChain()
```
  - getTokenstatus
```
public ETokenStatus getTokenstatus()
```
  - getQuoteSymbolLength
```
public int getQuoteSymbolLength()
```
  - getQuotedString
```
public String getQuotedString()
```
  - getTextWithoutQuoted
```
public String getTextWithoutQuoted()
```
    String literal in SQL usually inside ``. Quoted identifier in SQL usually surrounded by [ and ], This method only returns the text inside those surroundings.
    
    Returns:
    
    text inside `` and []
  - toScript
```
public String toScript()
```
    The string text of this token
    
    Returns:
    
    the string text of this token
  - appendText
```
public void appendText(TSourceToken st)
```
  - appendText
```
public void appendText(String text)
```
  - insertText
```
public void insertText(TSourceToken st)
```
  - setTextWithBackup
```
public void setTextWithBackup(String newText)
```
  - isChangedInAsCanonical
```
public boolean isChangedInAsCanonical()
```
  - setTokenstatus
```
public void setTokenstatus(ETokenStatus tokenstatus)
```
  - restoreText
```
public void restoreText()
```
  - getNodesEndWithThisToken
```
public Stack<TParseTreeNode> getNodesEndWithThisToken()
```
    A list of nodes whose end token is this token.
    
    Returns:
    
    A list of nodes whose end token is this token.
  - getNodesStartFromThisToken
```
public Stack<TParseTreeNode> getNodesStartFromThisToken()
```
    A list of node whose start token is this token
    
    Returns:
    
    A list of node whose start token is this token
  - setDbObjType
```
public void setDbObjType(int dbObjType)
```
    Deprecated. use setDbObjectType(gudusoft.gsqlparser.EDbObjectType) instead.
    
    Parameters:
    
    dbObjType - the database object type
  - getDbObjType
```
public int getDbObjType()
```
    Token in a TObjectName has the same database object type as the objectName. Please use TObjectName.getDbObjectType() instead of this method if possible.
    
    Returns:
    
    the type of the database object
  - setDbObjectType
```
public void setDbObjectType(EDbObjectType dbObjectType)
```
    Set the database object type of this token
    
    Parameters:
    
    dbObjectType - database object type
  - getDbObjectType
```
public EDbObjectType getDbObjectType()
```
    Token in a TObjectName has the same database object type as the objectName. Please use TObjectName.getDbObjectType() instead of this method if possible.
    
    Returns:
    
    the type of the database object
  - setDbvendor
```
public void setDbvendor(EDbVendor dbvendor)
```
    The database vendor which the SQL script includes this token will run against
    
    Parameters:
    
    dbvendor - the database vendor such as Oracle, DB2 and so on.
  - getDbvendor
```
public EDbVendor getDbvendor()
```
    The database vendor which the SQL script includes this token will run against
    
    Returns:
    
    dbvendor the database vendor such as Oracle, DB2 and so on.
  - setString
```
public void setString(String str)
```
    set new string of this token
    
    Parameters:
    
    str - the new string text
  - toString
```
public String toString()
```
    The original string text for this token.
    
    Overrides:
    
    toString in class Object
    
    Returns:
    
    the string text
  - toStringDebug
```
public String toStringDebug()
```
    String text with the debug information such as coordinate, token code, token type
    
    Returns:
    
    the string value with full debug info
  - isnonsolidtoken
```
public static boolean isnonsolidtoken(ETokenType tokentype)
```
    Space, return, comments are treated as non-solid token by default
    
    Parameters:
    
    tokentype - token type
    
    Returns:
    
    true if token type is not one of ttwhitespace,ttreturn,ttsimplecomment,ttbracketedcomment
  - isnonsolidtoken
```
public boolean isnonsolidtoken()
```
    Is this token a solid token or not.
    
    Returns:
    
    true if it's a non-solid token.
  - issolidtoken
```
public boolean issolidtoken()
```
    Is this token a non-solid token or not.
    
    Returns:
    
    true if it's a solid token.
  - getTokensAfter
```
public TSourceTokenList getTokensAfter()
```
    Used in sql formatter package only.
    
    Returns:
    
    source token list
  - getTokensBefore
```
public TSourceTokenList getTokensBefore()
```
    Used in sql formatter package only
    
    Returns:
    
    source token list
  - setReplaceToken
```
public void setReplaceToken(TSourceToken replaceToken)
```
    Used in sql formatter package only
    
    Parameters:
    
    replaceToken - replaced token
  - getReplaceToken
```
public TSourceToken getReplaceToken()
```
    Used in sql formatter package only
    
    Returns:
    
    replaced token
  - nextToken
```
public TSourceToken nextToken()
```
  - searchTokenAtTheEndOfSameLine
```
public TSourceToken searchTokenAtTheEndOfSameLine()
```
  - searchToken
```
public TSourceToken searchToken(int targetTokenCode,
                                int range,
                                int stopTokenCode,
                                boolean stopAtSemiColon)
```
    Search a token before or after this token in the same source token list. The result token should has the tokencode equals to the targetTokenCode in a specified range.
    
    Parameters:
    
    targetTokenCode, - the token code need to be searched
    
    range, - > 0, search token start from the next token and forward, = 0, just compare with this token, < 0, search from the previous token and backword.
    
    Returns:
    
    the token with the same token code, otherwise, return null.
  - searchToken
```
public TSourceToken searchToken(int targetTokenCode,
                                int range)
```
  - searchToken
```
public TSourceToken searchToken(String targetTokenText,
                                int range,
                                int stopTokenCode,
                                boolean stopAtSemiColon)
```
    Search a token before or after this token in the same source token list. The result token should has the string text equals to the targetTokenText in a specified range.
    
    Parameters:
    
    targetTokenText, - the target string text
    
    range, - > 0, search token start from the next token and forward, = 0, just compare with this token, < 0, search from the previous token and backword.
    
    Returns:
    
    the token with the same token code, otherwise, return null.
  - searchToken
```
public TSourceToken searchToken(String targetTokenText,
                                int range)
```
  - searchTokenAfterObjectName
```
public TSourceToken searchTokenAfterObjectName()
```
    Search the first non-solid token after the next objectName.
    Take this SQL for example:
```
 return new scott.func(x1);
 
```
    If this token is new, then call searchTokenAfterObjectName will return ( token.
    Returns:
    
    solid token after the next objectName
  - nextSolidToken
```
public TSourceToken nextSolidToken(int pstep)
```
  - nextSolidToken
```
public TSourceToken nextSolidToken()
```
    The next token whose tokentype is not ttreturn,ttwhitespace,ttsimplecomment and ttbracketedcomment.
    
    Returns:
    
    the next solid token, returns null if not found.
  - nextSolidToken
```
public TSourceToken nextSolidToken(boolean treatCommentAsSolidToken)
```
    The next token whose tokentype is not ttreturn,ttwhitespace,ttsimplecomment and ttbracketedcomment.
    
    Parameters:
    
    treatCommentAsSolidToken, - set to true will treat comment token as a solid token
    
    Returns:
    
    the solid token if found, otherwise, return null
  - prevSolidToken
```
public TSourceToken prevSolidToken()
```
    The previous token whose tokentype is not ttreturn,ttwhitespace,ttsimplecomment and ttbracketedcomment.
    
    Returns:
    
    the solid token if found, otherwise, return null
  - prevSolidToken
```
public TSourceToken prevSolidToken(boolean treatCommentAsSolidToken)
```
    The previous token whose tokentype is not ttreturn,ttwhitespace,ttsimplecomment and ttbracketedcomment
    
    Parameters:
    
    treatCommentAsSolidToken - set to true will treat comment token as a solid token
    
    Returns:
    
    the solid token if found, otherwise, return null
  - isFirstTokenOfLine
```
public boolean isFirstTokenOfLine()
```
    Check to see if this token is the first token in a line of the input SQL
    
    Returns:
    
    true if this is the first token of a line, otherwise, return false
  - isLastTokenOfLine
```
public boolean isLastTokenOfLine()
```
    Check to see if this token is the last token of in a line in the input SQL
    
    Returns:
    
    true if this is the last token of a line, otherwise, return false
  - setLinkToken
```
public void setLinkToken(TSourceToken linkToken)
```
    Create a link between two tokens. Make it easy to access another linked token. Usually, those are two tokens like the parenthesis in this SQL: (select * from t)
    
    Parameters:
    
    linkToken - the token need to be linked
  - getLinkToken
```
public TSourceToken getLinkToken()
```
    Gets the linked token. Take this SQL for example: (select * from t), if this token is '(', then you call this method will return ')' token.
    
    Returns:
    
    the paired parenthesis in select, expression
  - toUnQuotedString
```
public String toUnQuotedString()
```
    Deprecated. since 2.5.3.4
    
    Remove double quote "", bracket quote [] , left/right brace {} from a delimited identifier and return string text of this identifier.
    
    Returns:
    
    string text of this token

Class TSourceToken

Field Summary

Constructor Summary

Method Summary

Methods inherited from class Object

Field Detail

location

prevTokenCode

tokencode

lineNo

columnNo

offset

tokentype

container

posinlist

astext

tokenstatus

dolqstart

stmt

tag

Constructor Detail

TSourceToken

TSourceToken

Method Detail

clone

concatInChain

insertANewTokenAfterMe

insertANewTokenBeforeMe

updateNodeEndWithThisToken

updateNodeStartWithThisToken

removeFromChain

setPrevTokenInChain

setNextTokenInChain

getPrevTokenInChain

getNextTokenInChain

getTokenstatus

getQuoteSymbolLength

getQuotedString

getTextWithoutQuoted

toScript

appendText

appendText

insertText

setTextWithBackup

isChangedInAsCanonical

setTokenstatus

restoreText

getNodesEndWithThisToken

getNodesStartFromThisToken

setDbObjType

getDbObjType

setDbObjectType

getDbObjectType

setDbvendor

getDbvendor

setString

toString

toStringDebug

isnonsolidtoken

isnonsolidtoken

issolidtoken

getTokensAfter

getTokensBefore

setReplaceToken

getReplaceToken

nextToken

searchTokenAtTheEndOfSameLine

searchToken

searchToken

searchToken

searchToken

searchTokenAfterObjectName

nextSolidToken

nextSolidToken

nextSolidToken

prevSolidToken

prevSolidToken

isFirstTokenOfLine

isLastTokenOfLine

setLinkToken