Package gudusoft.gsqlparser

Class TSourceToken

Object
gudusoft.gsqlparser.TSourceToken
All Implemented Interfaces:
Cloneable
public class TSourceToken extends Object implements Cloneable
Represents a source token which is the basic syntactic unit of SQL. A token can be a key word, an identifier, a quoted identifier, a literal (or constant), or a special character symbol. Tokens are normally separated by whitespace (space, tab, newline), but need not be if there is no ambiguity (which is generally only the case if a special character is adjacent to some other token type).

The parse tree node consists of source tokens.


A list of source token will be available after parse or tokenize the input SQL. 

 

   TGSqlParser sqlparser = new TGSqlParser(EDbVendor.dbvoracle);
   sqlparser.sqltext = "select col from t";
   int ret = sqlparser.parse();
   if (ret == 0){
    for(int i=0;i<sqlparser.sourcetokenlist.size();i++){
      TSourceToken st =  sqlparser.sourcetokenlist.get(i);
      System.out.println(st.tokentype.toString()+" "+st.toString());
    }
   }else{
     System.out.println(sqlparser.getErrormessage());
   }
Get a list of source tokens after call the method TGSqlParser.parse() or just call TGSqlParser.tokenizeSqltext() if you only need to access tokens of input SQL without generating the full parse tree.

tokencode is the unique id represents the type of token, some typical tokens are: whitespace, return, keyword, identifier. This value is mainly used by the parser internally.

tokentype uniquely identify the token type in a more meaningful way. It's more easier to use this field in your program than tokencode.

  • Field Details

    • location

      public ESqlClause location

    • prevTokenCode

      public int prevTokenCode

    • tokencode

      public int tokencode
      Unique id of this token used by parser internally. check available value start from TBaseType.cmtslashstar

    • lineNo

      public long lineNo
      the line number of the first character in this token

    • columnNo

      public long columnNo
      the column number of the first character in this token

    • offset

      public long offset
      Token's offset from the beginning of the input query. 
           
    public void testOffset(){
        TGSqlParser sqlparser = new TGSqlParser(EDbVendor.dbvoracle);
        sqlparser.sqltext = "select f from t\n" +
                "where f>1\n";
        assertTrue(sqlparser.parse() == 0);
        for (int i=0;i<sqlparser.sourcetokenlist.size();i++){
            TSourceToken st = sqlparser.sourcetokenlist.get(i);
            String textFromOffset = sqlparser.sqltext.toString().substring((int)st.offset,(int)st.offset+st.toString().length());
            assertTrue(st.toString().equalsIgnoreCase(textFromOffset));
        }
    }

    • tokentype

      public ETokenType tokentype
      Uniquely identify the token type in a more meaningful way. It's more easier to use this field in your program than tokencode. check available value in ETokenType

    • container

      public TSourceTokenList container
      Container for this token which is a list of source token, this is the reference to TGSqlParser.sourcetokenlist

    • posinlist

      public int posinlist
      Index of this token in the container, start from 0 
           
    public void testPosinList(){
        TGSqlParser sqlparser = new TGSqlParser(EDbVendor.dbvoracle);
        sqlparser.sqltext = "select f from t\n" +
                "where f>1\n";
        assertTrue(sqlparser.parse() == 0);
        for (int i=0;i<sqlparser.sourcetokenlist.size();i++){
            assertTrue(i == sqlparser.sourcetokenlist.get(i).posinlist);
        }
    }

    • astext

      public String astext
      The text content of this token.

      This public field is maintained for backwards compatibility with existing code. New code should use getter/setter methods instead of direct field access.

      Recommended Usage:

      Since:
      1.0 (public field maintained for backwards compatibility)
      See Also:

    • tokenstatus

      public ETokenStatus tokenstatus
      Maintenance the status of this token during lex and parsing. Used by the parser internally.

    • dolqstart

      public String dolqstart
      Start part of Dollar-quoted String Constants of PostgreSQL. While the standard syntax for specifying string constants is usually convenient, it can be difficult to understand when the desired string contains many single quotes or backslashes, since each of those must be doubled. To allow more readable queries in such situations, PostgreSQL provides another way, called "dollar quoting", to write string constants. A dollar-quoted string constant consists of a dollar sign ($), an optional "tag" of zero or more characters, another dollar sign, an arbitrary sequence of characters that makes up the string content, a dollar sign, the same tag that began this dollar quote, and a dollar sign.

      For example, here are two different ways to specify the string "Dianne's horse" using dollar quoting: 
           $$Dianne's horse$$
     $SomeTag$Dianne's horse$SomeTag$
      This field will return $$ and $SomeTag$ accordingly.

    • insqlpluscmd

      public boolean insqlpluscmd

    • stmt

      public TCustomSqlStatement stmt
      SQL statement that owns this token.

    • tag

      public int tag
      Space to save a value for temporary use

  • Constructor Details

    • TSourceToken

      public TSourceToken()
      Class constructor

    • TSourceToken

      public TSourceToken(String s)
      Class constructor, set a string value.
      Parameters:
      s - the string value this toke represent for.

  • Method Details

    • clone

      public TSourceToken clone()
      Overrides:
      clone in class Object

    • concatInChain

      public static void concatInChain(TSourceToken st1, TSourceToken st2)

    • insertANewTokenAfterMe

      public void insertANewTokenAfterMe(TSourceToken newToken)

    • insertANewTokenBeforeMe

      public void insertANewTokenBeforeMe(TSourceToken newToken)

    • updateNodeEndWithThisToken

      public void updateNodeEndWithThisToken()

    • updateNodeStartWithThisToken

      public void updateNodeStartWithThisToken()

    • removeFromChain

      public void removeFromChain()

    • setPrevTokenInChain

      public void setPrevTokenInChain(TSourceToken prevTokenInChain)

    • setNextTokenInChain

      public void setNextTokenInChain(TSourceToken nextTokenInChain)

    • getPrevTokenInChain

      public TSourceToken getPrevTokenInChain()

    • getNextTokenInChain

      public TSourceToken getNextTokenInChain()

    • getTokenstatus

      public ETokenStatus getTokenstatus()

    • getQuoteSymbolLength

      public int getQuoteSymbolLength()

    • getQuotedString

      public String getQuotedString()

    • getTextWithoutQuoted

      public String getTextWithoutQuoted()
      String literal in SQL usually inside ``. Quoted identifier in SQL usually surrounded by [ and ], This method only returns the text inside those surroundings.
      Returns:
      text inside `` and []

    • toScript

      public String toScript()
      The string text of this token
      Returns:
      the string text of this token

    • appendText

      public void appendText(TSourceToken st)

    • appendText

      public void appendText(String text)

    • insertText

      public void insertText(TSourceToken st)

    • setTextWithBackup

      public void setTextWithBackup(String newText)

    • isChangedInAsCanonical

      public boolean isChangedInAsCanonical()

    • setTokenstatus

      public void setTokenstatus(ETokenStatus tokenstatus)

    • restoreText

      public void restoreText()

    • getNodesEndWithThisToken

      public Stack<TParseTreeNode> getNodesEndWithThisToken()
      A list of nodes whose end token is this token.
      Returns:
      A list of nodes whose end token is this token.

    • getNodesStartFromThisToken

      public Stack<TParseTreeNode> getNodesStartFromThisToken()
      A list of node whose start token is this token
      Returns:
      A list of node whose start token is this token

    • setDbObjType

      public void setDbObjType(int dbObjType)
      Deprecated.
      use setDbObjectType(gudusoft.gsqlparser.EDbObjectType) instead.
      Parameters:
      dbObjType - the database object type

    • getDbObjType

      public int getDbObjType()
      Token in a TObjectName has the same database object type as the objectName. Please use TObjectName.getDbObjectType() instead of this method if possible.
      Returns:
      the type of the database object

    • setDbObjectType

      public void setDbObjectType(EDbObjectType dbObjectType)
      Set the database object type of this token
      Parameters:
      dbObjectType - database object type

    • getDbObjectType

      public EDbObjectType getDbObjectType()
      Token in a TObjectName has the same database object type as the objectName. Please use TObjectName.getDbObjectType() instead of this method if possible.
      Returns:
      the type of the database object

    • setDbvendor

      public void setDbvendor(EDbVendor dbvendor)
      The database vendor which the SQL script includes this token will run against
      Parameters:
      dbvendor - the database vendor such as Oracle, DB2 and so on.

    • getDbvendor

      public EDbVendor getDbvendor()
      The database vendor which the SQL script includes this token will run against
      Returns:
      dbvendor the database vendor such as Oracle, DB2 and so on.

    • setString

      public void setString(String str)
      set new string of this token
      Parameters:
      str - the new string text

    • toString

      public String toString()
      The original string text for this token.
      Overrides:
      toString in class Object
      Returns:
      the string text

    • toStringDebug

      public String toStringDebug()
      String text with the debug information such as coordinate, token code, token type
      Returns:
      the string value with full debug info

    • isnonsolidtoken

      public static boolean isnonsolidtoken(ETokenType tokentype)
      Space, return, comments are treated as non-solid token by default
      Parameters:
      tokentype - token type
      Returns:
      true if token type is not one of ttwhitespace,ttreturn,ttsimplecomment,ttbracketedcomment

    • isnonsolidtoken

      public boolean isnonsolidtoken()
      Is this token a solid token or not.
      Returns:
      true if it's a non-solid token.

    • issolidtoken

      public boolean issolidtoken()
      Is this token a non-solid token or not.
      Returns:
      true if it's a solid token.

    • getTokensAfter

      public TSourceTokenList getTokensAfter()
      Used in sql formatter package only.
      Returns:
      source token list

    • getTokensBefore

      public TSourceTokenList getTokensBefore()
      Used in sql formatter package only
      Returns:
      source token list

    • setReplaceToken

      public void setReplaceToken(TSourceToken replaceToken)
      Used in sql formatter package only
      Parameters:
      replaceToken - replaced token

    • getReplaceToken

      public TSourceToken getReplaceToken()
      Used in sql formatter package only
      Returns:
      replaced token

    • nextToken

      public TSourceToken nextToken()

    • searchTokenAtTheEndOfSameLine

      public TSourceToken searchTokenAtTheEndOfSameLine()

    • searchToken

      public TSourceToken searchToken(int targetTokenCode, int range, int stopTokenCode, boolean stopAtSemiColon)
      Search a token before or after this token in the same source token list. The result token should has the tokencode equals to the targetTokenCode in a specified range.
      Parameters:
      targetTokenCode - , the token code need to be searched
      range - , > 0, search token start from the next token and forward, = 0, just compare with this token, < 0, search from the previous token and backword.
      Returns:
      the token with the same token code, otherwise, return null.

    • searchToken

      public TSourceToken searchToken(int targetTokenCode, int range)

    • searchToken

      public TSourceToken searchToken(String targetTokenText, int range, int stopTokenCode, boolean stopAtSemiColon)
      Search a token before or after this token in the same source token list. The result token should has the string text equals to the targetTokenText in a specified range.
      Parameters:
      targetTokenText - , the target string text
      range - , > 0, search token start from the next token and forward, = 0, just compare with this token, < 0, search from the previous token and backword.
      Returns:
      the token with the same token code, otherwise, return null.

    • searchToken

      public TSourceToken searchToken(String targetTokenText, int range)

    • searchTokenAfterObjectName

      public TSourceToken searchTokenAfterObjectName()
      Search the first non-solid token after the next objectName.

      Take this SQL for example: 
       return new scott.func(x1);
      If this token is new, then call searchTokenAfterObjectName will return ( token.
      Returns:
      solid token after the next objectName

    • nextSolidToken

      public TSourceToken nextSolidToken(int pstep)

    • nextSolidToken

      public TSourceToken nextSolidToken()
      The next token whose tokentype is not ttreturn,ttwhitespace,ttsimplecomment and ttbracketedcomment.
      Returns:
      the next solid token, returns null if not found.

    • nextSolidToken

      public TSourceToken nextSolidToken(boolean treatCommentAsSolidToken)
      The next token whose tokentype is not ttreturn,ttwhitespace,ttsimplecomment and ttbracketedcomment.
      Parameters:
      treatCommentAsSolidToken - , set to true will treat comment token as a solid token
      Returns:
      the solid token if found, otherwise, return null

    • prevSolidToken

      public TSourceToken prevSolidToken()
      The previous token whose tokentype is not ttreturn,ttwhitespace,ttsimplecomment and ttbracketedcomment.
      Returns:
      the solid token if found, otherwise, return null

    • prevSolidToken

      public TSourceToken prevSolidToken(boolean treatCommentAsSolidToken)
      The previous token whose tokentype is not ttreturn,ttwhitespace,ttsimplecomment and ttbracketedcomment
      Parameters:
      treatCommentAsSolidToken - set to true will treat comment token as a solid token
      Returns:
      the solid token if found, otherwise, return null

    • isFirstTokenOfLine

      public boolean isFirstTokenOfLine()
      Check to see if this token is the first token in a line of the input SQL
      Returns:
      true if this is the first token of a line, otherwise, return false

    • isLastTokenOfLine

      public boolean isLastTokenOfLine()
      Check to see if this token is the last token of in a line in the input SQL
      Returns:
      true if this is the last token of a line, otherwise, return false

    • setLinkToken

      public void setLinkToken(TSourceToken linkToken)
      Create a link between two tokens. Make it easy to access another linked token. Usually, those are two tokens like the parenthesis in this SQL: (select * from t)
      Parameters:
      linkToken - the token need to be linked

    • getLinkToken

      public TSourceToken getLinkToken()
      Gets the linked token. Take this SQL for example: (select * from t), if this token is '(', then you call this method will return ')' token.
      Returns:
      the paired parenthesis in select, expression

    • toUnQuotedString

      public String toUnQuotedString()
      Deprecated.
      since 2.5.3.4
      Remove double quote "", bracket quote [] , left/right brace {} from a delimited identifier and return string text of this identifier.
      Returns:
      string text of this token

    • getAstext

      public String getAstext()
      Text representation for this token.

    • setAstext

      public void setAstext(String astext)