https://github.com/andialbrecht/sqlparse/blob/9f44d54c07180b826a6276d3acf5e1458b507c3f/sqlparse/utils.py#L81-L90

FOO has a type of Name while $FOO has a type of Name.Placeholder. As FOO.BAR is parsed as an IdentifierList, I think $FOO.BAR should be parsed as IdentiferList as well.

Line 90 does not handle a tuple of Tokens such as (Token.Name, Token.Symbol) properly. In such a case, it would wrap t into a list, which looks like a bug to me.

I think this line should be written to wrap t into a list only if t is a single Token item. Otherwise, do not wrap it. For this intent, I'd suggest to change this line as follows: types = [t, ] if t and not isinstance(t[0], tuple) else t

Hey,

I just did my first test with a JOIN. At the end you see the "ON", which could look better with an "\n" infront. < snip > JOIN (SELECT hour(TIMESTAMP) AS HOUR, round(avg(value), 2) AS Home_Consumtion_P FROM history WHERE TIMESTAMP > §timestamp_begin§ AND TIMESTAMP < DATE_ADD(§timestamp_begin§,INTERVAL 1 DAY) AND DEVICE = @generator AND READING = @Home_Consumtion_P AND VALUE > 10 GROUP BY 1) X3 ON X1.HOUR = X2.HOUR AND X1.HOUR = X3.HOUR) X;

You did a graet job! Christian

This PR makes the Lexer a singleton class. This object carries the configured syntax as instance attributes. A library user who has non-standard syntax requirements is able to adapt the behavior of the Lexer to meet her needs. As an example for how to do this, please see the relevant test:

def test_configurable_syntax():
    sql = """select * from foo BACON SPAM EGGS;"""
    tokens = sqlparse.parse(sql)[0]

    assert list(
        (t.ttype, t.value) for t in tokens if t.ttype not in sqlparse.tokens.Whitespace
    ) == [
        (sqlparse.tokens.Keyword.DML, "select"),
        (sqlparse.tokens.Wildcard, "*"),
        (sqlparse.tokens.Keyword, "from"),
        (None, "foo BACON"),
        (None, "SPAM EGGS"),
        (sqlparse.tokens.Punctuation, ";"),
    ]

    Lexer().add_keywords(
        {
            "BACON": sqlparse.tokens.Name.Builtin,
            "SPAM": sqlparse.tokens.Keyword,
            "EGGS": sqlparse.tokens.Keyword,
        }
    )

    tokens = sqlparse.parse(sql)[0]

    assert list(
        (t.ttype, t.value) for t in tokens if t.ttype not in sqlparse.tokens.Whitespace
    ) == [
        (sqlparse.tokens.Keyword.DML, "select"),
        (sqlparse.tokens.Wildcard, "*"),
        (sqlparse.tokens.Keyword, "from"),
        (None, "foo"),
        (sqlparse.tokens.Name.Builtin, "BACON"),
        (sqlparse.tokens.Keyword, "SPAM"),
        (sqlparse.tokens.Keyword, "EGGS"),
        (sqlparse.tokens.Punctuation, ";"),
    ]
    # reset the syntax for later tests.
    Lexer().default_initialization()

The split does not work properly for the below Oracle Stored procedure

CREATE OR REPLACE EDITIONABLE PROCEDURE "OWB_ADMIN"."AZBJ_RI_LIMIT_PROC" AS week_date_first DATE := TRUNC (azbj_pme_api.opus_date - 7); week_date_last DATE := TRUNC (azbj_pme_api.opus_date - 1); BEGIN logtrace ('LOG', 10001, 'PROC AZBJ_RI_LIMIT_PROC STARTED', 'AZBJ_RI_LIMIT_PROC'); BEGIN EXECUTE IMMEDIATE ('DROP TABLE STG_RI_LIMIT_DATA PURGE'); EXCEPTION WHEN OTHERS THEN NULL; END; END;

The below one REPLACE PROCEDURE is split into multiple as in screenshot

REPLACE PROCEDURE WDP1POMD_OBJECT.TYPE_DO_DELETE( P_OBJECT_TYPE_CODE VARCHAR(1)) SQL SECURITY CREATOR #MAIN# : BEGIN DECLARE GV_PACKAGE_NAME VARCHAR(50) DEFAULT 'WDP1POMD_OBJECT_TYPE'; DECLARE gv_module_name VARCHAR(50); DECLARE EXIT HANDLER FOR SQLState '02000' RESIGNAL SQLSTATE 'T7665' SET MESSAGE_TEXT = '- 20011, Record not found.'; DECLARE EXIT HANDLER FOR SQLException BEGIN CALL WDP1POMD_UTIL.omd_fatal_log2(GV_PACKAGE_NAME || '.' || GV_MODULE_NAME, SQLCODE); RESIGNAL; END; SET GV_MODULE_NAME = 'DO_DELETE';

DELETE FROM WDP1TOMD.OMD_OBJECT_TYPE WHERE OBJECT_TYPE_CODE = P_OBJECT_TYPE_CODE; END;