Git Product home page Git Product logo

antlr-plsql's Introduction

antlr-plsql

Build Status PyPI version

Development

ANTLR requires Java, so we suggest you use Docker when building grammars. The Makefile contains directives to clean, build, test and deploy the ANTLR grammar. It does not run Docker itself, so run make inside Docker.

Build the grammar

# Build the docker container
docker build -t antlr_plsql .

# Run the container to build the python grammar
# Write parser files to local file system through volume mounting
docker run -it -v ${PWD}:/usr/src/app antlr_plsql make build

Set up the Python module

Now that the Python parsing files are available, you can install them with pip:

pip install -r requirements.txt
pip install -e .

And parse SQL code in Python:

from antlr_plsql import ast
ast.parse("SELECT a from b")

Using the AST viewer

If you're actively developing on the ANLTR grammar or the tree shaping, it's a good idea to set up the AST viewer locally so you can immediately see the impact of your changes in a visual way.

  • Clone the ast-viewer repo and build the Docker image according to the instructions.
  • Spin up a docker container that volume mounts the Python package, symlink-installs the package and runs the server on port 3000:
docker run -it \
  -u root \
  -v ~/workspace/antlr-plsql:/app/app/antlr-plsql \
  -p 3000:3000 \
  ast-viewer \
  /bin/bash -c "echo 'Install development requirements in development:' \
    && pip install --no-deps -e app/antlr-plsql \
    && python3 run.py"

When simultaneously developing other packages, volume mount and install those too:

docker run -it \
  -u root \
  -v ~/workspace/antlr-ast:/app/app/antlr-ast \
  -v ~/workspace/antlr-plsql:/app/app/antlr-plsql \
  -v ~/workspace/antlr-tsql:/app/app/antlr-tsql \
  -p 3000:3000 \
  ast-viewer \
  /bin/bash -c "echo 'Install development requirements in development:' \
    && pip install --no-deps -e app/antlr-ast \
    && pip install --no-deps -e app/antlr-plsql \
    && pip install --no-deps -e app/antlr-tsql \
    && python3 run.py"
  • If you update the tree shaping logic in this repo, the app will auto-update.
  • If you change the grammar, you will have to first rebuild the grammar (with the antlr_plsql docker image) and restart the ast-viewer container.

Run tests

# Similar to building the grammar, but running tests
# and not saving the generated files
docker build -t antlr_plsql .
docker run -t antlr_plsql make build test

Or run the test locally, first build the grammar then run:

pytest

Travis deployment

  • Builds the Docker image.
  • Runs the Docker image to build the grammar and run the unit tests.
  • Deploys the resulting python files to PyPi when a new release is made, so they can be installed easily.

antlr-plsql's People

Contributors

bogdanfloris avatar ddmkr avatar filipsch avatar hermansje avatar machow avatar timsangster avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

antlr-plsql's Issues

Difference in parsing WHERE if using parentheses and not

SELECT id FROM artists WHERE id > 100

image

versus

SELECT id FROM artists WHERE (id > 100)

image

Notice that the first image shows where_clause on the edge, while the second shows where_clause[0]. Not sure if there is ever a use-case for multiple WHERE's

ORDER BY parsing is incorrect

from datacamp/sqlwhat#44

edit:
UNION is treated as compound statement, so for two select statements, a trailing ORDER BY is the outermost piece. It's probably okay for now, but the ORDER BY will incorrectly be part of the final SELECT statement. Example below.

SELECT a FROM b
UNION
SELECT x FROM y
ORDER BY c

the postgres ast is

  [{"SelectStmt"=>
     {"sortClause"=>
       [{"SortBy"=>
          {"node"=>
            {"ColumnRef"=>
              {"fields"=>[{"String"=>{"str"=>"c"}}], "location"=>47}},
           "sortby_dir"=>0,
           "sortby_nulls"=>0,
           "location"=>-1}}],
      "op"=>1,
      "larg"=>
       {"SelectStmt"=>
         {"targetList"=>
           [{"ResTarget"=>
              {"val"=>
                {"ColumnRef"=>
                  {"fields"=>[{"String"=>{"str"=>"a"}}], "location"=>7}},
               "location"=>7}}],
          "fromClause"=>
           [{"RangeVar"=>
              {"relname"=>"b",
               "inhOpt"=>2,
               "relpersistence"=>"p",
               "location"=>14}}],
          "op"=>0}},
      "rarg"=>
       {"SelectStmt"=>
         {"targetList"=>
           [{"ResTarget"=>
              {"val"=>
                {"ColumnRef"=>
                  {"fields"=>[{"String"=>{"str"=>"x"}}], "location"=>29}},
               "location"=>29}}],
          "fromClause"=>
           [{"RangeVar"=>
              {"relname"=>"y",
               "inhOpt"=>2,
               "relpersistence"=>"p",
               "location"=>36}}],
          "op"=>0}}}}],

support for case when and subqueries

Currently code with CASE..WHEN or subqueries in them seem to produce Unshaped nodes.

Case When Example

SELECT name, continent, code, surface_area,
    CASE WHEN surface_area > 2000000 THEN 'large'
       WHEN surface_area > 350000 THEN 'medium'
       ELSE 'small' END AS geosize_group
FROM countries;

Subquery Example

SELECT local_name, subquery.lang_num
FROM countries,
  (SELECT code, COUNT(*) AS lang_num
   FROM languages
   GROUP BY code) subquery
WHERE countries.code = subquery.code;

cc @ismayc

Bad support for aliases inside join statements

Look at this example. The alias of the inner join is parsed properly, but the tree representation turns it into an Unshaped node while it should be an AliasExpr.

@machow The plsql.g4 file doesn't need changing for this, I think, but the ast.py file does. Is this correct? Do you have time to pair on this at some point?

testing deployments to PyPI

Yesterday, I updated antlr-plsql to create a Speaker instance from antlr_plsql/speaker.yml. However, because I didn't tell the package to contain the speaker.yml file, it did not include it in the PyPI release. This caused the library to raise an error on import.

Unfortunately, because the unit tests passed on every step leading up to release, and the exercise validator isn't testing courses-intro-sql yet, I didn't notice until people were testing course exercises.

It might not be worth the extra effort, but before deploying I can pin to courses-intro-sql to make sure the exercises pass.

grammar uses binary_expression instead of expression often

For example, standard_function often allows an argument to be concatenation (which is now just another name for binary_expression). E.g.

COUNT '(' ( '*' | (DISTINCT | UNIQUE | ALL)? concatenation) ')' over_clause?

However, the official postgres parser allows any expression, rather than just concatentation. E.g.

SELECT COUNT( a AND b) FROM x

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.