Git Product home page Git Product logo

sample-sql-translator's Introduction

SQL Translator

Overview

This package is a general-purpose SQL parser, translator, analyzer, and formatter.

Note: This is not an officially supported Google product.

It is intended to -

  • Provide an example of a comprehensive recursive descent parser using SQL.
  • Provide a template for SQL parser, rewriting, analysis, and formatting.

The dialect of SQL primarily based on BigQuery Standard SQL, with some additional features.

Why a recursive descent parser?

After many attempts at using LL and LR parsers over many years -- the common tools used today -- I've switched to using exclusively hand-crafted recursive descent parsers.

Some good background of the challenges of parser/lexer tools can be found in this blog post.

It is also noteworthy that GCC and clang have both switched to hand-crafted recursive descent paresrs. See gcc and clang.

This provides an example of a complex parser that, I hope, is easy to read, maintain, and extend. The only state is maintained in lexer.py that provides the fundamental methods for finding identifiers, strings, and numbers.

Potential Use Cases

There are many potential use cases for extending this code:

  • Providing a formatter tool as part of continuous integration.
  • Analyzing code for table dependencies and other interesting information. Search the AST for any information you'd like.
  • Translating SQL -- possibly custom SQL dialect -- into different target dialects.

Note that in all cases it will take extending the parser and formatting as needed. This is not officially supported Google product.

Running examples

Please see examples.

Components

sql_parser

sql_parser is a pure recursive-descent parser that generates Python frozen dataclass objects as an AST.

The AST also contains methods to generate SQL formatted by rfmt.

The data used:

  • Frozen Python Dataclass nodes as the abstract syntax tree.
  • Lexer that contains the string, list of comments (for association), and current position.

The only two variables modified is the parsing position and list of comments. The rest of the parsing state is the current call-stack and execution point.

sql_rewriter

sql_rewriter contains examples of of re-written the AST for different dialects.

Some examples include:

  • Converting CASE statements into IF() functions.
  • Converting colon cast to CAST() functions.

It also includes an extraction algorithm for detecting dependency between tables, and a graph minimisation algorithm for producing a graphviz comaptible graph.

rfmt

rfmt is copied from https://github.com/google/rfmt.

This is a general purpose formatting library. More information on its approach can be found in the research paper.

sample-sql-translator's People

Contributors

mescanne avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.