pnathan / cl-linq Goto Github PK

Can recursive joins be implemented without ctes?

postmodern and cl-mssql both have the option to return alists.

grouping by alist key would be kiiind of nice.

Referencing #3, #6, #7.

Problem Number One: Columns are unlabelled.

As our dear forefathers figured out long ago, columns of data need labelling. However, if a 2D array or list of lists is passed in; there are no labels native to the data format. It is plausible that a "header row" ala CSV can be accepted and used. Reviewing cl-linq-tests.lisp, it's completely clear that indexes into data alone will not suffice.

Problem Number Two: Database systems emit alists.

This is a good thing - it's easy to debug - and a bad thing - cl-linq doesn't understand them.

Problem Number Three: This code is slow. Postgres can run multiple times quicker than this code on data sets under 1M rows. This may be a generally unsolvable problem when taking & giving out 2D lists/vectors. Most optimizations in comp sci demand preconditions on data (sorting, etc).

Solution Number One: Data Frame object.

This takes a page from R and would require that a query take either a 2D thingie (as now) or a data frame object. A dataframe object has not only data, but labelled columns and the capacity to add indexes to columns. An interesting exploration would be to allow row/column swaps to handle OLAP-style queries quickly.

Solution Number Two: Context through QUERY. Instead of using Query as a simple sql-esque starting point, add several options to it that would significantly enhance usefulness. E.g., &key Sort-before-use; &key list-of-columns.

Solution Number Three: Stop attempting to mimic SQL.
Instead of using SQLish approaches, decompose the system into components:
one function to GROUP-BY, one function to WHERE, one function to SELECT-COLUMNS, etc. This would diminish the opportunity for interprocedural optimization, but would provide a flexibility of end-use.

Overall Solution Requirements : Algorithmic revisitation/restrictions.

• Ensure that algorithms are optimal and unneeded elements are filtered out
• Conversion to vectors ASAP?
• Remove using Lists.

Partial issue: lists are real handy, but vectors are faster (yet have lousy sets of functions supporting them). Should generic infrastructure be built up to allow generic ad-hoc dispatch? this will slow things down.

Example:

(defmethod head ((object vector)) (aref object 0))
(defmethod head ((object list)) (car object))

this is silly. should be easy to fix in code.

desperately needs a having clause.

A query should be "order-by"-able.

Add ASDF integration & packaging.

Add implicit parallelization if lparallel is around.