quarylabs / quary Goto Github PK

For middling data AWS Athena is an excellent low cost warehouse/lakehouse that works with open table formats like iceberg, hudi and delta lake.
Would definitely get us to try quarry if Athena was supported!

Whether this is a bug or multiple or not a bug is arguably subjective, but I wanted to tried to give quary a chance and didn't do so great due to multiple paper cuts and uncertainties, and I thought it might be useful to you as an outsider's perspective.

I have done a bunch of data work, but haven't used dbt before either, which might explain some of the issues or not.

TL;DR: I tried all three ways to try out quary, only the command line worked, and that can easily be made much better.

Current Behavior:

First blood

Initially I tried to just use the tool from the command line and rely on its internal documentation + https://github.com/quarylabs/quary/.

• First thing I tried after deciding to give it a shot is cargo install quary, and was surprised to find it not published.

• Next I started from the current repo, which offered installs either through brew or the vscode extension.

• I don't love vscode, so first I tried cargo build --release. That didn't work.

• I looked at the subcrates and guessed cargo install --path rust/cli would be it, that worked (after updating to latest stable rust).

• Not sure where to go from there, I tried quary --help.

  • "a very fast and useful database tool" doesn't tell me what it does. Very fast at deleting DBs? ;)

  • After trying a few different things, I came back and tried quary init, which filled my directory with what looks like a complete small project based on the template. This is pretty nice, actually, better than what I did before, which was to clone the template myself. The reason I didn't try this first is because the help doesn't explain that's what it does. In my experience most tools with an init, like git init, create a valid but empty project. The help entry of "Initialize a new project in the current directory" could have turned out to be either.

  • Once in the template project, I was not quite sure what to do.

    • compile? what even needs compilation?

      • after running this tells me it processed 10 models and 65 tests and completed successfully.
      • It does not seem to have affected my template directory at all, which makes me wonder where the outputs of compilation went.
      • I'm guessing that we are using an in memory sqlite DB for everything, therefore no trace is left of the compile, but I'm only guessing that because I peeked at the quary.yaml.
      • Will that be a problem later because everything we built is gone? or will later steps invoke compile anyway if needed, so I didn't need to run compile at all in the first place? No idea.

    • build? "build the seed and model views".

      • The seed seems to be csv files, what there needs building? maybe in other cases seeds are sometimes views?
      • The output is very terse, just progress bar that immediately goes to 29. No idea 29 what. Would be nicer if it looked more like the output to compile, at least when passed --verbose.

    • test? sounds good.

      • Have to admit I initially ran it without -s, which failed with an error that a table is missing, which was pretty opaque at that time.
      • After looking again at https://github.com/quarylabs/quary/, I saw the -s and that passed the tests.
      • I am now speculating that because we're using an in memory DB, the views are missing, so we need to "Rather than running the tests against the views,s run them against the source tables by building the model in CTEs".
      • Note that if I had read that help about -s before knowing about the in memory db thing, I would not have had an idea that that will solve my problem, this issue is not mentioned in the internal or repo help.
      • While I'm following the repo README, I see quary run, try it, and it is unrecognized.

So at this point, I think using the command line might work, I can't rely on internal documentation to get along. That's ok, gotta start somewhere. Maybe I should be using the vscode extension anyway?

VSCode...

• So I fired up VSCode on my machine, opened a project folder, got the recommendation to install the extension, agreed, tried to run some Quary commands and...

• Got a lot of "initializing extensions" that didn't go anywhere.

• Killed VSCode, brought it back up on the folder, tried again, and immediately got an option to "sign in to quary".

  • Wait, why do I need to sign in to Quary? Almost no extensions need a login. What is the sign in for exactly? what information will be sent to quary? Hmm, its not explained before the login prompt, not a great vibe, but let's try to find out.
  • https://github.com/quarylabs/template/tree/main doesn't mention the signin at all.
  • The extension documentation says: "Open the Sample Project: Access our template repository in GitHub's VSCode instance with no account required." so no explanation of what happens when one does sign in, which is a bit off putting.
  • Just looking at the command palette, it looks like there are a lot of options there, so maybe the command line version is going to be a pretty limited way of using quary?
  • https://www.quary.dev/ does not answer this :/
  • https://www.quary.dev/docs/documentation/installation doesn't mention an account being required either.
  • Ok... let's trust it. Tried to sign it, it sends me at a website I do not recognize, shows me a code and asks to confirm I see it in extension authenticator, but I see no such code in vscode at all. Eventually it times out.
  • I give up on that.

VSCode on web

• OK, this won't depend on specifics of my PC, let's try it!
• Click the link to https://github.dev/quarylabs/template/blob/main/models/shifts_summary.sql
• Get a github.dev workspace, looks promising, until I try to run a command, wherein sign in again doesn't work: now every time I try to run a command, the Welcome tab reopens blank.

Expected Behavior:

• cargo install quary works, and quary help is enough to get an idea how it works as a system.
• vscode extension installation works, and then commands immediately work. If a signin is required, I know what functionality that serves and what terms it commits me to.
• web demo... just works, unless microsoft is again messing with firefox users.

Steps To Reproduce:

See above

Environment:

• OS: PopOS which is approximately Ubuntu 22.04

Anything else:

I know this is a pre-release, so these are just the kinds of problems it is useful to catch. I don't know enough about your strategy to know whether using the open source part would be reasonable for me, but look forward to learning more.

hi I wonder if it's possible to run one model from CLI. it'll be an analog of dbt

dbt run --models model_name

thank you

I ran quary build and got a result as a view in postgres but it's not a materialized view. I checked quary/rust/core/src/database_postgres and it has

 fn automatic_cache_sql_create_statement(
        &self,
        model: &str,
        model_cache_name: &str,
    ) -> Vec<String> {
        vec![format!(
            "CREATE OR REPLACE VIEW {} AS SELECT * FROM {}",

in dbt I can add materialized: materialized_view to dbt_project.yml but don't see such an option in quary.
Which config I need to change to get a materialized view.
Thank you