devrel-project-demos's People
devrel-project-demos's Issues
Error running motherduck-dagster-hybrid-compute pipeline
I've cloned the repository to wsl2 on my windows machine, I created a .env file containing the following:
# MotherDuck / DuckDb Connection String
#
# Can either be a simple <filename>.duckdb, or a connection string to MotherDuck.
#
# References:
# - https://motherduck.com/docs/authenticating-to-motherduck/#authentication-using-a-service-token
#
DUCKDB_DATABASE="local.duckdb"
# Evidence Connection Variables
#
# References
# - https://motherduck.com/docs/integrations/evidence/
# - https://docs.evidence.dev/cli#environment-variables
#
EVIDENCE_SOURCE__DAGSTER_HYBRID_DEMO__TYPE="duckdb"
EVIDENCE_SOURCE__DAGSTER_HYBRID_DEMO__FILENAME="local.duckdb"
I ran make install
and then make dev
, after which I clicked materialize all in the dagster webpage and get the following error:
Invalid Input Error: Python Object "birds" of type "AssetsDefinition" found on line "/home/maxisq/dev/dagster-project-demos/motherduck-dagster-hybrid-compute/motherduck_dagster_hybrid/assets.py:145" not suitable for replacement scans.
Make sure that "birds" is either a pandas.DataFrame, duckdb.DuckDBPyRelation, pyarrow Table, Dataset, RecordBatchReader, Scanner, or NumPy ndarrays with supported format
Line 145 in assets.py doesn't appear to mention a replacement scan at all and googling "dagster AssetDefinition type not suitable for replacement scan error" seems to show not relevant results.
Can anyone advise on how I can fix this error?
[motherduck-demo] it's not clear how a user could configure this project to run locally
Description
There are two configuration files that have to be modified to allow this project to run locally (./.env
and ./reports/sources/dagster_hybrid_demo/connection.yaml
), and this isn't very clear to the user.
Firstly, setting DUCKDB_DATABASE="local.duckdb"
as is explained in the .env.example
file will result in the dbt assets failing to materialize as I believe dbt assumes this path is relative to its runtime directory, meaning it can't locate the database. Instead, it's mandatory to use the absolute path in the .env
file like so: DUCKDB_DATABASE="/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/local.duckdb"
.
Secondly, the EVIDENCE_SOURCE__DAGSTER_HYBRID_DEMO__FILENAME
environment variable isn't referenced in the evidence project files isn't respected by evidence meaning it will fail to find the local database and dagster will fail to materialize the evidence assets. Because of this it's necessary to modify the connection.yaml
file in the evidence sources directory, where you unintuitively have to use a relative path, using an absolute paths will result in evidence modifying the path provided at run time which causes the same error.
Proposed Solution
Firstly we can change the instructions in .env.example
to properly indicate how to configure the project for local use, using absolute paths in the .env
file.
Secondly we can either find a way to configure evidence such that it reads from the EVIDENCE_SOURCE__DAGSTER_HYBRID_DEMO__FILENAME
environment variable properly, or give users instructions to modify the connection.yaml
file to point to the local duckdb file using a relative path.
[motherduck-demo] Error when evaluating SSR module ReferenceError: window is not defined
Appearance
"500 | Window is not defined" when starting evidence web server.
Reproduction Steps
Clone repository, cd
into motherduck demo, run make evidence-install
and make evidence
as described in the Makefile to start the evidence web server in developer mode.
Logs
(venv) maxisq@MS-V-Desktop:~/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute$ make evidence
npm --prefix ./reports run dev -- --port 4000
> [email protected] dev
> evidence dev --open / --port 4000
Starting Evidence development workspace - this can take up to 90 seconds
[!] Unable to load source manifest
This likely means you have no source data, and need to generate it.
Running npm run sources will generate the needed data. See npm run sources --help for more usage information
Documentation: https://docs.evidence.dev/core-concepts/data-sources/
Cleared query cache
Forced re-optimization of dependencies
VITE v5.1.6 ready in 1050 ms
➜ Local: http://localhost:4000/
➜ Network: use --host to expose
➜ press h + enter to show help
Unable to load manifest, do you need to generate sources?
3:22:06 PM [vite] Error when evaluating SSR module /@fs/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/@evidence-dev/core-components/dist/unsorted/ui/QueryViewerSupport/QueryDataTable.svelte: failed to import "echarts"
|- ReferenceError: window is not defined
at detect (/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:123:54)
at /home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:97:9
at /home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:22:68
at Object.<anonymous> (/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:25:2)
at Module._compile (node:internal/modules/cjs/loader:1368:14)
at Module._extensions..js (node:internal/modules/cjs/loader:1426:10)
at Module.load (node:internal/modules/cjs/loader:1205:32)
at Module._load (node:internal/modules/cjs/loader:1021:12)
at cjsLoader (node:internal/modules/esm/translators:366:17)
at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:315:7)
3:22:06 PM [vite] Error when evaluating SSR module /@fs/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/@evidence-dev/core-components/dist/unsorted/ui/QueryViewer.svelte: failed to import "/@fs/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/@evidence-dev/core-components/dist/unsorted/ui/QueryViewerSupport/QueryDataTable.svelte"
|- ReferenceError: window is not defined
at detect (/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:123:54)
at /home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:97:9
at /home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:22:68
at Object.<anonymous> (/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:25:2)
at Module._compile (node:internal/modules/cjs/loader:1368:14)
at Module._extensions..js (node:internal/modules/cjs/loader:1426:10)
at Module.load (node:internal/modules/cjs/loader:1205:32)
at Module._load (node:internal/modules/cjs/loader:1021:12)
at cjsLoader (node:internal/modules/esm/translators:366:17)
at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:315:7)
3:22:06 PM [vite] Error when evaluating SSR module /@fs/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/@evidence-dev/core-components/dist/unsorted/ui/index.js: failed to import "/@fs/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/@evidence-dev/core-components/dist/unsorted/ui/QueryViewer.svelte"
|- ReferenceError: window is not defined
at detect (/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:123:54)
at /home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:97:9
at /home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:22:68
at Object.<anonymous> (/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:25:2)
at Module._compile (node:internal/modules/cjs/loader:1368:14)
at Module._extensions..js (node:internal/modules/cjs/loader:1426:10)
at Module.load (node:internal/modules/cjs/loader:1205:32)
at Module._load (node:internal/modules/cjs/loader:1021:12)
at cjsLoader (node:internal/modules/esm/translators:366:17)
at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:315:7)
3:22:06 PM [vite] Error when evaluating SSR module /@fs/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/@evidence-dev/core-components/dist/unsorted/index.js: failed to import "/@fs/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/@evidence-dev/core-components/dist/unsorted/ui/index.js"
|- ReferenceError: window is not defined
at detect (/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:123:54)
at /home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:97:9
at /home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:22:68
at Object.<anonymous> (/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:25:2)
at Module._compile (node:internal/modules/cjs/loader:1368:14)
at Module._extensions..js (node:internal/modules/cjs/loader:1426:10)
at Module.load (node:internal/modules/cjs/loader:1205:32)
at Module._load (node:internal/modules/cjs/loader:1021:12)
at cjsLoader (node:internal/modules/esm/translators:366:17)
at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:315:7)
3:22:06 PM [vite] Error when evaluating SSR module /@fs/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/@evidence-dev/core-components/dist/index.js: failed to import "/@fs/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/@evidence-dev/core-components/dist/unsorted/index.js"
|- ReferenceError: window is not defined
at detect (/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:123:54)
at /home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:97:9
at /home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:22:68
at Object.<anonymous> (/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:25:2)
at Module._compile (node:internal/modules/cjs/loader:1368:14)
at Module._extensions..js (node:internal/modules/cjs/loader:1426:10)
at Module.load (node:internal/modules/cjs/loader:1205:32)
at Module._load (node:internal/modules/cjs/loader:1021:12)
at cjsLoader (node:internal/modules/esm/translators:366:17)
at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:315:7)
3:22:06 PM [vite] Error when evaluating SSR module /src/pages/+layout.svelte: failed to import "/@fs/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/@evidence-dev/core-components/dist/index.js"
|- ReferenceError: window is not defined
at detect (/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:123:54)
at /home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:97:9
at /home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:22:68
at Object.<anonymous> (/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:25:2)
at Module._compile (node:internal/modules/cjs/loader:1368:14)
at Module._extensions..js (node:internal/modules/cjs/loader:1426:10)
at Module.load (node:internal/modules/cjs/loader:1205:32)
at Module._load (node:internal/modules/cjs/loader:1021:12)
at cjsLoader (node:internal/modules/esm/translators:366:17)
at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:315:7)
3:22:06 PM [vite] Error when evaluating SSR module /src/pages/+error.svelte: failed to import "/@fs/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/@evidence-dev/core-components/dist/index.js"
|- ReferenceError: window is not defined
at detect (/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:123:54)
at /home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:97:9
at /home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:22:68
at Object.<anonymous> (/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:25:2)
at Module._compile (node:internal/modules/cjs/loader:1368:14)
at Module._extensions..js (node:internal/modules/cjs/loader:1426:10)
at Module.load (node:internal/modules/cjs/loader:1205:32)
at Module._load (node:internal/modules/cjs/loader:1021:12)
at cjsLoader (node:internal/modules/esm/translators:366:17)
at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:315:7)
^Cmake: *** [Makefile:30: evidence] Interrupt
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.