Comments (5)
I just see this now. I will ensure we will have a glob
flag upstream in Polars which users can toggle off.
from dagster.
It seems that polars is interpreting square brackets [ ]
as valid shell-glob syntax.
I would recommend that this pattern is not used for DynamicOutput
persistent object paths as it is inevitable that a third party package that uses glob would fail.
This is from the glob man page:
Character classes
An expression "[...]" where the first character after the leading
'[' is not an '!' matches a single character, namely any of the
characters enclosed by the brackets. The string enclosed by the
brackets cannot be empty; therefore ']' can be allowed between
the brackets, provided that it is the first character. (Thus,
"[][!]" matches the three characters '[', ']', and '!'.)
Ranges
There is one special convention: two characters separated by '-'
denote a range. (Thus, "[A-Fa-f0-9]" is equivalent to
"[ABCDEFabcdef0123456789]".) One may include '-' in its literal
meaning by making it the first or last character between the
brackets. (Thus, "[]-]" matches just the two characters ']' and
'-', and "[--0]" matches the three characters '-', '.', and '0',
since '/' cannot be matched.)
https://man7.org/linux/man-pages/man7/glob.7.html
Here is the core bug being tracked in polars
from dagster.
I just see this now. I will ensure we will have a
glob
flag upstream in Polars which users can toggle off.
@ritchie46
Although I agree that a glob
flag would be useful in polars, it seems that in the case of Dagster
, the convention of using square brackets is really the key issue here.
For instance, say an operation created as set out partitioned outputs with the pattern:
output/[partition_A]/[partition_B]/*.parquet
one would like the ability to read all files in lowest folder root.
Perhaps it should be considered invalid syntax to write files containing square brackets and reserve []
to act as correct glob functionality for reads
from dagster.
Compiled and tested against polars master branch. Can confirm the issue has been fixed.
It should be resolved with the latest polars release
from dagster.
Scrap that latest got released two hours ago.
from dagster.
Related Issues (20)
- New API authentication option for Airbyte OSS HOT 4
- Attempting to show unified asset health for assets with dimension of different lengths HOT 1
- `AutomationCondition` support for `dbt_assets` HOT 1
- Logs generated during evaluation of sensor not shown in UI if they originate from within a configurable resource
- DuckDB connection_config parameter not being recognized HOT 2
- retry_on_asset_or_op_failure = false is not working HOT 2
- AssetsDefinitions underlying ops does not support "detached" outs 1.8+
- `MarkdownMetadataValue` does not properly display new lines `<br />`
- Enable max run time tracking at Op level
- AzureBlobComputeLogManager fails on missing secret_key HOT 2
- Job execution not started with `Iteration is not allowed on @records` error on Dagster 1.8 HOT 10
- Please add pandera schema to list of allowed metadata values for v2
- [dagster-polars] Upstream partitionned Polars assets cannot be optional HOT 2
- [dagster-pandera / dagster-polars] Asset with Pandera Polars schema fails to be materialized with PolarsIOManager HOT 1
- Manual backfill of partitioned dbt job spawns infinite amount of runs
- Able to report partial materialisation of graph_asset
- clicking external asset causes 'Asset must be part of at least one job'
- Add timezone argument to AutomationCondition.on_cron()
- Dagster daemon logging does not recover from error situation
- Allow passing shell=True to Popen from PipesSubprocessClient
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dagster.