Comments (5)
deequ is only designed to handle one dataframe at the moment
adding the FK as a column to that dataframe with a left join before the processing and converting the hasForeignKey
into a isComplete("fk")
is a workaround i can think of with the current set of tools you provide
from deequ.
FK validation constraint might looks like:
// when one source/target column involved in the relation
.hasForeignKey(customers, "customerId" -> "id")
// when multiple source/target columns
.hasForeignKey(customers, ("customerId", "version") -> ("id", "version"))
Do you think about something like that ?
from deequ.
In general yes, the problem is that deequ is only designed to handle one dataframe at the moment, so its not totally obvious how to extend this to another one.
from deequ.
Closing this due to lack of interest.
from deequ.
any further progress on this? It is really good to have feature.
from deequ.
Related Issues (20)
- Compliance calculation result HOT 1
- numerical statistical indicators have lost precision
- [FEATURE] Supporing Aggregation metrics for a group
- [FEATURE] Filter condition is ignored when filtering records based on row-level checks HOT 5
- Anomaly checks when fails
- containsCreditCardNumber analyser constraint doesnt support for JCB credit card
- Performance impact when trying to generate profiling report for more than 200 columns HOT 2
- Is AggregateMatch type check supported in the library? HOT 1
- [FEATURE] Cross-building via Mill HOT 5
- How to use Deequ to implement a custom return result set and return the correct and incorrect number of each check result
- Java null pointer issue , while creating sparksession , with deequ jar
- [BUG] Spark 3.4 and Deequ breeze version conflict HOT 1
- [FEATURE] Can we enhance `VerificationSuite` to supports more than one Dataframe?
- Custom user analyzers
- Support for Custom SQL Execution in Deequ Library
- Question: DQ over time
- [FEATURE] Extend RatioOfSums to support other aggregations
- [FEATURE] Support Wilson Score Interval for RetainCompletenessRule
- [BUG] Row-level filtering marking the records as pass when null values are present in the column
- Why is `Distance` not an analyzer?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deequ.