Comments (5)
Hi @bashtage! Can you clarify what you mean by "missing values"? Do you mean imputation in the original data set? Or missing values in a second dataset that you are massaging to look like the first? Or just any case where the are nulls in the data?
from formulaic.
patsy
has an input called NA_action
that lets you tell it what to do with NAs. I was trying to use formulas in linearmodels but noticed that I am strict with dropping and prefer to raise. I didn't see any obvious way to achieve the same result in formulaic.
from formulaic.
Ah! After thinking about this a little, here is how I'd like to implement this.
I will add a new stateful transform called impute
which allow you to do things like: a + impute(b, mean)
. It will keep track of its state, and passing future dataframes into the formula will remember which value to impute when data is missing. If after the columns are all evaluated (but before encoding) there are any null values, the materialiser will either: ignore the nulls, drop the nulls or raise, depending on the argument passed to the materialiser during its construction.
In full, then, you would something like:
from formulaic import model_matrix
df = ...
model_matrix('a + impute(b, mean)', df, na_action='drop'|'raise'|'ignore')
In this example, the na_action
could only be triggered based on a
(since b would have null values imputed).
Does that sound good?
from formulaic.
That sounds like an excellent solution.
As a wish list item, it would be great if impute
could have an interface so that it would be possible for end users to supply their own imputers. For example, statsmodels as MICE which could be wrapped if there was an interface to do sophisticated imputing, or someone might want to use a PCA-based computer, or one that uses some non-model data such as a regression projection.
from formulaic.
I was imagining that the mean
passed above is a python function that exists in the namespace (it could have been np.mean
, etc). So that should cover all local imputations (imputations local to a column), or non-local imputations where the function already has the relevant context (via a closure or some such). Is that sufficient?
If not, what would a non-local imputation look like (as I imagine would be the case for PCA) expressed in a formula?
from formulaic.
Related Issues (20)
- drop both columns in dependent variable and design matrix when missings occur HOT 5
- DOC: Explicitly mention support for multiple variables on the left hand side HOT 3
- Terms not being evaluated in get_model_matrix() HOT 2
- 17 tests fail: ModuleNotFoundError: No module named 'interface_meta' HOT 2
- How can the encoding choices for one dataset be reused for another? HOT 3
- Intercept term breaks when RHS formula begins with a parentheses HOT 2
- How do I set the reference level for a categorical term? HOT 4
- Support for sympy >= 1.10 HOT 3
- ENH: Preserve variable order as they appear in formulas HOT 5
- 2 tests fail HOT 1
- Interaction between two categorical covariates sometimes switches order, causing error HOT 3
- Intercept is not added after being removed HOT 4
- Proposal: support columns representing multiple features HOT 3
- Formulaic struggles with NAs and `poly()` syntax HOT 3
- Escaped variables and functions HOT 3
- How to include structural zeros? HOT 1
- Retain Column Names for sparse model matrices HOT 4
- Formulaic not raising an exception when required fields are missing in the dataset HOT 2
- Allow formatting the categorical encoded variables HOT 4
- Throw error when formula has parameters that are not available HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from formulaic.