Comments (3)
Hi @MartinStancsicsQC !
Thanks for reaching out, and apologies for the delay. It's exciting to see the extensibility of formulaic being exploited in the wild! And I'm more than happy to extend Formulaic's support for these kinds of use-cases.
One thing I didn't fully understand is why you want to treat a categorical column as if it were one-hot encoded when it was not. Why not just encode the column directly as a categorical/factor column, and then let tabmat take care of it directly? We'd need a tiny amount of extra metadata for the structure enforcement, but otherwise this should work just fine. Today you can achieve most of this passthrough (modulo enforcement) by just wrapping your incoming vector in a FactorValues(...)
class, and indicating that they are already encoded. We can make this more naturally a part of the categorical coding functionality using something so you can do something like C(X, contr.none)
.
In [1]: import pandas; from formulaic import FactorValues, model_matrix
In [2]: df = pandas.DataFrame({'X': ['a', 'b', 'c', 'a', 'b', 'c']})
In [3]: model_matrix("FactorValues(X, encoded=True)", df)
Out[3]:
Intercept FactorValues(X, encoded=True)
0 1.0 a
1 1.0 b
2 1.0 c
3 1.0 a
4 1.0 b
5 1.0 c
We can extend FactorValues
with a factor_levels
attribute that we can check during structure enforcement, which would be populated by C
.
Would that work for you? Or is there something I am missing? If not, I can have this added in the next patch release :).
from formulaic.
Hey @matthewwardrop, thanks so much for the response. I really like the idea of FactorValues
having some metadata about categorical levels, which can be set in C()
or encode_contrasts
. Let me elaborate a bit on our use case to make sure I understand the proposed solution :)
First, let me clarify the need for encoding, because I now realize that I've omitted an important piece of information: the materializer we are building takes a pandas.DataFrame
as its input, and outputs a tabmat.MatrixBase
object. So each pandas column has to be encoded in the sense of being converted to the appropriate tabmat column type. In the case of categoricals, it happens here, and I think this conversion fits very nicely into formulaic's structure. The only discrepancy is that formulaic sees the resulting object as one column, while tabmat treats it as a set of dummies (e.g., tabmat.CategoricalMatrix(["a", "b", "a"]).shape == (3, 2)
as opposed to pandas.Categorical(["a", "b", "a"]).shape == (3, )
).
Now, if I understand your idea correctly, encode_contrasts
would be able to set some metadata about which dummy columns a tabmat categorical matrix actually represents (i.e. factor_levels), and _enforce_structure
would perform its checks based on that. If that is the case, I think that would be a very nice solution to one part of this problem.
It would also be super useful if the ModelSpec.structure
would then be populated based on this metadata (so that, for example, ModelSpec.column_names
can be used as coefficient names later on). I'm thinking of something like
...
spec = spec.update(
structure=[
EncodedTermStructure(
term,
[st.copy(without_values=True) for st in scoped_terms],
list(
itertools.chain.from_iterable(
_name_columns(name, value) for name, col in scoped_cols.items()
)
),
)
for term, scoped_terms, scoped_cols in cols
],
)
...
def _name_columns(name, col):
if value.__formulaic_metadata__.factor_levels is not None:
for level in col.__formulaic_metadata__.factor_levels:
yield col.__formulaic_metadata__.name_format.format(name, level)
else:
yield name
replacing these lines. How does that look to you? (Apologies if something does not make sense, I'm still in the process of getting my head around the internals of formulaic.)
It's also perfectly fine for us to to make these changes downstream if you think it is too specific functionality, or you'd like to have them tried out somewhere before adding them to formulaic.
from formulaic.
Sorry for the latency. Life has been pretty hectic recently. This is going to miss the next patch release of formulaic, but I have read through this again and portions of tabmat
and I think I understand what is going on.
Firstly, tabmat
looks awesome. Kudos to you for your work there. I might even pick it up and start playing with it in some compressed regression routines I built. And as I've said before, I'm excited that Formulaic is proving useful and extensible in the way I had hoped.
Secondly, adding support for these kind of data structures makes a lot of sense to me. I'll noodle on how to best integrate it, but I do have some follow up questions (which might be answered by digging into tabmat more... but perhaps it is more expedient to simply ask): How do interaction terms work?
from formulaic.
Related Issues (20)
- How to include structural zeros? HOT 1
- Retain Column Names for sparse model matrices HOT 4
- Formulaic not raising an exception when required fields are missing in the dataset HOT 2
- Allow formatting the categorical encoded variables HOT 4
- Throw error when formula has parameters that are not available HOT 2
- Support polars HOT 4
- Dropping Indices via "+0" or "-1" and reference levels for categoricals HOT 1
- Extending `formulaic` to work with other input types HOT 2
- Handling individual columns that can expand into multiple columns HOT 7
- Support the hashing trick as an encoding strategy for categorical features HOT 6
- `model_spec.transform_state` bugged when formula is not correctly written HOT 1
- Is there a way to get the baseline value for categorical variables? HOT 7
- Add . operator HOT 1
- Suggestions for creating `get_feature_names_out` for Scikit Learn ColumnTransformer compatibility? HOT 3
- Is it possible to define custom operators? HOT 2
- Is it possible to force the `Formula` class to not expand categorical variables? HOT 3
- Add required variables to the `Formula` class HOT 6
- Potential Bug / different defaults for Intercept / Reference Levels when using `Formula.get_model_matrix()` with categoricals HOT 2
- Potential bug in Interacting variables via `:` syntax for categorical variables HOT 3
- Incompatibility with pandas development version HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from formulaic.