Potential bug in Interacting variables via `:` syntax for categorical variables about formulaic HOT 3 CLOSED

Hi Alex! Thanks for reaching out again.

This is also not a bug, and is documented abstractly here: https://matthewwardrop.github.io/formulaic/guides/grammar/#behaviours-and-conventions

The patsy documentation related to this is here: https://patsy.readthedocs.io/en/latest/formulas.html#redundancy-and-categorical-factors.

In short, though, we want to ensure that the matrices generated are (at least structurally) full rank. Since categorical encodings span the intercept, including the full Cartesian product of values would span the intercept column, leading to non-full-rank model matrices. Instead, we give you full set of features that span the requested Cartesian product.

If you omit the intecept, you will see that the Cartesian product is more like you expect.

You can also disable this behaviour by passing ensure_full_rank=False to the model_matrix or get_model_matrix functions, for example:

At the cost of guaranteeing structurally that the model matrix will not be full rank. In (e.g.) linear regression, this will cause X.T @ X to not be invertible and introduce a gauge degree of freedom in your model.

R is inconsistent with this kind of behaviour. It tries to reduce the rank of categorical variables on their own, but gives up for higher order interactions. We just do it properly.

from formulaic.

Ah, ok, this makes perfect sense! Thank you for your swift response =) My incorrect assumption was that formulaic mimics R's model.matrix(), but obviously it is based on patsy. I will have to think through if I want to set ensure_full_rank = False as a default for pyfixest (for perfect compatibility with fixest / R), or if I should keep with the formulaic defaults. Both fixest and pyfixest include a function to detect and drop multicollinear variables, so I (can) in principle ensure that the design matrix is of full rank even after calling formulaic.get_model_matrix(..., ensure_full_rank = False). I should also spend some time to read through the patsy docs!

As a side point, I've worked quite a bit with formulaic in the past week, and I found myself several times thinking that working with formulaic is really quite a pleasant experience! Starting from a already high base, using some of the "more advanced" features of the package has clearly increased my appreciation of your work even further! =)

from formulaic.

Thanks for the compliment!

It's worth noting, though, that even when you set ensure_full_rank=False it won't match R exactly in all cases... Since R does sometimes try to reduce rank of categorical encodings to maintain full rank... It just isn't as consistent in doing so. There is no option in Formulaic that perfectly matches R's inconsistent behaviour.

from formulaic.