Extending `formulaic` to work with other input types about formulaic HOT 2 OPEN

Hi @bnjhng. Thanks for reaching out.

In terms of the general role of materialization, did you read the generic docs here: https://matthewwardrop.github.io/formulaic/guides/formulae/#materialization ?

In terms of the right solution to this, your approach isn't crazy! It likely misses some edge-cases, but it probably works in the majority of cases. I've been tossing up adding support for dicts like this by just casting them to a pandas DataFrame in the default pandas materializer. Is there a reason that you want to avoid this?

from formulaic.

In terms of the general role of materialization, did you read the generic docs here: https://matthewwardrop.github.io/formulaic/guides/formulae/#materialization ?

Thanks for linking it! Somehow I missed that section as I was going through the docs.

I've been tossing up adding support for dicts like this by just casting them to a pandas DataFrame in the default pandas materializer. Is there a reason that you want to avoid this?

We have found that very often, when performing operations that do not involve row indexing (i.e., the vast majority of data transformations), working in dictionary of numpy arrays has speed advantage over pandas DataFrame.

It likely misses some edge-cases, but it probably works in the majority of cases.

This is indeed our conclusion! The main limitation we have encountered so far is with categorical encoding. And we have isolated the main issue to the following:

>>> from formulaic.materializers.types import FactorValues
>>> import pandas as pd
>>> import numpy as np
>>> 
>>> # while this works as expected:
>>> print(pd.Series(FactorValues(pd.Series(["a", "b", "c"]))))
0    a
1    b
2    c
dtype: object

>>> # this doesn't give the expected results:
>>> print(pd.Series(FactorValues(np.array(["a", "b", "c"]))))
0     abc
1    bc
2     c
dtype: object

>>> # and this straight up errors out:
>>> print(pd.Series(FactorValues(np.array([1, 2, 3]))))
TypeError: Argument 'values' has incorrect type (expected numpy.ndarray, got FactorValues)

The doctoring of FactorValues says that it is:

A convenience wrapper that surfaces a FactorValuesMetadata instance at
<object>.__formulaic_metadata__. This wrapper can otherwise wrap any
object and behaves just like that object.

But clearly, in the case of numpy.ndarray, the wrapper doesn't behave just like numpy.ndarray. To get the code above to work properly, one way is to explicitly call the __wrapped__ attribute:

# both of the following works as expected:
>>> print(pd.Series(FactorValues(np.array(["a", "b", "c"])).__wrapped__))
0    a
1    b
2    c
dtype: object

>>> print(pd.Series(FactorValues(np.array([1, 2, 3])).__wrapped__))
0    1
1    2
2    3
dtype: int64

However, there are places in the formulaic code that simply does pandas.Series(data) instead of pandas.Series(data.__wrapped__), for example here and here.

Would it be possible to fix this limitation with formulaic?

from formulaic.