jasonkessler / scattertext-pydata Goto Github PK

Hello,
I use the Jupyter Notebook with my own data. When running the following part of the code

def normcdf(x):
    return norm.cdf(x, x.mean(), x.std())
term_freq_df['eight_precision_normcdf'] = normcdf(term_freq_df['eight_precision'])
term_freq_df['eight_freq_pct_normcdf'] = normcdf(term_freq_df['eight_freq_pct'])
term_freq_df['eight_scaled_f_score'] = hmean([term_freq_df['eight_precision_normcdf'], term_freq_df['eight_freq_pct_normcdf']])
term_freq_df.sort_values(by='eight_scaled_f_score', ascending=False).iloc[:10]

I get the following error

The column categories run from 'zero' to 'eight'. Any suggestion to overcome this problem is welcome

I am trying to import scattertext on AWS SageMaker with a conda_python3 kernel. I downgraded numpy to 1.20.3 and installed a certain daal version based on the error messages.
!pip install --upgrade numpy==1.20.3
!pip install daal==2021.2.3

Now I get the error message "ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject"

This still seems like a numpy package collision. Is there a Requirements.txt for scattertext?

Hi Jason, just want to say thanks for your codes! I have been looking for a long time for something like this and was almost going to give up. Now, I just hope it will work on my dataset. By the way, your corpus code does not work and needs to be updated. It should be:

corpus = st.CorpusFromPandas(convention_df, category_col='party', text_col='text',nlp=nlp).build()

instead of

corpus = st.CorpusFromParsedDocuments(convention_df, category_col='party', parsed_col='parsed').build()

This is great! How can one incorporate dimensionality reduction into the pipeline? For substantive and speed reasons, I'd like to exclude the most and least common words:

corpus = st.CorpusFromPandas(df,
category_col='country',
text_col='text',
nlp=nlp,
# can we discard 1st and 99th percentile of words here?
).build()