vectorizing_text.ipynb Cell #4 error about ml-powered-applications HOT 5 CLOSED

Thank you for reporting the error @SSJUSA. While UMAP claims to be pretty memory efficient (see their benchmark), it can still get pretty memory hungry!

To get the notebook running, I suggest switching to a faster embedding method. Here, TruncatedSVD should do the trick.

To use that, replace cell 4 with:

from sklearn.decomposition import TruncatedSVD
umap_embedder = TruncatedSVD()
umap_bow = umap_embedder.fit_transform(bag_of_words)

You will likely also need to change cell 8 to use TruncatedSVD by replacing it with:

umap_embedder = TruncatedSVD()
umap_emb = umap_embedder.fit_transform(embeddings)

If that does not fix your issue, you can instead reduce the dataset size by selecting the first 2000 rows for example. To do so, add a new line in the first cell following the format_raw_df call:
df = df.head(2000)

from ml-powered-applications.

from ml-powered-applications.

The error ValueError: Found array with 0 feature(s) (shape=(5676, 0)) while a minimum of 2 is required. makes it seem like the embeddings variable has rows of size 0. I think this may be a different issue than the previous one.

If you run embeddings.shape in a notebook cell right before cell 8 should be able to see whether they are of the correct size. On the full dataset, the results should be (5676, 300). If you've run df = df.head(2000) it should be (265, 300).

If they are not, try running train_author[train_author["is_question"]]["body_text"].shape which should be the same number of rows (but only one dimension). So for the full dataset you should see (5676,) and (265,) for the partial one.

Finally, you may want to take another look at the spaCy docs and reinstall the large model by running python -m spacy download en_core_web_lg from the command line.

from ml-powered-applications.

from ml-powered-applications.

This is resolved quickly by the author @hundredblocks . Thanks.
This is closed.

from ml-powered-applications.