import from scanpy about sccoda HOT 7 CLOSED

Hi,
before answering your question, let me quickly clarify your confusion with the two repositories: SCDCdm_public is an old version of the project and not maintained anymore. This repository (scCODA) is the current one. Also, as already mentioned in your other issue (#15), you should use the dev branch atm, as we are still in development.

Regarding your issue:
It looks like something goes wrong with transforming your covariates from a subdictionary of your scanpy objects. It is intended that all the scanpy objects you want to use have the covariates saved as a subdictionary of adata.uns.
Could you show me adata.uns for one of your samples and tell me what part of it you want to use as your covariates in scCODA?

from sccoda.

thanks for the clarification, Johannes!

actually, switching to scCODA from the dev branch as suggested in #15 affected the behavior of scdcdm.util.cell_composition_data.from_scanpy_list as well. Depending on whether I pass adata or [adata] to the samples= argument, the command either gets stuck in the loop forever without any error message:

In [6]: dat.from_scanpy_list(adata,'leiden','Condition')
/opt/anaconda3/lib/python3.7/site-packages/anndata/_core/anndata.py:1056: FutureWarning: is_categorical is deprecated and will be removed in a future version.  Use is_categorical_dtype instead
  if not is_categorical(df_full[k]):

or it shows the following error message:

In [7]: dat.from_scanpy_list([adata],'leiden','Condition')
Traceback (most recent call last):

  File "<ipython-input-7-b2e2010d4dfa>", line 1, in <module>
    dat.from_scanpy_list([adata],'leiden','Condition')

  File "/Applications/python_modules/scCODA/scCODA/sccoda/util/cell_composition_data.py", line 81, in from_scanpy_list
    covariate_data = covariate_data.fillna(0)

  File "/opt/anaconda3/lib/python3.7/site-packages/pandas/core/frame.py", line 4327, in fillna
    downcast=downcast,

  File "/opt/anaconda3/lib/python3.7/site-packages/pandas/core/generic.py", line 6083, in fillna
    value=value, limit=limit, inplace=inplace, downcast=downcast

  File "/opt/anaconda3/lib/python3.7/site-packages/pandas/core/internals/managers.py", line 595, in fillna
    "fillna", value=value, limit=limit, inplace=inplace, downcast=downcast

  File "/opt/anaconda3/lib/python3.7/site-packages/pandas/core/internals/managers.py", line 409, in apply
    applied = getattr(b, f)(**kwargs)

  File "/opt/anaconda3/lib/python3.7/site-packages/pandas/core/internals/blocks.py", line 1779, in fillna
    values = values.fillna(value=value, limit=limit)

  File "/opt/anaconda3/lib/python3.7/site-packages/pandas/core/arrays/categorical.py", line 1721, in fillna
    raise ValueError("fill value must be in categories")

ValueError: fill value must be in categories

I copied adata.uns['Condition'] from adata.obs since anndata.AnnData.concatenate, which I ran earlier, stores sample labels in adata.obs.

My adata.uns is very long, but here is a representative chunk of it:

 'Condition': AAAGGTAGTGGCACTC-0     W_d0
 GAAGTAATCTTACCAT-0     W_d0
 AGATAGACAGACAAGC-0     W_d0
 CCTAAGACAAGCACAG-0     W_d0
 ATCTCTACAGCTACAT-0     W_d0

 TACGGTAAGGTTGTTC-14    W_d0
 TTTACTGAGCGTCGAA-14    W_d0
 AGCGTCGCATCACCAA-14    W_d0
 ACGGAAGAGCCTGCCA-14    W_d0
 GCCGTGAAGCCGCTTG-14    W_d0
 Name: caste, Length: 85942, dtype: category
 Categories (2, object): ['G_d30', 'W_d0']}

as a workaround, I just ran a quick for loop with np.count_nonzero to create a pandas.DataFrame containing cell counts in each cluster x condition and then passed it to scdcdm.util.cell_composition_data.from_pandas.

from sccoda.

Hi @bsierieb1,
the endless loop is definitely not intended, I will add a condition to avoid this in the future!

However, I am still unsure whether you can use dat.from_scanpy_list here. Do you have an anndata object for each sample or do you have one anndata object that includes all your samples?
What dat.from_scanpy_list does is that it applies dat.from_scanpy to each element in samples. Therefore, the compositional dataset is created "row-by-row".

Additionally, I could not find where in your adata.uns the covariate information (preferably a string that is either 'G_d30' or 'W_d0') is located. So far, I assumed that one would have a subdictionary with all metadata information (i.e. covariates) in adata.uns, which can simply be used as a row of the covariate data frame, but this does not seem to be the case here.

Could you tell me if there is a way to extract your condition from adata.uns? I'd like to make dat.from_scanpy_list more flexible regarding inclusion of covariates.

from sccoda.

hi Johannes,

I only have one anndata object, not a list. However, I was using dat.from_scanpy_list instead of dat.from_scanpy because dat.from_scanpy returns a different output according to readthedocs. Instead of returning a CompositionalData object, like all the other dat.from_ functions, dat.from_scanpy returns a np.array with cell counts per cluster and a list with covariates. In my case, the np.array with cell counts per cluster is not split by condition, so it is not useful, and instead of the covariate list it simply returns adata.uns['Condition'].

Maybe the problem is that my covariate information is not a dictionary, as what you said you expect, but a pandas series? The reason it is in this format is because I started with a bunch of anndata objects (different libraries from different conditions). At some point, I concatenated them using anndata.AnnData.concatenate, which adds sample IDs to a pandas dataframe in anndata.obs. Then I generated my covariate values from the sample IDs (knowing which library corresponds to which condition).

Sorry, I just realized I made a typo in my previous comment. I said dat.uns['Condition'], but what I meant was that this is my adata.uns['Condition']:

 'Condition': AAAGGTAGTGGCACTC-0     W_d0
 GAAGTAATCTTACCAT-0     W_d0
 AGATAGACAGACAAGC-0     W_d0
 CCTAAGACAAGCACAG-0     W_d0
 ATCTCTACAGCTACAT-0     W_d0
...
 TACGGTAAGGTTGTTC-14    W_d0
 TTTACTGAGCGTCGAA-14    W_d0
 AGCGTCGCATCACCAA-14    W_d0
 ACGGAAGAGCCTGCCA-14    W_d0
 GCCGTGAAGCCGCTTG-14    W_d0
 Name: caste, Length: 85942, dtype: category
 Categories (2, object): ['G_d30', 'W_d0']}

from sccoda.

Dear @bsierieb1,
using anndata.AnnData.concatenate and having one object for all samples is actually a scenario that I did not consider when writing these functions. dat.from_scanpy is only supposed to be used within dat.from_scanpy_list or dat.from_scanpy_dict. Therefore, I don't think that scCODA currently has an automatic data conversion method for your data.

You're right, the error was caused by the series in adata.uns, but even if this was not the case, dat.from_scanpy_list([adata]) would have given you only the cell type counts over all samples.

I'll try to come up with a solution to this, but for now I'd suggest you use your workaround

from sccoda.

great, thanks! it would be great if scCODA could handle concatenated anndata objects, too, because I suspect a lot of people may first do UMAP and clustering on a concatenated data set, and then perform various downstream analyses such as compositional analysis.

from sccoda.

Version 0.1 (#17) addresses this exact task in the re-written function from_scanpy, which allows you to specify a column in adata.obs that contains the sample assignment. Covariates can be passed as a key in adata.uns (as before) or as a separate DataFrame.
I hope this solves the issue!

from sccoda.