Count matrix to cistopic link broken + scalability to large datasets question about scenicplus HOT 4 OPEN

Hi @wgao688

You can still find the tutorial here: https://github.com/aertslab/pycisTopic/blob/old/notebooks/Toy_melanoma-RTD.ipynb

We have ran pycisTopic successfully with >100k cells, also we are improving the topic modeling step: https://github.com/aertslab/pycisTopic/tree/polars_1xx

Besides number of cells, topic modelling also scales with the number of regions. How many regions are you using? Especially when you use the tile matrix from snapATAC2 this number can be very large (I would suggest to use a count matrix based on consensus peaks, either from snapATAC2 or pycisTopic, if this is possible).

All the best,

Seppe

from scenicplus.

With the Polars 1xx branch it is now possible to make a Mallet corpus file from a binary count matrix file in Matrix Market format:

    Expose creation of Mallet corpus file from pycistopic CLI interface:
    
        pycistopic topic_modeling create_mallet_corpus
    
    Usage:
    
      Create binary accessibility matrix in Matrix Market format:
    
        import pycisTopic.fragments
        import scipy
    
        counts_fragments_matrix, cbs, region_ids = pycisTopic.fragments.create_fragment_matrix_from_fragments(
            "fragments.tsv.gz",
            "consensus_regions.bed",
            "cbs.tsv"
        )
    
        # Create binary matrix:
        binary_matrix = counts_fragments_matrix.copy()
        binary_matrix.data.fill(1)
    
        # Write binary matrix in Matrix Market format.
        scipy.io.mmwrite("binary_accessibility.mtx", binary_matrix)
    
      Create Mallet corpus file from binary accessibility matrix in Matrix Market format:
    
        $ pycistopic topic_modeling create_mallet_corpus -i "binary_accessibility.mtx" -o "corpus.mallet"

aertslab/pycisTopic@2d54473

from scenicplus.

Thanks Seppe and Gert, I will try the code above. I am working with about 300K peaks for 300,000 cells. Should I subset to the most highly variable peaks (e.g., 50K, 100K)?

I also wish to handle batch effects (due to individual donor and technology (multiome vs. ATAC only). I see from the SCENIC+ paper that you used harmonypy on the scaled cell–topic matrix for the mouse cerebellum dataset. I also saw in a previous Github question that you do not perform batch correction typically (#134). Is there anything specific that you recommend for batch correction?

from scenicplus.

@ghuls I am trying to run pycistopic topic_modeling create_mallet_corpus -i "binary_accessibility.mtx" -o "corpus.mallet" but I am not seeing the create_mallet_corpus option available. Is there something wrong with my download of the polars_1xx branch?

git clone --branch polars_1xx --single-branch https://github.com/aertslab/pycisTopic.git
cd pycisTopic/
pip install -e . 

Thanks

from scenicplus.