Normalizing Hi-C contact counts efficiently
Home Page: http://members.cbio.mines-paristech.fr/~nvaroquaux/iced/
License: Other
Hi,
I am trying to install iced in python 2.7 but met with the following error. Do you know what happened?
ERROR: Failed building wheel for iced
Failed to build iced
ERROR: Could not build wheels for iced which use PEP 517 and cannot be installed directly
Dear @NelleV
Some questions about the HiC normalization have confused me a long time. As the normalization is critical for the correct identification of different chromation interactions between samples, so I here want to ask you for help. Thank you very much.
1 Does ICE normalization take sequence depth into account?
2 Is there any differences between normalization by chromosome and by genome-wide? Which pipeline can generate more comparable results among different chromosomes?
sincerely,
Zheng zhuqing
Hi, thanks for the normalization algorithm. I found an issue about the element-wise product(**2, iced/iced/normalization/init.py line 101).
If the input of X is the type of matrix, the expression "**2" will make a production of matrices.
If the input of X is the type of ndarray the expression "**2" will make element-wise production.
I think the "**2" in the algorithm is supposed to be element-wise production. So, it might be the explicit expression of element-wise production or check the type of X.
Reproduction:
A = np.matrix(np.ones((2,2)))
print(A**2)
[[2. 2.]
[2. 2.]]
A = np.asarray(np.ones((2,2)))
print(A**2)"
[[1. 1.]
[1. 1.]]
Thanks.
Tutorial explaining:
Hi, I'm trying to use iced 0.5.8 on the raw matrix output of HiC-Pro. Hower, I'm getting inconsistent bin counts.
If I run iced like:
ice --results_filename {output.icematrix} --filter_low_counts_perc 0.02 \
--filter_high_counts_perc 0 --max_iter 100 --eps 0.1 --base 0 \
--remove-all-zeros-loci --output-bias 1 {input.matrix} > {log} 2>&1the output files look like this:
(iced) [bcro-big ice-matrix main*]$ wc -l N2P_D0_1000000*
3594245 N2P_D0_1000000.icematrix
3115 N2P_D0_1000000.icematrix.biases
3597360 total
(iced) [bcro-big ice-matrix main*]$ head -n 2 N2P_D0_1000000*
==> N2P_D0_1000000.icematrix <==
1 1 3590.512980
1 2 1465.522789
==> N2P_D0_1000000.icematrix.biases <==
nan
3.456610107735157977e-01
However, if I check the raw matrix:
[bcro-big 1000000 main*]$ wc -l *
3114 N2P_D0_1000000_abs.bed
3594245 N2P_D0_1000000.matrix
3597359 total
[bcro-big 1000000 main*]$ head -n 2 *
==> N2P_D0_1000000_abs.bed <==
chr1 0 1000000 1
chr1 1000000 2000000 2
==> N2P_D0_1000000.matrix <==
1 1 429
1 2 481
So the size of the matrices is the same, the bin numbering seems to be ok, both of them start at 1, but:
--base 0 to get 1 based bin numbersThe resulting bias file breaks the hicpro2fithic.py script from HiC-Pro. Some HiC-Pro issues might be relevant: this, this, this.
Can I just simply remove the first line of the bias file? It seems to always contain a nan regardless of sample or resolution.
Hi!
I had an issue with pysam.
I had this error message:
File "/home/apps/HiC-Pro/HiC-Pro_2.7.4b/scripts/mergeSAM.py", line 220, in
outfile = pysam.AlignmentFile(output, "wb", template=hr1)
AttributeError: 'module' object has no attribute 'AlignmentFile'
make: *** [bowtie_pairing] Error 1
My pysam version was 0.7.5
$ python
Python 2.7.2 (default, Feb 3 2012, 08:01:33)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-51)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
import pysam
print pysam.version
0.7.5
Pysam 0.7.5 does not have AlignmentFile.
Now I use Pysam 0.9 and it works great!
Thank you Nicolas and Dave for your help!!! :)
Could you rename/copy the COPYING-file to LICENSE? This makes it easier to find.
Hi,I am trying to use iced module to normalized Hi-C data.However , there is something wrong 'ImportError: ……/anaconda2/lib/python2.7/site-packages/iced/normalization.so: undefined symbol: PyUnicodeUCS2_Format
make: *** [ice_norm]'
What does that error mean?How can I fix it?
Thanks!
I am thinking how should I do if I want to compare 2 experiments with different sequence depths. Here is two ways come up to me but I am not sure whether they are correct:
Set the total_counts parameter of each experiments's normalization be the same. Then I can compare the normalized counts of them?
Output the bias of each experiment. And transform the raw count map into probability matrix by calculating Tij=Oij/BiBj. Is this calculation for probability matrix correct? Or there is some ways to calculate the probability matrix from the output normalized count matrix?
Thanks
In the docs you link to this issue-tracker: https://github.com/hiclib/pastis.
I think it would be more logical to link to the issue-tracker of iced.
We should have 0 warnings when building the documentation
Hi,
I used HiC-Pro to get the ICE normalization matrix implemented by hiclib, I used 1kb resolution, but meet the memory problem, so I split the genome matrix into 24 matrices per chromosome, and get the 24 ICE matrices per chromosome, but the ICE assumption of equal visibility (genome-wide) change to equal visibility (per chromosome). Dose this will effect my result greatly? I don't know whether this is right.
Hi Nelle,
There is a small bug in the verbose option using python 3.7, with --verbose 1
Using iced version 0.5.6
Loading files...
Normalizing...
Filter 9 out of 237 bins ...
Matrix is triangular superior
Traceback (most recent call last):
File "/home/nservant/Apps/conda/envs/hic-pro/bin/ice", line 97, in <module>
verbose=args.verbose, eps=args.eps, output_bias=True)
File "/home/nservant/.local/lib/python3.7/site-packages/iced/normalization/__init__.py", line 129, in ICE_normalization
if verbose > 1 and old_bias is not None:
TypeError: '>' not supported between instances of 'str' and 'int'
Dear @NelleV
After installing the latest iced using pip, I found it gives me following warnings when running ice.
/anaconda3/bin/ice:63: SyntaxWarning: "is" with a literal. Did you mean "=="?
if "--filtering_perc" is None and "--filter_low_counts_perc" not in sys.argv:
/anaconda3/lib/python3.8/site-packages/iced/normalization/_ca_utils.py:8: UserWarning: The API of this module is likely to change. Use only for testing purposes
warnings.warn(
usage: ICE normalization [-h] [--results_filename RESULTS_FILENAME] [--filtering_perc FILTERING_PERC] [--filter_low_counts_perc FILTER_LOW_COUNTS_PERC]
[--filter_high_counts_perc FILTER_HIGH_COUNTS_PERC] [--remove-all-zeros-loci] [--max_iter MAX_ITER] [--eps EPS] [--dense]
[--output-bias OUTPUT_BIAS] [--verbose VERBOSE]
File to load
ICE normalization: error: the following arguments are required: File to load
Best wishes,
Zheng zhuqing
When installing iced using pip:
$ pip install iced
...
Building wheels for collected packages: iced
Building wheel for iced (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: /home/tgg/venv-test/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-jD4T4X/iced/setup.py'"'"'; __file__='"'"'/tmp/pip-install-jD4T4X/iced/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-vT3ByC
cwd: /tmp/pip-install-jD4T4X/iced/
Complete output (7 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-jD4T4X/iced/setup.py", line 112, in <module>
setup_package()
File "/tmp/pip-install-jD4T4X/iced/setup.py", line 104, in setup_package
from numpy.distutils.core import setup
ImportError: No module named numpy.distutils.core
----------------------------------------
ERROR: Failed building wheel for iced
Running setup.py clean for iced
Failed to build iced
Installing collected packages: iced
Running setup.py install for iced ... error
ERROR: Command errored out with exit status 1:
command: /home/tgg/venv-test/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-jD4T4X/iced/setup.py'"'"'; __file__='"'"'/tmp/pip-install-jD4T4X/iced/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-N3SMcO/install-record.txt --single-version-externally-managed --compile --install-headers /home/tgg/venv-test/include/site/python2.7/iced
cwd: /tmp/pip-install-jD4T4X/iced/
Complete output (7 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-jD4T4X/iced/setup.py", line 112, in <module>
setup_package()
File "/tmp/pip-install-jD4T4X/iced/setup.py", line 104, in setup_package
from numpy.distutils.core import setup
ImportError: No module named numpy.distutils.core
----------------------------------------
ERROR: Command errored out with exit status 1: /home/tgg/venv-test/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-jD4T4X/iced/setup.py'"'"'; __file__='"'"'/tmp/pip-install-jD4T4X/iced/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-N3SMcO/install-record.txt --single-version-externally-managed --compile --install-headers /home/tgg/venv-test/include/site/python2.7/iced Check the logs for full command output.
The issue appears to be related to numpy not being present before iced is installed. This specific example is intended to be a minimal example and is somewhat contrived, but the same issue arises when doing a standard pip install -r requirements.txt with a requirements.txt file that includes both iced and numpy: pip builds wheels for all packages first before installing any packages, so even though the requirements.txt file includes numpy, pip will attempt to build the wheel for iced before numpy is installed.
A typical "old-school" workaround is to perform the installation in two steps, e.g.:
$ pip install $(grep requirements.txt numpy) && pip install -r requirements.txt
It is possible to avoid this extra step and recover the simplicity of "pip install -r requirements.txt" by declaring that the installation depends on numpy using PEP 518. The implementation for this could simply be to add a pyproject.toml file with the following contents:
[build-system]
requires=['setuptools', 'wheel', 'numpy']
This would have no impact on users using older versions of the Python packaging tools (these older versions just ignore pyproject.toml), but users using newer versions would be able to directly install iced using pip without knowing anything about the confusing history of Python packaging.
All in the title.
Hi :
the script python /data1/liuzunpeng/04_Softwares/iced-0.5.1/iced/scripts/ice --results_filename hic_results/matrix/$sample/iced/${res}/${sample}_${res}_iced.matrix --filter_low_counts_perc 0.02 --filter_high_counts_perc 0 --max_iter 100 --eps 0.1 --remove-all-zeros-loci --output-bias 1 --verbose 1 hic_results/matrix//${sample}/raw/${res}/${sample}_${res}.matrix
the error iced.sh: line 9: 25355 Segmentation fault
When I cat ice ,I found the line 9 is from iced.io import loadtxt, savetxt
But the python /data1/liuzunpeng/04_Softwares/iced-0.5.1/iced/scripts/ice -h worked ,and the python ~/iced-0.5.1/iced/tests/test_filter.py also worked .
Can you give me some advices?
Thanks for your time !
Best wishes
Qianzhao
When I try to manually do ice normalization on matrix files from HiC-Pro ice fails with the following error:
Traceback (most recent call last):
File "/home/balthasar/.local/bin/ice", line 95, in <module>
counts, bias = iced.normalization.ICE_normalization(
File "/home/balthasar/.local/lib/python3.8/site-packages/iced/normalization/__init__.py", line 66, in ICE_normalization
is_symetric_or_tri(X)
File "/home/balthasar/.local/lib/python3.8/site-packages/iced/utils/validation.py", line 13, in is_symetric_or_tri
raise ValueError("The matrix should be symmetric")
ValueError: The matrix should be symmetric
The matrix file in question was made by downloading an allValidPairs file from Github and then building the matrix file with the build_matrix script from HiC-Pro.
When issuing the ice command like this:
ice output.matrix
The above error appears after a while.
iced is not compatible with the recent numpy versions
iced/iced/normalization/__init__.py
Line 79 in 30c5673
It should be corrected to X.eliminate_zeros() without the assignment operator as it is an in-place function.
With the assignment operator, it assigns a None type object to X if no zeros were eliminated.
I'm experiencing an import issue with iced and io.
File "/project/umw_rene_maehr/programs/HiC-Pro/scripts/ice", line 5, in <module>
import numpy as np
File "/share/pkg/python/2.7.9_packages/numpy/1.9.2/lib/python2.7/site-packages/numpy/__init__.py", line 170, in <module>
from . import add_newdocs
File "/share/pkg/python/2.7.9_packages/numpy/1.9.2/lib/python2.7/site-packages/numpy/add_newdocs.py", line 13, in <module>
from numpy.lib import add_newdoc
File "/share/pkg/python/2.7.9_packages/numpy/1.9.2/lib/python2.7/site-packages/numpy/lib/__init__.py", line 8, in <module>
from .type_check import *
File "/share/pkg/python/2.7.9_packages/numpy/1.9.2/lib/python2.7/site-packages/numpy/lib/type_check.py", line 11, in <module>
import numpy.core.numeric as _nx
File "/share/pkg/python/2.7.9_packages/numpy/1.9.2/lib/python2.7/site-packages/numpy/core/__init__.py", line 46, in <module>
from numpy.testing import Tester
File "/share/pkg/python/2.7.9_packages/numpy/1.9.2/lib/python2.7/site-packages/numpy/testing/__init__.py", line 13, in <module>
from .utils import *
File "/share/pkg/python/2.7.9_packages/numpy/1.9.2/lib/python2.7/site-packages/numpy/testing/utils.py", line 15, in <module>
from tempfile import mkdtemp
File "/share/pkg/python/2.7.9/lib/python2.7/tempfile.py", line 32, in <module>
import io as _io
File "/share/pkg/python/2.7.9_packages/iced/0.2.1/lib/python2.7/site-packages/io/__init__.py", line 1, in <module>
from .fastio_ import loadtxt, savetxt
File "__init__.pxd", line 155, in init iced.io.fastio_ (iced/io/fastio_.c:4789)
AttributeError: 'module' object has no attribute 'dtype'
Have you encountered this before, and can you recommend a fix? Thanks!
Need to fix this…
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.