FlowUtils is a Python package containing various utility functions related to flow cytometry analysis, primarily focused on compensation and transformation tasks commonly used within the flow community.
Home Page: https://flowutils.readthedocs.io
License: BSD 3-Clause "New" or "Revised" License
i tried to import flowkit but there is an error like:
ImportError: cannot import name 'logicle_c' ithink it is coming from
\flowutils\transforms.py", line 7, in
from flowutils import logicle_c
Did it happen before? not sure where it can come from
Hello,
panelbuilder has some nice unmixing options that go beyond OLS. I haven't tried too much of it, but especially removing specific spectra from the data (e.g. an autofluorescence from single stain cells) seems interesting. I'll probably reproduce a lot of what they do in Python anyway, would you be interested in me adding that functionality to the package? Their special unmixing would add a torch dependency, but that could be treated as an optional dependency. (I could technically also implement it in pure numpy, but GPU support is pretty useful if the data get large.)
Greetings
Valentin
Currently, the compensate function takes a numpy data array as an input along with the list of column indices that will be compensated. However, it only returns the compensated columns, requiring the caller to re-assemble the original data array with the compensated columns. This is a bit awkward and the function should return the whole data array.
Add docstrings for functions to include the descriptions of arguments and return values.
Hi,
I've encountered some differences when calculating the logicle transform using different methods. I had seen that flowutils matches the GatingML 2.0 spec, but I get differences when computing with mpmath, a Python package for arbitrary-precision floating-point arithmetic.
When I tell mpmath to use 30 digits of decimal precision, I get these results for the test values for logicle(x,1000,1,4,1) in the GatingML 2.0 spec:
| x | expected | computed with mpmath |
|---|---|---|
| -10 | 0.254059 | 0.244811788147158329560715914228 |
| -5 | 0.318389 | 0.31741253160831268687487017787 |
| -1 | 0.383001 | 0.382999362288320181332800213487 |
| 0 | 0.4 | 0.400000000000000022204460492503 |
| 0.3 | 0.405107 | 0.405107011048485188577537974655 |
| 1 | 0.416999 | 0.41699870734437975458826375087 |
| 3 | 0.450318 | 0.450317517410592531905350830829 |
| 10 | 0.545941 | 0.545940641054175536655407791803 |
| 100 | 0.791638 | 0.791638120758736567347236966264 |
| 1000 | 1 | 1.0 |
The differences seem to only occur at the negative x values, where only the first two digits after the decimal point are matching. I have also tried this with larger values of decimal precision up to 10,000 but have not seen the mpmath results match the GatingML 2.0 spec. Have you encounter this? I've pasted my mpmath python script below. Any thoughts?
Thanks!
import mpmath as mp
################################################################################
# for logicle(x,1000, 1, 4, 1)
GATING_ML_TABLE = [
(-10, 0.254059),
(-5, 0.318389),
(-1, 0.383001),
(0, 0.4),
(0.3, 0.405107),
(1, 0.416999),
(3, 0.450318),
(10, 0.545941),
(100, 0.791638),
(1000, 1),
]
################################################################################
def find_d(w, b):
f = lambda x : 2 * (mp.log(x) - mp.log(b)) + w *(x+b)
result = mp.findroot(f,0.5)
return result
def B(y, T, W, M, A):
w = W / (M + A)
x2 = A / (M+A)
x1 = x2 + w
x0 = x2 + mp.mpf(2.0)*w
b = (M+A) * mp.log(mp.mpf(10.0))
# d such that 2 * (log(d) - log(b)) + w*(d+b) = 0 ## need to solve with newton's method or similar
d = find_d(w, b)
ca = mp.exp(x0 * (b+d))
fa = mp.exp(b*x1) - ca/ mp.exp(d*x1)
a = T / (mp.exp(b) - fa - ca/mp.exp(d))
c = ca*a
f = fa*a
return a * mp.exp(b*y) - c* mp.exp(-d * y) - f
def logicle(x, T, W, M, A):
f = lambda xx : B(xx, T, W, M, A) - x
#result = mp.findroot(f, 0.5)
result = mp.findroot(f, 0.9)
return result
mp.mp.dps = 30
mp.pretty = True
for x, y in GATING_ML_TABLE:
print("For {x}, expected {y}, got ".format(x=x, y=y), logicle(x, 1000, 1, 4, 1))I am having the following error when installing FlowUtils using Python 3.7 64 bits
logicle.c
flowutils/logicle_c_ext/logicle.c(213): error C2057: expected constant expression
flowutils/logicle_c_ext/logicle.c(213): error C2466: cannot allocate an array of constant size 0
flowutils/logicle_c_ext/logicle.c(213): error C2133: 'tmp_taylor': unknown size
flowutils/logicle_c_ext/logicle.c(321): error C2057: expected constant expression
flowutils/logicle_c_ext/logicle.c(321): error C2466: cannot allocate an array of constant size 0
flowutils/logicle_c_ext/logicle.c(321): error C2133: 'tmp_taylor': unknown size
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Professional\\VC\\Tools\\MSVC\\14.21.27702\\bin\\HostX86\\x64\\cl.exe' failed with exit status 2
Any idea?
Hi Scott,
First of all thanks for putting together this package that makes data analysis very easy. All of my colleagues are using it on Mac. I run under a Windows with a VM on ubuntu and when I tried to install the package using python 3.8 I received this error message.
I would be very grateful if you could help me out solving the issue,
Thanks a million for your help as I'm very new to code.
Input
python3.8 -m pip install flowutils
ERROR: Failed building wheel for flowutils
Running setup.py clean for flowutils
Failed to build flowutils
Installing collected packages: flowutils
Running setup.py install for flowutils ... error
ERROR: Command errored out with exit status 1:
command: /usr/bin/python3.8 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-w2f4txmj/flowutils_ce8ef7c331e14639a53a1273947fce47/setup.py'"'"'; file='"'"'/tmp/pip-install-w2f4txmj/flowutils_ce8ef7c331e14639a53a1273947fce47/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-gepqjqt8/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/customer/.local/include/python3.8/flowutils
cwd: /tmp/pip-install-w2f4txmj/flowutils_ce8ef7c331e14639a53a1273947fce47/
Complete output (20 lines):
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.8
creating build/lib.linux-x86_64-3.8/flowutils
copying flowutils/compensate.py -> build/lib.linux-x86_64-3.8/flowutils
copying flowutils/transforms.py -> build/lib.linux-x86_64-3.8/flowutils
copying flowutils/init.py -> build/lib.linux-x86_64-3.8/flowutils
running build_ext
building 'flowutils.logicle_c' extension
creating build/temp.linux-x86_64-3.8
creating build/temp.linux-x86_64-3.8/flowutils
creating build/temp.linux-x86_64-3.8/flowutils/logicle_c_ext
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/customer/.local/lib/python3.8/site-packages/numpy/core/include -Iflowutils/logicle_c_ext -I/usr/include/python3.8 -c flowutils/logicle_c_ext/_logicle.c -o build/temp.linux-x86_64-3.8/flowutils/logicle_c_ext/_logicle.o -std=c99
flowutils/logicle_c_ext/_logicle.c:1:10: fatal error: Python.h: No such file or directory
#include <Python.h>
^~~~~~~~~~
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3.8 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-w2f4txmj/flowutils_ce8ef7c331e14639a53a1273947fce47/setup.py'"'"'; file='"'"'/tmp/pip-install-w2f4txmj/flowutils_ce8ef7c331e14639a53a1273947fce47/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-gepqjqt8/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/customer/.local/include/python3.8/flowutils Check the logs for full command output.`
Hi,
first, thanks very much for this effort to provide open facs analysis tools! We are in need for some of the transformation functions (e.g., Logicle), offered in the higher-level module flowkit. However I got an error message in the installation process. The error is caused by the flowutils installation. The relevant line might be:
flowutils/logicle_c_ext/_logicle.c:120:5: error: implicit declaration of function 'hyperlog_inverse' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
hyperlog_inverse(t, w, m, a, xc, n);
Do you know what's going on here and can help us out? Thanks a lot in advance. The full error message is below.
My system is macOS Catalina, python 3.8.8, numpy 1.19.2.
Collecting flowutils
Using cached FlowUtils-0.9.1.tar.gz (11 kB)
Requirement already satisfied: numpy>=1.7 in ./opt/anaconda3/envs/sc_env/lib/python3.8/site-packages (from flowutils) (1.19.2)
Building wheels for collected packages: flowutils
Building wheel for flowutils (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: /Users/user/opt/anaconda3/envs/sc_env/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/d6/cn_12s513fj8dymm3xy_tdy9mskn84/T/pip-install-4ev728v7/flowutils_c66ffa7ac87d488f9f05705abe9735a0/setup.py'"'"'; __file__='"'"'/private/var/folders/d6/cn_12s513fj8dymm3xy_tdy9mskn84/T/pip-install-4ev728v7/flowutils_c66ffa7ac87d488f9f05705abe9735a0/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/d6/cn_12s513fj8dymm3xy_tdy9mskn84/T/pip-wheel-dpauxvfu
cwd: /private/var/folders/d6/cn_12s513fj8dymm3xy_tdy9mskn84/T/pip-install-4ev728v7/flowutils_c66ffa7ac87d488f9f05705abe9735a0/
Complete output (31 lines):
running bdist_wheel
running build
running build_py
creating build
creating build/lib.macosx-10.9-x86_64-3.8
creating build/lib.macosx-10.9-x86_64-3.8/flowutils
copying flowutils/transforms.py -> build/lib.macosx-10.9-x86_64-3.8/flowutils
copying flowutils/__init__.py -> build/lib.macosx-10.9-x86_64-3.8/flowutils
copying flowutils/compensate.py -> build/lib.macosx-10.9-x86_64-3.8/flowutils
running build_ext
building 'flowutils.logicle_c' extension
creating build/temp.macosx-10.9-x86_64-3.8
creating build/temp.macosx-10.9-x86_64-3.8/flowutils
creating build/temp.macosx-10.9-x86_64-3.8/flowutils/logicle_c_ext
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/user/opt/anaconda3/envs/sc_env/include -arch x86_64 -I/Users/user/opt/anaconda3/envs/sc_env/include -arch x86_64 -I/Users/user/opt/anaconda3/envs/sc_env/lib/python3.8/site-packages/numpy/core/include -Iflowutils/logicle_c_ext -I/Users/user/opt/anaconda3/envs/sc_env/include/python3.8 -c flowutils/logicle_c_ext/_logicle.c -o build/temp.macosx-10.9-x86_64-3.8/flowutils/logicle_c_ext/_logicle.o -std=c99
In file included from flowutils/logicle_c_ext/_logicle.c:2:
In file included from /Users/user/opt/anaconda3/envs/sc_env/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4:
In file included from /Users/user/opt/anaconda3/envs/sc_env/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12:
In file included from /Users/user/opt/anaconda3/envs/sc_env/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822:
/Users/user/opt/anaconda3/envs/sc_env/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
#warning "Using deprecated NumPy API, disable it with " \
^
flowutils/logicle_c_ext/_logicle.c:120:5: error: implicit declaration of function 'hyperlog_inverse' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
hyperlog_inverse(t, w, m, a, xc, n);
^
flowutils/logicle_c_ext/_logicle.c:120:5: note: did you mean 'wrap_hyperlog_inverse'?
flowutils/logicle_c_ext/_logicle.c:95:18: note: 'wrap_hyperlog_inverse' declared here
static PyObject *wrap_hyperlog_inverse(PyObject *self, PyObject *args) {
^
1 warning and 1 error generated.
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Failed building wheel for flowutils
Running setup.py clean for flowutils
Failed to build flowutils
Installing collected packages: flowutils
Running setup.py install for flowutils ... error
ERROR: Command errored out with exit status 1:
command: /Users/user/opt/anaconda3/envs/sc_env/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/d6/cn_12s513fj8dymm3xy_tdy9mskn84/T/pip-install-4ev728v7/flowutils_c66ffa7ac87d488f9f05705abe9735a0/setup.py'"'"'; __file__='"'"'/private/var/folders/d6/cn_12s513fj8dymm3xy_tdy9mskn84/T/pip-install-4ev728v7/flowutils_c66ffa7ac87d488f9f05705abe9735a0/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/d6/cn_12s513fj8dymm3xy_tdy9mskn84/T/pip-record-jq9qby7j/install-record.txt --single-version-externally-managed --compile --install-headers /Users/user/opt/anaconda3/envs/sc_env/include/python3.8/flowutils
cwd: /private/var/folders/d6/cn_12s513fj8dymm3xy_tdy9mskn84/T/pip-install-4ev728v7/flowutils_c66ffa7ac87d488f9f05705abe9735a0/
Complete output (31 lines):
running install
running build
running build_py
creating build
creating build/lib.macosx-10.9-x86_64-3.8
creating build/lib.macosx-10.9-x86_64-3.8/flowutils
copying flowutils/transforms.py -> build/lib.macosx-10.9-x86_64-3.8/flowutils
copying flowutils/__init__.py -> build/lib.macosx-10.9-x86_64-3.8/flowutils
copying flowutils/compensate.py -> build/lib.macosx-10.9-x86_64-3.8/flowutils
running build_ext
building 'flowutils.logicle_c' extension
creating build/temp.macosx-10.9-x86_64-3.8
creating build/temp.macosx-10.9-x86_64-3.8/flowutils
creating build/temp.macosx-10.9-x86_64-3.8/flowutils/logicle_c_ext
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/user/opt/anaconda3/envs/sc_env/include -arch x86_64 -I/Users/user/opt/anaconda3/envs/sc_env/include -arch x86_64 -I/Users/user/opt/anaconda3/envs/sc_env/lib/python3.8/site-packages/numpy/core/include -Iflowutils/logicle_c_ext -I/Users/user/opt/anaconda3/envs/sc_env/include/python3.8 -c flowutils/logicle_c_ext/_logicle.c -o build/temp.macosx-10.9-x86_64-3.8/flowutils/logicle_c_ext/_logicle.o -std=c99
In file included from flowutils/logicle_c_ext/_logicle.c:2:
In file included from /Users/user/opt/anaconda3/envs/sc_env/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4:
In file included from /Users/user/opt/anaconda3/envs/sc_env/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12:
In file included from /Users/user/opt/anaconda3/envs/sc_env/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822:
/Users/user/opt/anaconda3/envs/sc_env/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
#warning "Using deprecated NumPy API, disable it with " \
^
flowutils/logicle_c_ext/_logicle.c:120:5: error: implicit declaration of function 'hyperlog_inverse' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
hyperlog_inverse(t, w, m, a, xc, n);
^
flowutils/logicle_c_ext/_logicle.c:120:5: note: did you mean 'wrap_hyperlog_inverse'?
flowutils/logicle_c_ext/_logicle.c:95:18: note: 'wrap_hyperlog_inverse' declared here
static PyObject *wrap_hyperlog_inverse(PyObject *self, PyObject *args) {
^
1 warning and 1 error generated.
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /Users/user/opt/anaconda3/envs/sc_env/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/d6/cn_12s513fj8dymm3xy_tdy9mskn84/T/pip-install-4ev728v7/flowutils_c66ffa7ac87d488f9f05705abe9735a0/setup.py'"'"'; __file__='"'"'/private/var/folders/d6/cn_12s513fj8dymm3xy_tdy9mskn84/T/pip-install-4ev728v7/flowutils_c66ffa7ac87d488f9f05705abe9735a0/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/d6/cn_12s513fj8dymm3xy_tdy9mskn84/T/pip-record-jq9qby7j/install-record.txt --single-version-externally-managed --compile --install-headers /Users/user/opt/anaconda3/envs/sc_env/include/python3.8/flowutils Check the logs for full command output.
Hi, I have mac M1 with aarch64.
Previously flowkit and flowutils worked for me on python 3.10, currently it complains on wrong binary in a strange way:
Error message:
ImportError: dlopen(/Users/axelr/pll/venv311/lib/python3.11/site-packages/flowutils/logicle_c.cpython-311-darwin.so, 0x0002): tried: '/Users/axelr/pll/venv311/lib/python3.11/site-packages/flowutils/logicle_c.cpython-311-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/axelr/pll/venv311/lib/python3.11/site-packages/flowutils/logicle_c.cpython-311-darwin.so' (no such file), '/Users/axelr/pll/venv311/lib/python3.11/site-packages/flowutils/logicle_c.cpython-311-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))
Part of traceback:
File "/Users/axelr/pll/venv311/lib/python3.11/site-packages/flowkit/__init__.py", line 4, in <module>
from ._models.sample import Sample
File "/Users/axelr/pll/venv311/lib/python3.11/site-packages/flowkit/_models/sample.py", line 19, in <module>
from .._models.transforms import _transforms
File "/Users/axelr/pll/venv311/lib/python3.11/site-packages/flowkit/_models/transforms/__init__.py", line 2, in <module>
from ._transforms import \
File "/Users/axelr/pll/venv311/lib/python3.11/site-packages/flowkit/_models/transforms/_transforms.py", line 4, in <module>
import flowutils
File "/Users/axelr/pll/venv311/lib/python3.11/site-packages/flowutils/__init__.py", line 4, in <module>
from . import compensate, transforms, gating
File "/Users/axelr/pll/venv311/lib/python3.11/site-packages/flowutils/transforms.py", line 8, in <module>
from . import logicle_c
Setup:
M1
Python 3.11.1 (installed from brew)
Numpy 1.24.1
Flowutils were installed as:
# same error in all three cases
pip install flowutils
pip install flowutils --no-binary flowutils
form source, using instruction here: https://github.com/whitews/FlowUtils#from-github-source-codeHi,
I wanted to use the flowutils logicle transformation in stead of the one provided from flowCore. In both cases i use the default parameters (m=4.5 , t=26144 ,w=0.5, a=0). But the results differ. I expected the same output cause it uses the same transformation function.
Python Code:
flow_data = flowio.FlowData(str(file_path), ignore_offset_error=True)
columns = [channels[key]["PnN"] for key in sorted(channels, key=int)]
events = numpy.reshape(flow_data.events, (-1, flow_data.channel_count))
markers_to_transform = [marker["PnN"] for marker in channels.values() if not(re.match(r"^time$", marker["PnN"], re.IGNORECASE))]
fluoro_indices = []
for channel in flow_data.channels:
if flow_data.channels[channel]['PnN'] in markers_to_transform:
fluoro_indices.append(int(channel) - 1)
fluoro_indices.sort()
transformed_events = flowutils.transforms.logicle(events, fluoro_indices)
df = pandas.DataFrame(transformed_events, columns=columns)Result(df.head(5)):
R Code:
library(flowCore)
path <- "/home/max/Cytolytics/Samples/samples/raw/Sample_C-066C_006.fcs"
flowframe <- read.FCS(path)
tf <- transformList(cols[1:19], logicleTransform(), transformationId = "logicle")
out_frame <- transform(flowframe, tf)Results:
Is there something I'm missing or why is the scaling different.
Thanks and best wishes
Max
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.