novartis / cellxgene-gateway Goto Github PK

Hello everyone,

I am trying to use your software to put online a series of .h5ad files for my lab.

I have created a separate conda environment to run cellxgene gateway and followd the installation instruction, plus a put a bunch of .h5ad files in a custom directory.

the files are correctl seen by cellxgene-gateway. However, when I attempt to open them, I obtain the following error:

The cellxgene gateway stays up, but the port into which is running (5005) goes offline.

On my command shell, I instead obtain the following error messages:

Can you help me figure out what is happening?

Thanks a lot,

Daniele

can I add protected different password for my different datasets?

Hi cellxgene-gateway team,

I'd like to host a cellxgene server that can be accessed by anyone. I'm working from a secure server and I'm not super computer savvy, is there any chance you guys could specify a few of the parameters "for dummies"? Specifically:

EXTERNAL_HOST: this should be the host name of my secure server if I understand correctly, and how do I know which port to use?
GATEWAY_IP - the command socket.gethostbyname(socket.gethostname()) doesn't seem to work for me, is there any other way to find the IP address?
GATEWAY_PORT: how do I know what this should be?

Any help would be greatly appreciated! And really cool that you guys set this up.

Best,
Lisa

Allowing uploads of data is a potential security risk on the open internet. The proposal here is to disable uploads by default and add an add GATEWAY_ENABLE_UPLOAD environment variable to enable them. It is critical that the environment variable disables all the routes open to abuse (eg, mkdir).

Hi !
This tool is very useful, but I encountered some problems in the process of using it.
I have Mouse.h5ad in the file, but I can't open it.
The error is as follows：

Hello, your project is amazing and we've had success self-hosting with it internally. We're currently going through the code and adding some reformatting to the filecrawl page using filecrawl.html and filecrawl.py files. We are looking into how to reformat the item tree portion of the page. We might create many annotations for one data set so we'd like to put the "new" make annotation link to always appear at the front of the annotations list instead of the end

Trying to use annotations, I have a server setup with a reverse proxy, i'll substitute the domain I'm using with example.com

When I click on the dataset to view without annotation it works fine. When i click 'new' I get the box to write the name. But then the browser is redirected to a URL like:

"GET /view/10x/pbmc3k_annotations/:/example.com:5005/view/10x/pbmc3k_annotations/:/view/10x/pbmc3k_annotations/test01.csv/ HTTP/1.1" 404 -

Any ideas?

Thanks

EDIT:
if I then strip the URL to just the domain and the part after the final colon (/view/10x/pbmc3k_annotations/test01.csv/) it's working

Setup:

cellxgene --version
[cellxgene] Version 0.16.6

python -c 'import cellxgene_gateway; print(cellxgene_gateway.version)'
0.2.2

export CELLXGENE_LOCATION=/usr/local/bin/cellxgene
export CELLXGENE_DATA=/data/cellxgene_data
export EXTERNAL_HOST='example.com:5005'
export EXTERNAL_PROTOCOL='https'
export GATEWAY_ENABLE_ANNOTATIONS=true
export GATEWAY_ENABLE_BACKED_MODE=true

If you miss the exact moment when the counter hits 5 it never reloads.

Currently a given dataset get stuck in a failed mode if the cellxgene server fails.

Proposed fix:
Add a "relaunch" endpoint that will terminate the current server and mark the cache entry as "terminated", then redirect to the "view" endpoint.
We can add a link to the "relaunch" page to the failure page.
The actual cache entry will be pruned by the background thread since we don't want to actually wait for the server to shutdown.

Major impact:
New "terminated" cache entry status will need to be accounted for in other code.

Previously I could have subdirectories in cellxgene_data/ and linking to the subdirectory would only show the h5ad files in that subdirectory. But after updating beyond 0.2.3 clicking into a subdirectory is still showing the full filecrawl for all subdirectories?

Here I have clicked on 01, but it still shows the files for 02/ etc...

Just an FYI... I'm going to try and find some time to deep-dive this, but maybe y'all can find the problem quicker?

Currently we display an index page for the entire cellxgene_data folder. The proposal is to display indexes for intermediate folders if they are entered.

subdir gets printed in h5ad filename and it says it couldn't find the item and lists the subdir twice

if I have multiple nested subdirs... cellxgene-gateway fails to run with a exception S3 url doesn't exist and lists the first subdir twice in the exception...
getting: Exception: S3 url 's3://XXX/PublishedData/PublishedData/2018CellReports' does not exist.

The "PublishedData" folder only exists once...

installing on Ubuntu/WSL, installation fails due to a typo in setup.py. Root cause, I think: readme file name has different case in setup.py.

• file in repo is Readme.md
• file in setup.py is README.md

so on a posix filesystem, they are different file names.

Relevant portions of the install output, showing the error:

(venv) # pip install git+https://github.com/Novartis/cellxgene-gateway
Collecting git+https://github.com/Novartis/cellxgene-gateway
  Cloning https://github.com/Novartis/cellxgene-gateway to /tmp/pip-req-build-qb38q_fr
Requirement already satisfied: cellxgene in ./venv/lib/python3.6/site-packages (from cellxgene-gateway==0.1) (0.11.2)
Requirement already satisfied: flask in ./venv/lib/python3.6/site-packages (from cellxgene-gateway==0.1) (1.1.1)

[SNIP]


    byte-compiling /mnt/c/Users/bruce/projects/cellxgene-gateway/venv/lib/python3.6/site-packages/cellxgene_gateway/subprocess_backend.py to subprocess_backend.cpython-36.pyc
    byte-compiling /mnt/c/Users/bruce/projects/cellxgene-gateway/venv/lib/python3.6/site-packages/cellxgene_gateway/util.py to util.cpython-36.pyc
    running install_data
    error: can't copy 'README.md': doesn't exist or not a regular file
    
    ----------------------------------------
Command "/mnt/c/Users/bruce/projects/cellxgene-gateway/venv/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-req-build-qb38q_fr/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-n3hx8scf/install-record.txt --single-version-externally-managed --compile --install-headers /mnt/c/Users/bruce/projects/cellxgene-gateway/venv/include/site/python3.6/cellxgene-gateway" failed with error code 1 in /tmp/pip-req-build-qb38q_fr/

with uptime in seconds, launch time in ISO9601, and cache stats

A bug was likely introduced in v0.3.2 causing process cache pruning to fail with error like the following:

 Traceback (most recent call last):
   File "/home/docker/anaconda/envs/VIP/lib/python3.8/site-packages/cellxgene_gateway/prune_process_cache.py", line 42, in prune
     logger.info(f"pruning process {process.pid} ({process.key.dataset})")
 AttributeError: 'CacheKey' object has no attribute 'dataset'

This will enable users to free up memory and also reload data if they have changed it on disk.

Hi, my lab is trying to set up a self-hosted cellxgene instance via reverse proxy and we've run into this issue below. Our current goal is to at least get an instance of cellxgene running remotely via a VM. We've tried using the cellxgene-gateway command but localhost:5005 gives us a "connection refused" error. We've tried troubleshooting this problem by turning off our firewall, enabling ports, and allowing SELinux to permissive but none of these methods have worked. Additionally, cellxgene also gives us an error when trying to launch: "unable to retrieve http://localhost:5005/: Connection refused". Any help would be greatly appreciated, because we're a bit stuck!

[exampleuser ~]$ export CELLXGENE_LOCATION=/usr/local/bin/cellxgene
[exampleuser ~]$ export CELLXGENE_DATA=../cellxgene_data
[exampleuser ~]$ cellxgene-gateway
2022-07-15 09:48:56,187:cellxgene_gateway:INFO:Got required env: {'CELLXGENE_LOCATION': '/usr/local/bin/cellxgene'}
2022-07-15 09:48:56,187:cellxgene_gateway:INFO:Got optional env: {'EXTERNAL_HOST': 'localhost:5005', 'EXTERNAL_PROTOCOL': None, 'GATEWAY_IP': None, 'GATEWAY_PORT': 5005, 'GATEWAY_EXTRA_SCRIPTS': None, 'GATEWAY_TTL': None, 'GATEWAY_ENABLE_ANNOTATIONS': False, 'GATEWAY_ENABLE_BACKED_MODE': False, 'CELLXGENE_ARGS': None, 'CELLXGENE_DATA': '../cellxgene_data', 'PROXY_FIX_FOR': 0, 'PROXY_FIX_PROTO': 0, 'PROXY_FIX_HOST': 0, 'PROXY_FIX_PORT': 0, 'PROXY_FIX_PREFIX': 0}

• Serving Flask app "cellxgene_gateway.gateway" (lazy loading)
• Environment: production
  WARNING: This is a development server. Do not use it in a production deployment.
  Use a production WSGI server instead.
• Debug mode: off
  2022-07-15 09:48:56,192:werkzeug:INFO: * Running on http://0.0.0.0:5005/

Hi,

I am running the gateway with cellxgene==1.0.0 on an AWS EC2 instance using docker. If I open the .h5ad files on S3 storage, the annotation and gene-set files are created but they aren't filled in. The EC2 instance has full read/write access to the S3 bucket. I also keep getting a 500 error every 2-3 seconds once I start annotating the data:

2022-02-15 15:11:58,004:werkzeug:INFO:10.252.248.24 - - [15/Feb/2022 15:11:58] "PUT /source/s3/view/Human_PBMC_HO_aggr_annotations/berguner.csv/api/v0.2/annotations/obs?annotation-collection-name=berguner HTTP/1.1" 500 -

This doesn't happen while working on local data. Do you have any idea why this might happen? How can I debug this?

https://chanzuckerberg.github.io/cellxgene/posts/annotations

Hi,

I installed cellxgene-gateway in its own conda environment on Windows 10.
After I try to run cellxgene-gateway, all the .h5ad files show up in the file crawler.
However, clicking any one of them gives me the following error:

This is the output in the terminal:

2021-09-29 10:00:18,469:werkzeug:INFO: * Running on http://0.0.0.0:5005/ (Press CTRL+C to quit) 2021-09-29 10:00:24,521:werkzeug:INFO:127.0.0.1 - - [29/Sep/2021 10:00:24] "←[37mGET / HTTP/1.1←[0m" 200 - 2021-09-29 10:00:26,100:werkzeug:INFO:127.0.0.1 - - [29/Sep/2021 10:00:26] "←[37mGET /filecrawl.html HTTP/1.1←[0m" 200 - view path=My_AnnData.h5ad/, source_name=None, dataset=Z:\AnnData\My_AnnData.h5ad, annotation_file= None, key=My_AnnData.h5ad, source=local

2021-09-29 10:00:30,877:cellxgene_gateway:INFO:launching yes | C:\foobarbaz\anaconda3\envs\cellbygene-gateway\Scripts\cellxgene.exe launch Z:\AnnData\My_AnnData.h5ad --port 8000 --host 127.0.0.1 --disable-annotations

Exception in thread Thread-6: Traceback (most recent call last): File "c:\foobarbaz\anaconda3\envs\cellbygene-gateway\lib\threading.py", line 926, in _bootstrap_inner self.run() File "c:\foobarbaz\anaconda3\envs\cellbygene-gateway\lib\threading.py", line 870, in run self._target(*self._args, **self._kwargs) File "c:\foobarbaz\anaconda3\envs\cellbygene-gateway\lib\site-packages\cellxgene_gateway\subprocess_backend.py", line 82, in launch raise ProcessException.from_cache_entry(cache_entry) cellxgene_gateway.process_exception.ProcessException

These are my conda environment variables

CELLXGENE_DATA = Z:\AnnData
CELLXGENE_LOCATION = C:\foobarbaz\anaconda3\envs\cellbygene-gateway\Scripts\cellxgene.exe

This error happened already yesterday, when I was able to run cellxgene launch My_AnnData.h5ad successfully. The problem seems therefore unrelated to today's cellxgene bug.
To confirm: conda env config vars set CELLXGENE_ARGS=--no-upgrade-check also does not solve the problem.
Any suggestions would be appreciated!

When a file has many annotations, cellxgene gateway will hang after it is reloaded a few times. This seems to be fixed by reading from the stderr and stdout of the process until it exits.

There was a bug in referring to the cache object.

Error https://github.com/Novartis/cellxgene-gateway/runs/5547618966?check_suite_focus=true
The issue is that jinja2-2.11.3 calls soft_unicode in markupsafe, but "soft_unicode" was removed in latest markupsafe (v2.1)

See similar issue aws/aws-sam-cli#3661

This will create a seam that allows subclasses to modify the process creation to e.g. inject credentials. This literally covers moving the subprocess.Popen from the launch mention into a separate method.

Hello,

Thank you for developing this beautiful software. Is there any way to combine cellxgene-gateway and cellxgene_VIP?

Best wishes,
Polina

Hi, and again congratulation for this wonderful tool, it has been very helpful so far

Would it be possible for you to include the reading of .cxgs (such as the one derived from cellxgene convert ( in the upcoming releases)?

I'm asking this because we found a pesky bug in cellxgene that can only be solved by converting the .h5ad to a .cxg file and insofar the developers told me the bugfix would tike quite along time.

Thanks for your help!

Daniele

Hello, I like your gateway. I was wondering if there was a way to save the state of an analysis, not just gene sets I have created. (Or is this a question to ask at the cellxgene github?) Thank you in advance.

Edit: I also can't seem to find a way to save a selected subset of cells, that seems like it would be a nice feature.

Initial report:

The file crawler lists all of the file, but the links / URLs in the files are wrong.
When I click on a file name, I get a 404. The reason is that the URLs are missing a slash. Example:
a file called pbmc3k_processed.h5ad has a link http://localhost:5005/viewpbmc3k/pmbc3k_processed.h5ad Note the missing slash between view and pbmc3k
If I manually correct that URL in my browser, cellxgene loads fine.

The underlying issue is that the CELLXGENE_DATA had a trailing slash... the code should be resilient to this variation.

hello,

I managed to run cellxgene-gateway on my local server.

Everything seems to work fine except for the GATEWAY_ENABLE_ANNOTATIONS variable. I set it to 1 in order to enable annotations. Additionally, I passed the --annotations-dir (with a valid path) to the CELLXGENE_ARGS variable.

When I launch cellxgene-gateway I see that GATEWAY-ENABLE-ANNOTATIONS is set to True.

however, each cellxgene process is running with the --disable-annotations flag (--annotations-dir is passed correctly).

For the record, I'm running the tool into a conda environment and cellxgene version is 16.04.

Glad if you could provide any help on this.

Daniele

Hello and thanks for a great tool.

I'm currently using our tool to display data in s3 folder.

In case when a new file is produced and uploaded to s3, the list of files is not updated and there is no way to display that data. Unfortunately, restarting gateway is non-trivial. Can you consider adding a button that would update the list of files? Thank you!

Hi cellxgene-gateway team,
I'd like to host a cellxgene server that can be accessed by anyone. But I'm not super computer savvy.I changed some environment variables, but still cannot be accessed through the public network ip:202...**:8516.

Thanks

With some investigation, this is due to the assumption that the subprocess has the original process id + 2. Possible remediation steps include

• adjust launch procedure so only a single subprocess is launched
• check ports are clear before launching processes
• do not clear cache entries that do not have process ids

On a fresh install of cellxgene-gateway, I get the following error:

ERROR: numba 0.54.0 has requirement numpy<1.21,>=1.17, but you'll have numpy 1.21.2 which is incompatible.

However, if I install celllxgene version 0.18.0, and then upgrade to cellxgene 0.19.0, it seems to work. This is because numba is downgraded to 0.52.0 by this process. Thus a current workaround would be to downgrade numba to 0.52.0

Currently we use environment variables to indicate the external protocol and host. These values could also be taken from Host and X-Forwarded-Proto headers.
https://flask.palletsprojects.com/en/0.12.x/deploying/wsgi-standalone/#proxy-setups

This is to make it easy to upload data programmatically when the servers are behind a load balancer.

Hi,
We would like to have it running in our k8s cluster, is it possible that we spin up a new container for each dataset?
Thanks.

Hi,

Thank you for developing the application! cellxgene-gateway deploys the browser in development; how to deploy this on production server?

Thanks!
Parul

First of all, great tool you created! Very useful.

I'm hosting the gateway on our hpc and it is working but I have the issue that is not pointing to the right IP address (although when you type it in in chrome it actually loads under this IP address).
This is quite annoying because every time I want to open a dataset from the file crawler it opens it under localhost and I have to change it to the correct IP address.
Any idea on your side how we can fix this?

(cxg_VIP) [glanl@by08tq ~]$ cellxgene-gateway
2020-09-23 17:46:13,765:cellxgene_gateway:INFO:Got required env: {'CELLXGENE_LOCATION': '/gpfs01/home/glanl/miniconda3/envs/cxg_VIP/bin/cellxgene', 'CELLXGENE_DATA': '/gpfs01/bhcbio/flatfile/single_cell_datasets'}
2020-09-23 17:46:13,765:cellxgene_gateway:INFO:Got optional env: {'EXTERNAL_HOST': '10.190.91.50:5005', 'EXTERNAL_PROTOCOL': 'http', 'GATEWAY_IP': '10.190.91.50', 'GATEWAY_PORT': 5005, 'GATEWAY_EXTRA_SCRIPTS': None, 'GATEWAY_TTL': None, 'GATEWAY_ENABLE_UPLOAD': False, 'GATEWAY_ENABLE_ANNOTATIONS': True, 'GATEWAY_ENABLE_BACKED_MODE': False, 'CELLXGENE_ARGS': None}
 * Serving Flask app "cellxgene_gateway.gateway" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
2020-09-23 17:46:13,769:werkzeug:INFO: * Running on http://0.0.0.0:5005/ (Press CTRL+C to quit)

If a directory path is more than two elements deep, then the second element is used for all deeper entries.
For example, if there is a file at "foo/bar/baz/test.h5ad", the directory entry for baz will show up as bar.

running a clean install of cellxgene-gateway==0.3.7, ships with cellxgene==0.19.0

when I click a dataset in the File Crawler, the server refuses to launch cellxgene:

Traceback (most recent call last):
File "/usr/local/bin/cellxgene", line 8, in <module> sys.exit(cli())
  File "/home/ubuntu/.local/lib/python3.8/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) 
    File "/home/ubuntu/.local/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx)
      File "/home/ubuntu/.local/lib/python3.8/site-packages/click/core.py", line 1256, in invoke Command.invoke(self, ctx)
        File "/home/ubuntu/.local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params)
          File "/home/ubuntu/.local/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) 
            File "/usr/local/lib/python3.8/dist-packages/server/cli/cli.py", line 30, in cli log_upgrade_check()
              File "/usr/local/lib/python3.8/dist-packages/server/cli/upgrade.py", line 25, in log_upgrade_check if version_gt(latest_release, __version__):
                File "/usr/local/lib/python3.8/dist-packages/server/cli/upgrade.py", line 80, in version_gt for left, right in zip(split_version(left_version), split_version(right_version)):
                  File "/usr/local/lib/python3.8/dist-packages/server/cli/upgrade.py", line 76, in split_version return [int(match.group(group)) for group in ["major", "minor", "patch"]]
                    File "/usr/local/lib/python3.8/dist-packages/server/cli/upgrade.py", line 76, in <listcomp> return [int(match.group(group)) for group in ["major", "minor", "patch"]]

AttributeError: 'NoneType' object has no attribute 'group'

as far as I can tell, cellxgene-gateway keeps looking for a sever update or upgrade and has not formatted the query correctly...?

Sometimes it is handy to override the cellxgene configuration in jobs that are launched. This could be accomplished by adding a new CELLXGENE_CONFIG environment variable that could be used to pass the --config-file parameter to cellxgene processes when they are launched.

Hi,
I tried the option 1 and 1 in Install Cellxgene-gateway setion, but cannot install due to AttributeError: type object 'Distribution' has no attribute '_finalize_feature_opts'.
Then I tried the Development install, and in the step 3 got the same error.
Please help to resolve it. Thank you~

Example:

if I want to launch Cellxgene using the following command args

--disable-diffexp
--disable-annotations
--max-category-items 500

This could be implemented using just one env variable such as "CELLXGENE_ARGS" perhaps?

Thanks

I'm setup with a reverse proxy and version 0.2.2 had no issue, after upgrading to 0.2.3 I have an error when I try to create a new annotation, loading existing annotations seems fine.

export CELLXGENE_LOCATION=/home/jonathan/venv/main/bin/cellxgene
export CELLXGENE_DATA=/data/cellxgene_data
export EXTERNAL_HOST='my.url.com'
export EXTERNAL_PROTOCOL='https'
export GATEWAY_ENABLE_ANNOTATIONS=true
export GATEWAY_ENABLE_BACKED_MODE=true

cellxgene-gateway

cellxgene_gateway:INFO:launching yes | /home/jonathan/venv/main/bin/cellxgene launch /data/cellxgene_data/10x/pbmc3k.h5ad --port 8010 --host 127.0.0.1 --annotations-file /data/cellxgene_data/10x/pbmc3k_annotations/test2.csv --backed
werkzeug:INFO:IP - - [26/Mar/2021 09:51:38] "GET /view/10x/pbmc3k_annotations/test2.csv HTTP/1.1" 302 -
2021-03-26 09:51:38,687:werkzeug:ERROR:IP- - [26/Mar/2021 09:51:38] code 400, message Bad request syntax ('\x16\x03\x01\x02\x00\x01\x00\x01ü\x03\x03\x83\x1dú\x84¢ Fd\x8b¿\x88\x1f»\x14\x05{³\x15É\x17\x8e-BMåÿPø\x03²ú ^\x8b£)\\ou«²W\x11\x8b3à\x87_ês')
2021-03-26 09:51:38,687:werkzeug:INFO:IP- - [26/Mar/2021 09:51:38] "ü�ú�¢ Fd�¿�»{³É�-BMåÿPø²ú ^�£)\ou«²W�3à�_ês" HTTPStatus.BAD_REQUEST -
2021-03-26 09:51:38,714:werkzeug:ERROR:IP- - [26/Mar/2021 09:51:38] code 400, message Bad request syntax ('\x16\x03\x01\x02\x00\x01\x00\x01ü\x03\x03m2Ó\x10äèQõQÊ\x1a\x1a°¬Æ;\x19¢"\x00û <1UUû\x8eÖTk- àb¢ö\x97Ùçe]\x98\rR*7!\x0b\x98\t*ÁZw\x96j¯Æ§Ø\x8d¼')
R*7!-03-26 09:51:38,714:werkzeug:INFO:IP- - [26/Mar/2021 09:51:38] "üm2ÓäèQõQÊ°¬Æ;¢"û <1UUû�ÖTk- àb¢ö�Ùçe

Although is possible to provide the env variable: GATEWAY_HOST

port and ip are ignored and seen in line: https://github.com/Novartis/cellxgene-gateway/blob/master/cellxgene_gateway/gateway.py#L235

I was wondering how we could keep the cache process for longer?

I saw in 'prune_process_cache.py' line 21:

self.expire_seconds = 3600

Please could you let me know if I could change this value to something larger, e.g. 7200? Is there any other place I need to change?

Many thanks

Brian

File crawler order appears to be based on created time?

Anyway to get the option to order alphabetically?

We can populate this at the transition from loading->loaded to keep the overhead low.