Git Product home page Git Product logo

micas's Introduction

MICAS v0.1 Manual

  1. About MICAS
  2. Installation
    2.1. Automated Installation
    2.2. Manual Installation
  3. Running MICAS
    3.1. MICAS Setup
    3.2. MICAS Analysis
  4. Example Workflow
  5. Citation
  6. Feedback and bug reports

1. About MICAS

MinION Classification & Alerting System is a web application meant to be run simultaneously with the MinION DNA sequencer. This app provides an alerting system through which a scientist performing DNA sequencing runs could be notified minknow style notifications or via a log file. The alerts could be set to respond to any particular sequences of interest arising in their sample. Our team believes that this will enable researchers to use their time more efficiently by allowing them to focus on more important matters in the meantime, rather than waiting around for significant sequences.

2. Installation

There are two ways to get up-and-running with MICAS. You can either install it using our automated installation script. It will detect the current environment and install required packages accordingly. Otherwise, you can follow our detailed guide on how to install the software.

2.1. Automated Installation

The automated installation script has been developed for MacOS and Linux users who would like to get MICAS ready without worrying about what components are required to get it to work. The script has been designed to detect the environment as well as the programs that are installed and adjusts its commands accordingly. Assuming MICAS is being downloaded from GitHub, here are the set of commands one has to run to initiate the installation script, which will take care of remaining steps:

  1. Cloning the latest MICAS software

    git clone https://github.com/coadunate/MICAS.git

  2. Entering into the cloned directory

    cd MICAS

  3. Running the installation script

    ./install.sh

After the installation has finished, you should be ready to start using MICAS.

2.2. Manual Installation

3. Running MICAS

Once you have successfully installed MICAS and are ready to run it, you can run the pre-made script called start.sh, which should run the necessary programs and serve the webpage at:http://127.0.0.1:5000/. When you subscribe to the above URL on your favourite web browser, you'll land on setup page. This indicates that MICAS installation was successful and you are ready to do science.

NOTE: Unfortunately, the start.sh script currently does not function, and needs to be fixed. Please use the bellow instructions regarding "Running MICAS For Debugging".

3.1. Running MICAS For Debugging

Occasionally, one may feel inclined to run each of the individual portions of MICAS within there own terminals, for easier error hanbdling. This can be achieved by running the following commands, in any order:

Start Redis server

redis-server

This step could fail if redis is not installed.

Start Celery

cd ./server/app/main/utils
celery -A tasks worker --loglevel=INFO

This step could fail if celery is not installed.

Start Front End

cd ./frontend
npm run start

Start MICAS Server

python server/micas.py

With all the above running in seperate terminals, you can now delight in viewing each applications error streams with relative ease. NOTE: The above is completely optional, and use of start.sh is reccomended for most users.

3.2. MICAS Setup

The setup page is meant to setup the alerting system for a particular organism for a particular MinION run. There are three components of the setup page which need to be filled in with relevent information before analysis.

1. Database Selection
In this section you can select the database within which to search for your alerts in your MinION run. For example, if you performing MinION run of nCOVID-19 virus, you would most certainly want to check off Virus in your database selection.

You also have a choice to insert your own FASTA sequences as databases, which may or may not be included as alerts.

2. Alerting
Once the database selection process has been complete, you can configure your alerting information. The primary notification system is the via notification through logs, indicated by the micas location, and via the UI on minknow GUI. There is also a possibility to add more alering sequences in this component, which will be monitored during the MinION run.

3. Configuration
Finally, in the confiuration section, you have to choose the directory to which MinION reads will be deposited so that MICAS can access it and analyze. You also have to provide an empty directory where MICAS can store its database and other temporary files.

3.3. MICAS Analysis

4. Example Workflow

5. Citation

6. Feedback and bug reports

micas's People

Contributors

dependabot[bot] avatar matthewlinks avatar syncrotron avatar tayabsoomro avatar

Stargazers

avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar

Forkers

tayabsoomro

micas's Issues

Experiment centrifuge run with combined and individual indexes

Do an centrifuge experiment with combined index including bacterial genomes and B. anthracis reads and get the centrifuge output file. Do another two centrifuge runs with two different indexes: one with bacterial genomes and another with B. anthracis reads and get the centrifuge output file. Merge the two centrifuge files that you get from the individual runs and compare that with the original combined run to see if there is a difference. If there isn't, we can use this approach for centrifuge classification, because it will reduce the bacterial database downloading step as we can provide user's with NCBI genomes indexes.

Send task for sequence classification to the background

The set of commands that do the classification step by running centrifuge command take longer for the longer databases. Because of that, it takes a long time to populate the classification files, which the UI uses to show the visualizations.

Those set of commands should be run in the background through celery because when ~ 4000 files are going to be dumped into the minion_reads location, the application is going to crash while dealing with the classification of that many files in the foreground process, which I experienced the other day when my computer shut down thrice while doing the classification.

Unhandled Rejection (Error): Network Error

When running MICAS in docker with the hosts root bound to /MICAS/data (to allow MICAS to grab sequence data as it becomes available), MICAS encounters an unhandled error once the button is pressed after an email is provided in setup. The error is as follows:

Unhandled Rejection (Error): Network Error
validateLocations
src/modules/setup/setup-steps/summary/summary.component.tsx:33

  30 | locationData.append('App', locations.micasLocation);
  31 | locationData.append('Queries', queryFiles);
  32 | 
> 33 | return axios({
     | ^  34 |     method: 'POST',
  35 |     url: 'http://localhost:5000/validate_locations',
  36 |     data: locationData,

./src/modules/setup/setup-steps/summary/summary.component.tsx/</SummaryComponent/</<
src/modules/setup/setup-steps/summary/summary.component.tsx:77

  74 | 
  75 | useEffect(() => {
  76 |     (async () => {
> 77 |         const res = await validateLocations(add_databases, databaseSetupInput.locations)
     | ^  78 |         const v_code = res.data.code
  79 | 
  80 |         if (v_code === 0) { // if locations are valid

./src/modules/setup/setup-steps/summary/summary.component.tsx/</SummaryComponent/<
src/modules/setup/setup-steps/summary/summary.component.tsx:94

  91 |             console.log(data)
  92 |         })
  93 | 
> 94 |     })();
     | ^  95 | 
  96 | }, [started])
  97 |

Perhaps this is an error in the locations provided, however I checked there validity from the running container, and everything looks fine from there.

Be able to re-start the polilng system on any project

Currently, the only way to start the polling system is to go through the database setup process. However, there are going to be times where we need to resume the analysis on an old project. For that, we would like to have some functionality in the main UI to be able to revive any old projects and start the polling on that project so that any new reads are going to be analyzed.

Download database progress report

Add a functionality to get the # of reads that are left to be downloaded from the total number of reads. This can be done by parsing the file download_err.txt file inside micas_data/database directory.

SankeyPlot should also show unclassified reads

Currently the code responsible for generating JSON data, extracted for Kraken report file, for Sankey visualization discards the unclassified reads and for that reason, no visualization is shown until a sequence is classified. We ideally would also want to show a branch of unclassified reads along with the reads that have been classified.

Here is the message that gets shown when there are only unclassified reads:

screen shot 2018-08-11 at 11 31 30 pm

Remove magic indexing from server/app/main/utils/FASTQFileHandler.py

Within FASTQFileHandler, os.popen calls awk print to dump the contents of the minimap2 report file. To find match_alert_header, num_match, and tot_len (of which later two are used in presence threshold calculations) are found within the awk pipe via magic indexing. If awk was ever to fail or produce something unexpected, the indexing would fail, causing MICAS to fail to notify the end user of a threshold being reached.

This could be fixed via some form of named indexing or error handling.

Set up config file for Twilio

Create a config file that MICAS reads for sending alerts to user's cellphone. The config file will contain the information required by Twilio to send a SMS message.

Also integrate the sending of SMS message upon alert detection.

Sankey visualization is not legible for porecamp data

The Sankey visualization needs to be altered in a way that it is legible, because with sequence runs having longer kraken report outputs, the Sankey visualization becomes less legible. Example shown below for porecamp data:

screen shot 2018-08-13 at 7 18 53 pm

SankeyPlot branch width

Fix SankeyPlot branch width so that it is proportional to the number of reads that a particular branch contains.

This includes thinking of a way to normalize the number of reads that a branch contains. It is currently set so that the normalized values are with range [0,20] and the input values are the "number of fragments assigned directly to this taxon", the value is derived from kraken output.

The formula is y = 0.2x where x is the number of fragments assigned directly to this taxon and y is the normalized width of a particular branch.

Change alert threshold visualization code from fusioncharts to D3js

The alert threshold visualization currently uses fusioncharts. They are used because of their ability to draw a line horizontally through a vertical bar chart indicating the threshold value. For consistency purposes, the alert threshold visualization needs to be implemented in react-vis as it is being used by every other visualization in the application.

Update depreciated requirments

Many of the python packages used throughout the project have since been updated. A review of these depreciated packages must be made, with updates to current versions of the packages where possible.

Briefly addressed in issue #32, but further work is needed.

Stream the celery output into the webpage

The celery output which gives information on the database downloading needs to be added into the web interface so that the users are aware of the downloading process. Here's an example of the celery log:

[2021-06-06 14:43:34,061: INFO/MainProcess] Connected to redis://localhost:6379//
[2021-06-06 14:43:34,071: INFO/MainProcess] mingle: searching for neighbors
[2021-06-06 14:43:35,095: INFO/MainProcess] mingle: all alone
[2021-06-06 14:43:35,107: INFO/MainProcess] [email protected] ready.
[2021-06-06 14:43:35,109: INFO/MainProcess] Task app.main.tasks.int_download_database[7bb9333c-265d-4a2b-bd99-1f1c04dc9e13] received
[2021-06-06 14:43:35,111: WARNING/ForkPoolWorker-8] DOWNLOAD DATABASE: Downloading taxonomy files
Downloading NCBI taxonomy ...
x nodes.dmp
x names.dmp
[2021-06-06 14:43:48,224: WARNING/ForkPoolWorker-8] DOWNLOAD DATABASE: Successfully downloaded taxonomy files
[2021-06-06 14:43:48,225: WARNING/ForkPoolWorker-8] DOWNLOAD DATABASE: Created seqid2taxid.map file
[2021-06-06 14:43:48,227: WARNING/ForkPoolWorker-8] DOWNLOAD DATABASE: No queries provided, skipping.
[2021-06-06 14:43:48,227: WARNING/ForkPoolWorker-8] DOWNLOAD_DATABASE: Downloading bacteria database(s) from NCBI
[2021-06-06 14:43:48,228: WARNING/ForkPoolWorker-8] CMD:
[2021-06-06 14:43:48,228: WARNING/ForkPoolWorker-8] ['centrifuge-download -o /Users/tayabsoomro/Documents/Projects/Files-MICAS/micas_data/database/library -m -d bacteria refseq']

Central Logging System

There are various tools that MICAS uses and the status, process and warnings and progress reports from these tools need to be reported to the user for documentation purposes. Therefore, we need a central logging system for MICAS, to which every tool sends log messages. These messages then need to be displayed on the front-end for the user to see.

SocketIO blocked by CORS

Description
When I load the front-end webpage at localhost:3000/, I get the following error:

Access to XMLHttpRequest at 'http://localhost:5000/socket.io/?EIO=3&transport=polling&t=NnW2sUh' from origin 'http://127.0.0.1:3000' has been blocked by CORS policy: The value of the 'Access-Control-Allow-Credentials' header in the response is '' which must be 'true' when the request's credentials mode is 'include'. The credentials mode of requests initiated by the XMLHttpRequest is controlled by the withCredentials attribute.

The error seems to suggest that the front-end (localhost:3000) SocketIO resources are being blocked by the back-end API (localhost:5000), even though the cross-origin resource sharing (CORS) has been enabled for the localhost:3000 (https://github.com/coadunate/MICAS/blob/master/server/app/__init__.py#L19)

Make the timeline functional

The timeline component of this application needs to be functional. Need to put some thought on how to record the comparison of time between the MinION and MICAS.

Porecamp data run

Do a porecamp data run with a script that automatically pumps data into minion reads directory to mimic minion run. List out the issues that you find during the run.

Database index is not being built.

After the database gets downloaded, the index file doesn't get built. Here is the error that it throws:

No output file specified!
Centrifuge version 1.0.4 by Daehwan Kim ([email protected], http://www.ccb.jhu.edu/people/infphilo)
Usage: centrifuge-build [options]* --conversion-table <table file> --taxonomy-tree <taxonomy tree file> <reference_in> <cf_index_base>
    reference_in            comma-separated list of files with ref sequences
    centrifuge_index_base          write cf data to files with this dir/basename
Options:
    -c                      reference sequences given on cmd line (as
                            <reference_in>)
    -a/--noauto             disable automatic -p/--bmax/--dcv memory-fitting

...

Convert project to exclusivly use conda

Right now we are using both python virtual environments and conda, which splinters development and requirement tracking. There needs to be some effort to unify the virtual environments used within the project to only utilize conda. Thus, the following changes need to me made:

  • Remove all references to python virtualenv from the install.sh
  • Generate YAML file for conda requirments

npm install "Cannot find module [X]"

Occasionally, when you run the install script or attempt to npm install from inside the frontend folder, npm will get confused and throw an error message like:
Cannot find module 'react-dev-utils/crossSpawn'

A temporary workaround is to delete the node_modules from within frontend, however a more long term solution should be found.

Add a check to see if the URL parameters are present before getting them

Current State
There are various instances in routes.py file where we are fetching the URL parameters without checking to see whether they exist. This causes the API to throw a python error which does not make sense.

Solution
In order to avoid that, we need to add various checks to ensure that the parameters that we are accessing actually do exist in the dictionary.

Retrieving data from sankey.data

Need to update the code at route: /get_sankey_data in routes.py which imports the Sankey visualization data. The current implementation assumes that everything inside sankey.data is in one line, which it should not. The code should read all lines and parse everything as JSON object.

Set up email configuration

Setup and test email configuration so that email is sent out (if email notifications are enabled by the user) when alert is detected during the sequence run.

Celery throws EOF error when downloading database

Celery throws following error when downloading database:

[2021-07-26 16:19:12,635: WARNING/ForkPoolWorker-8] DOWNLOAD DATABASE: Successfully downloaded taxonomy files
[2021-07-26 16:19:12,636: WARNING/ForkPoolWorker-8] DOWNLOAD DATABASE: Created seqid2taxid.map file
[2021-07-26 16:19:12,637: WARNING/ForkPoolWorker-8] DOWNLOAD DATABASE: Entering queries into taxonomy files.
[2021-07-26 16:19:14,836: ERROR/ForkPoolWorker-8] Task app.main.tasks.int_download_database[7016c878-fe6b-4698-b7e4-aa7958c10346] raised unexpected: SyntaxError('unexpected EOF while parsing', ('<unknown>', 0, 0, ''))
Traceback (most recent call last):
  File "/Users/tayabsoomro/anaconda3/envs/micas_env/lib/python3.6/site-packages/celery/app/trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/Users/tayabsoomro/anaconda3/envs/micas_env/lib/python3.6/site-packages/celery/app/trace.py", line 650, in __protected_call__
    return self.run(*args, **kwargs)
  File "/Users/tayabsoomro/Documents/Projects/MICAS-Coadunate/server/app/main/utils/tasks.py", line 86, in int_download_database
    list_of_sequences = ast.literal_eval(list_of_sequences.strip())
  File "/Users/tayabsoomro/anaconda3/envs/micas_env/lib/python3.6/ast.py", line 48, in literal_eval
    node_or_string = parse(node_or_string, mode='eval')
  File "/Users/tayabsoomro/anaconda3/envs/micas_env/lib/python3.6/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 0

    ^
SyntaxError: unexpected EOF while parsing
 SyntaxError('unexpected EOF while parsing', ('<unknown>', 0, 0, ''))
Traceback (most recent call last):
  File "/Users/tayabsoomro/anaconda3/envs/micas_env/lib/python3.6/site-packages/celery/app/trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/Users/tayabsoomro/anaconda3/envs/micas_env/lib/python3.6/site-packages/celery/app/trace.py", line 650, in __protected_call__
    return self.run(*args, **kwargs)
  File "/Users/tayabsoomro/Documents/Projects/MICAS-Coadunate/server/app/main/utils/tasks.py", line 86, in int_download_database
    list_of_sequences = ast.literal_eval(list_of_sequences.strip())
  File "/Users/tayabsoomro/anaconda3/envs/micas_env/lib/python3.6/ast.py", line 48, in literal_eval
    node_or_string = parse(node_or_string, mode='eval')
  File "/Users/tayabsoomro/anaconda3/envs/micas_env/lib/python3.6/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 0

Here's the code which throws the error: https://github.com/coadunate/MICAS/blob/master/server/app/main/utils/tasks.py#L86

Check `npm` is installed

When running the install.sh script, around step 7 and 8 npm is required from some server setup stuff. If you don't have npm, the script fails in a way that is hard to spot (sanitized for privacy):

[ DEBUG ][ Fri Jun 25 11:26:22 CST 2021 ] -- Step 7: Installing frontend dependencies
[ DEBUG ][ Fri Jun 25 11:26:22 CST 2021 ] -- npm install --prefix $HOME/WebstormProjects/MICAS/static $HOME/WebstormProjects/MICAS/static
bash: line 1: npm: command not found
[ DEBUG ][ Fri Jun 25 11:26:22 CST 2021 ] -- Step 8: Building the front-end ReactJS application
bash: line 1: npm: command not found

Should probably check npm is installed, and install it if required. Related or sub-issue to #31

Warning that centrifuge-build is not installed should be more clear

Running install.sh, it is difficult to see the one line warning that centrifuge-build is not installed:

[ DEBUG ][ Fri Jun 25 13:20:09 CST 2021 ] -- Step 10: Checking if centrifuge is installed
/Users/shorovatin/WebstormProjects/MICAS/install.sh:hash:137: no such command: centrifuge-build

This should probably be made a fatal_error, as the system is dependent on centrifuge.

Centrifuge fails to compile on WLS

This is bug is a direct consequence of me trying to compile and run MICAS and its dependencies in windows linux subsystem. It gets through most of the install.sh script without error, however when it comes to making centrifuge, some internal confusion on the part of centrifuges makefile and WLS cause the dump of the following error:

[ DEBUG ][ Thu Aug  5 14:33:34 CST 2021 ] -- Step 10: Checking if centrifuge is installed
./install.sh: line 125: hash: centrifuge-build: not found
[ DEBUG ][ Thu Aug  5 14:33:34 CST 2021 ] -- curl -L https://codeload.github.com/infphilo/centrifuge/tar.gz/v1.0.3 | tar zx -o
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  684k    0  684k    0     0   614k      0 --:--:--  0:00:01 --:--:--  614k
[ DEBUG ][ Thu Aug  5 14:33:36 CST 2021 ] -- making Centrifuge....
[sudo] password for shoro:
make: Entering directory '/home/shoro/MICAS/centrifuge-1.0.3'
/usr/bin/g++  -O3 -m64 -msse2 -funroll-loops -g3 -DCOMPILER_OPTIONS="\"-O3 -m64 -msse2 -funroll-loops -g3 -DPOPCNT_CAPABILITY\"" -DPOPCNT_CAPABILITY \
-fno-strict-aliasing -DCENTRIFUGE_VERSION="\"1.0.3-beta\"" -DBUILD_HOST="\"`hostname`\"" -DBUILD_TIME="\"`date`\"" -DCOMPILER_VERSION="\"`/usr/bin/g++  -v 2>&1 | tail -1`\"" -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE   -D
BOWTIE_MM   -DCENTRIFUGE -DBOWTIE2 -DBOWTIE_64BIT_INDEX -DNDEBUG -Wall \
 -I third_party  \
-o centrifuge-class centrifuge.cpp \
ccnt_lut.cpp ref_read.cpp alphabet.cpp shmem.cpp edit.cpp bt2_idx.cpp reference.cpp ds.cpp limit.cpp random_source.cpp tinythread.cpp qual.cpp pat.cpp read_qseq.cpp ref_coord.cpp mask.cpp pe.cpp aligner_seed_policy.cpp scoring.cpp pres
ets.cpp simple_func.cpp random_util.cpp outq.cpp centrifuge_main.cpp \
-lpthread
In file included from blockwise_sa.h:29,
                 from bt2_idx.h:44,
                 from centrifuge.cpp:34:
diff_sample.h: In function ‘void calcExhaustiveDC(T, bool, bool)’:
diff_sample.h:175:6: warning: this ‘if’ clause does not guard... [-Wmisleading-indentation]
  175 |      if(!diffs[d1]) diffCnt++; diffs[d1] = true;
      |      ^~
diff_sample.h:175:32: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘if’
  175 |      if(!diffs[d1]) diffCnt++; diffs[d1] = true;
      |                                ^~~~~
diff_sample.h:176:6: warning: this ‘if’ clause does not guard... [-Wmisleading-indentation]
  176 |      if(!diffs[d2]) diffCnt++; diffs[d2] = true;
      |      ^~
diff_sample.h:176:32: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘if’
  176 |      if(!diffs[d2]) diffCnt++; diffs[d2] = true;
      |                                ^~~~~
In file included from centrifuge.cpp:34:
bt2_idx.h: In static member function ‘static std::pair<Ebwt<index_t>*, Ebwt<index_t>*> Ebwt<index_t>::fromStrings(const EList<std::__cxx11::basic_string<char> >&, bool, int, int, bool, int32_t, int32_t, int32_t, const string&, bool, in
dex_t, index_t, index_t, int, uint32_t, bool, bool, bool)’:
bt2_idx.h:1053:3: warning: ‘template<class> class std::auto_ptr’ is deprecated [-Wdeprecated-declarations]
 1053 |   auto_ptr<stringstream> ss(new stringstream());
      |   ^~~~~~~~
In file included from /usr/include/c++/9/memory:80,
                 from bt2_idx.h:28,
                 from centrifuge.cpp:34:
/usr/include/c++/9/bits/unique_ptr.h:53:28: note: declared here
   53 |   template<typename> class auto_ptr;
      |                            ^~~~~~~~
In file included from centrifuge.cpp:34:
bt2_idx.h:1057:3: warning: ‘template<class> class std::auto_ptr’ is deprecated [-Wdeprecated-declarations]
 1057 |   auto_ptr<FileBuf> fb(new FileBuf(ss.get()));
      |   ^~~~~~~~
In file included from /usr/include/c++/9/memory:80,
                 from bt2_idx.h:28,
                 from centrifuge.cpp:34:
/usr/include/c++/9/bits/unique_ptr.h:53:28: note: declared here
   53 |   template<typename> class auto_ptr;
      |                            ^~~~~~~~
In file included from centrifuge.cpp:47:
classifier.h: In member function ‘virtual int Classifier<index_t, local_index_t>::go(const Scoring&, const Ebwt<index_t>&, const Ebwt<index_t>&, const BitPairReference&, WalkMetrics&, PerReadMetrics&, HIMetrics&, SpeciesMetrics&, Rando
mSource&, AlnSinkWrap<index_t>&)’:
classifier.h:431:45: error: the value of ‘rank’ is not usable in a constant expression
  431 |                     while(_hitMap[i].rank < rank) {
      |                                             ^~~~
classifier.h:427:21: note: ‘uint8_t rank’ is not const
  427 |             uint8_t rank = 0;
      |                     ^~~~
classifier.h:431:45: error: type/value mismatch at argument 1 in template parameter list for ‘template<class> struct std::rank’
  431 |                     while(_hitMap[i].rank < rank) {
      |                                             ^~~~
classifier.h:431:45: note:   expected a type, got ‘rank’
centrifuge.cpp: In function ‘void parseOptions(int, const char**)’:
centrifuge.cpp:1494:2: warning: ‘template<class> class std::auto_ptr’ is deprecated [-Wdeprecated-declarations]
 1494 |  auto_ptr<Presets> presets(new PresetsV0());
      |  ^~~~~~~~
In file included from /usr/include/c++/9/memory:80,
                 from bt2_idx.h:28,
                 from centrifuge.cpp:34:
/usr/include/c++/9/bits/unique_ptr.h:53:28: note: declared here
   53 |   template<typename> class auto_ptr;
      |                            ^~~~~~~~
centrifuge.cpp: In function ‘void multiseedSearchWorker(void*)’:
centrifuge.cpp:2333:2: warning: ‘template<class> class std::auto_ptr’ is deprecated [-Wdeprecated-declarations]
 2333 |  auto_ptr<PatternSourcePerThreadFactory> patsrcFact(createPatsrcFactory(patsrc, tid));
      |  ^~~~~~~~
In file included from /usr/include/c++/9/memory:80,
                 from bt2_idx.h:28,
                 from centrifuge.cpp:34:
/usr/include/c++/9/bits/unique_ptr.h:53:28: note: declared here
   53 |   template<typename> class auto_ptr;
      |                            ^~~~~~~~
centrifuge.cpp:2334:2: warning: ‘template<class> class std::auto_ptr’ is deprecated [-Wdeprecated-declarations]
 2334 |  auto_ptr<PatternSourcePerThread> ps(patsrcFact->create());
      |  ^~~~~~~~
In file included from /usr/include/c++/9/memory:80,
                 from bt2_idx.h:28,
                 from centrifuge.cpp:34:
/usr/include/c++/9/bits/unique_ptr.h:53:28: note: declared here
   53 |   template<typename> class auto_ptr;
      |                            ^~~~~~~~
centrifuge.cpp: In function ‘void driver(const char*, const string&, const string&)’:
centrifuge.cpp:2976:9: warning: ‘template<class> class std::auto_ptr’ is deprecated [-Wdeprecated-declarations]
 2976 |         auto_ptr<BitPairReference> refs;
      |         ^~~~~~~~
In file included from /usr/include/c++/9/memory:80,
                 from bt2_idx.h:28,
                 from centrifuge.cpp:34:
/usr/include/c++/9/bits/unique_ptr.h:53:28: note: declared here
   53 |   template<typename> class auto_ptr;
      |                            ^~~~~~~~
In file included from centrifuge.cpp:47:
classifier.h: In instantiation of ‘int Classifier<index_t, local_index_t>::go(const Scoring&, const Ebwt<index_t>&, const Ebwt<index_t>&, const BitPairReference&, WalkMetrics&, PerReadMetrics&, HIMetrics&, SpeciesMetrics&, RandomSource
&, AlnSinkWrap<index_t>&) [with index_t = long unsigned int; local_index_t = short unsigned int]’:
centrifuge.cpp:2660:94:   required from here
classifier.h:299:38: warning: comparison of integer expressions of different signedness: ‘size_t’ {aka ‘long unsigned int’} and ‘const THitInt’ {aka ‘const long int’} [-Wsign-compare]
  299 |                     if(coords.size() > rp.ihits) {
      |                        ~~~~~~~~~~~~~~^~~~~~~~~~
In file included from centrifuge.cpp:47:
classifier.h:411:30: warning: comparison of integer expressions of different signedness: ‘int’ and ‘size_t’ {aka ‘long unsigned int’} [-Wsign-compare]
  411 |                     if(i + 1 < _hitMap.size()) {
classifier.h:432:48: warning: comparison of integer expressions of different signedness: ‘int’ and ‘size_t’ {aka ‘long unsigned int’} [-Wsign-compare]
  432 |                         if(_hitMap[i].rank + 1 >= _hitMap[i].path.size()) {
classifier.h:442:55: warning: comparison of integer expressions of different signedness: ‘int’ and ‘size_t’ {aka ‘long unsigned int’} [-Wsign-compare]
  442 |                     uint64_t parent_taxID = (rank + 1 >= _hitMap[i].path.size() ? 1 : _hitMap[i].path[rank + 1]);
classifier.h:475:63: warning: comparison of integer expressions of different signedness: ‘int’ and ‘size_t’ {aka ‘long unsigned int’} [-Wsign-compare]
  475 |                         uint64_t cur_parent_taxID = (rank + 1 >= _hitMap[i].path.size() ? 1 : _hitMap[i].path[rank + 1]);
classifier.h: In instantiation of ‘void Classifier<index_t, local_index_t>::searchForwardAndReverse(index_t, const Ebwt<index_t>&, const Scoring&, RandomSource&, const ReportingParams&, index_t) [with index_t = long unsigned int; local
_index_t = short unsigned int]’:
classifier.h:238:13:   required from ‘int Classifier<index_t, local_index_t>::go(const Scoring&, const Ebwt<index_t>&, const Ebwt<index_t>&, const BitPairReference&, WalkMetrics&, PerReadMetrics&, HIMetrics&, SpeciesMetrics&, RandomSou
rce&, AlnSinkWrap<index_t>&) [with index_t = long unsigned int; local_index_t = short unsigned int]’
centrifuge.cpp:2660:94:   required from here
classifier.h:850:50: warning: comparison of integer expressions of different signedness: ‘long unsigned int’ and ‘const THitInt’ {aka ‘const long int’} [-Wsign-compare]
  850 |                        hit.size() + rchit.size() > rp.ihits) {
In file included from centrifuge.cpp:41:
aln_sink.h: In instantiation of ‘void AlnSinkSam<index_t>::appendMate(Ebwt<index_t>&, BTString&, const Read&, const Read*, TReadId, AlnRes*, AlnRes*, const AlnSetSumm&, const PerReadMetrics&, SpeciesMetrics&, size_t) [with index_t = lo
ng unsigned int; BTString = SStringExpandable<char, 1024, 2>; TReadId = long unsigned int; size_t = long unsigned int]’:
aln_sink.h:1407:9:   required from ‘void AlnSinkSam<index_t>::append(BTString&, size_t, const Read*, const Read*, TReadId, AlnRes*, AlnRes*, const AlnSetSumm&, const PerReadMetrics&, SpeciesMetrics&, bool, size_t) [with index_t = long
unsigned int; BTString = SStringExpandable<char, 1024, 2>; size_t = long unsigned int; TReadId = long unsigned int]’
aln_sink.h:1392:15:   required from here
aln_sink.h:2302:18: warning: comparison of integer expressions of different signedness: ‘int’ and ‘size_t’ {aka ‘long unsigned int’} [-Wsign-compare]
 2302 |  for (int i=0; i < this->tab_fmt_cols_.size(); ++i) {
In file included from blockwise_sa.h:29,
                 from bt2_idx.h:44,
                 from bt2_idx.cpp:25:
diff_sample.h: In function ‘void calcExhaustiveDC(T, bool, bool)’:
diff_sample.h:175:6: warning: this ‘if’ clause does not guard... [-Wmisleading-indentation]
  175 |      if(!diffs[d1]) diffCnt++; diffs[d1] = true;
      |      ^~
diff_sample.h:175:32: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘if’
  175 |      if(!diffs[d1]) diffCnt++; diffs[d1] = true;
      |                                ^~~~~
diff_sample.h:176:6: warning: this ‘if’ clause does not guard... [-Wmisleading-indentation]
  176 |      if(!diffs[d2]) diffCnt++; diffs[d2] = true;
      |      ^~
diff_sample.h:176:32: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘if’
  176 |      if(!diffs[d2]) diffCnt++; diffs[d2] = true;
      |                                ^~~~~
In file included from bt2_idx.cpp:25:
bt2_idx.h: In static member function ‘static std::pair<Ebwt<index_t>*, Ebwt<index_t>*> Ebwt<index_t>::fromStrings(const EList<std::__cxx11::basic_string<char> >&, bool, int, int, bool, int32_t, int32_t, int32_t, const string&, bool, in
dex_t, index_t, index_t, int, uint32_t, bool, bool, bool)’:
bt2_idx.h:1053:3: warning: ‘template<class> class std::auto_ptr’ is deprecated [-Wdeprecated-declarations]
 1053 |   auto_ptr<stringstream> ss(new stringstream());
      |   ^~~~~~~~
In file included from /usr/include/c++/9/memory:80,
                 from bt2_idx.h:28,
                 from bt2_idx.cpp:25:
/usr/include/c++/9/bits/unique_ptr.h:53:28: note: declared here
   53 |   template<typename> class auto_ptr;
      |                            ^~~~~~~~
In file included from bt2_idx.cpp:25:
bt2_idx.h:1057:3: warning: ‘template<class> class std::auto_ptr’ is deprecated [-Wdeprecated-declarations]
 1057 |   auto_ptr<FileBuf> fb(new FileBuf(ss.get()));
      |   ^~~~~~~~
In file included from /usr/include/c++/9/memory:80,
                 from bt2_idx.h:28,
                 from bt2_idx.cpp:25:
/usr/include/c++/9/bits/unique_ptr.h:53:28: note: declared here
   53 |   template<typename> class auto_ptr;
      |                            ^~~~~~~~
make: *** [Makefile:265: centrifuge-class] Error 1
make: Leaving directory '/home/shoro/MICAS/centrifuge-1.0.3'

No expectation for this to be resolved, merely documenting known erroneous behaviour.

Legacy code from Centrifuge

MICAS/server/app/main/utils/tasks.py

Lines 79 to 142 in 57338a1

if bacteria or archaea or virus:
# Construct db_string first
db_list = []
if bacteria:
db_list.append('bacteria')
if archaea:
db_list.append('archaea')
if virus:
db_list.append('viral')
db_string = ",".join([str(x) for x in db_list])
logger.debug("DOWNLOAD_DATABASE: Downloading " + db_string + " database(s) from NCBI")
self.update_state(
state="PROGRESS",
meta={
'percent-done': 55,
'message' : "Downloading " + db_string + " database(s) from NCBI",
'project_id' : project_id
}
)
cmd = ['centrifuge-download -o ' + app_location_database + 'library -m -d ' + db_string + ' refseq']
logger.debug(f"CMD:\n {cmd}")
outfile = open(app_location_database + 'seqid2taxid.map', 'a+')
err_file = open(app_location_database + 'download_err.txt', 'w+')
try:
# Download the database.
download_db_cmd_output = subprocess.Popen(
cmd,
shell=True,
stdout=outfile,
stderr=err_file
)
download_db_cmd_output.communicate()
download_db_cmd_output.wait()
except (OSError, subprocess.CalledProcessError) as exception:
logger.error(str(exception))
return "ER1"
else:
logger.debug("DOWNLOAD_DATABASE: Successfully downloaded " + db_string + " database(s) from NCBI")
self.update_state(
state="PROGRESS",
meta={
'percent-done': 90,
'message' : "Successfully downloaded " + db_string + "database(s) from NCBI",
'project_id' : project_id
}
)
import glob
# Putting all the query sequences in one, input_sequences file.
logger.debug("DOWNLOAD_DATABASE: Concatenating all the sequence files.")
self.update_state(state="PROGRESS",
meta={'percent-done': 95, 'message': "Concatenating all the sequence files.",
'project_id' : project_id}
)
with open(app_location_database + 'input_sequences.fa', 'wb') as outfile:
for filename in glob.glob(app_location_database + 'library/*/*.fna'):
with open(filename, 'rb') as readfile:
shutil.copyfileobj(readfile, outfile)

The above code chunk is left over from our old use of centrifuge. This now does nothing but produce errors, and should be removed or updated.

Refine install script

  • Make sure that the redis binary is installing correctly
  • Make sure that the centrifuge binary is installing corrrectly

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.