monarch-initiative / phenopacket-scraper-core Goto Github PK

Extracts information from life-science websites and texts, generating phenopackets with the extracted information and correct external ontology references.

To get the project's source code, clone the github repository:

$ git clone https://github.com/monarch-initiative/phenopacket-scraper-core.git

First, you need to create a virtual environment and activate it.

$ [sudo] pip install virtualenv
$ virtualenv -p python3 venv
$ source venv/bin/activate
(venv)$

Next, install all the dependencies in the environment.

(venv)$ venv/bin/pip install -r requirements.txt

Clone the phenopacket-python repository:

$ git clone https://github.com/phenopackets/phenopacket-python.git

Install it in your virtual environment:

(venv)$ cd phenopacket-python
(venv)$ python setup.py install

Add this to the end of ~/.profile or ~/.bash_profile file to add phenopacket-python directory to your python environment variables:

$ export PYTHONPATH=$PYTHONPATH:[path of phenopacket-python directory]

For Example:

$ export PYTHONPATH=$PYTHONPATH:/Users/Gauss/Home/phenopacket-python

Now, install the application into the virtual environment.

(venv)$ cd phenopacket-scraper-core
(venv)$ python setup.py install

(venv)$ pps --help
(venv)$ pps scrape -u (url)

Example:

(venv)$ pps scrape -u http://molecularcasestudies.cshlp.org/content/early/2016/02/09/mcs.a000786.abstract
(venv)$ pps -q scrape -u (Url)

Title: Mutations in the substrate ...

Abstract:
We describe a large Lebanese fa...

HPO Terms:
Diffuse cerebellar atrophy
Generalized hypotoni...

To use files with list of URLs as input:

(venv)$ pps scrape -f (Filename)

Example:

(venv)$ pps scrape -f testurls.txt

testurls.txt:

http://molecularcasestudies.cshlp.org/content/early/2016/02/09/mcs.a000786.abstract
http://molecularcasestudies.cshlp.org/content/2/2/a000703.abstract
http://molecularcasestudies.cshlp.org/content/2/2/a000620.abstract
http://molecularcasestudies.cshlp.org/content/2/1/a000661.abstract

To scrape required data from a HTML file:

(venv)$ pps scrape -d (Filename)

To store the output in a file:

(venv)$ pps scrape -u (Url) -o (Filename)
(venv)$ pps scrape -d (Filename) -o (Filename)
(venv)$ pps scrape -f (Input_filename) -o (Output_filename)

This will create two files for now, (Filename)_abstract.txt will contain the abstract and the (Filename)_hpo_terms.txt will contain the hpo terms.

Sci-graph Annotation:

(venv)$ pps annotate -u (url)

[{u'start': 4, u'token': {u'terms': [u'TORC1 complex'], u'id': u'GO:0031931', u'categories': [u'cellular component']}, u'end': 10}, {u'start': 11, u'token': {u'terms': [u'inhibitor'], u'id': u'CHEBI:35222', u'categories': [u'chemical role']}, u'end': 20}, {u'start': 72, u'token': {u'terms': [u'multiple'], u'id': u'PATO:0002118', u'categories': [u'qua......

HPO Terms:
Neoplasm
Breast carcinoma
Carcinoma
increased carcinoma incidence

Phenopacket Generation:

(venv)$ pps phenopacket -u (url)
(venv)$ pps phenopacket -d (html_filename)

{
"entities": [
  {
    "id": "http://molecularcasestudies.cshlp.org/content/2/1/a000661.abstract",
    "type": "paper"
  }
],
"id": "gauss-packet",
"phenotype_profile": [
  {.....

To store the output in a file:

(venv)$ pps annotate -u (Url) -o (Filename)
(venv)$ pps phenopacket -u (Url) -o (Filename)

Finally, when done, deactivate your virtual environment:

(venv)$ deactivate
$