brunoamaral / gregory Goto Github PK

This does not feel right because on clinical trials it should not be a source. Maybe it should be "sponsor" or "published_in"

• 1. Registo nacional de ensaios clínicos (https://www.rnec.pt/pt_PT)
• 2. MS Journal (https://journals.sagepub.com/home/msj)
• 3. MS and Related Disorders Journal (https://www.msard-journal.com/)

Gregory needs to be agnostic in order to be applied to any number of subjects.

Right now, stop words, or stop sentences, are hardcoded into the javascript:

https://github.com/brunoamaral/gregory/blob/main/assets/js/gregory.js#L5-L20

Maybe move this information into config.toml or another single configuration file that makes sense.

The node-red feedparser doesn't let us add an RSS url to it, so instead we will be using /python-ml/feedreader.py

• List all articles
• relevancy API
• Remove pagination from articles
  https://api.gregory-ms.com/articles/all[](https://api.gregory-ms.com/articles/id/19)

Example: https://api.gregory-ms.com/articles/all

• List article that matches the {ID} number.

https://api.gregory-ms.com/articles/id/{ID}

Example: https://api.gregory-ms.com/articles/id/19[](https://api.gregory-ms.com/articles/keyword/myelin)

• List all articles by keyword.

https://api.gregory-ms.com/articles/keyword/{keyword}

Example: https://api.gregory-ms.com/articles/keyword/myelin

• List related articles by keywords

POST https://gregory-ms.com/articles/related/[](https://api.gregory-ms.com/articles/relevant)

Expects a json object of keywords in the post body.

{ "keywords": ['trials','gait rehabilitation','multiple sclerosis'] }
https://gregory-ms.com/articles/related/

• List all relevant articles.

These are articles that we show on the home page because they appear to offer new courses of treatment.

https://api.gregory-ms.com/articles/relevant[](https://api.gregory-ms.com/articles/source/1)

Example: https://api.gregory-ms.com/articles/relevant

Articles’ Sources

• List all articles from specified {source}.

https://api.gregory-ms.com/articles/source/{source_id}

Example: https://api.gregory-ms.com/articles/source/1[](https://api.gregory-ms.com/articles/sources)

• List all available sources.

https://api.gregory-ms.com/articles/sources[](https://api.gregory-ms.com/trials/all)

Example: https://api.gregory-ms.com/articles/sources

Trials

• List all trials.

https://api.gregory-ms.com/trials/all[](https://api.gregory-ms.com/trials/keyword/myelin)

Example: https://api.gregory-ms.com/trials/all

• List all trials by keyword.

https://api.gregory-ms.com/trials/keyword/{keyword}

Example: https://api.gregory-ms.com/trials/keyword/myelin[](https://api.gregory-ms.com/trials/source/pubmed)

Trials’ Sources

• List all trials from specified {source}.

https://api.gregory-ms.com/trials/source/{source}

Example: https://api.gregory-ms.com/trials/source/pubmed[](https://api.gregory-ms.com/trials/sources)

• List all available sources.

https://api.gregory-ms.com/trials/sources

Example: https://api.gregory-ms.com/trials/sources

• - RNEC
• - FirstWord Pharma
• - EMA
• - Champalimaud Foundation
• - CEIC : Doesn't seem to have any public database
• - CUF
• - Novartis

New suggestions today (2022-03-15)

• https://www.nejm.org/
• https://www.bmj.com/

Currently we need to add users to the mailing lists manually. This would allow them to subscribe on their own.

Requires

1. Spam protection
2. Ability to add existing users to new lists
3. Ability to unsubscribe from a list by email link

Example:

[
    {
        "source": "CUF",
        "link": "https://www.example.com"
    },
    {
        "source": "ClinicalTrials.gov",
        "link": "https://www.example.com"
    },
    {
        "source": "Novartis",
        "link": "https://www.example.com"
    }
]

Other relevant information, the link of the search page and keywords we use.

depends on #5

$ sudo docker-compose up

Creating volume "gregory_flows" with local driver
Creating volume "gregory_python" with local driver
Creating node-red ... error

ERROR: for node-red  Cannot create container for service node-red: failed to mount local volume: mount ./docker-python:/var/lib/docker/volumes/gregory_python/_data, flags: 0x1000: no such file or directory

ERROR: for node-red  Cannot create container for service node-red: failed to mount local volume: mount ./docker-python:/var/lib/docker/volumes/gregory_python/_data, flags: 0x1000: no such file or directory
ERROR: Encountered errors while bringing up the project.

https://www.clinicaltrialsregister.eu/ctr-search/search?query=multiple+sclerosis

These results are available as an RSS Feed.
https://www.clinicaltrialsregister.eu/ctr-search/rest/feed/bydates?query=multiple+sclerosis

reasons for it:

• better handling of timestamp data
• equal integration with metabase
• faster (?) response time

best approach would be psql -d gregory -f ./docker-data/gregory.db but it results in syntax errors because of the html values in some columns.

Example https://gregory-ms.com/articles/1/

some of the articles listed seem that they were not marked as relevant.

I must have missed something when I wrote the models.

from django.db import models
class Categories(models.Model):
	category_id = models.AutoField(primary_key=True)
	category_name = models.CharField(blank=True, null=True,max_length=200)
	category_description = models.TextField(blank=True, null=True)

	def __str__(self):
		return self.category_name

	class Meta:
		managed = True
		verbose_name_plural = 'categories'
		db_table = 'categories'

class Articles(models.Model):
	article_id = models.AutoField(primary_key=True)
	title = models.TextField(blank=False, null=False, unique=True)
	summary = models.TextField(blank=True, null=True)
	link = models.URLField(blank=False, null=False, max_length=2000)
	published_date = models.DateTimeField(blank=True, null=True)
	discovery_date = models.DateTimeField()
	source = models.ForeignKey('Sources', models.DO_NOTHING, db_column='source', blank=True, null=True)
	relevant = models.BooleanField(blank=True, null=True)
	ml_prediction_gnb = models.BooleanField(blank=True, null=True)
	ml_prediction_lr = models.BooleanField(blank=True, null=True)
	noun_phrases = models.JSONField(blank=True, null=True)
	categories = models.ManyToManyField(Categories)
	entities = models.ManyToManyField('Entities')
	sent_to_admin = models.BooleanField(blank=True, null=True)
	sent_to_subscribers = models.BooleanField(blank=True, null=True)
	sent_to_twitter = models.BooleanField(blank=True, null=True)
	doi = models.CharField(max_length=280, blank=True, null=True)

	def __str__(self):
		return str(self.article_id)

	class Meta:
		managed = True
		# unique_together = (('title', 'link'),)
		verbose_name_plural = 'articles'
		db_table = 'articles'


class Entities(models.Model):
	entity = models.TextField()
	label = models.TextField()


	class Meta:
		managed = True
		verbose_name_plural = 'entities'
		db_table = 'entities'


class Sources(models.Model):
	TABLES = [('articles', 'Articles'),('trials','Trials')]


	source_id = models.AutoField(primary_key=True)
	source_for = models.CharField(choices=TABLES, max_length=50, default='articles')
	name = models.TextField(blank=True, null=True)
	link = models.TextField(blank=True, null=True)
	language = models.TextField()
	subject = models.TextField()
	method = models.TextField()
	

	def __str__(self):
		return self.name

	class Meta:
		managed = True
		verbose_name_plural = 'sources'
		db_table = 'sources'


class Trials(models.Model):
	trial_id = models.AutoField(primary_key=True)
	discovery_date = models.DateTimeField(blank=True, null=True)
	title = models.TextField(blank=False,null=False, unique=True)
	summary = models.TextField(blank=True, null=True)
	link = models.URLField(blank=False, null=False, max_length=2000)
	published_date = models.DateTimeField(blank=True, null=True)
	source = models.ForeignKey('Sources', models.DO_NOTHING, db_column='source', blank=True, null=True)
	relevant = models.BooleanField(blank=True, null=True)
	sent = models.BooleanField(blank=True, null=True)
	sent_to_twitter = models.BooleanField(blank=True, null=True)
	sent_to_subscribers = models.BooleanField(blank=True, null=True)

	def __str__(self):
		return str(self.trial_id) 

	class Meta:
		managed = True
		verbose_name_plural = 'trials'
		db_table = 'trials'

We are using https://github.com/explosion/spaCy to detect the noun phrases in the title of articles. This information is then used to list related articles on each page.

Half of the build process is running spacy.io, so it should be included in the node-red flows to save that information in the database.

We could run it as a separate script, but I don't want to split the different processing steps between the container and the host server.

options:

• lunr.js
• Algolia

https://www.mssociety.org.uk/research/explore-our-research/emerging-research-and-treatments/explore-treatments-in-trials

The goal is to list the current research as listed by MS Society with a listing of trials and published articles.

• Bexarotene
• Metformin
• Tcelna
• Simvastatin
• Ocrelizumab

Right now, we are fetching articles/all?format=json which uses Django Rest Framework to dump the whole database.

Fetching from the postgres database directly will speed up the build and cut down on processing.

https://github.com/brunoamaral/gregory/blob/211806446e0584a9819ae5f949b580b7567eb5db/gregory-pg.schema.sql

Anyone reading this, I could really use some help making sure the new tables and columns are correct.

Main requirement is that the value for an article's title or URL needs to be unique.
Diagram below shows a possible DB model for version 7.

select trial_id,published_date,discovery_date from trials where published_date = '';

161, mostly from Novartis.

This is a caveat where the system fails to include articles in the corresponding category if the noun used is different. For example Ocrelizumab and Ocrevus, or Natalizumab and Tysabri. These nouns correspond to a single medication, respectively, however, in the current state, Gregory can only identify them as being separate entities.

this will be a by product of #5

maybe using Metabase to avoid extra coding