peer35 / cosmin Goto Github PK

COSMIN database of systematic reviews. Based on Blacklight + ActiveRecord for record management

Ruby 95.28% JavaScript 0.15% CoffeeScript 0.02% CSS 0.34% HTML 2.77% XSLT 1.18% Rich Text Format 0.01% SCSS 0.17% Dockerfile 0.05% Shell 0.03%

cosmin's Introduction

README

For some documentation see the wiki

cosmin's People

Contributors

Watchers

cosmin's Issues

Remove check on duplicate Endnote numbers

Referenties uit endnote kunnen op dit moment alleen in de CDB ingevoegd worden, als het record nummer er nog niet in zit. dit is eigenlijk heel onhandig, omdat de endnote bestanden steeds anders zijn, en alle refs steeds opnieuw beginnen met nummeren. Zou het mogelijk zijn om wel dubbele nummer te kunnen invoeren?

Refactor the instruments table

Currently instrument names, variants, aliases and versions are stored as a single string in the name field. It would be better to have a single instrument table, linked to a separate table with the name variants or versions.
An alternative could be to store the name and variants as a jsonb object, but only if we're certain the amount of querying will be limited. https://guides.rubyonrails.org/active_record_postgresql.html#json-and-jsonb

The instrument names could still be shown as single strings in the front end.

Updating instruments takes a long time to updare assciated records in SOLR

It takes ~1.5 second to update an associated record. Some instruments have 20 associated records, that's 30 seconds!

Possible solutions:

Do a single add, build an array of records to update first.
Simplify the add. I now GET the complete records from SOLR, append or delete the instrument and the add the complete record. Isn't there a simpler update command?
Update SOLR asynchronously so the user does not have to wait.

RSOLR is only used in the model records.rb so it shouldn't be too difficult to rewrite. Might need to look at a different gem like sunspot.

Create an API (or export) to GET instruments and records

Preferably authenticated with a key
Exact output to be determined

https://stevepolito.design/blog/build-an-api-in-rails-with-authentication seems to be a reasonable implementation.

Create rake task for reindexing

Using the indexall url takes forever, this will allow starting a reindex in the background.

tokenizer deprecated

TokenizerFactory is using deprecated 6.1.0 emulation. You should at some point declare and reindex to at least 7.0,&#8203; because 6.x emulation is deprecated and will be removed in 8.0

Solr loaded a deprecated plugin/analysis class [solr.TrieIntField]. Please consult documentation how to replace it accordingly.

Update rails to ~>5.0

Improve performance of instrument list

https://database.cosmin.nl/instruments

Check server performance (atp)
Improve query
instruments_controller.rb

...
        @instruments, @alphaParams = Instrument.all
                                         .alpha_paginate(params[:letter], {:default_field => '0-9', :include_all => false, :js => false, :bootstrap3 => true}) { |instrument| instrument.name }
        @instruments.sort_by! { |m| m.name.downcase }
...

This queries all instruments in batches of 500 and then filters them. Slow.

views/instruments/index.html.erb

...
          <td><%= instrument.records.count %></td>
...
          <td>
            <% if instrument.records.count == 0 %>
              <%= link_to 'Delete', instrument, method: :delete, data: {confirm: 'Are you sure?'} %>
            <% end %>
          </td>
...

Results in 2 queries per row (although the second one is from cache). Also slow

Use jquery datatables with paginated queries.

Address Blacklight 8 deprecation warnings

cosmin-app          | DEPRECATION WARNING: Blacklight 8 will filter out non-search parameter, including: id. (called from permit_search_params at /usr/local/bundle/gems/blacklight-7.35.0/lib/blacklight/parameters.rb:77)

cosmin-app          | DEPRECATION WARNING: Blacklight 8 will filter out non-search parameter, including: utf8. (called from permit_search_params at /usr/local/bundle/gems/blacklight-7.35.0/lib/blacklight/parameters.rb:77)

Backend record "show" page does not exist

See for example https://database.cosmin.nl/records/3016

User gets dropped their when they click cancel on the edit page.

Create a frontend url like https://database.cosmin.nl/?instrument_id=XX

Add instrument id to SOLR index
Add a custom route to the catalog controller

Update single record in SOLR takes too much time

Started PATCH "/records/289" for 130.37.15.198 at 2023-01-31 08:23:59 +0000
I, [2023-01-31T08:23:59.167496 #13]  INFO -- : Processing by RecordsController#update as HTML
I, [2023-01-31T08:23:59.168461 #13]  INFO -- :   Parameters: {"utf8"=>"✓", "authenticity_token"=>"g+7BS7YNQtFp88oz0iJuJ48nAHPjNKn7BlgALjGsYLcVfS38gAUkJ4u4XvyQqu93GG0E+8Tn1JpsXIKHUFhSWw==", "record"=>{"endnum"=>"298", "title"=>"A systematic review of the scales used for the measurement of cancer-related fatigue (CRF)", "doi"=>"10.1093/annonc/mdn537", "url"=>["<Go to ISI>://WOS:000262718000003", "https://www.ncbi.nlm.nih.gov/pubmed/18678767"], "abstract"=>"Background: Fatigue in cancer is very common and can be experienced at all stages of disease and in survivors. There is no accepted definition of cancer-related fatigue (CRF) and no agreement on how it should be measured. A number of scales have been developed to quantify the phenomenon of CRF. These vary in the quality of psychometric properties, ease of administration, dimensions of CRF covered and extent of use in studies of cancer patients. This review seeks to identify the available tools for measuring CRF and to make recommendations for ongoing research into CRF.", "issn"=>"0923-7534", "issue"=>"1", "journal"=>"Annals of Oncology", "accnum"=>"WOS:000262718000003", "author_str"=>"Minton, O.;Stone, P.", "pubyear"=>"2009", "startpage"=>"17-25", "ss"=>["Physical state", "Emotional state", "Cognitive/mental state"], "fs"=>["Physical functioning", "Cognitive/mental functioning"], "age"=>["Children (0-18)", "Adults (18-65)", "Seniors (65+)"], "disease"=>["Neoplasms and related symptoms"], "pnp"=>["Patient Reported Outcome"], "tmi"=>["1 - Questionnaires"], "instrument_ids"=>["", "1520", "463", "6930", "2356", "6931", "6940", "2366", "733", "6936", "2370", "736", "2372", "4330", "6922"], "admin_notes"=>"Converted from old COSMIN database\r\ntest2", "status"=>"published"}, "commit"=>"Update Record", "id"=>"289"}
I, [2023-01-31T08:23:59.194074 #13]  INFO -- : <ActionController::Parameters {"endnum"=>"298", "title"=>"A systematic review of the scales used for the measurement of cancer-related fatigue (CRF)", "doi"=>"10.1093/annonc/mdn537", "url"=>["<Go to ISI>://WOS:000262718000003", "https://www.ncbi.nlm.nih.gov/pubmed/18678767"], "abstract"=>"Background: Fatigue in cancer is very common and can be experienced at all stages of disease and in survivors. There is no accepted definition of cancer-related fatigue (CRF) and no agreement on how it should be measured. A number of scales have been developed to quantify the phenomenon of CRF. These vary in the quality of psychometric properties, ease of administration, dimensions of CRF covered and extent of use in studies of cancer patients. This review seeks to identify the available tools for measuring CRF and to make recommendations for ongoing research into CRF.", "issn"=>"0923-7534", "issue"=>"1", "journal"=>"Annals of Oncology", "accnum"=>"WOS:000262718000003", "pubyear"=>"2009", "startpage"=>"17-25", "ss"=>["Physical state", "Emotional state", "Cognitive/mental state"], "fs"=>["Physical functioning", "Cognitive/mental functioning"], "age"=>["Children (0-18)", "Adults (18-65)", "Seniors (65+)"], "disease"=>["Neoplasms and related symptoms"], "pnp"=>["Patient Reported Outcome"], "tmi"=>["1 - Questionnaires"], "instrument_ids"=>["1520", "463", "6930", "2356", "6931", "6940", "2366", "733", "6936", "2370", "736", "2372", "4330", "6922"], "admin_notes"=>"Converted from old COSMIN database\r\ntest2", "status"=>"published", "bpv"=>[], "ghp"=>[], "oql"=>[], "author"=>["Minton, O.", "Stone, P."]} permitted: true>
I, [2023-01-31T08:23:59.236790 #13]  INFO -- : add to solr index 289
I, [2023-01-31T08:24:13.219273 #13]  INFO -- : Redirected to http://adminyoda.labs.vu.nl:3013/catalog/289
I, [2023-01-31T08:24:13.221120 #13]  INFO -- : Completed 302 Found in 14052ms (ActiveRecord: 25.8ms

On the producton server it goes to 30 seconds and even over 60 seconds for other records.