Script of adding new organism must able to receive three arguments from command line , in case the input of display name has second Species name.

There are two ways to make it :

1. Write our own script to scrape the wiki data via wikipedia API

2. Find the open source that someone publish to pypi
   ref :　https://github.com/goldsmith/Wikipedia/blob/master/wikipedia/wikipedia.py　

Also make the script for blast and hmmer that can add the database by one command and can be easier to integrate with CWL.

ideal command : python manage.py addblast [Genus] [species] -t [sequence_type] -f [fasta_filename]

As the number of organisms growing, I believe that searchBar is better than a scrollBar in the aspect of working efficiency for BLAST user

• Link:
  https://i5k.nal.usda.gov/webapp/blast/

• The result might look like this:
  

As described in this stackoverflow question:
https://stackoverflow.com/questions/33784362/django-migrations-best-practice

The same question comes to my mind when I first started working on this project.
During the configuration step, we need to makemigrations and migrate, which is different from my previous experience of other django project (although I didn't work on model part of MTV framework). Maybe we should consider to follow this kind of best practice?

Due to "manage.py" module we can not directly import argparse to get the argument we want from command line.

Need to put our function under "Command" Class which inherit the "BaseCommand" Class
and use its own "add_arguments(self,parser)" to get the argument,

e.g. python manage.py addorganism Anoplophora glabripennis

Not sure why I'm getting 404 errors. I reran the command to populate the Sequences database. :

06/Aug/2018 15:53:42] "GET /favicon.ico HTTP/1.1" 404 2980
Not Found: /blast/task/api/seq/gnl|Eurytemora_affinis|euraff_Scaffold502/
[06/Aug/2018 15:54:34] "GET /blast/task/api/seq/gnl|Eurytemora_affinis|euraff_Scaffold502/?format=fasta HTTP/1.1" 404 4146
Not Found: /blast/task/api/seq/gnl|Eurytemora_affinis|euraff_Scaffold579/
[06/Aug/2018 15:54:37] "GET /blast/task/api/seq/gnl|Eurytemora_affinis|euraff_Scaffold579/?format=fasta HTTP/1.1" 404 4146

We recently had issues with using our normal header in the Django app, specifically the blast-results page. This was caused by some hard coded assertions in the kendo.js code that lays out the divs for the panels in the page.

I would like to replace the kendo js with grid-css for layout. There are several benefits to using this including the ability to mix absolute and fractional sizes in a layout, and since it is css it is native to the browser and does not require additional code or us to post additional license information.

Some references:

https://christianheilmann.com/2017/06/21/css-vs-javascript-trust-vs-control/

Guides for implementing grid-css

http://cssgridgarden.com/
https://medium.com/flexbox-and-grids/how-to-efficiently-master-the-css-grid-in-a-jiffy-585d0c213577

@hsiaoyi0504 mentioned that this is not universally available, so I checked with caniuse.com. They report that is should be supported in ~87% of browser installations.

https://caniuse.com/#feat=css-grid

1. need to check the connection to WIKI api and NCBI api
2. make sure organism name is correct
3. make sure to add the organism to the database

done

Test if linkouts appear when ENABLE_JBROWSE_INTEGRATION = False

Needed for 508 (accessibility) compliance. https://www.w3schools.com/tags/att_img_alt.asp

• https://github.com/NAL-i5K/genomics-workspace/blob/1783dad9ec25424053a8223a1ecef29c649dc5c0/sphinx_doc/user_guide.rst

• https://github.com/NAL-i5K/genomics-workspace/blob/aaa4f1997394a9bb26071949742f10471f0596e5/blast/templates/blast/main.html

• https://github.com/NAL-i5K/NAL-genomics-workspace/blob/e1950045575d35e8d0e29bf7cc316a113d37a92e/clustal/templates/clustal/manual.html

• https://github.com/NAL-i5K/genomics-workspace/blob/aaa4f1997394a9bb26071949742f10471f0596e5/hmmer/templates/hmmer/manual.html

For adding blast db we need to split to 2 step,

1. Pick the organism from organism table to add to the blast database
2. Can execute makeblastdb separately
3. Populate to the sequence table

docker is very useful to rebuild the environment and whole configuration so I want to apply it to our genomics-workspace.

reference : 1. https://github.com/twtrubiks/docker-tutorial
2. https://docs.docker.com/get-started/

1. Legacy blast is a very clean and stable interface that also works well on smaller screens. (General display)
2. Vigilance on simple mouse-overs without clicking seems to be necessary to ensure that I continue to select the intended organism in setting up a blast query. (Submission page)
3. When working on complicated genes, which require iterative blasting or blasting for specific exons or motifs individually, the back browser functionality of the legacy system recalls the species and the type of blast (e.g., tblastn) automatically, which saves time and worry. (Going from results back to submission and keeping some settings see #119 (closed))
4. Also, there never seems to be smooth scrolling navigability within the blast search results subwindows, and where results always require up-down and left-right scrolling. (Results page)
5. I repeatedly struggle to select the results I want to view and to ensure that I'm actually at the top of the blast hits and haven't inadvertently scrolled down without noticing. (results page)

The version of jquery we're using now has security issues, and we'll need to update it before any more deployments.

To let the user know for the internet connection error , we can add more conditional expression to the script and also check the data extracted from wiki and NCBI are correct or not.

As shown in following lines:
The ex_nucleotide is a peptide sequence, not a nucleotide sequence. I will fix it later.

When I worked on #631 today, I found that jquery-ui.css (used in result page of blast search) is a customized version of jquery-ui.css, although its name looks like without customized. We should figure out customized part and separate that part from library. That will make our future upgrade more easily. To figure out which part is customized, I noticed that the orange background in result page of blast search comes from that file.

On our production, we replace

{% block head-scripts %}{% endblock %} in layout.html

into

An usage of old-styled url pattern I found when I traced the current codebase. As described here. The usage of patterns() is deprecated in 1.8, and will be removed in 1.10. It will not be a issue if we don't upgrade to 1.10. It's just documented here for future reference.

The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.

For the ideal result of all the command for handle django database:

addorganism : python manage.py addorganism Aethina tumida

addblast : python manage.py addblast Aethina tumida -t peptide protein -f GCF_001937115.1_Atum_1.0_genomic.fna

makeblastdb : python manage.py blast_utility GCF_001937115.1_Atum_1.0_genomic.fna -m

populatesequence : python manage.py blast_utility GCF_001937115.1_Atum_1.0_genomic.fna -p

addhmmer : python manage.py addhmmer Aethina tumida -f GCF_001937115.1_Atum_1.0_genomic.fna

make blast is_shown:
python manage.py blast_shown GCF_001937115.1_Atum_1.0_genomic.fna --shown true/false
delete feature:
python manage.py delete -o Aethina tumida

python manage.py delete -b GCF_001937115.1_Atum_1.0_genomic.fna

python manage.py delete -h GCF_001937115.1_Atum_1.0_genomic.fna

There are two files recording python dependencies, requirements.txt and requirements_new.txt. It seems that requirements_new.txt was accidentally added one year ago. It should be removed because it may cause some confusion. It's a minor change, but I think it can be totally removed from git history.

There are another name for molecule_type : ('nucl',Nucleotide'), ('prot','Peptide')
need to identify the input to these two type

Inspired by #271, we will create a searchBar for quickly search the organism we are interested in.
I want to discuss about the order of organisms?
Should it still be alphabetically?
Or it is nicer to order by popularity, and how do we define popularity?

Just document things can be improved in front end of blast app.

1. use prop() rather than attr() for disabled, checked, and selected
   such as
   

   genomics-workspace/blast/static/blast/scripts/blast-multi.js

   Line 210 in b719f8e

   $('.program').attr('disabled', false).removeClass('disabled-radio');

   
   

   genomics-workspace/blast/static/blast/scripts/blast-multi.js

   Line 212 in b719f8e

   $('.blastp').attr('disabled', 'disabled').addClass('disabled-radio');

   
   It relates to performance and avoid error.

2. Move css on template to specific css file
   such as
   

   genomics-workspace/blast/templates/blast/main.html

   Lines 21 to 30 in b719f8e

   	<style>
   	.nav-tabs>li.active>a, .nav-tabs>li.active>a:hover, .nav-tabs>li.active>a:focus {
   	box-shadow: inset 0 -2px 0 #317eac;
   	color: #317eac;
   	border: 0px;
   	}
   	.txt:hover {
   	text-decoration: underline;
   	}
   	</style>

   
   

   genomics-workspace/blast/templates/blast/main.html

   Line 66 in b719f8e

   <legend>BLAST Databases <a href="https://i5k.nal.usda.gov/content/blast-tutorial" target="_blank"><span style="float:right"><span class="glyphicon glyphicon-book"></span> Tutorial</span></a></legend>

   
   

   genomics-workspace/blast/templates/blast/main.html

   Line 106 in b719f8e

   <span class="load-nucleotide">Or</span> <span class="load-peptide">load</span> it from disk <span style=" cursor: pointer; font-weight:bold; color: blue;" class="load-nucleotide txt"> (load example) </span>

It will make it more easier to maintain css and avoid error.

3. Use :disabled css selector, rather than add disabled class dynamically.
   https://developer.mozilla.org/zh-TW/docs/Web/CSS/:disabled

from results page back to submission page
We want the state still preserve.

I thought we fixed this to be a calculated value instead of static.

Line in question:

test the deployment on dev

The install guide has a mistake.

The line virtualenv py2.7

Should read:
virtualenv -p python2.7 py2.7

Like we discussed today, perhaps we don't need to have APIs.
Here are some possible options. I would update if I have more information:

1.using python manage.py shell < myscript.py: https://stackoverflow.com/a/16853799/5363040

2.using RunScript of django-extensions.
See an example here: https://github.com/django-extensions/django-extensions/blob/master/docs/runscript.rst.

The underlying mechanism is the same as 1, but based on a python package.

3.use loaddata, dumpdata of django commands: https://www.creativesoftware.com/blog/8-useful-django-dumpdata-and-loaddata-commands. It's the mechanisms used by Han Lin before and there are lots of scripts/command weren't maintained for a long time: https://github.com/NAL-i5K/genomics-workspace/tree/e9d355bc51118d3f7a4b56b94717d740853bf6f6/util.

django customize manage.py command :https://docs.djangoproject.com/en/1.11/howto/custom-management-commands/

I think I found a bug of blast-app front-end. When you insert some query into query sequence text area. If it disable some options of blast programs, and you click the reset button. All forms will be reset. However, some of options of blast programs will still be disabled, as that shown in following screenshot:

[BLAST query page(submission page)]
link:
https://github.com/r06942072/frontend_BLASTquery

Naming list of smart UI components

• 4 blocks:
  • yourQuery
  • ourDb
  • program
  • timeTravel

Functionality list

• #271, add Searching bar, and remain scrolling bar
• List all organisms and BLAST databases
• User can select multiple databases
• User can enter multiple query sequences
• Text box for query input
• Sample sequences for input
• Support only one type of database at a time (nucleotide or peptide)
• Disable BLAST programs based on the database and query type (nucleotide or peptide)

Link:

• BLAST webapp
  https://i5k.nal.usda.gov/webapp/blast/
• BLAST js original code
  https://github.com/NAL-i5K/genomics-workspace/blob/master/blast/static/blast/scripts/blast-multi.js
• Hmmer js original code
  https://github.com/NAL-i5K/genomics-workspace/blob/master/hmmer/static/hmmer/scripts/hmmer-multi.js

The idea of this is to have a npm script that can clean up files copied from node_modules folder to the place we want by webpack. This enhancement have at least following two benefits:

1. Reduce the complexity of developing front-end javascript/css based on new npm packages
2. boost the robustness of the deployment through deploy.sh

That is because currently we need to manually clean up files collected by webpack.

High level usage

Run this command: npm run clean.

When we want update (deploy) new genomics-workspace from one version to another version, the procedure will be following:

1. npm run clean
2. use git switch to newer version
3. npm run build
4. deploy.sh

Low level details

The current implementation in my mind is to have a script would remove files listed in .gitignore, but would not remove media folder.

Just placed some document materials about testing. This will be add to readthedocs later.

To test whole repo-wide settings:
python manage.py test i5k.tests

To test single app, for example, to test blast app:
python manage.py test blast.tests

To test specific test case, for example, TestClickAll test case of blast app:
python manage.py test blast.tests.TestClickAll

When working on complicated genes, which require iterative blasting or blasting for specific exons or motifs individually, the back browser functionality of the legacy system recalls the species and the type of blast (e.g., tblastn) automatically, which saves time and worry. (Going from results back to submission and keeping some settings see below)

1. integrate the duplicate module from addblast and addhmmer
2. fix django function
3. fix trailing space issue
4. consider to move CWL project under django app

ideal command :
python manage.py delete [organism]
python manage.py delete [blastdb]
python manage.py delete [hmmerdb]

advanced :
delete multiple db or species in the same time
python manage.py delete [organism1] [organism2] . . .
python manage.py delete [blastdb1] [blastdb2] . . .

another delete command:
python manage.py delete -org [organism1]
python manage.py delete -b [blastdb1]
python manage.py delete -h [hmmerdb1]

advanced command:
python manage.py delete -org [org1] [org2] . . .
python manage.py delete -b [blastdb1] [blastdb2] . . .
python manage.py delete -h [hmmerdb1] [hmmerdb2] . . .

When the species is deleted, the cascade field and data should be deleted at the same time.
As same as the situation of species, it should deletes the cascading table when blastdb and hmmerdb be deleted but not the species.

add the feature that can generate the short_name automatically

In order to work with CWL, make the argument input available, via the argument "Genus_species" ( the name from CWL definition) we can get the display name of organism

Using web scraping to get the description of organism and the tax_id for each organism
reference:　

1. https://medium.freecodecamp.org/how-to-scrape-websites-with-python-and-beautifulsoup-5946935d93fe
2. https://ithelp.ithome.com.tw/articles/10190994
3. https://medium.com/pyladies-taiwan/%E5%BE%9Edcard%E7%B6%B2%E7%AB%99%E7%9C%8B%E7%88%AC%E8%9F%B2%E5%85%A5%E9%96%80-65105f0ddac
4. https://github.com/1040003585/WebScrapingWithPython/tree/master/2.%E6%95%B0%E6%8D%AE%E6%8A%93%E5%8F%96

After loading hundreds of BLAST databases, our sequences table is getting really, really big.

I think the only reason the sequences table exists is for doing lookup to get the sequence form the input fasta file.

With BLAST+, the blastdbcmd program can return a sequence from the ID, removing the need for the table, and the extra steps on setting up a new database.

What functions use the sequences table?
Can these functions be refactored to use existing BLAST apps, or simplified?

@deming7h777 commented on Thu Oct 11 2018

There are some file have been removed but the configuration is still in settings.py.(ex. jquery-ui.js,...)
It may confuse someone who want to figure out the whole genomics-workspace.

@hsiaoyi0504 commented on Fri Oct 12 2018

@deming7h777 Can you elaborate a little bit ? It seems to me that jquery-ui.js is still useful.

@deming7h777 commented on Fri Oct 12 2018

I can't find the jquery-ui.js in blast/script directory and even all of the repo, can you give me some suggestion

@hsiaoyi0504 commented on Fri Oct 12 2018

It's because it's installed by webpack:

@deming7h777 commented on Sat Oct 13 2018

I have seen that, i mean i don't find any script or html import jquery-ui.js

@hsiaoyi0504 commented on Sat Oct 13 2018

It's imported by django-pipeline through

@hsiaoyi0504 commented on Sat Oct 13 2018

BTW, this perhaps is a good alternative of django-pipeline: https://django-compressor.readthedocs.io/en/latest/. One important feature attracts me is that

JS/CSS belong in the templates
Every static combiner for Django I’ve seen makes you configure your static files in your settings.py. While that works, it doesn’t make sense. Static files are for display. And it’s not even an option if your settings are in completely different repositories and use different deploy processes from the templates that depend on them. - django-compressor

@deming7h777 commented on Mon Oct 15 2018

thanks for reminding !

Currently, we include the DAP government code into our template

https://github.com/NAL-i5K/NAL-genomics-workspace/blob/5b6e56257be1094e0c70ce8c2e45cda41db26bef/app/templates/app/layout.html#L19

but based on digital-analytics-program/gov-wide-code#65, we can now track this dependency by package.json and include it into app-layout.js generated by django-pipeline

Related links:

https://stackoverflow.com/questions/39732397/install-specific-branch-from-github-using-npm
https://stackoverflow.com/questions/14187956/npm-install-from-git-in-a-specific-version

when running setup.py it's be good to output some messages to show what's going on.

As shown in following line, load-peptide class binds to the txt load. I will fix it later.