1. git clone this repository, create a virtual environment in your home dir and install the required packages

   # git clone [email protected]:eddyod/pipeline_utility.git
   git clone https://github.com/eddyod/pipeline_utility.git
   sudo python3 -m venv /usr/local/share/pipeline
   cd pipeline_utility
   source /usr/local/share/pipeline/bin/activate
   pip install -r prerequirements.txt
   pip install -r requirements.txt

2. We are currently using Ubuntu 18.04 as of October 2020. Either install this on your local machine or install it as a VM with Virtualbox or VMware. Note, using Ubuntu 20.04 also works, and since our servers will eventually get upgraded to that, you may as well install 20.04
3. Create this directory to start with: sudo mkdir -p /net/birdstore/Active_Atlas_Data/data_root/pipeline_data/DK52/preps/CH1
4. Make yourself user: sudo chown -R $(id -u):$(id -g) /net
5. Get some thumbnails to start with rsync -auv ratto.dk.ucsd.edu:/net/birdstore/Active_Atlas_Data/data_root/pipeline_data/DK52/preps/CH1/thumbnails/ /net/birdstore/Active_Atlas_Data/data_root/pipeline_data/DK52/preps/CH1/thumbnails/
6. You can now experiment with some of the thumbnails for DK52

1. Clone the repository, use the same virtualenv as above. You might need to install some more packages.

   git clone [email protected]:eddyod/ActiveBrainAtlasAdmin.git
   source /usr/local/share/pipeline/bin/activate

• For complete instructions, look at this page: https://www.digitalocean.com/community/tutorials/how-to-install-mariadb-on-ubuntu-20-04 Step-by-step guide:

1. Install and run mysql

   sudo apt update
   sudo apt install mariadb-server
   sudo mysql_secure_installation
   sudo mysql -u root -p

2. Create a new user and a new database:

   CREATE USER 'dklab'@'localhost' IDENTIFIED BY '<your_password_here>';
   GRANT ALL ON active_atlas_development.* TO 'dklab'@'localhost';
   
   CREATE DATABASE active_atlas_development;

3. Disconnect the database.
4. Setup the database user by creating a file: ~/.my.cnf in your home directory on your local machine:

   [client]
   user                        = dklab
   password                    = <your_password_here>
   port                        = 3306
   host                        = localhost
   

5. Fetch the database with the last backup from ratto (to current directory), and import it to the database:

   last_backup=`ssh ratto ls -1tr /net/birdstore/Active_Atlas_Data/data_root/database/backups/ | tail -1`
   rsync -auv ratto:/net/birdstore/Active_Atlas_Data/data_root/database/backups/$last_backup ./
   gunzip < $last_backup | sed 's/\DEFINER\=`[^`]*`@`[^`]*`//g' | mysql active_atlas_development

6. Test by going into the database and running some commands:

   mysql active_atlas_development

   In SQL prompt:

   show tables;

1. Here is a list of the software we use on a daily basis
2. Visual Code - IDE for python and typescript. This is free and works on most OSs.
3. Dbeaver - database GUI tool
4. imagemagick - used for converting images.
5. matlab - we are just starting to use this. UCSD license is also available
6. jupyter notebooks
7. Fiji, port of ImageJ
8. 3D Slicer
9. Gimp - image editing software
10. Geeqie - image viewer

1. The base directory is located on birdstore at: /net/birdstore/Active_Atlas_Data/data_root/pipeline_data
2. All brains are located in the base directory.
3. To view the post tif pipeline process go here: Neuroglancer process
4. The directory structure of a 3 channel brain will look like this:

1. Annotation keys are viewable in the database: https://activebrainatlas.ucsd.edu/activebrainatlas/admin/neuroglancer/structure/

1. The development and production databases are backed up multiple times each day on basalis
2. If you need a backup, look on basalis at: /net/birdstore/Active_Atlas_Data/data_root/database/backups/
3. The development database is named active_atlas_development
4. The production database is named active_atlas_production

1. Refer Checking for existing SSH keys and Generating a new SSH key and adding it to the ssh-agent for setting up the SSH on your local machine.
2. Substitute hostX and domainname names below with real names
3. Appending the following test in the SSH config file ~/.ssh/config to allow SSH server name aliasing

Host host1
  HostName host1.domainname
  User <AD_username_here>

Host host2
  HostName host2.domainname
  User <AD_username_here>

Host host3
  HostName host3.domainname
  User <AD_username_here>

Then copy the SSH identity to the remote server, enter your AD password when prompted.

for server in host1, host2 host3; do
    ssh-copy-id -i $server
done

Now you should be able to SSH into the servers without password.