global19 / icity Goto Github PK

This file describes scripts designed to search for genes linked to set of the 
baits defined by user. Scripts given in order of their use general workflow 
of the pipeline. Pipeline takes as input: BLAST database, coordinates of 
coding sequences of proteins present in the database and set of baits 
(coordinates in contigs).

Scripts designed to be used in Unix environment and require following programs
to be installed and path to executables should be exported with "export PATH=":
Python 3.xx: https://www.python.org/downloads/ (scripts provided for protocol were implemented with python 3.4)
NCBI BLAST suite: ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/LATEST/ (version 2.7.1 was used for example dataset and scripts provided).
Clustering tools: mmseqs2 (preferred) can be downloaded at https://github.com/soedinglab/MMseqs2 (mmseqs version used in protocol was downloaded at Jule 2018)  
Sequence alignment: MUSCLE http://www.drive5.com/muscle/ (MUSCLE v3.7 version was used in the protocol)

All the scripts can be found at:
ftp://ftp.ncbi.nih.gov/pub/wolf/_suppl/icityNatProt/


icity.py
Python batch file that run all computational steps of the pipeline.  Requires all python files described below.
Script reads config.py file for input parameters and executes following steps of the pipeline:
	Step 5, Identify protein-coding genes around the baits
	Step 6, Collect protein IDs from bait neighborhoods
	Step 7, Get protein sequences from the database
	Step 8, Run clustering with permissive parameters
	Step 9, Create protein profiles from representatives of a permissive cluster
	Step 10, Run BLAST search for generated protein profiles
	Step 11, Sort blast hits between clusters
	Step 12, Calculate relevance metrics for all protein clusters

Example:
python icity.py
results will be stored in files specified in config.py (default files are: Relevance.tsv - contains data for ICITY 
metric for all clusters, VicinityPermissiveClustsLinear.tsv - contains information about cluster members)


config.py
Configuration file for icity.py. Have the following fields:
ICITY_CONFIG_INPUT - Parameters of the pipeline
#Parameter Name					Value for the example dataset	Description
PTYFile							Database/CDS.pty				Path to the file generated in Step 1
SeedsFile						Seeds.tsv						Path to the file generated in Step 3
NeighborhoodVicinitySize		10000							Offset around seed (base pairs)
PathToDatabase					Database/ProteinsDB				Path to the database generated in Step 1
PermissiveClusteringThreshold	0.3								Sequence similarity clustering threshold
SortingOverlapThreshold			0.4								Overlap threshold to sort BLAST hits
SortingCoverageThresold			0.25							Coverage threshold to sort BLAST hits

ICITY_CONFIG_OUTPUT - Output files of the pipeline
#Parameter Name				Example Value						Description
ICITYFileName				Relevance_09.tsv					File with ICITY values for protein clusters 
VicinityClustersFileName	VicinityPermissiveClustsLinear.tsv	File with protein clusters information 

ICITY_CONFIG_TEMPORARYFILES - List of temporary files generated by the pipeline


SelectNeighborhood.py
  -h, --help  show this help message and exit
  -p P        PTYDataFileName, complete pty for contigs. PTY file should 
			  contain following values: 
			  LocusID	ORFStart..ORFStop	Strand	OrganismID	ContigID	Accession Number	GeneratedGI			  
			  separated by tab symbol.
  -s S        SeedsFileName, seeds tsv file. File should contain following
              values: LociID	ContigID	Start	Stop
			  separated by tab symbol
  -o O        ResultFileName, output pty file that contain following values:
              GI	ORF Coordinates	Strand	Genome	Contig
			  separated by tab values
  -d D        Offset around seed (base pairs)
Script takes subset of pty file in vicinity of baits and saves to file 
specified.

Example: To select proteins in +-10kb vicinity of the seeds in example dataset 
run and save it into Vicinity.tsv file run:
python SelectNeighborhood.py -p Database/CDS.pty -s Seeds.tsv -o Vicinity.tsv -d 10000


sh RunClust.sh
-	Argument 1: FASTA file name
-	Argument 2: Sequence similarity clustering threshold
-	Argument 3: Result clusters FileName

This script call will cluster protein sequences presented in file FASTA file 
using sequence similarity cut off saves results into file specified.

Example: to cluster protein sequences presented in file Vicinity.faa using 
sequence similarity cut off equal to 0.3 and saves results into 
VicinityPermissiveClustsLinear.tsv run following command
sh RunClust.sh Vicinity.faa 0.3 VicinityPermissiveClustsLinear.tsv


RemoveFASTAIDRedundency.py
  -h, --help  show this help message and exit
  -f F        FASTA file name
Script removes everything except first ID in FASTA ID line in the file

Example:
python RemoveFASTAIDRedundency.py -f Vicinity.faa > VicinityShortID.faa
This call will reduce following FASTA ID
>gi|1000270260|gb|AAD36845.1| AAD36845.1 N-acetyl-gamma-glutamyl-phosphate reductase [Thermotoga maritima MSB8]
to
>1000270260


ConvertOutput.py:
  -h, --help  show this help message and exit
  -f F        Cluster file name
Script converts mmseq tab separated file output to file where each line present
one cluster. Output file will contain cluster id separated by tab character 
with cluster members separated by space character. Linear format is needed for 
scripts used below.

Example: 
python ConvertOutput.py -f VicinityPermissiveClusts.tsv > VicinityPermissiveClustsLinear.tsv
Will convert VicinityPermissiveClusts.tsv to new format in
VicinityPermissiveClustsLinear.tsv


MakeProfiles.py
  -h, --help  show this help message and exit
  -f F        Clusters file name
  -c C        Folder name where profiles will be saved
  -d D        Path to protein database
Script will create protein profile for proteins from genomic database using 
MUSCLE for each permissive cluster in and save it to CLUSTERS folder with 
“.ali” extension and CLUSTER_ prefix with line number after as cluster ID, if 
directory don’t exists it will create it.

Example:
python MakeProfiles.py -f VicinityPermissiveClustsLinear.tsv -c CLUSTERS/ -d Database/ProteinsDB
Script will make profiles for each line presented in 
VicinityPermissiveClustsLinear.tsv and save it to CLUSTERS folder.


RunPSIBLAST.py
  -h, --help  show this help message and exit
  -c C        Folder name where profiles stored
  -d D        Path to protein database
Script will run PSIBLAST for each cluster present in specified folder and save
BLAST hits to the same folder with .hits extension with following format:
ProteinID	BLAST Score	Alignment Start	Alignment Stop	Alignment Sequence	CLUSTERID	Contig	Is in Vicinity Islands	ORF Start	ORF Stop	Distance to the bait

Example:
python RunPSIBLAST.py -c CLUSTERS/ -d Database/ProteinsDB
Will take sequence alignments in CLUSTERS folder, run PSIBLAST and save result
with .hits extension to the same folder


GetIcityForBLASTHits.py
  -h, --help  show this help message and exit
  -f F        Sorted PSIBLAST hits file name
  -o O        Result tsv file
  -d D        Genomic database
  -c C        Permissive clusters file name
Calculates number of different proteins at the baits, in all genomic database
and calculates median distance to the baits using sorted PSIBLAST search
results. Saves it to the file with following format:
Cluster ID	Effective size in vicinity of baits	Effective size in entire database	Median distance to bait (in ORFs)	Icity

Example:
python GetIcityForBLASTHits.py -f ClusterHitsFileName -o ResultFileName -d PathToDatabase -c PermissiveClustersFileName


CalculateICITY.sh
-	Argument 1: Clusters folder path
-	Argument 2: Path to protein database
-	Argument 3: Path to the file with clusters information
-	Argument 4: Result file name with clusters relevance information

Script run GetIcityForBLASTHits.py for each cluster blast hits file specified
in the folder. Saves results into separate file for each cluster in the same 
folder with CLUSTER_ prefix followed by the number of the cluster in cluster 
file and .tsv extension. Then merges all files into one and saves it into file
provided as 4th argument.

Example:
sh RunEffectiveSizeEstimation.sh CLUSTERS/Sorted/