PowerBLAST: RELEASE 2 (1/6/97) (README last modified 2/23/99) PowerBLAST can now be run with either the command line interface ('pblcmd') or a REAL graphical user interface ('powblast'). You still need connection to both network BLAST server and Entrez network service to run the program. Please read the manual carefully. If you have any questions about the display of the TEXT alignment, please check out the examples in this file. For questions and bug report, please send email to: blast-help@ncbi.nlm.nih.gov ############################################################ New Features ############################################################ 1)Graphical User Interface. 2)Search with Multiple BLAST Programs. 3)Save the Settings for Both PowerBLAST Specific Options and BLAST Search Parameters. 4)Options for Dumping out a HTML Page for TEXT Alignment 5)Option for Monitoring the Search Process 6)Better Manual ############################################################ Content of README ############################################################ The README includes 1)Manual for Graphical User Interface (powblast) 2)Manual for Command Line Interface (pblcmd) 3)Examples for the TEXT output format 4)Some Excerpt from the draft of the PowerBLAST paper that explains the system and algorithm of PowerBLAST ############################################################ Manual for Graphical User Interface ############################################################ PowerBLAST start up with a window for setting up the PowerBLAST special options as well as the BLAST search parameters. *********************** Set up BLAST Parameters *********************** Push the Button "Blast Program" on the top right of the window and you will have a new window titled "Parameter and DataBase for Blast Search". This window allows you to set parameters for multiple database searches with multiple BLAST programs. It has two sections. The top is to set parameters for searching against nucleotide databases, and the bottom for searching against protein databases. a)Select databases On the top section, you can select multiple databases by checking the boxes such as "nr", "sts", and "est". On the bottom, you can check the boxes such as "nr", "swissprot", and "pdb". In addition to the check boxes, you can type the name of a BLAST search database in the dialog box "other". That gives the flexibility of including a new search database if it is not covered by the check boxes. b)BLAST programs You can check "BLASTN" and "TBLASTN" for searching against nucleotide databases. You can check "BLASTX" and "BLASTP" for searching against protein databases. If none of the BLAST program is selected, PowerBLAST will run BLASTP for a protein query sequence and BLASTN for a nucleotide query sequence. PowerBLAST also checks the consistency between query and the selected BLAST program. For example, if the query sequence is a protein and if one of the selected program is BLASTN, it will skip the BLASTN option. c)Parameters for BLAST search Set the parameters for BLAST search in the dialog box following the program name. !!!!NOTE!!!! The behavior of an empty dialog box is different in this version than the previous one. If no parameter is set, PowerBLAST will use the default setting in the regular BLAST search. In the old version, the default for BLASTN and BLASTX were set with high cutoff score (M=1 N=-3 S=40 S2=40 for BLASTN and S=90 S2=90 -filter=seg for BLASTX). If you are processing a large genomic sequence, those two sets of parameters work quite well, and you may consider to keep them as your default search parameter. For searching against protein databases, it will be good to set -filter=seg all the time to filter the low complexity regions in a protein sequence. If you have questions for setting up the BLAST search parameter, push the button "Help" at the bottom of the window to obtain the email address for help message of BLAST search. d)Action Buttons If you push the "Cancel" button, the option will be reverted to the previous setting. If you Push "Accept", the new setting will be effective. If you push "Help", you can get the email address for help message of BLAST search. ^^^^ TIPS ^^^^ If it is the FIRST time for you to run PowerBLAST, it is strongly recommended that you set up BLAST options and push "Accet". In the main window, push "Save Setting". The current setting will become the default setting when you run PowerBLAST next time. All you need to do is just load the query sequence. *********************** Input Query sequences *********************** You can load the query sequences either a) from an input file or b) by "pasting" to the Window. The input can be a FASTA formatted sequence file (contain one or multiple sequences), or a list of accession numbers, or a list of gis. If the input is from a FILE, it can also be a list of file names. If Accessions or gis are supplied, the query sequences will be fetched from the Entrez server directly. a) Type in the file name or Push the Button "Read Input File" to load the input file. b) Click at the empty panel underneath "Or Paste Query Formatted As:". This is important!!! If you don't place the cursor properly, you can paste the buffer in the wrong place !!! After that, go to the Pulldown Menu "Edit" and select "Paste" to paste the buffer to the panel. Only three formats are supported for this option: FASTA, GI or Accessions. The default is set to FASTA. You can specify the format by selecting from the pulldown list. If you Push the button "Clear Window", the panel will become empty again so you can correct the error. ******** Monitor ******** You can monitor the PowerBLAST search process by selecting "Use Monitor" or disable it by de-selecting the option. On Unix machines, sometimes, the monitor can be quite annoying. And you have to be prepared to Click OK when a query has no hit in the database. *********************** Mask the Repeat Region *********************** Type in the file name of a FASTA formatted repeat sequence library or load it by pushing the button "Mask Repeats". A sample file for human repeats, humrep.fsa, is supplied in the data directory. ***************** Gapped Alignment ***************** Select one of the Radio Buttons, "None", "SIM", "SIM2","SIM3", for Gapped Alignment Algorithm. SIM works for DNA-DNA and protein-protein alignment. SIM2 and SIM3 work for DNA-DNA alignment only. Note!! There is NO gapped alignment algorithm available for DNA-protein (BLASTX) and protein-DNA (TBLASTN) alignment. ********************* Other Filter Options ********************* Select "Low Complexity" region to filter the low complexity region in a DNA sequence with dust. Select "Self Hit" if your query is a GenBank sequence and you don't want to see any hits to its self. ************************* Organism Specific Search ************************* You can restrict the BLAST search results to include or exclude a specific group of organism. Use the radio button "None", "Include", "Exclude" to make the choice and put the organism name (either taxname, such as homo sapien or common name, such as mammal will do) in the dialog box. ******************* Output File Format ******************* The output from PowerBLAST will be saved in one or more selected file format: "TEXT" (file extention .ali), "HTML" (file extension .html), "Seq-align" (file extension .sat), and "Seq-entry' (file extension .ent). Both Seq-align(*.sat) and Seq-entry (*.ent) can only be viewed by Chromoscope. The TEXT file can be viewed directly and the HTML file can be opened with a WWW browser, such as netscape. It has hotlinks to the sequences in the public databases. *************** Action Buttons *************** The "Search" Button is disabled when there is no query sequence. It will be activated when there is an input file or the panel for pasting the results contains data. The "Save Setting" Button is very useful!!! It will save all the parameters that you have set (which include the BLAST search parameters as well as the other powerBLAST specific options) in the powblast configuration file and the next time you run the program, all the options will come up as the default setting. All you need to do is to load the query sequence. If you do NOT "Save Setting", next time, it will start with the same parameters as the current start-up. The "Quit" Button is used for quit the program. ********************** Some Advanced Options ********************** a)TEXT alignment The default formatting will display the annotated features together with alignments. You can disable this function by going to the Pulldown menu "Option" and deselect "Show Feature". b) Error Log file If you are processing a large amount of queries at one time, you may find the monitor annoying and turned it off. However, you may still want to know the records that found no hit. Go to the pulldown menu File and select "Save Error Log" to open an error log file for recording the sequences with no hit. ############################################################ Manual for Command Line Interface ############################################################ The Name of the Program is called "pblcmd". The argument list gets much more complicated compared with the previous version because new options are available To review the parameter list, type pblcmd - and you will get: power blast arguments: -i The file name for power blast job [String] -c Reset the options 0=No 1=Reset 2=Reset+Save 3=Modify 4=Modify+Save [Integer] Optional default = 0 range from 0 to 4 -l The repeat FASTA library file for filtering [String] Optional -d dust the sequence before blast [T/F] Optional default = TRUE -f filter the blast output with the organism? 0=NO 1=Keep 2=Filter [Integer] Optional default = 0 range from 0 to 2 -o the name for organism for filtering [String] Optional -s compute gapped alignment 0=No 1=sim1 2=sim2 3=sim3 [Integer] Optional default = 0 range from 0 to 3 -a export the results as 1=text(*.ali) 8=HTML(*.html) 2=Seq-align(*.sat) 4=Seq-entry(*.ent) [Integer] Optional default = 0 range from 0 to 15 -b type of blast 0=default 1=blastn 2=blastp 4=blastx 8=tblastn [Integer] Optional default = 0 range from 0 to 15 -N Search Nucleotide databases: 1=nr 2=est 4=sts 8=month 16=htgs 32=vector 64=mito 128=kabat 512=pDB epd=1024 yeast=2048 gss=4096 alu=8192 [String] Optional default = 1 -A Search Protein databases: 1=nr 8=month 128=kabat 256=swissprot 512=pdb yeast=2048 alu=8192 [String] Optional default = 1 -n Parameters for BLASTN, use quote [String] Optional -x Parameters for BLASTX, use quote [String] Optional -p Parameters for BLASTP, use quote [String] Optional -t Parameters for TBLASTN, use quote [String] Optional -q filter out the GenBank query itself [T/F] Optional default = FALSE -m Enable the Monitor [T/F] Optional default = TRUE ************ OPTIONS ************ -i: the file name for the query sequence(s), which can be FASTA formatted file with multiple sequences, a list of accessions/locus/gis, or a list of file names. -c: options for save the settings -c0 take the default settings from the config file. -c1 reset all the parameters by taking the values from the command line -c2 same as c1 and save the settings to the config file. -c3 take the default settings from the config file, modify the values with the user setting in the command line -c4 same as c3 and save the settings to the config file If it is the FIRST time for you to run pblcmd, it is strongly recommended that you use -c2 to set up your options in the most of the search fields. The settings will be saved into the config file and next time you run the program, if you choose -c0, it will automatically set up the previous options as the default. This option is a little bit awkward. I tried to mimic the GUI interface for saving the settings and being able to modify the some but not all the values. -l a FASTA formatted repeat library file for human repeats, humrep, is included in this package. If you want to filter human repeats, just do lhumrep. -d -dT mask the low complexity region in DNA query sequences by the dust program. -dF no dusting -f -f0 No organism filtering -f1 Include organism -f2 Exclude organism -o Name of the organism. Use quotes. -o"human" -s -s0 Do not run gapped alignment -s1 Run SIM -s2 Run SIM2 -s3 Run SIM3 -a The format for output files. You can select to save multiple formats by adding the numbers together. If you select -a5, it will produce both the TEXT alignment (*.ali file) and the Seq-entry ASN.1 file (*.ent file) -a1 TEXT alignment with the extension .ali -a8 HTML page with the extension .html -a2 ASN.1 Seq-align file with the extension .sat. You can view it in Chromoscope. -a4 Seq-entry ASN.1 file with the extension .ent. You can view in Chromoscope. -b BLAST programs. You can select multiple BLAST programs by adding the numbers together -b0 default. Use BLASTN for a DNA query and BLASTP for a Protein query. -b1 BLASTN -b2 BLASTP -b4 BLASTX -b8 TBLASTN PowerBLAST also checks the consistency between query and the selected BLAST program. For example, if the query sequence is a protein and if one of the selected program is BLASTN, it will skip the BLASTN option. -N the Nucleotide Databases for BLAST Search. You can run searches against multiple databases by adding the numbers together. -A the Protein Databases for BLAST Search. The settings are similar to -N. -n Parameters for BLASTN search. -x Parameters for BLASTX search. -p Parameters for BLASTP search. -t Parameters for TBLASTN search. !!!!NOTE!!!! The behavior of the unspecified choice for setting BLAST parameter is different in this version than the previous one. If no parameter is set, PowerBLAST will use the default setting in the regular BLAST search. In the old version, the default for BLASTN and BLASTX were set with high cutoff score (M=1 N=-3 S=40 S2=40 for BLASTN and S=90 S2=90 -filter=seg for BLASTX). If you are processing a large genomic sequence, those two sets of parameters work quite well, and you may consider to keep them as your default search parameter. For searching against protein databases, it will be good to set -filter=seg all the time to filter the low complexity regions in a protein sequence. -q -qT if the query is a GenBank sequence, filter the hits to itself -qF keep the hits -m -mT monitor the process -mF turn off the monitor ******************* EXAMPLE for pblcmd ******************* pblcmd -iH_214K23.seq -c2 -lhumrep -dT -f1 -o"human" -s2 -a5 -b5 -N3 -A257 -n"M=1 N=-3 S=40 S2=40" -x"S=90 S2=90 -filter=seg" -mT For this setting, it will reset the parameters and save them into the configuration file. It takes the input sequence file H_214K23.seq, run against the human repeat library to find the repeats, mask the low complexity regions in the query with dust. Keep only the human hits from BLAST search. Run SIM2 to produce gapped alignment. Save the results in both the TEXT and the ASN.1 file. Search both nr and est database for BLASTN and the parameter for BLASTN is "M=1 N=- 3 S=40 S2=40". For BLASTX, the parameter is "S=90 S2=90 - filter=seg". It will run with a monitor. It is a long parameter list. But once it is set with -c2, the next time, all you need is to run the search with pblcmd -iinput file to get the same results. ############################################################ Examples for TEXT output of the Alignment ########################################################### a) a simple DNA-DNA alignment 10 20 30 40 | | | | | | | | 12> 297 aattaaactgtatattctggataaataaaattatttcgac L24443> 1347 ........................................ D31734> 1344 ........................................ 3'UTR > 1344 **************************************** polyA_sign > 1367 ****** U25274> 1262 ...................a.....at... ****************** Sequence Identity ****************** In this output format, 12 is the query sequence. L24443, D31734, and U25274 are the BLAST hits. All the resides of the query sequence are displayed, while in the hit sequences, only the mismatched residues are displayed (the a and at in U25274). The identical residues are displayed as dots ".". The ">" symbol shows the orientation of the alignment. ">" for the plus strand and "<" for the minus strand. The number followint ">" indicates the position in the sequence. ******************** Feature Information ******************** For sequence D31734, there are two annotated features at the region where there is high similarity to the query. Both are marked as "*" underneath the DNA sequence. one is 3'UTR. 3'UTR > 1344 **************************************** The other is the polyA signal polyA_sign > 1367 ****** b) the combined view of BLASTN and BLASTX 10 20 30 40 | | | | | | | | 214K> 8837 ttgggtttctagactaaatacagtgtgggaatacacaata X03557> 192 ...aa..c......a.......................-- 56-KDa> 43 I E F L D K Y S V G I H N \ | cc 56-KDa> 48 T G05877> 24 ...an..c......a.......................-- \ | cc ______________________________________________________ frame=+1> I G F L D * I Q C G N T Q Y P09914 43 . E . . . 307041 43 . E . . . A25407 43 . E . . . _______________________________________________________ frame=+3 > W V S R L N T V W E Y T I P09914 42 Q I E F . D . K Y S V G . 307041 42 Q I E F . D . K Y S V G . A25407 42 Q I E F . D . K Y S V G . The results from BLASTN and BLASTX are separated by the line ____ into three panels. The top shows the results from BLASTN, the middle and the bottom show the results from BLASTX with frame = +1 and frame = +3, respectively. ******* BLASTN ******* The query sequence 214K has two BLASTN hits: X03557 and G05877 in this region. The gapped alignments were computed by SIM2, and the alignments were displayed as multiple pair- wise alignment. A gap on the master sequence, i.e. the query sequence, is displayed as an insertion in the matching sequence. At position 8852 of the query sequence, both of the hit sequences contain 2-bp insertions represented by \. | cc At the end of line, both have 2-bp gaps represented by dashes (--). In the aligned region, the mRNA sequence X03557 has a coding region feature, which is presented by labeling each amino-acid in the middle of the 3-base codon. As a result, the protein sequence displayed in this panel is derived from the annotation on the DNA sequence. The BLASTX display, the conceptual translation with the specified reading frame is displayed underneath the separation line. The conceptual translation is compared with matching sequences from the protein database. Identical residues are labeled by dots. In this view, there are 3 protein sequences, P09914, 307041, A25407, all of which align to the query sequence in both frame +1 and frame +3. The alignments for frame 1 translation stop at position 8852 on the query sequence, which corresponds the 2-bp gap in the query sequence (displayed as 2-bp insertions on the matching sequences). ############################################################ Algorithms: Excerpt from the draft of PowerBLAST paper ############################################################ METHODS Figure 1 illustrates the data processes in PowerBlast. Prior to a BLAST search, SIM2 computes repeat regions in the query sequence and the results are automatically annotated as repeat features in the query sequence. Those, together with the low complexity regions in a DNA sequence identified by dust (Kuzio, unpublished), are masked in a copy of the query sequence which will be sent to the BLAST server for database search. Four types of BLAST search may be conducted with PowerBLAST: BLASTN compares a nucleotide query to a nucleotide database; BLASTP compares a protein query to a protein database; BLASTX compares a translated nucleotide query to a protein database; TBLASTN compares a protein query to a translated nucleotide database. Large sequences are split into overlapping pieces and the results are merged at the end. An interface was developed to enable searches against multiple databases with multiple BLAST programs (Figure 2). Organism specific results can be obtained at any level of taxonomy index by filtering the HSP alignments inclusively or exclusively with Entrez Taxonomy Server. A suite of SIM algorithms (SIM, SIM2, SIM3) may be selected to compute more refined gapped alignments. The details of repeat filtering, process of large sequences, organism filtering and gapped alignments are described below. Filter Repeat Region To identify repeat regions in the query sequence, PowerBLAST uses the SIM2 algorithm to compute the top n non-intersecting gapped alignments between the query sequence and repeat sequences in a user supplied FASTA library file. A sample file for human repeat sequences, humrep (Makalowski, unpublished), is included in the package. In order to reduce false positive and false negative results, various parameters were tested in a experiment that compares the ALU repeats identified by SIM2 with the annotations in the public records ( Makalowski and Zhang, unpublished) and the optimal choice is the combination of scores>=20 and sequence identity > 65%. The end points of the alignments are taken as repeat regions, and if there are tandem repeats of the same repeat element, the leftmost and rightmost positions will be recorded as the end points of a single repeat region. The repeat regions will also be annotated automatically as features on the query sequence. Since repeat features are derived from the gapped alignments, the query sequence will be broken into overlapping pieces if its length exceeds 10,000bp because it is faster to compute alignments multiple times than to process the whole sequence at one time. Processing Large Genomic Sequence The memory and CPU-time requirements vary with the type of BLAST program as well as the composition and length of the query, PowerBlast uses an empirically derived maximum search size for each BLAST program. For BLASTN, the maximum size is 8000bp; for BLASTP, it is 4000aa; for BLASTX, it is 3000bp; and for TBLASTN, it is 2000aa. If the query sequence exceeds the threshold, it is broken into overlapping pieces and each piece is submitted as a separate query to the Network BLAST server. When the entire sequence is processed, the HSPs from the same match sequence are sorted by locations. If two neighboring HSPs overlap and cover the same diagonal, they will be merged into a larger HSP. The statistics from the HSP that has a higher score is assigned to the new HSP as an approximation of the real statistical value. Organism Filtering PowerBLAST employs two strategies for organism filtering to achieve the most efficient network communication with Entrez Taxonomy Server. If the selected organism has less than 1000 records in the public databases, all the Ids are loaded in memory. The BLAST hits will be compared locally with the list of the Ids. Otherwise, the Ids of the matching sequences will be sent over the network to Entrez server for evaluation. The user may choose either to include or exclude a certain taxonomy class. Gapped Alignment Three algorithms, SIM, SIM2, SIM3, can be selected to compute gapped alignments between the query sequence and the database matches. The original unmasked query sequence is used as the input to the SIM programs to ensure that the repeat regions are included in the alignments. SIM is a space efficient algorithm that generates the top n non-intersecting Smith-Waterman alignments between DNA-DNA or protein-protein sequences. However, it may be too slow for long sequences. SIM2 and SIM3 are much faster than SIM, but they only compute DNA-DNA alignments. SIM2 improves the speed by first constructing the n best non-intersecting chains of "fragments". It then applies the traditional dynamic programming algorithm to compute an optimal gapped alignment in a region delimited by the chain. SIM3 computes global alignments for sequences that have high similarity; it can be only used when a high cutoff score is set for the BLAST search. HSPs from a BLAST search supply the orientation and approximate range as input to the SIM programs so that the computation is much more efficient than aligning the entire sequences. They are sorted by location, and the gaps between the neighboring HSPs are analyzed to determine if more than one alignment needs to be computed because a large gap may impose a heavy penalty that terminates the alignment. The threshold is set to be 200 with the default setting of the SIM programs. The ends of the HSPs are extended (1000 bp for DNA sequences, 100 aa for protein sequences) so that the SIM programs will be able to compute more accurate end points.