Hi, Thank you for the development and maintenance of this useful package. With the latest GitHub version of pypdb, I tried to run MMseqs2 searches of many sequences using Query as below, but the command returned an error in some cases. This error seems to be dependent on query sequences. Here is a reproducible example with one of the problematic sequences.

Query

from pypdb import Query

aa_query = 'MGEILYFDTVLAPLSLFLPIGYHAYLWQCFKSKPSHTYIGIDALRRKGWFLDMKEDVDQKGMLAIQSVRNTLMSTIFIASIAVLVSMALAALTNNAYNASQLFRSAFFGSQIGGIVVLKYGSASLFLLVSFLCSSMAVGFLIDANFLINIGIGQFSSPAYTQTIFERGFTLALIGNRMLCMTFPLILWIFGPVSMALSSLALVWGLYELDFPGKLPSVKHG'
q = Query(aa_query, query_type='sequence', return_type='polymer_entity')
out = q.search()

/Users/kf/miniconda3/envs/pymol/lib/python3.8/site-packages/pypdb/util/http_requests.py:65: UserWarning: Too many failures on requests. Exiting...
  warnings.warn("Too many failures on requests. Exiting...")
/Users/kf/miniconda3/envs/pymol/lib/python3.8/site-packages/pypdb/pypdb.py:292: UserWarning: Retrieval failed, returning None
  warnings.warn("Retrieval failed, returning None")
Traceback (most recent call last):
  File "/Users/kf/Dropbox/repos/csubst/csubst/csubst", line 309, in <module>
    args.handler(args)
  File "/Users/kf/Dropbox/repos/csubst/csubst/csubst", line 34, in command_site
    main_site(g)
  File "/Volumes/kfssd1/Dropbox/repos/csubst/csubst/main_site.py", line 658, in main_site
    g['pdb'] = pdb_sequence_search(g)
  File "/Volumes/kfssd1/Dropbox/repos/csubst/csubst/main_site.py", line 605, in pdb_sequence_search
    best_hit = mmseqs2_out['result_set'][0]
TypeError: 'NoneType' object is not subscriptable

Process finished with exit code 1

perform_search

After reading #26, I also checked perform_search, but it ended up with another error as below. I would appreciate it if you could give me any advice. Thank you.

from pypdb.clients.search.search_client import perform_search
from pypdb.clients.search.operators.sequence_operators import SequenceOperator

aa_query = 'MGEILYFDTVLAPLSLFLPIGYHAYLWQCFKSKPSHTYIGIDALRRKGWFLDMKEDVDQKGMLAIQSVRNTLMSTIFIASIAVLVSMALAALTNNAYNASQLFRSAFFGSQIGGIVVLKYGSASLFLLVSFLCSSMAVGFLIDANFLINIGIGQFSSPAYTQTIFERGFTLALIGNRMLCMTFPLILWIFGPVSMALSSLALVWGLYELDFPGKLPSVKHG'
seq_op = SequenceOperator(sequence=aa_query, identity_cutoff=0.99, evalue_cutoff=1000)
out = perform_search(search_operator=seq_op, return_with_scores=True)

Querying RCSB Search using the following parameters:
 {"query": {"type": "terminal", "service": "sequence", "parameters": {"evalue_cutoff": 1000, "identity_cutoff": 0.99, "target": "pdb_protein_sequence", "value": "MGEILYFDTVLAPLSLFLPIGYHAYLWQCFKSKPSHTYIGIDALRRKGWFLDMKEDVDQKGMLAIQSVRNTLMSTIFIASIAVLVSMALAALTNNAYNASQLFRSAFFGSQIGGIVVLKYGSASLFLLVSFLCSSMAVGFLIDANFLINIGIGQFSSPAYTQTIFERGFTLALIGNRMLCMTFPLILWIFGPVSMALSSLALVWGLYELDFPGKLPSVKHG"}}, "request_options": {"return_all_hits": true}, "return_type": "entry"} 

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/kf/miniconda3/envs/pymol/lib/python3.8/site-packages/pypdb/clients/search/search_client.py", line 183, in perform_search
    return perform_search_with_graph(query_object=search_operator,
  File "/Users/kf/miniconda3/envs/pymol/lib/python3.8/site-packages/pypdb/clients/search/search_client.py", line 271, in perform_search_with_graph
    for query_hit in response.json()["result_set"]:
  File "/Users/kf/miniconda3/envs/pymol/lib/python3.8/site-packages/requests/models.py", line 910, in json
    return complexjson.loads(self.text, **kwargs)
  File "/Users/kf/miniconda3/envs/pymol/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/Users/kf/miniconda3/envs/pymol/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Users/kf/miniconda3/envs/pymol/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Can we access structure's pH with something like pypdb.describe_pdb(pdb_id)?
Thank you very much.

The following snippet from the demos now fails:

qq = make_query('protein')
kk = do_search(qq)
list_taxa(kk)

with the following traceback:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-33-b148cd1dd2a8> in <module>()
----> 1 list_types(kk)

/Users/william/python_files/pypdb/pypdb/pypdb.py in list_types(pdb_list)
   1129     infotypes = []
   1130     for pdb_id in pdb_list:
-> 1131         all_info = get_all_info(pdb_id)['polymer']
   1132         if type(all_info)==list:
   1133             infotypes.append(all_info[0]['@type'])

KeyError: 'polymer'

This means that the RCSB PDB must have changed the syntax of the output XML files. A future version should be able to dig up this information in the raw XML returned by the search, regardless of whatever file structure the RCSB is currently using.

Whenever I try to run this example:

search_dict = make_query('T[AG]AGGY',querytype='MotifQuery')
found_pdbs = do_search(search_dict)
print(found_pdbs)

Instead of this:
['3LEZ', '3SGH', '4F47']

I get really broken id's like this:
['EZ:1', '3SGH:1', '4F47:1']

Same thing happens when I try a different sequence.

Hello!
I've tired to install the package, but demos.ipynb didn't work until I used the command
pip install --user .

When I used pip install pypdb, python failed to find pypdb.clients module and when downloaded repository and used python setup.py install, python failed to find pypdb.clients.search.operators module.

My os is macOS Mojave 10.14.6, I use conda 4.9.2.

Hello,

Searching only returns 100 results. Is it possible to get all the results and download them?

Hi there,

I'd love to help with the refactor to use the new RPDB API to keep this package up-to-date and useable.

Are there any functions / etc you'd like me to start with?

Cheers,

• lacoperon

Ran into an issue searching with database today - noticed that this module queries V1 API which was removed yesterday and replaced by V2. Looks like the module here needs to be updated to reflect this

I was trying to use PyPDB to search for a protein sequence and get a list of the matching PDB IDs and BLAST scores (i.e. expectation values). I wasn't able to do this with PyPDB, but I eventually got this working. The key was using a GET request to access the PDB API instead of the POST XML interface (which doesn't return BLAST scores, for some reason). I wrote a new function pdb_blast_search() to do this, and figured I would share it, in case you want to incorporate it into PyPDB at some point.

Cheers,
Dan

import pypdb
import untangle
import requests
import pandas as pd
import numpy as np

# set up variables
ALDHseq = 'MTKIANKYEVIDNVEKLEKALKRLREAQSVYATYTQEQVDKIFFEAAMAANKMRIPLAKMAVEETGMGVVEDKVIKNHYASEYIYNAYKNTKTCGVIEEDPAFGIKKIAEPLGVIAAVIPTTNPTSTAIFKTLIALKTRNAIIISPHPRAKNSTIEAAKIVLEAAVKAGAPEGIIGWIDVPSLELTNLVMREADVILATGGPGLVKAAYSSGKPAIGVGAGNTPAIIDDSADIVLAVNSIIHSKTFDNGMICASEQSVIVLDGVYKEVKKEFEKRGCYFLNEDETEKVRKTIIINGALNAKIVGQKAHTIANLAGFEVPETTKILIGEVTSVDISEEFAHEKLCPVLAMYRAKDFDDALDKAERLVADGGFGHTSSLYIDTVTQKEKLQKFSERMKTCRILVNTPSSQGGIGDLYNFKLAPSL'
ADHseq = 'PEKIYIKRGCLPVALDELKNVMGKKKAFIVTDNFLYNNGYTKPITDKLDEMGIVHKTFFDVSPDPSLASAKAGAAEMLAFQPDTIIAVGGGSAMDAAKIMWVMYEHPEVDFMDMAMRFMDIRKRVYTFPKMGQKAYFIAIPTSAGTGSEVTPFAVITDEKTGIKYPLADYELLPDMAIVDADMMMNAPKGLTAASGIDALTHALEAYVSMLATDYTDSLALRAIKMIFEYLPRAYENGASDPVAREKMANAATIAGMAFANAFLGVCHSMAHKLGAFYHLPHGVANALMINEVIRFNSSEAPTKMGTFPQYDHPRTLERYAEIADYIGLKGKNNEEKVENLIKAIDELKEKVGIRKTIKDYDIDEKEFLDRLDEMVEQAFDDQCTGTNPRYPLMNEIRQMYLNAYYGGAKK'
adhEseq = 'MTKIANKYEVIDNVEKLEKALKRLREAQSVYATYTQEQVDKIFFEAAMAANKMRIPLAKMAVEETGMGVVEDKVIKNHYASEYIYNAYKNTKTCGVIEEDPAFGIKKIAEPLGVIAAVIPTTNPTSTAIFKTLIALKTRNAIIISPHPRAKNSTIEAAKIVLEAAVKAGAPEGIIGWIDVPSLELTNLVMREADVILATGGPGLVKAAYSSGKPAIGVGAGNTPAIIDDSADIVLAVNSIIHSKTFDNGMICASEQSVIVLDGVYKEVKKEFEKRGCYFLNEDETEKVRKTIIINGALNAKIVGQKAHTIANLAGFEVPETTKILIGEVTSVDISEEFAHEKLCPVLAMYRAKDFDDALDKAERLVADGGFGHTSSLYIDTVTQKEKLQKFSERMKTCRILVNTPSSQGGIGDLYNFKLAPSLTLGCGSWGGNSVSDNVGVKHLLNIKTVAERRENMLWFRTPEKIYIKRGCLPVALDELKNVMGKKKAFIVTDNFLYNNGYTKPITDKLDEMGIVHKTFFDVSPDPSLASAKAGAAEMLAFQPDTIIAVGGGSAMDAAKIMWVMYEHPEVDFMDMAMRFMDIRKRVYTFPKMGQKAYFIAIPTSAGTGSEVTPFAVITDEKTGIKYPLADYELLPDMAIVDADMMMNAPKGLTAASGIDALTHALEAYVSMLATDYTDSLALRAIKMIFEYLPRAYENGASDPVAREKMANAATIAGMAFANAFLGVCHSMAHKLGAFYHLPHGVANALMINEVIRFNSSEAPTKMGTFPQYDHPRTLERYAEIADYIGLKGKNNEEKVENLIKAIDELKEKVGIRKTIKDYDIDEKEFLDRLDEMVEQAFDDQCTGTNPRYPLMNEIRQMYLNAYYGGAKK'
eCutoff = 1e-20

pdbUrl = 'http://www.rcsb.org/pdb/rest/getBlastPDB1'


# do a BLAST search for a given sequence, return results as a dataframe
def pdb_blast_search(seq, eCutoff):
    payload = {
            'sequence': seq,
            'eCutoff':str(eCutoff),
            'matrix':'BLOSUM62',
            'outputFormat':'XML' # or 'HTML' or nothing for text format
            }
     
    # create the url with parameters
    r = requests.get(pdbUrl, params = payload)

    # perform the web request and parse the resulting xml into a python object
    ro = untangle.parse(r.url)
    hits = ro.BlastOutput.BlastOutput_iterations.Iteration.Iteration_hits.Hit
    
    # make a dataframe with PDB IDs and BLAST search e-values
    resultList = []
    for hit in hits:
        pdbId = hit.Hit_def.cdata[0:4]
        eVal = float(hit.Hit_hsps.Hsp.Hsp_evalue.cdata)
        resultList.append([pdbId, eVal])

    df = pd.DataFrame(resultList, columns = ['PDB_ID', 'eValue'])

    # the PDB search doesn't parse the e-value cutoff correctly, so we have to do it again
    df2 = df.loc[df['eValue'] < eCutoff, :]

Bug Description

When calling the search method on a Query object that would return no results, an error similar to the following is raised

/home/o-linux/anaconda3/envs/proteins/lib/python3.7/site-packages/pypdb/util/http_requests.py:61: UserWarning: Too many failures on requests. Exiting...
  warnings.warn("Too many failures on requests. Exiting...")
Traceback (most recent call last):
  File "pypdb/mwe.py", line 4, in <module>
    found_pdbs = Query('McNoman, Norman', query_type='AdvancedAuthorQuery').search()
  File "/home/o-linux/anaconda3/envs/proteins/lib/python3.7/site-packages/pypdb/pypdb.py", line 253, in search
    if response.status_code == 200:
AttributeError: 'NoneType' object has no attribute 'status_code'

System info

• Operating system: Linux Debian, kernel 5.9.0-5-amd64
• Anaconda version: 4.9.2
• Python version: 3.7.9
• pypdb version: 2.0, from conda-forge

Duplication

This issue can be duplicated by using this python script

from pypdb import *
found_pdbs = Query('McNoman, Norman', query_type='AdvancedAuthorQuery').search()

Hello!
I'm trying to use the function get_blast() for getting a fasta file from the pdb site. But the following error is showing:
"name 'get_blast' is not defined".
How could I fix that?
My os is Ubuntu 18.04.
Thank you!!!

import pypdb
p = pypdb.describe_pdb('2H8N')
print(p['resolution']) # prints 2.60
print(p['nr_atoms']) # prints 0

Same thing with 4lza.

It's currently not possible to negate arbitrary text operators like 'contains_word', even though the API supports this. library only lets you use not_equal

pypdb.describe_pdb('5JHL').get('release_date') returns a None value.

A user emailed requesting the following functionality, which might be possible to do with the new API

We would like to query RCSB advanced search, pairing specific protein sequences with "number of distinct protein entities" (as the second parameter) to return a list of identical/similar proteins. We would then use the top query result.

Dear William,

I'm glad, that I discovered your package PyPDB! It seems to be really useful for me as a Bioinformatician.
I tried some of the sample codes and I encountered an error, while trying this sample code bellow. Is it because of the new API or is there a mistake on my side?
(The sample codes from the first section "Search functions that return lists of PDB IDs" work fine for me. I am using pypdb 2.0.)

Code:

Error:

Thank you very much in advance.
Best regards,

Marketa Pavlikova

Dear pypdb developers,

I am looking for an operation that retrieves non-polymer molecule information (chemical name, InChI, etc.) given a pdb code query. I noticed that the describe_chemical has been commented out, and this RCSB page shows that the API for web retrieval has been deprecated.

I am wondering if new open-source tools have been developed to adapt to these changes. If this is an unsolved issue, I am willing to contribute new functionalities to this repository. Thanks.

Hi, the function to download a given pdb file fails as the result from (the generic link in the examples)
http://www.rcsb.org/pdb/download/downloadFile.do?fileFormat=structfact&structureId=2F5N
now leads to an Error page.

I'm using Python 3.6.

Cheers,

Hi! We are super excited about the new rate limited requests! Any chance you have an estimated date for the new release? 1.300 is a year old now! Thanks!

When I search a Protein in PDB Bank, the result could be viewed by ligand, is there any way to retrieve these ligand SMILES by pypdb?

Hi,

I'm trying to search an attribute (rcsb_polymer_entity.rcsb_polymer_name_combined.names) which returns a list, but I can't seem to get it to work with any of the operators.

Any suggestions?

Dan

The get_all_info method results in a TypeError when the get_info method returns None. The code is attempting to access the result of the get_info method as a dictionary, which is not possible when its value is None.
This can probably be fixes with some simple exception handling code.

Hello. I want to know whether I can use PyPDB to search for PDB files via their Enzyme Classification number. Currently, I am able to search it with the normal query search but the results are more accurate if EC Number was searched specifically.

For instance, the url for the query search of EC number 2.3.2.5 in the PDB database is:
https://www.rcsb.org/search?request=%7B%22query%22%3A%7B%22type%22%3A%22group%22%2C%22nodes%22%3A%5B%7B%22type%22%3A%22group%22%2C%22nodes%22%3A%5B%7B%22type%22%3A%22terminal%22%2C%22service%22%3A%22text%22%2C%22parameters%22%3A%7B%22attribute%22%3A%22rcsb_polymer_entity.rcsb_ec_lineage.id%22%2C%22negation%22%3Afalse%2C%22operator%22%3A%22in%22%2C%22value%22%3A%5B%222.3.2.5%22%5D%7D%7D%5D%2C%22logical_operator%22%3A%22and%22%2C%22label%22%3A%22text%22%7D%5D%2C%22logical_operator%22%3A%22and%22%7D%2C%22return_type%22%3A%22entry%22%2C%22request_options%22%3A%7B%22paginate%22%3A%7B%22start%22%3A0%2C%22rows%22%3A25%7D%2C%22results_content_type%22%3A%5B%22experimental%22%5D%2C%22sort%22%3A%5B%7B%22sort_by%22%3A%22score%22%2C%22direction%22%3A%22desc%22%7D%5D%2C%22scoring_strategy%22%3A%22combined%22%7D%2C%22request_info%22%3A%7B%22query_id%22%3A%22e3de02ba278f11e20b06a6b6cc018332%22%7D%7D

which shows that rcsb_polymer_entity.rcsb_ec_lineage.id is used to access the query.
It would be really helpful for my project if I could search with EC numbers.
Thank you for taking the time

Hello William,

Hope you are doing fine. In the old version and API it was possible to obtain "chain ID"s through get_all_info("PDB_ID")['polymer']['chain'] functionality.

Unfortunately i couldn't spot a similiar way to obtain chain IDs with curren get_info("PDB_ID") is this a shortcoming of new API or is there a way to obtain chain ID (such as "A", "B", "C", etc.)

I'm looking for a way to automize obtainment of second column through API.

Kind regards.

Dear William,
Thanks for creating 'pypdb' module for such a large range of protein data.

As you can see from the picture, the PDB allows downloading either ideal CCDs or instance coordinates.

In the current state of pypdb development, do you think there is/will be a possibility to download specifically only ligands coordinates (.sdf of .mol2) similarly to the "get_pdb_file" function?
This would speed up the process of collecting crystallographic reference libraries. In addition, I noticed that these coordinates are already represented in Kekulè form without therefore adding an extra step of preparing the downstream molecules.
Best regards
Atilio

Hi, recently we got the "AttributeError: 'NoneType' object has no attribute 'status_code'" error for everything in our data, which worked before.
I tested with the demo code in pydpb and the same error showed up.
found_pdbs = Query(27499440, "PubmedIdQuery").search() print(found_pdbs[:10])
Can you have a look?
Thanks!

Hi there,

It seems that among the changes that were introduced with the new RCSB API, the structure of the range node was also changed.

RangeOperator.to_dict() currently outputs the following:

{
  "operator": "range", 
  "attribute": "cell.length_a", 
  "value": {
    "from": 80, 
    "to": 84, 
    "include_lower": true, 
    "include_upper": true
  }
}

while the same request using the Advanced Search outputs the following:

{
  "operator": "range",
  "attribute": "cell.length_a",
  "negation": false,
  "value": [
    80,
    84
  ]
}

Using the option range (upper included) results in "operator": "range_closed" instead of "range".

Thus, running the following code returns a 400 status:

from pypdb.clients.search.search_client import perform_search_with_graph, QueryNode, SearchService, ReturnType
from pypdb.clients.search.operators import text_operators

cell_a_operator = text_operators.RangeOperator(
    attribute='cell.length_a',
    from_value=80,
    to_value=84,
    include_upper=True
)

results = perform_search_with_graph(
    query_object=QueryNode(SearchService.TEXT, cell_a_operator),
    return_type=ReturnType.ENTRY
)

{
  "status" : 400,
  "message" : "Invalid operator value: search on [ cell.length_a ] attribute with [ range ] operator expects a date in ISO 8601 format: '2019-01-01T00:00:00Z' or a valid date math expression: 'now-1w', 'now+1M', 'now-2y', or '2020-06-01||-1w'.",
  "link" : "https://search.rcsb.org/redoc/index.html"
}

I'm using pypdb version 2.0 installed from PyPI.

Hi there,

For a project I'm doing, I'm using the RCSB API to get information about the particular chain-level information associated with a given PDB stored in the Data Bank.

For example, using various APIs, you can get the chain name (ie 'A'), chain description (ie 'yeast S12 protein', taxonomy (ie 'S. cerevisiae'), and if you do a join on two queries, other information like the sequence associated with said chain ('ATGAAATAG').

I was wondering if this sort of query would be useful with respect to this API, given that I've already pretty much implemented the logic in a repo which I wrote to query the RCSB API (having not heard about your project until now).

Thanks,

• Elliot

It looks like several of the lookup functions, like find_authors() and find_papers() are now timing out when I run them on my system. These functions involve putting in a large number of queries in order to aggregate results from many places in the PDB, and so it is possible that the PDB does not appreciate the volume. I'll look into bundling requests better for these functions.

Hello I've been using this module for about 4 months and I'm running a monthly script to scrap coronavirus-related data. I hadn't have any problems before but get_all_info function gives Type Error without any reason.

Here is the error:

I'm also using the latest version. (1.310)

I find that "get_ligands" is not defined (running in Jupyter Notebooks- use "!pip install pypdb" and "from pypdb import * "). On github pypdb.py the function is in comments, but it is not actually in code. Am I missing something? If not, is there another way to do this?
Thanks.

hi,
I'm wondering how I can search for PDBIDs using ligand SMILES. For example:

from pypdb import *
#
found_pdbs = Query("Clc1nc(Br)nc2nc[nH]c12").search()
print(found_pdbs[:10])

I'm able to find the PDBID I want using the 'chemical similarity' option on the 'https://www.rcsb.org/' website. However, I'm unable to achieve the same result using the code above. Can you please advise on the appropriate query_type to use in this case?

Hello,

I am trying to run the demos jupyter notebook and I get an error when running the second cell:
found_pdbs = Query("ribosome").search()
print(found_pdbs[:10])

I get:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-4-53b8e7e3615e> in <module>
----> 1 found_pdbs = Query("ribosome").search()
      2 print(found_pdbs[:10])

~/anaconda3/lib/python3.7/site-packages/pypdb/pypdb.py in search(self, num_attempts, sleep_time)
    251          data=query_text)
    252 
--> 253         if response.status_code == 200:
    254             pass
    255         else:

AttributeError: 'NoneType' object has no attribute 'status_code'

Any idea where the issue could come from?
Thanks you.

Is there a way to retrieve a list of PDBs for a given organism?

Similar to PDB's internal "Source Organism Taxonomy Name".