According to function stringify_affiliation, we return ' '.join(filter(None, parts)). So, it solve problem if we have multiple children node but for some case it returns excess space for example University , Turku , Finland. We should return more reliable string format.

In function pretty_print_xml, it does not print pretty output but just a long string.

Started hapenning a few days ago. Tried reinstalling a few libs, but to no avail. Do you have a suggestion?

Traceback (most recent call last):
  File "main.py", line 11, in <module>
    from util import *
  File "/home/sevajuri/projects/zsl/util.py", line 8, in <module>
    import pubmed_parser as pp
  File "/home/sevajuri/anaconda2/envs/tf/lib/python2.7/site-packages/pubmed_parser/__init__.py", line 7, in <module>
    from .pubmed_oa_parser import list_xml_path, \
  File "/home/sevajuri/anaconda2/envs/tf/lib/python2.7/site-packages/pubmed_parser/pubmed_oa_parser.py", line 2, in <module>
    from lxml import etree
ImportError: libicui18n.so.56: cannot open shared object file: No such file or directory

For author_list, it should be this following format instead, which I think it is easier to do post-process (i.e. linking author to affiliation).

[[first_name_1, last_name_1, aff_1], [first_name_1, last_name_1, aff_2], [first_name_2, last_name_2, aff_1]]

For affiliation_list, it should be list instead of dictionary format. Then we can add more function to return link of both list.

We have a function to parse table from Pubmed OA subsets. However, right now, it returns only tables that have one layer (only list of values per one column).

However, tables from Pubmed OA could be multilayers which we currently discard those rows of column names. I would like to have function that parse those table in a format that we can use pandas to convert it right away (e.g. JSON format or a list that we can convert it easily)

Hi, I found the code works well with MEDLINE 2016 baseline. However, when I tried to apply it to the MEDLINE 2017 baseline, only PMIDs were parsed, other fields are blank. Anyone aware of that? Thanks!

Chengkun

Let's add comments to each function and clean the code overall.

Hi, first of all big thanks for this life-saver of a package.

I think there is some problem with parsing XML for structured abstracts. Consider the following example:

         <Abstract>
          <AbstractText Label="" NlmCategory="UNASSIGNED">
            <b>Patient: Female, 16</b>
            <b>Final Diagnosis: Pelvic mass</b>
            <b>Symptoms: None</b>
            <b>Medication: None</b>
            <b>Clinical Procedure: CT • MRI</b>
            <b>Specialty: Diagnostic radiology • pediatrics.</b>
          </AbstractText>
          <AbstractText Label="OBJECTIVE" NlmCategory="OBJECTIVE">
            <b>Unusual presentation of unknown etiology, Rare disease, Mistake in diagnosis.</b>
          </AbstractText>
          <AbstractText Label="BACKGROUND" NlmCategory="BACKGROUND">Müllerian anomalies encompass a wide variety of malformations in the female genital tract, usually associated with renal and anorectal malformations. Of these anomalies, approximately 11% are uterus didelphys, which occurs when midline fusion of the müllerian ducts is arrested to a variable extent.</AbstractText>
          <AbstractText Label="CASE REPORT" NlmCategory="METHODS">We report the case of a 16-year-old female with uterine didelphys, jejunal malrotation, hematometra, hematosalpinx, and bilateral subcentimeter homogenous circular cystic-like renal lesions, who initially presented with left lower quadrant abdominal pain, non-bloody vomiting, and a history of irregular menstrual periods. Initial CT was confusing for an adnexal cystic mass, but further imaging disclosed the above müllerian anomalies.</AbstractText>
          <AbstractText Label="CONCLUSIONS" NlmCategory="CONCLUSIONS">Müllerian anomalies may mimic other, more common, adnexal lesions; thus, adequate evaluation of suspicious cystic adnexal masses with multiple and advanced imaging modalities such as MRI is essential for adequate diagnosis and management.</AbstractText>
        </Abstract>

The parse returned by medline_parser is as follows:

'Patient: Female, 16\n            Final Diagnosis: Pelvic mass\n            Symptoms: None\n            Medication: None\n            Clinical Procedure: CT \u2022 MRI\n            Specialty: Diagnostic radiology \u2022 pediatrics.'

As you can see, it completely misses a major portion of the text. I wonder if this is the case for all structured abstracts or only limited ones. As additional info, the file I'm using is ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/medline17n0763.xml.gz and the PMID of the abstract is 23826455.

Thanks!

• For Pubmed OA dataset, we want to extract information from acknowledgement section text
• For Medline dataset, it has GrantID section that we can write a parser to extract these number.

note that for now, I couldn't find example of xml file that has Grant or GrantID section. Feel free to let me know if anyone have example of xml file where we can parse GrantID section out

cc. @daniel-acuna

I believe (I could be wrong) there are three small mistakes on lines 192, 246 and 312 of pubmed_oa_parser.py in the parse_references, parse_paragraph and parse_pubmed_caption functions, respectively.

The code references a function join, but I have been unable to locate it. Perhaps this should be corrected to reference to the string method join (i.e., " ".join()) ?

I think we can speed up the parser a lot by parallelize this L105

path_list = [folder for folder in fullpath if os.path.splitext(folder)[-1] == ('.nxml' or '.xml')]
Is this code correct? ('.nsml', '.xml') can only return .nxml, so basically it can not get the .nxml from the path.
Do you mean path_list = [folder for folder in fullpath if os.path.splitext(folder)[-1] in ('.nxml', '.xml')]?

I attach code to do that here but still haven't cleaned it up. @davidbrandfonbrener can you take a look?

import pubmed_parser as pp
from lxml import etree

def join(l):
    return ' '.join(l)

path_xml = pp.list_xml_path('data/')
#tree = etree.parse('data/pntd.0002065.nxml')
tree = etree.parse(path_xml[0])
references = tree.xpath('//ref-list/ref[@id]')
dict_refs = list()
for r in references:
    ref_id = r.attrib['id']
    for rc in r:
        if 'publication-type' in rc.attrib.keys():
            if rc.attrib.values() is not None:
                journal_type = rc.attrib.values()[0]
            else:
                journal_type = ''
            names = list()
            for n in rc.findall('name'):
                name = join([t.text for t in n.getchildren()][::-1])
                names.append(name)
            try:
                article_title = rc.findall('article-title')[0].text
            except:
                article_title = ''
            try:
                journal = rc.findall('source')[0].text
            except:
                journal = ''
            try:
                pmid = rc.findall('pub-id[@pub-id-type="pmid"]')[0].text
            except:
                pmid = ''
            dict_ref = {'ref_id': ref_id, 'name': names, 'article_title': article_title, 
                        'journal': journal, 'journal_type': journal_type, 'pmid': pmid}
            dict_refs.append(dict_ref)

Web of Science data is probably valid to just few people since you need the license. However, there is example XML file avialable at https://github.com/yadudoc/wos_builder. Should we add function to parse WoS to Pubmed Parser?

Now the script here only download subset of the data.

These include the following:

• MedlineTA
• NlmUniqueID
• ISSNLinking
• Country

These data can be used as a linkage to Web of Science (WoS) dataset later on for people who own WoS dataset.

For a (small?) subset of documents, only part of the abstract is extracted (e.g. PMID 24653627, 23357879, 27983391, 26762307, 28005260, 22351618, 23456555,18006916,25371446)

Lists returned by parse_pubmed_paragraph does not include paragraphs if they do not have any references. I think it would be useful if paragraphs with no references can also be returned.

We can loop through each sec and grab title for that section. Then in each section, we go through each paragraph can get all references id and full text.

Some people might use this in publications, we should let them spread the credit :)

here is one example that it gives error Acta_Psychiatr_Scand/Acta_Psychiatr_Scand_2010_Feb_121(2)_152-156.nxml. I'll update the issue soon.

I wrote a script right here https://gist.github.com/titipata/3f896ab5017e95fbd8aeec25bc283b44. We can put as function parse_pubmed_table. Anyone want to help adding it to the library?

Now include_path=False is in parse_pubmed_xml_to_df, not in parse_pubmed_xml. It would be nicer to move directly into parse_pubmed_xml for PySpark user to specify as a flag. We should change README specification also.

The API here cannot convert all PMID input. I was trying to parse citations from given set of PMIDs but it only returns subset of PMIDs that I provided. One possibility is to host pair of PMIDs/PMCs somewhere on the cloud and provide similar API or source file that user can use to convert PMID to PMC.

There is a problem with some of the pubdate fields in the output. It is not pulling the correct year and instead is splitting the text based off of " " and grabbing the first chunk of text. Because of this you end up with results for pubdate like ["Summer","Winter"]. Some example pmid's this happens for is [28599031,28599032,28599033, etc]. Could you please update to match on some form of regular expression like "\d{4}" instead of splitting on the whitespace and just grabbing the first chunk?

@daniel-acuna catched this error when we run output = pp.parse_pubmed_xml('data/1472-6831-8-11.nxml') and then check an output type i.e. list(map(type, out.values()))

Output type is

[str,
 list,
 lxml.etree._ElementUnicodeResult,
 str,
 list,
 str,
 str,
 str,
 lxml.etree._ElementUnicodeResult,
 str]

We have to turn this type into proper string.

Right now, the parse accepts a path or file-like. We should allow strings too.

I think I'll use urllib instead of requests in order to reduce required packages.

In my environment python-2.7 is not able to find the module urllib.request required by pubmed_web_parser.py.

If I replace the urllib import statement as suggested in this discussion then python2 is able to find the request module:

try:
    from urllib.request import urlopen
except ImportError:
    from urllib2 import urlopen

urlopen is used by the function parse_outgoing_citation_web in the same file. For all http requests from this file requests library is used except the request made from the function parse_outgoing_citation_web. It could be possible to use the requests library here as well? to avoid the import.

--mahmut

There are some error related when parsing some PMID including PMID29323847 (book)

Below I've copied my python instance. I'm trying to parse medline data. I've done this with your pubmed and medline parser on the listed machine as well as on a ubuntu server with the same error. I've also generated a file using the R programming language. If you are familiar with that, the package I used is called easyPubMed. I used the batch_pubmed_download() function.

Anyways I'd really like to use your code, especially as it links the authors with their affiliated institutions.
I'm new to XML parsing so I have no idea what I'm doing in that respect.

Python 3.6.0 (v3.6.0:41df79263a11, Dec 23 2016, 08:06:12) [MSC v.1900 64 bit (AMD64)] on win32

>>>import pubmed_parser as pp
>>>pp.parse_pubmed_xml('C:\\Users\\Work\\Downloads\\medline16n0902.xml')

Error: it was not able to read a path, a file-like object, or a string as an XML
Traceback (most recent call last):
  File "C:\Program Files\Python36\lib\site-packages\pubmed_parser-0.1-py3.6.egg\pubmed_parser\utils.py", line 14, in read_xml
    tree = etree.parse(path)
  File "src\lxml\lxml.etree.pyx", line 3427, in lxml.etree.parse (src\lxml\lxml.etree.c:81101)
  File "src\lxml\parser.pxi", line 1811, in lxml.etree._parseDocument (src\lxml\lxml.etree.c:117832)
  File "src\lxml\parser.pxi", line 1837, in lxml.etree._parseDocumentFromURL (src\lxml\lxml.etree.c:118179)
  File "src\lxml\parser.pxi", line 1741, in lxml.etree._parseDocFromFile (src\lxml\lxml.etree.c:117091)
  File "src\lxml\parser.pxi", line 1138, in lxml.etree._BaseParser._parseDocFromFile (src\lxml\lxml.etree.c:111637)
  File "src\lxml\parser.pxi", line 595, in lxml.etree._ParserContext._handleParseResultDoc (src\lxml\lxml.etree.c:105093)
  File "src\lxml\parser.pxi", line 706, in lxml.etree._handleParseResult (src\lxml\lxml.etree.c:106801)
  File "src\lxml\parser.pxi", line 633, in lxml.etree._raiseParseError (src\lxml\lxml.etree.c:105612)
OSError: Error reading file 'medline16n0902.xml': failed to load external entity "medline16n0902.xml"
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\Program Files\Python36\lib\site-packages\pubmed_parser-0.1-py3.6.egg\pubmed_parser\medline_parser.py", line 354, in parse_medline_xml
    tree = read_xml(path)
  File "C:\Program Files\Python36\lib\site-packages\pubmed_parser-0.1-py3.6.egg\pubmed_parser\utils.py", line 17, in read_xml
    tree = etree.fromstring(path)
  File "src\lxml\lxml.etree.pyx", line 3213, in lxml.etree.fromstring (src\lxml\lxml.etree.c:78994)
  File "src\lxml\parser.pxi", line 1848, in lxml.etree._parseMemoryDocument (src\lxml\lxml.etree.c:118325)
  File "src\lxml\parser.pxi", line 1729, in lxml.etree._parseDoc (src\lxml\lxml.etree.c:116883)
  File "src\lxml\parser.pxi", line 1063, in lxml.etree._BaseParser._parseUnicodeDoc (src\lxml\lxml.etree.c:110870)
  File "src\lxml\parser.pxi", line 595, in lxml.etree._ParserContext._handleParseResultDoc (src\lxml\lxml.etree.c:105093)
  File "src\lxml\parser.pxi", line 706, in lxml.etree._handleParseResult (src\lxml\lxml.etree.c:106801)
  File "src\lxml\parser.pxi", line 635, in lxml.etree._raiseParseError (src\lxml\lxml.etree.c:105655)
  File "<string>", line 1
lxml.etree.XMLSyntaxError: Start tag expected, '<' not found, line 1, column 1

Incorporate created functionality from https://github.com/titipata/pubmed_parser_web to this repository

In both Pubmed Open Access and MEDLINE open access, when we parse abstract text. It mostly includes blank space and new line \n. We can preprocess that later using nltk whitespace tokenizer. However, it would be nice to return string without extra-space right away.

We select all string under affiliation where it comes with number. It should be a way to remove it from parser itself or we might use regular expression to remove it.

Right now, the function parse_pubmed_references return empty list of name. This happens in example folder mds526.nxml.

here is an example link that we want to parse http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2631471/citedby/

Some website require PMCID instead of PMID.

We want to grab figure label, caption and xlink:href for that file so we can relate caption with images later.

There can be one or more section in single abstract. For example, we can have BACKGROUND, METHOD, RESULT or just Abstract and RESULT in single abstract etc. We want to add some tagger when we parse these abstracts in case people want to analyze only few parts of abstract.

I'm still not sure which format should we return in this case.

Hey guys,
I'm getting the following error if I try to use your code.
Whats the problem?

Best regards

AttributeError                            Traceback (most recent call last)
<ipython-input-2-9968ea229488> in <module>()
      1 ## settings
      2 file = "medline17n1086.xml";
----> 3 pubmed_dict = pp.parse_pubmed_xml(file) # dictionary output

/usr/local/lib/python3.5/dist-packages/pubmed_parser-0.1-py3.5.egg/pubmed_parser/pubmed_oa_parser.py in parse_pubmed_xml(path, include_path)
    108         journal = ''
    109 
--> 110     dict_article_meta = parse_article_meta(tree)
    111     pub_year_node = tree.find('//pub-date/year')
    112     pub_year = pub_year_node.text if pub_year_node is not None else ''

/usr/local/lib/python3.5/dist-packages/pubmed_parser-0.1-py3.5.egg/pubmed_parser/pubmed_oa_parser.py in parse_article_meta(tree)
     56     """
     57     article_meta = tree.find('//article-meta')
---> 58     pmid_node = article_meta.find('article-id[@pub-id-type="pmid"]')
     59     pmc_node = article_meta.find('article-id[@pub-id-type="pmc"]')
     60     pub_id_node = article_meta.find('article-id[@pub-id-type="publisher-id"]')

AttributeError: 'NoneType' object has no attribute 'find'

Getting

> raise SyntaxError("cannot use absolute path on element")
> SyntaxError: cannot use absolute path on element
> 

Fixed when replacing all xpaths starting with // into .//

article_meta = tree.find('//article-meta') to article_meta = tree.find('.//article-meta')

It'd be useful to be able to get MeSH IDs as well as heading names. I've used the following quick-and-dirty change to medline_parser.py but there's probably a more elegant way.

mesh_terms_list = [m.find('DescriptorName').attrib.get('UI','') + ":" + m.find('DescriptorName').text for m in mesh.getchildren()]

In [41]: pp.parse_medline_xml('/home/docClass/files/pubmed/pubmed18n1040.xml.gz')
Error: it was not able to read a path, a file-like object, or a string as an XML
File "", line 1
XMLSyntaxError: Start tag expected, '<' not found, line 1, column 1

Source: ftp://ftp.ncbi.nlm.nih.gov/pubmed/updatefiles/pubmed18n1040.xml.gz

I tried to run the following test. Seems like it should have worked. Any suggestions welcome. Thanks!!

from pubmed_oa_parser import parse_pubmed_paragraph
from os import system

if __name__=="__main__":
 
    system("wget 'https://www.ncbi.nlm.nih.gov/pmc/oai/oai.cgi?verb=GetRecord&identifier=oai:pubmedcentral.nih.gov:156895&metadataPrefix=pmc' -O ./data/PMC156895.nxml") 

    dicts_out = parse_pubmed_paragraph('data/PMC156895.nxml', all_paragraph=False)

Output

--2017-06-14 22:23:06--  https://www.ncbi.nlm.nih.gov/pmc/oai/oai.cgi?verb=GetRecord&identifier=oai:pubmedcentral.nih.gov:156895&metadataPrefix=pmc
Resolving www.ncbi.nlm.nih.gov... 130.14.29.110, 2607:f220:41e:4290::110
Connecting to www.ncbi.nlm.nih.gov|130.14.29.110|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/xml]
Saving to: ‘./data/PMC156895.nxml’

./data/PMC156895.nxml                        [    <=>                                                                            ]  66.07K  67.0KB/s    in 1.0s    

2017-06-14 22:23:10 (67.0 KB/s) - ‘./data/PMC156895.nxml’ saved [67658]

<lxml.etree._ElementTree object at 0x1068596c8>
Traceback (most recent call last):
  File "scrape.py", line 9, in <module>
    dicts_out = parse_pubmed_paragraph('data/PMC156895.nxml', all_paragraph=False)
  File "/Users/fenwick/Library/Mobile Documents/com~apple~CloudDocs/other/web_scrape/pubmed_oa_parser.py", line 253, in parse_pubmed_paragraph
    dict_article_meta = parse_article_meta(tree)
  File "/Users/fenwick/Library/Mobile Documents/com~apple~CloudDocs/other/web_scrape/pubmed_oa_parser.py", line 57, in parse_article_meta
    pmid_node = article_meta.find('article-id[@pub-id-type="pmid"]')
AttributeError: 'NoneType' object has no attribute 'find'

How should we process the delete citations?

Sometimes the update XML comes with "deleted" citations (like this example), and it would be good to know which PMID were deleted.

For example, the stats for the update file medline16n0906.xml available [here ftp://ftp.nlm.nih.gov/nlmdata/.medlease/gz/medline16n0906_stats.html] says that there are 8809 citations and 353 delete citations. If we process the XML with pubmed_parser, we correctly get 8809-353 = 8456 records. Use the code below to test this:

# adapted from http://stackoverflow.com/questions/18772703/read-a-file-in-buffer-from-ftp-python
from ftplib import FTP
import gzip
import StringIO

def open_ftp_data(server, path, binary=True):
    ftp = FTP(server)
    ftp.login()

    data_io = StringIO.StringIO()
    def handle_data(more_data):
        data_io.write(more_data)
    if binary:
        resp = ftp.retrbinary("RETR " + path, callback=handle_data)
    else:
        resp = ftp.retrlines("RETR " + path, callback=handle_data)
    data_io.seek(0) # Go back to the start
    ftp.close()
    return data_io

import pubmed_parser as pp
binary_file = open_ftp_data('ftp.nlm.nih.gov', 'nlmdata/.medlease/gz/medline16n0906.xml.gz')
zippy = gzip.GzipFile(fileobj=binary_file)
medline_xml = zippy.read()
dict_records = pp.parse_medline_xml(medline_xml)
print("pubmed_parser records processed: {}".format(len(dict_records)))

Output

pubmed_parser records processed: 8456

We can find the deleted citations by simply

from lxml import etree
root = etree.fromstring(medline_xml)
print("Delete citations {}".format(len(root.xpath('//DeleteCitation/PMID'))))

Output

Delete citations 353

For several PMC files I get an error while reading in the content. list below:

PMC4569614.nxml 
PMC5362956.nxml 
PMC4162892.nxml 
PMC4569628.nxml 
PMC5348996.nxml 
PMC5362810.nxml 
PMC5352161.nxml 
PMC4522714.nxml 
PMC5352154.nxml 
PMC5363022.nxml 
PMC4522719.nxml 
PMC5346358.nxml 

P.S.
Ill post the traceback when I run it again.