Possible error while parsing structured abstracts. about pubmed_parser HOT 23 CLOSED

Yes, there's a dilemma here. Maybe we should simply use Label because` that's the way it's displayed on the website as well?

Or do something like NLMCategory - Label: and reduce it to just one of them in case of repetitions like OBJECTIVE - OBJECTIVE:

What do you think?

from pubmed_parser.

The check for UNASSIGNED is still failing. Replacing 'is not' by != fixes it. Also, we want to remove the content under UNASSIGNED section right? In which case, we may want to put abstract_list.append(stringify_children(abstract).strip()) inside the if condition.

No problem! Thanks for being so prompt on this. Let me know if I can contribute in any other way.

from pubmed_parser.

Okay, that works too :)

from pubmed_parser.

Thanks for reporting @RudrakshTuwani! Is there anyway that you can upload XML somewhere so that I can try to fix the parser?

from pubmed_parser.

See the updated description.

from pubmed_parser.

That's perfect. I'll fix that by the weekend. Bug me if I forget tho!

from pubmed_parser.

No problem, thanks a lot!

Do you think this is the case for all structured abstracts or it's specific to a few?

from pubmed_parser.

There are a lot of abstract like this. I haven't calculated the exact number. I guess it should be roughly a million from full Medline dataset.

from pubmed_parser.

Okay, thanks!

from pubmed_parser.

I seem to have found the bug.

if article.find('Abstract/AbstractText') is not None:
        abstract = stringify_children(article.find('Abstract/AbstractText')).strip() or ''
elif article.find('Abstract') is not None:
        abstract = stringify_children(article.find('Abstract')).strip() or ''
else:
        abstract = ''

So, the first if block matches the first tag under structured abstract and returns it. This can be solved by removing the condition for matching "AbstractText". Can you tell me why we are even matching AbstractText? Since, just matching on Abstract also returns all the relevant information.

from pubmed_parser.

Yes @RudrakshTuwani, you're right. I'm fixing that part now.

from pubmed_parser.

I fix it in commit b59e150, let me know if it works for you!

from pubmed_parser.

Yes, this seems to be working. Thanks a lot!

One suggestion for future, maybe we can add the corresponding field for the structured abstract in the beginning of the sentence. If you want, I can work on it and send a PR later.

from pubmed_parser.

Haha, you read my mind! I did that add just push it, see commit 34dae58

from pubmed_parser.

Haha, yes this seems to be working perfectly! Thanks a lot, man!

from pubmed_parser.

@RudrakshTuwani, sorry, I still see the problem in my parser atftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/medline17n0843.xml.gz. I'll fix it and close the issue soon.

from pubmed_parser.

Oh, can you send me the PMID? I'll have a look as well.

from pubmed_parser.

This is 26368793, so now I parsed at got multiple METHODS section as

OBJECTIVE
To examine interleukin-12 (IL-12), IL-18, IFN-γ, intracellular adhesion molecule-1 (ICAM-1), leukemia inhibitory factor (LIF), and migration inhibitory factor (MIF) levels in precisely-timed blood and endometrial tissue samples from women with idiopathic recurrent pregnancy loss (RPL).


METHODS
Case-control study.


METHODS
University hospital.

It should be a better way to concatenate all the same section in one long string.

from pubmed_parser.

We are actually doing it correctly, you can see at: https://www.ncbi.nlm.nih.gov/pubmed/?term=26368793. However, I chose to get abstract.attrib.get('NlmCategory', '') instead of says Label="BACKGROUND". Still thinking what is the best way to do that. Label will be good for someone who wants to display the abstract. However, NlmCategory is also good if you want to make each sections consistent.

from pubmed_parser.

So I fixed it in f3fe97f as

dicts_out = pp.parse_medline_xml('data/medline16n0902.xml.gz', year_info_only=False, nlm_category=False)

If you set nlm_category=True, it will give result using NLM category instead.

from pubmed_parser.

Okay, I fixed it in a slightly different way. The output I get is the following:

OBJECTIVE: Unusual presentation of unknown etiology, Rare disease, Mistake in diagnosis. 

BACKGROUND: Müllerian anomalies encompass a wide variety of malformations in the female 
genital tract, usually associated with renal and anorectal malformations. Of these anomalies, 
approximately 11% are uterus didelphys, which occurs when midline fusion of the müllerian ducts is 
arrested to a variable extent. 

METHODS "CASE REPORT": We report the case of a 16-year-old female with uterine didelphys, 
jejunal malrotation, hematometra, hematosalpinx, and bilateral subcentimeter homogenous circular 
cystic-like renal lesions, who initially presented with left lower quadrant abdominal pain, non-bloody 
vomiting, and a history of irregular menstrual periods. Initial CT was confusing for an adnexal cystic 
mass, but further imaging disclosed the above müllerian anomalies. 

CONCLUSIONS: Müllerian anomalies may mimic other, more common, adnexal lesions; thus, 
adequate evaluation of suspicious cystic adnexal masses with multiple and advanced imaging 
modalities such as MRI is essential for adequate diagnosis and management.

Label if not equal to NLM category are added in quotation marks in front of the NLMCategory.

I guess we can close this now.

from pubmed_parser.

Definitely, we can close it for now. I'll discuss with @daniel-acuna a bit more about this issue. Let me know there is any use cases that you want output to be in particular format. Thanks again for the issue tho!

from pubmed_parser.

Fixed is not to !=, I amend the commit!

I think putting abstract_list.append(stringify_children(abstract).strip()) outside is correct cause we always add text into the list. However, if section is UNASSIGNED, I just skip it :)

from pubmed_parser.