convert_pdf may capture errant pages and make the created file too long about bankruptcy-parser HOT 11 CLOSED

How does this feel to you, @flooie ?

from bankruptcy-parser.

This is not doubting this - but could you add some screenshots of examples pages for reference?

from bankruptcy-parser.

Also @anthonysalvato3 did you see my last message on the closed PR? Should we just wrap that one up if we have a completed solution?

never mind - see it now

from bankruptcy-parser.

It feels like if we have test cases on all our PRs, it'd be really easy to know if this kind of logic was working.

from bankruptcy-parser.

Here is an example situation (with an unsupported form, but still applicable) where if I want to parse 206A/B, the text shows in the last 300 characters of this page, so it counts.

from bankruptcy-parser.

Thanks - I found an example in our test files as well. Im looking at a fix - because I think your example code - will leave it vulnerable to another potential bug.

from bankruptcy-parser.

@anthonysalvato3

What about this ...

we change drop the page requirement because I came across an example that does not use page 1 of X in reviewing other form names. And just create a nice regex.

This regex pattern finds occurrences of a target form with optional word boundaries or parentheses around it (see some form with wrapped parentheses, while also excluding matches where the words "Official Form" come after the target string. This avoids finding anything but the final form.

    pages = []
    with pdfplumber.open(filepath) as pdf:
        for page in pdf.pages:
            text = page.extract_text()
            if text[-300:].find(form) == -1:
                continue
            if text[-300:].lower().find("page") == -1:
                continue
            pages.append(page.page_number - 1)

    if not pages:
        return False

to

    pages = []
    with pdfplumber.open(filepath) as pdf:
        for page in pdf.pages:
            text = page.extract_text()
            pattern = r"(?<!\w)(?:\(|\b){}(?:\)|\b)(?!\w(.*Official Form))".format(
                re.escape(form)
            )
            if not re.findall(pattern, text[-300:]):
                continue
            pages.append(page.page_number - 1)

    if not pages:
        return False

from bankruptcy-parser.

I simplified it a little more in the PR.

Added two tests. Would appreciate your feedback and testing though @anthonysalvato3

from bankruptcy-parser.

I didnt think the looping would avoid the issue of one page forms - having a second or third incorrectly labeled page. And I thought the regex pattern that only finds the form if it is the last usage of official form on the page was cleaner.

from bankruptcy-parser.

I only have 1 test example of my own but it works on that, and I like the solution because mine wasn't very well thought-out.

from bankruptcy-parser.

@anthonysalvato3 do you use the docker image or would a pypi bump be sufficient for your current usage

from bankruptcy-parser.