Feature request: rearrange paragraphs for minimum difference about dinglehopper HOT 5 OPEN

An interesting more recent addition in the scientific community is the Flexible character accuracy measure.

As I have a similar problem and need a solution I will try to integrate the Flexible Character Accuracy as option for Dinglehopper.

from dinglehopper.

I'm leaning towards providing the UWER (unordered word error rate) in dinglehopper to resolve this.

Thoughts:

• I don't think a layout analysis feature - which reordering the paragraphs is - would be appropriate here in an evaluation tool. If there's a simple algorithm that solves most issues, there should a separate tool to do this in the OCR-D community.
• Just trying all permutations of paragraphs is IMHO no good as this would be on the order of O(m!) for m paragraphs

from dinglehopper.

I must agree, calculation of reliable accuracy rates with wrong segmentation order is beyond the possibilities of dinglehopper. The sheer amount of possible segmentation classes/errors is escalating way too quickly!

As always when it comes to the topic of evaluation, the PRImA group have some good publications about this, e.g. The Significance of Reading Order in Document Recognition and its Evaluation and Scenario Driven In-Depth Performance Evaluation of Document Layout Analysis Methods.

The typical solution for this adopted in other evaluation tools is to include the Bag-of-words (BOW) metric, which is easy to compute and could probably be supported by dinglehopper too.

An interesting more recent addition in the scientific community is the Flexible character accuracy measure.

from dinglehopper.

Flew through this paper. Does compare strings of GT with substrings of OCR (in case of erroneously joined columns).

( I assume the "equal-length distance" editDist(..., substr(..., t2.length )) is because of runtime considerations, but in theory this is does not need to be same length; I would suggest word boundaries. )

I'll still think such a flexible comparison is essential - before runnning ocr-d in production - to verify the workflows in use.

from dinglehopper.

@jbarth-ubhd

Flew through this paper. Does compare strings of GT with substrings of OCR (in case of erroneously joined columns).

Simplified Explanation: FCA compares a line from GT with all lines from OCR and either finds a satisfying match or splits the GT line into smaller fragments based on the best match found. There are more steps and some implementation details only visible in the Java Implementation.

( I assume the "equal-length distance" editDist(..., substr(..., t2.length )) is because of runtime considerations, but in theory this is does not need to be same length; I would suggest word boundaries. )

I am confused by your mentioning of the "equal-length distance"... maybe you confuse it with the splitting of lines into smaller fragments?

from dinglehopper.