opendata / ssn-redaction Goto Github PK

Our present regex ((?!666|000|9\d{2})\d{3}(-?)(?!00)\d{2}(-?)(?!0{4})\d{4}) allows hyphens optionally. But this allows just one hyphen, which probably isn't realistic. SSNs are likely to have zero hyphens or two hyphens, but not one. In testing, this made a false match, in a footer at the bottom of a page (reading 15351125-781074). I suggest that, if feasible, we allow either zero hyphens or two hyphens, but not one.

Review at least a few dozen SSN-bearing PDFs, testing the candidate regular expression against real data, to fine-tune it.

After the target text has been identified and replaced (#8), and the background image modified to overwrite the original text (#6), save the modified PDF.

People sometimes use spaces instead of hyphens.

Use the regex to identify the target text within the PDF text.

OCR software (including, notably, Tesseract) sometimes gets a little too clever, and believes that hyphens are actually em dashes, en dashes, or minus signs. (Possibly other characters too, I'm not sure.) e.g., 012—34—5678 instead of 012-34-5678. These, of course, are not found by our regex.

I suggest that we convert the character set down to ASCII (assuming that Ruby can do such a thing), so that all hyphen-like characters become, simply, hyphens.

Virtually any 10-plus-digit number is going to be matched, in part, by our regular expression. We really only want to match 9 digit numbers that are not preceded by or followed by digits. I've tried this:

(?:\D)(?!666|000|9\d{2})\d{3}(-?)(?!00)\d{2}\1(?!0{4})(?:\D)

but that also matches the preceding and following characters, returning results like A123-45-6789,. (The ?: ensures that the group isn't stored in memory by regex, but they're still being returned as part of the result.) Figure out how to require the presence of \D before and after, but not return them.

Are we concerned that OCR software frequently replaces letters with numbers? O for 0 and I for 1 are both very common transpositions. Or do we regard OCR as the user's problem? That is, is it up to them to get their OCR correct?