grammatek / regina_normalizer Goto Github PK

Current approach to separate characters in tokens includes:

• separate upper case ('FTSE' -> 'F T S E'; 'AIDS' -> 'A I D S')
• separate predefined abbreviations ('ehf' -> E H F)
• separate all chars in a URL, except for a handful of listed URLs (like 'ruv.is'), so 'fotbolti.is' becomes 'f o t b o l t i punktur is'

Problems:

• uppercase abbreviations or acronyms are not always spelled out: 'FTSE -> 'futsi' 'AIDS' -> 'eids', i.e. need to distinguish between true acronyms (like AIDS) and initialisms, and then know of exceptions like 'FTSE'
• lower case tokens can be abbreviations (not listed) that need to be spelled out: 'þth'
• URLs are spoken as words if they are pronouncable as such, like 'fotbolti'