tsroten / dragonmasher Goto Github PK

nice to meet you @tsroten, my name is Tony Narlock and have come to find and admire your CJK libraries. I have a similar project at https://cihai.git-pull.com. It's still in early phases

Goal is to be a successor to cjklib

Some notes on the approach I'm taking / state of dealing with data:

• backing data with sqlalchemy (like CJKLib did)
• datasets pulled in are being distributed as data packages (for the sake of conservancy and the time it takes to normalize them, may as well make them available agnostically)
• backed by downloading UNIHAN (using https://unihan-etl.git-pull.com)
  some notes on that: i found that UNIHAN actually bakes in quite a lot of structured information into its fields. but i also believe it makes the ideal "spine" to pull in the rest of the CJK data sources against.
• re: UNIHAN
  • it's also not trivial to use rdbms as a backend, it's not simple as dumping data in, there's some parts of UNIHAN such as variants that need to be read into to calculate joins
  • UNIHAN is inherently unique and not a flat, tabular dataset at all. so cihai doesn't handle it like it would a flat source like CEDict. A lot of effort is going into building a front-end to store/query against it.
• for other data sources, considering baking in https://github.com/frictionlessdata/datapackage-py to the system.
• risk: some other data sets which come up may not lend themselves to RDBMS. So I plan to document trial/error with it so no one exerts their effort twice
• if my sqlalchemy / unihan expedition falls flat, i may end up back here. there's a need for a unified front for cjk in python
• bikeshed: using MIT license (was formerly BSD)
• CFDict has gone missing from the internet