[multiline] wordwrap for non-single-width characters (CJK) about visidata HOT 4 CLOSED

Can you provide sample data like that used in the screenshot?

from visidata.

yes i've updated issue

from visidata.

I looked into this a bit, for Python 3.10.12. The trouble seems to be the use of textwrap.TextWrapper._wrap_chunks(). The wrapping calculations there use len() for strings, where visidata would use their display width dispwidth(s). As a quick check, I seem to get proper wrapping when I use a modified version of _wrap_chunks() for sheets.py.

def dispwidth_wrap_chunks(self, chunks):
    len = dispwidth
    # the rest of the function is unmodified from the original _wrap_chunks()
    ...
textwrap.TextWrapper._wrap_chunks = dispwidth_wrap_chunks

So one solution would be to subclass textwrap.TextWrapper to override _wrap_chunks() with a version that uses dispwidth() instead of len(). Then replace textwrap.wrap() with wtw = WideTextWrapper(); wtw.wrap(). Overriding like that is a solution that textwrap puts forward:

If you want to completely replace the main wrapping algorithm,
you'll probably have to override _wrap_chunks().

from visidata.

Several other projects subclass TextWrapper to override _wrap_chunks(), in order to handle variable width characters:

• sphinx-doc/sphinx
• paaguti/python-tabulate
• facebook/sapling
• etc

Here is the Python PR (open since 2021) that could fix it: python/cpython#28136

It's not perfect, so it's not been merged. I suggest we throw our weight behind that, and/or help it become more perfect. In the meantime, if someone wants to take that code into VisiData immediately, in visidata/_textwrap.py, such that if/when the Python PR is merged, we can use the Python implementation for newer versions and then ultimately delete our interim implementation, then we can merge that to close this issue and do some field testing for the Python PR and prove its mettle.

from visidata.