Encoding Problem with French caracters in headers about panflute HOT 13 CLOSED

ok I solved the problem : If the filter is not executable, the shebang is ignored and the default python env is used:
jgm/pandoc#3174

A simple chmod a+x /tmp/headers.py fixed issue.

Indeed the filters works fine with a python3 environment...

You should problably write somewhere in the documentation that Python2 is not well supported.

Thanks for your time and for this software !

from panflute.

Hi Daamien,

Do you have access to py3? If so, could you try running it there to see if the issue persists?

The py2 version of Panflute should work, but that version is mostly built automatically with 3to2, so I'm not that surprised to see unicode issues. In this specific case, the bug might be related with json.load missing an encoding option, but as per the py2 manual, json.load should have defaulted to utf-8 encoding and not ascii encoding. Since I don't have py2 installed on my laptop, I'm not sure if I can fix the issue (unless it also occurs on py3), but if you or another user submit a PR I can merge it for sure.

Best,
Sergio

from panflute.

ok I'll give it a try....

If I install a python3 on machine with python2, how can I force pandoc to use python3 to interpret the filter ?

from panflute.

Yes, there are several ways for this depending on your system:

On *nix, you could use shebang lines: https://docs.python.org/3/using/windows.html#shebang-lines

On windows, I think there is a new utility called "py" that comes with the latest version of py2 or py3, which also allow shebangs.

from panflute.

Sorry to bother with that but pandoc seems to use python2 only....

The shebang is there

cat /tmp/headers.py 
#!/usr/bin/python3

panflute is installed with pip3

# pip3 install panflute
Requirement already satisfied (use --upgrade to upgrade): panflute in /usr/local/lib/python3.4/dist-packages
Cleaning up...

But pandoc still fails like this

pandoc tests/french_title.md --filter /tmp/headers.py -o test.pdf
Traceback (most recent call last):
  File "/tmp/headers.py", line 3, in <module>
    from panflute import run_filter, Header
ImportError: No module named panflute
pandoc: Error running filter /tmp/headers.py
Filter returned error status 1

In the python3 console, i can load panflute :

# python3
Python 3.4.2 (default, Oct  8 2014, 10:45:20) 
[GCC 4.9.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import panflute
>>>

from panflute.

Great to hear that! And yes, I agree about python 2 support (which as of now, mostly depends on external commiters)

from panflute.

I faced Very similar situation(Japanese) :)

from panflute.

Hi Kazuki,

Does it work on py3?

from panflute.

no worries
no problem it worked with py3
I was also using py2

from panflute.

There are similar issues with autofilters and unicode literals in the Python source, which can be traced there:

I fixed it with open(fn, encoding="utf-8"), but I'm not sure if that's the best solution (this ignores any encoding headers in the python source)

from panflute.

I fixed it with open(fn, encoding="utf-8"), but I'm not sure if that's the best solution (this ignores any encoding headers in the python source)

@damiendr I wouldn't mind adding it, but one question: is this a py2 or py3 issue?

from panflute.

This is with Python 3.

from panflute.

Making the python filters executable solved my problem too, that should be added to the installation documentation.

from panflute.