Ordinary html file not recognized as utf-8 encoded, stopping gray-matter to parse YAML about metalsmith HOT 3 CLOSED

Hey,

Thanks for your response.

After a little deep dive, I figured out a way. I found another library utf-8-validate to validate the source files. This library works well with all the source files I had including the one I generated from Redoc.

So, basically I created a plugin called frontmatterParser which will use this utf-8-validate library instead of is-utf8 library and to use my plugin I set the property frontmatter to false. This solved my problem.

from metalsmith.

Hi there,

Nothing does an "automatic utf-8 conversion". Having a <meta charset="utf-8"> tag does not convert the document to utf-8, it only informs the browser that the document should be interpreted as utf-8.

Metalsmith does a utf-8 check to know if the file is a text file or a binary file because binary files like images cannot contain front-matter and they need to be skipped. There are a lot of character encodings but UTF-8 is the most widely used and safe to detect.

So in this case either/or:

• Redoc generates your metalsmith source files in a non-utf-8 encoding. You can check this by opening the document with an editor like Notepad++ which has charset re-encoding support. If I'm not mistaken (but I should test) Notepad on some Windows versions also saves text files as non-UTF8 by default (so if you opened a file with Notepad & saved it in another encoding without knowing it, the build will fail.
• Redoc generates files with NULL-terminated strings for which there is a known bug in the is-utf8 package: wayfind/is-utf8#6

from metalsmith.

Closing as stale, feel free to comment if it needs to be reopened

from metalsmith.