Report is creating errors when run with non ASCII characters about scubagear HOT 7 CLOSED

Thank you for opening the issue. This issue is cause by a similar problem that occurred in issue #12 .
However, in this instance the OPA error is being thrown because of non-English language characters being interpreted as non-UTF-8 characters.

We'll look into this and follow up.

from scubagear.

@hervecinetic was this fixed by the v0.2.0 release?

from scubagear.

Took a quick look at this while I was renaming it. I think the problem is that rego expects utf8 (per its error) but PowerShell ConvertTo-Json generates a .NET String, which is apparently a utf-16 based construct in some way. But what I don't understand is that we were getting ASCII compatible binary output for english language before, which was not what i would have expected if it was actually serializing as UTF-16. So I don't understand the full picture here yet, nor am I an expert in this topic.

Sources:

https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/convertto-json?view=powershell-7.3
https://learn.microsoft.com/en-us/dotnet/api/system.string?view=net-7.0

Crutch, you may have already come to the same conclusion but in the event you haven't looked yet hopefully this is helpful. This seems to be a candidate solution, but I don't have time to test it:

https://stackoverflow.com/questions/23256911/encode-a-string-in-utf-8

from scubagear.

One more note on this one. Orchestrator.psm1 is responsible for writing the unified string of provider data (from the various ConvertTo-Json calls) to a file using Set-Content

Set-Content by default encodes using "utf8NoBom", per documentation

So that explains why we weren't seeing utf-16 providersettingsexport.json, but not quite why rego can't parse the non-ascii characters. So likely will need to reproduce this and then analyze the json encoding more carefully. I'll leave it there, but wanted to post this update since this was nagging at me from earlier today.

from scubagear.

One more note on this one. Orchestrator.psm1 is responsible for writing the unified string of provider data (from the various ConvertTo-Json calls) to a file using Set-Content

Set-Content by default encodes using "utf8NoBom", per documentation

Took a quick look.
Set-Content by default encodes using "utf8NoBom" for PowerShell version 7.2+.
However, it encodes using "Default" (usually ANSI) for PowerShell version 5.1 Set-Content 5.1 documentation
This is likely the source of the issue.

from scubagear.

Nice find. Yes, I agree. That seems the likely cause. A fix may just be to change to UTF8 for encoding since that is what rego is expecting.

from scubagear.

Changed all Set-Content to encode using utf8 or utf8NoBom if available.

from scubagear.