[GENERAL ISSUE] Shouldn't `unresolved` be not `FAIL` but `ERROR` in unit testing? about cloudformation-guard HOT 3 CLOSED

#355 (comment)

With the current version a single property value check will complete the exists and value check in one operation.

So, is the behavor of AWS Config wrong?

from cloudformation-guard.

Hi @yukihiko-shinoda no, I don't believe this should be an error. This is consistent throughout the guard evaluation engine when using either test or validate command.

I am going to go ahead and close out this issue since this seems to be proper behaviour imo.

Feel free to reopen if need be.

from cloudformation-guard.

I think this was AWS Config's bug and it was fixed now.

I understood that it's proper to Gurad evaluation engine marks unresolved as FAIL.
However, AWS Config custom policy rule behaved as SKIP at least in Tokyo region until 2023-07-28.

I already asked AWS support about this issue.
They answered that, at this time, if you try to evaluate a property that is not included in the setting item with the Config custom policy rule, the behavior will not be included in the evaluation results that can be obtained with GetComplianceDetailsByConfigRule.

On 2023-07-28, AWS Config custom policy rule suddenly started to detect the resource which had skiped as noncompliant with following message:

Attempting to retrieve array index or key from map at path = /****/**** , Type was not an array/object map, Remaining Query = Level

from cloudformation-guard.