Valid YAQL in With Items Input Fails about orquesta HOT 8 OPEN

After some discussion and investigation with @jamesdreid I was able to identify the items() behaviour has changed between yaql v1.1.3 and yaql v2.0.0

YAQL is currently pinned as yaql>=1.1.0
According to pypi, yaql was updated to 2.0.0 on 12 Jul 2021, which is just after st2 v3.5 and would have been included in the st2 v3.6

from orquesta.

Sorry, the issue was first identified in our workflows after an upgrade from 3.4.1 to 3.6. I do not know if it was there in 3.5 as we jumped straight to 3.6 on the system where these failing workflows were running.

from orquesta.

@nzlosh Any next steps? Do we need to pin yaql in the requirements to fix it?

from orquesta.

I've looked into this further, this is the commit for yaql v2.0.0

The abstract base classes previously defined in 'collections' were moved
to 'collections.abc' in 3.3. The aliases will be removed in 3.10.

-    return isinstance(obj, collections.Iterator)
+    return isinstance(obj, collections.abc.Iterator)

If we pin to 1.1.3, builds will fail against py3.10 (which will be required to support ubuntu 22.04 and drop ubuntu 18.04). I think we're at the point where we'll need to fix the orquesta code. Which will be related to how it's detecting/handling collections.

from orquesta.

Looks like this PR may have been related to the attempted fix for a similar issue that may need some updates. #191

from orquesta.

Plus it appears that Orquesta may be importing "collections" as well and may need to be updated in a manner similar to the YAQL lib.
https://github.com/userlocalhost/orquesta/blob/4bc557574042a970a2101d15fecf626d0f93a4e7/orquesta/expressions/functions/workflow.py#L15

from orquesta.

Agreed. Let's fix orquesta, not the pin.

from orquesta.

It seems #191 is a partial fix. Testing a few expressions shows the following:

YAQL list ✔️

    action: core.local
    with: <% [1,2] %>
    input:
      cmd: echo <% item() %>

YAQL dict keys ❌

    action: core.local
    with: <% { "a" => 1, "b" => 2 }.keys() %>
    input:
      cmd: echo <% item() %>
'TypeError: The value of "<% { "a" => 1, "b" => 2 }.keys() %>" is not type of list.'

YAQL dict keys using X in format ❌

    action: core.local
    with: keyname in <% { "a" => 1, "b" => 2 }.keys() %>
    input:
      cmd: echo <% item() %>
'TypeError: The value of "<% { "a" => 1, "b" => 2 }.keys() %>" is not type of list.'

YAQL cast dict keys to list ✔️

    action: core.local
    with: <% { "a" => 1, "b" => 2 }.keys().toList() %>
    input:
      cmd: echo <% item() %>

Jinja list ❌

    action: core.local
    with: {{ [1,2] }}
    input:
      cmd: echo <% item() %>

"Failed to load workflow definition because while constructing a mapping
  in "<unicode string>", line 7, column 12
found unacceptable key (unhashable type: 'list')
  in "<unicode string>", line 7, column 14."

Jinja list using X in format ✔️

    action: core.local
    with: keyname in {{ [1,2] }}
    input:
      cmd: echo <% item() %>

stdout: '{keyname: 1}'

Jinja dict keys ❌

    action: core.local
    with: {{ {"a": 1, "b": 2}.keys() }}
    input:
      cmd: echo <% item() %>

"Failed to load workflow definition because while parsing a flow mapping
  in "<unicode string>", line 7, column 12
did not find expected ',' or '}'
  in "<unicode string>", line 7, column 30."

Jinja dict keys using X in format ❌

    action: core.local
    with: keyname in {{ {"a": 1, "b": 2}.keys() }}
    input:
      cmd: echo <% item() %>

"Failed to load workflow definition because mapping values are not allowed in this context
  in "<unicode string>", line 7, column 29."

Jina cast dict keys to list using X in format ❌

    action: core.local
    with: keyname in {{ {"a": 1, "b": 2}.keys() | list }}
    input:
      cmd: echo <% item() %>

"Failed to load workflow definition because mapping values are not allowed in this context
  in "<unicode string>", line 7, column 29."

It's not immediately obvious to me where the root cause of this issue is.

from orquesta.