csml-by-clevy / csml-engine Goto Github PK

Describe the bug
A clear and concise description of what the bug is.

A bot with a function containing a condition check can not be built and returns an error!

Bot validation failed with 2 errors:
  In Default.start:4:11 - use valid actions
  In Default:0:0 - LINTER: Flow 'Default' need to have a 'start' step

To Reproduce
Steps to reproduce the behavior (for example, paste a flow that demonstrates the issue):

fn myfunc():
  if (1 < 2) return "something"
  return "something else"

start:
  say myfunc()
  goto end

Expected behavior
A clear and concise description of what you expected to happen.

Should build without error!

If I need to validate numbers on user input, things can get unnecessarily complicated.

For example:
I'm looking for ints between 8 and 25, user inputs: "14.3"

if (!event.is_number()) goto invalid // if event is "toto" next line fails
if (event.to_int() != event) goto invalid // number is not an int
if (event < 8 || event > 25) goto invalid // number is not within boundaries

A better option would be:

if (!event.is_int()) goto invalid
if (event < 8 || event > 25) goto invalid

A similar option to check if event.is_float() would be great as well.

Is your feature request related to a problem? Please describe.
When I want to skip a loop without exiting, there is no concept for a continue keyword. This would make loops more readable when you need to exit early.

Describe the solution you'd like

foreach (item) in [1, 2, 3, 4] {
  if (item % 2) continue
  say "{{item}} is odd"
}

Describe alternatives you've considered
This adds one extra line and level of indentation and is not quite as readable:

foreach (item) in [1, 2, 3, 4] {
  if (!(item % 2)) {
    say "{{item}} is odd"
  }
}

Is your feature request related to a problem? Please describe.
Would be great to be able to have variables in goto. Currently it's just stupidly complex to do something like:

goto random(X, Y)

and when matching various buttons, it would be great to not have to just iterate across all buttons like so:

if event.match(btn1) goto X
if event.match(btn2) goto Y
...

but just go for something like goto btn.some_param

Describe the solution you'd like
there has to be some extrapolation of variable, as otherwise goto something will try to find a step called something. Let's try to find a good way to do it :-)

Describe the bug
break doesn't work in function scope

To Reproduce
fn test():
do vec = [1,2,3]
foreach (elem) in vec {
break
}

Expected behavior
break command is expected to work

Describe the solution you'd like
A connector to Amazon DynamoDB similar to the default MongoDB connector would be great to run in an AWS-friendly environment with a serverless database. This helps reduce hosting costs.

Describe alternatives you've considered

• using mongodb may be expensive for smaller bots
• using the http microservice connector requires hosting an additional component and is slower as additional boilerplate/networking is required

Currently it's impossible to use double quotes and other escaped chars inside a string in CSML. It would be great if we could actually do something like:

say "Hi \"there\" \nI want a newline"

Which should print:

Hi "there"
I want a newline

Describe the bug
When I try to say Url() with a special utf-8 character in it, it fails and doesn't get displayed at all.

To Reproduce

start:
  say "Bienvenue sur le test **d'autodiagnostic COVID-19**, réalisé en partenariat avec **l'Institut Pasteur**, **l'AP-HP** et le **Ministère de la santé**."
  say Typing(2000)
	say "⚠️ Attention, ce site d’information **n’est pas un dispositif médical**, il ne délivre pas d’avis médical."
	say Typing(2000)
	say "**Avant de commencer, veuillez lire le préambule à l'utilisation du service :**"
	say Url("https://bit.ly/préambule")

Expected behavior
I expect the url function to urlencode the link in the message. However, it should display the original text in the message.

To demonstrate, I expect something like this:

<a href="https://bit.ly/pr%C3%A9ambule">https://bit.ly/préambule</a>

Additional context
I discovered the bug in the test channel in CSML Studio, but it also fails on the covidbot.fr website.

Workaround
You can urlencode the url yourself like this, but it doesn't display the urldecoded version in the message itself.

say Url("https://bit.ly/pr%C3%A9ambule")

Describe the bug
Code that normally display warnings don't do that in the function scope

To Reproduce
fn test():
do value

Expected behavior
the engine is expected to return some type of warning or error for this type of code

Is your feature request related to a problem? Please describe.
If more than one flow have the same command, it will always match the same flow. It would be ideal if all matching flows could randomly match that command.

Describe the solution you'd like
https://github.com/CSML-by-Clevy/csml-engine/blob/master/csml_engine/src/utils.rs#L229-L236
This section here matches the first flow that it found with the given command. If several flows have the same command, the same one will always be matched with the command. Ideally, it would be best to get one at random.

Describe alternatives you've considered
The only possible way to achieve this currently is by having one command go to one flow, and from that flow randomly go to one of the wanted targets. Not very convenient.

Describe the bug
When using the word match as a variable name, the CSML raise an error upon building.
Here is the error shown in the console:

22:45:52: Bot validation failed with 1 error:
  In Default.start:7:5 - at line 7,
  do match = "OM/PSG"
    ^
expect valid argument after action keywords. Example: say value

To Reproduce
do match = "OM/PSG"

Expected behavior
Well, either match is a reserved name, or it's not, it just needs to be clear.

Is your feature request related to a problem? Please describe.
Impossible to do a basic auth with HTTP requests now as it requires to base64 encode the Authorization payload.

Describe the solution you'd like

Base64("xxxx").encode()
Base64("xxxx").decode()

Describe alternatives you've considered
Custom Fn() but that sucks :-)

the debug messages should be improved, standardized and generally more helpful. For instance:

conversation_end => false
http post callback_url - 0.90
conversation_end => true
http post callback_url - 0.25
Total Time interpret step start - 0.544

Does not provide any help, nor guidance on what the numbers mean, where we are at, etc.

A better solution could be:

init: received event - {timestamp}
init: conversation exists with ID {CONV_ID} - {timestamp}
{CONV_ID}: entering flow X - {timestamp}
{CONV_ID}: entering step X - {timestamp}
{CONV_ID}: calling callback_url X - {timestamp}
{CONV_ID}: leaving flow X - {timestamp}
{CONV_ID}: conversation end - {timestamp}

etc.

This way, we can actually find the conversation traces with DB entries.
Actual format of messages to be discussed in the comments :-)

Describe the bug
https://github.com/CSML-by-Clevy/csml-engine/blob/master/csml_interpreter/src/data/memories.rs#L12

This struct contains a single Memory and should not use the plural form

Describe the bug
Array.find() should return [] when the value is not found within the source array, but in reality it returns null.
Doc reference: https://docs.csml.dev/language/standard-library/array-methods#find

To Reproduce

do val = ["Batman", "Robin", "Superman", "Batman"]
do a = val.find("Ironman")
say "{{a}}" // null

Is your feature request related to a problem? Please describe.
Navigating between flows is missing the possibility to directly reach a specific step inside the target flow, instead of always the default "start" step.

Describe the solution you'd like
Something along the lines of the following would be very useful:

goto stepname@flowname // reach the requested step in target flow

Some special cases:

goto @flowname // = goto flow flowname
goto stepname@ // = goto stepname
goto @ // = goto start

Describe the bug
event keyword can not be used as one of the params of a function

To Reproduce

fn myfunc(param):
  return "hi {{param}}"

start:
  say "Hello"

  do x = myfunc(event)
  say "result is: {{x}}"

  goto end

This prevents build with error: In Default.start:7:22 - reserved keyword can't be used as identifier

Expected behavior
It should build with no error!

Additional context
Assigning event to another variable first works

  do myparam = event
  do x = myfunc(myparam)

Is your feature request related to a problem? Please describe.
Currently, outside of external Fn() calls, it's impossible to create reusable parts of code to perform repeatable actions. It would be extremely useful to be able to create local functions, in pure CSML, to perform reusable operations. In general, the ability to write functions is sorely missing from the language.

Describe the solution you'd like
Some way of declaring a function:

fn MyAdd(param1, param2):
    return param1 + param2

Then later in the same flow:

say "your result is {{MyAdd(currentPoints + newPoints)}} total points."

Describe alternatives you've considered
The only current alternative is using external functions, which are slower (because HTTP latency), and require more boilerplate (because it requires another language runtime).

Describe the bug
When we call a function with by adding an argument name, the chatbot silently crashes.

To Reproduce

start:
  say myFn(arg="foo")

fn myFn(arg):
  return arg

Expected behavior
This syntaxe is not unusual in programming, I would expect the CSML to let me know that it is not possible to write it this way instead of crashing silently

Is your feature request related to a problem? Please describe.
In order to prevent repetition of code and to have a cleaner CSML it will be helpful to have an import system for the functions

Describe the solution you'd like
Some way of importing a function:

import {step_name as new_step_name, ..}

If from flow is specified the scope of the search will only the flow otherwise the entire bot will be the scope

import step_name from flow

Describe alternatives you've considered
without the import system the only solution will be to have the same function copied in all the flows that need it

Is your feature request related to a problem? Please describe.
event.match() is great if you know what all arguments you want to match, but it's hard to use programmatically if your happen to have an array of possible values (which can not be flattened into individual arguments).

Let's say we have an array of buttons:

do btns = [ Button("A"), Button("B") ]
say Question("click a button", buttons=btns)
hold

The current only way to know which button was clicked is to iterate over the array and check each individual item, which is cumbersome:

foreach (btn) in btns {
  if (event.match(btn)) {
    say "matching {{btn.title}}"
  }
}

Describe the solution you'd like
Either:

• a way to "flatten" the array into individual arguments, like the ... operator in JS: event.match(...btns)
• or a separate method to match an array: event.match_array(btns)

Describe the bug
Saying "Hi" does not trigger a flow with a command of "hi"

To Reproduce
Steps to reproduce the behavior (for example, paste a flow that demonstrates the issue):

Set any command on a flow and try saying that command with other case

Expected behavior
A clear and concise description of what you expected to happen.

Saying "Hi", "HI", "hI" should have the same effect as "hi" if the command is "hi"

Describe the bug
Linter shows error in the wrong flow

To Reproduce
Steps to reproduce the behavior (for example, paste a flow that demonstrates the issue):

Validate the following bot data:

{
  "id": "fb1324fd-7618-48f1-a8bb-4d5eed0a583d",
  "organization_id": "0b50ddf9-052b-4dd6-9901-459caa15ed33",
  "name": "mybot",
  "description": "somedescription",
  "default_flow": "442f0e44-a330-435d-a07a-e87a4b47a14d",
  "created_at": "2020-08-23T21:44:06.323Z",
  "updated_at": "2020-08-28T18:08:20.912Z",
  "flows": [
    {
      "id": "442f0e44-a330-435d-a07a-e87a4b47a14d",
      "bot_id": "fb1324fd-7618-48f1-a8bb-4d5eed0a583d",
      "organization_id": "0b50ddf9-052b-4dd6-9901-459caa15ed33",
      "name": "Default",
      "description": "Default custom flow",
      "commands": [
        "/default"
      ],
      "content": "tutu:\n  say \"Hello\"\n  goto end",
      "created_at": "2020-08-23T21:44:06.410Z",
      "updated_at": "2020-08-23T21:44:06.410Z"
    },
    {
      "id": "d111452e-aec4-4638-a8b2-afbb3aae048f",
      "bot_id": "fb1324fd-7618-48f1-a8bb-4d5eed0a583d",
      "organization_id": "0b50ddf9-052b-4dd6-9901-459caa15ed33",
      "name": "otherflow",
      "commands": [
        "/otherflow"
      ],
      "content": "start:\n  say \"hello\"\n  goto end",
      "created_at": "2020-08-24T15:42:58.384Z",
      "updated_at": "2020-08-24T15:42:58.384Z"
    }
  ]
}

As you can see, "Default" does not have a "start" step (only a "tutu" step), and start is mandatory in CSML. So rightfully so, I get a validation error:

The errors returned are as follows

{
  "valid": false,
  "errors": [
    {
      "flow": "otherflow",
      "step": "start",
      "line": 0,
      "column": 0,
      "message": "LINTER: Flow 'Default' need to have a 'start' step"
    }
  ]
}

However errors[0].flow says that we should look at the flow otherflow, but the error is in Default flow (the message is right).

Expected behavior

errors[0].flow should be "Default", not "otherflow"

Is your feature request related to a problem? Please describe.
getting the current time or converting timestamps to various formats requires using an external function, which is cumbersome, requires external code in a different programming language, requires an external Fn runtime, and adds delays. It would be useful to add some way to get timestamps, convert those in various formats, get timediffs, and optionally change timezone...

ESPECIALLY useful for calendar-types of chatbot (book an event on date X)

Describe the solution you'd like
Some inspiration:

• JS: built-in Date, date-fn, momentjs...
• rust: chrono, chrono-tz
  etc.

Describe alternatives you've considered
Fn("getTime", ...)

Describe the bug
Debug(1, 2, 3) => sometimes { args: [ '1', '2', '3' ] }, sometimes { args: [ '3', '1', '2' ] }...

To Reproduce
say Debug(1, 2, 3) multiple times

Expected behavior
{ args: [ '1', '2', '3' ] } everytime

Is your feature request related to a problem? Please describe.
Linked to #166 , it would be nice to be able to automatically perform HTTP basic auth with a specific header method, instead of having to do HTTP().set({Authorization: "Basic base64(username:password)"}).get().send()

Describe the solution you'd like
HTTP().auth(username, password).get().send()

Describe the bug
Doing a HTTP().post() without a body crashes:

usage: post(body: object) => http object at line 2, column 83 in step [start] from flow [Default]
[send] is not a method of Null at line 2, column 59 in step [start] from flow [Default]

To Reproduce
This works:

start:
  do x = HTTP("https://reqbin.com/echo/post/json").post({}).send()
  say "response: {{x}}"
  goto end

This doesn't:

start:
  do x = HTTP("https://reqbin.com/echo/post/json").post().send()
  say "response: {{x}}"
  goto end

Expected behavior
An empty body is valid and should be accepted

Describe the bug
As expected, naming a step "flow" is not allowed, but the error message is not very clear.

flow:
^
invalid argument. One of the action keywords [say, do, if, ...] is missing

To Reproduce

start:
  goto end

flow:
  say "hi"

Expected behavior
Something like "Invalid step name" would be better

Describe the bug
Fail to execute correct CSML steps after wrong action is use in a CSML function in another bot

To Reproduce
bot: A
fn toto():
say "plop"
return 1

start:
if (a) say "Hi"
goto end

bot: B
start:
if (a) say "Hi"
goto end

Expected behavior
I expect bot B to work even if bot A fails

One of the most common data validation tasks is checking if a user input is a valid email. The current solution with string.contains(@) is clearly not great, and it's a bit cumbersome to ask beginners to use regex.

Let's add a built-in function to perform this task: var.is_email() -> Boolean

A good enough regex for this would be: /^.+@.+\..+$/ (checks strings in the form of [email protected] where xxx is at least 1 character). It's not the definitive best regex but it is good enough, and expert users who are looking for a specific issue in emails should use something more expert, such as https://emailregex.com/.

Would be great to have users install the CSML engine in their own app with a crate :-)

Is your feature request related to a problem? Please describe.
It's currently impossible to use hold inside a foreach. However it's not always possible or easy to go around this limitation easily, so a way to use hold inside loops would be greatly appreciated.

Describe the solution you'd like

start:
  remember arr = ["cheese", "strawberries"]
  remember answers = []
  foreach (food) in arr {
    say "do you like {{food}}"
    hold
    do answers[food] = event
  }

Describe alternatives you've considered

• instead of using foreach, we can simply increment over array indexes and hold there, but it should really be handled by the engine itself:

start:
  remember arr = ["a", "b", "c"]
  remember answers = {}
  remember idx = 0
  goto iter

iter:
  if (idx >= arr.length()) goto finished
  say "item: {{arr[idx]}}"
  hold
  do answers[arr[idx]] = event
  do idx = idx + 1
  goto iter

finished:
  say "finished: {{answers}}"

• it's also possible to use goto and move to a different step for every iteration, but it's very cumbersome and not developer-friendly, and it does not solve all problems

Is your feature request related to a problem? Please describe.
In async mode (with a callback_url) CSML server could very easily be triggered by a SNS notification event.

Describe the solution you'd like

• add a way to register CSML server as a SNS http target
• respond to events sent by SNS

Additional context
CSML Studio runs on SNS and we would like to use the native implementation of CSML server instead of the node binding 😉

Is your feature request related to a problem? Please describe.
When using the bot to connect to some thrid-party services, it would be very useful to be able to initialize the bot with environment variables that can be used in any flow, any step.
The main issue with the current way of doing things is that you must remember the variable on every entry point of the chatbot to make sure that it is there when you need to use it, AND it also requires setting a variable in the current user's memory.

Describe the solution you'd like
something like say "{{_env.myvar}}" would be neat. One way to initialize would be to send the env variables in the POST call of each request or in the bot's definition as in #156 .
I'm fine with having only strings in env vars (as in the shell's env vars which can not contain objects and ints etc.)

Describe alternatives you've considered
Starting the flows with remember myvar = "xxx" works but requires setting the value in clear text in the bot + setting the value as a user conversation attribute, which can be an issue when handling data export requests.

If we need to force-cast a value to an int, we are expected to use the method val.to_int(). However, if val = "string", this operation throws an exception.

do x = "string".to_int()

Expected behavior:

x = 0 or x = Null

Describe the bug
packages are directories csml_manager, csml_interpreter but the names are inconsistent: csmlrustmanager and csmlinterpreter.

Expected behavior
use standard notation with underscore: csml_interpreter, csml_manager

Is your feature request related to a problem? Please describe.
It's currently a bit cumbersome to apply something to an array and retrieve a result at the end. For instance, if I want to do the simple javascript code: const x = [1, 2].map(i => i * 2), the same in CSML is:

do arr = []
foreach (i) in [1, 2] {
  arr.push(i * 2)
}

which is much longer and not as elegant.

In general, CSML lacks the ability to handle first-class functions.

Describe the solution you'd like
JS or rust-like notation is fine:

do res = [1, 2].map(| a | a * 2) // rust
do res = [1, 2].map(a => a * 2) // JS

and with a CSML native functions:

import myfunc
do res = [1, 2].map(myfunc)

Describe the bug
When checking if event.is_int() an error is returned:
[is_int] is not a method of Object at line 7, column 24 in step [start] from flow [Default]

To Reproduce
Click on a button payload and try to parse the corresponding event.

After the hold of this bot:

start:
  say Question("Check int", buttons=[Button("42")])
  hold
  say "is int? {{event.is_int()}}"
  goto start

click on the button or try sending this event

{ "content_type": "payload", "content": { "payload": "42" } }

Expected behavior
It should return a boolean

Describe the bug
A clear and concise description of what the bug is.

say "hello {{ myfunc(1, 2 }}" (note the missing )) should error during build OR at least at runtime, but nothing happens.

To Reproduce

fn myfunc():
  return "hi there"

start:
  say "Hello"

  do x = myfunc()
  say "result is: {{ myfunc( }}" // add or remove parentheses here!

  goto end

Expected behavior
Either a syntax error at build time OR some error/warning at runtime OR a Null value

Describe the solution you'd like
For for users that are on hold if the developer update the csml flow but don't change the step code in witch the user is on hold it will be better if they continue the conversation instead of resetting the flow and if the developer update the step then de user only need to redo the step and not the flow.

The signature of the start_conversation method currently goes something like this (in pseudo-code):

fn start_conversation( CsmlEvent, { ...CsmlEvent, bot: CsmlBot } )

Granted, the first parameter is a serde_json struct and the second is a dedicated CsmlData, but they both have the same contents "duplicated", which is a waste and makes it harder to integrate. It should probably be something along the lines of:

fn start_conversation( CsmlEvent, CsmlBot )

Describe the bug
When trying to build this code below, the following misleading error about step syntaxe is returned :
In myFlow.start:3:12 - Flow's must have steps declared as follow 'step_name:' previous to any action

To Reproduce
Build this flow

start:
  do foo "bar"
  goto end

Expected behavior
The error should underline the fact that the syntax error is on line 2

Additional context
N/A

To print any value, you currently need to say "{{value}}", which can not be differentiated from a legit say instruction and shows a non-pretty-printed value.

A Debug() component would be able to take any value as a parameter, format it properly and wrap it unchanged in a dedicated component that could be left unimplemented by the consuming channel, allowing the developers to leave debug calls that could only appear on the channel when in "debug" mode.

Is your feature request related to a problem? Please describe.
Currently, the only way to use CSML engine is by embedding it into another rust project or using one of the available bindings to incorporate it into a nodejs / docker-based project.

Adding an HTTP interface would make usage of CSML engine so much more developer friendly!

Describe the solution you'd like
Add HTTP endpoints to validate and run CSML chatbots

Describe alternatives you've considered
Using nodejs bindings or docker. Not as convenient as something native!

CSML request metadata is ignored, and is built from event.content instead. So for instance:

{ 
  ...,
  "metadata": { "example": "value" },
}

when used as say "{{_metadata.example}}" will not print "value" but instead "Null"

The current behavior of the match keyword works for single options but makes it hard to find what the input matches if there are several options. For instance:

say Question(
  "What do you like best?", 
  buttons=[
    Button("Pizza") as pizza,
    Button("Salad") as salad,
    Button("Cheese") as cheese
  ]
)
hold
if (event match pizza) goto ValidChoice
else if (event match salad) goto ValidChoice
else if (event match pizza) goto ValidChoice
else goto NotValid

A better option would be to be able to find if the user selected an option, like for instance:

if (event.match_any(pizza, salad, cheese)) goto ValidChoice
else goto NotValid

We could further improve this behavior by having it return the actual item that matches:

do found = event.match_any(pizza, salad, cheese)
if (found) say "Found: {{found.title}}"
else say "Not found"

Describe the bug
When I check if a variable has been initialized or not, I receive a runtime error message, saying the variable does not exist yet. It's expected that it does not exist, so it should not send an error. At most a warning could be useful, but probably nothing at all is required as this will be triggered for every user the first time they arrive there.

To Reproduce
Steps to reproduce the behavior (for example, paste a flow that demonstrates the issue):

start:
  if (!has_been_there) {
    say "it's your first time here!"
    remember has_been_there = true
  }
  else {
    say "you've been there already"
  }
  goto end

First time:

Second time:

It might be helpful in some instances to warn that you are using an uninitialized variable, but in this case it just does not make sense.

Expected behavior
A clear and concise description of what you expected to happen.

I would expect nothing to happen at all here: the variable does not exist, so !has_been_there should be true, and continue. Consider this variable to be Null (same as when a Fn() crashes, for example)?

Is your feature request related to a problem? Please describe.
It's currently impossible to natively iterate over the letters of a word.

Describe the solution you'd like

foreach (letter) in string {
  say "letter: {{letter}}"
}

Describe alternatives you've considered
"string".split("") does generate an iterable array but it contains a first and last empty items (["", "s", "t", "r", "i", "n", "g", ""]) which makes it less easy to manipulate.

Implicit recursive is a good idea on paper but is often misunderstood. The basic idea was to be able to have automatic temporary placeholders in a flow A while the bot would run through a second flow B, and if that flow B were to hit a goto end without holding anywhere, the bot would return and continue from the placeholder in A.

The main idea behind it was: if the user is in the middle of a long/complex flow, types "help", the bot might return with i.e a single block of text explaining its purpose without requiring any input. In that scenario, it makes sense to come back to where the user was after traversing the "help" flow and go on with the initial flow from there without going back to the beginning.

However, this sort of magical behavior is confusing developers who are simply trying to end a conversation in a flow that just happens to follow from another flow, and does not have any hold in it. We could make a better job at explaining it but in the end we think it is just a bad design decision, so we are removing this feature that is barely used. The same effect will be doable in a future version with other mechanisms (importable modules, variable navigation, etc.).

Describe the bug
As of now, as far as the documentation goes, the only way to check if a key exists within an object is to write if(myObject.myKey), this throws an error because myKey does not exist.

To Reproduce

do myObj = {}
if (myObj.myKey) say "This key exists"
else say "it does not :("

Expected behavior
It'd be nice to either have the csml return null or undefined or anything else along these lines, or to have some sort of method that checks if the key exists.