Release Process for Frame

Merge source to GitHub

On your local computer:

1. Update Cargo.toml file with new version attribute
2. Update README.md with new release notes.
3. Do git checkout main.
4. Merge any feature branches and remove them.
5. Do git fetch origin --all --tags.
6. Do git merge origin/main.
7. Resolve any merge conflicts.
8. Run cargo test from the root directory.
9. If all tests pass, do git push origin main.

Steps to repro:

1. Create frame spec in Atom and save
2. Launch FMM plugin
3. set language to C#
4. Select all code and copy
5. paste to another file

Expected:

Content to paste

Actual:

Doesn't copy

To support a system "manager" concept (MOM - Machine Operating Machine) it would be useful to have parameters to the system that are independent of the interface for the state machine.

#TrafficLight[manager:`*MOM`]

This would declare a new member variable manager of type *MOM that would be added to the constructor or equivalent method/function for the system.

Alternatively, create a -system- block which might be more flexible:

    -system- 

    #[a b c]:# --- factory pattern. Typically would be `SysInstance NewSysInstance(a b c)`

It is often desirable to handle an incoming event in a different state. This proposal introduces the "transition event forward" notation that enables forwarding events to the new current state after a transition.

$S1
   |tick| ->  =>  $S2$ ^

$S2
   |tick| handleForwardedEvent() ^

Need to handle

1. Transition event forward for '>' w/ exit event handler
2. Change state
3. Forbid enter event parameters with an enter event transition event forward.

$S1
    |>| [x] ("exit msg") -> => $S2 ^
    |<| [msg] print(msg) ^
    

$S2
   |>| [x] dosomething(x) ^

Frame expressions seem to support method call dot-notation. However, this notation is not parsed correctly when the LHS of the dot-expression is a disambiguated variable using one of Frame's disambiguation sigils.

For example, the following method call is parsed correctly, regardless of the scope of the variable a:

a.foo()

However, the following method calls are not parsed correctly:

$[a].foo()
$.a.foo()
||[a].foo()
||.a.foo()

Note that surrounding the disambiguated variables in parentheses does not resolve the problem.

Interestingly, method calls applied to the domain-variable sigil do parse correctly:

#.a.foo()

Need to support the following:

var a = `[]`

In a string matching block, an empty pattern // yields a parse error. Based on the scanner code, it appears that this should instead match the empty string (which makes sense).

However, one can still match on the empty string in a multi-string matching pattern. For example, the pattern /|foo/ will match both the empty string and the string "foo".

I suggest making assignments statements, rather than expressions.

In languages where assignments are expressions, using an assignment in "expression position" (e.g. as an argument to a function or on the RHS of another assignment) is usually considered bad style at best and error-prone at worst. IMO it's better to just rule such uses out in the syntax of the language.

Besides enforcing saner programming style, restricting assignments to statements also significantly eases things on the implementation side. For example, in the Rust backend, we have to handle assignments carefully due to ownership. Restricting assignments to statements makes this much easier.

Currently, the Rust backend produces bad code for most non-statement-style assignment expressions. I'll file a separate issue for that, but I think the better fix would be to rule out these cases on the grammar/parser side, as described here.

Also related to: #36 -- When fixing the grammar of assignments, the LHS of an assignment statement should also be restricted to the set of things that can be assigned to.

(Side note: having the draft grammar document was really helpful for understanding what Frame is intended to support!)

Different bad outcomes from the following reuse of the identifier x:

1. Errors out improperly:

#SymbolBugs1

    -machine-

    $S1
        |Foo|
            x ? a() :: ^

        |x| ^

##

Symbol table crasher:

#SymbolBugs2

    -machine-

    $S1
        |Foo|
            t ? x ? -> $S2 ^ : -> $S3 ^  ::  :: ^

        |x| ^

##

Changing |x| to |y| fixes both issues.

Investigate if it is possible to use @ as a type symbol for the target language FrameEvent type.

var frame_event:@ = @ --- the frame_event variable is of FrameEvent type and assigned the in scope FrameEvent instance

Currently transitions have optional labels to provide more informative transition descriptions:

-> "Label" $S2

States are currently rendered in UML with their programmatic names. It would be nice to be able to modifiy that in a similar manner to transitions:

$S1 "The very first state"

Problem

Often event handlers share some logic in common. Currently, there's no great way to encapsulate this shared logic in Frame.

The best current solution is to define an additional event in the interface block, implement a handler for that event that contains the shared logic, then directly invoke this handler from the handlers with the shared functionality.

For example:

-interface-
foo
bar
shared

-machine-
|foo|
    ... stuff specific to foo ...
    shared ()
|bar|
    ... stuff specific to bar ...
    shared ()
|shared|
    ... shared logic, e.g. decide whether to transition ...

However, there are some drawbacks of this approach. The obvious drawback is that shared shows up in the public interface of the machine, when really it's a subroutine that shouldn't be invoked from the outside.

A much less obvious drawback is that since shared is just a normal event, it's handler may or may not transition, and so we have to enforce some limitations around this kind of usage to ensure that we don't end up executing handler code after a transition (which can lead to bad situations, such as a double transition).

Currently, the Rust backend treats this case conservatively, treating direct handler invocations (like the use of shared () above) essentially as terminators. So you cannot, for example, write:

|baz|
    ... stuff specific to baz ...
    shared ()
    ... more stuff specific to baz (this code can never be executed in the Rust backend) ...

We could loosen this restriction by tracking whether or not we've transitioned at runtime, but then that leads to potentially confusing behavior where code after a handler will sometimes be executed and sometimes not.

Solution? Internal/private events

I'm not sure what the best solution is here, but one idea is to provide a way to define "internal" or "private" events. This would solve the issue of polluting the public interface, and Mark has said that this would also be useful for realizing the actor model in Frame.

However, there is still the second problem described above, of tracking whether or not a handler has transitioned. I think there are four ways to resolve this:

1. Disallow internal/private events from transitioning. I think this probably rules out too many useful cases.
2. Conservatively require invocations of internal/private events to occur in terminator position, similar to the current solution in the Rust backend. I think this probably also rules out too many useful cases (e.g. having an internal event that just updates state variables but never transitions).
3. Track at runtime whether a handler has transitioned, and stop execution of the calling handler if the callee transitioned. This is a bit more work and could lead to some confusing behavior, but admits the most cases.
4. Separate internal/private events into those that may transition (i.e. they have a transition statement somewhere in them) and those that don't (no transition statements). Potentially transitioning events may be triggered only in terminator position, while non-transitioning events may be triggered anywhere. This would be a simple static approach that probably gives us most of the useful cases.

My vote is for resolution 3 or 4, leaning toward 4. But maybe there's some other solution altogether?

Draft - In Progress

Purpose

To present a high-level proposal for a wholistic software system design language and runtime environment for deploying controller logic and infrastructure to a diverse set of target environments (front-end, cloud, embedded, etc.) architected in accordance with the actor model.

Background

Classical systems engineering organizes design artifacts into two broad types - structural and behavioral. Structural artifacts describe what a system is made of while behavioral artifacts define what a system does.

Currently tools such as Terraform and others can define the structural aspects of software infrastructure but lack the ability to specify software system behavior. In contrast, the Frame language describes system behavior but has no concept of system structure or how to manipulate it.

Additionally, Frame notation is currently limited to describing the behavior of a single "controller", which may or may not be the complete behavioral logic of even one system. Often (perhaps usually) interesting software systems are best described as a system of systems that cooperate to achieve some larger purpose (such as flying an airplane). FSDL will expand Frame notation to address this additional scope in its original mission.

By including other declarative languages as first class aspects of the technology as well as expanding the capabilties of Frame notation itself, FSDL seeks to define unified and universal software system design language.

FSDL

FSDL will define standardized software architecture components, specification languages and toolchains for building systems of systems. FSDL declarative, polyglot specification language includes both Frame notation for defining system behavior in conjunction with popular declarative languages for defining system structure.

System Controller Package (SCP)

The System Controller Package (SCP) is a bundle of resources and code containing a SysOS deployment for a target environment. The contents of the bundles are environment specific, but must minimally contain the SysOS runtime for the target environment and the Main Controller (MaC).

SCP Components

1. System Operating System (SOS) runtime
2. A Main Controller (MaC).
3. The Effector
4. Machine Operating Machine (MOM)
5. System Controllers (SC) (optional).
6. Message specifications.
7. Channels, both internal and external to the system.
8. A persistence mechanism for system state (optional).
9. A distributed lock mechanism (optional).

System Operating System (SOS)

System Operating System (SOS) is the runtime environment enabling SCP components to operate in a target environment.

Main Controller

The Main Controller (MaC) is the first instantiated controller and will bootstrap the environment based on configuration data. It manages the lifecycle of all other controllers as well as establishes and/or verifies the system infrastructure.

The first responsibility of the MaC upon bootup is to determine if system the state needs to be retrieved from persistent storage. If so, it loads the system's persisted state and initializes it. Otherwise the system is booted into a default state.

The order of instantiation may be important. If it is critical that all controllers are operational before the system as a whole can perform certain operations, the MaC will need to be aware of such requirements and progressively instantiate the communication infrastructure as appropriate.

Alternatively, if it is permitted for the system to begin operations in a non-deterministic manner, then the order of instantiation and operational readiness of both the controllers and communications infrastructure can be ignored.

Effector

The Effector is an object that receives commands from the controllers and performs operations. It should not make contextual decisions and simply performs the requested operation and return data and results to the controller.

The Effector may be a composite of objects rather than a monolith, an approach which makes the runtime optimization of the system more flexible and optimizable.

Machine Operating Machine (MOM)

In most environments, controllers will not be first class objects. Therefore some management logic must exist to create, initialize and destroy the controller as well as provision it to communicate with the rest of the system.

MOM will be an environment specific runtime that handles these duties and includes the main entry point or equivalent for the environment.

System Controllers (SC)

System Controllers (SC), or just controllers, are Frame generated state machines that manage an external domain or simply serve as a mechanism for internal state or data management.

Messages

Messages should be specified using Protocol Buffers. This will enable easy support of both gRPC and REST protocols.

Channels

Channels are an abstraction of communication paths both internally and externally. Ideally they should be usable via standardized, simple Frame notation for sending and receiving messages. Channels should be definable using a standard interface API as well as with declarative configuration languages like Terraform.

Persistant Storage

SysOS will define an interface for persisting system state that can support a wide variety of standard data storage technologies (SQL, NoSQL, filesystem).

See:

        } else if self.match_token(&vec![TokenType::GTTok]) {
            return Ok(Some(TerminatorExpr::new(
                Continue,
                None,
                self.previous().line,
            )));
        }

Elsewhere it properly accepts ':>'.

1. Capitalize interface methods

Frame can return values from handlers using the following notation ^(foo).

However, the syntax of the return value seems to be restricted to literal values and variable references, and does not include arbitrary expressions.

Thus, the following statement is currently a parse error:

^(a+b)

The workaround is simple since one can just introduce a local variable and then reference that in the return statement. For example:

var r = a + b
^(r)

However, it seems like it should be easy to generalize the value of the return statement to allow arbitrary expressions.

There are no tests covering Frame's state history features. Considering the representations of states and state contexts has changed since this code was originally written, it is likely that this feature is broken.

I'm guessing this is already known as Rust support is incomplete, but I thought I'd drop an issue as it's one of the few remaining issues I'm seeing. Presently, I can compile Rust output from Frame files that don't include heirarchical state machines, but if there's hierarchy they fail on *_state() functions that include this line in interface event handler code:

StateContext stateContext = null;

Error: error: expected one of '!', '.', '::', ';', '?', '{', '}', or an operator, found 'stateContext'

Explore including some kind of mechanism for remapping types in a spec to a language specific type.

I would think a simple yaml file format which can either live in a spec, be referenced by one or exist in an enclosing yaml file that also contains the spec.

The Rust backend is designed around the assumption that assignments are statements. However, in the current version of Frame's syntax, they are actually expressions. Therefore, the Rust backend does not generate the correct code when an assignment occurs as the argument to an action/event or as the RHS of another assignment.

I think that this should rather be fixed by changing the syntax to forbid such uses, as described here: #64

However, I'm filing this issue to track the problem with the Rust backend in case the proposal in #64 is decided against.

$S1
|>| -> "" $S2 ^

Overview

Currently Frame supports "code literal" tokens using backticks:

    -actions-

    print[msg:&String] {`
        println!("{}", &format!("{}",msg));
    `}

This notation passes through the contents between the backticks.

A cleaner approach might be to use attributes that specify the target language and are only used by the indicated language generators:

    -actions-

    #[language=rust]
    print[msg:&String] {
        println!("{}", &format!("{}",msg));
    }

    #[language=python3]
    print[msg] {
        print(msg)
    }

If the language generator does not find a match, it will generate an empty action that can be overridden.

We have a scenario where we need to know the file name of the Frame spec that generated a particular state machine. This can easily be added to the MachineInfo struct available via the runtime system (in the Rust backend).

It'd be easy to store the whole source file path there, but that might encourage some hacky/non-portable uses, so not sure whether that's a good idea... Maybe it's OK to provide it as long as it's just documented accordingly?

It's possible to write a syntactically invalid Frame program that is silently parsed as something other than what the programmer intended. Instead, the program should be rejected with a parse error of some kind.

Here is one example reported by @fdlg, though I've encountered others:

-> !foo() || bar() ? -> "Label" $State : :: ^

The conceptual error here is the initial ->, which was mistakenly inserted through an editing error. To the parser, this should be recognized as an error because the text after the initial -> is not a valid transition target.

However, instead Frame v0.6.0 happily generates the following code:

if self.foo() || self.bar() {
    // Start transition
    // Label
    ...

Note that the initial -> is ignored and the ! is dropped in the generated program.

This is an especially nefarious case because the generated program is almost what the programmer actually intended, but with a crucial negation omitted.

Instead, the program should be rejected with a parse error.

I've been toying with some ideas for syntax for a standardized messaging system based on protobuf3. The first class entities so far are 1) channels 2) message types and 3) communication operators.

<@> --- configurable channel type
:@message --- Reference to a protobuf type
msg_instance @> channel_instance --- send a message on a channel

import Person.proto

--- <@> is the channel token

var global_channel:<@> = <@>(<<config data>>)

#CommExample

    -machine-

    $S1
        |e1|
            var channel:<@> = <@>(<<config data>>)
            var sendPerson:@Person = @Person(
                first_name:Mark
                last_name:Truluck
            )

            var recvPerson:@Person

            --- <@ and @> are the "I/O" tokens

            channel <@ sendPerson --- send operation
            sendPerson @> global_channel --- send operation
            channel @> recvPerson --- receive operation
            recvPerson <@ global_channel --- receive operation
            ^
##

@walkie Thoughts on this initial direction and syntax?

The runtime interface now supports registering callbacks to be notified when events are sent, when they're handled, and when transitions occur. I think the next logical step is registering callbacks to be notified when actions are invoked.

The runtime interface for the event system was designed to accommodate actions too, so this should be really easy to add whenever we decide we want/need it.

Thoughts I've had on a more generalized use of the event dispatch operator in a more general purpose way.

Currently it is only defined in indicating a strict call chain from "child" state to a "parent" state.

$A => $B

Expanding on this could include using it as an event handler statement that could "route" the current event to other states:

$A
   |e| x ? => $B : => $C :: ^

The original Harel paper discusses what I interpret to be similar ideas in section 6.2 on "overlapping states". http://www.inf.ed.ac.uk/teaching/courses/seoc/2005_2006/resources/statecharts.pdf

One general nuance (which exists with the current strict hierarchy as well) to this is to determine if an event has been "handled". Currently there is no standardized way to know. One thought I have is to add an additional boolean "handled" attribute to the metadata on the FrameEvent that is autoset under defined circumstances, most typically when it is routed to an event handler after a message selection occurs.

The token for this attribute on an event could be @| (seems elegant to me IF it doesn't clash with other syntax. The idea is the most important thing though).

So this would be some ways to use this token:

@| ?  handled() : not_handled() ::
@| = true
var handled = @| 

A typical situation would be to dispatch an event to another state and want to test if it had been handled. This could be accomplished with a special test operator ?|.

$A
   |e|
       => $B ?|  ^("b")  :: => $C ?| ^("c") :: d() ^("d")

As often we will just want to return if it has been handled, we can combine testing and returning if true using another operator ?|^ which wold combine the test and return terminator statement and be used like this:

$A
   |e|
       => $B  ?|^ => $C ?|^ d() ^

Or with return expressions:

$A
   |e|
       => $B  ?|^("b") => $C ?|^("c") d() ^("d")

Enable stdin/out as an option to support use cases for CLI as well as other tools.

7.4 - Support golang.

This should be invalid:

1 = 1

Currently, the accept() method in the NodeElement trait is implemented for all nodes as a simple double-dispatch to the corresponding visit_* method.

However, since Rust does not have inheritance, this double-dispatch is actually redundant. Any use of accept() could be replaced by a direct call to the corresponding visit_* method for that node.

Much more useful would be if each accept() method instead invoked the visit_* methods for all of the child nodes of the current node.

The default implementation of each visit_* method in AstVisitor could then be changed to invoke accept on its argument node, so that the default behavior of an empty visitor would be a simple, effectless traversal of the AST.

This would be a huge improvement to Frame's visitor interface. Currently, every visitor is responsible for traversing the AST, which is verbose, error-prone, and a maintenance problem since any change to the AST necessarily affects every visitor. Moving the traversal into the accept() methods captures the traversal of the AST once in a way that can be easily shared among all visitors. This would make it easier to create new visitors (since they can just "link in" to the traversal wherever they need to by overriding the corresponding visit_* methods), and would significantly ease the maintenance of existing visitors.

Note that this would not negatively impact the flexibility of visitors to perform a custom traversal, if needed. Each invocation of accept() would simply invoke the visit_* methods of its children, returning control to the visitor. That is, each invocation of accept() performs one level of traversal. So visitors retain ultimate control of the traversal by choosing when to invoke accept() and when to call the corresponding visit_* methods directly, in order to handle special cases.

Transition code should be generated that only send exit/enter events if exit/enter handlers exist for the current/target states. The condition where enter/exit eventhandlers for parent states must be taken into account as well.

For visualization and debugging, it could be useful to have access to the history of Frame events for a given state machine.

This would be a bit more work than it seems because currently events live on the stack in the Rust backend, so we'd need to change the generated code to use Rc everywhere for events when the runtime system is enabled.

trafficLight = New(# data)

Errors out.

The implementation of the change-state operation in the Rust backend only supports transitions to states that do not require a state context, that is, states with no state variables or state parameters.

The relevant function is generate_state_ref_change_state. The code includes a TODO comment, so this was a known issue; just documenting it here.

This is related to issue #22. If the generation of state contexts is refactored and separated from the implementation of transitions, then it should be possible to reuse it to fix initial state context and to finish the implementation of change-state.

This does not generate a print action:

-actions-

print[msg:string] {` `}

It would be very helpful to have a grammar for Frame's syntax.

Currently, Frame's grammar is documented only informally on the Frame Notation page. However, some aspects of the grammar are very under-specified there.

The biggest gap is Frame's expression syntax. A grammar for that piece of the language would be especially helpful.

Fortunately, Frame expressions seem to follow standard conventions, so guessing at the syntax has mostly worked out. However, the lack of a grammar significantly impacts debugging since it's impossible to determine whether errors are bugs in the Frame spec, a conceptual misunderstanding of how Frame works, or bugs in the Frame implementation. It also makes filing issues and working on Frame itself more difficult.

A line ending in a variable reference followed by a transition with an exit argument is incorrectly parsed as a function call.

For example, consider the following sequence of Frame statements:

var tmp = 5 * arg
(10) -> (tmp) $Bar(x)

These are incorrectly parsed as if this was written:

var tmp = 5 * arg(10)
-> (tmp) $Bar(x)

On the other hand, if the previous line ends in a literal number, the program is parsed correctly:

var tmp = arg * 5
(10) -> (tmp) $Bar(x)

Working proposal for a composition model for implementing actions via a separate, underived object rather than strictly relying on inheritance from the controller. This might more easily enable architectures where the object isn't owned/allocated/managed by the controller at all but instead is managed by the environment and is communicated with via messaging.

The current Golang implementation uses this approach, though I think I may have just discovered how golang can "inherit" behavior.

More to come...

This would be very useful when generating visualizations to show how we got to the current state. Once we're saving the last transition, it seems a small step to save the whole (or a finite amount of) history, though I'm not sure what the use case for that is.

This was a precursor decl syntax:

.foo:int = 1
rather than
var foo:int = 1

Look for:
else if self.match_token(&vec![DotTok]) {
return if self.match_token(&vec![IdentifierTok]) {

Currently, terminators are not allowed inside of branch statements. Since branches are statement-level, it seems like this should be permitted.

For example, this Frame spec currently returns a (confusing) Frame parse error:

#TerminatorsInBranch
    -interface-
    Foo

    -machine-
    $S1
        |Foo|
            true ?
                -> $S2 ^
            :
                -> $S3 ^
            ::

    $S2
    $S3

    -actions-
    -domain-
##

Instead one must move the terminators out of the branch to write this:

#TerminatorOutOfBranch
    -interface-
    Foo

    -machine-
    $S1
        |Foo|
            true ?
                -> $S2
            :
                -> $S3
            ::
            ^
    $S2
    $S3

    -actions-
    -domain-
##

Enabling within-branch terminators would give handlers finer grained control over whether to propagate a message on to a parent handler in a hierarchical state machine. For example, one could write the following handler, which is currently a parse error:

    |Foo| [b:bool]
            b ?
                log("continue") :>
            :
                log("return") ^
            ::

I've fixed a couple bugs related to this issue already, but it seems it's still around.

This conditional:

!mode_normal() || none_source() ? -> "Mode != Normal || Source == None" $Entry : :: ^

Generates this code on v0.6.0:

            FrameMessage::FormingSourceChanged => {
                if self.mode_normal() || self.none_source() {
                    // Start transition
                    // Mode != Normal || Source == None

This case can probably be simplified a bit.

Thanks to @fdlg for reporting.

Problem

Currently Frame has no standardized system initialization notation. Additionally Frame treats the start state differently than other states in that it does not receive a '>' message upon startup (it can receive one if it is later transitioned back into from another state however).

#FizzBuzz[name:string] 
~@(|<<| [msg:string])
##

This notation assumes the existence of CreateXXX() and DestroyXXX() functions.

The create function will:

1. instantiate the system controller
2. directly inject a '>' event with the passed in parameters.

The destroy function will:

1. directly inject the defined event into the system.

In this way the creation/destruction of the system can be managed independent of the interface.

Frame has a single enter and exit event for each state, which can be handled to perform work when the state is entered or exited, respectively. However, state charts allow multiple entry/exit actions to be defined.

If we solve #59, then this functionality could be recovered with "internal events" by writing something like:

$Foo
  |>|
    action1 ()
    action2 ()

Where action1 and action2 are two different entry actions (represented in Frame as internal events) for state Foo.

However, there is still the challenge of how to show these entry actions in visualizations. A visualization of the above state Foo should include a label like entry/ action1, action2 (for example, see the description of action notation here).

As a special case, we could list all internal events called from enter/exit event handler as entry/exit actions in visualizations.

Another possible (but more complex) solution would be to provide direct support for registering multiple enter/exit events to be triggered for a given state.

In the Rust backend, the initial state context is initialized differently than subsequent state contexts, which causes problems if the initial state contains state variables (or any other features that rely on state contexts).

For example, the following Frame spec generates Rust code that does not compile.

#InitContextBad
    -interface-
    -machine-
    $S
        var x:i16 = 0
    -actions-
    -domain-
##

This could be fixed by refactoring the Rust visitor to use the same code path to generate the initial state context as is used for subsequent state contexts.

However, in the short term, a simple workaround exists. Just include a trivial initial state that immediately transitions into the "real" initial state.

#InitContextWorkaround
    -interface-
    -machine-
    $I
        |>| -> $S ^
    $S
        var x:i16 = 0
    -actions-
    -domain-
##

Currently the code doesn't follow the rust style guide so running cargo fmt to help automate styling results in modifications to a lot of files.

Would be nice to make it conformant running cargo fmt once and cheking in the results to allow devs to use cargo fmt going forward