microsoft / powerapps-tooling Goto Github PK

Is there a PowerShell cmdlet to automate the download / upload of .msapp files?

Adding a responsive container pulls in some extra default properties.

In canvasmanifest.json:
Lots of volatile perf counters, etc that don't below. Move to entropy?

In resources.json:
(the msapp has an image, it looks like the RootPath for the image has a volatile guid)

In data source, extract the “API Definitions” (ie, TableDefinition,Swagger,Wadl,etc) to a separate file. The API def for CDS Contacts is 1.6 MB.
Also need to consider whether these files can be optional (can they just be deleted?) - What happens when an API definition changes?

Remove the Console.WriteLines and use a proper error reporting callback.
The host (PAC CLI) can then implement this callback and show errors properly.

Template.json doesn't round-trip the Components information.

Template.json gets fully sharded to xml files in the \pkg dir, so it doesn't even appear in the source file.
That's losing the Components information.

We're also wrongfully excluding it from checksum because of the escaped xml. The checksum should cover everything.

better source span on yaml.

Do a duplicate control definition in pa.yaml and verify we get meaningful source locations..

When using -pack NewPathToApp.msapp PathToSourceFolder

It will be useful to create the NewPathToApp folder if it doesn't exist, instead of erroring out.

The basic problem is that we load the entire msapp into memory (CanvasDocument).

And then oom at places like this:

                        case FileKind.AppCheckerResult:
                            var appChecker = Encoding.UTF8.GetString(entry.ToBytes());
                            app._entropy.AppCheckerResult = appChecker;
                            break;

We should calculating this in a streaming way rather than ToBytes().
(sarif file can be huge)

We should scrutinize usage of FileEntry.ToBytes, and probably need to be lazier and have documents backed by disk.

For diagnostics, provide a checksum per-file.
Still keep the top-level msapp checksum for easy compare., But if that's different, then we can look at per-file checksums. This can help diagnose things like which Screen or Data source was changed.

The Sarif file is a noisy diff. It's 1-line json file, and changes each time we export.

Setup some backcompat tests to verify that the sources are stable.

Hello,

Having read the blog post at https://powerapps.microsoft.com/en-us/blog/source-code-files-for-canvas-apps/, I have some feedback for you to consider, in order of importance.

1. Don't build extra tools to compensate for a file format that isn't source code/git friendly to begin with. The process of packing and unpacking the .msapp files is what I consider a legacy concept taken from Dynamics 365/CDS/Dataverse/PowerApps/(insert your preferred technology name here) solutions. You don't need to do this! Instead, make the server-side export process for an .msapp itself place the unpacked file's contents directly in the .msapp file (a zip file renamed to .msapp). Then, it's just a matter of unzipping the file before editing on the local machine and saving to source control. No extra specialized tooling is then required beyond using a zip/unzip program/command already built into every OS and scripting language. If you need the files to be packed into some other format, do that yourself on the server after the .msapp file containing the unpacked files is uploaded. Don't offload the responsibility for this cumbersome process to everyone else when you can handle it yourself. Remove the barriers to adopting source control by removing unnecessary steps.
2. It seems like you're using YAML to create a domain specific language that is meant to be a less verbose data structure format than JSON or XML would be. But, you're admittedly saying in https://github.com/microsoft/PowerApps-Language-Tooling/blob/master/docs/YAMLFileFormat.md that YAML standards like escaping and comments aren't supported. All the rules and exceptions and special cases for how the file format works makes me think this isn't the right file format to use and you're twisting YAML into doing something else as a convenience to you rather than a convenience to the users who will be working with these source code files. Why don't you come up with your own DSL where everything can be expressed in the most logical syntax possible without any special rules or edge cases meant to satisfy your YAML file format rules. Work with Microsoft research or the F#/C# compiler teams to do this if the PowerApps team doesn't know how to do this themselves.
3. Make sure there is a compiler and there's IDE support with intellisense/autocomplete, go to definition, etc. Then the files can be edited with guidance and compiled/validated on a user's (i.e. developer's) local machine without wondering if what's being typed in is valid. Raw text files that can be edited by humans without any local validation is going to result in errors no matter how much guidance is provided. This is also going to be needed to deal with merging files as mentioned at https://github.com/microsoft/PowerApps-Language-Tooling#merging-changes-from-studio.
4. Last thing to add just so I can say I've said it, add support for a real programming language so we don't have to use expressions. I'm not as serious about this one, but this is truly what every developer who is forced to build canvas apps is thinking when they're told to build one. There are more of us than you may think. There are dozens of beautiful programming languages that people love and we could do things so much easier with them than with than expressions. Something as simple as trying to express conditional logic with expressions is way too difficult. You could use C#, JavaScript, TypeScript, Python, Ruby, and so many more, and convert them to WASM so they run on the browser. You could make the properties with expressions be function calls to a namespace and function by name with supplied parameters, and the rest of the logic exists in the source code files, and by doing so enable code reuse.

Thanks for your consideration.

Alan

Writing errors to Console.Error instead of Console.Out helps with logging when calling the tool from batches or from custom wrapper tools.
I have noticed this kind of "problematic" implementation in SolutionPackager.exe from Microsoft.CrmSdk.CoreTools too.

When consuming a component via a libraries, in the Consumer, there’s this in propertiesJson:

"LibraryDependencies": "[{\"LibraryId\":\"737b973d-124d-4927-83a4-799e6b60c49b\",\"LibraryLogicalName\":\"\",\"OriginalComponentDefinitionTemplateId\":\"51d8f4a68c934385b451a1bedba9d8ed\",\"SourceLastTimeSaved\":\"2021-01-05T20:12:30.194+00:00\",\"RefusedSourceLastTimeSaved\":\"0001-01-01T00:00:00\",\"InstanceCount\":1}]",

[
{
"LibraryId": "737b973d-124d-4927-83a4-799e6b60c49b",
"LibraryLogicalName": "",
"OriginalComponentDefinitionTemplateId": "51d8f4a68c934385b451a1bedba9d8ed",
"SourceLastTimeSaved": "2021-01-05T20:12:30.194+00:00",
"RefusedSourceLastTimeSaved": "0001-01-01T00:00:00",
"InstanceCount": 1
}
]

This guid points back to Component1.json.

Action:

1. Need to shard Properties.LibraryDependencies. This is nested json. Maybe this is a new “ComponentsReference.json” file?
2. Do we do anything special for the copy of the component in the Consumer?

Using pasopa -unpack outputs the error:

Unhandled exception. System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at PASopa.Program.Main(String[] args) in C:\repos\PASopa\PowerApps-Language-Tooling\src\PASopa\Program.cs:line 51

Expected output:

A required argument -unpack is missing.

Usage: 
-unpack PathToApp.msapp PathToNewSourceFolder
-unpack PathToApp.msapp  // infers source folder

Since this will eventually make its way into SolutionPackager - maybe this isn't a priority?!

The following code gives the error Error PA3003: Parse error: Missing ':'. If this is a multiline property, use | at Screen1.pa.yaml:2,18-2,18

Screen1 As screen:
    # Some comment
    sldSlider As slider:
        # Another comment
        Max: =sldSlider.Width
       ...

but this code packs ok:

Screen1 As screen:
# Some comment
    sldSlider As slider:
        # Another comment
        Max: =sldSlider.Width

When I unpack after a pack with comments - the pa.yaml is missing all the detal after the comment and is just:

Screen1 As screen:

Are yaml comments supported in pa.yaml files?

In the bleow, CustomProperty1 is an Input property and CustomProperty2 is an Output property. Yet they show up the same way, which is very confusing.

Component1 As CanvasComponent:
  CustomProperty1: ="Text, Modified"
  CustomProperty2: ="Text, Modified"

It should be something closer to:

Component1 As CanvasComponent:
    CustomProperty1:
        InputDefault: ="Text, Modified"
    CustomProperty2: ="Text, Modified"

In the future, we may be supporting separated Input and Output properties:

Component1 As CanvasComponent:
    CustomProperty3:
        InputDefault: ="Text, Modified"
        Output: ="Text, Modified"

Not something blocking public preview, but we should look at this as we add the other metadata for property parameters.

I added a new control into a screen.pa.yaml (copied from an existing control but with a new Unique ID - and then packed.

    # Existing control
    img_1 As image:
        Image: =powerapps
        ZIndex: =2
        Y: =162
        X: =varImg2X

    # New control
    img_2 As image:
        Image: =powerapps
        ZIndex: =2
        Y: =250
        X: =varImg3X

The following error was thrown, but the app loads ok:

Error   PA3007: MsApp is corrupted. Duplicate Control Unique Ids
Warning PA2001: Checksum mismatch. Sources have changed since when they were unpacked.
Warning PA2001: Checksum mismatch. File Controls\3.json checksum does not match on extract

Are there some additional steps I need to do when adding a new control via code - rather than via studio?
If I round-trip - I get a new entry in editorstate.json for the new control.

https://github.com/microsoft/PowerApps-Language-Tooling/blob/0a2ff0f7ed6f248b6ff12fd54a30695ba930b1b5/src/PASopa/Program.cs#L126

This line echos Pack. Should be Make

Fixing this should help while debugging.

Determine what to do with .editorState files.

Maybe also split the .editorstate.json into a subfolder under /src/editorstate so it's less crowded.

• if you delete the .pa.yaml, we'll just ignore the corresponding .editorstate
• if you delete the .editorstate, we'll create a bunch of defaults that may not be the same as the original.

This should not happen in well-behaved scenarios. But this can happen if somebody manually edits the files (such as trying a manual textual merge).

Our build infrastructure, /targets/Program.cs takes a hard dependeny on Git being on the path:

static string RootDir = Read("git", "rev-parse --show-toplevel", noEcho: true).Trim();

If it's not on the path, we get a cryptic error.

Should PASopa project be converted to a proper dotnet cli tool?

For reference : https://github.com/dotnet/command-line-api

fyi @MikeStall

YamlWriter test is failing on windows when download from Zip due to CRLF issues:

Expected:<P0: =abc\nObj1:\n P1a: |-\n =A#B\n P1b: =B\n Obj2:\n P2a: =A\n P1c: =C\n>.
Actual: <P0: =abc\r\nObj1:\r\n P1a: |-\r\n =A#B\r\n P1b: =B\r\n Obj2:\r\n P2a: =A\r\n P1c: =C\r\n>."}

There are open compliance tasks that need to be reviewed for your PowerApps-Language-Tooling repo.

Action required: 4 compliance tasks

To bring this repository to the standard required for 2021, we require administrators of this and all Microsoft GitHub repositories to complete a small set of tasks within the next 60 days. This is critical work to ensure the compliance and security of your microsoft GitHub organization.

Please take a few minutes to complete the tasks at: https://repos.opensource.microsoft.com/orgs/microsoft/repos/PowerApps-Language-Tooling/compliance

• The GitHub AE (GitHub inside Microsoft) migration survey has not been completed for this private repository
• No Service Tree mapping has been set for this repo. If this team does not use Service Tree, they can also opt-out of providing Service Tree data in the Compliance tab.
• No repository maintainers are set. The Open Source Maintainers are the decision-makers and actionable owners of the repository, irrespective of administrator permission grants on GitHub.
• Classification of the repository as production/non-production is missing in the Compliance tab.

You can close this work item once you have completed the compliance tasks, or it will automatically close within a day of taking action.

If you no longer need this repository, it might be quickest to delete the repo, too.

GitHub inside Microsoft program information

More information about GitHub inside Microsoft and the new GitHub AE product can be found at https://aka.ms/gim or by contacting [email protected]

FYI: current admins at Microsoft include @MattB-msft, @JimDaly, @davidjenni, @MikeStall, @lajusp

Review the file format to ensure it plays well with source control and Pull Request merging. Ensure each Directory falls into the following category, where all files in the directory have the same rules:

Ensure we can round-trip components.

This is not the ideal experience, we'd prefer the definition be

Component1 As UXComponent

And the use be

Component1_Instance As Component1

Improve the real .msapp format to make conversions easier.

As general principles:

• avoid double json encoding because that breaks checksums.
• separate out volatile information (timestamps, per-environment) from stable information.
• remove fields that are redundant or don't need to be emitted.

Some specific things:

• Double encoding in PCF controls
• Double encoding in Properties.json
• Test cases. (see updates in #52)

If a property is empty, then consider removing it from the pa.yaml

E.g Could HoverFill and HoverBorderColor be removed to lower verbosity?

        imgBox As image:
            Image: =box.img
            ImagePosition: =ImagePosition.Stretch
            X: =0
            Y: =0
            Width: =1024
            Height: =32
            HoverFill: =
            HoverBorderColor: =
            ZIndex: =1

YAML is like marmite - you either love it or hate it! I think there an opportunity here for an open community discussion around the use of YAML as the 'source code format' - possibly from the perspective of:

1. Diff/Merge - we obviously want it easily diff/merge in source control and ALM - would alternatives (e.g. JSON/XML/Other) make it easier or harder to diff/merge?
2. Authoring Experience - if it is intended that this 'language' be used to write apps 'by hand' - then is the format going to support that and be easy to learn? Should there be an editor that understands the format and can provide IntelliSense?

A number of starting points to discuss spring to mind:

1. Verbosity - Some formats like XML are very verbose - JSON can also be quite verbose especially where you have string escaping inside string escaping.
   E.g Consider the following YAML

imgControl As image:
    Image: =If(varCurrentMode=ThisItem.ani,ThisItem.imgSel,ThisItem.imgOff)
    ZIndex: =1
    Width: =82
    Height: =151
    ImagePosition: =ImagePosition.Fill
    OnSelect: |
        // explode all lemmings
        UpdateIf(colLemmings,ani<>"" && Not(ani in ["explode","explode_done","splut_done"]),{
            countdown:10
        });

In JSON the OnSelect value would look far more verbose with lots of escaping - this does make it harder to read.

{
  "imgControl As image": {
    "Image": "If(varCurrentMode=ThisItem.ani,ThisItem.imgSel,ThisItem.imgOff)",
    "ZIndex": 1,
    "Width": 82,
    "Height": 151,
    "ImagePosition": "ImagePosition.Fill",
    "OnSelect": "// explode all lemmings\n                    UpdateIf(colLemmings,ani<>\"\" && Not(ani in [\"explode\",\"explode_done\",\"splut_done\"]),{\n                        countdown:10\n                    });"
  }
}

What do folks think - is there a better option to YAML from this perspective?

2. Indentation/Escaping/New Lines - If this format is intended to be written by hand then how hard will the format be for people to learn? Especially where there are multi-line expressions that all need to be indented the same as the parent containing YAML element
   E.g. Consider the following that will pack:

        imgControl As image:
            Image: =If(varCurrentMode=ThisItem.ani,ThisItem.imgSel,ThisItem.imgOff)
            ZIndex: =1
            Width: =82
            Height: =151
            ImagePosition: =ImagePosition.Fill
            OnSelect: |
                =Set(varCurrentMode,ThisItem.ani);
                If(ThisItem.ani="nuke",

but the following will throw the error Can't have empty multiline expressions. or Missing ':'. If this is a multiline property, use | at ... (because the =Set is not indented correctly).

        imgControl As image:
            Image: =If(varCurrentMode=ThisItem.ani,ThisItem.imgSel,ThisItem.imgOff)
            ZIndex: =1
            Width: =82
            Height: =151
            ImagePosition: =ImagePosition.Fill
            OnSelect: |
=Set(varCurrentMode,ThisItem.ani);
            If(ThisItem.ani="nuke",

This will make it difficult to copy/paste code between expressions - without some IDE tooling.

What do folks think about the alternatives to YAML from the perspective of indentation/escaping/newlines?

3. Consistency - How important do folks value this? Currently, some of the elements are unpacked as json and some as yaml - should there be a sinle format for all elements apart from resources such as images?
   For perspective - for a large app, the majority of the files in the unpacked files are JSON and XML (Data sources, Resource metadata, etc.)

What do folks think - is consistency important?

We should strip LocalDataBaseReference and Local* from Properties.json (CanvasManifest.json). These are noisy diffs and already shared to other files.

If I change screen order, editorstate.json Index changes. What this means is that if I insert a screen into a 30 screen app, 29 editorstate json files will all change.

We should have a single place that lists the screen order (either separate file or an array in CanvasManifest.json)

We have some hacky cases where we use REgex.

To repro, run PASopa.exe from a CLI w/o any args

Expected: usage is shown
Actual: Crashes w/ a stacktrace

Adding a control to screen1 will change the PublishOrderIndex.editorstate for screen2.

What be the cause of the warning:

Warning PA2001: Checksum mismatch. File Controls\194.json checksum does not match on extract

I also get messages like the following:

FAIL: hash mismatch: Controls\145.json
FAIL: hash mismatch: Controls\194.json
FAIL: hash mismatch: Controls\22.json
FAIL: hash mismatch: Controls\3.json
FAIL: hash mismatch: Controls\88.json
FAIL: hash mismatch: Properties.json
FAIL: hash mismatch: References\Templates.json
FAIL!!

This happens on first extract where there were no existing files extracted.

I have noticed that the order of the controls in a screen yaml element are not ordered by zOrder (and hence the order shown in studio). This then results in very noise diffs when unpacking.