croconut / godot-tester Goto Github PK

View Code? Open in Web Editor NEW

36.0 36.0 14.0 1.82 MB

A Github Action to handle testing Godot applications with GUT

License: MIT License

Dockerfile 0.12% GDScript 93.25% JavaScript 6.63%

godot

godot-tester's People

Contributors

Stargazers

Watchers

Forkers

adarapata otaviosoato zutatensuppe you-win db0 onepersonhere ericdowney ghgsrt rgonzaleztech gamesforgood rainbowcow-studio tysterman74 dhleong gslogimaker

godot-tester's Issues

CI run fails: .stex files missing

There's something odd about my Godot project. Whether I download/invoke Godot through my own bash scripts, or via this command, the tests hang, and early on, I see this message:

ERROR: Failed loading resource: res://.import/protagonist.png-bc286cb64aca4b0378c3f1855cbf21b8.stex. Make sure resources have been imported by opening the project in the editor at least once.

I can reproduce this locally on Windows with the Ubuntu shell; the file is there, on disk, and running the same command again, doesn't cause that error to appear. It looks like - for some reason - running Godot twice, would solve this. (I tried increasing the import-time value, but that doesn't seem to do anything.)

Am I doing something wrong? Any ideas how I can get this working?

Linux Standard as an option

theres pretty major differences between standard and headless in terms of visual based testing. I'll see if the standard linux version works in a container.

Parse exported XML instead of output for build pass/failure

I saw #20, stating that GUT 7.3.0 broke the build. Awhile back GUT got a junit-xml compliant export. You'll have to do a little more work to determine if all the tests passed, since there is no summary in the XML, but parsing this output will be more reliable across GUT builds.

You can find more information about it on the wiki

Actions stuck on unzipping

I'm trying to create my first pipeline using godot-tester, but it's failing because it's getting stuck during unzipping

Run croconut/[email protected]
/usr/bin/docker run --name e28490be9d109211014fcd93cde7beaf172b45_05c85b --label e28490 --workdir /github/workspace --rm -e INPUT_IMPORT-TIME -e INPUT_TEST-TIMEOUT -e INPUT_MINIMUM-PASS -e INPUT_TEST-DIR -e INPUT_VERSION -e INPUT_RELEASE_TYPE -e INPUT_PATH -e INPUT_IS-MONO -e HOME -e GITHUB_JOB -e GITHUB_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_REPOSITORY_OWNER -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RETENTION_DAYS -e GITHUB_RUN_ATTEMPT -e GITHUB_ACTOR -e GITHUB_WORKFLOW -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GITHUB_EVENT_NAME -e GITHUB_SERVER_URL -e GITHUB_API_URL -e GITHUB_GRAPHQL_URL -e GITHUB_REF_NAME -e GITHUB_REF_PROTECTED -e GITHUB_REF_TYPE -e GITHUB_WORKSPACE -e GITHUB_ACTION -e GITHUB_EVENT_PATH -e GITHUB_ACTION_REPOSITORY -e GITHUB_ACTION_REF -e GITHUB_PATH -e GITHUB_ENV -e RUNNER_OS -e RUNNER_ARCH -e RUNNER_NAME -e RUNNER_TOOL_CACHE -e RUNNER_TEMP -e RUNNER_WORKSPACE -e ACTIONS_RUNTIME_URL -e ACTIONS_RUNTIME_TOKEN -e ACTIONS_CACHE_URL -e GITHUB_ACTIONS=true -e CI=true -...
downloading godot from https://downloads.tuxfamily.org/godotengine/3.2.2/Godot_v3.2.2-stable_linux_headless.64.zip ...
unzipping ...

I've waited over an hour for this to proceed, but it never does.

Feature Request: Log level

It could be very useful to set the level of logs in the workflow file like you can do in Godot.

When there is no error, log level at 1 is enough, but if something fails, having logs level to 2 or 3 can come handy.

ignore test code / game data in release

Should generally be able to manage this with a . gitattributes file. Should ignore everything that isn't absolutely required for end user use as an action.

Forgot to remove `IGNORE_ERROR` input 😅

$IGNORE_ERROR is no longer used since the results come from the XML file rather than STDOUT.

How to disable logging for asserts?

Currently in my test suite i have about 7k asserts (integration testing).

The problem is the gut print output (I believe) is crashing the runner, so the task gets terminated early.

Is there anyway to disable output apart from failures?

Thanks

Example below:

In this example, I turned off assert statement comments to reduce the amount of lines printed - previously it would get stuck at around 600, now we are up to over 2k.

Unable to Download Godot

Noticed the following issue two days ago trying to use the actions as normal with a pull request:

2m 17s
Run croconut/godot-tester@v5
  with:
    version: 3.5
    release_type: stable
    is-mono: false
    path: .
    import-time: 100
    test-timeout: 60
    minimum-pass: 0.99
    test-dir: res://Test
    assert-check: true
    max-fails: 1
    result-output-file: test_results.xml
    direct-scene: false
    config-file: res://.gutconfig.json
/usr/bin/docker run --name dadd4faca31dd2175147bcb2c3c4198046a812_ab61fc --label dadd4f --workdir /github/workspace --rm -e "INPUT_VERSION" -e "INPUT_RELEASE_TYPE" -e "INPUT_IS-MONO" -e "INPUT_PATH" -e "INPUT_IMPORT-TIME" -e "INPUT_TEST-TIMEOUT" -e "INPUT_MINIMUM-PASS" -e "INPUT_TEST-DIR" -e "INPUT_ASSERT-CHECK" -e "INPUT_MAX-FAILS" -e "INPUT_RESULT-OUTPUT-FILE" -e "INPUT_DIRECT-SCENE" -e "INPUT_CONFIG-FILE" -e "INPUT_CUSTOM-GODOT-DL-URL" -e "HOME" -e "GITHUB_JOB" -e "GITHUB_REF" -e "GITHUB_SHA" -e "GITHUB_REPOSITORY" -e "GITHUB_REPOSITORY_OWNER" -e "GITHUB_REPOSITORY_OWNER_ID" -e "GITHUB_RUN_ID" -e "GITHUB_RUN_NUMBER" -e "GITHUB_RETENTION_DAYS" -e "GITHUB_RUN_ATTEMPT" -e "GITHUB_REPOSITORY_ID" -e "GITHUB_ACTOR_ID" -e "GITHUB_ACTOR" -e "GITHUB_TRIGGERING_ACTOR" -e "GITHUB_WORKFLOW" -e "GITHUB_HEAD_REF" -e "GITHUB_BASE_REF" -e "GITHUB_EVENT_NAME" -e "GITHUB_SERVER_URL" -e "GITHUB_API_URL" -e "GITHUB_GRAPHQL_URL" -e "GITHUB_REF_NAME" -e "GITHUB_REF_PROTECTED" -e "GITHUB_REF_TYPE" -e "GITHUB_WORKFLOW_REF" -e "GITHUB_WORKFLOW_SHA" -e "GITHUB_WORKSPACE" -e "GITHUB_ACTION" -e "GITHUB_EVENT_PATH" -e "GITHUB_ACTION_REPOSITORY" -e "GITHUB_ACTION_REF" -e "GITHUB_PATH" -e "GITHUB_ENV" -e "GITHUB_STEP_SUMMARY" -e "GITHUB_STATE" -e "GITHUB_OUTPUT" -e "RUNNER_OS" -e "RUNNER_ARCH" -e "RUNNER_NAME" -e "RUNNER_ENVIRONMENT" -e "RUNNER_TOOL_CACHE" -e "RUNNER_TEMP" -e "RUNNER_WORKSPACE" -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN" -e "ACTIONS_CACHE_URL" -e "ACTIONS_RESULTS_URL" -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/M2/M2":"/github/workspace" dadd4f:aca31dd2175147bcb2c3c4198046a812  "--godot-version" "3.5" "--release-type" "stable" "--path" "." "--is-mono" "false" "--import-time" "100" "--test-timeout" "60" "--minimum-pass" "0.99" "--test-dir" "res://Test" "--direct-scene" "false" "--assert-check" "true" "--max-fails" "1" "--config-file" "res://.gutconfig.json" "--custom-godot-dl-url" "" "--result-output-file" "test_results.xml"
Downloading Godot from:  https://downloads.tuxfamily.org/godotengine/3.5/Godot_v3.5-stable_linux_headless.64.zip
(node:1) [DEP0147] DeprecationWarning: In future versions of Node.js, fs.rmdir(path, { recursive: true }) will be removed. Use fs.rm(path, { recursive: true }) instead
(Use `node --trace-deprecation ...` to show where the warning was created)
node:internal/validators:372
    throw new ERR_INVALID_CALLBACK(callback);
    ^

TypeError [ERR_INVALID_CALLBACK]: Callback must be a function. Received undefined
    at makeCallback (node:fs:190:3)
    at Object.unlink (node:fs:1751:14)
    at ClientRequest.<anonymous> (/usr/lib/DownloadGodot/helpers/PerformDownload/PerformDownload.js:15:16)
    at ClientRequest.emit (node:events:513:28)
    at TLSSocket.socketErrorListener (node:_http_client:494:9)
    at TLSSocket.emit (node:events:513:28)
    at emitErrorNT (node:internal/streams/destroy:157:8)
    at emitErrorCloseNT (node:internal/streams/destroy:122:3)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  code: 'ERR_INVALID_CALLBACK'
}

I noticed the endpoint at https://downloads.tuxfamily.org/ seems to be down when I try to access it manually, so I attempted to define a custom custom-godot-dl-url param to download directly from the godot releases page. Unfortunately I still seem to be noticing weird errors when attempting to download Godot:

I have attached my current YAML file as well as the output when testing locally. Is anyone else able to recreate this issue or point out something I might have overlooked?

unit-test.txt
output.txt

Thanks in advance!

move to containerless

To move to containerless, possible due to our use of node and system packages already available to a test runner, we'll need to start bundling the node packages.

Would recommend esbuild and development of a custom release build script.

Should also use packages native to node / the test runner env as much as possible - will likely move to octocat over yargs for prod deploy, if yargs was used outside of testing.

Pass rate is considering "pending" tests as failed checks

In your entrypoint.sh you have:

test_failed_string="- test"
[...]
    elif [[ $temp =~ ^$test_failed_string ]] ; then
        FAILED=$((FAILED+1))
[...]

But this will report a false positive on this perfectly passing report.

*** Run Summary ***
res://tests/integration/test_piles.gd
- test_popup_discard_view
    [Pending]:  Drawing a card from the pile, picks it from the popup
res://tests/integration/test_scripting_engine_general.gd
- test_basics
    [Pending]:  Empty does not create a ScriptingEngine object
res://tests/integration/test_scripting_engine_nested.gd
- test_draw_more_cards_than_pile_max
    [Pending]:  Test good execution of nested tasks
    [Pending]:  Test failing cost in nested cost task aborts parent script
res://tests/integration/test_scripting_engine_tasks_execute_scripts.gd
- test_no_costs_are_paid_when_target_costs_cannot_be_paid
    [Pending]:  is_cost == true execute_tasks: No costs were paid because target card could not pay costs
    [Pending]:  is_cost == false execute_tasks: Cost were paid because target card could not pay costs
    [Pending]:  is_cost == true execute_tasks: costs were paid when subject == boardseek

The script needs to be able to distinguish reports of pending tests from actual faulted tests.

Personally, I think this extra check on your end is not necessary. Have you seen instances where you will catch a failure by grepping for those lines starting with - test?

Error when trying to make .cli_support

Hey! I'm facing this error when trying to use the action.

Downloading Godot from:  https://downloads.tuxfamily.org/godotengine/4.1/Godot_v4.1-stable_linux.x86_64.zip
(node:1) [DEP0147] DeprecationWarning: In future versions of Node.js, fs.rmdir(path, { recursive: true }) will be removed. Use fs.rm(path, { recursive: true }) instead
(Use `node --trace-deprecation ...` to show where the warning was created)
Unzipping Godot...
Godot unzipped to:  /tmp/Godot_v4.1-stable_linux.x86_64
exePath:  /tmp/Godot_v4.1-stable_linux.x86_64
node:fs:1382
  handleErrorFromBinding(ctx);
  ^

Error: ENOENT: no such file or directory, mkdir '/github/workspace/addons/gut/.cli_support'
    at Object.mkdirSync (node:fs:1382:3)
    at addRebuilderSceneToProjectAddonFolder (/usr/lib/RunGodotImport/helpers/ImportRebuilderHelpers/ImportRebuilderHelpers.js:[24](https://github.com/Ocacho-Games/Home-Godot/actions/runs/5671711739/job/15369416244#step:7:25):12)
    at runGodotImport (/usr/lib/RunGodotImport/RunGodotImport.js:13:5)
    at runTests (/index.js:13:5) {
  errno: -2,
  syscall: 'mkdir',
  code: 'ENOENT',
  path: '/github/workspace/addons/gut/.cli_support'
}

This is the setup I have for the action

    - name: Passing GUT Tests
      uses: croconut/godot-tester@v5
      with:
        # required
        version: "4.1"

In case this helps, this is a photo of my repository hierarchy. The .godot is inside the godot folder and I'm executing the action from .github folder

Thanks in advance!

Docker image node install deprecated

While running a unit test on v5, I noticed this delay. If I get a chance to work on this, I'll open a PR for it.

  #7 [ 3/12] RUN curl -sL https://deb.nodesource.com/setup_16.x | bash -
  #7 0.274 
  #7 0.274 ================================================================================
  #7 0.274 ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
  #7 0.274 ================================================================================
  #7 0.274 
  #7 0.274                            SCRIPT DEPRECATION WARNING                    
  #7 0.274 
  #7 0.274   
  #7 0.274   This script, located at https://deb.nodesource.com/setup_X, used to
  #7 0.274   install Node.js is deprecated now and will eventually be made inactive.
  #7 0.274 
  #7 0.274   Please visit the NodeSource distributions Github and follow the
  #7 0.274   instructions to migrate your repo.
  #7 0.274   https://github.com/nodesource/distributions
  #7 0.274 
  #7 0.274   The NodeSource Node.js Linux distributions GitHub repository contains
  #7 0.274   information about which versions of Node.js and which Linux distributions
  #7 0.274   are supported and how to install it.
  #7 0.274   https://github.com/nodesource/distributions
  #7 0.274 
  #7 0.274 
  #7 0.274                           SCRIPT DEPRECATION WARNING
  #7 0.274 
  #7 0.274 ================================================================================
  #7 0.275 ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
  #7 0.275 ================================================================================
  #7 0.275 
  #7 0.275 TO AVOID THIS WAIT MIGRATE THE SCRIPT
  #7 0.275 Continuing in 60 seconds (press Ctrl-C to abort) ...
  #7 0.275

I can clean up the issue description as needed. Just dropping this here for now to inform.

result-output-file not working sometimes

Hey again!

This is my configuration:

  pass_tests:
    name: Passing GUT Tests
    runs-on: ubuntu-latest
    permissions: write-all
    steps:
      - uses: actions/[email protected]
      - run: touch ./godot/test_results.xml
      - run: ls && ls ./godot
      - uses: croconut/godot-tester@v5
        with:
          version: "4.1"
          path: "godot"
          import-time: "5"
          test-timeout: "45"
          minimum-pass: "1.0"
          test-dir: "res://test"
          config-file: "res://.gut_editor_config.json" 
          result-output-file: "test_results.xml"

After the tests are passed, I got this error:

/usr/lib/AnalyzeTestResults/AnalyzeTestResults.js:26
        throw new Error('No test results found');
              ^

Error: No test results found
    at analyzeTestResults (/usr/lib/AnalyzeTestResults/AnalyzeTestResults.js:26:15)
    at runTests (/index.js:15:21)

Which doesn't make a lot of sense cause I'm creating the file with the command touch ./godot/test_results.xml I also run the ls command to make sure the file was there.
I also tried creating the file on the root folder instead of ./godot but same result.

Run ls && ls ./godot
  ls && ls ./godot
  shell: /usr/bin/bash -e {0}
Home
godot
scripts
addons
export_presets.cfg
icon.svg
icon.svg.import
project.godot
test
test_results.xml

Thanks in advance!

convert entrypoint to js

The entrypoint file is in bash, which is an unwieldy, barely readable language. Despite the efforts to make bash more friendly, JavaScript would make for a much more flexible/maintainable language.

That said, when using the node ecosystem, updates/maintenance essentially becomes required so I'm very open to changing to a slower moving language.

Coverage exceeds 100% when using parameterization tests.

I'm using a translation tool, so sorry if it's hard to understand.

Thank you for making such a great product. 👍

When I include parameterization or perform multiple assertions in a single method, the rate becomes an inappropriate value.

Here is the current formula

rate = Tests / Passing asserets.

However, since passing asserets counts the number of assert methods, it may exceed the number of Tests.

I fixed it in the following way

adarapata#1

However, this is not a complete solution because when a timeout occurs, it is not included in Passing asserts nor Failing asserts.

However, one problem has been solved and I am hoping that this will be taken care of once and for all. What do you think?

Writing a file fails test

Description

I have a test, which writes a file to a path similar to res://tests/temp/config.cfg.
I also run

func after_each():
    directory.remove(path)

to remove the file after every single test execution.

This works fine on my local machine. However, it does not with godot-tester.

I assume this is due to file system permissions on the container?

Expected Behaviour

File gets written in test
Test can assert that the file exists
File will be removed after test

Actual Behaviour

Test fails:

[Failed]:  
      at line -1

Repro

Comment out this test and run on GitHub Action

Repro Test: https://github.com/Structed/godot-playfab/blob/db0ca9f1e1ee170080394fa8ed21bd7a22778a34/addons/godot-playfab/test/unit/PlayFabClientConfig/test_PlayFabClientConfigLoader.gd#L13

integrate build process into release / tagging

When tagging on the main branch, should be able to auto gen release notes and create a build to upload to GitHub actions.

With the move to containerless, we either need to build and upload manually for releases or add it as a tag action... let's just add as an action.

See:
https://github.com/marketplace/actions/build-and-tag

Broken with GUT 7.3.0

Looks like in the the new version of GUT, the final output changed in a way that breaks the godot-tester

Totals
Scripts:          37
Passing tests     546
Failing tests     0
Pending:          8
Asserts:          4323/0

Warnings/Errors:
* 33 Warnings.


546 passed 0 failed.  Tests finished in 987.8s


Hypnagonia Exited Gracefully.
ERROR: Condition "_first != nullptr" is true.
   at: ~List (./core/self_list.h:108)
ERROR: Condition "_first != nullptr" is true.
   at: ~List (./core/self_list.h:108)
ERROR: Condition "_first != nullptr" is true.
   at: ~List (./core/self_list.h:108)
WARNING: ObjectDB instances leaked at exit (run with --verbose for details).
     at: cleanup (core/object.cpp:2064)
ERROR: Resources still in use at exit (run with --verbose for details).
   at: clear (core/resource.cpp:417)
ERROR: There are still MemoryPool allocs in use at exit!
   at: cleanup (core/pool_vector.cpp:63)
Note: Tests may appear to pass on SCRIPT ERROR, but they were just ignored by GUT
This action ensures those will fail

Tests failed due to timeout or there were no tests to run

This happened as soon as I upgraded, so I'm fairly confident it's caused by the new GUT version

Relevant run

Catching failed scripts ignored by GUT can lead to false positives

E.g. see this

 script error found at SCRIPT ERROR: Invalid get index 'temp_properties_modifiers' (on base: 'previously freed instance').
failed test count increased: 1

This failed because GUT had torn down the test setup, probably while some yields were still waiting. Then when those yields returned, they caused this SCRIPT ERROR. But this is OK, since we don't care about these timing issues during testing. However the tester still reports this as a failure, even through I've set assert-check: true

link: https://github.com/db0/godot-card-game-framework/runs/4340432389?check_suite_focus=true

Godot 4

Any interest in porting this to Godot 4?

Check for Script Errors in output and fail pipeline.

First off it is great to see a tool I started spawning more tools! Thanks for using GUT and making this.

I've been thinking a lot about bitwes/Gut#210 which has to do with errors in scripts causing a test to exit early and appear as passing. There's been a bunch of ideas floated around but the most bulletproof approach I can think of (given Godot limitations) would be to detect Godot script errors in the output. Can you get to this output in a Github Action and then fail the pipeline if you find specific strings in it?

(BTW I found two Actions for GUT so I'm going to copy/paste this directly into the other one too. I'm not pitting you against each other, I just don't know anything about Github Actions and I'm not comfortable picking a favorite tool-spawn (hmmm now I'm thinking about a Tool and Spawn collab). I'm sure both pipelines are great. When I searched the marketplace my response was "There's two? THERE'S TWO!". Thanks for making <current tool> and for using GUT!)

parallelize tests

Should swap to a job matrix and have them run in parallel.

It's just slow waiting for all the tests to complete. Should also check if Godot 4 has an stdout for when it's completed the import process, or a way to trigger that.

add verbose flag - default off

Should allow more transparency as to what we're doing so users can debug issues easier.

Specifically, I think we're missing logs for running gut / Godot since they're in subprocesses - we'll want to pipe output streams back to the main process.

croconut / godot-tester Goto Github PK

godot-tester's People

Contributors

Stargazers

Watchers

Forkers

godot-tester's Issues

Description

Expected Behaviour

Actual Behaviour

Repro

Recommend Projects

Recommend Topics

Recommend Org