jonasjurczok / faketorio Goto Github PK

Running the /faketorio command should execute all features and scenarios.

To generate all info needed for a release this command should do the following:

• collect changes from the changelog (last tag -> now)
• create changelog entry
• changelog entry should follow a simple syntax to distinguish bugfixes from features and breaking changes
• bump mod version accordingly

workflow:

• work on feature branches and merge to master
• locally create new branch
• run faketorio prepare-release
• commit, PR, merge
• build release

simple workflow:

• work on feature branch
• when work finished run prepare release
• include changelog and info.json changes in the PR

While working on factorio-todo-list on a mac which has factorio installed via steam, I discovered that faketorio doesn't quite work out of the box.

First Issue - confirm launch with steam

The first issue is that Steam asks the user to confirm (by clicking a button!!!) the launch with cli args every time. I don't think this is something faketorio can avoid, so feel free to disregard :)

Second Issue - spaces in paths break things

Faketorio doesn't seem to handle spaces in path names very well. This is relevant because the path to the factorio executable on mac+steam is /Users/<user_name>/Library/Application Support/Steam/steamapps/common/Factorio/factorio.app/Contents/MacOS/factorio Faketorio can't seem to follow that path. Similar issue exists with pointing to the mods directory.
workaround: I made a symlink in my home directory that points to the 'app data' directory that contains mods and saves (/Users/<user_name>/Library/Application Support/factorio/), and put a symlink in there that links to the executable directly, so it's all in one place and there're no pesky spaces in the path. Then I updated my .faketorio file to point to the symlinks. This allowed faketorio to launch the game with the new mod build, but added a third issue:

Third Issue - won't create the blank savegame

Faketorio doesnt actually create the 'testing sandbox' savegame in this environment, so upon launching it shows the 'can't find specified save' error. I suspect this also has to do with the complicated filestructure.
workaround: The error isn't fatal, so I just load up an existing game or make a new one and run /faketorio ¯\_(ツ)_/¯

search for a gui element by a given id

You have an example where a function myMod.is_setting_enabled() gets mocked. How is myMod exposed in this case?

I've barely started using faketorio, but it looks like it always requires manual interaction in order to run tests. Is this true? It would be really nice to have it start Factorio, run the tests, and then exit with a report and appropriate exit code.

mocks should be able to fail

The ingame testing engine should recognize failures and successes and report the overall testsuite result to the user.

Minor thing, but when I run /faketorio in-game, the results window appears, but there doesn't seem to be a way to dismiss it, if I want to run the tests again. (If I missed it, apologies)

For CI systems like travis it's difficult to get the full client for a couple reasons

• One needs to be logged in to download the full factorio
• Downloading 800 megabytes repeatedly from Wube servers would be slow (and lots of traffic)
• re-hosting the full version of factorio on an easier to access host seems like trouble

It would be nice if there was a way to run tests against the headless server. I got this working locally by doing the following

• call factorio.run() on the first tick
• intentionally error(...) after all specs have run to stop (crash) the server
• use --start-server instead of --load-game
• use a modified server-settings.json that does things like disabling matchmaking and auto-pause

The only major issue with this approach is that it's not possible to get a player in game. This means no GUI testing, or testing of anything that relies on a player.

The benefits are pretty nice however

• completely automated testing is possible
• quicker game loading
• easier to set up on CI.

• documentation for logging
• log level to log output

The coverage for that file is abysimal.
There are two options:

• change all other tests to call that function instead of interacting with the system directly
• write specific tests for that file.

Currently faketorio prints only the last line of the stacktrace in the error message.
This makes understanding the context of the error quite hard.

Faketorio should print the whole stacktrace.

using faketorio.check()/uncheck() does not trigger a on_gui_checked_state_changed event like it should do.

I really like faketorio's publishing abilities, and I've restructured my mod for it. The one thing I miss from when I just had my working copy in the mods folder, is being able to make edits to control.lua while Factorio is running.

Would you consider making a development mode for the "run" command which symlinks instead of copying script files? I understand that this probably isn't compatible with your current method of staging, but it would make iteration as fast as if the originals were in the mod folder.

The ui has been added in #17 but still needs documentation in the README

The faketorio code is producing a lot of debug messages.
It would be cool to be able to selectively disable some of them.

The easiest way would be a log level like mechanic so that the user can enable or disable logging during the tests as she pleases.

Sometimes it is necessary to mock certain behavior.
It would be cool to have something along the lines of when(myMod.myFunction).thenReturn()

Of course removing the mock should also be possible.

It would be cool to have a before_each/after_each functionality so that controlling test behaviour is a little bit easier.

Issuing a click on a checkbox is not working.
Faketorio needs a way to select/deselect checkboxes.

Where should the control lua be?
Either in the root or in the source folder?

luarocks install faketorio ZLIB_DIR=C:\MinGW\msys\1.0\bin ZLIB_INCDIR=C:\MinGW\msys\1.0\include

Some tests need setup, or rely on a specific set of data being in a save. For example, backwards compatibility tests could be made by loading a save from an earlier version of the mod.

To this end, it would be much better to be able to load a file automatically for a given set of tests than to have to do manual setup each time.

In the tests it has to be possible to click on elements by its id.

The mocking in faketorio is useful, but it would be awesome if I could use the same API as in my busted tests: https://olivinelabs.com/busted/#spies-mocks-stubs

It should be possible to enter text into textboxes

The location of the config should be changable.

Especially for longer running testsuites it would be cool to see the errors on the fly and not just only after the testrun is complete.

Add configuration for letting people include custom files into the combined zip

Is there a way to wait one or more frames during a test? It would be nice to be able to handle interactions over time in order to avoid faking everything.

It should be possible to spawn new players and control them

I think the standard place for a .faketorio file should be in the user's home directory, since it contains information which is not specific to any mod. It would be useful if faketorio looked there by default, so I didn't have to pass the path in with every invocation.

The build should produce a zip file for the package goal

It would be cool to provide an existing savegame to run the mod/tests.

It would be really cool to have a graphical representation of what happens during a faketorio run. And not just output in the chat.

Add support for the changelog.txt and thumbnail.png files on the root level.

In the forum it was requested to be able to click on certain children or say something like "click on the first child of the third child of this element"

Its hard to see what scenarios have been run and which not.

It would be cool to have a "default log" of all executed features/scenarios.

There should be support for graphics and other resources

Faketorio needs a way to assert results. Either we pull in a lib (complicated) or just copy the needed basic asserts (easy)

find by id should return nil if the element does not exist.
assert_element_exists should fail if element is not there.

to make dev iterations shorter copy should also grab the tests.

as completion to the other assert functions

It would be cool to generate a luacov coverage report.
It should be fairly simple to integrate luacov

Depends on #40

The two functions should also default to the first player if none is provided.

faketorio needs mod_gui as a capability