cycling74 / max-test Goto Github PK

https://github.com/Cycling74/max-test/wiki/v2-Overview

There are wrinkles in floating point comparison still.
Perhaps the code from Jamoma2 can be helpful here for potential improvements.

This is a bit sporadic, although on my El Capitan machine I have one checkout of max-test which always works and another which always fails. On my Sierra machine it always fails.

In Sierra, the process runs - very slowly - for a while. See screenshot. The last meaningful message is:

[0901/184516:WARNING:dns_config_service_posix.cc(217)] Failed to read DnsConfig.

(That's nearly an hour into the running time.)

Currently, the test runner always returns 0:

This means that, for example, if max-test is used in GitHub Actions, the action always succeeds regardless of whether any tests fail.

A potential workaround might be to check the result of the MAXTEST_PASS environment variable:

However, environment variables set this way are not (and indeed can’t be) exported to the parent process, as can be seen by running:

ruby -e 'ENV["XXX_RUBY_ENV_TEST_XXX"] = "success"'
echo "XXX_RUBY_ENV_TEST_XXX is \"$XXX_RUBY_ENV_TEST_XXX\""

In addition to the logging to a database, i’d like to see printing to the Max window, with a conclusion when all asserts in the patch has been performed about how many assertions where run, and how many failed. That would be the quickest way to see how a test fares when running it manually, in particular if the test patch contains enough assertions that they need to be embedded in sub patches.

Running test.rb with Ruby 2.6.3 produces errors like this:

#<Thread:0x00007ff18f05e280@packages/max-test/ruby/test.rb:102 run> terminated with exception (report_on_exception is true):
Traceback (most recent call last):
	7: from packages/max-test/ruby/test.rb:103:in `block in establishCommunication'
	6: from /Users/benbracken/.rvm/gems/ruby-2.6.3/gems/rosc-0.1.3/lib/osc/udp.rb:18:in `serve'
	5: from /Users/benbracken/.rvm/gems/ruby-2.6.3/gems/rosc-0.1.3/lib/osc/udp.rb:18:in `loop'
	4: from /Users/benbracken/.rvm/gems/ruby-2.6.3/gems/rosc-0.1.3/lib/osc/udp.rb:20:in `block in serve'
	3: from /Users/benbracken/.rvm/gems/ruby-2.6.3/gems/rosc-0.1.3/lib/osc/server.rb:33:in `dispatch'
	2: from /Users/benbracken/.rvm/gems/ruby-2.6.3/gems/rosc-0.1.3/lib/osc.rb:242:in `decode'
	1: from /Users/benbracken/.rvm/gems/ruby-2.6.3/gems/rosc-0.1.3/lib/osc.rb:242:in `new'
/Users/benbracken/.rvm/gems/ruby-2.6.3/gems/rosc-0.1.3/lib/osc.rb:242:in `initialize': no implicit conversion of nil into String (TypeError)

Perhaps newer versions of Ruby are being more strict somewhere?

The tests complete and successful logs are written, so it seems like a fairly cosmetic issue.

If Max is running when the tests are started, it seems that a second copy of the same Max is launched for testing. (Seems benign, so far.)

Can't run on a Max with spaces in its name. (Probable cause: adoption of IO.popen.)

yedikule:ruby nick$ ruby test.rb "/Applications/Max bb_mxj_less.app/"
Max Automated Test Runner
  Starting the OSC Server...

  Launching Max...
test.rb:174:in `popen': No such file or directory - /Applications/Max (Errno::ENOENT)
    from test.rb:174:in `launchMax'
    from test.rb:194:in `<main>'

should just have it assert that there is a test.terminate object in the patcher -- and fail when it is missing, and then move on. Right now the whole thing stalls waiting for a termination that isn't coming.

currently it just covers single floats, but ideally could also cover:

• lists of floats
• mixed lists
• audio signal vectors
• possibly wrap the functionality of string.equals

thus providing a unified equality object

When I'm running tests I'm seeing several patchers from distinct test projects open at the same time, although they're generally all closed at the end. This is causing issues with (for example) clashes between names/ports in send/receive pairs.

Hi all,

anybody has instructions on how to build max-test for 64-bit Windows? I am using VS2017 but I have progressed very little.

Regards,
Juan.

It seems that the test process hangs when first opening Max, until I focus and move the mouse pointer in one of Max's windows. Then it runs to completion.

OS X 10.11.6.

Hi folks, I followed the tutorial video and got some errors attempting to run the tests following the steps in the video.

The first issue is just that the video is missing the critical step of making the max-test-config.json file. Maybe it can be updated or something added in the description? The error is not too obvious, without doing it I got:

Result:
1: from /usr/local/lib/ruby/gems/2.5.0/gems/sqlite3-1.4.2/lib/sqlite3/database.rb:147:in new' /usr/local/lib/ruby/gems/2.5.0/gems/sqlite3-1.4.2/lib/sqlite3/database.rb:147:in initialize': no such table: tests (SQLite3::SQLException)

After fixing this, I still got errors, though my tests worked. I'm not sure what happened, but it doesn't look like the clean output in the video.

Full Output:
`
MacBook-Pro-7:ruby iainduncan$ ruby test.rb "/Applications/"
Max Automated Test Runner
MAX FOLDER
/Applications
Starting the OSC Server...

Launching Max...
Establishing Communication with Max...
2020-04-18 08:31:02.840 Max[63851:40438231] packages: package name (Video and Graphics) is different from folder name (Video & Graphics)
2020-04-18 08:31:02.853 Max[63851:40438231] packages: package name (Max-SDK) is different from folder name (max-sdk-8.0.3)
2020-04-18 08:31:03.745 Max[63851:40438231] max_startmaxserver
2020-04-18 08:31:04.536 Max[63851:40438231] clang using Sandy Bridge workaround
2020-04-18 08:31:04.709 Max[63851:40438231] sqlite version 3.8.5
2020-04-18 08:31:05.376 Max[63851:40438231] Max: starting maxdb update (schema version 8.1.2b)
2020-04-18 08:31:05.410 Max[63851:40438231] code: en-CA, name English
2020-04-18 08:31:06.806 Max[63851:40438231] test.port sending on 4792
2020-04-18 08:31:06.817 Max[63851:40438231] test.port listening on 4791
2020-04-18 08:31:06.977 Max[63851:40438231] maxworkspace_restore: vol=-674 filename=maxworkspace-63851.txt fullpath=Macintosh HD:/Users/iainduncan/Library/Application Support/Cycling '74/Max 8/Crash Recovery/maxworkspace-63851.txt
Sending ping to Max.
/usr/local/lib/ruby/gems/2.5.0/gems/rosc-0.1.3/lib/osc.rb:297: warning: constant ::Fixnum is deprecated
Ping successfully returned.

2020-04-18 08:31:09.429 Max[63851:40438451] Max: maxdb update complete
Waiting for the Max database to complete...
Sending query to Max.

Telling Max to run all of the tests for us...
2020-04-18 08:31:29.599 Max[63851:40438231] doesn't understand "won't-understand-BOOGLE"
2020-04-18 08:31:30.041 Max[63851:40438231] index out of range for key test-
2020-04-18 08:31:30.515 Max[63851:40438231] @ loadbang: Kung
2020-04-18 08:31:30.515 Max[63851:40438231]
2020-04-18 08:31:30.515 Max[63851:40438231] @ set: Kung
2020-04-18 08:31:30.515 Max[63851:40438231]
2020-04-18 08:31:30.515 Max[63851:40438231] @ set: Fu
2020-04-18 08:31:30.515 Max[63851:40438231]
2020-04-18 08:31:31.303 Max[63851:40438231] ad_spin: looks like audio callbacks have stopped
2020-04-18 08:31:34.075 Max[63851:40438231] integration tests complete

RESULTS

Executed 9 Tests with 39 Assertions
All assertions passed.  Congratulations!

pass
Full test results can be found @
/Volumes/Macintosh HD/Applications/Macintosh HD--Applications--maxtestresults.db3
and explored in your favorite SQLite database client.

#<Thread:[email protected]:102 run> terminated with exception (report_on_exception is true):
Traceback (most recent call last):
7: from test.rb:103:in block in establishCommunication' 6: from /usr/local/lib/ruby/gems/2.5.0/gems/rosc-0.1.3/lib/osc/udp.rb:18:in serve'
5: from /usr/local/lib/ruby/gems/2.5.0/gems/rosc-0.1.3/lib/osc/udp.rb:18:in loop' 4: from /usr/local/lib/ruby/gems/2.5.0/gems/rosc-0.1.3/lib/osc/udp.rb:20:in block in serve'
3: from /usr/local/lib/ruby/gems/2.5.0/gems/rosc-0.1.3/lib/osc/server.rb:33:in dispatch' 2: from /usr/local/lib/ruby/gems/2.5.0/gems/rosc-0.1.3/lib/osc.rb:242:in decode'
1: from /usr/local/lib/ruby/gems/2.5.0/gems/rosc-0.1.3/lib/osc.rb:242:in new' /usr/local/lib/ruby/gems/2.5.0/gems/rosc-0.1.3/lib/osc.rb:242:in initialize': no implicit conversion of nil into String (TypeError)
`

e.g. when inside of an app bundle it will write someplace wonky and hostile to apple-security requirements.