Zip the result strings when the slave is uploading to the server, as well as 
when the server is storing the string.

We need a /monitor page that is independent of /stats, in the sense that 
monitoring is to be alerted when something goes wrong, and the sysadmin/SRE can 
take actions on what he sees, e.g. a slave is missing, there's a high queue 
length, etc. Stats is more about making as much data available to improve 
performance overall and use the data to do longer term decisions.

Manifest file should just use the json file format in the code, this will 
modify test_request_message.py

If a slave fails to PublishResults, the PublishInternalResults should send the 
results to a different location (probably related to the monitoring page) and 
have swarm retry the runner.

By having swarm retry the runner, we hide the internal failure from the user.

Some team need logs from application running as opposed to logs generated by 
tests via the stdout/stderr, so we should add a new field to the Swarm file to 
specify the path of text files to be loaded after the tests have run so that 
their text content can be read and then appended to the test result 
stdout/stderr collected by the local test runner.

In the mean time, we could simply provide a small Python script that receives 
the path of a log file as a command line argument and just reads the file and 
outputs it to stdout... There could be other command line arguments to specify 
header/footer text as needed.

That should do it.

Some of the runner stats are split up by dimensions, but this is only possible 
for runners who were able to convert their dimensions from the string into a 
list of hashes.

If the dimensions weren't converted to hashes than we can't use them in queries 
so they just get skipped.

What steps will reproduce the problem?
1. Add an 'output_destination' field to a test run, include a 'size' field '10'
2. Check that 'size' gets sent to the swarm slave
3. ???

What is the expected output? What do you see instead?
Packetized output expect, but the entire output gets sent.

When download from Google Storage fails, retry by resuming download from last 
fetched offset (instead from fetching it from scratch).

It may improve performance and reliability for cases when Google Storage acting 
flaky.

Currently the local test runner has a hard coded value indicating how often it 
be polling back to the main server when it is running a test so the server 
doesn't think it timed out.

This should be replaced with another value that is passed in the swarm file to 
the localtestrunner from the server (possibly just another variable in the 
TestRun class) 

What steps will reproduce the problem?
1. add 'cleanup': 'data' to testrun file
2. run swarm test

What is the expected output? What do you see instead?
zip files and all unzipped files should be deleted.  Instead all files still 
exist until we run out of space.

What version of the product are you using? On what operating system?


Please provide any additional information below.

We should add a feature to Swarming that will -- when requested by a user -- 
synchronize a remote machine to the correct current time (or a specific 
date/time) before running the actions.  This will be useful whenever remote 
machines may not have the correct current system time or some actions need to 
be run at a given time/date.

We should allow people to download unzipped file, and also copy them from a 
file path as opposed to an HTTP download.

Also add support to the swarm generator so that files can be copied from 
anywhere to added to a zip file, not just local files.

Currently local_test_runner.py logs its data to a console stream and a 
temporary file. When it is run remotes on linux there is no console stream so 
only the temporary file can provide data and it isn't easy to find (it is a 
temp file made by the system).

This should be changed so that it is easy for a user to quickly find the log 
file (possibly by just creating a log file in the directory you run swarm in).

Currently, if we can't find an remote machine for a request (e.g., when we get 
the following error:

WARNING  2010-09-16 19:56:03,004 machine_manager.py:333] Can't open request, 
exception: Cannot find a matching image in the lab for given dimensions (11)

), the test stays pending... It shouldn't... We should have a way to tell in 
the UI that will never get run, and we should stop trying to get a remote 
machine for it.

The local test runner can't completely cleanup the root folder on the remote 
machine since it's sitting in it. The Swarming server should take care of it 
using a new API to recursively delete a folder on the XMLRPC server running on 
the remote machine.

Now that the exit codes of test runs are passed to the result URL, we could 
display them on the Swarm server main page...

We may want to check if the persisted data is there or not for currently 
existing servers... or not... :-) 

When a subporcess command fails, the local test runner may try to cleanup temp 
files before the one used as the stdout pipe gets released by the other process 
(easier to reproduce on Windows)...

For example:

D:\src\swarming\src>python local_test_runner.py -v -f sample_run.swarm
root        : INFO     Test case: My First Swarm Test Case starting to download 
data
root        : INFO     Downloading: D:\src\swarming\src\print_a_lot.zip from 
http://swarming.googlecode.com/files/print_a_lot.zip
root        : INFO     Running tests from My First Swarm Test Case test case
root        : INFO     Test Print 10*.1 second
root        : ERROR    Execution of ['blu', 'python', '-u', 'print_a_lot.py', 
'-t 0.1', '-n 10'] raised exception: [Error 2] The system cannot find the file 
specified.
Traceback (most recent call last):
  File "local_test_runner.py", line 346, in _RunCommand
    stdin=subprocess.PIPE)
  File "D:\src\depot_tools\python_bin\lib\subprocess.py", line 595, in __init__
    errread, errwrite)
  File "D:\src\depot_tools\python_bin\lib\subprocess.py", line 804, in _execute_child
    startupinfo)
WindowsError: [Error 2] The system cannot find the file specified
root        : ERROR    [Error 32] The process cannot access the file because it 
is being used by another process: 
'c:\\users\\mad\\appdata\\local\\temp\\tmpvps5kq'
Traceback (most recent call last):
  File "local_test_runner.py", line 637, in main
    (success, result_codes, result_string) = runner.RunTests()
  File "local_test_runner.py", line 487, in RunTests
    env=dict(test_env_vars))
  File "local_test_runner.py", line 350, in _RunCommand
    CleanupTempFiles()
  File "local_test_runner.py", line 332, in CleanupTempFiles
    os.remove(stdout_file_name)
WindowsError: [Error 32] The process cannot access the file because it is being 
used by another process: 'c:\\users\\mad\\appdata\\local\\temp\\tmpvps5kq'

Some tools scripts (e.g., remote_test_run.py) don't work from the tools folder 
because they import stuff in common. Same thing for the local_test_runner.py 
under test_runners, it must be copied to the top level folder to work, or run 
in an environment where the root of swarming (above common) is a part of the 
Python path we should fix that, somehow.

The local test runner support multiple scheme to return test results:
http://code.google.com/p/swarming/source/browse/trunk/src/test_runner/local_test
_runner.py#555

But the Swarm server doesn't and it should...
http://code.google.com/p/swarming/source/browse/trunk/src/server/test_manager.py
#509

Added a helper class to generate TestRunner instances for tests, since there 
are quite a few tests that are repeating this code (because TestRunner also 
required a TestRequest model).

Users may need to do system level things like restart the machine it is 
currently running on (and resume after it restarts). Swarming need to support 
this somehow.

Some suggestions so far:
 - Add an option/flag to .swarm files that does this as an action.
 - Write a module that runs on the target hive machine, which exposes an API for the user's action to call against to control the remote machine.

When swarm is polled it will use that url as the swarm server url and pass that 
to all swarm bots.

If the server is running locally and is polled by going to localhost/ then the 
swarm bots will be told that the swarm server url is localhost/.

There should be some logging (or restrictions on using localhost) to prevent 
this from happening because it isn't to clear why the machines fail to return 
in this case.

So I'm encountering a problem where local_testrunner tries to unzip to a locked 
file:

2011-08-29 18:32:09,079 root         ERROR    Failed to unzip 
/swarm_tests/fh3ix19hyt.zip
Exception: [Errno 26] Text file busy: '/swarm_tests/src/out/Release/chrome'
Traceback (most recent call last):
  File "/swarm_tests/local_test_runner.py", line 434, in DownloadAndExplodeData
    zip_file.extractall(self.data_dir)
  File "/usr/lib/python2.6/zipfile.py", line 938, in extractall
    self.extract(zipinfo, path, pwd)
  File "/usr/lib/python2.6/zipfile.py", line 926, in extract
    return self._extract_member(member, path, pwd)
  File "/usr/lib/python2.6/zipfile.py", line 970, in _extract_member
    target = file(targetpath, "wb")
IOError: [Errno 26] Text file busy: '/swarm_tests/src/out/Release/chrome'

While the crashing is normal since it can't unzip there anyways, swarm seems to 
register this as a "passing test", when its more like an error (I'm getting a 
post to /result_url with an exit code of 0, and the swarm frontend indicates 
that the test passed).  Should that be the case?

There are a few places (mainly in the local test runner) where we do string 
concatenation with +=... This is expensive, we should use the StringIO class 
instead.

The RetryHandler sets the machine_id to 0 when it should just reset the start 
time to identify that a test should be retried. It used to be that a machine_id 
of 0 meant unassigned, but not anymore...

We need a /monitor page that is independent of /stats, in the sense that 
monitoring is to be alerted when something goes wrong, and the sysadmin/SRE can 
take actions on what he sees, e.g. a slave is missing, there's a high queue 
length, etc. Stats is more about making as much data available to improve 
performance overall and use the data to do longer term decisions.

We need to sort it out with ereporter2 too and this needs to be done on both 
servers.

Since the swarm server can no longer directly connect to a slave, it can't kill 
any running processes once they've timed out.

This means it is possible for the server to mark as test as timed out, but then 
have the test still return at a later time.

Currently we just consider this a repeated result (since we added a result of 
timing out earlier) and the fact the we just got data from a timed out test 
isn't recorded anywhere.

It is not clear what (if anything), should be done in this case, so I'm just 
adding this bug to remind us of this issue.

We currently use cron tasks for that, and they can't run more often than once a 
minute.

While running the Swamring server as an AppEngine dev server, we could set a 
cron job on the same machine from a straight WGET to tasks/poll to a "for loop" 
that does is it 6 times at 10 seconds interval and have the cron job set to run 
every minute.

Once we run in AppEngine, we will need to find another solution. The problem 
with having our tasks run every minute and making it do stuff for a whole 
minute is that we will run out of quota, or will block the users access (I 
think)... So we need to investigate how we will fix that in the future when the 
Swarming server runs on a real AppEngine instance.

So that we can more easily filter them when looking for results.

The Test Manager use an appengine Model called IdleMachine to keep track of 
remote machines that are idle and ready to run configuration tests.  The TRM 
often has the unique id of a machine and retrieves it from the data store using 
code such as this:

 idle_machine = IdleMachine.gql('WHERE id = :1', info.id).get()

This type of request is inefficient.  It would be better to use the machine id 
in the name of the object so that it can be retrieved with code like this:

 idle_key = db.Key.from_path(IdleMachine.kind(), info.id)
 idle_machine = IdleMachine.get(idle_key)

The same applies to the TestRunner model.

Swarm currently use c:\7zip by default as the way to unzip downloaded archives. 
Even though it provides a customization of the unzipper to use, it should 
simply use the zip module of Python instead.

The binaries attribute in TestConfiguration or TestCase (found in 
test_request_message.py) is no longer needed and it should be removed.

A problem can arise where an newer test request can take a swarm bot before an 
older test does, even if they have the exact same requirements.

This occurs because when a machine is returned after a test run, the old test 
won't try to take it until the next poll, but when the new test arrives it will 
immediately see if there is a free machine available and if there is it will 
use it.

The code should be modify so that the old test request will never be preempted 
by a newer one, since this could led to starvation.

In test_manager.py, in the function _EmailTestResults a call to mail.send_mail 
is in a try block where we catch all exceptions. Even though the exceptions are 
logged this can result in a valid (but not directly related to send_mail) 
exception getting caught.

The catch should be updated so that it only catches the exceptions that we 
actually expect from mail.send_mail

Currently if swarm allocates a machine and the machine fails to get the file 
uploaded, swarm counts this as a failure and stops. The test didn't have a 
chance to run though and the end user might (incorrectly) think that something 
is wrong with there test, when the issue was really with swarm.

Swarm should still count the run as a failure, but if the failure wasn't due to 
a task then swarm should try to run the test.

Extra points if the end user is unaware that this occurred.

In case the local test runner gets in weird trouble because of very badly 
behaving tests, it would be useful to send the tests output logs incrementally 
back to the server so that we can get some information as to how far the tests 
were able to go.

What steps will reproduce the problem?
1. Run swarm
2. Run many (500+) tests
3. Load the swarm's main status page

What is the expected output? What do you see instead?
Should load fairly quickly, but instead takes about 5-10 minutes.

What version of the product are you using? On what operating system?


Please provide any additional information below.

When running on Mac, the Local Test Runner fails to get any output from the 
steps it runs. The steps are correctly run.

After some investigation the problems seemed to be if the Local Test Runner 
attempts to read the file that acts as a buffer for the stdout from the child 
step process, then the child process can no long write to the file.

If the Local Test Runner waits until the child process is done, it can retrieve 
all the output.

I'll submit a temporary work around to make Local Test Runner always wait for 
the child process to finish before reading so we can get the output on mac.

Until this issue is fixed we will be unable to provide streaming output on macs.

Users should be able to tell Swarm to fetch data from a URL with specific 
headers. The files could available at a given URL, with a TOKEN=xxx header to 
the GET request.

It should be easy for someone with a set of machines to make them available to 
a Swarm server.

We could reuse the machine provider currently written for the Chrome Golo, and 
write a simple Python base web server that would provide the machine in the 
same way that the Chrome Golo machine provider uses vSphere.

Currently, if we lose the AppEngine storage for some reason, we lose access to 
the provided machines because we don't remember requesting them.

There should be an API on the machine provider that allows us to list the 
existing instances we have and the server could use it to reconnect to them.

We need to make sure that if two or more people are using Swarming 
simultaneously on the same configuration, we don't have to wait until the first 
user's action are completed to run the second user's actions and there aren't 
any possible collisions. Swarming should allocate a new machine for the second 
user to run actions (but maybe should use the original machine if it's 
available before the machine provider allocates the new one).

Need I say more? :-)

When tests that run through Swarm fail, the user of those tests will most 
likely want to debug those failures.  Moreover, the user may not have easy 
access to the particular machine configuration on which the failure manifests.

There's a great opportunity here to make things really easy for the user.  
Suppose that Swarm automatically requests a machine instance from the machine 
provider that has the relevant configuration, then sets up that instance with 
the associated test data, so the tests are all ready to be run.  Then Swarm can 
provide the machine connection details to the user (e.g., the IP address of the 
instance), and the user can simply log into the machine and immediately start 
running the tests to repro/debug the failure.

This could be a compelling advantage of Swarm for configuration testing - 
making it really easy to debug failures on different configs that the user may 
not otherwise have easy access to.  However, there are important considerations 
to be made, such as whether there are enough machines available to be used for 
this purpose in the machine provider, how long a user should have access to a 
machine, and how to release the machine back to the provider when the user is 
done.

There are now 3 different variables controlling what is visible on the main 
page (show_success, sort_by and page). 

Currently changing anyone of these directly (through the page) results in the 
other two getting clobbered. This should be fixed so they can all work together 
nicely.

Currently, we validate a bit loosely the content of the output_destination 
dictionary in the test message request object. We should add better validation 
and also accept any type that could successfully convert to int/float.

The swarming tests use self.testbed.init_all_stubs() which is known to be a tad 
slower than just enabling the stubs that are actually needed.

So remove this call and replace it with only the subset of stubs that are 
needed for all unit tests under services/swarming/

We currently have this API, for which all entry points receive a test 
identifier key.

- show_message
 which returns the whole test case swarm file that was used to generate this test.

- get_result
 which returns the result that were sent to the resul_url from the local test runner.

- cancel
 which cancels a pending test so that it doesn't get sent to a machine next time around if one is available.

- retry
 which resends the test to a machine to try it again.

  We should clean it up a bit so that it's easier to use programatically and not just from the server front-end, and add more functionality to it like updating a test or interrupt it in flight, not just cancel pending ones, etc...

Remove the binaries attribute

We should remove the duality of data and binaries attributes. The TestCase and 
Configuration expose both data and binaries whereas the TestRun only has the 
data attribute.
Since we use both data and binaries for the same thing, might as well only have 
a data array and merge all the info in it (as we currently do when we create 
TestRuns from a TestCase).

Remove mentions of tests

Swarm can be used for sharding anything, not just tests, so might as well 
remove the testing terminology.

If a slave fails to PublishResults, the PublishInternalResults should send the 
results to a different location (probably related to the monitoring page) and 
have swarm retry the runner.

By having swarm retry the runner, we hide the internal failure from the user.