seattletestbed / integrationtests Goto Github PK

We have a DynDNS-like service, Zenodotus, running on blackbox.poly.edu. We need an integration test to ensure that the zenodotus service is up and working correctly.

More information on the Zenodotus service can be found here:

• https://seattle.poly.edu/wiki/Zenodotus
• https://seattle.poly.edu/wiki/zenodotus_server

It seems that there are some integration tests that have been failing on blackbox, but no notification has been going out. We need to update these tests to ensure that they all have the right credentials to send email notifications and also ensure they are working correctly.

One test in particular that I found was not sending out notification is the time_servers_running test, that checks to make sure that there are sufficient time servers up and running. The cron log shows that the test notified the people on the notification list, but I don't seem to have received these emails.

When running the retrieve_url("https://selexor.poly.edu:8888/") from selexor_active.py the following error occurs:
Traceback (most recent call last):
File "/home/john/a.py", line 29, in
main()
File "/home/john/a.py", line 6, in main
retrieve_url("https://selexor.poly.edu:8888/")
File "/home/john/a.py", line 12, in retrieve_url
download_process = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
File "/usr/lib/python2.7/subprocess.py", line 679, in init
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

The current integration tests are structured in such a way that every integration test has its own set of repy dependencies. This makes updating the e-mail list a pain, as well as makes the directories for each test case very cluttered.

We should reorganize it so that the integration tests share the repy dependencies (files generated by preparetest.py). Also, we should only include the actual test cases in their own directories as well.

I propose having an integration test that will pull nodemanager logs from our nodes (softwareupdater.old and nodemanager.old, *.new files if they exist as well) periodically, to potentially uncover issues that we may not be aware of. Monzur mentioned he had a script that was used for this purpose before, so if possible we should get that integrated into our monitoring test suite and into our repository.

We should create a test that checks the time needed to talk to the (publicly run) NTP servers we use in our NTP library.

This is related to / motivated by SeattleTestbed/attic#1363.

The test_time_tcp integration test checks (among other things) if the replies it gets from querying multiple time servers are within a reasonable time interval. This is implemented by comparing the difference between maximum and minimum received timestamp with the time elapsed between the first and the last server contact. The idea is that if the server clocks and local clock are progressing at the same rate, and the server clocks are set correctly, then the difference and time elapsed should be identical (up to some slack).

The current implementation fails to send notifications if the slack becomes excessive, because it references undeclared variables:

https://github.com/SeattleTestbed/integrationtests/blob/master/time/test_time_tcp.py#L88

I get this error when trying to run the time server integration test. It seems like it terminates before it can send out an error e-mail.

integrationtester@blackbox:~/cron_tests/timeserver_tests$ python test_time_servers_running.py
Tue Jan  7 12:50:34 2014 : Looking up time_servers
Traceback (most recent call last):
  File "test_time_servers_running.py", line 44, in <module>
    main()
  File "test_time_servers_running.py", line 34, in main
    servers = centralizedadvertise_lookup("time_server")
  File "/home/integrationtester/cron_tests/timeserver_tests/centralizedadvertise_repy.py", line 52, in centralizedadvertise_lookup
    sockobj = timeout_openconn(servername,serverport, timeout=10)
  File "/home/integrationtester/cron_tests/timeserver_tests/sockettimeout_repy.py", line 95, in timeout_openconn
    tsock.connect((desthost, destport))
  File "/home/integrationtester/cron_tests/timeserver_tests/sockettimeout_repy.py", line 203, in connect
    self._openconn()
  File "/home/integrationtester/cron_tests/timeserver_tests/sockettimeout_repy.py", line 277, in _openconn
    self.sockobj = openconn(destip, destport)
  File "/home/integrationtester/cron_tests/timeserver_tests/emulcomm.py", line 1339, in openconn
    comminfo[handle]['socket'].connect((desthost,destport))
  File "<string>", line 1, in connect
socket.gaierror: (-2, 'Name or service not known')

The following needs to be done:

• Make integration test runnable
• Report all exceptions via e-mail notification

Currently, the test_lookup_node_states integration test only queries the "central" advertise server, but not central_v2, or the UDP central advertise server. Therefore, if nodes fail to advertise on centraladvertise.poly.edu (which isn't totally unlikely given that it is a single-threaded server that has issues under load, see SeattleTestbed/advertiseserver#22) the integration tests will under-count the number of nodes available, and send out notifications.

(However, we'll need to verify that the nodes "missing" from the central advertise server really fail due to server load, and don't have other issues.)