Local development: make triage and cleanup cron more reliable (not in production) about clusterfuzz HOT 13 CLOSED

Do you see any files in local/storage/local_gcs/test-blobs-bucket/objects/. if yes, that confirms those testcase files got saved. If this does not happen, some issue with your port forwarding setup.

if you see files, next thing to try is go to http://localhost:9002/cron, then scroll to bottom and run triage cron manually. wait a minute or two, then refresh http://localhost:9000 to see if testcase came there.

from clusterfuzz.

This crashing testcase didn't appear in "Testcases" page. Is this expected behavior (uploaded case won't add into testcase collections) ?

No. When you go to locahost:9000/upload-testcase do you see the testcase you uploaded? What happens when you click on it? That same testcase isn't showing up when you go to localhost:9000/testcsases?

from clusterfuzz.

Do you see any files in local/storage/local_gcs/test-blobs-bucket/objects/. if yes, that confirms those testcase files got saved. If this does not happen, some issue with your port forwarding setup.

I saw lots of files in local/storage/local_gcs/test-blobs-bucket/objects/ but the testcase I'm interested in is not there. Are those objects minimized testcase, unminimized, or both?

Because my bots run on localhost, so it should be not related to port forwarding.

I will try triage cron. However, they didn't appear for many hours. I guess it won't help.

No. When you go to locahost:9000/upload-testcase do you see the testcase you uploaded? What happens when you click on it? That same testcase isn't showing up when you go to localhost:9000/testcsases?

Right, I can see my uploaded testcase in /upload-testcase and the bots have already verified they crash. Some of them are labeled "Confirmed" and some of them are labeled "Duplicated".
However, they don't show in /testcases page.

from clusterfuzz.

oops, they appear after triage cron manually!

from clusterfuzz.

@kcwu - how long was your run_server running. did you see any stack that can tell how triage cron died ? can you start up a fresh run_server instance with --bootstrap, and retry just to see if triage cron can go down again ?

from clusterfuzz.

My run_sever has run for more than 20 hours (before #177). The console log is longer than my terminal scrollback history (about 2.5 hours), so I don't have stack. Is the server log written to disk somewhere?

From existing console log, there is only one "GET /triage" request in past 2.5 hours, which is my manual one. In other words, the cron stopped for some reason.

I guess #177 make the cron down.

from clusterfuzz.

More info, there are no "cron-service" messages in server's console log in past 2.5 hours. So it's not only triage cron down, it's cron fully stopped.

After server restarted, I can see "cron-service" messages in sever's console log again.

from clusterfuzz.

can you leave the server running for another 20 hours with fuzzing ongoing. lets see if triage goes down for you. you can check triage cron running in run_server log on console.

from clusterfuzz.

Closing for now since #177 is unsupported usecase for local. If you can still reproduce it, we will reopen.

from clusterfuzz.

This reproducible.

1. restart server
2. there are several "cron-service: "GET /triage HTTP/1.1" 200 -" in console log after server start (about 1 per minutes)
3. one /triage 503 error
4. no more /triage requests (at least 1.5 hours). Since the /triage cron is scheduled as 1 per hour, I guess it's really dead.

| INFO     2019-02-14 02:16:47,515 module.py:861] cron-service: "GET /triage HTTP/1.1" 503 59
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "src/local/butler/run_server.py", line 106, in trigger
    response = urllib2.urlopen(request, timeout=request_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 503: Service Unavailable

BTW, similar error happened for /cleanup as well. That is,

1. server start
2. first /cleanup appear 1 minute after server start
3. the /cleanup 503, identical message as above
4. no more /cleanup

from clusterfuzz.

What caused the 503 ? Any explanation can help to fix.

from clusterfuzz.

@kcwu @mhlakhani - this should be fixed now. please comment back if you see any similar issues. locally, crons should continue to work now, even in case of error.

from clusterfuzz.

thanks @inferno-chromium! Will let you know here if I see it again.

from clusterfuzz.