The server started successfully and the webinterface is reachable. But we cannot generate anything, the on the webinterface given examples do not work. openllm query
did not work either.
`openllm query "Tell me the truth!"`
Timed out while connecting to localhost:3000:
Timed out waiting 30 seconds for server at 'localhost:3000' to be ready.
Traceback (most recent call last):
File "/home/openllm/openllm_environment/bin/openllm", line 8, in <module>
sys.exit(cli())
File "/home/openllm/openllm_environment/lib64/python3.9/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/home/openllm/openllm_environment/lib64/python3.9/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/home/openllm/openllm_environment/lib64/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/openllm/openllm_environment/lib64/python3.9/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/openllm/openllm_environment/lib64/python3.9/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/home/openllm/openllm_environment/lib64/python3.9/site-packages/openllm/cli.py", line 380, in wrapper
return func(*args, **attrs)
File "/home/openllm/openllm_environment/lib64/python3.9/site-packages/openllm/cli.py", line 353, in wrapper
return_value = func(*args, **attrs)
File "/home/openllm/openllm_environment/lib64/python3.9/site-packages/openllm/cli.py", line 328, in wrapper
return f(*args, **attrs)
File "/home/openllm/openllm_environment/lib64/python3.9/site-packages/openllm/cli.py", line 1345, in query
openllm[client.framework], # type: ignore (internal API)
File "/home/openllm/openllm_environment/lib64/python3.9/site-packages/openllm_client/runtimes/http.py", line 52, in framework
return self._metadata["framework"]
File "/home/openllm/openllm_environment/lib64/python3.9/site-packages/openllm_client/runtimes/base.py", line 102, in _metadata
return self.call("metadata")
File "/home/openllm/openllm_environment/lib64/python3.9/site-packages/openllm_client/runtimes/base.py", line 143, in call
return self._cached.call(f"{name}_{self._api_version}", *args, **attrs)
File "/home/openllm/openllm_environment/lib64/python3.9/site-packages/openllm_client/runtimes/base.py", line 151, in _cached
self._client_class.wait_until_server_ready(self._host, int(self._port), timeout=self._timeout)
File "/home/openllm/openllm_environment/lib64/python3.9/site-packages/bentoml/_internal/client/http.py", line 67, in wait_until_server_ready
raise TimeoutError(
TimeoutError: Timed out waiting 30 seconds for server at 'localhost:3000' to be ready.
Examples/readyz from the Webgui:
```python
2023-06-27T08:14:33+0200 [WARNING] [cli] No known supported resource available for <class 'types.OptRunnable'>, falling back to using CPU.
2023-06-27T08:14:34+0200 [INFO] [cli] Environ for worker 0: set CPU thread count to 16
2023-06-27T08:14:34+0200 [WARNING] [cli] No known supported resource available for <class 'types.OptRunnable'>, falling back to using CPU.
2023-06-27T08:14:34+0200 [INFO] [cli] Prometheus metrics for HTTP BentoServer from "_service.py:svc" can be accessed at http://localhost:3000/metrics.
2023-06-27T08:14:34+0200 [INFO] [cli] Starting production HTTP BentoServer from "_service.py:svc" listening on http://0.0.0.0:3000 (Press CTRL+C to quit)
2023-06-27T08:14:43+0200 [INFO] [api_server:llm-opt-service:16] 123.123.123.123:60081 (scheme=http,method=GET,path=/,type=,length=) (status=200,type=text/html; charset=utf-8,length=2859) 0.470ms (trace=9a0ba0a3ad009c89aebddd40fa94efa0,span=f5801bc407cca64c,sampled=1,service.name=llm-opt-service)
2023-06-27T08:14:44+0200 [INFO] [api_server:llm-opt-service:16] 123.123.123.123:60081 (scheme=http,method=GET,path=/static_content/swagger-ui.css,type=,length=) (status=200,type=text/css; charset=utf-8,length=143980) 6.529ms (trace=a044f5c38e8f8b15777f10f6a61e5e5b,span=d7759d1a32791968,sampled=1,service.name=llm-opt-service)
2023-06-27T08:14:44+0200 [INFO] [api_server:llm-opt-service:16] 123.123.123.123:60081 (scheme=http,method=GET,path=/static_content/index.css,type=,length=) (status=200,type=text/css; charset=utf-8,length=1125) 1.547ms (trace=13289048526d6668c8c59767576296b6,span=7480ec06dd86230a,sampled=1,service.name=llm-opt-service)
2023-06-27T08:14:44+0200 [INFO] [api_server:llm-opt-service:16] 123.123.123.123:60083 (scheme=http,method=GET,path=/static_content/swagger-initializer.js,type=,length=) (status=200,type=application/javascript,length=383) 1.395ms (trace=0f6357f294a61675d8b97a2d388116a6,span=e07593e79162735e,sampled=1,service.name=llm-opt-service)
2023-06-27T08:14:44+0200 [INFO] [api_server:llm-opt-service:16] 123.123.123.123:60081 (scheme=http,method=GET,path=/static_content/swagger-ui-bundle.js,type=,length=) (status=304,type=,length=) 0.877ms (trace=d076ac9766e7be7e4b142a7305daa54e,span=133caf1d609a19aa,sampled=1,service.name=llm-opt-service)
2023-06-27T08:14:44+0200 [INFO] [api_server:llm-opt-service:16] 123.123.123.123:60081 (scheme=http,method=GET,path=/static_content/swagger-ui-standalone-preset.js,type=,length=) (status=304,type=,length=) 0.881ms (trace=1d5078f1aee1173ee967829168d0ac01,span=0b4cd579c15c02f6,sampled=1,service.name=llm-opt-service)
2023-06-27T08:14:44+0200 [INFO] [api_server:llm-opt-service:16] 123.123.123.123:60081 (scheme=http,method=GET,path=/static_content/favicon-96x96.png,type=,length=) (status=200,type=image/png,length=5128) 3.940ms (trace=f141d17fbcbba97061c02f688df3659b,span=95f42383d374830e,sampled=1,service.name=llm-opt-service)
2023-06-27T08:14:44+0200 [INFO] [api_server:llm-opt-service:16] 123.123.123.123:60083 (scheme=http,method=GET,path=/static_content/favicon-32x32.png,type=,length=) (status=200,type=image/png,length=1912) 4.628ms (trace=766eb0b7b95b86767e67bafdc784f69f,span=8cd6fe4667b2d25d,sampled=1,service.name=llm-opt-service)
2023-06-27T08:14:44+0200 [INFO] [api_server:llm-opt-service:15] 123.123.123.123:60084 (scheme=http,method=GET,path=/docs.json,type=,length=) (status=200,type=application/json,length=8166) 16.775ms (trace=36ee0fd098a50c360588732a6eff754e,span=92bfc9de114573d5,sampled=1,service.name=llm-opt-service)
2023-06-27T08:14:57+0200 [INFO] [runner:llm-opt-runner:1] _ (scheme=http,method=GET,path=http://127.0.0.1:8000/readyz,type=,length=) (status=404,type=text/plain; charset=utf-8,length=9) 1.229ms (trace=825cdd8b43c097cf0fe9e7a89664d8c7,span=bf06eb4b65e76b31,sampled=1,service.name=llm-opt-runner)
2023-06-27T08:14:57+0200 [INFO] [api_server:llm-opt-service:16] 123.123.123.123:60109 (scheme=http,method=GET,path=/readyz,type=,length=) (status=503,type=text/plain; charset=utf-8,length=22) 77.489ms (trace=825cdd8b43c097cf0fe9e7a89664d8c7,span=023a43fb11c48835,sampled=1,service.name=llm-opt-service)
2023-06-27T08:26:51+0200 [INFO] [api_server:llm-opt-service:15] 123.123.123.123:60622 (scheme=http,method=POST,path=/v1/metadata,type=text/plain,length=4) (status=200,type=application/json,length=731) 5.742ms (trace=7214af978f9398a2d7b8a9feaebc215e,span=b9bf604323a0daa2,sampled=1,service.name=llm-opt-service)
2023-06-27T08:27:13+0200 [INFO] [api_server:llm-opt-service:15] 123.123.123.123:60651 (scheme=http,method=POST,path=/v1/metadata,type=text/plain,length=20) (status=200,type=application/json,length=731) 4.030ms (trace=e7360b8ce2bbd984fbe1a93dc0bc3b18,span=ac391d391a530d3b,sampled=1,service.name=llm-opt-service)
2023-06-27T08:27:59+0200 [INFO] [runner:llm-opt-runner:1] _ (scheme=http,method=GET,path=http://127.0.0.1:8000/readyz,type=,length=) (status=404,type=text/plain; charset=utf-8,length=9) 0.848ms (trace=3d67a35c7cae0ea522a4752ee78ad8c9,span=37de79502ec15faf,sampled=1,service.name=llm-opt-runner)
2023-06-27T08:27:59+0200 [INFO] [api_server:llm-opt-service:16] 127.0.0.1:58740 (scheme=http,method=GET,path=/readyz,type=,length=) (status=503,type=text/plain; charset=utf-8,length=22) 4.934ms (trace=3d67a35c7cae0ea522a4752ee78ad8c9,span=f05679dc029016fd,sampled=1,service.name=llm-opt-service)
2023-06-27T08:28:00+0200 [INFO] [runner:llm-opt-runner:1] _ (scheme=http,method=GET,path=http://127.0.0.1:8000/readyz,type=,length=) (status=404,type=text/plain; charset=utf-8,length=9) 0.706ms (trace=6feded7ab0a1ebb5850e8e07205137ce,span=545366f670aa6670,sampled=1,service.name=llm-opt-runner)
2023-06-27T08:28:00+0200 [INFO] [api_server:llm-opt-service:16] 127.0.0.1:58750 (scheme=http,method=GET,path=/readyz,type=,length=) (status=503,type=text/plain; charset=utf-8,length=22) 3.470ms (trace=6feded7ab0a1ebb5850e8e07205137ce,span=bfcf97403b38166f,sampled=1,service.name=llm-opt-service)
2023-06-27T08:28:01+0200 [INFO] [runner:llm-opt-runner:1] _ (scheme=http,method=GET,path=http://127.0.0.1:8000/readyz,type=,length=) (status=404,type=text/plain; charset=utf-8,length=9) 0.727ms (trace=56d62f117e9c36e4601564f3b4546e8b,span=80f35138b90a4d11,sampled=1,service.name=llm-opt-runner)
2023-06-27T08:28:01+0200 [INFO] [api_server:llm-opt-service:16] 127.0.0.1:58764 (scheme=http,method=GET,path=/readyz,type=,length=) (status=503,type=text/plain; charset=utf-8,length=22) 3.528ms (trace=56d62f117e9c36e4601564f3b4546e8b,span=2b0e2872be7a99f0,sampled=1,service.name=llm-opt-service)
2023-06-27T08:28:02+0200 [INFO] [runner:llm-opt-runner:1] _ (scheme=http,method=GET,path=http://127.0.0.1:8000/readyz,type=,length=) (status=404,type=text/plain; charset=utf-8,length=9) 0.692ms (trace=29a97e0ce492e2e54d0e47792360f091,span=2a83ddb3c9fec1f0,sampled=1,service.name=llm-opt-runner)
2023-06-27T08:28:02+0200 [INFO] [api_server:llm-opt-service:16] 127.0.0.1:58770 (scheme=http,method=GET,path=/readyz,type=,length=) (status=503,type=text/plain; charset=utf-8,length=22) 3.434ms (trace=29a97e0ce492e2e54d0e47792360f091,span=6b1596a336c1d33f,sampled=1,service.name=llm-opt-service)
2023-06-27T08:28:03+0200 [INFO] [runner:llm-opt-runner:1] _ (scheme=http,method=GET,path=http://127.0.0.1:8000/readyz,type=,length=) (status=404,type=text/plain; charset=utf-8,length=9) 0.698ms (trace=c63dee9e650759297b537c42d531bf9f,span=148454642c31920d,sampled=1,service.name=llm-opt-runner)
2023-06-27T08:28:03+0200 [INFO] [api_server:llm-opt-service:16] 127.0.0.1:58784 (scheme=http,method=GET,path=/readyz,type=,length=) (status=503,type=text/plain; charset=utf-8,length=22) 3.481ms (trace=c63dee9e650759297b537c42d531bf9f,span=3685ee5030f131ba,sampled=1,service.name=llm-opt-service)
2023-06-27T08:28:04+0200 [INFO] [runner:llm-opt-runner:1] _ (scheme=http,method=GET,path=http://127.0.0.1:8000/readyz,type=,length=) (status=404,type=text/plain; charset=utf-8,length=9) 0.703ms (trace=3d4f393d8e9d87c9489b80a2602c3d72,span=76da5185f188a13f,sampled=1,service.name=llm-opt-runner)
2023-06-27T08:28:04+0200 [INFO] [api_server:llm-opt-service:16] 127.0.0.1:58786 (scheme=http,method=GET,path=/readyz,type=,length=) (status=503,type=text/plain; charset=utf-8,length=22) 3.489ms (trace=3d4f393d8e9d87c9489b80a2602c3d72,span=1f55db959f3f0e8c,sampled=1,service.name=llm-opt-service)
2023-06-27T08:28:05+0200 [INFO] [runner:llm-opt-runner:1] _ (scheme=http,method=GET,path=http://127.0.0.1:8000/readyz,type=,length=) (status=404,type=text/plain; charset=utf-8,length=9) 0.688ms (trace=a79e3934f5765460d9453d5d22c70244,span=484c8de3837265e3,sampled=1,service.name=llm-opt-runner)
2023-06-27T08:28:05+0200 [INFO] [api_server:llm-opt-service:16] 127.0.0.1:58790 (scheme=http,method=GET,path=/readyz,type=,length=) (status=503,type=text/plain; charset=utf-8,length=22) 3.438ms (trace=a79e3934f5765460d9453d5d22c70244,span=6708a4fbbd796615,sampled=1,service.name=llm-opt-service)
2023-06-27T08:28:06+0200 [INFO] [runner:llm-opt-runner:1] _ (scheme=http,method=GET,path=http://127.0.0.1:8000/readyz,type=,length=) (status=404,type=text/plain; charset=utf-8,length=9) 0.693ms (trace=6ab794b317bb28b325ce65004f5adf95,span=bf26e25355405222,sampled=1,service.name=llm-opt-runner)
2023-06-27T08:28:06+0200 [INFO] [api_server:llm-opt-service:16] 127.0.0.1:58792 (scheme=http,method=GET,path=/readyz,type=,length=) (status=503,type=text/plain; charset=utf-8,length=22) 3.643ms (trace=6ab794b317bb28b325ce65004f5adf95,span=76c919c82c4d3ded,sampled=1,service.name=llm-opt-service)
2023-06-27T08:28:07+0200 [INFO] [runner:llm-opt-runner:1] _ (scheme=http,method=GET,path=http://127.0.0.1:8000/readyz,type=,length=) (status=404,type=text/plain; charset=utf-8,length=9) 0.728ms (trace=1f0affee83d5baa143853575680021b8,span=403ddab11fe64cdc,sampled=1,service.name=llm-opt-runner)
2023-06-27T08:28:07+0200 [INFO] [api_server:llm-opt-service:16] 127.0.0.1:58808 (scheme=http,method=GET,path=/readyz,type=,length=) (status=503,type=text/plain; charset=utf-8,length=22) 4.121ms (trace=1f0affee83d5baa143853575680021b8,span=b4981d0ffb1e79d5,sampled=1,service.name=llm-opt-service)
2023-06-27T08:28:08+0200 [INFO] [runner:llm-opt-runner:1] _ (scheme=http,method=GET,path=http://127.0.0.1:8000/readyz,type=,length=) (status=404,type=text/plain; charset=utf-8,length=9) 0.684ms (trace=79bee6a0c93b14dc8a0d52298af710a8,span=a2a6853a038532b1,sampled=1,service.name=llm-opt-runner)
2023-06-27T08:28:08+0200 [INFO] [api_server:llm-opt-service:16] 127.0.0.1:58820 (scheme=http,method=GET,path=/readyz,type=,length=) (status=503,type=text/plain; charset=utf-8,length=22) 3.425ms (trace=79bee6a0c93b14dc8a0d52298af710a8,span=63935df1dd7a4b6f,sampled=1,service.name=llm-opt-service)
2023-06-27T08:28:09+0200 [INFO] [runner:llm-opt-runner:1] _ (scheme=http,method=GET,path=http://127.0.0.1:8000/readyz,type=,length=) (status=404,type=text/plain; charset=utf-8,length=9) 0.713ms (trace=3a17ccfc409db03a76f4c1e99d2f7abd,span=bf36173e51d2114f,sampled=1,service.name=llm-opt-runner)
2023-06-27T08:28:09+0200 [INFO] [api_server:llm-opt-service:16] 127.0.0.1:42736 (scheme=http,method=GET,path=/readyz,type=,length=) (status=503,type=text/plain; charset=utf-8,length=22) 3.536ms (trace=3a17ccfc409db03a76f4c1e99d2f7abd,span=54f8a583b1e7f02f,sampled=1,service.name=llm-opt-service)
2023-06-27T08:28:10+0200 [INFO] [runner:llm-opt-runner:1] _ (scheme=http,method=GET,path=http://127.0.0.1:8000/readyz,type=,length=) (status=404,type=text/plain; charset=utf-8,length=9) 0.711ms (trace=08a437207ba29e28532062be8544627c,span=f66c358bbd8aa2eb,sampled=1,service.name=llm-opt-runner)