Service pipeline stops serving static files after new data committed about pachyderm HOT 9 OPEN

An immediate workaround would be to exec python app.py, though we tested that internally and didn't get that to work in every case. It worked with my random version of python3 and flask, but not with someone else's.

The problem here is that we send SIGKILL to bash, not python, and bash just leaves python running when it gets SIGKILL. With exec python, bash "turns into" python, and then we kill that. (But if python spawns additional processes to handle HTTP requests, we can't kill those.)

Longer term, PR #8385 fixes this problem in general. I'll let you know when that's available to test in a nightly release.

from pachyderm.

My fix is in v2.5.0-nightly.20221130, which you can install with something like helm install my-pachyderm pachyderm/pachyderm --version 2.5.0-nightly.20221130. Don't use the nightly on your production cluster or anything, you won't be able to downgrade from the nightly build to the stable release. (You might also need a new version of pachctl, which you can grab from here: https://github.com/pachyderm/pachyderm/releases/tag/v2.5.0-nightly.20221130)

I've tested it with a scenario similar to yours, so I think it should fix things, but I'm interested in actual real world feedback of course 😂

from pachyderm.

Thanks @dsgibbons for submitting this. I'll talk to the team and we will get back to you shortly.

from pachyderm.

Great, thank you very much for that @jrockway. I'll try the short-term fix and see if that helps. I look forward to testing your PR.

from pachyderm.

Looks good. I'll raise this with my team and see if we can test this fix next week. Thank you!

from pachyderm.

That is good to hear. Feel free to @jrockway if you have anything to report!

from pachyderm.

@jrockway We tried pulling the latest helm chart version, but it only seems to go up to 2.4.1 (which we currently have installed). Is your public helm chart updated to include nightly builds? Are the pachd container images for nightly builds public? If so, where can we find them?

from pachyderm.

We update the helm chart and Dockerhub with prerelases. Apparently the chart version contains the commit id, so 2.5.0-nightly.20221205-1f8686882d20250a137667d7a13027e6198da5f5 instead of just 2.5.0-nightly.20221205 is required. "helm repo search pach/pachyderm --devel" will show the latest prerelease. (But the name is too long to copy-paste out of that output, so that's fun.)

I don't know why that's the case. Probably a mistake on our end.

from pachyderm.

Ok I've finally gotten around to testing this fix. It works in the sense that the service pipeline doesn't go down if a new datum is added. Unfortunately, the new datum is not recognised by the service pipeline - a restart of the service pipeline is required to locate the new datum in the static/ folder. Perhaps symlinking to static/ is not the best way to have the service pipeline update to serve new artefacts? Do you have any suggestions?

from pachyderm.