Base Livepeer Docker image
An Ubuntu 16.04 Docker image with Node.js 8.x, ffmpeg-static and IPFS installed. This image can be used as a base for other Livepeer related Docker images. verification-computation-archive currently uses this image as a base.
This image can be found on Docker Hub.
We should do what the title says.
Also, we should link to their docs.
livepeer/monitoring that provides that Postgres url to GrafanaWe just got access to the necessary APIs for this... Surely there's a Prometheus exporter somewhere for this data? Would be super useful to track.
> kubectl top pod
NAME CPU(cores) MEMORY(bytes)
api-694d8f77b4-7hmhp 10m 75Mi
api-694d8f77b4-8k62l 7m 67Mi
api-694d8f77b4-wvl78 53m 72Mi
broadcaster-bulbasaur-5cb6d64fbb-hnzrl 70m 330Mi
broadcaster-charmander-7f78bcb5f4-kv486 108m 293Mi
broadcaster-squirtle-84d7df674b-452s6 6667m 297Mi
broadcaster1-554bd5c5c7-rc2m5 1m 13Mi
external-dns-85646cb9b-5rzg9 24m 19Mi
f1l1b0x-test-7768548988-77pvv 2m 64Mi
ivan-acb-test-74c87874cb-dtxc4 808m 156Mi
mainnet-orchtran-68c85f7686-w79t2 4m 25Mi
mainnet-prometheus-7699fdf5c8-qbwpx 21m 89Mi
nginx-ingress-controller-675b647f6c-dk4vs 523m 1524Mi
nginx-ingress-controller-675b647f6c-n9xd6 589m 1539Mi
nginx-ingress-controller-675b647f6c-p6pm5 474m 1568Mi
orchestrator-1070-cc65f96db-6kpgn 79m 948Mi
orchestrator-1070-cc65f96db-9bbz5 82m 902Mi
orchestrator-1070-dualcore-55446d8744-4d6sc 22m 1168Mi
orchestrator-1070-dualcore-55446d8744-97ntx 42m 921Mi
orchestrator-1070-dualcore-55446d8744-9d47d 5m 890Mi
orchestrator-1070-dualcore-55446d8744-9g8gk 6m 975Mi
orchestrator-1070-dualcore-55446d8744-b46dw 30m 1328Mi
orchestrator-1070-dualcore-55446d8744-bkfhm 1m 9Mi
orchestrator-1070-dualcore-55446d8744-bmdsp 12m 901Mi
orchestrator-1070-dualcore-55446d8744-cfxmj 59m 752Mi
orchestrator-1070-dualcore-55446d8744-dghrz 5m 842Mi
orchestrator-1070-dualcore-55446d8744-dghs9 34m 756Mi
orchestrator-1070-dualcore-55446d8744-dm8sm 29m 751Mi
orchestrator-1070-dualcore-55446d8744-dmxqq 31m 957Mi
orchestrator-1070-dualcore-55446d8744-gq4cw 90m 1175Mi
orchestrator-1070-dualcore-55446d8744-kbmpm 204m 852Mi
orchestrator-1070-dualcore-55446d8744-khczx 294m 1107Mi
orchestrator-1070-dualcore-55446d8744-lgvmx 35m 901Mi
orchestrator-1070-dualcore-55446d8744-lt487 197m 848Mi
orchestrator-1070-dualcore-55446d8744-m74dt 5m 719Mi
orchestrator-1070-dualcore-55446d8744-ntfpn 162m 995Mi
orchestrator-1070-dualcore-55446d8744-pnr82 5m 994Mi
orchestrator-1070-dualcore-55446d8744-q9r94 214m 739Mi
orchestrator-1070-dualcore-55446d8744-ql4h5 116m 821Mi
orchestrator-1070-dualcore-55446d8744-rtq2t 5m 735Mi
orchestrator-1070-dualcore-55446d8744-s4qv9 1m 8Mi
orchestrator-1070-dualcore-55446d8744-txfm4 81m 820Mi
orchestrator-1070-dualcore-55446d8744-v5mqf 82m 814Mi
orchestrator-1070-dualcore-55446d8744-v8gl2 152m 909Mi
orchestrator-1070-dualcore-55446d8744-w2vs5 86m 755Mi
orchestrator-1070-dualcore-55446d8744-xk59t 8m 983Mi
orchestrator-1070-dualcore-55446d8744-xlkdf 148m 763Mi
orchestrator-1070-dualcore-65f7bc8d9d-x4ss6 2000m 7077Mi
orchestrator-1080-7b9b9c8948-f9lcg 163m 1560Mi
orchtran1-76c66958f8-6wvsm 1m 7Mi
prometheus-7cd759bbbf-llx46 80m 133Mi
rinkeby-orchtran-7bd9976679-jltx6 4m 15Mi
rinkeby-prometheus-c5dc6969d-n7kkq 17m 78Mi
First alert is the query that detects MistProcLivepeer excessive restarts, indicating (maybe) a bframe or audio timestamp issue.
lvl=eror msg="Failed to read plugin provisioning files from directory" logger=provisioning.plugins path=/etc/grafana/provisioning/plugins error="open /etc/grafana/provisioning/plugins: no such file or directory"
Getting that command after running the example standalone command:
docker run --net=host --env LP_MODE=standalone --env LP_NODES=localhost:7935 livepeer/monitoring:latest
To be filled in.
Ideally, we can use the most recent version of geth, but at the very least we should be using a post-Constantinople version.
Fails to start Grafana with the following error:
service init failed: Alert notification provisioning error: alert validation error: Could not find webhook url property in settings
My docker-compose config:
services:
livepool_monitor:
container_name: livepool_monitor
image: livepeer/monitoring:latest
#network_mode: host
ports:
- '9191:9090'
- '3001:3000'
environment:
LP_MODE: 'standalone'
LP_NODES: '127.0.0.1:7935'
LP_PROMETHEUS_ENDPOINT: 'http://localhost:9191'
volumes:
- ./livepool_monitoring/log:/var/log/grafana
- ./livepool_monitoring/data:/data
- ./livepool_monitoring/data/plugins:/etc/grafana/provisioning/plugins # hack to pass issue #83In particular what I'm looking for is P50, P90, and P95 segment delivery times
Could we integrate Loki? I really took a liking to it recently. Thank you ! 🙏
We have an orchestrator overview dashboard now which shows orchestrator specific data. We should create a broadcaster overview dashboard as well that shows broadcaster specific data.
@jailuthra and @yondonfu will pair on this after the metric as added to go-livepeer.
/config-generator/src/supervisord.js:17
obj['program:loki'].command = `/usr/bin/loki --config.file=/etc/loki/loki.yaml`
^
TypeError: Cannot set properties of undefined (setting 'command')
at Object.generate (/config-generator/src/supervisord.js:17:33)
at generate (/config-generator/src/generate.js:143:41)
at Object.<anonymous> (/config-generator/src/generate.js:160:1)
at Module._compile (node:internal/modules/cjs/loader:1097:14)
at Object.Module._extensions..js (node:internal/modules/cjs/loader:1149:10)
at Module.load (node:internal/modules/cjs/loader:975:32)
at Function.Module._load (node:internal/modules/cjs/loader:822:12)
at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
at node:internal/main/run_main_module:17:47
Missing data in the Grafana dashboard when running the container in standalone mode with a local orchestrator. Nothing shows up in any of the orchestrator related panels.
I think this is due to the Prometheus expressions used in the dashboard template. For example, this line uses a regex to match for a livepeer_node_type label. But, that label only exists in a Kube environment where Prometheus creates that label based on a Kube service name.
There is currently a percentile latency alert supported by monitoring. In most cases, the ratio between segment duration and round trip latency is more relevant because the absolute round trip latency can vary depending on the segment duration. There should be a ratio alert added.
Currently, the Grafana dashboard uses the livepeer_transcode_latency_seconds metric in latency focused graphs. This metric represents the transcode latency from the time the segment is produced from the segmenter until one of the transcoded renditions is received. This metric does not capture latency induced by additional steps after a rendition is received (i.e. waiting for all renditions to be received before playlist insertion and verification when running in on-chain mode). I think using the livepeer_overall_transcode_latency_seconds metric would capture this information.
The geth-with-protocol image is currently built using geth v1.9.11. As of geth v1.9.15, decoded contract revert reasons are returned so upgrading to this version will be helpful for debugging contract reverts.
This works better: sum(rate(livepeer_orchestrator_swaps[1m]))
Or we could do one with sum() and one without sum to see both
Would be nice to see how many we were handling over time.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.