edgurgel / verk_web Goto Github PK
View Code? Open in Web Editor NEWA dashboard for the job processing system that just verks! 🧛
Home Page: https://hex.pm/packages/verk_web
License: MIT License
A dashboard for the job processing system that just verks! 🧛
Home Page: https://hex.pm/packages/verk_web
License: MIT License
Not sure if this is an issue with my setup but trying to run the master
branch of VerkWeb locally doesn't work for me:
$ mix deps.get
$ npm install && bower install
$ mix phoenix.server
[info] GET /
[debug] Processing by VerkWeb.PageController.index/2
Parameters: %{}
Pipelines: [:browser]
[info] Sent 500 in 12ms
[error] #PID<0.2415.0> running VerkWeb.Endpoint terminated
Server: localhost:4000 (http)
Request: GET /
** (exit) an exception was raised:
** (ArgumentError) argument error
(stdlib) :ets.select(:queue_stats, [{{:"$1", :"$2", :"$3", :"$4", :_, :_}, [], [{{:"$1", :"$2", :"$3", :"$4"}}]}])
(verk) lib/verk/queue_stats_counters.ex:23: Verk.QueueStatsCounters.all/0
(verk) lib/verk/queue_stats.ex:22: Verk.QueueStats.all/0
(verk_web) web/controllers/page_controller.ex:6: VerkWeb.PageController.index/2
(verk_web) web/controllers/page_controller.ex:1: VerkWeb.PageController.action/2
(verk_web) web/controllers/page_controller.ex:1: VerkWeb.PageController.phoenix_controller_pipeline/2
(verk_web) lib/phoenix/router.ex:261: VerkWeb.Router.dispatch/2
(verk_web) web/router.ex:1: VerkWeb.Router.do_call/2
(verk_web) lib/verk_web/endpoint.ex:1: VerkWeb.Endpoint.phoenix_pipeline/1
(verk_web) lib/plug/debugger.ex:123: VerkWeb.Endpoint."call (overridable 3)"/2
(verk_web) lib/phoenix/endpoint/render_errors.ex:34: VerkWeb.Endpoint.call/2
(plug) lib/plug/adapters/cowboy/handler.ex:15: Plug.Adapters.Cowboy.Handler.upgrade/4
(cowboy) src/cowboy_protocol.erl:442: :cowboy_protocol.execute/4
Is there something else I need to do to run this in development?
Move queue/:queue
to queue/:queue/busy
and put enqueued jobs on queue/:queue
. Use recently added Verk.Queue
to list (edgurgel/verk#18)
Since Endpoint is all macros it checks :authorization
config during verk_web compile time. Since compiling dependency happens only once, changing config after dependencies have been compiled has no effect. I think the reasonable solution would be to have a plug that checks the config on init. BasicAuth does exactly that, but because of https://github.com/edgurgel/verk_web/blob/master/lib/verk_web/endpoint.ex#L39 it is not even loaded.
Is there a way to customize mounting options? Trying to integrate it into phoenix app.
Would be nice to protect Verk UI with simple http auth or something
In exq_ui i can simply add necessary stuff via pipelines:
https://github.com/akira/exq_ui#using-with-plug
GenStage architecture was implemented in the Verk 0.13.6
It's hard to know how long ago a job was started/enqueued by looking only the UTC time.
Related to edgurgel/verk#49
It should have something like: "This job is likely gone (aka processed)"
Still trying to triage this, but upgrading to 0.13.3 breaks my application by preventing startup.
Startup crash log: https://gist.github.com/keyan/29f1c503cc3de1865bbc9531e33692e8
My mix.lock
: https://gist.github.com/keyan/3d372b125579e684116a5f0f9713eb09
I might have to make some application changes on my end in order to work with the new version. At the very least I think the changes were large enough and added additional features and so warrant a minor version bump.
Initially reported here: #62
We should be able to clear a list of enqueued jobs for a queue.
Hi! Love Verk. Question. Is there a way to mount the verk route under a protected scope as opposed to using basic auth? We use pow to protect routes and would like to put the /verk route under an authenticated scope instead if basic auth.
something like:
scope "/", do
pipe_through [:browser, :protected]
forward("/verk", VerkWeb.MountRoute)
end
It would be interesting to know how many workers this instance has and how many are idle.
Let's say I have 2 nodes, node_id: "1"
and node_id: "2"
running on the same redis server.
Currently, the "Failed", "Enqueued", "Scheduled" jobs properly list all jobs in the cluster (that is, jobs saved in Redis, no matter on what node is the VerkWeb being run).
Unfortunately, the "Running jobs" count and list lists only jobs running in the given node.
It would be cool if it would list all the jobs that are inprogress
in all currently living nodes.
Possibly related to edgurgel/verk#157 ?
P.S. Until we implement it, we could mention it in the README and/or in VerkWeb interface.
Sometimes you want to select all jobs and then unselect one. This can be quite tedious 😢
Maybe a checkbox placed inside the table header?
This is more to start a discussion if this is a wanted feature.
Here is my current implementation, that so far looks promising...
The inspiration came from:
this problem is not properly solved yet:
What was done:
It depends on my fork of Verk atm, that's why it makes no sense to open a pull request...
@edgurge what are your thoughts about this feature?
Hi!
So we have an issue, which is, we currently run our Elixir app on Heroku (not for much longer but we are currently stuck) and we want to use Heroku workers to handle the job load. However, it seems I can't configure our "web" app to not accept worker requests but still host verk_web
on the same app.
We have the dashboard working but it shows no queues in the web interface because we have used the VERK_DISABLED example in the docs to disable it on our API instance. The problem is that we need to host the verk_web
panel on the API instance and host the workers on separated machines but since we return []
for no queues, the dashboard thinks there is nothing there. We definitely don't want the API app to act as a worker.
Do you have any advice???
Failed to use "cowboy" (version 2.6.1) because
plug_cowboy (version 2.0.1) requires ~> 2.5
verk_web (versions 1.0.0 to 1.4.2) requires ~> 1.0
mix.lock specifies 2.6.1
Hi,
The newest release of phoenix (including cowboy2) no longer has the method Phoenix.Router.Route.forward/4
. I receive the following error trying to redirect to /verk
.
function Phoenix.Router.Route.forward/4 is undefined or private
It is happening inside the plug/mount.ex
file in the defp forward
method.
EDIT: For now, I got around it by adding this to my router directly.
scope "/", VerkWeb do
forward("/verk", Endpoint)
end
I have verk_web configured as a separate app in our umbrella project (not mounted in any phoenix apps).
We have one proxy app that kindof plugs requests to the other phoenix endpoints.
Shortened:
defmodule ProxyWeb.Endpoint do
@domains %{
"admin.app1.com" => AdminWeb.Endpoint,
"verk.app1.com" => VerkWeb.Endpoint
}
def init(opts), do: opts
def call(conn, _) do
with domain <- get_domain(conn),
endpoint <- Map.get(@domains, domain, @default_host) do
endpoint.call(conn, endpoint.init(:supervisor, endpoint))
end
end
end
VerkWeb is totally fine here, except for websockets:
[info] Converted error Phoenix.Router.NoRouteError to 404 response
[info] GET /socket/websocket
I'm not quite sure what I should do from here, but happy to provide more detail or even PR if we find it's a real issue.
I have several instances running a Verk-backed application and I’d like to see the stats for the machines in aggregate. I’m having a couple issues.
Is there a way to run a single VerkWeb node that gets stats for all the processors? When I tried running VerkWeb on it’s own I get an argument error:
18:36:07.483 [error] #PID<0.337.0> running VerkWeb.Endpoint terminated
Server: localhost:4000 (http)
Request: GET /
** (exit) an exception was raised:
** (ArgumentError) argument error
(stdlib) :ets.select(:queue_stats, [{{:"$1", :"$2", :"$3", :"$4", :_, :_}, [], [{{:"$1", :"$2", :"$3", :"$4"}}]}])
(verk) lib/verk/queue_stats_counters.ex:23: Verk.QueueStatsCounters.all/0
(verk) lib/verk/queue_stats.ex:22: Verk.QueueStats.all/0
(verk_web) web/controllers/page_controller.ex:6: VerkWeb.PageController.index/2
(verk_web) web/controllers/page_controller.ex:1: VerkWeb.PageController.action/2
(verk_web) web/controllers/page_controller.ex:1: VerkWeb.PageController.phoenix_controller_pipeline/2
(verk_web) lib/phoenix/router.ex:261: VerkWeb.Router.dispatch/2
(verk_web) web/router.ex:1: VerkWeb.Router.do_call/2
Unfortunately it looks like VerkWeb
relies on the :queue_stats
ets
table to get queue specific stats, rather than using the data in Redis. The “All-time stats” numbers are coming from Redis and are reporting correctly, but queue specific stats are only showing jobs processed by the node on which that VerkWeb instance is running. Is this expected behavior?
The inspect feature makes no sense for jobs that are not running (scheduled, retries, enqueued)
I have the following code in endpoint.ex
:
defmodule App.Endpoint do
use Phoenix.Endpoint, otp_app: :app
if Application.get_env(:verk_web, :authorization) do
IO.puts("+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")
IO.inspect(Application.get_env(:verk_web, :authorization))
use VerkWeb.Mount, path: "/admin/verk"
IO.puts("+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")
else
raise "Verk authorization"
end
If I execute this in dev
it ask for authorization. When I compile to prod
I can see this message:
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[realm: "Admin", username: "admin", password: "password"]
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
If I execute a remote console in the prod
server I can get Application.get_env(:verk_web, :authorization)
correctly. However I does not ask for authorization. I don't have more ideas to solve this.
I use AWS LoadBalancer and Docker with AWS ECS.
Any help it is very welcome :)
Thanks in advance.
Error occurs on application startup
Error log:
02:15:10.347 [info] Application verk_web exited: VerkWeb.start(:normal, []) returned an error: shutdown: failed to start child: VerkWeb.Endpoint
--
| | ** (EXIT) an exception was raised:
| | ** (UndefinedFunctionError) function Plug.Cowboy.child_spec/1 is undefined (module Plug.Cowboy is not available)
| | Plug.Cowboy.child_spec([scheme: :http, plug: {VerkWeb.Endpoint, []}, options: [dispatch: [_: [{:_, Phoenix.Endpoint.Cowboy2Handler, {VerkWeb.Endpoint, []}}]], port: 4000, otp_app: :verk_web]])
| | (phoenix 1.4.17) lib/phoenix/endpoint/cowboy2_adapter.ex:44: Phoenix.Endpoint.Cowboy2Adapter.child_spec/3
| | (phoenix 1.4.17) lib/phoenix/endpoint/supervisor.ex:108: anonymous fn/6 in Phoenix.Endpoint.Supervisor.server_children/4
| | (elixir 1.10.4) lib/enum.ex:2111: Enum."-reduce/3-lists^foldl/2-0-"/3
| | (phoenix 1.4.17) lib/phoenix/endpoint/supervisor.ex:99: Phoenix.Endpoint.Supervisor.server_children/4
| | (phoenix 1.4.17) lib/phoenix/endpoint/supervisor.ex:59: Phoenix.Endpoint.Supervisor.init/1
Elixir 1.10.4
relevant mix.exs:
def application do
[
applications: [
:verk,
:verk_web,
]
]
end
def deps do
{:verk, "~> 1.7.1"},
{:verk_web, "~> 1.5.0"}
end
relevant releases.exs:
config :verk_web, VerkWeb.Endpoint,
http: [port: System.get_env("PORT", "4000") |> String.to_integer()],
url: [host: System.get_env("URL_HOST", "localhost")],
server: true,
pubsub: [name: VerkWeb.PubSub, adapter: Phoenix.PubSub.PG2]
I've tried several things, but nothing really works. It does seem like some sort of a dependency problem, but I really can't figure it out. Seems like this error is related to phoenix 1.4, but since I don't have it as an explicit dependency I'm not sure what to do. Any help will be greatly appreciated. Thanks in advance :)
Internal links in the VerkWeb are behaving weirdly, see screenshot:
page_path(@conn, :index)
-> /
(incorrect)mount_static_path(@conn, "/js/app.js")
-> /verk/js/app.js
(correct)Details:
mounted into endpoint via
defmodule App.Web.Endpoint do
use Phoenix.Endpoint, otp_app: :app
use VerkWeb.Mount, path: "/verk"
...
I have an older project with phoenix 1.2 and verk 0.13.5 and everything works as expected.
git diff between these versions don't show any related changes so it's probably behavior change on phoenix/plug side.
Did anyone encounter something similar? Thoughts on debugging?
P.S. related - https://github.com/edgurgel/verk_web/blob/master/web/views/layout_view.ex#L5 - that's why mount_static_path
works well.
Similar to edgurgel/verk#54
Hey Ed,
When you enqueue or schedule a job you get back a job id that's a big long number, however, there doesn't look like there is a way to delete a job by that ID. My ideal case is to store that number in my db table so I can delete the future scheduled job if I need to.
I had a look in verk_web, found this:
. which removes a job when you pass in the whole job, on a queue named 'schedule'.There was also a delete_job function I found, but that required the original json, not just the ID.
Appreciate advice on where to go next.
Thanks!
Similar to Retries
It would be neat to add a Dockerfile and publish to docker hub a standalone verk web server.
Using x from now
format.
#84 here is the PR
Similar implementation to Retries
Hello,
It seems it does not work for the newest versions of Phoenix:
Failed to use "phoenix" (version 1.3.0) because
phoenix_live_reload (version 1.1.1) requires ~> 1.0
verk_web (version 1.1.0) requires ~> 1.2.1
mix.exs specifies ~> 1.3.0
Any plan or solutions about this?
Thanks for your time
Add a retry button so we can retry instantly some jobs that failed
Hello,
Thanks for writing this very useful library!
I have a use case where I want to manually run the scheduled job earlier than the scheduled time through the interface.
I imagine a button next to the details button on the scheduled jobs tab. This would have a javascript "Are you sure" alert and then just enqueue the scheduled job immediately.
I'd be interested in writing a PR for this if you think it would be something you'd want to add.
When running the phoenix server. I see plenty of those messages:
[warn] Ignoring unmatched topic "rooms:jobs" in xxx.UserSocket
As a beginner to elixir phoenix, some help would be awesome.
Thanks
I think it would be really useful to see some stats around "dead" (as Verk calls them) jobs on VerkWeb. As of now, the only job failure stats we get are related to failed attempts, even when subsequent retries may succeed. It is difficult to tell how many job failures ended with the job reaching the max retry count and dying.
It is not entirely clear to me how difficult it would be but it very helpful to have another tab, "Dead", that has similar data as the "Retries" tab.
** (ArgumentError) lists in Phoenix.HTML and templates may only contain integers representing bytes, binaries or other lists, got invalid entry: {:a, "b"}
https://github.com/edgurgel/verk_web/blob/master/web/templates/dead/index.html.eex#L66
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.