darthsim / hivemind Goto Github PK

View Code? Open in Web Editor NEW

972.0 8.0 29.0 1.26 MB

Process manager for Procfile-based applications

License: MIT License

Go 85.55% Shell 14.45%

procfile

hivemind's Introduction

Hivemind

Hivemind is a process manager for Procfile-based applications. At the moment, it supports Linux, FreeBSD, and macOS.

Procfile is a simple format to specify types of processes your application provides (such as web application server, background queue process, front-end builder) and commands to run those processes. It can significantly simplify process management for developers and is used by popular Platforms-as-a-Service, such as Heroku and Deis. You can learn more about the Procfile format here.

There are some good Procfile-based process management tools, including foreman by David Dollar, which started it all. The problem with most of those tools is that processes you want to manage start to think they are logging their output into a file, and that can lead to all sorts of problems: severe lagging, losing or breaking colored output. Tools can also add vanity information (unneeded timestamps in logs). Hivemind was created to fix those problems once and for all.

See this article for a good intro and all the juicy details! Introducing Overmind and Hivemind

Enter Hivemind

Hivemind uses pty to capture process output. That fixes any problem with log clipping, delays, and TTY colors other process management tools may have.

If you would like a process management tool with a lot of features, including tmux support, restarting and killing individual processes and advanced configuration, you should take a look at Hivemind's big brother — Overmind!

Installation

With Homebrew (macOS)

brew install hivemind

Download the latest Hivemind release binary

You can download the latest release here.

From Source

You need Go 1.11 or later to build the project.

$ GO111MODULE=on go get -u -f github.com/DarthSim/hivemind

Note: You can update Hivemind the same way.

Usage

Hivemind works with a Procfile. It may look like this:

web: bin/rails server
worker: bundle exec sidekiq
assets: gulp watch

To get started, you just need to run Hivemind from your working directory containing Procfile.

$ hivemind

If Procfile isn't located in your working directory, or named it non-standard as Procfile.dev, you can specify the path to it: [Fun Fact: Name of the Procfile is arbitrary and can be anything, although it is a best practice to name it as Procfile for sanity]

$ hivemind path/to/your/Procfile
$ hivemind path/to/your/Procfile.dev

Run hivemind --help to see other options. Note that every Hivemind option can be set with corresponding environment variable.

Environment

If you need to set specific environment variables before running a Procfile, you can specify them in the .env file in the current working directory. The file should contain variable=value pairs, one per line:

PATH=$PATH:/additional/path
PORT=3000
HIVEMIND_TITLE=my_awsome_app

Author

Sergey "DarthSim" Aleksandrovich

Highly inspired by Foreman.

Many thanks to @antiflasher for the awesome logo.

License

Hivemind is licensed under the MIT license.

See LICENSE for the full license text.

hivemind's People

Stargazers

Watchers

hivemind's Issues

Add vendoring workflow to README.md

Currently there's no vendoring workflow noted in the README.md

At a best guess the gvt tool seems compatible, but gvm is mentioned in the commit logs.

Could this be clarified?

Timeout when committing large files via gRPC in background jobs

We're looking at replacing foreman with Hivemind at GitHub and we're running into timeouts when trying to commit large files via gRPC in background jobs. Once we hit the timeout, Hivemind becomes unresponsive and we have to manually kill it. The jobs are able to commit small files so it seems like something about the file size is causing the timeout. Maybe there's a config setting or something similar we can change that will let the commit finish for large files?

We haven't had this issue with either foreman or Overmind.

Building on Apple Silicon

Anyone successful so far in building Hivemind with Apple Silicon (using Go 1.16.beta1/related issue)?

Using hive mind with overseer

https://github.com/jpillora/overseer

Overseer can run a process but also upgrade it by polling to an external server for new binaries. In so doing it does not drop connections because of its forking style. You read about this on the main page.

Hivemend seems to more of a process manager and so it seems that the two can be combined together really nicely.

You should not to change either library.

Raising this for discussion as I have tried this out and it seems to work.
Have not tried with overmind / tmux yet.

Fault creating process

Just upgraded to the latest and running within a Docker container built from either the official Node 8.12 image or the official Ruby 2.6 image, after installing Go, Hivemind faults on startup:

panic: runtime error: invalid memory address or nil pointer dereference
react_1  | [signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x402fee]
react_1  |
react_1  | goroutine 1 [running]:
react_1  | panic(0x565100, 0xc420010050)
react_1  | 	/usr/lib/go-1.7/src/runtime/panic.go:500 +0x1a1
react_1  | main.newProcess(0xc420010750, 0x3, 0xc420010755, 0xb, 0x2, 0xc4200105c0, 0x6, 0x1388, 0xc42000a800, 0xc4200105c0)
react_1  | 	/go/src/github.com/DarthSim/hivemind/process.go:32 +0x34e
react_1  | main.newHivemind(0x0, 0x0, 0x7ffedfba6edb, 0xc, 0x0, 0x0, 0xc4200105c0, 0x6, 0x1388, 0x64, ...)
react_1  | 	/go/src/github.com/DarthSim/hivemind/hivemind.go:52 +0x335
react_1  | main.main.func1(0xc42009c000, 0xc42009c000, 0xc420035a87)
react_1  | 	/go/src/github.com/DarthSim/hivemind/main.go:67 +0x16d
react_1  | github.com/DarthSim/hivemind/vendor/gopkg.in/urfave/cli%2ev1.HandleAction(0x55e720, 0xc42000a1a0, 0xc42009c000, 0x0, 0x0)
react_1  | 	/go/src/github.com/DarthSim/hivemind/vendor/gopkg.in/urfave/cli.v1/app.go:485 +0xd4
react_1  | github.com/DarthSim/hivemind/vendor/gopkg.in/urfave/cli%2ev1.(*App).Run(0xc4200981a0, 0xc42000a140, 0x2, 0x2, 0x0, 0x0)
react_1  | 	/go/src/github.com/DarthSim/hivemind/vendor/gopkg.in/urfave/cli.v1/app.go:259 +0x74f
react_1  | main.main()
react_1  | 	/go/src/github.com/DarthSim/hivemind/main.go:72 +0x6e9

All was good until the latest change.

[Bug] Hivemind prints nothing

Hi,

First, thank you for this project, I've been using personally for a couple years now, I find it handy to spin up rails server, webpack, sidekiq all in one go. 🙇

I tried implementing a new Procfile.dev for a new project, this one is not Rails it's a node.js project and whenever I type hivemind Procfile.dev or anything in that folder, nothing happens, just plain no output.

I switched to the frontend folder of that project, added a Procfile.dev there and it worked as expected.

Back to the folder with issue I tried hivemind -h --help --verbose --version, asd asdnasdm never got any output.

Any idea what could be happening in this case? I don't know if the binary checks for some thing and does not run or I might have some permission missing.

I installed via homebrew v 1.1.0

Ignore release command?

Hi,

thanks for hivemind, I really enjoy using it!

I've recently added a release command to my Procfile (see https://devcenter.heroku.com/articles/release-phase). And now hivemind stops working as it recognizes the task exit and shuts everything down.

Would it make sense to ignore release by default? That's the behavior of heroku local

Cheers,
Florian

Option to skip reading .env?

Hi folks,

Would it be possible to add a flag that skips reading a local .env file? It seems to get loaded no matter what, which isn't always desired.

It would also be good to avoid setting things like $PORT too

Example:

Procfile

test: sh -c 'echo FOO is $FOO'

.env

FOO=BAR

❯ hivemind -l test
test | Running...
test | FOO is bar
test | Process exited

Hivemind process exits 0 regardless of child process status

To my understanding, the parent process should exit with the same status as the child process it's running (if there's only one child process maybe), or some other non-zero status.

Example:

Procfile:

test: exit 1

❯ hivemind -l test
test | Running...
test | exit status 1

❯ echo $?
0

Thanks in advance!

Installation via `go get` as documented in README fails

With go 1.13 on Centos 7

[vagrant@localhost ~]$ go version
go version go1.13.6 linux/amd64
[vagrant@localhost ~]$ go get -u -f github.com/DarthSim/hivemind
# github.com/DarthSim/hivemind
/opt/gopath/src/github.com/DarthSim/hivemind/main.go:26:5: app.Author undefined (type *cli.App has no field or method Author)
/opt/gopath/src/github.com/DarthSim/hivemind/main.go:27:5: app.Email undefined (type *cli.App has no field or method Email)
/opt/gopath/src/github.com/DarthSim/hivemind/main.go:33:17: cannot use cli.StringFlag literal (type cli.StringFlag) as type cli.Flag in array or slice literal:
        cli.StringFlag does not implement cli.Flag (Apply method has pointer receiver)
/opt/gopath/src/github.com/DarthSim/hivemind/main.go:33:36: unknown field 'EnvVar' in struct literal of type cli.StringFlag
/opt/gopath/src/github.com/DarthSim/hivemind/main.go:34:17: cannot use cli.StringFlag literal (type cli.StringFlag) as type cli.Flag in array or slice literal:
        cli.StringFlag does not implement cli.Flag (Apply method has pointer receiver)
/opt/gopath/src/github.com/DarthSim/hivemind/main.go:34:40: unknown field 'EnvVar' in struct literal of type cli.StringFlag
/opt/gopath/src/github.com/DarthSim/hivemind/main.go:35:14: cannot use cli.IntFlag literal (type cli.IntFlag) as type cli.Flag in array or slice literal:
        cli.IntFlag does not implement cli.Flag (Apply method has pointer receiver)
/opt/gopath/src/github.com/DarthSim/hivemind/main.go:35:32: unknown field 'EnvVar' in struct literal of type cli.IntFlag
/opt/gopath/src/github.com/DarthSim/hivemind/main.go:36:14: cannot use cli.IntFlag literal (type cli.IntFlag) as type cli.Flag in array or slice literal:
        cli.IntFlag does not implement cli.Flag (Apply method has pointer receiver)
/opt/gopath/src/github.com/DarthSim/hivemind/main.go:36:37: unknown field 'EnvVar' in struct literal of type cli.IntFlag
/opt/gopath/src/github.com/DarthSim/hivemind/main.go:36:14: too many errors
[vagrant@localhost ~]$ ( cd /opt/gopath/src/github.com/DarthSim/hivemind/ && git status -v && git log | head -n5 )
# On branch master
nothing to commit, working directory clean
commit 10d1fa0ff72d08fd1110f2c17a5bc0b1714af4af
Author: DarthSim <[email protected]>
Date:   Wed Dec 18 16:59:01 2019 +0600

    Update readme

With go 1.12 on osx

is-mbp-timothy4:tmp timothy$ go version
go version go1.12.1 darwin/amd64
is-mbp-timothy4:tmp timothy$ go get -u -f github.com/DarthSim/hivemind
# github.com/DarthSim/hivemind
/Users/timothy/go/src/github.com/DarthSim/hivemind/main.go:26:5: app.Author undefined (type *cli.App has no field or method Author)
/Users/timothy/go/src/github.com/DarthSim/hivemind/main.go:27:5: app.Email undefined (type *cli.App has no field or method Email)
/Users/timothy/go/src/github.com/DarthSim/hivemind/main.go:33:17: cannot use cli.StringFlag literal (type cli.StringFlag) as type cli.Flag in array or slice literal:
	cli.StringFlag does not implement cli.Flag (Apply method has pointer receiver)
/Users/timothy/go/src/github.com/DarthSim/hivemind/main.go:33:36: unknown field 'EnvVar' in struct literal of type cli.StringFlag
/Users/timothy/go/src/github.com/DarthSim/hivemind/main.go:34:17: cannot use cli.StringFlag literal (type cli.StringFlag) as type cli.Flag in array or slice literal:
	cli.StringFlag does not implement cli.Flag (Apply method has pointer receiver)
/Users/timothy/go/src/github.com/DarthSim/hivemind/main.go:34:40: unknown field 'EnvVar' in struct literal of type cli.StringFlag
/Users/timothy/go/src/github.com/DarthSim/hivemind/main.go:35:14: cannot use cli.IntFlag literal (type cli.IntFlag) as type cli.Flag in array or slice literal:
	cli.IntFlag does not implement cli.Flag (Apply method has pointer receiver)
/Users/timothy/go/src/github.com/DarthSim/hivemind/main.go:35:32: unknown field 'EnvVar' in struct literal of type cli.IntFlag
/Users/timothy/go/src/github.com/DarthSim/hivemind/main.go:36:14: cannot use cli.IntFlag literal (type cli.IntFlag) as type cli.Flag in array or slice literal:
	cli.IntFlag does not implement cli.Flag (Apply method has pointer receiver)
/Users/timothy/go/src/github.com/DarthSim/hivemind/main.go:36:37: unknown field 'EnvVar' in struct literal of type cli.IntFlag
/Users/timothy/go/src/github.com/DarthSim/hivemind/main.go:36:14: too many errors
is-mbp-timothy4:tmp timothy$ ( cd /Users/timothy/go/src/github.com/DarthSim/hivemind/ && git status -v && git log | head -n5 )
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean
commit 10d1fa0ff72d08fd1110f2c17a5bc0b1714af4af
Author: DarthSim <[email protected]>
Date:   Wed Dec 18 16:59:01 2019 +0600

    Update readme

Sub-processes aren't in the same process group

Because subprocesses aren't in the same process group as hivemind, when hivemind gets killed for some reason, the subprocesses arn't killed and are disowned instead.

Could you please spawn subprocesses in the same process group?

use PORT= in .env

It appears that hivemind 1.0.4 always sets its own value for PORT, even if PORT=1234 is present in .env.

Would it be possible for hivemind to set PORT only if PORT is not already set? Changing this would also make this compatible with foreman's behavior on this particular matter (that is, it allows PORT to be configured via .env).

hivemind fails to start some processes properly, but foreman works as expected

Summary

The following Procfile works properly with foreman 0.87.2, but fails to execute both processes when called with hivemind 1.1.0 as installed via Homebrew. Go reports itself as go version go1.18.1 darwin/amd64.

Procfile

yard: open -a Safari.app http://localhost: 8088
coverage: open -a Safari.app coverage/index.html

Failures from Hivemind

With foreman start, a PID is created for each browser process, and the relevant pages are opened in Safari. However, with hivemind ./Procfile the first process starts correctly even though hivemind thinks it has immediately exited. The second process (coverage) never runs, and reports:

coverage: Running
yard: Running...
yard: Process exited
coverage: Interrupting...
coverage: Signal: interrupt

This intuitively seems to be an issue with jobs that fork or share a parent process, since in actuality the YARD server (started separately by Guard in this case) continues to run, but hivemind seems to think that because open has exited it should not start or continue the other defined processes defined in the Procfile.

If there's a work-around, or if I've misunderstood the documentation, please let me know. Otherwise, this seems like a bug for anything that might spawn background processes or have non-interdependent processes that should not fail to start or continue to run simply because another named process within the Procfile did.

Option to turn off log prefixes

My Heroku app processes emit logs in JSON format to be ingested by LogDNA parser. I'm currently using Honcho (Python port of Foreman) with --no-prefix option to ensure the logs are parsed correctly.

Is there any way to turn off log prefixes in hivemind?

hivemind takes 99% CPU

Hey, I noticed that hivemind takes 99% CPU all the time.
Initially, I thought it's something w/ my processes, but nope.

Procfile

fake: sleep 10000

results in

Same when I run my app.

macOS: 10.12.6
Processor: 2,9 GHz Intel Core i7
Memory: 16 GB 2133 MHz LPDDR3

hivemind hangs when handling more than 65536 bytes of output

Hivemind hangs when it gets more than 65536 bytes of output from the child process. The following procfile demonstrates the problem:

works: hexdump -v -n 32768 -e '1/1 "%02x"' /dev/urandom
hangs: hexdump -v -n 32769 -e '1/1 "%02x"' /dev/urandom

I'm on MacOS 10.15 Catalina.

Go module release

This tool is installable via brew. in order to maintain go 1.17 support please release a new version with go modules.

Homebrew/homebrew-core#47627

how to install binary on linux?

given i want to install a linux binary, what steps do i take? i'm on debian arm64. and would like to grab the binary so it's easy for me to use within github's codespaces. apologies, feeling like an absolute noob here!

$ dpkg --print-architecture
amd64
$ curl -OL "https://github.com/DarthSim/hivemind/releases/download/v1.1.0/hivemind-v1.1.0-linux-arm64.gz"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   667  100   667    0     0   4446      0 --:--:-- --:--:-- --:--:--  4446
100 1105k  100 1105k    0     0  3734k      0 --:--:-- --:--:-- --:--:-- 3734k
$ gunzip hivemind-v1.1.0-linux-arm64.gz 
$ chmod +x ./hivemind-v1.1.0-linux-arm64 
$ ./hivemind-v1.1.0-linux-arm64 
bash: ./hivemind-v1.1.0-linux-arm64: cannot execute binary file: Exec format error

Error installing Hivemind with Go 1.13

Running go get github.com/DarthSim/[email protected] I got the following error:

build github.com/DarthSim/hivemind: cannot load gopkg.in/urfave/cli.v1: cannot find module providing package gopkg.in/urfave/cli.v1

I found a similar bug report and worked around it with replace gopkg.in/urfave/cli.v1 => github.com/urfave/cli v1.19.1 (the same version used in hivemind's go.mod).

hivemind/go.mod

Line 9 in f675566

gopkg.in/urfave/cli.v1 v1.19.1

hivemind does not work correctly with webpack-dev-server.

Hello,

thanks for this great process manager!

After rails 5.1 introduced webpack, I tried to keep webpack-dev-server and rails 5 running in parallel. Unfortinually, with hivemind it does not work, webpack-dev-server just does not produce any output after starting.

Procfile:

web: bundle exec rails s
assets: ./bin/webpack-dev-server

Output after typing hivemind

web    | Running...
assets | Running...
assets |  10% building modules 1/1 modules 0 active                                         
assets | Project is running at http://localhost:8080/
assets | webpack output is served from http://localhost:8080/packs/
assets | Content not from webpack is served from /Users/lisovskiivladislav/work/factory/factory-management-backend/public/packs
assets | 404s will fallback to /index.html
web    | => Booting Puma
web    | => Rails 5.1.0 application starting in development on http://localhost:3000
web    | => Run `rails server -h` for more startup options
web    | Puma starting in single mode...
web    | * Version 3.8.2 (ruby 2.4.1-p111), codename: Sassy Salamander
web    | * Min threads: 5, max threads: 5
web    | * Environment: development
web    | * Listening on tcp://0.0.0.0:3000
web    | Use Ctrl-C to stop

If to launch 2 processes each in an own tab, it works.

Make logging of timestamps optional

Tools can also add vanity information (unneeded timestamps in logs).

Timestamps are not just vanity information, but can add vital context if you want to know when something was logged. As one of my colleagues put it:

Yeah I have great experiences with Hivemind but I must admit that I sometimes re-run tasks because I can't tell if they ran just now or an hour ago

I think making logging an optional feature would make hivemind strictly better than foreman, since it already is superior in terms of logging and killing processes correctly.

It would be great if you would consider adding this as a feature.

Add command line argument to set window title

Hi!
I have 4 tabs running hivemind in my terminal and its impossible to identify what is where.
It will be nice to have the ability to set terminal title with command line (--title for example).
If --title argument is missing, get current directory name and put it in terminal title like project | hivemind.

darthsim / hivemind Goto Github PK

hivemind's Introduction

Hivemind

Enter Hivemind

Installation

With Homebrew (macOS)

Download the latest Hivemind release binary

From Source

Usage

Environment

Author

License

hivemind's People

Stargazers

Watchers

Forkers

hivemind's Issues

Summary

Procfile

Failures from Hivemind

Recommend Projects

Recommend Topics

Recommend Org