Git Product home page Git Product logo

steeloverseer's Introduction

Steel Overseer

A file watcher and development tool, similar to Ruby's Guard.

The main idea is that you have steeloverseer watch your files and then execute a series of shell commands in response. The first command to fail short circuits the series. The watched files can be selected using regular expressions and the commands may include capture groups.

Build Status Build status Build status

Also see feedback and other tools.

Installation

Download and install the stack build tool.

stack install steeloverseer

This will create a binary deep inside ~/.stack/, and symlink to it at ~/.local/bin/sos.

Usage

See sos --help to get started:

Steel Overseer 2.0.2

Usage: sos [TARGET] [--rcfile ARG] [-c|--command COMMAND] [-p|--pattern PATTERN]
           [-e|--exclude PATTERN]
  A file watcher and development tool.

Available options:
  -h,--help                Show this help text
  TARGET                   Optional file or directory to watch for
                           changes. (default: ".")
  --rcfile ARG             Optional rcfile to read patterns and commands
                           from. (default: ".sosrc")
  -c,--command COMMAND     Add command to run on file event.
  -p,--pattern PATTERN     Add pattern to match on file path. Only relevant if
                           the target is a directory. (default: .*)
  -e,--exclude PATTERN     Add pattern to exclude matches on file path. Only
                           relevant if the target is a directory.

Patterns and Commands

Capture groups can be created with ( ) and captured variables can be referred to with \1, \2, etc. (\0 contains the entire match).

For example, for each change to a .c file in src/ (excluding files containing "_test"), we may want to compile the file and run its corresponding unit test:

sos src/ -c "gcc -c \0 -o obj/\1.o" -c "make test --filter=test/\1_test.c" -p "src/(.*)\.c" -e "_test"

Commands are run left-to-right, and one failed command will halt the entire pipeline.

The RCFile

As a shortcut, we may want to write the above only once and save it in .sosrc, which is an alternative to the command-line interface (yaml syntax):

- pattern: src/(.*)\.c
  exclude: _test
  commands:
  - gcc -c \0 -o obj/\1.o
  - make test --filter=test/\1_test.c

Then, we only need to run:

sos

to start watching the current directory. If you'd like to use multiple rcfiles, or just don't like the name .sosrc you can specify the rcfile on the command line like so:

sos --rcfile my-rcfile

Grammar

sosrc            := [entry]
entry            := {
                      pattern_entry,
                      exclude_entry?, -- Note: optional!
                      command_entry
                    }
pattern_entry    := "pattern" | "patterns" : value | [value]
exclude_entry    := "exclude" | "excludes" | "excluding" : value | [value]
command_entry    := "command" | "commands" : value | [value]
value            := [segment]
segment          := text_segment | var_segment
text_segment     := string
var_segment      := '\' integer

The .sosrc grammar is somewhat flexible with respect to the command specifications. Both singular and plural keys are allowed, and both strings and lists of strings are allowed for values.

Pipelining Explaned

Pipelines of commands are immediately canceled and re-run if a subsequent filesystem event triggers the same list of commands. Otherwise, commands are are enqueued and run sequentially to keep the terminal output clean and readable.

For example, we may wish to run hlint on any modified .hs file:

- pattern: .*\.hs
  command: hlint \0

We can modify foo.hs and trigger hlint foo.hs to run. During its execution, modifying bar.hs will enqueue hlint bar.hs, while modifying foo.hs again will re-run hlint foo.hs.

Transient Files

Sometimes text editors and other programs create short lived files in the directories that sos is watching. These can trigger sos to run your pipeline. This can often be avoided by using precise include syntax, ie adding explicit matchers like an end-line match:

- pattern: .*\.tex$ 

Alternatively you may use exclude syntax to exclude any transient editor files (eg here's an sosrc used for editing Haskell doctests and ignoring emac's flycheck files):

# This is for testing documentation
- patterns:
  - .*/[^_]*\.l?hs$
  excludes:
  - \#
  - flycheck
  commands:
  - stack exec doctest -- \0

For more info, see #38

steeloverseer's People

Contributors

fredefox avatar gabriella439 avatar joefiorini avatar mitchellwrosen avatar peterbecich avatar rvl avatar schell avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

steeloverseer's Issues

Make it work on GHC 8.2.2 w/ newer deps etc.

This patch works for me:

diff --git a/app/Main.hs b/app/Main.hs
index 5811ff5..339c1f7 100644
--- a/app/Main.hs
+++ b/app/Main.hs
@@ -14,6 +14,7 @@ import Control.Concurrent.STM.TMVar
 import Control.Concurrent.STM.TQueue.Extra
 import Control.Exception
 import Control.Monad
+import Control.Monad.Catch (MonadThrow(..))
 import Control.Monad.Managed
 import Data.ByteString (ByteString)
 import Data.List.NonEmpty (NonEmpty(..))
diff --git a/steeloverseer.cabal b/steeloverseer.cabal
index e6bea6c..63e6a82 100644
--- a/steeloverseer.cabal
+++ b/steeloverseer.cabal
@@ -47,13 +47,13 @@ library
         , regex-tdfa >= 1.2
         , semigroups >= 0.16
         , stm >= 2.4
-        , streaming >= 0.1.0 && < 0.1.5
+        , streaming
         , text >= 1.2
         , yaml >= 0.8
         , aeson         >= 0.8
         , ansi-terminal >= 0.6.2
         , containers    >= 0.5
-        , process       >= 1.6.0 && < 1.6.1
+        , process
         , unix
     hs-source-dirs:
           src
@@ -73,7 +73,7 @@ executable sos
         , regex-tdfa >= 1.2
         , semigroups >= 0.16
         , stm >= 2.4
-        , streaming >= 0.1.0 && < 0.1.5
+        , streaming
         , text >= 1.2
         , yaml >= 0.8
         , steeloverseer
@@ -107,7 +107,7 @@ test-suite spec
         , regex-tdfa >= 1.2
         , semigroups >= 0.16
         , stm >= 2.4
-        , streaming >= 0.1.0 && < 0.1.5
+        , streaming
         , text >= 1.2
         , yaml >= 0.8
         , steeloverseer

Commands with quotes fail to parse

On 85d4008, e.g.:

$ echo "cabal test --test-options='1234'\\'' 567'"
cabal test --test-options='1234'\'' 567'

$ sos . -c "cabal test --test-options='1234'\\'' 567'"
sos: Error parsing command 'cabal test --test-options='1234'\'' 567''

(1)$ 

Make a package in nixpkgs without GHC in closure

Hi, currently installing haskellPackages.steeloverseer adds up to 2,5Gb of disk storage because it has GHC in closure. The size makes this package hard to add to a development environment, because sharing it with people will imply them downloading 2,5Gb of data. Maybe there is a way to package this wonderful app as they packaged pandoc (it weighs around 200Mb)?
See NixOS/nixpkgs#34376

sos doesn't compile with megaparsec-5.0.0?!

~ $ stack install steeloverseer
Run from outside a project, using implicit global project config
Using resolver: nightly-2016-06-21 from implicit global project's config file: /home/simon/.stack/global-project/stack.yaml
megaparsec-5.0.0: configure
megaparsec-5.0.0: build
megaparsec-5.0.0: copy/register
steeloverseer-2.0: download
steeloverseer-2.0: configure
steeloverseer-2.0: build
Completed 2 action(s).

--  While building package steeloverseer-2.0 using:
      /home/simon/.stack/setup-exe-cache/x86_64-linux/setup-Simple-Cabal-1.24.0.0-ghc-8.0.1 --builddir=.stack-work/dist/x86_64-linux/Cabal-1.24.0.0 build --ghc-options " -ddump-hi -ddump-to-file"
    Process exited with code: ExitFailure 1
    Logs have been written to: /home/simon/.stack/global-project/.stack-work/logs/steeloverseer-2.0.log

    Configuring steeloverseer-2.0...
    Building steeloverseer-2.0...
    Preprocessing library steeloverseer-2.0...
    [1 of 5] Compiling Sos              ( src/Sos.hs, .stack-work/dist/x86_64-linux/Cabal-1.24.0.0/build/Sos.o )

    /tmp/stack27432/steeloverseer-2.0/src/Sos.hs:20:11: error:
        • Expecting two more arguments to ‘ParseError’
          Expected a type, but ‘ParseError’ has kind ‘* -> * -> *’
        • In the type ‘ParseError’
          In the definition of data constructor ‘SosCommandParseException’
          In the data declaration for ‘SosException’

If this is due to a missing or too large upper bound on megaparsec, it would be great if you could add a revision for that bound on Hackage!

Feature request: support REPLish workflow

I wonder if a typical ghci workflow can be automated with sos.

Usually you run a ghci process and:

  1. When *.hs changes, you send :r<CR> or possibly :r<CR>:main<CR> to ghci
  2. When *.cabal or *.yaml changes, you restart ghci
  3. Also you're able to type stuff into currently running ghci

Right now 2. is working perfectly, 3. works kind of weird (I'm not even sure how to describe it) and 1. is not covered at all.

Meanwhile, ghcid supports 1 and 2, but not 3 (by current design) and naturally only supports ghci. Ideally I'd like to be able to use sos to automate a REPL workflow for any language, e.g. restart the process on project.clj changes and send (run-all-tests)<CR> on other changes.

Crashes on emacs lock files. (symbolic link without a real target)

While editing a file, emacs creates a lock file (symbolic link) of the form .#filename → [email protected]:timestamp (not sure if it's a real timestamp or not).

While checking file status on files in the target directory, sos crashes with the following message:

sos: /path/to/file: getFileStatus: does not exist (No such file or directory)

Steps to reproduce:

  1. Create an invalid link.

    $ cd /tmp
    $ ln -s /invalid/target link
    
  2. Run sos

    $ sos .
    sos: /tmp/link: getFileStatus: does not exist (No such file or directory)
    

Make sos smarter about interactive processes

Right now sos doesn't deal very well with commands that read on stdin (like some stack exec my-thing at the end of a list of commands).

Somehow, these interactive processes aren't actually killed right now, so sos both leaks pids and also subsequent runs compete for input from stdin.

sos: addWatch: resource exhausted (No space left on device)

When running sos on Ubuntu-17.10 (x64), I occasionally get sos: addWatch: resource exhausted (No space left on device) at first I thought this was only when I was in tmux, but it looks like it happens in some non-deterministic fashion. Memory usage is <25% utilization and disk space is ~66% -- so I'm not sure what kind of space is being referred to in this error. I've never had this problem on Archlinux or Debian (so far).

If someone could point me in a direction, I can try to fix this myself (I imagine it is a bit difficult to simulate).

Zombie processes

After running a command, a zombie process stays.

32430 robin     20   0  127208   4812   3068 S   0.0  0.0   0:00.24                          `- bash                                                                                                                                                                                        
23581 robin     20   0  1.001t  34584  30164 S   0.0  0.3   0:00.09                              `- sos                                                                                                                                                                                     
23661 robin     20   0       0      0      0 Z   0.0  0.0   0:00.04                                  `- cabal                                                                                                                                                                               
24087 robin     20   0       0      0      0 Z   0.0  0.0   0:00.04                                  `- cabal                                                                                                                                                                               
24213 robin     20   0       0      0      0 Z   0.0  0.0   0:00.05                                  `- cabal

im using Steel Overseer 2.0.1.0
and the command sos -p '.*\.hs' -c 'cabal run'

/bin/sh: -c: line 0: syntax error near unexpected token `('

$ g++ 000.cpp && diff output.txt <(./a.out < input.txt)
1c1
< 9
\ No newline at end of file
---
> Hello

$ sos -p 000.cpp -c "g++ 000.cpp && diff output.txt <(./a.out < input.txt)"
Hit Ctrl+C to quit.

Modified: 000.cpp
[1/1] g++ 000.cpp && diff output.txt <(./a.out < input.txt)
/bin/sh: -c: line 0: syntax error near unexpected token `('
/bin/sh: -c: line 0: `g++ 000.cpp && diff output.txt <(./a.out < input.txt)'
Failure ✗ (1)

Incompatibility with fsnotify >= 0.3

    /tmp/stack19783/steeloverseer-2.0.2.0/app/Main.hs:258:5: error:
        • The constructor ‘FSNotify.Added’ should have 3 arguments, but has been given 2
        • In the pattern: FSNotify.Added path _
          In a case alternative:
              FSNotify.Added path _ -> S.yield (FileAdded (go cwd path))
          In the second argument of ‘S.for’, namely
            ‘(\case
                FSNotify.Added path _ -> S.yield (FileAdded (go cwd path))
                FSNotify.Modified path _ -> S.yield (FileModified (go cwd path))
                FSNotify.Removed _ _ -> pure ())’
        |
    258 |     FSNotify.Added    path _ -> S.yield (FileAdded    (go cwd path))
        |     ^^^^^^^^^^^^^^^^^^^^^^^^

Hackage update

The version in master seems to be working well, and has updated depencencies, would it be possible to upload it to Hackage?

Thank you!

Can't prevent sos from triggering on emacs lock files

Hi!
I have emacs creating files like .#myfile.txt when I am editing a file, and sos triggers on those (runs the command). This is not the behaviour I want, so I tried using -e to exclude those, but I just can't get it to work!
I tried following:

-e "#"
-e "\.#"
-e ".*\.#.*"

and non of those excludes the files in question from triggering sos.

The message I get is Added: foo/bar/.#yourfile.txt. So it seems to be triggering only on adding the emacs lock file, not modifying it. Is it possible that -e matches only modified files and not added ones?

Am I doing smth wrong? Thank you!

Btw. I am on Archlinux, I am running sos on a directory, and sos version is 2.0.2.

Design typed (client-server) interface to steeloverseer

I'm thinking of expressing the steeloverseer "server" as a stream of typed "messages", where each one is responded to by the client (front-end: terminal, ncurses, etc).

Haven't though about it too hard but here's what I've come up with:

Use a typed GADT for the message type, produced by the server process and tagged with the expected type of the client response.

data MessageF a where
  -- "I just started a job"
  JobStarted :: JobDescription -> MessageF () -- client responds with (), aka "ok"
  -- "The current job produced a line of output"
  Stdout :: Text -> MessageF ()
  -- "A job exited unsuccessfully, continue?"
  JobDied :: JobDescription -> MessageF Bool

The exact messages obviously need to be fleshed out a bit. Anyways, with this GADT, I think we can reuse the streaming machinery with a clever functor:

data Message a = forall x. Message (MessageF x) (x -> a)

Thus, the overall type of the server is something like

Stream Message IO ()

which expands to (basically)

  Pure ()
| Effect (IO (Stream Message IO ()))
| Step (forall x. (MessageF x) (x -> Stream (Message IO ())))

Not triggered for `git checkout`

I’m using the latest HEAD, 85d4008.

I’m first modifying Main.hs 3 times in Emacs, and then switch to some other terminal and do git checkout src/ and:

$ cd some-project/
$ sos src/ -p '\.hs$' -e '/\.#' -e '/flycheck_' -c :        Hit Ctrl+C to quit.

Modified: src/Main.hs
[1/1] :
Success ✓

Modified: src/Main.hs
[1/1] :
Success ✓

Modified: src/Main.hs
[1/1] :
Success ✓


# now I’ll do git checkout src/


# nothing happened
^C

Exclusion patterns

It would be great to be able to exclude patterns explicitly so we don't have to model that with regular expressions. Excluding based on a string makes for a pretty tough regex.

sos -p ".*\.l?hs$" -e "#" -e "flycheck" -c "stack build"

This sos command would only stack build when changes are made to a haskell file (possibly literate haskell) that doesn't contain a # or flymake, which are both emacs artifacts.

AsyncCancelled if command takes a while.

sos output.raw -c "sleep 0.15; echo 'test'"
Hit Ctrl+C to quit.

Modified: output.raw
[1/1] sleep 0.15; echo 'test'
AsyncCancelled
AsyncCancelled

Modified: output.raw
[1/1] sleep 0.15; echo 'test'
AsyncCancelled
AsyncCancelled

Modified: output.raw
[1/1] sleep 0.15; echo 'test'
AsyncCancelled
AsyncCancelled

Modified: output.raw
[1/1] sleep 0.15; echo 'test'
AsyncCancelled
AsyncCancelled

This only occurs if sleep 0.15 is present. echo 'test' is never run. This used to work without issue (I might have been on an older commit / version).

This occurs with the master branch (3eee0ec).

Actions don't restart after an `AsyncCancelled`

If a file change triggers a longer task, and an other file is changed sos interrupts the current task with an AsyncCancelled message.

I'd then expect it to immediately restart the relevant actions, as a watch file has changed. Instead, it waits until an other change is made, and then restarts.

I'm not sure if this is by design or a bug, but either way it would be nice to have the option to restart immediately after cancellation.

If this is a bug, it might be related to #38.

Special case for working on a single file script

When I'm working on a script named "foo.py", I'd like to use sos foo.py as a shorthand for sos foo.py -c foo.py. The former invocation does nothing, so this change should not break anything.

Sometimes it hangs

85d4008

I can’t quite get how this happens, but look at this screenshot:

For all following modifications of that .hs file, nothing happens: sos doesn’t output interrupted or anything like that. And the commands are not run.

In htop it has 8 instances of Cabal as its child processes, but they are not really Cabal, only named so (look at their mem stats):

steeloverseer-1.1.0.1 on hackage doesn't complie

Resolving dependencies...
Configuring steeloverseer-1.1.0.1...
Building steeloverseer-1.1.0.1...
Preprocessing executable 'sos' for steeloverseer-1.1.0.1...

src/Main.hs:11:8:
    Could not find module `SOS'
    Use -v to see a list of the files searched for.
ghc --make: /usr/hs/ghc/7.6.3/bin/ghc failure (return code=1)
Failed to install steeloverseer-1.1.0.1

cabal unpack confirms that there's no SOS.hs in hackage tarball.

Parallel execution of commands

If I understand correctly, currently it is possible to execute commands just one after the other.

Would it be interesting/possible to configure certain commands to run in parallel?

For example, I could have four commands a, b, c and d and I would like to run a, then b and c in parallel and only after they both complete, run d

Related work

I just stumbled on this and I sort of wish I had found it sooner! I just implemented a similar tool at https://github.com/grafted-in/twitch-cli. Mine is not as powerful but perhaps a bit simpler to use. I'm posting here in case there are ideas we can share.

twitch-cli is able to set up many pattern/command patterns and simply relies on you using something like ; or && to run commands in sequence. It uses Glob instead of regex and is therefore a bit simpler but less powerful.

Write tests

This app is very IO-heavy and side-effecty. Would be nice to refactor it to write some pure tests, be able to feed in pure (mocked) filesystem events, and stuff like that.

Some simple high-level properties to test:

  • The same job should not be enqueued twice (so e.g. saving multiple times in succession will not each trigger a separate unit test run)
  • A job identical to the one currently running should cause it to be canceled and restarted

Then there's all the untested IO ugliness:

  • Filesystem oddities (dead symlinks, permission errors, etc)
  • Failed spawned processes, or rogue processes that don't even respond to SIGINT (not sure if Haskell can do anything about these at all)
  • Async exceptions handling

Run Commands on First Start

It would be nice if there were an option to run the commands on first start before any of the files change. This is useful when starting up a server.

Special pattern for filename/basename?

Right now \0 refers to the entire match of the regex, which is not necessarily the entire file.

It might be nice to launch sos like so:

sos -p 'hs$' -c 'stack ghc <the-file>'

and have it run stack ghc on any modified file that ends in hs. However, I believe the simplest way to do this (currently) is:

sos -p '.*hs$' -c 'stack ghc \0'

Not too bad, but that extra .* can be annoying. Would it make sense to have magic variable names for the absolute path matched and/or the basename matched?

Crazy idea: reuse .gitignore

Maybe it’d make sense to allow reusing project’s .gitignore, as these are the files one’s most likely to manually be adding to -e exclusion patterns.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.