Git Product home page Git Product logo

deltareport's Introduction

Deltareport

Find file/dir changes and queue to Beanstalkd for processing.

This application does not spawn off go-routines, it does all in the main-routine.

Please don't run this application multiple times with the same delta.db!

Use of this source code is governed by a BSD-style license that can be found in the LICENSE file.

config.toml

# Directory to scan for additional Files-entries
# allowing a more modular approach
# (only read on start)
Confdir = "./conf.d"
# Path to auto-created filed file for deltareport
# to remember it's state
Db = "/var/deltareport/example.db"
# Read all .JS-files from the script.d-dir and
# if none have errors run them one by one and run
# to filter out text from the diffs
# (only read on start)
Scriptdir="./script.d"

# Output queues
# The queue type 'mail' and 'newline' indicate
# what type of JSON is added to the Beanstalkd queue.
[Queues.mail]
	[Queues.mail.admin]
		Beanstalkd = "127.0.0.1:11300"
		From = "support"
		To = ["[email protected]"]
		Subject = "[AUTOGEN] "

	[Queues.newline.sess]
		Beanstalkd = "127.0.0.1:11300"
		Queue = "linediffs"
	[Queues.newline.slack]
		Beanstalkd = "127.0.0.1:11300"
		Queue = "slack"

[[Files]]
	# Watch an individual file and send to queues.mail.admin any changed byte
	Path = "./test.txt"
	To = "admin"
	Recurse = false
	Linediff = false

[[Files]]
	# Watch a directory recursive and send changes files+lines to queues.newsline.sess
	# Filtering by file extension+regular expression

	Path = "./test.d"
	To = "sess"
	Recurse = true
	IncludeExt = [
		".txt",
		".log"
	]
	Regex = "/valid.txt$"
	# Add every changed line separately to the queue
	Linediff = true

[[Files]]
	# Queue to newline/slack and write to #channel
	# Filtering by file extension
	Path = "./test.d"
	To = "sess"
	Tags = ["channel"]
	Recurse = true
	IncludeExt = [
		".txt",
		".log"
	]
	# Add every changed file to the queue
	Linediff = false

Help

  • "Linediff": true If the diff is "msg1\nmsg2\nmsg3" then msg1/msg2 and msg3 are all added separately in the queue (creating 3 jobs in the queue instead of 1).
  • "Tags": ["channel"] Metadata that only the worker understands.
  • Watching same files/dirs multiple times? Yes possible.

How?

Using the keyvaluestore (Bolt) to remember the last read position and on change read all changes and write these to the assigned queue. It reads/loads it's status from ./delta.db.

Beanstalkd is used as a persistant queue between this application (diffing) and processing (workers).

Scripting engine?

Yes, there is a small JavaScript-engine (Otto, native Go) embedded into the application. By adding one or more JS-files to the Scriptdir these can be executed just before the diff it sent to the processor for sending.

The idea is very simple: Let the script 'filter' out anything you don't want reported by adjusting the input text and returning the new text.

Scripts get two variables: queue and body, where queue contains the channel it wants to send to and body contains the text it will send.

Example added in script.d/example.js where for the admin-queue the 'Hello world2!' line is removed.

Workers?

Datastructures

type Email struct {
	From string                 // Key that MUST match From in config
	To []string
	Subject string
	Html string
	Text string
	HtmlEmbed map[string]string // file.png => base64(bytes)
}
type LineDiff struct {
	Hostname string
	Path     string
	Line     string
	Tags     []string
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.