Git Product home page Git Product logo

haproxy-spoe-go's Introduction

Haproxy SPOE Golang Agent Library Go Report Card

Terms from Haproxy SPOE specification

* SPOE : Stream Processing Offload Engine.

    A SPOE is a filter talking to servers managed ba a SPOA to offload the
    stream processing. An engine is attached to a proxy. A proxy can have
    several engines. Each engine is linked to an agent and only one.

* SPOA : Stream Processing Offload Agent.

    A SPOA is a service that will receive info from a SPOE to offload the
    stream processing. An agent manages several servers. It uses a backend to
    reference all of them. By extension, these servers can also be called
    agents.

* SPOP : Stream Processing Offload Protocol, used by SPOEs to talk to SPOA
         servers.

    This protocol is used by engines to talk to agents. It is an in-house
    binary protocol described in this documentation.

This library implements SPOA for Golang applications

Install

go get -u github.com/negasus/haproxy-spoe-go

Example

Example from Section 2.5 SPOE specification describes a simple IP reputation service.

Here is a simple but complete example that sends client-ip address to a ip reputation service. This service can set the variable "ip_score" which is an integer between 0 and 100, indicating its reputation (100 means totally safe and 0 a blacklisted IP with no doubt).

Golang backend application for this example

package main

import (
	"github.com/negasus/haproxy-spoe-go/action"
	"github.com/negasus/haproxy-spoe-go/agent"
	"github.com/negasus/haproxy-spoe-go/request"
        "github.com/negasus/haproxy-spoe-go/logger"
	"log"
	"math/rand"
	"net"
	"os"
)

func main() {

	log.Print("listen 3000")

	listener, err := net.Listen("tcp4", "127.0.0.1:3000")
	if err != nil {
		log.Printf("error create listener, %v", err)
		os.Exit(1)
	}
	defer listener.Close()

	a := agent.New(handler, logger.NewDefaultLog())

	if err := a.Serve(listener); err != nil {
		log.Printf("error agent serve: %+v\n", err)
	}
}

func handler(req *request.Request) {

	log.Printf("handle request EngineID: '%s', StreamID: '%d', FrameID: '%d' with %d messages\n", req.EngineID, req.StreamID, req.FrameID, req.Messages.Len())

	messageName := "get-ip-reputation"

	mes, err := req.Messages.GetByName(messageName)
	if err != nil {
		log.Printf("message %s not found: %v", messageName, err)
		return
	}

	ipValue, ok := mes.KV.Get("ip")
	if !ok {
		log.Printf("var 'ip' not found in message")
		return
	}

	ip, ok := ipValue.(net.IP)
	if !ok {
		log.Printf("var 'ip' has wrong type. expect IP addr")
		return
	}

	ipScore := rand.Intn(100)

	log.Printf("IP: %s, send score '%d'", ip.String(), ipScore)

	req.Actions.SetVar(action.ScopeSession, "ip_score", ipScore)
}

API

Messages

Getting request data is possible through Request.Messages

Len() int

Returns count of messages in request

count := request.Messages.Len()

GetByName(name string) (*Message, error)

Returns message by name and error if not exists

mes, err := request.Messages.GetByName("get-ip-reputation")

GetByIndex(idx int) (*Message, error)

Returns message by index and error if not exists

mes, err := request.Messages.GetByIndex(0)

Message

Represents one message, sent by Haproxy

Message has two fields

  • Name string
  • KV *kv.KV

KV contains key-value data of message

Actions

Actions is used for sending a response to Haproxy

SetVar(scope Scope, name string, value interface{})

Set variable with name to value in specific scope (see bellow)

request.Actions.SetVar(action.ScopeSession, "ip_score", 10)

UnsetVar(scope Scope, name string)

Unset variable with name in specific scope

request.Actions.UnsetVar(action.ScopeSession, "ip_score")

Actions Scopes

  • ScopeProcess
  • ScopeSession
  • ScopeTransaction
  • ScopeRequest
  • ScopeResponse

KV (key-value)

Contains message key-value data sent by Haproxy

Get(key string) (interface{}, bool)

Returns value by name. If key doesn't exist, last returned value will be set to False

ipValue, ok := message.KV.Get("ip")

haproxy-spoe-go's People

Contributors

jan-dubsky avatar kinnou02 avatar lwimmer avatar negasus avatar pulzarraider avatar xani avatar y9mo avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

haproxy-spoe-go's Issues

ack frame write failed

Hey under heavy load on the server seems after sometime to spoa server cannot write to the connection as haproxy has closed it? Do you have any pointers or information on where I can look to debug this?

spoa-1      | time="2024-05-15T18:24:46Z" level=error msg="ack frame write failed: cannot write frame to connection: write tcp 10.5.5.254:9000->10.5.5.4:38230: use of closed network connection" worker=spoa1

Thank you for spending time on writing this package ๐Ÿ‘๐Ÿป

Memory leak?

Hi. We are using coraza-spoa which is using this library. Notices that this service is gradually increasing it's memory usage. This service is now enabled on a site that has a lot of incoming requests to process.

Took out a pprof file and a graph of the memory consumption.

profile001
Screenshot 2023-10-09 at 08 40 52

Together with this memory usage we notice this:
level=warning msg="spoe: error handling connection: disconnect error: I/O error"

Not sure if this is connected to haproxy-spoe-go or something else but that causes a lot of the requests to not be processed correctly (Failed to parse request body).

Any clue on how we can troubleshoot this further?

Agent graceful shutdown

Hey all ๐Ÿ‘‹๐Ÿป

Thank you for the amazing package made starting crowdsec-spoa a breeze, however, I was surprised we dont have a way to gracefully shutdown the server. If your open to it I could have a go?

I believe it should be something like a root context on the agent that is checked by the for loop on serve, however, because we dont keep track of any connections I believe we should also note if there is a response pending ๐Ÿคท๐Ÿป let me know how you feel about it ๐Ÿ‘๐Ÿป

Broken pipe on unix socket error handle worker: error send AgentDisconnect frame

When listening on a unix socket, I get this error a few seconds after each event is processed:

2020/12/14 18:37:58 error handle worker: error send AgentDisconnect frame: write unix /var/run/gohaproxy/agent.sock->@: write: broken pipe

Example code https://github.com/alechenninger/go-haproxy-agent/blob/cd1d9681bcfe818f134dc1bc2d2ba1ec4e5983f2/main.go#L43

I've tried both listneing on "unix" and "unix packet"

Haproxy configs here https://github.com/alechenninger/go-haproxy-agent/tree/cd1d9681bcfe818f134dc1bc2d2ba1ec4e5983f2/conf

No ip_score returned

Hello,

we're facing something weird, here is the haproxy part:
...
http-request capture var(sess.myiprep.ip_score) len 3
http-response set-header X-IP-Score "%[var(sess.myiprep.ip_score)]"
...

in the Go file, "ip_score" is correct (verified with a dedicated log file) but it seems not properly sent back to Haproxy by "req.Actions.SetVar(action.ScopeSession, "ip_score", ipScore)"

here is the iprep.conf:
[ip-reputation]
spoe-agent iprep-agent
messages check-client-ip
option var-prefix myiprep
option continue-on-error
timeout hello 2s
timeout idle 2m
timeout processing 10ms
use-backend agents
no log
spoe-message check-client-ip
args ip=src
event on-client-session if ! { src -f /etc/haproxy/whitelist.lst }

any idea ?

bye Fred

Bug with 64-bit varint encodings

The buffer created here

buf = append(buf, TypeInt64)
b := make([]byte, 8)
i := varint.PutUvarint(b, uint64(v))

Isn't big enough for 64-bit integers (which can take up to 10 bytes when encoded as varints).

Problem acquiring messages with binary fields

Hi,

Thanks for the work! Very useful :)

I'm trying to port the libmodsecurity SPOA to Golang using your library. It happens that when trying to acquire a message with a binary field (as req.hdr_bin) I get a panic, as following:

2019/11/13 21:22:57 listen 3000
panic: runtime error: index out of range [0] with length 0

goroutine 34 [running]:
github.com/negasus/haproxy-spoe-go/message.(*Messages).Decode(0xc00000e060, 0xc00001e0b2, 0x0, 0x1, 0x1, 0x53)
	/media/rkatz/data/rkatz/go/codes/haproxy-spoe-go/message/decode.go:22 +0x339
github.com/negasus/haproxy-spoe-go/frame.(*Frame).Read(0xc0000e8080, 0x562880, 0xc000072180, 0xc00010f000, 0xc00010c000)
	/media/rkatz/data/rkatz/go/codes/haproxy-spoe-go/frame/read.go:61 +0x3c4
github.com/negasus/haproxy-spoe-go/worker.(*worker).run(0xc0000e41e0, 0x0, 0x0)
	/media/rkatz/data/rkatz/go/codes/haproxy-spoe-go/worker/worker.go:53 +0x12c
github.com/negasus/haproxy-spoe-go/worker.Handle(0x5648e0, 0xc00010c000, 0x5461c0)
	/media/rkatz/data/rkatz/go/codes/haproxy-spoe-go/worker/worker.go:24 +0x66
created by github.com/negasus/haproxy-spoe-go/agent.(*Agent).Serve
	/media/rkatz/data/rkatz/go/codes/haproxy-spoe-go/agent/agent.go:31 +0xe7

I've been trying to dig into the code, but I'm not having success to dig what is going on. Is there any way to acquire binary fields?

This errors happens earlier, with no code, just with

mes, err := req.Messages.GetByName(messageName)
	if err != nil {
		log.Printf("message %s not found: %v", messageName, err)
		return
	}

Is there anything I'm missing?

Thank you!!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.