ngmoco / falcore Goto Github PK

Was wondering if it is possible to use this in combination with the Gorilla web toolkit at http://www.gorillatoolkit.org/

Thanks!

not trolling, but ab doc says:

-k
Enable the HTTP KeepAlive feature, i.e., perform multiple requests within one HTTP session. Default is no KeepAlive.
gleecology commented

More to the point, after the first conn, it fails (even whilst adding the Connection: header):

--debug--

ab -v 6 -n 3 -c 1 -k http://localhost:8000/

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)...INFO: POST header ==
GET / HTTP/1.0
Connection: Keep-Alive
Host: localhost:8000
User-Agent: ApacheBench/2.3
Accept: /

LOG: header received:
HTTP/1.1 200 OK
Content-Length: 12

hello world!
LOG: Response code = 200
apr_poll: The timeout specified has expired (70007)

...To this ignorant layman, KA looks broken.

This is basically a duplicate of issue #4. I would have re-opened it, but don't have permission.

C:\Users\Ralph\Temp>go version
go version go1
C:\Users\Ralph\Temp>go get github.com/ngmoco/falcore
# github.com/ngmoco/falcore
C:\Go\src\pkg\github.com\ngmoco\falcore\server.go:100: cannot use fd (type int) as type syscall.Handle in function argument

I'd like to propose we rename SimpleResponse to StringResponse. We can then add a BytesResponse with similar behavior. I'd also like to add a JSONResponse helper that takes an interface{} and returns (*http.Response, error). There's already a pull request for a RedirectResponse helper.

These are extremely common cases and we should support them. Renaming SimpleResponse might be painful, but it will be more clear. We could leave it as is for now, but it seems better to just go for it...

Thoughts?

Is it possible to extract parameters with the router to pass to a pipeline stage ?

Created this issue to discuss the possibility/likelyhood of getting a litte bit of refactoring commited. See this branch for the commits: https://github.com/gnanderson/falcore/tree/refactor

Will send pull requests to Dave and Scotts tree's for review.

This started off by a series of benchmarks using weighttpd against one of my own server binaries, I noticed that using weighttp with keep alive enabled gave a very unrealistically good score, but then I noticed that only half the requests were successful so there was something up server side. I tracked it down to the main server loop and the 'Connection' header check. While RFC2616 states that header field names should be case insensitive, it's vague about the header values and the upper case Keep-Alive check in the server loop was failing on every second request using weighttp.

So no problem, easy fix, but when I changed the header check to match lower case I noticed performance fall through the floor for persistent keep alive connections. I tracked that down to a combination of TCP_CORK plus the tweaked TCP stack setting on the linux box. With a bit of fiddling and some hit and miss I was back to my usual throughput.

Having said that, what I really missed was being able to bench the falcore.Server directly in the test suite. I started off trying to add a benchmark for falcore.Server, and this was OK for simple hello world filters but to use the file_filter meant the cyclic dependancy of importing file_filter which imports falcore which imports file_filter... made this impossible.

So one thing led to another, breaking the cyclic dependancy meant having to refactor the package imports, and api calls/references, and then it meant also refactoring the filter tests to work standalone without an http server, otherwise I would have been back to square one.

Anyway, there's a few things to look at in the refactored branch... Let me know what you think. It breaks the package imports for server binaries and there's a few API backwards incompatibilities, such things as each filter had a "Filter" struct which i renamed to FileFilter, CompressionFilter and so on.

Cheers the noo,
Graham

rather than starting a new falcore server.

can I attach falcore to the started net/http server?

this way I can use falcore with google appengine.

thanks.

I've noticed an intermittent failure of this test, here's a sample log from the latest failure.

[   51s] $WORK/github.com/ngmoco/falcore/static_file/_test/static_file.test
[   52s] --- FAIL: TestBasicFiles (0.02 seconds)
[   52s]   file_filter_test.go:144: custom mime type Error GETting file:unexpected EOF

I first saw the failure on the openSUSE Factory and openSUSE Tumbleweed builders. These are our unstable and rolling release platforms respectivly so after triggering a rebuild and not seeing a direct repeat of the failure I dismissed it as an issue with an unstable platform. Subsequently I have seen the issue maybe two or three time on the current stable release builders and now on an external distro build. Seeing the issue on an external distro prompted me to upstream it.

This intermittently appears on both 32 and 64 bit archs, and across at least the following platforms, but this should be considered with the knowledge that as in intermittent failure, the openSUSE platforms are the ones that get automatically triggered to rebuild more often which increases the likelyhood of occurence.

• openSUSE 12.2 RC
• openSUSE Factory
• openSUSE Tumbleweed
• RHEL 6

A first take suggests the error can only originate from the RoundTripper use but poking at the implemtation in the std libs:
http://golang.org/src/pkg/net/http/transport.go?s=3821:3892#L116
I see that there's the use of synch primatives, so this may be a subtle failure and possibly a bug in the test rather than falcore. Additionally we run our Go builds with -p 4 to match the allocated cores on the builders, so I'm not sure if that affects things.

From a trace:

[TRAC] 1c7de31dfc server.Init                    S=0 Tot=0.0003 %=44.39
[TRAC] 1c7de31dfc *auth_filter.Filter            S=0 Tot=0.0002 %=39.46
[TRAC] 1c7de31dfc *falcore.Pipeline              S=0 Tot=-8143286784.0000 %=-1384912741466112.00
...

Captain we've been hit!

Tried to install.

go get github.com/ngmoco/falcore

And got:

 # github.com/ngmoco/falcore
 C:\Go\src\pkg\github.com\ngmoco\falcore\server.go:68: cannot use int(srv.listenerFile.Fd()) (type int) as type syscall.Handle in function
 argument

Windows amd64 Go 1

The serve loop is currently very generous with clients that use keep-alive. This is, I think, a good thing, but it means the server can never gracefully shut down if there are long lived keep-alives.

I think the graceful shutdown procedure should be finish up any in flight requests and close connections as gracefully as possible. I can see a very clean approach to this by using a separate (permanent or temporary) go routine for reading. This will double the number of goroutines per connection, but it will also allow us to support http pipelining as we can separate read from write.

Any thoughts? Too much?

Could you please be so kind to explain how to use the hot restart functionality?

When sending SIGHUP to the hot_restart example binary, I receive an address already in use error like so:

./hot_restart 
47149 Received SIGHUP.  forking.
Forking now with socket: 7
47149 Forked pid: 47220 errno: <nil>
47220 Got socket FD: 7
47220 Could not start server: listen tcp <nil>:8090: address already in use47220 Exiting now

this is on osx 10.8

thanks a lot!

While trying to unit test my own request filters, I realized it's very difficult as I can't create my own complete falcore.Request objects.

running go build server.go in falcore/examples/basic_file_server produces

server.go:7:2: import "github.com/fitstar/falcore/compression": cannot find package
server.go:8:2: import "github.com/fitstar/falcore/static_file": cannot find package

running go get in the same directory produces

package github.com/fitstar/falcore/compression
    imports github.com/fitstar/falcore/compression
    imports github.com/fitstar/falcore/compression: import "github.com/fitstar/falcore/compression": cannot find package
package github.com/fitstar/falcore/compression
    imports github.com/fitstar/falcore/static_file
    imports github.com/fitstar/falcore/static_file
    imports github.com/fitstar/falcore/static_file: import "github.com/fitstar/falcore/static_file": cannot find package

The upstream has a reference to a single *net.TCPConn which it uses to set the deadline just before calling transport.RoundTrip. The transport will actually open new connections to the same server up to MaxIdleConnsPerHost if the current connection is already in use. Whenever a new connection is opened it overwrites the single reference. When an old connection is reused later, it times-out since the deadline was set on the new connection instead of the older one being used.

I can't think of an easy way to fix this. Maybe wrap the connection with one that checks a channel. Send a flag to the channel in the same place as current deadline place and the first connection used will pop the channel and set the deadline only on the first write, otherwise no deadline is set. I don't think this method is "thread" safe though.

Thoughts?

Scott, I think you're right that we should move most of the responders to a subpackage. I propose we keep SimpleResponse, StringResponse, ByteResponse, and PipeResponse in falcore proper and move all others to a new package, responder.

Thoughts?

Using this most basic falcore app:

package main

import (
    "net/http"
    "github.com/ngmoco/falcore"
)

func main() {

    pipeline := falcore.NewPipeline()
    pipeline.Upstream.PushBack(falcore.NewRequestFilter(func(req *falcore.Request)*http.Response {
        return falcore.SimpleResponse(req.HttpRequest, 200, nil, "OK\n")
    }))

    server := falcore.NewServer(8081, pipeline)
    server.ListenAndServe()
}

A simple request is very fast:

Daves-MacBook-Air-2:ka_test dgrijalva$ time curl http://localhost:8081/
OK

real    0m0.015s
user    0m0.008s
sys 0m0.005s

If you add a Connection:Keep-Alive header, it always takes just over 5 seconds:

Daves-MacBook-Air-2:ka_test dgrijalva$ time curl http://localhost:8081/ -H"Connection:Keep-Alive"
OK

real    0m5.298s
user    0m0.005s
sys 0m0.003s

Apparently some of my recent issues arise from the smelly armpit of the HTTP spec:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.2.3

I need to falcore to handle requests with this header but this type of request breaks the 1 request to 1 response paradigm that we have use for the pipeline design.

There are a few things we can do:

1. Have the server send the 100 response immediately to the client then pass the request to the pipeline. This is not exactly in the spirit of the "feature" since we never actually "determine if the origin server is willing to accept the request (based on the request headers)"
2. Run the pipeline twice. Once with the headers and let the pipeline/filters return the 100 status. The server detects a 100 and sends the response and then immediately runs the pipeline again with the same request minus the Expect header.
3. We expose the connection directly in falcore.Request, so let the filters send the 100 response manually then return the real response as normal. This seems a bit lazy to me since the average developer (aka me) would not know to handle it. But we could include an example of how to do it.

The Router stuff needs to be updated, clarified, and documented.

Original request:

I've been poring over the source for falcore for a while now and I can't work out how Routes work, and their relationship with pipelines.

How do you add a route to a server? Can a route select a pipeline? etc;

A good clear, but simple example would go a long way to helping newbs like myself understand how all this works.

P.S. While I'm at it, a good clear example of how the Proxy/Upstream stuff works would rule too :)

README.md refer to a not valid url blog

Hi guys,

This is more of a heads up, I've not quite got to the bottom of the problem yet.

After some tweaks to our core Go package RPM spec file, I was comfortable enough to re-enable building our Go repos for some other distros. A mime based file_filter test failed for the following distros:

https://build.opensuse.org/project/monitor?project=devel:languages:go

• CentOS 6
• RHEL 6
• ScientificLinux 6

Given that these three distro's are clones, I'll bet there's some problem with mime type detection on RHEL 6. If I find more, I'll update.

Cheers the noo,
Graham