Git Product home page Git Product logo

weasel's Introduction

yummy-weasel

A simple frontend (App Engine app) that serves content from a Google Cloud Storage (GCS) bucket, while allowing for:

  • HTTPS on custom domain
  • support for Service Worker (works only over HTTPS)
  • HTTP/2 push
  • more robust redirect from naked custom domain
  • serving directly from naked custom domain
  • keep the deployment/publishing flow using just the existing GCS bucket, which has proven to be fast and reliable, with partial content updates.
  • dynamic server-side logic

design

The design is simple. Suppose you have a static website www.example.com, served directly from a GCS gs://www.example.com.

A very simplified picture of the serving path can be shown as follows.

+-----+    GET http://www.example.com/dir/index.html
| GCS |  <-----------------------------------------+  visitor
+-----+                                                 :-/

Weasel enhancement consists of modifying the serving path to:

+-----+   (2) GET gs://www.example.com/dir/index.html
| GCS |  <------------------------------+--------+
+-----+                                 | yummy  |
                                        | weasel |
                                        +--------+
                                           ↑  |
                                      (1)  |  | (3)
         GET https://www.example.com/dir/  |  ↓ Link: <asset>; rel=preload

                                         visitor
                                           :-)
  1. A visitor requests the website page. Note that /index.html is now optional. Due to GCS restrictions, such suffixes were previously required for sub-folders, but with this approach they no longer need to be specified. Also, requests can (and will) be made over HTTPS.

  2. Weasel fetches the object content from the original GCS bucket and caches it locally. This step is necessary only if the object hasn't been cached already or the cache has expired. Cache expiration and invalidation is based on GCS object cache-control header settings.

  3. Weasel responds with the GCS object contents. Note that we can optionally push additional assets related to the requested file by using Link: <asset>; rel=preload header supported by GFE.

license

Apache License 2.0.

This is not an official Google product.

weasel's People

Contributors

mco-gh avatar x1ddos avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

weasel's Issues

app.yaml configuration example

hi,

if this is suppose to run on appengine, serving from static there should be an app.yaml example for deploying this?

if im incorrect could you point me in a direct how to deploy this repo?

Support for later App Engine Go runtimes

With Go 1.12+ runtimes, App Engine no longer supports calls to proprietary App Engine APIs, among many other changes. This has locked weasel to the Go 1.11 runtime. I would like to run this on the 1.14 runtime, and, from what I can see, these are the needed changes:

  • remove all uses of appengine packages and replace with standard Go packages & Google Cloud SDK
  • switch to Memorystore or a local cache (#7)

These changes will also make it easier to test locally.

Since this would be a breaking change, is this something you would be interested in? I would be happy to do the work if you are open to PRs.

Cache not flushed on updates

Each time I release a new version of the site I need to manually flush memcache from the console. Is that expected? It seems CacheKey should use the current version to build the key, this would guarantee that the correct files are being served.

Absolute redirects

There are a number of files in my GCS bucket that have moved or just don't exist anymore however there are outstanding external links to them.

It is fairly simple to add a section to config.json that lists "absolute" redirects where the redirect does not append the original request path.

I'm thinking something like:
config.json:

  "absolute-redirects": {
    "/getting-started.html": "https://goa.design/learn/guide",
    "/goagen.html": "https://goa.design/implement/goagen",
    "/vice-framework": "https://goa.design",
    "/swagger.html": "https://goa.design/design/swagger"
  },

The server code would then be tweaked to not append the original request path for these, something like:
server.go:

func init() {
// ...
    for path, redir := range config.AbsoluteRedirects {
        http.Handle(path, redirectHandler(redir, http.StatusMovedPermanently, true))
    }
    for host, redir := range config.Redirects {
        http.Handle(host, redirectHandler(redir, http.StatusMovedPermanently, false))
    }
//...
}

// ...

func redirectHandler(url string, code int, abs bool) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        u := url
        if !abs {
            u += r.URL.Path
        }
// ....

Would you consider a PR that does the above?

Proper 404 handling

I could be missing something but it doesn't seem like weasel handles 404s properly. The 404 configured in GCS doesn't seem to take effect and looking at the code I don't see where it would handle it.

404s could be handled by adding a section to the config, something like:
config.json

  "not-found": "404.html"

Which could be used by the server:
server.go

func serveObject(w http.ResponseWriter, r *http.Request) {
    if !weasel.ValidMethod(r.Method) {
        http.Error(w, "", http.StatusMethodNotAllowed)
        return
    }
    if _, force := config.tlsOnly[r.Host]; force && r.TLS == nil {
        u := "https://" + r.Host + r.URL.Path
        if r.URL.RawQuery != "" {
            u += "?" + r.URL.RawQuery
        }
        http.Redirect(w, r, u, http.StatusMovedPermanently)
        return
    }

    ctx := newContext(r)
    bucket := bucketForHost(r.Host)
    oname := r.URL.Path[1:]

    o, err := storage.OpenFile(ctx, bucket, oname)
    if err != nil {
        code := http.StatusInternalServerError
        if errf, ok := err.(*weasel.FetchError); ok {
            code = errf.Code
        }
        if code == http.StatusNotFound {  // *** START OF ADDED CODE ***
            o, err = storage.OpenFile(ctx, bucket, config.NotFound)
            if err == nil {
                goto serve
            }
        }   // *** END OF ADDED CODE ***
        serveError(w, code, "")
        if code != http.StatusNotFound {
            log.Errorf(ctx, "%s/%s: %v", bucket, oname, err)
        }
        return
    }
serve:  // *** ADDED ***
    if err := storage.ServeObject(w, r, o); err != nil {
        log.Errorf(ctx, "%s/%s: %v", bucket, oname, err)
    }
    o.Body.Close()
}

Would you consider a PR that does the above?

Example from server.go fails

I'm trying to run this with the example code from the comments in server/server.go but deploying it fails with the error message: app.go:7: undefined: weasel.Config

EDIT: Found it. Due to my inexperience with Go, I didn't know where to look first.

Anyway, the example code in the comment of server should be fixed to this:

```go
package app

import "github.com/google/weasel"
import "github.com/google/weasel/server"

func init() {
	conf := &server.Config{
		Storage: weasel.DefaultStorage,
		Buckets: map[string]string{
			"default": "<my GS bucket>",
		},
		HookPath: "/-/flush-gcs-cache",
	}
	server.Init(nil, conf)
}
```

goapp test fails

I'm getting the following when running goapp test:

❯ ~/go_appengine/goapp test
# github.com/goadesign/goa.design/appengine/server
package github.com/goadesign/goa.design/appengine/server (test)
        imports github.com/google/weasel/internal: use of internal package not allowed
FAIL    github.com/goadesign/goa.design/appengine/server [setup failed]

This used to work but it looks like the code is now using an internal package which goapp doesn't like...

LocalCache Implementation

I've integrated github.com/hashicorp/golang-lru as an in-memory local cache. Would you accept that as a PR?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.