ito-org / api-backend Goto Github PK

We need a function that fills up the number of reports when there are only a few of them. The reason is that you can't find out which reports come from one person alone by doing frequent queries.

General issue for discussion about scalability and other issues that come with it

• How do we make sure the API can handle the user loads, once the app goes live?
• How do we protect against DDoS attacks? Do we use a service like CloudFlare?
• How does the database cope with it?
• How does the TCN protocol deal with large user loads? From what I can tell, every client has to ask for every infected UUID. Depending on the scale, that's a lot of users requesting a lot of UUIDs.

There's no migration mechanism in place in case anything changes in the database schema, etc. Use https://github.com/golang-migrate/migrate or something similar for this.

A program flag or environment variable should be available for setting sslmode for Postgres. Right now, it's always set to "disable".

Currently, there's no validation and no rate limiting on the server.

Currently, the application is run by the root user in the container. This is a security issue and should be changed.

To decouple build and deployment, I would like to extend the CI pipeline with a job, that publishes the docker image to the project's GitHub docker registry. The idea is, that other repos can pick it up and implement the deployment.

This allows a separation of development and deployment, that I described here: #12.

Any thoughts about this?

I would like to pick up the discussion about hosting / cloud platforms / having a preproduction environment / etc, that came up in the last call. However, I take a step back here and focus on the underlying requirements for a hosting solution and how we might structure our repositories to integrate different/competing solutions.

Brainstorming:

• We don't know, who will host the backend in the future. It might be ito, it might be some public institution. At the moment, we cannot find the perfect fit for a hosting solution because we do not know the stakeholder requirements (e.g. no cloud hosting, regulations, etc). In this situation, it is probably a good idea to favor platform-agnostic solutions (like Docker) over platform-specific ones.
• There are however still good reasons to invest time and effort into hosting solutions right now:
  • We need something to pitch app+backend to interested adopters and journalists.
  • Hosting a staging / preproduction environment and using it with a staging build of the app is also a good idea in my opinion.
  • I can imagine, that stakeholders like governments/public institutions who want to officially adopt the "app" actually expect to get some kind of full-stack "turnkey solution" delivered. We would probably help them if we also provide a scalable hosting solution for their own infrastructure and/or some public cloud platform.
• However, when development will be done by ito but the hosting will be done by some other organization, we will have an organizational boundary between development and hosting. Maybe we should consider this in the design of our DevOps workflows, so that they also work with external hosting teams right from the beginning (e.g. how do we roll out a new backend version from ito to an external team).
• Looking at the already ongoing discussions and different solutions about where/how to host the backend (#11), it might make sense to build a repository structure that allow coexisting, competing solutions to be tried out.

So I thought about a way, how we can structure our repositories and workflows and came up with the following idea:

A new infrastructure repository:

• The repository provides a place for all kinds of infrastructure related assets:
  • Documentation
  • CI pipeline configurations for different cloud platforms
  • GitHub issues for infrastructure specific problems
• The idea is to separate development repositories like api-backend from infrastructure management. A deployment workflow for api-backend would look like this:
  • The CI pipeline of api-backend deploys a tagged docker image to the GitHub organization's docker registry and automatically opens a pull request on the infrastructure repo to use that new version.
  • Accepting the pull request triggers one or more infrastructure deployment pipelines, which will take the image and deploy it to some cloud platforms / servers.
• Competing solutions can coexist within the infrastructure repo. Which pipeline is run for which environment can be determined with some kind of combination of files, branches and pipeline triggers.
• Setting permissions for accepting pull requests on infrastructure is a way for restricting access to the infrastructure. Maybe it is necessary, that those permissions need to vary from those of the development repositories (e.g. maintainers on api-backend should just have contributor access on infrastructure).
• We can advertise the infrastructure repo as ito's turnkey solution for self-hosting. Other organizations can fork the repo and set up their own pipelines and deployment processes. ito can open pull requests on their infrastructure repository to hand over new versions of deployable artifacts (e.g. the backend).
• Because changes are applied via pull request, the git log of the infrastructure repo will become a public audit trail for all operational changes to the hosting infrastructure, which promotes transparency.
• The idea is based on the concept of GitOps.

Feel free to comment on this, please. If this idea finds some support, I would volunteer to spend some time on setting this up.

Add GitHub Actions workflow.

• Disable/Remove Python workflow
• Build Go code
• Run Postgres in Docker
• Run tests

When trying to build, the following error occurs gets emitted:

./main.go:57:14: undefined: StartServer

Reproduce

❯ docker run --rm -it -v $(pwd):/root golang:1.14 sh
# cd /root
# ls
CONTRIBUTING.md  LICENSE  README.md  db.go  go.mod  go.sum  main.go  server.go	server_test.go	tcn
# go get -v -t -d ./...
go: downloading github.com/gin-gonic/gin v1.6.2
go: downloading github.com/jmoiron/sqlx v1.2.0
go: downloading github.com/urfave/cli v1.22.4
go: downloading github.com/stretchr/testify v1.5.1
go: downloading github.com/mattn/go-isatty v0.0.12
go: downloading github.com/gin-contrib/sse v0.1.0
go: downloading github.com/golang/protobuf v1.4.0
go: downloading gopkg.in/yaml.v2 v2.2.8
go: downloading github.com/ugorji/go v1.1.7
go: downloading github.com/go-playground/validator/v10 v10.2.0
go: downloading github.com/json-iterator/go v1.1.9
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading golang.org/x/sys v0.0.0-20200413165638-669c56c373c4
go: downloading github.com/ugorji/go/codec v1.1.7
go: downloading github.com/cpuguy83/go-md2man/v2 v2.0.0
go: downloading google.golang.org/protobuf v1.21.0
go: downloading github.com/go-playground/universal-translator v0.17.0
go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go: downloading github.com/modern-go/reflect2 v1.0.1
go: downloading github.com/russross/blackfriday/v2 v2.0.1
go: downloading github.com/leodido/go-urn v1.2.0
go: downloading github.com/go-playground/locales v0.13.0
go: downloading github.com/shurcooL/sanitized_anchor_name v1.0.0
# go build -v .
github.com/go-playground/locales/currency
google.golang.org/protobuf/internal/flags
google.golang.org/protobuf/internal/fieldnum
github.com/gin-gonic/gin/internal/bytesconv
github.com/gin-gonic/gin/internal/json
github.com/gin-contrib/sse
github.com/go-playground/locales
github.com/leodido/go-urn
google.golang.org/protobuf/internal/detrand
github.com/go-playground/universal-translator
google.golang.org/protobuf/internal/errors
google.golang.org/protobuf/internal/pragma
google.golang.org/protobuf/internal/set
google.golang.org/protobuf/encoding/protowire
google.golang.org/protobuf/internal/genname
google.golang.org/protobuf/internal/version
github.com/ugorji/go/codec
google.golang.org/protobuf/reflect/protoreflect
gopkg.in/yaml.v2
github.com/go-playground/validator/v10
google.golang.org/protobuf/reflect/protoregistry
google.golang.org/protobuf/internal/encoding/messageset
google.golang.org/protobuf/internal/strs
google.golang.org/protobuf/internal/mapsort
google.golang.org/protobuf/internal/encoding/text
google.golang.org/protobuf/internal/fieldsort
google.golang.org/protobuf/runtime/protoiface
google.golang.org/protobuf/internal/descfmt
google.golang.org/protobuf/proto
google.golang.org/protobuf/internal/descopts
golang.org/x/sys/unix
google.golang.org/protobuf/internal/encoding/defval
github.com/ito-org/go-backend/tcn
github.com/jmoiron/sqlx/reflectx
google.golang.org/protobuf/encoding/prototext
google.golang.org/protobuf/internal/filedesc
github.com/jmoiron/sqlx
github.com/mattn/go-isatty
github.com/shurcooL/sanitized_anchor_name
github.com/russross/blackfriday/v2
google.golang.org/protobuf/internal/encoding/tag
google.golang.org/protobuf/internal/impl
github.com/cpuguy83/go-md2man/v2/md2man
github.com/urfave/cli
google.golang.org/protobuf/internal/filetype
google.golang.org/protobuf/runtime/protoimpl
github.com/golang/protobuf/proto
github.com/gin-gonic/gin/binding
github.com/gin-gonic/gin/render
github.com/gin-gonic/gin
github.com/ito-org/go-backend
# github.com/ito-org/go-backend
./main.go:57:14: undefined: StartServer