Git Product home page Git Product logo

greenplum-for-kubernetes's Introduction

Greenplum for Kubernetes

This repo houses the Helm charts as well as code and resources that go into the Docker images for a Greenplum on Kubernetes release.

git clone [email protected]:greenplum-db/greenplum-for-kubernetes.git

Table of Contents

Run Unit and Lint Tests

# runs only unit tests
make unit

# runs only linting
make lint

# runs unit and lint
make check

Deploy Greenplum for Kubernetes on Minikube

Setup

Install Minikube and it's pre-requisites by following instructions here. (If there are any problems, use minikube delete to delete the current minikube and then recreate it.)

Use HyperKit with minikube. It's much faster:

brew install docker-machine-driver-hyperkit
# Make sure to follow the `sudo chown` commands that brew outputs
minikube config set vm-driver hyperkit

Start minikube as below. You can adjust the resources to be lower if needed, but we recommend these resource settings for improved performance and enabling larger clusters.

minikube start --memory 8192 --cpus 8 --kubernetes-version=v1.16.7 --disk-size 80g

Confirm kubectl can access the minikube.

kubectl get nodes

NAME       STATUS    ROLES     AGE       VERSION
minikube   Ready     master    21d       v1.10.0

Authenticate with gcloud to pull required images from gcr.io

gcloud auth login
gcloud auth configure-docker

Build

Run the following command to change the local docker environment to point to the minikube docker. You only need to run this command once (per shell).

eval $(minikube docker-env)

(Note: to undo this docker setting in current shell, run eval "$(docker-machine env -u)")

Then, to build and upload the Greenplum images to minikube's docker registry.

make docker

The image should now have the tag greenplum-for-kubernetes:<version> as shown below:

$ docker images | grep greenplum
greenplum-operator                                  container-structure-test-in-docker                  2de1e0fa79b1        8 minutes ago       928MB
greenplum-operator                                  latest                                              24ca53fb90aa        8 minutes ago       111MB
greenplum-operator                                  v2.0.0-alpha.2.dev.7.gee6a7552                      24ca53fb90aa        8 minutes ago       111MB
greenplum-instance                                  container-structure-test-in-docker                  b90a82a6a746        10 minutes ago      928MB
greenplum-for-kubernetes                            latest                                              a2076cf10832        10 minutes ago      2.13GB
greenplum-for-kubernetes                            v2.0.0-alpha.2.dev.7.gee6a7552                      a2076cf10832        10 minutes ago      2.13GB

To clean up images:

# clean files and images directly created by our Makefile
make clean

# clean all the dangling docker images
make docker-clean

Deploy

Our make deploy target creates a regsecret used by the pods to download images. This requires a service account key for GCP. You must either place the key in a file named key.json inside the operator directory or a set the GCP_SVC_ACCT_KEY environment variable for this to work.

To install Greenplum cluster in the minikube environment, run the command below:

make deploy

You can access the Greenplum cluster with psql running in master-0:

kubectl exec -it master-0 -- bash -c "source /usr/local/greenplum-db/greenplum_path.sh; psql"

If you want to access the Greenplum service outside the minikube and you have a compatible "psql" executable in your path, you can do:

PGPORT=$(minikube service --format "{{.Port}}" --url greenplum) \
  PGHOST=$(minikube service --format "{{.IP}}" --url greenplum) \
  bash -c 'psql -U gpadmin -h $PGHOST -p $PGPORT'

To remove the Greenplum deployment:

make deploy-clean

Integration Test

The integration tests work by deploying Greenplum for Kubernetes and running checks against the cluster. Make sure you have performed the Build step before attempting to run the integration tests.

To run integration tests on minikube, execute the following command:

make integration

To run a specific integration test suite (from greenplum-operator/integration/), run:

make -C greenplum-operator integration-<suite> # e.g. `integration-ha`

Deploy Greenplum for Kubernetes on GKE

Setup

Create a cluster in GKE either with the web console or on the command line.

Run the following command to set the GKE cluster as your Kubernetes context for the command line

gcloud container clusters get-credentials <your-cluster-name> --project <your-project> --zone <cluster-zone>

Confirm kubectl can access the GKE cluster.

kubectl get nodes

NAME                                          STATUS   ROLES    AGE   VERSION
gke-test-default-pool-dba9314c-6xkg   Ready    <none>   20h   v1.15.9-gke.24
gke-test-default-pool-dba9314c-gjxc   Ready    <none>   20h   v1.15.9-gke.24
gke-test-default-pool-dba9314c-s83j   Ready    <none>   20h   v1.15.9-gke.24
gke-test-default-pool-dba9314c-tz4v   Ready    <none>   20h   v1.15.9-gke.24
gke-test-default-pool-dba9314c-wwc6   Ready    <none>   20h   v1.15.9-gke.24
gke-test-default-pool-dba9314c-x1cq   Ready    <none>   20h   v1.15.9-gke.24

Authenticate with gcloud to pull required images from gcr.io

gcloud auth login
gcloud auth configure-docker

Build

The following command will build the docker images for the Greenplum instance locally and then upload them to GCR so they can be used on your GKE cluster.

make gke-docker

The image should now have the tag greenplum-for-kubernetes:<version> as shown below:

$ docker images | grep greenplum
greenplum-operator                                  container-structure-test-in-docker                  2de1e0fa79b1        8 minutes ago       928MB
greenplum-operator                                  latest                                              24ca53fb90aa        8 minutes ago       111MB
greenplum-operator                                  v2.0.0-alpha.2.dev.7.gee6a7552                      24ca53fb90aa        8 minutes ago       111MB
gcr.io/gp-kubernetes/greenplum-operator             dev-24ca53fb90aa                                    24ca53fb90aa        8 minutes ago       111MB
greenplum-instance                                  container-structure-test-in-docker                  b90a82a6a746        10 minutes ago      928MB
greenplum-for-kubernetes                            latest                                              a2076cf10832        10 minutes ago      2.13GB
greenplum-for-kubernetes                            v2.0.0-alpha.2.dev.7.gee6a7552                      a2076cf10832        10 minutes ago      2.13GB
gcr.io/gp-kubernetes/greenplum-for-kubernetes       dev-a2076cf10832                                    a2076cf10832        10 minutes ago      2.13GB

A file will be created at /tmp/.gp-kubernetes_gke_image_tags containing the tags of the built images. This file will be used by the make gke-deploy target to reference the correct images.

To clean up images:

# clean files and images directly created by our Makefile
make clean

# clean all the dangling docker images
make docker-clean

Deploy

Our make gke-deploy target creates a regsecret used by the pods to download images. This requires a service account key for GCP. You must either place the key in a file named key.json inside the operator directory or a set the GCP_SVC_ACCT_KEY environment variable for this to work.

To install Greenplum cluster in the GKE environment, run the command below. This will set up parameters for the operator to use the docker images that were built with the gke-docker target.

make gke-deploy

You can access the Greenplum cluster with psql directly through the master pod with:

kubectl exec -it master-0 -- bash -c "source /usr/local/greenplum-db/greenplum_path.sh; psql"

To remove the Greenplum deployment:

make gke-deploy-clean

Integration Test

The integration tests work by deploying Greenplum for Kubernetes and running checks against the cluster. Make sure you have performed the Build step before attempting to run the integration tests.

To run integration tests on GKE, run the following command:

make gke-integration

To run a specific integration test suite (from greenplum-operator/integration/), run:

make -C greenplum-operator gke-integration-<suite> # e.g. `gke-integration-ha`

greenplum-for-kubernetes's People

Contributors

adamsdarlingtower avatar inviscid avatar khuddlefish avatar balinorliang avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.