Project for showcase of case study for Kubernetes based Project
To create a Kubernetes cluster creation configuration for kops with multiple instance groups, mixed instance groups, and lifecycle configurations (spot and on-demand), along with the cluster autoscaler enabled for all instance groups. Please follow the example below: In this example, I have defined three instance groups: ondemand-group, spot-group, and mixed-group. Each group specifies the minimum and maximum size, instance configurations, subnets, and the instance lifecycle (On-Demand or Spot). The mixed group includes a combination of On-Demand and Spot instances. Additionally, I have included the necessary IAM policies for autoscaling to work correctly. The clusterAutoscaler section enables the cluster autoscaler for all instance groups, with some configuration options like utilization thresholds and scan intervals.
Please note that we may need to adjust the instance types, AMIs, regions, and other parameters according to the specific requirements and environment.
Once we have created the configuration file, we can use kops to create the cluster using the following command:
kops create -f cluster-config.yaml
To achieve the requirements mentioned, follow the steps below:
Step 1: Create a Deployment with Nginx and the web application:
Create a file named deployment.yaml
and add the following content:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-webapp
spec:
replicas: 1
selector:
matchLabels:
app: nginx-webapp
template:
metadata:
labels:
app: nginx-webapp
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
volumeMounts:
- name: shared-files
mountPath: /usr/share/nginx/html
- name: webapp
image: your-web-application-image
ports:
- containerPort: 8080
volumeMounts:
- name: shared-files
mountPath: /usr/share/nginx/html
volumes:
- name: shared-files
emptyDir: {}
Replace your-web-application-image
with the actual image of the web application.
Step 2: Expose the application with a Service:
Create a file named service.yaml
and add the following content:
apiVersion: v1
kind: Service
metadata:
name: nginx-webapp-service
spec:
selector:
app: nginx-webapp
ports:
- protocol: TCP
port: 80
targetPort: 80
type: LoadBalancer
This will expose the service using a LoadBalancer type, which can be accessed externally.
Step 3: Implement autoscaling for the Deployment:
To implement autoscaling, we can use Kubernetes Horizontal Pod Autoscaler (HPA). Modify the deployment.yaml
file to add the HPA configuration:
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: nginx-webapp-autoscaler
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: nginx-webapp
minReplicas: 1
maxReplicas: 5
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 50
This configuration sets the minimum number of replicas to 1 and the maximum to 5. It also specifies that autoscaling should be based on CPU utilization, targeting an average utilization of 50%.
Step 4: Use Helm to render Kubernetes objects for reusability:
Helm is a package manager for Kubernetes that allows us to create reusable templates for Kubernetes objects. To use Helm, follow these steps:
-
Install Helm on our machine by following the Helm installation guide (https://helm.sh/docs/intro/install/).
-
Initialize Helm:
helm init
-
Create a Helm chart structure:
helm create mychart
-
Navigate to the chart directory:
cd mychart
-
Modify the generated
values.yaml
file to set our desired values for the deployment, service, and autoscaling. -
Create templates for the Deployment and Service in the
templates
directory. We can use the existing YAML files or split them into smaller template files. -
Render the Kubernetes objects using Helm:
helm template mychart/
This will generate the rendered YAML files based on our Helm chart.
We can customize the Helm chart and templates based on our specific needs.
That's it! We now have a deployment with Nginx and a web application running in the same pod, sharing public files through shared storage. The application is exposed with a Service object, and autoscaling is implemented. Ansible is used for basic configuration management, and Helm is used to render Kubernetes objects for reusability.
Here's an example of a web application using Python and Flask framework to parse and process CSV files, upload them to S3 storage, and implement S3 Glacier transition.
First, make sure we have Flask and Boto3 (AWS SDK for Python) installed. We can install them using pip:
pip install flask boto3
Next, create a new Python file, for example, app.py
, and add the following code:
from flask import Flask, render_template, request
import csv
import boto3
app = Flask(__name__)
# S3 Configuration
S3_BUCKET = "your-bucket-name"
S3_REGION = "us-east-1"
S3_GLACIER_TRANSITION_DAYS = 30 # Number of days to transition to S3 Glacier
# Initialize the S3 client
s3 = boto3.client("s3", region_name=S3_REGION)
@app.route("/")
def index():
return render_template("index.html")
@app.route("/process_csv", methods=["POST"])
def process_csv():
# Get the uploaded file
uploaded_file = request.files["csv_file"]
# Read and process the CSV file
csv_content = uploaded_file.read().decode("utf-8")
csv_lines = csv_content.split("\n")
# Print content of the lines to the browser
for line in csv_lines:
print(line)
# Upload the CSV file to S3
s3.upload_fileobj(uploaded_file, S3_BUCKET, uploaded_file.filename)
# Apply S3 Glacier transition
s3.put_object(
Bucket=S3_BUCKET,
Key=uploaded_file.filename,
StorageClass="GLACIER",
Transition={
"Days": S3_GLACIER_TRANSITION_DAYS,
"StorageClass": "GLACIER"
}
)
return "CSV file processed and uploaded to S3."
if __name__ == "__main__":
app.run()
In this code, we define the Flask routes for the homepage (/
) and the CSV processing endpoint (/process_csv
). The /
route renders an HTML template called index.html
, which we need to create in a templates
folder in the same directory as app.py
.
Create the templates/index.html
file and add the following HTML code:
<!DOCTYPE html>
<html>
<head>
<title>CSV Processor</title>
</head>
<body>
<h1>CSV Processor</h1>
<form action="/process_csv" method="post" enctype="multipart/form-data">
<input type="file" name="csv_file" accept=".csv" required>
<button type="submit">Upload and Process</button>
</form>
</body>
</html>
Make sure to replace "your-bucket-name"
with our actual S3 bucket name. Also, adjust the S3_REGION
if our bucket is located in a different region.
To run the web application, execute the following command in the terminal:
python app.py
The application will start, and we can access it in our web browser at http://localhost:5000
. You'll see a simple interface with an option to upload a CSV file.
When we upload a CSV file, it will be processed and the content of each line will be printed to the browser. The file will also be uploaded to the S3 bucket, and the S3 Glacier transition will be applied to the uploaded file.
Remember to have AWS credentials properly configured on the machine. We can use the AWS CLI or set the environment variables (AWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY
) for authentication.