The repository contains the sample code for the article

The repository contains the example of a microblogging service that uses a log-based architecture on Amazon MSK. It consists of a React app that allows you to publish and read articles. A service which implements the gRPC service to publish and subscribe to the messages (articles) that are created. And the AWS CloudFormation templates to run the service on the AWS cloud. The client uses gRPC for Web Clients to communicate with the pub/sub service.

• Protobuf protocol for the service
• React app with gRPC for Web Clients
• gRPC service to subscribe and publish articles
• AWS CloudFormation templates to run the service

The architecture for the service is provisioned by two CloudFormation stacks. A core stack that contains naive AWS components like VPC, NAT Gateway and Amazon MSK. And a second app stack, which provisions the app on Fargate with an Application Load Balancer

You can either run the example on

• AWS Cloud9
• Your machine.

We recommend using AWS Cloud9 to discover the example.

First, you need your own AWS account. Follow these steps if you do not have an account.

⚠️ Running the application will create and consume AWS resources. This will cost you money. Make sure you shut down/remove all resources once you are finished to avoid ongoing charges to your AWS account.

⚠️ The AWS Cloud9 workspace should be created by an IAM user with Administrator privileges, not the root account user. Please ensure that you are logged in as an IAM user, not the root account user.

• Open AWS Cloud9 in the AWS Console.
• Select Create environment to create a new workspace
• Name it mskworkshop, click Next step.
• Choose "t3.small" as Instance Type, take all default values and click Next Step
• On the overview double check your inputs and click Create Environment

💥 AWS Cloud9 provides a default auto-hibernation setting of 30 minutes for your Amazon EC2 instances created through Cloud9. With this setting, your EC2 instances automatically stop 30 minutes after you close the IDE and restart only when you reopen the IDE.

• When your workspace is ready, customize the environment by closing the Welcome tab, and opening up a new tab in the workspace.

• Your workspace should look like this now.

💥 If you prefer a different theme, you can choose one by selecting View > Themes > Solarized > Solarized Dark

• Run the following command to view the current version of the AWS CLI.

aws --version

• Update to the latest version.

pip install --user --upgrade awscli

• Confirm that you have a newer version running.

aws --version

The workshop needs some tools to be installed in the environment.

sudo yum install -y jq

This will install Node.js in your Cloud9 environment.

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash

Active the environment.

. ~/.nvm/nvm.sh

Use nvm to install a current version of Node.js.

nvm install node

Later we will use the yarn package manager for installing the client packages.

npm install yarn -g

⚠️ There is an extensive tutorial that walks through the steps of setting up Node.js on an Amazon EC2 instance.

You will need to clone the workshop to your AWS Cloud9 workspace.

# First, be sure you are in your environment directory
cd ~/environment

Clone the respository to your environment directory and change into the directory

git clone https://github.com/aws-samples/aws-msk-content-streaming aws-msk-content-streaming && cd $_

By default Cloud9 has 8GB storage attached. To build the containers you need more space.

make resize

This will resize your environment to 20GB storage.

If encounter an error that the /dev/nvme0n1 device does not exists, then this means you are not running on a Nitro-based architecture. Please replace the devices as explained here with the right ones.

Please run this command to generate SSH Key in Cloud9. This key will be used on the worker node instances to allow ssh access if necessary.

ssh-keygen

⚠️ Press enter 3 times to take the default choices

Upload the public key to your EC2 region.

aws ec2 import-key-pair --key-name ${C9_PROJECT} --public-key-material file://~/.ssh/id_rsa.pub

If you got an error similar to An error occurred (InvalidKey.Format) when calling the ImportKeyPair operation: Key is not in valid OpenSSH public key format then you can try this command instead.

aws ec2 import-key-pair --key-name ${C9_PROJECT} --public-key-material fileb://~/.ssh/id_rsa.pub

Set the environment variable for the KEY_PAIR.

export KEY_PAIR=${C9_PROJECT}

Running the script will deploy the application.

make deploy

This can take a while, as you will create a high available Kafka with Amazon MSK.

⚠️ The deploy scripts detect when you are running the deploy in a AWS Cloud9 workspace. It sets the PROJECT_NAME to your Cloud9 environment name, and extracts the AWS_ACCOUNT_ID and AWS_DEFAULT_REGION. You can override any of these variables

When the process is finished, you can start the React app. It will start the application with the REACT_APP_ENDPOINT environment variable which is set to the URL of the provisioned Application Load Balancer.

make start

When the application is finished to be installed you will see a message that it Compiled successfully!. You can access the preview by selecting Preview > Preview Running Runnin Application from the toolbar. This will open a new tab with the application.

⚠️ you cannot post new content yet. Because we do not have a custom domain, we have not enabled HTTPS with our service. You will have to access the preview URL with HTTP.

Either copy the full url (e.g. https://12345678910.vfs.cloud9.eu-west-1.amazonaws.com/) and replace https with http. Or click on the Pop Out Into New Window button next to the browser bar and then replace it.

You can now test it by creating an new item. Give it a title and add some content. If you have finished click Create Post.

You can access the app at localhost:3000 if you run the example on your local machine.

The last step is to cleanup your account. This will delete all created resources.

make delete

You can either run the example on your own machine, or run on AWS Cloud9. If you run it on your own machine you will have to install some additional tools.

• Docker
• AWS Command Line Interface
• Node.js and Yarn Package Manager
• Linux userland with bash
• GNU Make

You will have to clone the repository

git clone https://github.com/aws-samples/aws-msk-content-streaming aws-msk-content-streaming && cd $_

and manually set the environment variables for the project.

export PROJECT_NAME=<YOUR_PROJECT_NAME>
export AWS_ACCOUNT_ID=<YOUR_ACCOUNT_ID>
export AWS_DEFAULT_REGION=<YOUR_AWS_REGION>

You can then deploy the CloudFormation Stacks via.

make deploy

Start the development server at localhost:3000 if you run the example on your local machine.

make start

And jump to the cleanup if you have finished playing around with the example.

MIT