Terraform msk

Terraform msk DEFAULT


In my previous post — Install a Kafka Cluster on Ubuntu in AWS, I have detailed the steps to manually set up your messaging backbone in AWS. This is a great step to understand how Kafka works, but it takes a long time and is not scalable.

A Kafka Cluster has several components such as Brokers, Zookeepers, Workers running on multiple instances like the example below. The simplest way to provision, deploy and manage your platform is by using tools that will automate this task.

On this post, Iwill explain how can we use Terraform to quickly provision multiple instances and then install Kafka on these instances using Ansible to set up your Cluster.

First, let’s quickly understand what is Terraform and Ansible in a high-level view

Terraform is an open-sourceinfrastructure as code software tool created by HashiCorp. It enables users to define and provision infrastructure using a high-level configuration language known as Hashicorp Configuration Language (HCL), or optionally JSON.

Terraform supports a number of cloud infrastructure providers such as Amazon Web Services, Google Cloud Platform, Microsoft Azure.

Terraform generates an execution plan describing what it will do to reach the desired state, and then executes it to build the described infrastructure. As the configuration changes, Terraform is able to determine what changed and create incremental execution plans which can be applied.

Infrastructure is described using a high-level configuration syntax. This allows a blueprint of your datacenter to be versioned and treated as you would any other code. Additionally, infrastructure can be shared and re-used.

Ansible is a radically simple IT automation platform that makes your applications and systems easier to deploy. Avoid writing scripts or custom code to deploy and update your applications — automate in a language that approaches plain English, using SSH, with no agents to install on remote systems.

First, let’s install Terraform and Ansible


You can follow the instructions below or if you are on Mac, the simplest way is using brew:

brew install terraform

Confirm Terraform is correctly installed with:

terraform -v


Install Ansible using pip command. PIP is a for Python packages. macOS doesn’t have pip installed by default. You have to install it first.

sudo easy_install pip

Use pip to install Ansible

sudo pip install ansible

Let’s verify Ansible is installed

ansible --version

Now, let’s build the Kafka infrastructure using Terraform HCL.

The setup is based on the Enterprise Confluent Kafka using a public VPC and subnet.

All code is available in my GitHub

git clone [email protected]:mlomboglia/kafka-cluster-infra.git

Inside the terraform folder, there are the following files:

gateway.tf → Internet Gateway creation. There is no need to change this file

instances.tf → Instances to be created. There is no need to change this file

There are five types of instances with a short name in parenthesis:

  • Kafka Brokers (broker) → Host the Kafka broker service and maintain topic data
  • Zookeeper (zookeeper) →Host the quorum management service and manage topic metadata
  • Kafka Connect REST API (connect) → Kafka Connect, an open-source component of Kafka, is a framework for connecting Kafka with external systems such as databases, key-value stores, search indexes, and file systems.
  • REST Proxy (rest) → REST and HTTP/HTTPS access to additional Confluent Platform services
  • KSQL Server REST API (ksql) →KSQL is the streaming SQL engine that enables real-time data processing
  • Schema Registry REST API (schema) →The Schema Registry REST server uses content types for both requests and responses to indicate the serialization format of the data as well as the version of the API being used
  • Control Center (control_center) → Confluent Control Center is a web-based tool for managing and monitoring Apache Kafka®. Control Center facilitates building and monitoring production data pipelines and streaming applications.

public.tf → Subnet creation. For this example, I am using a public subnet

route.tf → Route table information

security_group.tf → Security group settings to allow for necessary network traffic into the instances. There is no need to change this file for the example.

Please keep in mind that ingress allows, so this is open to the public internet. You will need to restrict this to be used in a production environment.

vpc.tf → VPC creation

variables.tf contain the parameters for our terraform. You can modify this. For my example, I will provision 7 instances in AWS.

“rest” = 1
“connect” = 1
“ksql” = 1
“schema” = 1
“control_center” = 1
“broker” = 1
“zookeeper” = 1

With a prefix “staging”

Intance Type “t2.small” .

Important: Confluent recommends a minimum instance type “t2.large” to run Kafka, so if you modify this it will be out of AWS Free Tier and it will cost you money

Key Pair.

You will need an AWS Key Pair to create your instances. If you don’t have one yet, log in to your AWS account

EC2 -> Create Key Pair

Give a name and click Create.

Download your key, copy to a safe folder and update permissions

chmod 400 ~/.ssh/<Key Name>.pem

Update the section below of the variables.tf with your Key Name

variable "key_name" {
description = "Key Pair"
default = "<Your Key Name>"

Now, we are good to go.

Terraform init will download your dependencies

cd kafka-cluster-infra
terraform init

Terraform plan will create your deployment plan

terraform plan

Now, be careful on the next step. It will cost you money depending on your configuration.

terraform apply

Confirm with “yes”

if there is an error: ami is not found, go to your EC2 console, launch instance and update with the latest ami label:

Now you have provisioned all the instances in AWS.

Go your AWS and see all your instances running and take note of the public DNS (IPv4) of each instance. You will need this in the next section.

Now, let’s proceed to install Kafka using Ansible

To install Kafka, I will use the Confluent version.

1. Create a copy of the file.

cd ..
cd cp-ansible

cp hosts_example.yml hosts.yml

2. Use your favourite IDE and edit hosts.yml file entering the hosts from your AWS Terraform installation above following the labels for each instance type, looking similar to this with your Public DNS.

Be very careful to use correct indentation on YML files:

Uncomment the following line in the hosts.yml file adding your key path

ansible_ssh_private_key_file: '~/.ssh/<Key Path>.pem'

3. Confirm Ansible can connect over SSH

ansible -i hosts.yml all -m ping

Type “yes” to trust all servers.

If there is an issue connecting to an instance, try to SSH manually with the command below:

ssh -i <Key Path> [email protected]<Instance Public DNS>

4. Run the playbook

ansible-playbook -i hosts.yml all.yml

The Ansible playbook will perform the Kafka installation on all your instances.

After some time, you should see this message above. Everything installed!

Now, let’s test the installation:

  1. You can SSH to any of your instances to check the services are running using:
systemctl status confluent*
Sours: https://towardsdatascience.com/deploy-a-kafka-cluster-with-terraform-and-ansible-21bee1ee4fb

IAM-based client authentication

Describe the Feature

AWS have released IAM-based SASL security for MSK. It'd be great if we could configure support for this via the Cloudposse module.

Expected Behavior

Users should be able to specify IAM-based SASL security for clients as an alternative to SASL-SCRAM.

Use Case

We're currently looking at deploying an MSK cluster and would like the ability to configure client access via IAM instead of via SCRAM credentials in Secrets Manager.

Describe Ideal Solution

An option in the Cloudposse module to enable IAM-based SASL security that calls with the correct SASL options to enable IAM-based security.

Alternatives Considered

SASL-SCRAM is an existing client auth option but there's currently no way to update a SCRAM cluster to use IAM-based client security. IAM-based client security would add value for users that have a unified IAM-based security architecture.

Additional Context

Happy to raise a PR for this although I am (like everyone) tight for time and not a tf expert.

Sours: https://issueexplorer.com/issue/cloudposse/terraform-aws-msk-apache-kafka-cluster/20
  1. Kingston pa zillow
  2. Vans check platform
  3. Best investment accounts reddit

Terraform by HashiCorp¶

General information¶

You can use Terraform with an AWS provider or later to work with the EC2 API of the CROC Cloud,


You can see more examples of using Terraform in our official terraform-examples repository on GitHub. Here you also can find out supported and non-supported parameters for each resource.

Installation and configuration¶

Download and unpack Terraform executable file version for the target OS and CPU architecture.

Next, prepare a Terraform configuration file. For correct operation, specify the following parameters in the block:

variable "access_key"{} variable "secret_key"{} variable "region"{default="croc"} provider "aws"{ endpoints {ec2="https://api.cloud.croc.ru"}# NOTE: STS API is not implemented, skip validationskip_credentials_validation=true# NOTE: IAM API is not implemented, skip validationskip_requesting_account_id=true# NOTE: Region has different name, skip validationskip_region_validation=trueaccess_key= var.access_key secret_key= var.secret_key region= var.region }

Working with cloud resources¶

Create an instance from the specified template¶

When creating instances, specify the template ID, instance type, and subnet ID. Value of the parameter must be set to .

All instances in CROC Cloud run with monitoring enabled, and it cannot be disabled. The sourceDestCheck function is enabled by default for all newly created interfaces in subnets

resource "aws_instance""web1"{ami="cmi-012345678"instance_type="m1.medium"subnet_id="subnet-012345678"monitoring=true} resource "aws_instance""backend1"{ami="cmi-012345678"instance_type="m1.large"subnet_id="subnet-012345678"monitoring=truesource_dest_check=false}

Create a volume¶

and either (GiB) or are required.

resource "aws_ebs_volume""web1_volume1"{availability_zone="ru-msk-comp1p"size=1type="st2"} resource "aws_ebs_volume""backend1_volume1"{availability_zone="ru-msk-comp1p"size=32type="io2"iops=1600}

Create an instance snapshot¶

and are required. Instance must be stopped.

resource "aws_ami_from_instance""web1snap1"{name="test_name"source_instance_id="i-01234567"description="snap description"}

How to handle keys¶

The field is required.

resource "aws_key_pair""key-for-deploy"{key_name="deployer"public_key="ssh-rsa ......"}

Placement groups¶

Only strategy is available for use.

resource "aws_placement_group""web"{name="web-servers"strategy="distribute"}

Create a subnet and instance in VPC¶

data "aws_vpc""default_vpc"{default=true} resource "aws_subnet""subnet1"{vpc_id= data.aws_vpc.default_vpc.id cidr_block=""} resource "aws_instance""instance1"{ami="cmi-D7368411"instance_type="m1.2small"subnet_id= aws_subnet.subnet1.id depends_on=["aws_subnet.subnet1"]monitoring=truesource_dest_check=false}

© Copyright 2021, CROC Incorporated.

Built with Sphinx using a theme provided by Read the Docs.
Sours: https://docs.cloud.croc.ru/en/api/tools/terraform.html
Creating a module in Terraform - Getting started with Terraform Modules (part 1)

Securely Decoupling Kubernetes-based Applications on Amazon EKS using Kafka with SASL/SCRAM

Securely decoupling Go-based microservices on Amazon EKS using Amazon MSK with IRSA, SASL/SCRAM, and data encryption

As organizations scale and mature, they frequently endeavor to move away from a monolithic application architecture toward a distributed, microservices-based paradigm. As part of this transition, organizations regularly embrace modern programming languages and frameworks, adopt containerization, acquire a preference for open-source software components, and opt for asynchronous event-driven communication models. Regardless of the final architecture, organizations must continuously maintain a high level of application and infrastructure security.

This post will explore a simple Go-based application deployed to Kubernetes using Amazon Elastic Kubernetes Service (Amazon EKS). The microservices that comprise the application communicate asynchronously by producing and consuming events from Amazon Managed Streaming for Apache Kafka (Amazon MSK).

Authentication and Authorization for Apache Kafka

According to AWS, you can use IAM to authenticate clients and to allow or deny Apache Kafka actions. Alternatively, you can use TLS or SASL/SCRAM to authenticate clients, and Apache Kafka ACLs to allow or deny actions.

For this post, our Amazon MSK cluster will use SASL/SCRAM (Simple Authentication and Security Layer/Salted Challenge Response Mechanism) username and password-based authentication to increase security. Credentials used for SASL/SCRAM authentication will be securely stored in AWS Secrets Manager and encrypted using AWS Key Management Service (KMS).

Data Encryption

Data at rest in the MSK cluster will be encrypted at rest using Amazon MSK’s integration with AWS KMS to provide transparent server-side encryption. Encryption in transit of data moving between the brokers of the MSK cluster will be provided using Transport Layer Security (TLS 1.2).

Resource Management

AWS resources for Amazon MSK will be created and managed using HashiCorp Terraform, a popular open-source infrastructure-as-Code (IaC) software tool. Amazon EKS resources will be created and managed with , the official CLI for Amazon EKS sponsored by Weaveworks. Lastly, Kubernetes resources will be created and managed with Helm, the open-source Kubernetes package manager.

Demonstration Application

The Go-based microservices, which compose the demonstration application, will use Segment’s popular client. Segment is a leading customer data platform (CDP). The microservices will access Amazon MSK using Kafka broker connection information stored in AWS Systems Manager (SSM) Parameter Store.

Source Code

All source code for this post, including the demonstration application, Terraform, and Helm resources, are open-sourced and located on GitHub.


To follow along with this post’s demonstration, you will need recent versions of , , and installed and accessible from your terminal. Optionally, it will be helpful to have or , , and the AWS CLI v2 ().

To demonstrate the EKS and MSK features described above, we will proceed as follows:

  1. Deploy the EKS cluster and associated resources using ;
  2. Deploy the MSK cluster and associated resources using Terraform;
  3. Update the route tables for both VPCs and associated subnets to route traffic between the peered VPCs;
  4. Create IAM Roles for Service Accounts (IRSA) allowing access to MSK and associated services from EKS, using ;
  5. Deploy the Kafka client container to EKS using Helm;
  6. Create the Kafka topics and ACLs for MSK using the Kafka client;
  7. Deploy the Go-based application to EKS using Helm;
  8. Confirm the application’s functionality;

1. Amazon EKS cluster

To begin, create a new Amazon EKS cluster using Weaveworks’ . The default configuration file included in the project will create a small, development-grade EKS cluster based on Kubernetes 1.20 in . The cluster will contain a managed node group of three Amazon Linux 2 EC2 worker nodes. The EKS cluster will be created in a new VPC.

Set the following environment variables and then run the command to create the new EKS cluster and associated infrastructure.

export AWS_ACCOUNT=$(aws sts get-caller-identity \
--output text --query 'Account')
export EKS_REGION="us-east-1"
export CLUSTER_NAME="eks-kafka-demo"eksctl create cluster -f ./eksctl/cluster.yaml

In my experience, it could take up to 25-40 minutes to fully build and configure the new 3-node EKS cluster.

As part of creating the EKS cluster, will automatically deploy three AWS CloudFormation stacks containing the following resources:

  1. Amazon Virtual Private Cloud (VPC), subnets, route tables, NAT Gateways, security policies, and the EKS control plane;
  2. EKS managed node group containing Kubernetes three worker nodes;
  3. IAM Roles for Service Accounts (IRSA) that maps an AWS IAM Role to a Kubernetes Service Account;

Once complete, update your file so that you can connect to the new Amazon EKS cluster using the following AWS CLI command:

aws eks --region ${EKS_REGION} update-kubeconfig \
--name ${CLUSTER_NAME}

Review the details of the new EKS cluster using the following command:

eksctl utils describe-stacks \
--region ${EKS_REGION} --cluster ${CLUSTER_NAME}

Review the new EKS cluster in the Amazon Container Services console’s Amazon EKS Clusters tab.

Below, note the EKS cluster’s OpenID Connect URL. Support for IAM Roles for Service Accounts (IRSA) on the EKS cluster requires an OpenID Connect issuer URL associated with it. OIDC was configured in the file; see line 8 (shown above).

The OpenID Connect identity provider, referenced in the EKS cluster’s console, created by , can be observed in the IAM Identity provider console.

2. Amazon MSK cluster

Next, deploy the Amazon MSK cluster and associated network and security resources using HashiCorp Terraform.

Before creating the AWS infrastructure with Terraform, update the location of the Terraform state. This project’s code uses Amazon S3 as a backend to store the Terraform’s state. Change the Amazon S3 bucket name to one of your existing buckets, located in the file.

terraform {
backend "s3" {
bucket = "terrform-us-east-1-your-unique-name"
key = "dev/terraform.tfstate"
region = "us-east-1"

Also, update the variable in the file with the VPC ID of the EKS VPC created by in step 1.

variable "eks_vpc_id" {
default = "vpc-your-id"

The quickest way to obtain the ID of the EKS VPC is by using the following AWS CLI v2 command:

aws ec2 describe-vpcs --query 'Vpcs[].VpcId' \
--filters Name=tag:Name,Values=eksctl-eks-kafka-demo-cluster/VPC \
--output text

Next, initialize your Terraform backend in Amazon S3 and initialize the latest provider plugin with .

Use to generate an execution plan, showing what actions Terraform would take to apply the current configuration. Terraform will create approximately 25 AWS resources as part of the plan.

Finally, use to create the Amazon resources. Terraform will create a small, development-grade MSK cluster based on Kafka 2.8.0 in , containing a set of three broker nodes. Terraform will create the MSK cluster in a new VPC. The broker nodes are spread across three Availability Zones, each in a private subnet, within the new VPC.

It could take 30 minutes or more for Terraform to create the new cluster and associated infrastructure. Once complete, you can view the new MSK cluster in the Amazon MSK management console.

Below, note the new cluster’s ‘Access control method’ is SASL/SCRAM authentication. The cluster implements encryption of data in transit with TLS and encrypts data at rest using a customer-managed customer master key (CMS) in AWM KSM.

Below, note the ‘Associated secrets from AWS Secrets Manager.’ The secret, , contains the SASL/SCRAM authentication credentials — username and password. These are the credentials the demonstration application, deployed to EKS, will use to securely access MSK.

The SASL/SCRAM credentials secret shown above can be observed in the AWS Secrets Manager console. Note the customer-managed customer master key (CMK), stored in AWS KMS, which is used to encrypt the secret.

3. Update route tables for VPC Peering

Terraform created a VPC Peering relationship between the new EKS VPC and the MSK VPC. However, we will need to complete the peering configuration by updating the route tables. We want to route all traffic from the EKS cluster destined for MSK, whose VPC CIDR is , through the VPC Peering Connection resource. There are four route tables associated with the EKS VPC. Add a new route to the route table whose name ends with ‘’, for example, . Manually create the required route in this route table using the VPC console’s Route tables tab, as shown below (new route shown second in list).

Similarly, we want to route all traffic from the MSK cluster destined for EKS, whose CIDR is , through the same VPC Peering Connection resource. Update the single MSK VPC’s route table using the VPC console’s Route tables tab, as shown below (new route shown second in list).

4. Create IAM Roles for Service Accounts (IRSA)

With both the EKS and MSK clusters created and peered, we are ready to start deploying Kubernetes resources. Create a new namespace, , which will hold the demonstration application and Kafka client pods.

export AWS_ACCOUNT=$(aws sts get-caller-identity \
--output text --query 'Account')
export EKS_REGION="us-east-1"
export CLUSTER_NAME="eks-kafka-demo"
export NAMESPACE="kafka"kubectl create namespace $NAMESPACE

Then using , create two IAM Roles for Service Accounts (IRSA) associated with Kubernetes Service Accounts. The Kafka client’s pod will use one of the roles, and the demonstration application’s pods will use the other role. According to the eksctl documentation, IRSA works via IAM OpenID Connect Provider (OIDC) that EKS exposes, and IAM roles must be constructed with reference to the IAM OIDC Provider described earlier in the post, and a reference to the Kubernetes Service Account it will be bound to. The two IAM policies referenced in the commands below were created earlier by Terraform.

# kafka-demo-app role
eksctl create iamserviceaccount \
--name kafka-demo-app-sasl-scram-serviceaccount \
--namespace $NAMESPACE \
--region $EKS_REGION \
--cluster $CLUSTER_NAME \
--attach-policy-arn "arn:aws:iam::${AWS_ACCOUNT}:policy/EKSScramSecretManagerPolicy" \
--approve \
--override-existing-serviceaccounts# kafka-client-msk role
eksctl create iamserviceaccount \
--name kafka-client-msk-sasl-scram-serviceaccount \
--namespace $NAMESPACE \
--region $EKS_REGION \
--cluster $CLUSTER_NAME \
--attach-policy-arn "arn:aws:iam::${AWS_ACCOUNT}:policy/EKSKafkaClientMSKPolicy" \
--attach-policy-arn "arn:aws:iam::${AWS_ACCOUNT}:policy/EKSScramSecretManagerPolicy" \
--approve \
--override-existing-serviceaccounts# confirm successful creation of accounts
eksctl get iamserviceaccount \
--cluster $CLUSTER_NAME \
--namespace $NAMESPACEkubectl get serviceaccounts -n $NAMESPACE

Recall created three CloudFormation stacks initially. With the addition of the two IAM Roles, we now have a total of five CloudFormation stacks deployed.

5. Kafka client

Next, deploy the Kafka client using the project’s Helm chart, . We will use the Kafka client to create Kafka topics and Apache Kafka ACLs. This particular Kafka client is based on a custom Docker Image that I have built myself using an Alpine Linux base image with Java OpenJDK 17, . The image contains the latest Kafka client along with the AWS CLI v2 and a few other useful tools like . If you prefer an alternative, there are multiple Kafka client images available on Docker Hub.

The Kafka client only requires a single pod. Run the following commands to deploy the Kafka client to EKS using the project’s Helm chart, :

cd helm/# perform dry run to validate chart
helm install kafka-client-msk ./kafka-client-msk \
--namespace $NAMESPACE --debug --dry-run# apply chart resources
helm install kafka-client-msk ./kafka-client-msk \
--namespace $NAMESPACE

Confirm the successful creation of the Kafka client pod with either of the following commands:

kubectl get pods -n kafkakubectl describe pod -n kafka -l app=kafka-client-msk

The ability of the Kafka client to interact with Amazon MSK, AWS SSM Parameter Store, and AWS Secrets Manager is based on two IAM policies created by Terraform, and . These two policies are associated with a new IAM role, which in turn, is associated with the Kubernetes Service Account, . This service account is associated with the Kafka client pod as part of the Kubernetes Deployment resource in the Helm chart.

6. Kafka topics and ACLs for Kafka

Use the Kafka client to create Kafka topics and Apache Kafka ACLs. First, use the command to execute commands from within the Kafka client container.

kubectl get pods -n kafka -l app=kafka-client-msk | \
awk 'FNR == 2 {print $1}')kubectl exec -it $KAFKA_CONTAINER -n kafka -- bash

Once successfully attached to the Kafka client container, set the following three environment variables: 1) Apache ZooKeeper connection string, 2) Kafka bootstrap brokers, and 3) ‘Distinguished-Name’ of the Bootstrap Brokers (see AWS documentation). The values for these environment variables will be retrieved from AWS Systems Manager (SSM) Parameter Store. The values were stored in the Parameter store by Terraform during the creation of the MSK cluster. Based on the policy attached to the IAM Role associated with this Pod (IRSA), the client has access to these specific parameters in the SSM Parameter store.

export ZOOKPR=$(\
aws ssm get-parameter --name /msk/scram/zookeeper \
--query 'Parameter.Value' --output text)export BBROKERS=$(\
aws ssm get-parameter --name /msk/scram/brokers \
--query 'Parameter.Value' --output text)export DISTINGUISHED_NAME=$(\
echo $BBROKERS | awk -F',' '{print $1}' | sed 's/b-1/*/g')

Use the and commands to verify the environment variables have been retrieved and constructed properly. Your Zookeeper and Kafka bootstrap broker URLs will be uniquely different from the ones shown below.


To test the connection between EKS and MSK, list the existing Kafka topics, from the Kafka client container:

bin/kafka-topics.sh --list --zookeeper $ZOOKPR

You should see three default topics, as shown below.

If you did not properly add the new VPC Peering routes to the appropriate route tables in the previous step, establishing peering of the EKS and MSK VPCs, you are likely to see a timeout error while attempting to connect. Go back and confirm that both of the route tables are correctly updated with the new routes.

Kafka Topics, Partitions, and Replicas

The demonstration application produces and consumes messages from two topics, and . Each topic will have three partitions, one for each of the three broker nodes, along with three replicas.

Use the following commands from the client container to create the two new Kafka topics. When complete, confirm the creation of the topics using the option again.

bin/kafka-topics.sh --create --topic foo-topic \
--partitions 3 --replication-factor 3 \
--zookeeper $ZOOKPRbin/kafka-topics.sh --create --topic bar-topic \
--partitions 3 --replication-factor 3 \
--zookeeper $ZOOKPRbin/kafka-topics.sh --list --zookeeper $ZOOKPR

Review the details of the topics using the option. Note the three partitions per topic and the three replicas per topic.

bin/kafka-topics.sh --describe --topic foo-topic --zookeeper $ZOOKPRbin/kafka-topics.sh --describe --topic bar-topic --zookeeper $ZOOKPR

Kafka ACLs

According to Kafka’s documentation, Kafka ships with a pluggable Authorizer and an out-of-box authorizer implementation that uses Zookeeper to store all the Access Control Lists (ACLs). Kafka ACLs are defined in the general format of “Principal P is [Allowed/Denied] Operation O From Host H On Resource R.” You can read more about the ACL structure on KIP-11. To add, remove or list ACLs, you can use the Kafka authorizer CLI.

Authorize access by the Kafka brokers and the demonstration application to the two topics. First, allow access to the topics from the brokers using the environment variable (see AWS documentation).

# read auth for brokers
bin/kafka-acls.sh \
--authorizer-properties zookeeper.connect=$ZOOKPR \
--add \
--allow-principal "User:CN=${DISTINGUISHED_NAME}" \
--operation Read \
--group=consumer-group-B \
--topic foo-topic
bin/kafka-acls.sh \
--authorizer-properties zookeeper.connect=$ZOOKPR \
--add \
--allow-principal "User:CN=${DISTINGUISHED_NAME}" \
--operation Read \
--group=consumer-group-A \
--topic bar-topic
# write auth for brokers
bin/kafka-acls.sh \
--authorizer-properties zookeeper.connect=$ZOOKPR \
--add \
--allow-principal "User:CN=${DISTINGUISHED_NAME}" \
--operation Write \
--topic foo-topic
bin/kafka-acls.sh \
--authorizer-properties zookeeper.connect=$ZOOKPR \
--add \
--allow-principal "User:CN=${DISTINGUISHED_NAME}" \
--operation Write \
--topic bar-topic

The three instances (replicas/pods) of Service A, part of , produce messages to the and consume messages from the . Conversely, the three instances of Service B, part of , produce messages to the and consume messages from the .

Allow access to the appropriate topics from the demonstration application’s microservices. First, set the environment variable — the MSK cluster’s SASL/SCRAM credential’s username, stored in AWS Secrets Manager by Terraform. We can retrieve the username from Secrets Manager and assign it to the environment variable with the following command.

export USER=$(
aws secretsmanager get-secret-value \
--secret-id AmazonMSK_credentials \
--query SecretString --output text | \
jq .username | sed -e 's/^"//' -e 's/"$//')

Create the appropriate ACLs.

# producers
bin/kafka-acls.sh \
--authorizer kafka.security.auth.SimpleAclAuthorizer \
--authorizer-properties zookeeper.connect=$ZOOKPR \
--add \
--allow-principal User:$USER \
--producer \
--topic foo-topic
bin/kafka-acls.sh \
--authorizer kafka.security.auth.SimpleAclAuthorizer \
--authorizer-properties zookeeper.connect=$ZOOKPR \
--add \
--allow-principal User:$USER \
--producer \
--topic bar-topic
# consumers
bin/kafka-acls.sh \
--authorizer kafka.security.auth.SimpleAclAuthorizer \
--authorizer-properties zookeeper.connect=$ZOOKPR \
--add \
--allow-principal User:$USER \
--consumer \
--topic foo-topic \
--group consumer-group-B
bin/kafka-acls.sh \
--authorizer kafka.security.auth.SimpleAclAuthorizer \
--authorizer-properties zookeeper.connect=$ZOOKPR \
--add \
--allow-principal User:$USER \
--consumer \
--topic bar-topic \
--group consumer-group-A

To list the ACLs you just created, use the following commands:

# list all ACLs
bin/kafka-acls.sh \
--authorizer kafka.security.auth.SimpleAclAuthorizer \
--authorizer-properties zookeeper.connect=$ZOOKPR \
--list# list for individual topics, e.g. foo-topic
bin/kafka-acls.sh \
--authorizer kafka.security.auth.SimpleAclAuthorizer \
--authorizer-properties zookeeper.connect=$ZOOKPR \
--list \
--topic foo-topic

7. Deploy example application

We should finally be ready to deploy our demonstration application to EKS. The application contains two Go-based microservices, Service A and Service B. The origin of the demonstration application’s functionality is based on Soham Kamani’s September 2020 blog post, Implementing a Kafka Producer and Consumer In Golang (With Full Examples) For Production. All source Go code for the demonstration application is included in the project.

├── Dockerfile
├── README.md
├── consumer.go
├── dialer.go
├── dialer_scram.go
├── go.mod
├── go.sum
├── main.go
├── param_store.go
├── producer.go
└── tls.go

Both microservices use the same Docker image, , configured with different environment variables. The configuration makes the two services operate differently. The microservices use Segment’s client, as mentioned earlier, to communicate with the MSK cluster’s broker and topics. Below, we see the demonstration application’s consumer functionality ().

The consumer, above, and the producer both connect to the MSK cluster using SASL/SCRAM. Below, we see the SASL/SCRAM Dialer functionality. This type mirrors the API but is designed to open Kafka connections instead of raw network connections. Note how the function can access AWS Secrets Manager to retrieve the SASL/SCRAM credentials.

We will deploy three replicas of each microservice (three pods per microservices) using Helm. Below, we see the Kubernetes and resources for each microservice.

Run the following commands to deploy the demonstration application to EKS using the project’s Helm chart, :

cd helm/# perform dry run to validate chart
helm install kafka-demo-app ./kafka-demo-app \
--namespace $NAMESPACE --debug --dry-run# apply chart resources
helm install kafka-demo-app ./kafka-demo-app \
--namespace $NAMESPACE

Confirm the successful creation of the Kafka client pod with either of the following commands:

kubectl get pods -n kafkakubectl get pods -n kafka -l app=kafka-demo-service-akubectl get pods -n kafka -l app=kafka-demo-service-b

You should now have a total of seven pods running in the namespace. In addition to the previously deployed single Kafka client pod, there should be three new Service A pods and three new Service B pods.

The ability of the demonstration application to interact with AWS SSM Parameter Store and AWS Secrets Manager is based on the IAM policy created by Terraform, . This policy is associated with a new IAM role, which in turn, is associated with the Kubernetes Service Account, . This service account is associated with the demonstration application’s pods as part of the Kubernetes Deployment resource in the Helm chart.

8. Verify application functionality

Although the pods starting and running successfully is a good sign, to confirm that the demonstration application is operating correctly, examine the logs of Service A and Service B using . The logs will confirm that the application has successfully retrieved the SASL/SCRAM credentials from Secrets Manager, connected to MSK, and can produce and consume messages from the appropriate topics.

kubectl logs -l app=kafka-demo-service-a -n kafkakubectl logs -l app=kafka-demo-service-b -n kafka

The environment variable controls the frequency at which the microservices produce messages. The frequency is 60 seconds by default but overridden and increased to 10 seconds in the Helm chart.

Below, we see the logs generated by the Service A pods. Note one of the messages indicating the Service A producer was successful: . And a message indicating the consumer was successful: . Each message contains the name of the host container that produced and consumed it.

Likewise, we see logs generated by the two Service B pods. Note one of the messages indicating the Service B producer was successful: . And a message indicating the consumer was successful: .

CloudWatch Metrics

We can also examine the available Amazon MSK CloudWatch Metrics to confirm the EKS-based demonstration application is communicating as expected with MSK. There are 132 different metrics available for this cluster. Below, we see the and for each of the two topics, across each of the two topic’s three partitions, which are spread across each of the three Kafka broker nodes. Each metric shows similar volumes of traffic, both inbound and outbound, to each topic. Along with the logs, the metrics appear to show the multiple instances of Service A and Service B are producing and consuming messages.


We can also confirm the same results using an open-source observability tool, like Prometheus. The Amazon MSK Developer Guide outlines the steps necessary to monitor Kafka using Prometheus. The Amazon MSK cluster created by already has open monitoring with Prometheus enabled and ports and added to the necessary MSK security group by Terraform.

Running Prometheus in a single pod on the EKS cluster, built from an Ubuntu base Docker image or similar, is probably the easiest approach for this particular demonstration.

rate(kafka_server_BrokerTopicMetrics_Count{topic=~"foo-topic|bar-topic", name=~"BytesInPerSec|BytesOutPerSec"}[5m])
Sours: https://itnext.io/securely-decoupling-applications-on-amazon-eks-using-kafka-with-sasl-scram-48c340e1ffe9

Msk terraform

locals { bootstrap_brokers =try(aws_msk_cluster.default[0].bootstrap_brokers, "") bootstrap_brokers_list =local.bootstrap_brokers!=""?sort(split(",", local.bootstrap_brokers)) : [] bootstrap_brokers_tls =try(aws_msk_cluster.default[0].bootstrap_brokers_tls, "") bootstrap_brokers_tls_list =local.bootstrap_brokers_tls!=""?sort(split(",", local.bootstrap_brokers_tls)) : [] bootstrap_brokers_scram =try(aws_msk_cluster.default[0].bootstrap_brokers_sasl_scram, "") bootstrap_brokers_scram_list =local.bootstrap_brokers_scram!=""?sort(split(",", local.bootstrap_brokers_scram)) : [] bootstrap_brokers_iam =try(aws_msk_cluster.default[0].bootstrap_brokers_sasl_iam, "") bootstrap_brokers_iam_list =local.bootstrap_brokers_iam!=""?sort(split(",", local.bootstrap_brokers_iam)) : [] bootstrap_brokers_combined_list =concat(local.bootstrap_brokers_list, local.bootstrap_brokers_tls_list, local.bootstrap_brokers_scram_list, local.bootstrap_brokers_iam_list)}resource"aws_security_group""default" { count =module.this.enabled?1:0 vpc_id =var.vpc_id name =module.this.id description ="Allow inbound traffic from Security Groups and CIDRs. Allow all outbound traffic" tags =module.this.tags}resource"aws_security_group_rule""ingress_security_groups" { count =module.this.enabled?length(var.security_groups) :0 description ="Allow inbound traffic from Security Groups" type ="ingress" from_port =0 to_port =65535 protocol ="tcp" source_security_group_id =var.security_groups[count.index] security_group_id =join("", aws_security_group.default.*.id)}resource"aws_security_group_rule""ingress_cidr_blocks" { count =module.this.enabled&&length(var.allowed_cidr_blocks) >0?1:0 description ="Allow inbound traffic from CIDR blocks" type ="ingress" from_port =0 to_port =65535 protocol ="tcp" cidr_blocks =var.allowed_cidr_blocks security_group_id =join("", aws_security_group.default.*.id)}resource"aws_security_group_rule""egress" { count =module.this.enabled?1:0 description ="Allow all egress traffic" type ="egress" from_port =0 to_port =65535 protocol ="tcp" cidr_blocks = [""] security_group_id =join("", aws_security_group.default.*.id)}resource"aws_msk_configuration""config" { count =module.this.enabled?1:0 kafka_versions = [var.kafka_version] name =module.this.id description ="Manages an Amazon Managed Streaming for Kafka configuration" server_properties =join("\n", [forkinkeys(var.properties) :format("%s = %s", k, var.properties[k])])}resource"aws_msk_cluster""default" { count =module.this.enabled?1:0 cluster_name =module.this.id kafka_version =var.kafka_version number_of_broker_nodes =var.number_of_broker_nodes enhanced_monitoring =var.enhanced_monitoringbroker_node_group_info { instance_type =var.broker_instance_type ebs_volume_size =var.broker_volume_size client_subnets =var.subnet_ids security_groups = aws_security_group.default.*.id }configuration_info { arn = aws_msk_configuration.config[0].arn revision = aws_msk_configuration.config[0].latest_revision }encryption_info {encryption_in_transit { client_broker =var.client_broker in_cluster =var.encryption_in_cluster } encryption_at_rest_kms_key_arn =var.encryption_at_rest_kms_key_arn }dynamic"client_authentication" { for_each =var.client_tls_auth_enabled||var.client_sasl_scram_enabled||var.client_sasl_iam_enabled? [1] : []content {dynamic"tls" { for_each =var.client_tls_auth_enabled? [1] : []content { certificate_authority_arns =var.certificate_authority_arns } }dynamic"sasl" { for_each =var.client_sasl_scram_enabled? [1] : []content { scram =var.client_sasl_scram_enabled } }dynamic"sasl" { for_each =var.client_sasl_iam_enabled? [1] : []content { iam =var.client_sasl_iam_enabled } } } }open_monitoring {prometheus {jmx_exporter { enabled_in_broker =var.jmx_exporter_enabled }node_exporter { enabled_in_broker =var.node_exporter_enabled } } }logging_info {broker_logs {cloudwatch_logs { enabled =var.cloudwatch_logs_enabled log_group =var.cloudwatch_logs_log_group }firehose { enabled =var.firehose_logs_enabled delivery_stream =var.firehose_delivery_stream }s3 { enabled =var.s3_logs_enabled bucket =var.s3_logs_bucket prefix =var.s3_logs_prefix } } } tags =module.this.tags}resource"aws_msk_scram_secret_association""default" { count =module.this.enabled&&var.client_sasl_scram_enabled?1:0 cluster_arn = aws_msk_cluster.default[0].arn secret_arn_list =var.client_sasl_scram_secret_association_arns}module"hostname" { count =var.number_of_broker_nodes>0&&var.zone_id!=null?var.number_of_broker_nodes:0 source ="cloudposse/route53-cluster-hostname/aws" version ="0.12.0" enabled =module.this.enabled&&length(var.zone_id) >0 name ="${module.this.name}-broker-${count.index+1}" zone_id =var.zone_id records = [split(":", element(local.bootstrap_brokers_combined_list, count.index))[0]] context =module.this.context}
Sours: https://github.com/cloudposse/terraform-aws-msk-apache-kafka-cluster/blob/master/main.tf
Terraform Explained

AWS Classic

Manages an Amazon Managed Streaming for Kafka configuration. More information can be found on the MSK Developer Guide.

Create a Configuration Resource

Configuration Resource Properties

To learn more about resource properties and how to use them, see Inputs and Outputs in the Architecture and Concepts docs.


The Configuration resource accepts the following input properties:


All input properties are implicitly available as output properties. Additionally, the Configuration resource produces the following output properties:

Look up an Existing Configuration Resource

Get an existing Configuration resource’s state with the given name, ID, and optional extra properties used to qualify the lookup.

The following state arguments are supported:


MSK configurations can be imported using the configuration ARN, e.g.

Package Details

This Pulumi package is based on the Terraform Provider.
Sours: https://www.pulumi.com/docs/reference/pkg/aws/msk/configuration/

Now discussing:


1115 1116 1117 1118 1119