How to configure Amazon MSK (Kafka) Cluster

Owned by Ramya Raman (Unlicensed)
Last updated: Sept 30, 2024 by Eitan Dvir
2 min read

Please follow the steps below to create an Amazon Kafka cluster.

  1. Create a cluster a Cluster configurations and change auto.create.topics.enable = true

 

2. Create a cluster with a Plaintext Authentication.

3. Create a client as a consumer to see messages (Step 3: Create a client machine - Amazon Managed Streaming for Apache Kafka).

 

4 . After setting the client update: kafka_2.12-2.2.1/bin/client.properties

security.protocol=PLAINTEXT
sasl.mechanism=PLAIN

 

5. Configure the MCM Kafka agent with the cluster Plaintext - Private endpoint. example: ” b-3.mcmu.dzt62v.c9.kafka.eu-west-1.amazonaws.com“ Port: 9092

Please note:

  • If you do not specify the Topics then the MCM will create the topics. example: mcm9000.ChannelCaptions

  • “Ignore Notification Sets “ means that its the messages will be in a keep-alive state and will send messages regardless to the Trasholdset you have created.

 

6. Run the following command on the consumer to retrieve the Topics list (change Private endpoint to the plaintext endpoint):

./kafka-topics.sh --bootstrap-server "<Private endpoint>“--list

You will receive the example topic list:

mcm9000.ChannelCaptions
mcm9000.ChannelEvent
mcm9000.ChannelScte35
mcm9000.ChannelStatistics
mcm9000.OttDownloadStatistics

 

7. Select one of the Topics ann run the following command: (<change Private endpoint>,<change topic>)

./kafka-console-consumer.sh --bootstrap-server "<change Private endpoint>" --consumer.config client.properties --topic <change topic> --from-beginning

 

8. The output message you receive should look like this:

"event_number":86,"event_type":"Audio bitrate, max threshold","access_type":"MPEG-DASH","severity":"Notice","active":false,"trigger":false,"duration":8,"description":"Audio BR+ 241/192 K (P:6000)","details":"index: 1 cur: 241K exp: 192K","created_us":1679235989985123,"timestamp_us":1679235989985123}
{"channel":"ee32d72d-63c8-47bd-ac1e-833d72aaa73d","device":"bd4e80c9-9755-4ab7-91de-a4beb6178b3a","mcm-version":"6.1.0x1","protocol":"1.0","label":"MPEG-DASH Channel_4","url":"https://tv2-play-ew-live-aws-arn-dash.secure.footprint.net/__cl/cg:live/__c/tv2_fyn/__op/nd/__f/dash.mpd","tags":null,"profile":{"label":"Default","uuid":"ad9b9bad-ad9b-4bad-bd9b-9badad9b9bad"},"event_number":212,"event_type":"New HLS/Mpeg-Dash/MSS Block","access_type":"MPEG-DASH","severity":"Info","active":false,"trigger":true,"duration":0,"description":"New OTT Block (2623796419)","details":"val: 2623796419","created_us":1679235989985297,"timestamp_us":1679235989985297}
{"channel":"23a3da8d-aeb4-4f5d-949b-f4fd80c463bd","device":"bd4e80c9-9755-4ab7-91de-a4beb6178b3a","mcm-version":"6.1.0x1","protocol":"1.0","label":"MPEG-DASH Channel_3","url":"https://tv2-play-ew-live-aws-arn-dash.secure.footprint.net/__cl/cg:live/__c/tv2_fyn/__op/nd/__f/dash.mpd","tags":null,"profile":{"label":"Default","uuid":"46f6836d-cb70-4cbd-8741-d23dc6ca263d"},"event_number":86,"event_type":"Audio bitrate, max threshold","access_type":"MPEG-DASH","severity":"Notice","active":false,"trigger":false,"duration":8,"description":"Audio BR+ 250/192 K (P:6000)","details":"index: 1 cur: 250K exp: 192K","created_us":1679235990029512,"timestamp_us":1679235990029512}