Category: Apache Kafka

Apache Kafka related posts & articles.

Power your Kafka Streams application with Amazon MSK and AWS Fargate

Note: This blog post is a reproduction of an article I previously published on AWS Blog. I wanted to share it here for my readers who may not have seen it.

Introduction

Today, companies of all sizes across all verticals design and build event-driven architectures centered around real-time streaming and stream processing. Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a fully managed service that makes it easy for you to build and run applications that use Apache Kafka to process streaming and event data. Apache Kafka is an open-source platform for building real-time streaming data pipelines and applications. With Amazon MSK, you can continue to use native Apache Kafka APIs to build event-driven architectures, stream changes to and from databases, and power machine learning and analytics applications.

Continue reading “Power your Kafka Streams application with Amazon MSK and AWS Fargate”

Kafka Test Suite (Java)

One of the common issues that I had with Kafka was an integration test(Java). I was searching on the web for a library which will include not only Kafka broker but also Confluent Schema Registry. I didn’t find one, so I went to kafka-streams-examples and extracted classes needed for running Kafka cluster programmatically.
Continue reading “Kafka Test Suite (Java)”

Apache Kafka + Spring Cloud Task

Recently I was working on a piece of the project which required to perform a batch job, run application 1 time and once it successfully completed – terminate it. Spring Cloud Task fits perfect for this kind of requirement. Our usual stack of technologies is Spring Cloud Stream/Task and Apache Kafka. In my earlier posts, I showed you an example how to use Spring Cloud Stream + Apache Kafka. However, I never used Spring Cloud Task, so I start browsing for examples and… I found nothing. There is no example of Spring Cloud Task with Apache Kafka. At least I couldn’t find. So I dig into that and come up with a solution.
Continue reading “Apache Kafka + Spring Cloud Task”

Kafka CLI – Maximize

I was lucky to use Landoop Lenses for a few weeks on my current project. This is a user-friendly Kafka UI tool, comes with a ton of functionality. I was able to inspect and query topic messages, delete and create topics, check the status of consumer groups, look at Avro schema, quickly count messages in the topic, etc. Everything was a few clicks away. Well, this happy story ends here, sadly it was decommissioned recently.
Continue reading “Kafka CLI – Maximize”