Financial Data Analysis using Kafka and Spark Streaming

In my earlier posts on Apache Spark Streaming, we looked at how data can be processed using Spark to compute the aggregations and also store the data in a compressed format like Parquet for future analysis. We also looked at how data can be published and consumed using Apache Kafka which is a distributed message … Continue reading Financial Data Analysis using Kafka and Spark Streaming

Apache Kafka – Producers and Consumers

This post will provide a quick overview on how to write Kafka Producer and Kafka Consumer with a Kafka broker running locally. First, let's set-up the Kafka broker locally by downloading the TAR file and running the required scripts. Other option is to run Kafka broker locally using Docker image, however I'll stick to the … Continue reading Apache Kafka – Producers and Consumers