Financial Data Analysis using Kafka, Storm and MariaDB

In my previous posts, we looked at how to integrate Kafka and Storm for streaming loan data and cleansing the data before ingesting it into processing pipeline for aggregating the data. We also looked at how to leverage Liquibase for managing the relational database in form of immutable scripts that could be version controlled. This fits … Continue reading Financial Data Analysis using Kafka, Storm and MariaDB

Financial Data Analysis – Kafka, Storm and Spark Streaming

In my earlier posts, we looked at how Spark Streaming can be used to process the streaming loan data and compute the aggregations using Spark SQL. We also looked at how the data can be stored in file system for future batch analysis. We discussed how Spark can be integrated with Kafka to ingest the … Continue reading Financial Data Analysis – Kafka, Storm and Spark Streaming

Stream Processing using Storm and Kafka

In my earlier post, we looked at how Kafka can be integrated with Spark Streaming for processing the loan data. In the Spark streaming process, we are cleansing the data to remove invalid records before we aggregate the data. We could potentially cleanse the data in the pipeline prior to streaming the loan records in … Continue reading Stream Processing using Storm and Kafka

Queuing Theory for evaluating system performance in Event Driven Architecture – Part 1

Event Driven Architecture (EDA) is gaining a lot of popularity due to the benefits it offers to an enterprise to easily connect multiple disparate systems. Instead of creating a direct or point to point communication links between the systems that need to talk to each other, the communication process can be decoupled by having the … Continue reading Queuing Theory for evaluating system performance in Event Driven Architecture – Part 1