Financial Data Analysis – Kafka, Storm and Spark Streaming

In my earlier posts, we looked at how Spark Streaming can be used to process the streaming loan data and compute the aggregations using Spark SQL. We also looked at how the data can be stored in file system for future batch analysis. We discussed how Spark can be integrated with Kafka to ingest the … Continue reading Financial Data Analysis – Kafka, Storm and Spark Streaming

AWS Virtual Private Cloud

AWS Virtual Private Cloud (VPC) is a web service that allows provisioning of a logically isolated infrastructure in the public cloud with its own IP address range, subnets, internet gateway, ACLs and route table configuration. It can be thought of as an isolated data center in AWS. VPC does all the heavy lifting and makes it … Continue reading AWS Virtual Private Cloud

AWS – Simple Workflow Service

AWS Simple Workflow Service (SWF) is a web service that helps in coordinating the execution of automated tasks and human tasks within a workflow. The processing steps can be execution of scripts and code, web service calls, human action etc. SWF Workers and Deciders Workers are programs that interact with SWF service to get the tasks, … Continue reading AWS – Simple Workflow Service

AWS – Simple Notification Service

Message-oriented Middleware (MoM) supports a messaging type in which the messages can be broadcasted to multiple message consumers known as message subscribers. AWS Simple Notification Service is a type of messaging service in the cloud that is based on the pub-sub model. It allows the message publisher to send a message to a Topic which has multiple subscribers that … Continue reading AWS – Simple Notification Service

AWS – Simple Queue Service

Message-oriented Middleware (MoM) allows the various applications and microservices within an enterprise to communicate with each other by sending messages asynchronously. Some of the MoMs traditionally used by enterprises are Websphere MQ, Active MQ, TIBCO, Rabbit MQ etc. Message Queue is used for point to point communication between the message producer and message consumer. AWS … Continue reading AWS – Simple Queue Service

AWS – EC2, EBS and ELB

Elastic Compute Cloud (EC2) EC2 is a web service which provides re-sizable computing resources that could scale out or scale down very easily based on the varying work load. At a very basic level it's nothing but a virtual machine running as a guest on the host machine with the help of a hypervisor. Typically it … Continue reading AWS – EC2, EBS and ELB

AWS – Identity and Access Management

Identity and Access Management (IAM) is widely used in most of the enterprises to authenticate and authorize the users to grant access to applications and systems that supports various functions within the organization. It is one of the basic components when it comes to enterprise security and defense in-depth principles that organizations adopt to protect … Continue reading AWS – Identity and Access Management