December 01, 2019

Kafka Part 6: Assign and Seek

Assign

When we work with consumer groups, the partitions are assigned automatically to consumers and are rebalanced automatically when consumers are added or removed from the group.

ALSO READ: Kafka Consumer Group, Partition Rebalance, Heartbeat

Sometimes we need a single consumer that always read data from all the partitions in a topic, or from a specific partition in a topic. In this case, there is no reason for groups or rebalancing. We just assign the consumer-specific topic and/or partitions, consume messages, and commit offsets on occasion.

Kafka Part 5: Consumer Group, Partition Rebalance, Heartbeat

What is a Consumer Group?
Consumer Groups is a concept exclusive to Kafka. Every Kafka consumer group consists of one or more consumers that jointly consume a set of subscribed topics.

Let's say we have an application, which read messages from a Kafka topic, perform some validations and so some calculations, and write the results to another data store.

In this case our application will create a consumer object, subscribe to the appropriate topic, and start receiving messages, validating them and writing the results.

This may work well for a while, but imagine a scenario when the rate at which producers write messages to the topic exceeds the rate at which your application can validate them?

Kafka Part 4: Consumers

We have learned how to create Kafka Producer in the previous part of Kafka series. Now we will create Kafka consumer.

Reading data from Kafka is a bit different than reading data from any other messaging systems. Applications that need to read data from Kafka use a KafkaConsumer to subscribe to Kafka topics and receive messages from these topics.

In this blog post, we will discuss about the interview questions related to kafka Consumers and we will also create our own consumer.

Kafka Part 3: Kafka Producer, Callbacks and Keys

What is the role of Kafka producer?
The primary role of a Kafka producer is to take producer properties and record as inputs and write it to an appropriate Kafka broker. Producers serialize, partitions, compresses and load balances data across brokers based on partitions.

The workflow of a producer involves five important steps:
  1. Serialize
  2. Partition
  3. Compress
  4. Accumulate records
  5. Group by broker and send

Kafka Part 2: Kafka Command Line Interface (CLI)

Now we will create topics from CLI.
  • Open another command prompt, execute 'cd D:\Softwares\kafka_2.12-2.3.0\bin\windows'.
  • Make sure zookeeper and kafka broker is running.
  • Now execute 'kafka-topics --zookeeper 127.0.0.1:2181 --topic first_topic --create --partitions 3 --replication-factor 1'. This will create a kafka topic with name 'first_topic', with 3 partitions and replication-factor as 1 (in our case we cannot mention replication-factor more than 1 because we have started only one broker).
  • After executing the above command you will get a message 'Created topic first_topic'.
  • How can we check if our topic is actually created? In the same command prompt, execute 'kafka-topics --zookeeper 127.0.0.1:2181 --list'. This will list all the topics which are present.
  • To get the more details about the topic which is created, execute 'kafka-topics --zookeeper 127.0.0.1:2181 --topic first_topic --describe'

Kafka Part 1: Basics

What is Apache Kafka?
Apache Kafka is a publish-subscribe messaging system developed by Apache written in Scala. It is a distributed, partitioned and replicated log service. It is horizontally scalable, fault tolerant and fast messaging system.

Why Kafka?
Let's say we have a source system and a target system, where in the source consumes data from target system. In a simplest case, we have one source and one target system,so it would be easy for a source system to connect with the target. But now lets say there are x number of sources and y number of targets, and each source need to connect with all the targets. In this case it will become really difficult to maintain the whole system.


Lyrics and English Translation Of Baari




Tenu Takeya Hosh Hi Bhul Gayi,
Garm Garm Chaa Hath Te Dul Gayi,

As I saw you, I lost my senses,
which lead me to spill the hot tea on my hand.

November 28, 2019

BARC TV Ratings- Week 47, 2019

Top 10 Hindi GEC Channel
  1. Dangal : 1179064
  2. STAR Plus : 696143
  3. SONY SAB : 628896
  4. Zee TV : 609597
  5. Colors : 558278
  6. Sony Entertainment Television : 535514
  7. Big Magic : 470004
  8. STAR Bharat : 292995
  9. Colors Rishtey : 171323
  10. STAR Utsav : 164618

November 26, 2019

'Sarabjit' actor Trishaan Singh Maini roped in for #Mandi

We reported earlier that Mohini Shabade's web-series for Prime Flix underwent a name change from Brothel to #Mandi. Now we heard, Trishaan Singh Maini, who was seen in Vikram Bhatt's Hadh has been roped in to play one of the leads in the series.

Minissha Lamba will foray into the digital space with Ullu App's Kasak

Bollywood actress Minissha Lamba will foray into the digital space with Ullu App's upcoming web-series 'Kasak'.

November 24, 2019

Transwoman Ivanka Das to play a brothel owner in #Mandi

Multitalented transwoman Ivanka Das, who works as a dance choreographer and a model will be seen playing as a brothel owner in upcoming web-series '#Mandi'.

#Algorithms Part 3: Shortest Path Algorithms(Bellman-Ford Algorithm)

Bellman-Ford Algorithm
  • As we have seen in above example that Dijkstra Algorithm not always find the shortest path in case of negative edges. Bellman-Ford always give correct result even in case of negative edges (though it has its own drawback).
  • As Dijkstra says relax the vertices's only once, Bellman-Ford says we can relax vertices's for n number of times, so that we can get perfect shortest path to all the vertices's.
  • In Bellman-Ford we make list of all the edges and go one relaxing them for v-1 times (where v is the number of vertices) after which we will surely get the shortest path to all the edges even in case of negative edges.
Let's take an example:

We need to first make list of edges. In our case it would be (d,b), (c,d), (a,c), (a,b).

1). Take the list of all edges and mark the distances of all the vertices's as infinity, keep the first one ('a' in our case) as zero. Now we need to go and relax the vertices's for v-1 times (where v is number of vertices's), in our case it will be three. Lets start with d to b, d is infinity and b is infinity, so the shortest path from d to b would be infinity plus infinity which is infinity. Now we take c to d, again since both have infinity, so the shortest path at d would be infinity. Now take a to c, it would be 0+2, which is 2, we will change the shortest path at c from infinity to 2. Now we take a to b, and change the shortest path at b from infinity to 1. We have relaxed the vertices's for 1st time.

2). We repeat the process again, from d to b the shortest path would be infinity (value at d) minus 6 (since the edge has negative value), this will gives infinity. But the value at b is 1, which is less than infinity, so we are not going to change it. Now we take c to d, the shortest path would be 2+3 which is 5, shortest path at d was infinity, we will change it to 5. Now we take a to c, which gives shortest path as 2, which is already 2, so we are not going to change it. Same is the case from a to b. We have relaxed the vertices's for 2nd time.

3). Again we will go from d to b, the shortest path would be 5-6, thats -1. The shortest path at b is 1, which is greater than -1, so we will change it to -1. Then we will go from c to d, a to c, a to b, but the values remain unchanged.

4). We have relaxed vertices's for 3rd time. (v-1).

Bellman-Ford fails if there is a negative cycle in the graph. Like in the below case b-c-d has a cycle and total value is 1 + 3 + (-6), which is -3.


-K Himaanshu Shuklaa..

#Algorithms Part 1: Graphs, BFS,DFS, Dijkstra Algorithm, Bellman-Ford Algorithm

Graph Traversal
  • Its the process of visiting and exploring a graph for processing.
  • In this we visit and explore each vertex and edge in a graph such that all the vertices's are explored exactly once.
  • Visiting a node means selecting a node, where as exploring means exploring the children nodes of the node which we have visited.
Breadth First Search
  • Its an algorithm for traversing trees and graphs.
  • In BFS we start with the root node, explores all sibling nodes before moving on to the next level of siblings.
  • Queue is the main data structure which is used while performing BFS.
  • BFS is used as a crawlers in web-engines. It is the main algorithm which is used for indexing the web pages, it starts from the source page and follows all the links associated with that page. In this case each link is considered as a node in the graph.
  • BFS is also used in GPS navigation to find the neighboring locations.
  • BFS is also used in finding the shortest path

Binary Search Trees, Balance Search Trees, 2-3 ST, Red- Black BST

Tree Data Structure
  • If we compare sorted arrays with linked list: Search is fast in SA (O(logn)) and slow in LL (O(n)), but insert & delete is slow in SA (O(n)) and fast in LL (O(1)).
  • Binary search trees are very balanced in the sense that all these common operations like insert, delete, and search take about the same time (log n).
  • In Binary search trees we have the nodes, which contain the data. Nodes are connected to other nodes through edges. The top node of the tree is called the root node (in a tree data structure, there can only be one root node).
  • A node which has no children is called a leaf node. 
  • Any node may be considered to be the root of a subtree, which consists of its children and its children's children and so on. The level of a node tells us how far it is from the root. 

#Algorithms Part 4: Caching Algorithms in Java (LRU and LFU)

What is Cache?
  • As its expensive to every-time fetch data from the main storage, we story it in temporary location from where it can be retrieved faster. This temporary location is called Cache.
  • A Cache is made of pool of entries and these entries are a copy of real data which are in storage and it is tagged with a tag (key identifier) value for retrieval.
  • When an application receives a request to get a particular information, it is first checked in cache. If an entry is found with a tag matching with the request, it will be returned. Else it will be fetched from the main storage, kept in Cache and then returned to the client. 

Java Sorting Algorithms

Selection Sort
  1. The selection sort algorithm sorts an array by repeatedly finding the minimum element (considering ascending order) from unsorted part and putting it at the beginning. 
  2. The algorithm maintains two subarrays in a given array: (a).The subarray which is already sorted. (b). Remaining subarray which is unsorted.
  3. In every iteration of selection sort, the minimum element (considering ascending order) from the unsorted subarray is picked and moved to the sorted subarray.
  4. It is very simple to implement but it does not go well with large number of inputs.