The post-office & the postman

If we were to talk about old messaging system where there exist post-office, postman & mailbox. each component had its own functionality that we looked for when trying to visualize how those component where to interact in a computerized version.

Simple scenario:

1. Mail is added in mail box
2. Postman arrive pick mails from his area mailboxes and take them to the post-office.
3. Post-office organize mails by areas.
1. Postman takes mails related to his area "distribute it in mailboxes".
2. A person can go to post-office and  pick his own mail "in case of failure or wishes for early delivery".

Mapping in a computerized version:

• Scenario: Observer design pattern which can use push or pull scenario, to inform those whom are registered for an event about its occurrence.
• Component:
• Post-Office = Message-Broker
• Post-Office-Box = Message-Storage-Validity
• Mailbox = Topic/Queue
• Postman !!! where's the postman ?

Apache kafka act as a message broker which decouple message processing from publisher, also it can buffer unprocessed message, it follow pull data scenario "Consumer pull data". below is kafka capabilities:

• publish / subscribe: act as a messaging system Topic / Queue.
• Store stream: act as a storage system that can keep data for 2 day "configurable".
• Process stream: act as a decorator design pattern.

Apache flume act as the post-man, which will deliver the event from a point to another, it follow push data scenario, below is flume capabilities:

• deliver data from a point "source" to another "sink" (can be configured to push data to a port).
• Modify or drop events in the flow "act as a decorator".
• can store data in a partitions ( /dataDir/year=%y/  ).