Change

If we were to talk about old messaging system where there exist post-office, postman & mailbox. each component had its own functionality that we looked for when trying to visualize how those component where to interact in a computerized version. Simple scenario: Mail is added in mail box Postman arrive pick mails from his area mailboxes and take them to the post-office. Post-office organize mails by areas. Postman takes mails related to his area "distribute it in mailboxes". A person can go to post-office and  pick his own mail "in case of failure or wishes for early delivery". Mapping in a computerized version: Scenario: Observer design pattern which can use push or pull scenario, to inform those whom are registered for an event about its occurrence. Component: Post-Office = Message-Broker Post-Office-Box = Message-Storage-Validity Mailbox = Topic/Queue Postman !!! where's the postman ? Apache kafka act as a message broker which d

Data aggregation is one of the roads we use to understand our data diversity. SQL "Selective Query Language" is the easiest way we use to do so. below is how to map our SQL syntax to Pig or Spark. SQL Structure: What to retrieve, stating which column we choose to display from my data structure SQL: Select student, age From mathClass Pig: namedMathClass = foreach mathClass generate (chararray) $0 as student:chararray, (int) $2 as age:int ; Spark: namedMathClass = mathClass.map( row => row(0), row(2) ) Whether this row is to be added in our data-set or not "Condition" SQL: where age > 10 Pig: greater_10 = Filter namedMathClass by age > 10 ; Spark: greater_10 = namedMathClass.filter( col => col(1) > 10 ) How to aggregate, we group similar data together  in one bag then apply our aggregate function on this bags  SQL: Select age, Count(student) From mathClass group by age Pig: groupAge = Group mathClass by age; Iterate_Age = For