Batch Processing and Stream Processing (Async & Sync Messaging)

  • Fault tolerance—failed tasks will be recalled and re-executed
  • Idempotent operations—multiple executions of a job only perform once.
  • Common examples: backup systems, compressing logs
  • Map, Shuffle and Reduce
  • If a job is getting too big, we can create small parts called micro-batches
  • Synchronous communication
  • Stream Processing: convert the target data file into a set of events, events are created by one single producer (publisher).
  • If a job fails, reverse back to checkpoints
  • Buffer Messages in a Queue
  • Flow Control (when the slow consumer can not cope with the faster producer, simply block, can not receive any more data)
  • Fan-out —broadcast to several receivers without affecting each other
  • Example: email and web messages
  • Stream-based messaging (TCP — Transfer Control Protocol for data transfer)
  • UDP has no guarantee of all message delivery => good for distributing videos
  • After creating the socket, the server should start to wait for the connection Socket socket = new Socket(servername, port);



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Hanwen Zhang

Hanwen Zhang


Full-Stack Software Engineer at a Healthcare Tech Company | Document My Coding Journey | Improve My Knowledge | Share Coding Concepts in a Simple Way