Batch Processing and Stream Processing (Async & Sync Messaging)
An overview of Batch, Stream, Messaging, and WebSocket Connection
Batching Processing (Offline System)
- Takes a large amount of input data, and uses a background job to process them in a batch, and then sends it out maybe the next day
- Fault tolerance—failed tasks will be recalled and re-executed
- Idempotent operations—multiple executions of a job only performed once.
- Common examples: backup systems, compressing logs
MapReduce —a batch processing algorithm
- Split petabytes of data into smaller chunks for a specific task, and distribute those chunks in parallel processing among server machines
- Map, Shuffle, and Reduce
Fault Tolerance in Batch Processing
- If a job fails simply batch processor restart that job.
- If a job is getting too big, we can create small parts called micro-batches
- Involve with buffer object (raw data, bits, and bytes)
- The inbound stream of data is an input stream (keyboard).
- Write out to a file, and use an output stream.
- A pipe is a destination to write to (flow data from one object to another one). readStream.pipe(writableStream).
Stream Processing (Near Real-Time System)
- Asynchronous communication
- Synchronous communication
Batch Processing v.s. Stream Processing
- Batch Processing: convert the target data file into a set of records, a data can be created by multiple jobs.
- Stream Processing: convert the target data file into a set of events, events are created by one single producer (publisher).
The publisher produces the data and the Subscriber consumes the data.
Resolve Frequent Polling — by enabling the publisher to notify the subscriber whenever the information is available.
Fault Tolerance in Stream Processing
- Use checkpoint ID or use a time to label stream jobs and their order
- If a job fails, reverse back to checkpoints
queue up a certain amount of responses and then send them in a batch, the publisher will produce data that is put in a message bus, subscribers are not directly getting the message from the publisher, instead, get the message from a message bus
Message Bus — a sort of buffer that we put messages there, and pull the message when they are available to use
RabbitMQ — Software when the app needs to transfer messages between different software components
What happens when a receiver can not keep up with the sender?
- Drop Message (best way to deal with video data is to drop some packets)
- Buffer Messages in a Queue
- Flow Control (when the slow consumer can not cope with the faster producer, simply block, can not receive any more data)
Multiple Receiver Patterns
- Load balancing — add more receivers to process messages in parallel
- Fan-out —broadcast to several receivers without affecting each other
- Communication between two software components that one component is waiting for another component to respond.
- Example: email and web messages
- Packet-based messaging (UDP — User Datagram Protocol for data transfer)
- Stream-based messaging (TCP — Transfer Control Protocol for data transfer)
UDP v.s. TCP
- UDP is much faster than TCP
- UDP has no guarantee of all message delivery => good for distributing videos
Socket Connection — Server v.s. Client
- The first step is to create a server socket
Socket socket = serversocket.accept();
- After creating the socket, the server should start to wait for the connection
Socket socket = new Socket(servername, port);
- The client should request a server for a connection with the server name and port
ServerSocket serversocket = new ServerSocket(9000);