Pipeline Architectures

A pipeline consists of a sequence of filters connected by message queues called pipes.

There are four types of filters:

·        Producers are at the start of the pipeline. They perpetually produce messages and write them to an output pipe.

·        Consumers are at the end of the pipeline. They perpetually read messages from their input pipe and consume them.

·        Transformers perpetually read messages from their input pipes, transform them into different messages, then write the transformed message to their output pipe.

·        Testers perpetually read messages from their input pipe, test the message, and if it passes, write them to their output pipe.

 

The following sequence diagram shows one round of message processing in a pipeline consisting of four filters:

Problem: How does a filter know when to read from its input pipe?

Variation: In a demand-driven pipeline activity begins when the consumer reads from its input pipe. Redraw the above diagram assuming the pipeline is demand-driven.

Applications:

·        Signal Processing

Pipelines have always been used in signal processing labs like recoring studios, for example:

·        Language Processors

·        Unix pipelines:

% cat inFile | grep pattern | sort > outFile

·        Map-Reduce Architectures

Big data applications use platforms like Hadoop to create a tree-like structure of data processors. Each branch is a pipeline.

Transformers are called maps and consumers are called reducers.

·        Stream Pipelines in Java

Streams allow programmers to attach pipelines to a data source such as a file, list, or supplier function.

Example 1

List<Integer> nums = List.of(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
Integer sum = nums.stream().filter(n->n%2 == 0).
   map(n->n*n).reduce(0, (a, b)->a + b);
System.out.println("sum= " + sum); // 220 = 4 + 16 + 36 + 64 + 100

Example 2 Infinite streams!

class NatSupplier implements Supplier<Integer> {
    private Integer next = 0;
    @Override
    public Integer get() {
        return next++;
    }
}

Stream<Integer> nats = Stream.generate(new NatSupplier()).
   filter(n->n%2 == 0).map(n->n*n);
Iterator<Integer> p = nats.iterator();
for(int i = 0; i < 10; i++) {
    System.out.print(" " + p.next());
} // 0 4 16 36 64 100 144 196 256 324