Streaming data from a LEGO Truck 🚚

Using data streamed from our custom made LEGO Truck, we will present several Scala-based streaming event processing technologies such as Apache Kafka (Kafka Streams), Apache Storm, Apache Spark (Spark Streaming) and Apache Flink

We will show key differences and we will try to compare them in a fair manner on aspects like performance, implementation learning curve, stability, etc.

Project breakdown

• docs -> Contains demo slides. See README.md
• kafka -> Contains Kafka Streams Scala example. See README.md
• spark -> Contains Spark Streaming Scala example. See README.md
• storm -> Contains Storm Scala example. See README.md
• flink -> Contains Flink Scala example. See README.md

Technologies under evaluation

• Apache Kafka (Kafka Streams)
• Apache Spark (Spark Streaming)
• Apache Storm
• Apache Flink
• Akka Streams (not implemented)
• Apache Samza (not implemented)
• Apache Heron (Incubator) (not implemented)

Presentation

Slides on GitHub Pages

Pre-requisites

• Install Java
• Install SBT
• Download Kafka

How to run (example)

1. Start Zookeeper: ./bin/zookeeper-server-start.sh config/zookeeper.properties
   • Set default port 2182
2. Start Kafka broker: ./bin/kafka-server-start.sh config/server.properties
3. Create a Kafka producer: ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic SourceTopic
4. Start all stream processors:
5. Kafka streams: sbt run
6. Spark streaming: sbt run
7. Storm: sbt run
8. Flink: sbt run
9. Setup Kafka consumer: ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic SinkTopic

Disclaimer

The views, thoughts, and opinions expressed in this repository belong solely to the author, and not to the author’s employer, organization, committee or other group or individual.

License

This project is licensed under the terms of the MIT license. See LICENSE

🚚🚍🚚🚍🚚🚍🚚🚍🚚🚍🚚🚍