Loading…
This event has ended. Visit the official site or create your own event on Sched.
Wednesday, October 5 • 5:30pm - 6:20pm
Implementing an akka-streams materializer for big data

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Akka Streams provides a tremendously flexible architecture to build reactive pipelines that can be imported, exported and otherwise composed as partial DAGs. Its current, default implementation is ActorMaterializer which provides reactive streams across actors within a single JVM. 
Here we show how we implemented a GearpumpMaterializer which distributes reactive streams across a set of remote workers on the Apache Gearpump platform. We discuss how this was implemented and a number of challenges we faced with specific GraphStages and their semantics. Additional we cover how different materializer implementations can interoperate together to materialize different parts of the pipeline. We show that the changes we introduced internally within Akka Streams will enable other implementations of akka stream materializers and suggest a template based on our implementation. We will contribute the Gearpump materializer as open source to https://github.com/akka/akka-stream-contrib or make it available as part of an upcoming Apache Gearpump release.

Speakers
avatar for Kam Kasravi

Kam Kasravi

Senior Software Engineer, Intel Corp
Kam is working on deep learning systems and kubeflow


Wednesday October 5, 2016 5:30pm - 6:20pm CDT
Texas Ballroom