Building a stream processor for the future

When Vasiliki (Vasia) Kalavri attended her first programming class, she didn’t understand a word. At school, she had enjoyed mathematics and physics and had decided to study electrical engineering at the National Technical University of Athens. The curriculum included computer science, something Vasia Kalavri hadn’t encountered before. “I had a computer, but I only used it for things like text editing,” she recalls. “I had never programmed before. It was completely foreign to me. I wanted to understand it – out of stubbornness, as much as anything else.”

Stubbornness quickly turned into fascination, and Kalavri ended up pursuing a Master’s degree in computer science. She was then granted an Erasmus Mundus Doctoral Fellowship in Distributed Systems, allowing her to complete her doctoral degree at the KTH Royal Institute of Technology in Stockholm and the Université catholique de Louvain in Belgium. Deciding to pursue an academic career, she started looking for research groups that combined her two main fields of interest: stream processing and graph analysis. The Systems Group of the Computer Science Department at ETH Zurich attracted her interest, and so she applied for – and received – a Postdoctoral Fellowship at ETH, which supports excellent young researchers for two years.

An ambitious project

The project that brought Vasia Kalavri to Zurich is a stream processing system known as Strymon. “Stream processing is the continuous processing of data as it arrives in the system,” Kalavri explains. “It’s not a new idea; it has existed for many years. What is new is the large volume and velocity of data produced today. Nearly everything on the Web is a stream.”

To cope with the enormous amounts of data generated every day, new and more powerful stream processing systems are required. Strymon, developed by the Systems Group, is one of them. It is an order of magnitude faster and much more customisable than most commercially available stream processors, which has sparked a lot of interest from industry partners such as Google, VMware and the travel technology company Amadeus. Technology companies like these have many uses for stream processing, from the monitoring of data centre performance to fraud detection with online payments.

Since her arrival at ETH, Vasia Kalavri has taken the lead on parts of the project and has spearheaded the effort to develop an open-source version that users from academia and industry can adapt to their needs. With that first important milestone achieved, the Strymon team has been working on making their system more autonomous. “Strymon is a distributed system: it runs on many computers at once,” Kalavri elaborates. “Its workload – the amount of data coming in – is not always predictable. The system has to be ready to process any amount of data, but at the same time, we don’t want it to waste any computational resources when the volume of data is low.” Last year, the team “taught” Strymon to recognise changes in workload and to adapt its resource utilisation accordingly. “We also want it to recover from failures automatically, especially since it’s distributed. If one machine fails, we want Strymon to move the work it was doing onto other machines without losing any data or slowing down the system,” Kalavri states.

Despite the interest from industry partners, there are no specific plans to turn Strymon into a commercial project yet. “For that to happen, someone from the team would have to put their academic career on hold and focus on the commercial development,” Kalavri says.

Unique team spirit and nice weather

The researcher herself, who enjoys both research and teaching, is not done with academia just yet. As the two years of her Postdoctoral Fellowship at ETH Zurich come to an end in December 2019, she has accepted an Assistant Professorship at the Department of Computer Science at Boston University.