5.1: Design report data aggregator software

The data aggregator software task in work package 5 is progressing as planned. We decided to use Apache Kafka as the underlying technology for aggregation and streaming, and Google FlatBuffers as the serialisation library. Development work is going on with an agile approach focussed on early software tests and delivery, addressing the requirements in iterations. Collaboration tools are being used to track the progress of the project, discuss issues and maintain documentation, with virtual meetings held every two weeks.

Software modules to generate simulated data streams for integration and performance evaluations are ready to be put into our testing and deployment infrastructure. This infrastructure consists of a build server and with virtual machines for deploying and running integration tests as well as a physical lab space with three servers to be installed at the ESSIIP laboratory in Lund, where software from the Data Management group can be tested and integrated with real hardware and software from other ESS groups.

In the coming development cycles we are going to validate the technology choices and system design and architecture in both settings, addressing problems that might be discovered in the tests. The use of Kafka in projects handling large volumes of data is an indication that it can satisfy our requirements. Results from the tests and future project steps will be documented in JIRA, Confluence and Bitbucket.

DOI: https://doi.org/10.17199/BRIGHTNESS.D5.1