Federated Data Processing Systems

Objectives

Industry 4.0 ecosystems are becoming increasingly fragmented, dynamic, and collaborative. For instance, remote data sources such as sensors and cameras, which are widely spread, produce data to be handled by multiple applications owned by several participants that aim to collaborate, yet sometimes have conflicts of interest. Such intricacies exacerbate the need for more intelligent and federated systems capable of self-adapting to enable efficient and secure collaboration while safeguarding data-locality and without affecting or disrupting the running systems.

Currently, production-grade data systems focus on centralized setups and therefore struggle to provide a federated solution and address the above challenges. These systems exploit the traditional central cloud, which interferes with application performance and profitability due to bandwidth-limited and latency-prone edge links, network and computation costs, and limitations for knowledge sharing. Hence, systems must enable an adaptive usage of all layers of the network, leveraging federated application deployments across clouds, edges, and far edges.

We are creating a data fabric system that is distributed and federated by design and that dynamically adjusts to changes in the environment, offering real-time data processing for federated Industry 4.0 eco-systems. Our system adapts and grows dynamically without service disruption, achieving seamless and trusted distribution of analytics functions and machine learning models across federated actors and clouds.
 

Achievements and projects

The Nokia Bell team has created World Wide Streams (WWS), a large-scale and geo-distributed stream processing platform that facilitates the development of real-time applications, handles high volumes of data and media streams, and deploys applications across central clouds, edge clouds, or the far edge.

WWS applications are authored in XStream, a novel Bell Labs language, providing a flexible way to build dataflows with built-in and external operators. XStream mitigates the interplay with different programming languages, supporting the declaration of external operators, which are not implemented in XStream, and linking them together in a dataflow effortlessly. XStream primitives include stream processing operators (e.g., map, alter, join, and partition) and ML operators (e.g., object tracking, face recognition).

Current projects of the Federated Data Systems team go beyond WWS and XStream functionalities.

• OmniDT - a federated and distributed data fabric system for Digital Twins that enables interoperability across vendors
• Resilience and elasticity/evolvability of stateful applications (i.e., dataflows that maintain the application context and contain window, join, among other stateful operators) with minimal service disruption
   

Applications

Applications of federated and distributed digital twins are to be found in data intensive sectors like industrial automation, energy grid orchestration, intelligent transport systems, smart cities, …