Streaming telemetry with open source gNMIc

by Charuhas Ghatge

8 Dec 2023

The need to reliably monitor today’s ever-growing networks in the WAN, edge, and data center domains is more critical than ever. The collection of data from devices and their transmission to a receiver for monitoring needs to be done in real time using streaming telemetry. It is essential for network visibility and, in turn, helps in network operations, automation and planning.

One of the most robust frameworks for streaming data collection is Google’s remote procedure call (gRPC). There are a number of gRPC network management interface (gNMI) streaming telemetry collectors. The best of these gNMI collectors feature high availability and clustering of collector instances, rich data processing pipelines, sheer support for various outputs for metrics, and embedded caching. One of the most promising of these gNMI collectors, gNMIc is the open-source project being supported by Nokia.

What is streaming telemetry?

Streaming telemetry is the comprehensive practice of transmitting measurements from various network devices to a collector for storage and further processing, ultimately creating a monitoring and reporting system.

Steaming telemetry uses a push-based mechanism that transmits data automatically and continuously from various remote network devices such as routers and switches to a central repository. Selecting a proper telemetry architecture and protocol can potentially remove many issues—such as security, scaling, polling gaps, and resource utilization of the polled device—associated with sending and receiving streaming telemetry data.

Streaming telemetry and SNMP

The alternative to streaming telemetry data is the widely used SNMP (simple network management protocol) a pull-based model where the monitoring system periodically requests data from network devices. This method can cause delays in detecting issues and consumes considerable resources on both the monitored device and the monitoring system.

In contrast, streaming telemetry’s push-based approach allows network devices to stream data continuously and automatically to a centralized location. This approach enables near real-time network data collection, significantly improving the visibility and responsiveness of network monitoring. Streaming telemetry also leverages standardized data models, such as IETF YANG and OpenConfig, promoting consistency in the data collected across various devices and vendors.

What is gNMI?

gRPC Network Management Interface (gNMI) is a protocol to manage and report a network element’s configuration and operational data. gNMI is built on top of Google’s widely deployed gRPC protocol. gRPC provides protocol buffers (protobufs), a language neutral mechanism to transport structured data efficiently over networks making it an ideal method for streaming operational data from network elements. Listen to the Packet Pushers podcast “Configure Devices, Stream Telemetry With Nokia’s Free, Open-Source gNMIc” to learn more.

Figure 1. gNMI protocol stack

gNMI has a number of advantages:

A single service for state management (streaming telemetry and configuration)
Ideal protocol for streaming telemetry with its push mechanism from devices (vs pull from the client), making it efficient, near-real time, and reducing the processing load on both the network device (server) and the monitoring application (client)
Supports very efficient serialization and data access
3-10x smaller than XML
Offers an implemented alternative to NETCONF, RESTCONF.

gNMIc and OpenConfig

A few vendors and organizations have developed their own clients for gNMI and there is no standard that defines a way to develop a gNMI client. Nokia developed a gNMIc client and provided it to the open-source community OpenConfig. Although it is not a de facto standard for a gNMI client, gNMIc is widely used by customers, the open-source community, third-party developers and even networking vendors.

Nokia maintains the source code for any upgrades as well as for backward and forward compatibility with other open-source tools that gNMIc interfaces with. There is a single binary that can run on any platform and is not dependent on any other software components that need to be installed.

gNMIc vs gNMI

gNMIc is a client interface to gNMI. gNMIc is also a collector for the data transported by gNMIc and it provides Go APIs as well.

Figure 2. What is gNMIc and how does it relate to gNMI?

In simple terms, gNMIc is to gNMI what an SNMP client is to an SNMP agent. Like SNMP, gNMI and gNMIc have Get/Set. Unlike SNMP, the gNMIc provides an additional directive, Subscribe, which lets a client ‘subscribe’ to a stream of data (denoted by ‘path’ in the YANG model structure). The network device sends this stream of data either continuously, at prescribed intervals or when the data changes. View the quick reference to learn more about gNMIc.

gNMIc as a command line interface to gNMI

There are four basic gNMI RPC CLI commands of importance.

gNMI CapabilitiesRequest

This command gives a gNMI client information from the device about the gNMI versions it uses and the data models and encodings it supports.
gNMI GetRequest

To get the information from a device, there are two options: Get and Subscribe. The GetRequest is most often used when retrieving a small amount of data. The SubscribeRequest is used for retrieving larger amounts of data, such as streaming telemetry.
gNMI SetRequest

The gNMI SetRequest is used to set, modify or delete configuration parameters. It consists of an ordered set of edit operations. This procedure is atomic, which means that if any of the set requests fails, none of the edits get applied.
gNMI SubscribeRequest

The SubscribeRequest is used for larger queries of data such as telemetry. In the case of telemetry with gNMI, the subscriber (client) specifies a frequency of delivery, which could be:

ONCE: The data is returned immediately and only once for all specified paths
POLL: The data is returned from the device when polled with the current values for all specified paths
STREAM: The data is returned continuously. This could either be in the SAMPLE mode (where data is returned periodically as specified with a sample interval) or in the ON_CHANGE mode where the data is returned if there is a change in values.

gNMIc flexibility

The design goals of gNMIc are to be flexible, scalable, open, highly available and easily deployable.

Figure 3. gNMIc is a highly flexible and highly available collector of gNMI data

The gNMIc collector, as illustrated in Figure 3, provides flexibility to direct data to any of the entities that can consume it, such as:

Any database such as Prometheus or influxdb
Message queue such as NATS or Kafka
Terminal (to display on a console) or File – to be processed and consumed by any application.

Please note that the data, once received can be directed to multiple streams, for example, to Prometheus, Kafka and influxdata, simultaneously.

Figure 4. Versatility in pre-processing the data before sending to output streams

gNMIc provides further capability to process the data it receives from the devices before it sends it to the database. The output data may not be in the format that the database expects or the data needs to be manipulated either enriched or converted to a different type (e.g., from string to number). To facilitate this manipulation, ‘processors’ need to be defined. There can be more than one processor, and they are executed in the order they are defined.

Figure 5. Scalability and high availability

gNMIc is designed so that multiple instances of gNMIc can form a cluster. Clustering many instances can provide the scale needed to manage thousands of devices. Clustering also provides HA (high availability) functionality so that if an instance goes down another member of the cluster can take over. gNMIc also allows automatic target (devices) distribution so that the load among the cluster members is shared equitably.

Easy deployment

Nokia’s open source gNMIc is compiled as a standalone binary so that it can be deployed and executed on any system, and it has no external dependencies on any other software components.

Summary

Telemetry involves the collection of data from devices and their transmission to a receiver for monitoring. The ability to collect data in real time is essential for network visibility, which in turn helps in network operations, automation and planning. Traditional protocols such as SNMP are insufficient to provide the required streaming telemetry. The gNMI protocol is a unified management protocol for configuration management and streaming telemetry. The gNMI specification is openly available at the OpenConfig repository.

Although the gNMI protocol is well defined, the client to consume the gNMI data is not and all the vendors use their own implementations. Nokia developed gNMIc and has contributed it to the OpenConfig community. It is widely accepted by Nokia’s customers, the open-source community, third-party developers and competitors alike.