Powering routers for 400GE and beyond
The revolutionary space and power efficiency of 400GE pluggable coherent optics enable router interfaces to significantly increase their reach and achieve the same port density as short-reach gray router optics. But while 400ZR and 400ZR+ router-pluggable coherent optics can be equipped in the same QSFP-DD cages as 400GE short-reach gray optics, their power consumption and heat dissipation are much higher. With 400GE deployments picking up steam and 800GE not far off, there is growing concern about routers’ ability to power and cool these high-performance optics.
Power consumption of router optics
The increasing silicon densities brought by Moore’s Law have steadily produced ever-faster and more power-efficient routers. Routers built with state-of-the-art silicon such as Nokia FP5 consume less than 0.1 W per gigabit forwarding capacity, or 40 W per 400GE port, without the optics. Unfortunately, these efficiency gains must all be reinvested to power pluggable optics that support faster line rates, more digital signal processor (DSP) capabilities and an extended reach (coherent line optics). Coherent 400ZR+ optics consume 19–23 W per QSFP-DD cage, while the design target for 800GE optics lies in the range of 23–25W per QSFP-DD800 cage. Although the first 800GE optics (available mid-2022) will be 2x400 and 8x100 breakout optics, 1x800GE optics will follow soon after.
The net result is that the power dissipated by pluggable router optics has steadily increased over time and is a significant dimensioning factor for system power and cooling capacity (Figure 1).
Figure 1. Increasing power consumption of pluggable optic
This is not to say that high-speed optics are less power-efficient, because the opposite is true. One port with 400GE optics consumes up to 50 percent less power than using four ports with 100GE optics, and 800GE optics will reduce power consumption by another 25– 40 percent compared to using 400GE optics. Even so, continuing to scale router capacity in the existing space and power footprint is a formidable engineering challenge that has far-reaching implications for equipment design practices. Adequate power and cooling are critical requirements for network operators seeking to take advantage of the CAPEX and OPEX savings of IP-optical integration.
Designing routers with foresight
The operational deployment lifecycle of routing platforms in carrier networks easily spans a decade, during which platforms must be field-upgradable to meet ever-increasing traffic demands. It is relatively cheap and easy to equip higher-density line cards powered by faster silicon. But upgrades to power and cooling are more cumbersome, and replacing the chassis is prohibitively expensive and operationally disruptive.
Remedies such as introducing port configuration restrictions or cranking up fan speeds may leave valuable system capacity stranded or compromise Network Equipment-Building System (NEBS) compliance. Routers that have reached their capacity ceiling can sometimes be repurposed to address more moderate requirements in other parts of the network, but a forklift is still required to solve the principal platform scaling issue.
The challenge is to continue scaling capacity in the existing footprint, and to accommodate this need in forward-looking system design practices that are cost-competitive, eco-sustainable and able to produce more enduring returns on investments. The thermal designs of the chassis, line cards and pluggable optics are all critical scaling factors. Together, they determine a router’s ability to efficiently cool all interface ports equipped with coherent optics. State-of-the-art routers such as the Nokia 7750 SR-s use an orthogonal cross-connect design to ensure an uninterrupted front-to-back airflow through the chassis. Conventional chassis designs that use a backplane or midplane need to pull the hot air up through the chassis before it can be expelled.
Decoupling the power subsystem from the line card shelf can enable a more flexible and cost-efficient scaling of power entry requirements as the system evolves. It also provides investment protection when a line card shelf is swapped out for a higher-capacity variant. Inserting a switch fabric module in the rear of the system helps to maximize the useful real estate on the front of the line card shelf. Extra-wide line card slots increase air intake and allow belly-to-belly placement of small form-factor pluggable (SFP) cages on a double-sided printed circuit board (PCB) for better cooling of optics and application-specific integrated circuits (ASICs), as shown in Figure 2).
Figure 2. Line card design options
There are many more thermal design features that can help optimize cooling and avoid hot spots, including the design and operation of fans and power entry modules. However, these features are beyond the scope and intent of this blog. Ultimately, it is a combination of many design factors that determines a system’s current capabilities and future potential.
Depending on their role and performance objectives, different systems may successfully leverage different designs. Routers that are cost-optimized for data center networks typically have stacked SFP cages and narrow slots because they need only equip short-reach gray optics with more modest power and cooling performance requirements. On the other hand, high-capacity aggregation, edge and core routers designed for wide area carrier networks must support high-power, long-reach coherent optics that can dissipate twice as much power per port.
The Nokia 7750 SR-s service routers are designed to support pluggable coherent optics that consume up to 26 W per QSFP-DD cage at normal operating temperature (i.e., at 40 C) with a margin to spare, and without running fans at full speed. The current line cards based on Nokia FP4 silicon allow 400GE QSFP56-DD and CFP2 digital coherent optics (DCOs) to be mixed and matched in the same chassis to support different link capacity–reach requirements.
Next-generation line cards based on the Nokia FP5 silicon support QSFP-DD800 pluggable router optics with 2x400GE and 8x100GE breakout initially, and will support 800GE DCOs as they become available in the market. So when 800GE comes around, it’s good to know there is no need to shop around for a new chassis!
To learn how Nokia’s next generation routing technology can help you deliver sustainable growth, please read the eBook “Master the unexpected with Nokia FP5: The power of network processor innovation”.
