How CSPs can use NetOps to speed up service delivery

The move to the cloud and subsequent rise of webscale providers ushered in a new approach to compute and  application operations, known as DevOps.

The key foundation for DevOps is a cultural one that transforms the way IT developers, testers and operators collaborate during the development and delivery of IT changes.

By working cohesively within an Agile framework, these teams roll the new application or change into production with the help of toolsets that enable increased automation, version control and collaboration. This approach tears down of the boundaries between the develop, test and operational processes, with developers taking accountability for the operational capabilities of the solution.

This is a completely different world for traditional communications service providers (CSPs), which have skills based around large waterfall deployment and upgrade projects that take months or sometimes years to complete. The resource intensiveness of this model means that a CSP may make the conscious decision to upgrade a network platform only every two to three years, skipping major system upgrades and sacrificing feature velocity because of the complexity and cost of these traditional project processes.

There is a huge gap in implementing a change in culture and processes between an enterprise IT team and the network design and operations teams within a CSP. For a start, the enterprise team is focused on its own IT environment, whereas the CSP has thousands of customer services that rely on its stability and quality delivery. So completely changing the culture around how CSP networks are designed, deployed and operated is a tough shift for CSPs to make—but it can be done.

One innovative CSP that made this jump is Spark New Zealand (formerly Telecom NZ). Spark’s leadership changed the company’s business operations to Agile in 2018. This change has enabled significant growth in culture, employee engagement and customer satisfaction. McKinsey wrote a great article on this journey.

Just as DevOps has changed the delivery of IT projects, we are seeing NetOps as a similarly agile and dynamic way to manage IP networks. NetOps includes the same foundation as DevOps: a shift in culture and a new set of operations and automation tools.

At this point, some may ask: If the old way works, why change?

There are a couple of ways to answer this question, and depending on who in the CSP organization is asking, the answers could be tilted towards the fiscal benefits or the to-market and competitive benefits. The latter is what I'll focus on.

All CSPs are feeling the competitive threat of the cloud and the over-the-top application providers. Competition is coming from many angles. It could be directly from large webscale providers and X-as-a-Service vendors, from ICT vendors with enterprise-focused SD-WAN and cloud security offerings, or from nimbler CSPs in the mobile and fixed services space. The result is that enterprises perceive that the cloud or application provider delivers more value than the CSP.

The world where CSPs compete with one another for customers is waning. The competitive landscape is shifting to digital and cloud native providers. It’s an unfortunate, and in many cases unjustified, shift in perceived value, but it’s affecting all industries.

Think about when you buy something from Amazon Prime. If your package arrives early, you think Amazon is great. If it arrives late, you get grumpy at the courier service. The same is true with enterprise IT. If a new SaaS application can be purchased in a week, then why does it take a month or longer to get the network to support it?

NetOps brings responsive, customer-focused networking to the CSP but it requires changes in culture and operational toolsets. I'm no expert in cultural change but do look at the Spark NZ example. What I can talk to are the new operational toolsets and a focus area that will deliver operational cadence and shift the value back into the CSP network.

To start, the network needs to be broken down into domains and a view needs to be taken on each domain in relation to NetOps service delivery. When we think about CSP IP networks, there are four domains:

• IP core, which includes the MPLS P routers and core optical transport layer

• IP services edge, where MPLS provider edge (PE) routers perform service edge functions such as BNG, ePC, IP-VPN/VPLS services and internet peering

• IP aggregation, which provides the metro ethernet aggregation layer between the access and service edge domains

• IP access, which connects the 'last mile' fixed (dark/lit fiber and GPON) and cellular (mobile broadband and fixed wireless access) connections to the network.

Each domain will have differing operational requirements that drive the need for NetOps toolsets and operational culture change. For instance, major changes to the IP core domain are infrequent, so it’s probably not the first place to implement NetOps. But with IP aggregation, there is a set of market drivers that could benefit from NetOps.

High-touch items such as the management of alarms, deviations of configurations, and platform and node upgrades are a great starting point. Streamline this area and you will go a long way towards getting agility into the IP aggregation domain.

Step one here is to build the IP aggregation network with routing nodes powered by an extensible network operating system (NOS).

The need for NOS extensibility has grown from the world of webscale and cloud providers, where individually managed microservices are fundamental to their architectures. The fast-paced world of cloud networking requires a massive increase in the use of automated workflows and toolsets, along with a move to a more machine-to-machine-based operating environment. That’s not to say that human-based operations are less important, but the number of network services and level of telemetry data demand machine-based automation to separate the signals from the noise.

With an extensible NOS framework such as Nokia SR Linux, the operating system is architected with modularity and microservices at its core. It delivers isolated protocol functions, telemetry at scale, deep service visibility and a new approach to configuration and operability that utilizes NetOps tools like YANG and Python-based CLI. It also adopts a community-based feel with toolsets such as the Nokia NetOps Development Kit (NDK) for CSPs to create and share their own network applications and workflows.

With an extensible NOS in place at the IP aggregation edge, the toolsets required to manage alarms, analyze and report on configuration deviations and perform platform upgrades can all be simplified.

The extensible NOS brings power to the network operations team and provides them with a pre-analyzed and conditioned set of information that they or higher-level tools can use to make informed decisions on the network. The result is that operators become automation engineers with a higher understanding of the running condition of the network. To see this in action, watch this humorous explanation of how NetOps provides an evolutionary step forward in operations.

With these toolsets and a matching culture change in place, communications service providers can move on to step two, which is to upgrade and streamline processes, including the legacy process of upgrades every 2 to 3 years, though the adoption of Continuous Integration/Continuous Delivery (CI/CD) models.

The results are a more consistent workload for the service development and operations teams. Gone are the days of peak and trough waterfall projects with the need to hire external skillsets for the duration of the upgrade, only to let them go and lose that institutional knowledge. It’s time to move to a NetOps model where staff skills are iteratively built around CI/CD, and where team members are constantly immersed in automated iterations on the network. 

One of the many things the cloud has brought is an acceleration in network service consumption. This has put significant demands on legacy operations.

As a CSP, you need your network to be more dynamic in the face of market demands, which requires a rethink of the culture and toolsets.

The only path forward is NetOps, but it’s a big step. That’s why it makes sense to choose a single domain such as IP aggregation to start and build the tools around. From there, you can expand NetOps to increase the efficiency across the whole network so you can be in a better position to deliver as cloud networking demand increases.