Network Automation Makes for a Smooth Operator

For a telecom to become a smooth operator under 5G, the CSP must embrace network automation. du Telecom’s Basel Elabed explains that the first hurdle is human, but that ultimately companies can embrace the evolution of the industry from the ground up.  

Below is a transcript of this podcast. Some parts have been edited for clarity.  

Michael Hainsworth: For a telecommunication service provider to become a smooth operator. The CSP needs to embrace network automation. The quantitative benefits of network engineers, focusing on revenue, generating projects, instead of monitoring that infinite number of screens filled with never-ending blinking alerts are real, but it's not just about security. It's about a wholesale rebuilding of the way a network operator operates. Nobody knows that quite as well as Basel Elabed, the director of transport planning at du. His work in the United Arab Emirates has helped the company build on its expertise, reduce the time it takes to roll out new network services and ensure customers get the service they expect when they expect it. And he tells me automation has evolved dramatically over the years, especially with the introduction of 5G.

Basel Elabed: Automation has evolved quite a bit. We started off with the complete manual way of executing our day-to-day work and projects to somewhat semi automated platforms with semi-automated work execution. And eventually we're looking at going into full automation, having a majority of our day-to-day tasks done through NSP (Network Services Platform) and tools like an NSP.

MH: It's the kind of thing where we really don't have a choice. Do we, we have to go out of,

BE: We have to automate, there's quite a bit of resistance coming into automation with the mentality and you know, the human elements of jobs and job security. So there's quite a bit of resistance, but we've been put in a spot now where we don't have a choice, especially with evolving networks, 5G coming into the picture. You know, the customer's expectations are changing so rapidly that we don't have a choice anymore. But the key element to automation is the human factor and how acceptable automation is to humans because ultimately we can make it or break it. So we can keep complaining about that. It's not there a hundred percent. So the thing with automation and in our journey, what we have learned, is that we as humans tend to want something that will do the job a hundred percent or not want it at all. Or that's the excuse we use that I cannot do this because it's on a hundred percent automated, which has to be changed. The human aspect has to change it.

MH: I can imagine there is some pushback to your point. There has been some concern about job losses, but automation really isn't just about saving money. It's not about the job situation. It's about increasing network resiliency. It's about accelerating time to revenue.

BE: It is about that. It’s about increasing resiliency. It's about like you mentioned revenue. It's also about human free rollout or a human free intervention with all the changes that happen. Because the biggest contribution to outages is human intervention because it's the human intervention and the mistakes that cause generally the outages compared to when a device misbehaves. So, with automation, the message that has to be passed around. And I think we, as an industry, as telecom industry we've lagged in that perspective is that you've got to evolve as a network engineer. I studied programming in university and I went on to work as a network engineer and a network engineer 10 years ago is not the same network engineer that is out there today because you're expected to do a lot more because of all the visibility that you have in the network.

You're not expected just to connect a site A to B, you're supposed to do it in a smart manner, which can increase efficiencies and bring new revenue streams. So I think the emphasis, although it boils down to cost saving is more on how do we re skill our staff to be able to adapt to this change and in the process, give them new skills to really start working on different jobs and in different markets. If you look at the likes of Amazon and Google and so on and so forth, they have nailed it pretty much. They've done a good job at it. They've hit the nail on the head where their staff are re-skilled now to do more than just be in network engineer.

MH: We should probably get a misconception out of the way as well. Automation isn't as much about a software package that you buy. you install, you plug into the network as much as it is about delivering a project.

BE: One thing is for sure, that it's not an off the shelf product, the misconception is that I’ll bring NSP or a tool like NSP into the network and you know, all magic and it's all there. You know, again, as an industry, we've failed to really talk about this in detail, because you know, when the CXO level, they meet at a round table. Everyone sort of looks at the, oh, it's a click away. So you just click the button and it's there. But when you get into it, you know, like they say, the devil's in the detail. So when you really get down to it, every network is different than, and there, you know, I have different ways of rolling out due to my constraints compared to my competitors. And so that governs how a typology of the network is. And like you said, it's delivering an end to end project rather than delivering a subset of a task, but it gets built on tasks. So you can't expect also to be a hundred percent automated from day one. It's a journey that you have to take. And you slowly, slowly, you know, it's like building a building, you can't build a 50 story building with the top layer first. You start with the foundation and then first floor, then the second floor and you, and you go on, go on, go on from there. 

MH: I like your analogy of building a 50 story tower and you don't start at the top. You start with the foundation. What makes for a strong foundation for automation?

BE: Information. Information related to our customers, topology, information related to our network. What we have deployed, where. Topology is at the essence of network automation. Let me talk about network automation. So, having the right topology, what has been implemented on ground versus what we think on paper. So there's a big deviation between when we sit with our partners, like Nokia, set out an LOD, a low level design, expecting that we're going to roll out that design, that we've put on paper compared to what actually happens on ground. And the changes are not necessarily happening due to human errors, but it's the external effects that cause the changes. So there's a big deviation between what we think is there and what has been implemented. So when we started that automation journey, we suddenly find out that we have five or six different topologies.

And then we don't have enough information about our customers or because of the manual setup of automations, that there are different let's say, naming standards. So we had a big problem with our inventory project, where we could not really define our you know, devices properly because different people were using different naming conventions because of the human aspect. So, you know having accurate information is really key and having accurate topology is, again, another of foundation stone for you to build on. If you have that nailed down, you will go a long way in a very short span of time and get it delivered.

MH: I read a fascinating report from Analysis Mason that concluded that time to revenue aspect of automation falls 88% with time to revenue defined as the length of time from a contract execution before the relationship begins to provide revenue, how does network automation accomplish this?

BE: I'll give you an example, which emphasizes on what [Analysis] Mason said in their report. We used to do software upgrades in our network in a very manual way. So the pre-check, depending on the device of course, but let me talk about the big routers. So doing changes before we went ahead and did the changes we had to do a lot of pre-check them and post checks. And sometimes these pre-checks would go into an eight hour cycle to make sure that we're good to do an upgrade with automation that eight hours went down to 40 minutes that applied to pre and post. So we had another eight hour cycle post upgrade to make sure that everything is working as it was before. So again, that went down to 40 minutes. So previously doing an upgrade of a core router or an aggregator was a daunting task. And it would, it would be a full day, if not more, that would be required. Now we do three or four a night. So having new software rolled out into the network means I have more features and capabilities that I can offer my customers, which in turn, boils down to a faster time to market and revenue and better revenue for the organization

MH: What are the best practices associated with automation being executed to help reduce time to revenue. What are the tricks? What are the traps

BE: We're still learning. I'll be very honest. We're still learning the tricks, the traps but generally it is not to wait to get that full building. Let's start off with whatever is automated out there and really use it. And then slowly, slowly work on that picture, perfect world that we require. So a lot of the use cases that get developed, I'll give you an example. We started our enterprise automation journey and we said, okay, as a stepping stone, let's, let's only do a subset of the use cases. So let's only do activating an Internet customer, and then let's build on from there. It was countered and people were saying, no, no, we won't use this till we have the whole enterprise suite covered, which really puts you at, you know, on the back foot, because that's going to take a lot of time.

It's not a walk in the park and it's not a day's job that you do it. So it's a really use what is available. And go ahead with that till you build till you build the full suite, a suite of services that you require. So really I would say that there's the trick to accept what have, and really start using it and not to fall in the trap of having the complete implementation done from the get go. The other thing that also that I would like to talk about in terms of tricks is to make sure whatever is getting deployed newly into the network is automated from day one and not to wait because otherwise it's a catch-up, you know, you will always be catching up with what is coming in new. So really having these to nail down will help quite a bit.

MH: Also back to your analogy of the 50 story tower, that that's a fascinating twist to it. The idea that once you've built that foundation, you start putting a couple of floors in and you don't have to wait until that is 50 stories tall to start enacting automation. You can do that in the lobby.

BE: Oh, for sure. So you see, once you have the foundation nailed down, like having, having the customer information and topology nailed down across the network, be it IP, optical, or microwave or radio, there's nothing stopping you, cherry picking, which use cases you want at which part of the network and automating those to have that entire building. So the lobby can be a 10 meter by 10 meter room, or it can be a 50 meter by 50 meter room, depending on the use cases that are out there for you to pick on. So, you know, in our journey, we started automation use cases on our enterprise segment, and we started doing some of our mobile rollout, automating some of our mobile rollout, because that is our revenue stream on when a mobile site goes live is when we start capitalizing on the income that comes out from it. So the quicker that site is live, the more incumbent is for us. So we don't have to really wait to finish the enterprise to start the mobile. They can fo simultaneously all you need is the teams to work on it and have the different point of view to get the job done.

MH: With automation it's critical to service fulfillment under 5G it's estimated operational costs can be cut by 86%. What are the key issues?

BE: Not only with 5G with overall is that you know, we get our network rolled out as quick as possible. So if we look at it from 5G perspective previously in enodeb were rolled out at a gig, and that was looked at enough capacity per site, but now 5G comes in and you have 10 gigs. You have 25 gigs that have to be rolled out. So really we are going through a complete change on the full mobile access network because of 5G. So rolling it out in the fastest possible manner with the least amount of outages is really key to the whole story.

MH: What are the biggest operational costs that can be automated?

BE: It's a wide range of things. Software upgrades are our first thing that can be automated and help with cost reduction. Link upgrades. So really looking at the network and figuring out a  which links are saturated and how quickly you can upgrade those links. It can incur an upfront CapEx cost, but it will reduce your time to market because now the upgrades are, are done quicker. So you have the ports available. You have the fiber available, activating a port is a click away, honestly, speaking as long as you have that topology information nailed down. So doing software upgrades, doing link upgrades. Traffic load balancing is another use case that we're heavily looking at. And we really looking at having analytics pumped into load balancing. So depending on the traffic type and the analytics that you cover for that, you gather from the network load balancing different traffic types, different traffic on different links, depending on the shortest path, depending on the latency, depending on things like that. You see all these use cases couple of years ago were talked about, but today are  really being kicked in and are being asked either from 5G networks or from our enterprise customer. So our enterprise customers are no longer, you know, waiting. They're the ones who are demanding now and asking for all this information to be shared with them, not only how the networks are using it, but they also want a visual insight on what they're doing and how they can enhance it even more.

MH: I can imagine provisioning is a critical component as well, that requires automation.

BE: Oh, for sure. Be it our internal. So our mobile network is our internal customer for external customers. Be it our service layers. So service layers are our internal customers. Also, everyone is asking to have their project completed in the fastest possible time with the least complications and with the least human errors. So really you, you look at all the aspects. If you look at a time to market, if you look at human errors, and you look at faster deployment, automation is at the cornerstone of all of this to make sure that your project is executed in the fastest possible way. So with tools like NSP and like, we're really seeing that fiction turned into reality,

MH: I suppose, the most understood value in network automation is for security and resilience. It's not much of a surprise that the labor time and the cost associated with alarms falls, 98%.

BE: Yes. Previously, we ran an exercise where, where we looked at, how long does it take for an engineer to sift through and really look at alarms that matter to them versus the alarms that don't. And we found, you can save around four to five hours of an engineer's time, just by suppressing the alarms. You could save even more by correlating how a certain link, which has been cut off from the network is causing issues across the services that you have been providing on that link. So, and direct response to this is updating a customer with a message telling him that we know that you have an outage on your fixed services and we are working on it. So that alone saves a bombardment of you have calls on your call center, which again is only achieved if you have the information, you have the correlation and you have the automation to send out that message. 

So there, there are indirect and direct aspects that come into play here with automation, which sometimes becomes hard to quantify. So yes, alarm suppression is something that can be quantifiable, but, but calls to your call center, you know, again, is some customers call on some sort of a situation becomes hard to justify, but for sure it does have an impact on the customer experience. So when you send out a message to a customer that you're working on his issue, you know, it's a happier customer and our customer who knows that his services is really being looked at, rather than him calling and complaining that his services is not up. So that's an intangible you know, outcome of automation.

MH: So when we come full circle here, I suppose one of the most important takeaway elements to this conversation is that when a CSP needs to start with network automation, implementing a single use case, that helps you prioritize quick wins is going to have the biggest impact, just get at it, start at the basics, work your way up.

BE: Low-hanging fruit are always the ones that you should be running after, rather than building that 50th floor. And let's go back to the analogy of the tower.