Good network management and monitoring has always been crucial to the running of a carrier’s business, and the heart of any network management strategy is the network operation centre (NOC).
Traditionally, the NOC has had a fairly straightforward function. In the NOC of everybody’s imagination, a small team of people work in shifts to keep a watchful eye over a large bank of screens, with little to occupy them until an alarm of some sort signals a network outage.
However, we are rapidly entering a new era where mission critical services and applications are required to run 24 hours a day, seven days a week. As a result, the function and importance of the NOC is changing fast.
“In the old days, operating backbone networks was a rigid and predictable business without much ever changing in terms of how networks were managed and how they behaved,” says John Wick, senior vice president of the network division at Syniverse, the IPX backbone operator. “There wasn’t too much for a NOC to do. However, IP has changed all that – you now have up to 10Gbps in a single interface carrying an infinite number of services.”
A new type of NOC?
Richard Brandon, chief marketing officer for MLL Telecom, a UK-based provider of managed network services, argues that a whole new type of NOC is now emerging, in response to wider evolutionary trends within the telecoms sector.
Brandon says that a number of things have changed in the market to create the need for a next-generation NOC: “The importance of the cloud is perhaps an overdone topic, but it has nevertheless driven the need for greater resilience,” he points out. “Data now needs to be mirrored, in case of a problem. The use of browser-based systems also creates a need for resilience and for optimised network performance, particularly with so much remote access onto a range of devices. You can now offer people a view of the network on their tablet. Network managers demand a high-level view of what’s going on with a network. They want the ability to correlate different sets of information so that they can look for trends.”
NOCs, he says, have always been there to pick up alarms, but now there’s a need for them to see changes in trends and so spot anomalies before they happen: “Better to prevent that than have to send an engineer in to fix things,” he claims. “It’s more about health management now than it is about alarm monitoring. You’ve got to be looking out for things like congestion.”
Speaking of MLL’s own NOC, Brandon notes: “We build and manage special networks for other operators, often mobile operators, as well as other organisations like local government bodies. They might be looking for a shared network for economies of scale, perhaps connecting shared data centre space. You need the right NOC to manage that.”
Customer-driven change
Ethan Blodgett, vice president of network operations and customer care with US and European network operator Masergy, says the changes in the nature and role of the NOC are for the most part customer-driven: “Going back 15 or 20 years, a typical NOC (GNOC) was all about monitoring the backbone or core or transport part of the network. It wasn’t customer facing at all,” he says. “Now customers are expecting our GNOC to be aware of network problems before they are.”
Blodgett says that 15 years ago when he joined the industry, company goals were different: “There’s been drastic change, and a lot more automation to meet that change. Automation has been used in the past, but more now as these networks have become mission-critical. In any case we can’t just hire 20 people to throw at a problem. It’s about speed and accuracy. Outage is not tolerated and fault tolerance is expected. The customer doesn’t want to be spotting the problem themselves and then chasing you down. Now it’s us calling them, and escalating internally until a problem has gone.”
Customers, says Blodgett, want tools and toolsets that give them information quickly, and they want support at a much more granular level: “Our customers are in the main Fortune 5000 multinationals, based in the US or the UK – companies with 100 or so worldwide sites in major cities,” he says. “Our base line product has full network monitoring factored in as standard. With a lot of carriers that’s an added extra.”
Supporting networks is an increasingly global business, demanding a NOC that matches that need, argues Ashwini Bakshi, head of global managed services with Nokia Siemens Networks, a company that handles outsourced network management on behalf of a range of carrier customers. He says NSN has five GNOCs: in Sao Paulo, Chennai and Noida (both in India), Lisbon and Moscow.
“We remotely manage 540,000 network elements and 200 million subscribers on our customers’ networks through the GNOCs,” he says. “We employ 2,550 experts, and handle close to 100 million alarms per month.”
Bakshi relates the fast-changing role of the GNOC in part to seismic changes in the market for mobile services: “Since the debut of the iPhone 4 and the iPad, and other launches of that kind, there’s been a major impact from data traffic on networks,” he says. “This has necessitated investment in capacity, naturally. But it also means that the whole design and operation of networks is now approached in a slightly different fashion. With a traditional NOC, the main role is surveying the network for faults and putting up alarms where they are found. Depending on the severity, this will either mean a first-time fix or sending someone out. The big issue now though is not really outage, but performance. Alarms are no longer enough to point out problems, and the NOC has got to be a lot more proactive. The NOC needs a new set of skills and tools, and we’ve been developing these.”
He says all five NSN GNOCs feature alarms that trigger when capacity utilisation reaches 60%, and again at 75%: “At 95%, you need crash action to relieve the pressure,” he says. “You need tools that scan daily, and compare readings with yesterday, last week, last year and so on. You’ve got to look at how traffic is shifting in the metro areas, such as between morning and evening when data flows reverse. NOCs need to offer this level of granularity, and it demands new skill sets from the people who work there too. It’s about customer care as much as tools, and we work hard on that.”
The mobile data era
The onset of the mobile data era has made the job of monitoring the performance of cellular networks a great deal more challenging for the operator, agrees Richard Stone, mobile solutions director at software vendor Compuware.
He believes that unlike previous voice-centric network management models, where operators essentially had complete control of the end-user experience, mobile data services have introduced a high level of complexity, and consequent problems for the NOCs and GNOCs tasked with making that experience as positive as possible.
“This is because the ‘service delivery chain’ for mobile data involves multiple devices, applications, networks and content providers – most of which are out of the NOC’s direct control and influence,” he says. “The only way for NOCs to ensure a good mobile data experience is to understand how each part of the delivery chain impacts performance on the end-user. Subsequently, the best way for NOC teams to gain an insight into problems is by starting with the end-user’s experience of mobile data services.”
He says that much of Compuware’s current work involves helping mobile operators improve the performance of data services through supplying solutions for application performance management. “The end-user experience is our focus,” he says. “The old ways of managing application performance is obsolete. The days of ‘voice only’ and the era when the network was the solution are gone. Data has changed everything dramatically. Now the mobile operator has got to monitor the device, the application, the carrier network and the content that the customer is trying to access. When a wired broadband network goes wrong, the carrier does not necessarily get the blame. But when the same thing happens to a wireless network, the mobile operator is always seen to be at fault. They face complex problems – just look at billing alone.”
He claims Compuware’s solution lets the operator take a user’s view of the problem: “They need to understand the whole chain, not just the bit they actually own.”
Stone’s vision for the future is one where the mobile world and the cloud will merge to become one: “Mobile operators will be the centre of this new ecosystem, which is why we are focussed on them. This is a huge opportunity.”
Size is everything
When it comes to monitoring and managing networks, size is everything. And that, maintains Chuck Kerschner, director of AT&T’s New Jersey-based GNOC is the case whether the challenge is wired or unwired.
“Our GNOC monitors and manages the data and voice traffic flowing across AT&T’s domestic and global networks. We have a sweeping wall of 141 giant screens showing different aspects of network activity, network topography and news events. It is the largest and most sophisticated command-and-control centre of its kind anywhere in the world.”
The AT&T GNOC, he says, operates like a thin layer on top of all the company’s networks, giving an overview at Layer 1, 2, 3 and 4: “There are additional consoles with hooks into the individual networks. We can see out across all the different activities. From the customer point of view, the applications they use are changing and embedding deeper into their business. We need to encompass wireline and wireless too. We’re a global company, so what the GNOC needs to provide is a worldwide seamless solution. There are no more siloes.”
Kerschner says that AT&T’s whole view of its networks has had to change too to be a lot more application-focussed: “We use every tool at our disposal, as it just takes one incident somewhere in the world to make a difference somewhere quite different. The way in which faults are monitored and notified is changing too, with Twitter having an increasingly important role. It’s a paradigm shift in communications.”
A paradigm shift
The speed at which the paradigm is shifting is well known to UK-based QiComm, which provides wholesale carrier services, operates as an MVNO, and provides systems integration, wireless network deployment and managed switching services. From a data centre deep in the heart of London’s Canary Wharf, it controls a fully redundant, carrier class DWDM transmission network on a dark fibre ring, connecting all the major London telehouses.
“We’ve got a 24-hour NOC based in London and Manila – 12 hours in one and then a handover,” says CTO Chris Joseph. “A while back there was internal criticism that we were being reactive and not proactive enough. I’ve been trying to upskill the NOC to change this perception. I’ve also ended the siloes in the NOC, separating voice, transport, IP core, IT services and so on. There’s a demand now that a NOC should focus on network performance, and not just availability and meeting SLAs. That has to happen too, but for us now it’s about monitoring flows of voice traffic, and seeing how that flow affects services. What’s needed is a full end-to-end picture now. Customers are more demanding, and that’s not a bad thing as far as I’m concerned.”
Mark Leonard, EVP of the Infrastructure Services Unit for Colt Telecom, believes that changes in the way networks are managed are leading inexorably to the emergence of what he calls the ‘next-generation service provider’, supported in turn by ‘the next-generation NOC’.
“On top of our fibre infrastructure we’ve got 19 data centres which means we offer managed IT services on top of being a network operator,” he claims. “We think a true next-generation service provider should not only offer networks but other services on top of the stack, with a wrap of experienced professionals who know how services should be delivered. We’re investing in capabilities to bring all those worlds together. It’s a work in progress, but at the moment we’re probably ahead of where demand is.”
He says Colt is supporting this evolutionary move by changing the focus of its NOC to services restoration rather than technology restoration: “Customers are not buying a circuit so much as a solution,” he observes. “We’ve moved that support now to just the one NOC. Next-generation NOCs, like ours, will enable carriers to manage their current and future networks and ensure network reliability and efficiency to help reduce churn and increase opex savings.”
The future of NOCs
So who, given this rapidly changing landscape, are the companies whose NOCs will best meet the needs of tomorrow’s communications market? Are they massive and global or small, disruptive and focussed? Who is getting left behind?
Syniverse’s Wick says his company is up to speed with what is demanded of the NOC of tomorrow, but fears that doesn’t apply across the industry: “It’s an ecosystem we understand well, but some operators on the wholesale side still need to adapt their processes somewhat,” he points out. “Your NOC needs to be staffed by the same calibre of people on a Sunday morning as on a Wednesday afternoon. You don’t always see this.”
Blodgett of Masergy wonders if the successful network management providers of the future will be the same carrier names that have dominated the last couple of decades: “The winning network companies of the future will be those that focus on the customer experience,” he maintains. “You want someone who is a proper partner. A network operator that employs 100,000 people is a big ship that doesn’t change direction easily. We’re a sailing boat. Customers are tired of the old mantra of ‘buy from us because there isn’t any choice’. There is now.”
Network management ‘begins at the design stage’
The first step towards ensuring network reliability is giving careful consideration to network management in the design stage, maintains Jay Stewart, director of Ethernet solutions at vendor JDSU.
“This means that OAM considerations and visibility into network performance need to be considered very early on in the network planning stage and not as an afterthought,” he says. “The requirements of the NOC should be an integral part of this process. Due to the complexity of IP networks, it is imperative that both service reliability and network reliability are carefully measured,” the systems specialist adds.
Stewart says another strategy is to ensure that analysts can drill down and harvest raw data about any of the services that run on the network: “This means taking data from multiple sources such as the network and back office systems and mapping it to the IP services to allow operational efficiencies in the NOC,” he argues. “Melding together the network, service and customer experience layers will be key, especially as we move towards more dynamic responsive network designs with automated optimisation. It will also be important to have the ability to understand changes in the network in real-time instead of the traditional 15 minute window.”
Best practices for network operators, he says, should include ensuring that they have the ability to monitor and measure the actual service; maintaining the ability to measure the impact of dynamic changes to the network with both dynamic service activation and monitoring; and designing the network with enough touch points to allow specialists to track a service through the network.
