Interview: Pierre-André Rulmont of BICS

Published on:

Transforming a 25 billion minute network to NGN in two nights

Jamie Anderson interviews Pierre-André Rulmont, the chief technology officer of Belgacom’s wholesale division, about a daring transformation to its all-IP network — in just two days

Pierre-André Rulmont (above) is CTO of BICS, a member of the Belgacom group that operates in the international carrier business. He has been within the company for 10 years, acting in many roles from engineering to IT management, and has been CTO for the past two years.

Can you talk about the kind of business that BICS is in?
Pierre-André Rulmont:
BICS is in the business of international carrier services. We deliver global solutions for voice, messaging, roaming, connectivity and mobile financial services to hundreds of telecommunication providers around the world.
This means that our main activity is to interconnect fixed and mobile operators to provide international services. Typically our customers are incumbent operators but also any size and type of mobile operators from anywhere around the globe.
BICS a joint venture of Belgacom, Swisscom and MTN with a continued focus on value added services for mobile operators and a strong growth strategy that enabled us to reach a world-leading position both on the international voice and mobile data markets.

What is the size of the business?
BICS’s core business is international wholesale telephony services. In this market we are in the top five. To give some kind of reference we have about 5% market share as this is still a very fragmented market. We run more than 25 billion minutes of international telephony traffic on a yearly basis. Our network is interconnected to more than 600 operators around the world.

Can you give us the background to your NGN transformation project?
The company has been growing tremendously over the last 10 years. When I joined the company BICS was running a fraction of the current volume of traffic on 20 year old TDM switching infrastructure.
BICS is operating in a very competitive international market. The success of the company is based on balancing constantly decreasing margins with strong traffic growth, not only via organic growth but also consolidation with the carrier activities of Swisscom and the MTN group. This was putting a lot of pressure on the legacy infrastructure inherited from the past.
The other key aspect in this project is the industry move to all-IP. The international market is certainly on the leading edge. Everybody is talking a lot about it and every operator in the world knows that its network over time will be all IP. The question is what will be the speed of this transformation of the legacy network, especially among the incumbent operator community.
BICS had a lot of legacy infrastructure from the past, representing our core network and was of course an asset in terms of connectivity and in terms of connected operators.
Transforming this to all-IP was a real challenge especially for such a heavily interconnected network. This is a typical international carrier’s issue as most of domestic networks only interconnect with a limited set of other networks.

What kind of transformation strategy did you choose?
We decided to go for what we called a “flash cutover”, meaning to change the complete switching infrastructure in two nights — that is, 50% of the capacity per night. This was highly ambitious compared to the traditional way of transforming a network which usually consists of commissioning a new platform and migrating the customers one by one over a fairly lengthy period of time.
This decision for a cut-over approach was mainly driven by the need of limiting the business impact and the difficulty to build a strong transformation business case. First of all, a slow transition would have in our case meant an expensive transformation in terms of time to deliver and manpower to be allocated to the migration of hundreds of interconnection around the world.
Secondly, a slow transition is straightforward in a fairly static network but in our business we need to realise hundreds of routing updates on a daily basis.
This means that during a traditional transformation process, our network operations teams would have had to operate the legacy and new infrastructures, manage two routing environments, and on top of this the interaction between these while moving customer from one platform to the other. This would have been a very tricky juggling process that could have negatively impacted our business agility.
That being said, most of the carriers around the world still choose incremental transformation of their network infrastructure, as the flash cut-over approach seems too high risk.

Can you talk about the implementation part?
The flash cut-over approach required us to build beside the existing platform a completed mirrored infrastructure which had exactly the same size, same configuration, and same features, while at the same time being completely integrated in the OSS and BSS environment while staying offline.
The flash cut over could then take place, and during one night we switched over 50% of the old infrastructure to the new one and the remaining 50% a second night six weeks after the first one.
Two nights were required to mitigate the risk but also to ensure service continuity to our customer as the transition process took around 10 hours. During the transition, all of our customers had dual connections and we performed the transition during offpeak periods to ensure that no customer traffic would be negatively affected.
From a timing standpoint, the construction of the new infrastructure, the commissioning and configuration took nine months, followed by another three months of preparation for the cut-over itself.

What were the biggest challenges?
The technology learning curve is a big challenge. In a more incremental approach, network operations teams are used to master new technologies with a slow introduction process, starting with a small configuration, adding a few beta customers and growing progressively while solving scalability and integration issues. In a flash cutover everything has to be anticipated and it all happens in one night.
The other big challenge was managing the reference data. Mirroring infrastructure is not just copying the configuration from one switch and putting it on to another. You need to be able to link it to your customer database.
We managed to mitigate the risks of errors in this space using transmission equipment to copy the signalling of the production environment to the offline new infrastructure, while observing the behaviour via our monitoring systems.
The last big challenge was what we refer to as the residual traffic issue. When you are the single gateway of all the telephony of a particular country you need to support all the traffic that gets in and out of that particular country.
And there are plenty of applications that actually you as an operator don’t know about. Customised alarm modems and proprietary or encrypted faxes are typical traffic that runs very well on traditional infrastructure, and are very challenging to NGN infrastructure.
On top of this, you cannot detect these issues during the six weeks’ observation period because those fax-like devices have retry features that will, in combination with the network internal traffic distribution mechanism, ensure that the call will reach sooner or later the old switch. As a consequence, you can only detect them when 100% of your migration is done and no way back possible.

What were the main people and change management issues that you encountered?
Inside our technical community when we came with this idea of doing a flash cutover everyone was saying “this is crazy”. This is not really the typical way of bringing an infrastructure of such a size into service.
Even though we had a strong business drive, it was very important to get the technical community onboard and to convince them that it was worth taking the risk. The internal acceptance process of such a decision is, after vendor selection, the second most critical success factor.
Network operations, engineering and IT people are very different communities. They don’t mix and match spontaneously that well, and sometimes don’t even speak the same language. It is therefore very important to identify and empower the resources that show interest in both fields and provide them end to end responsibilities.
With regard to the transition process itself, having a strong and stable monitoring environment is also key. Disconnecting 50% of the network capacity in such a short time generates a massive amount of events in all network monitoring systems, meaning that you could lose control of the network itself.
And finally, this is what allows declaring the success of the transition. The cutover is a 10 hour process with 200 steps.
After that, you have a window of one hour to evaluate the situation and from that point, if it doesn’t work perfectly, you need to roll back and bring the old switch into service.
Your network monitoring environment needs to be able to recover very quickly, and you need to have all the tools that give you the visibility on the quality of service being provided. This is the project success indicator.

What would be the main piece of advice you would give to another CTO who would be considering a similar ‘big-bang’ approach?
We have all experienced, in any type of organisation or technology transformation, that there is always a good reason to postpone things. My main piece of advice is to go for an ambitious way forward, and pursue a vision that the organisation and its people will have pride in delivering. That certainly helps to overcome the levers of resistance that elements of the organisations tend to fall back on.
This also certainly helps to build a business case. Many business cases fail due to lack of ambition and therefore lack of means to realise the transformation. With an ambitious plan, you have the resources and can therefore limit the potentially risky impacts on the business.
The second aspect is to understand that any transformation is about people and their emotions. If you want to have an ambitious project, you certainly should not push down with your view and try to impose your way of moving forward. Show your team that you have an ambition for them to realise a very visionary project, but let them come up with the best way to serve the business and really give their best in order to support the challenge of the company.
If you have an ambitious project but your people don’t find the vision and the approach legitimate, then they will not support you.
The last point relates to the way this project was managed from the perspective of change. To ensure the huge shift, it is obvious change management methodologies should be applied.
In this particular project one very interesting exercise was to put people around the table and ask them to focus on one subject: “What if things go very bad?”
We also spent a lot of time doing simulations because this remained a complex process, and simulations made people more comfortable. Having a strong fallback scenario also allowed us to take more risks, and increases the commitment of the project team.
How has the success of the transformation been measured?
The success of any such transformation project is ultimately measured by the network KPIs, but also by the lack of “noise” or disruption to clients the day after the completion of the work.
Perhaps the most significant sign of the success of our project was the fact that we did not have to set up even a single emergency client meeting due to unexpected behaviour or outages. So from this perspective, we can say that the changeover was an unmitigated success.
We had no network crash and our customers appreciated not being affected. But at the same time we were very transparent with them, explaining the process, how ambitious it was, and all customers have appreciated it.
Some of them have even said they would love to be able to do the same. GTB 
See also
Belgacom upgrades wholesale network to 40 gigabits (GTB report)
Belgacom and MTS wholesale merger (GTB report) 
Industry prepares for IP and carrier ethernet services (GTB report)

Professor Jamie Anderson is adjunct professor of strategic management at TiasNimbas, the business school of Tilburg University and Eindhoven University of Technology