The IP SLA Investigation: How a Configuration Nobody Wanted to Question Was Costing Everyone

A BIAMIC case study in diagnostic leadership and holding a finding under pressure

Technology | Service Delivery | Contract Compliance | Vendor Management

The Contract Nobody Could Deliver

When a contractor is brought in to take over a struggling service delivery operation, the first instinct is usually to look at what is visibly broken. The complaints. The escalations. The open tickets.

The more important question is what is contractually committed and whether the organisation is anywhere near delivering it.

Reading the contract carefully, in this case, revealed something more uncomfortable than oversight. The organisation was meeting approximately 35% of its contractual obligations. Not because the people were incompetent or indifferent. But because the commitments made at the negotiating table had never been validated against what the delivery operation was actually capable of delivering. The gap was not discovered after the fact. It had been there since the day the contract was signed. The people who negotiated the agreement and the people who had to deliver it were never in the same room at the same time before the ink dried.

Most of the gaps responded to attention. Processes were adjusted. Reporting was built. Standards were applied. The compliance picture improved steadily.

One area would not move. And it would not move because the tools being used to measure it were fundamentally incapable of measuring what the contract required.

You cannot close a compliance gap you cannot measure. And you cannot measure it with tools that were not designed to see it.

The Decision Nobody Wanted Made

The organisation had a standard observability platform. It was the approved tool. It was what everyone used. It was what the internal teams knew and what the executive layer expected to see in every dashboard.

It could not produce the IP SLA data required under the contract. Not a configuration issue. Not a skills gap. A fundamental capability limitation.

The contractual requirement was specific: measurement and reporting of jitter, packet loss, and latency across QoS classifications for voice, video, signalling traffic, and three priority levels of data. The approved platform could not do this. The contract said it had to be done.

The decision to procure an alternative platform was not made lightly. It was made against significant internal resistance, under executive pressure to remain within the approved standard, and on a contractor basis, meaning the person making the call had no permanent authority to absorb the consequences if it went wrong.

Reluctant approval was eventually obtained. The new platform was implemented. The data began to flow.

There was a further dimension to the pressure being applied. As the IP SLA violations became visible and the service credit liability grew, voices within the organisation advocated for removing the offending clause from the contract altogether. The logic was commercially straightforward: no clause, no violation, no service credits. What it would have meant in practice was walking back a commitment to a client who was already experiencing the failure the clause was designed to protect against, and leaving the underlying configuration problem permanently unaddressed. The clause stayed. The problem was fixed instead.

The pressure to reverse the platform decision continued for years afterwards. That is the nature of decisions made against institutional preference. Even when they are right, they are not quickly forgiven.

It is worth noting that doing the right thing inside an organisation that would have preferred a different decision can carry a personal cost. That cost is real, and it should be understood before the decision is made, not discovered afterwards.

Doing the right thing inside an organisation that has not yet decided it wants the right thing done is rarely comfortable and never quick.

The Data That Should Not Have Been There

Once the observability platform was producing the IP SLA data the contract required, the reporting told a clear and troubling story.

QoS appeared to be functioning correctly for data traffic. But for voice, video, and signalling, the jitter, packet loss, and latency values were consistently and significantly outside the bounds agreed in the contract.

Service Level Agreement violations were being recorded. Service credits were accruing. The client was benefitting, financially, from a compliance failure that was costing the organisation real money on a continuing basis.

The connectivity vendor, whose infrastructure carried the traffic in question, was asked about the readings. Their position was unambiguous: the configuration was correct. The values being reported by the tool were the tool’s problem, not theirs.

The investigation that followed would last weeks. Every element of the data collection was examined. Every parameter was tested. Every configuration on the measurement side was validated.

The tool was doing exactly what it was configured to do. The data it was collecting was accurate. The values it was reporting reflected what was actually happening on the network.

When the tool has been validated and the data is accurate, the only remaining question is why the network is behaving the way the data says it is.

The Diagnostic Process: Weeks of Work, Three Minutes to Prove

Why the Remote Argument Was Going Nowhere

Technical disputes between parties with financial interests in the outcome have a predictable pattern. Each side presents data that supports its position. The other side questions the data. Nobody changes their mind. The argument continues until someone either gives up or escalates.

This investigation had reached that point. The connectivity vendor’s senior management had been through multiple heated discussions. Their position had not moved. The configuration was correct. The tool was wrong.

Continuing the remote argument was not going to resolve anything. A different approach was needed.

The Joint Session: Agreement Before Evidence

A meeting was arranged at a mutually accessible location with direct access to the network infrastructure and the observability tools. Representatives of all parties attended. The atmosphere reflected exactly what it was: a room full of people who each believed they were right and who had financial reasons to remain so.

The ego and tension in the room were palpable. The connectivity vendor’s regional MD opened with claims and accusations. The instinct of every party was to defend their position before examining any evidence.

The approach taken to manage that room was deliberate and specific.

Rather than presenting data or making assertions, the first question asked was this: can we agree, collectively, on exactly how the IP SLA elements should be configured on the routers?

This was not an accusation. It was a request for shared ground. Before anyone looked at anything, everyone in the room would agree on what correct looked like.

After some resistance, agreement was reached. The correct configuration was documented. Everyone in the room had endorsed it.

Establish what correct looks like before looking at anything. Once the standard is agreed, the evidence either meets it or it does not. There is no longer a debate about the standard itself.

Looking at the Evidence

A site in a nearby region that was showing the problematic data was selected. The observability data for that site was examined collectively. Every party in the room could see the values. Every party agreed the values significantly exceeded the agreed thresholds.

The router configuration for that site was then pulled up. The PE router. The CE router. Examined against the configuration that everyone in the room had just agreed was correct.

It was not configured correctly.

The process was repeated with additional sites. Different locations. The result was consistent in its outcome but not in its detail. The configuration was wrong at every site examined, but not wrong in the same way. There was no standard error. There was no standard at all. Each site reflected the interpretation of whichever engineer had configured it. The IP SLA implementation across the estate was ungoverned, inconsistent, and had apparently never been audited against any agreed baseline.

The network engineer present was asked to amend the configuration on one of the sites to match what had been agreed.

The next polling cycle ran within three minutes. The values for that site were now well within the agreed thresholds. The SLA violation for that site was gone.

The same amendment was applied to additional sites in the room. The same result followed each time.

Three minutes. The problem that had persisted for months, that had been argued about in meeting after meeting, that had cost significant money in service credits, was fixed in three minutes once the correct configuration was applied.

The Resolution and What It Cost

Agreement was reached in the room, though not warmly. The connectivity vendor committed to amending the configuration across all affected sites. They were not pleased. The admission that configurations had been applied inconsistently, without governance or standard, across an entire estate they were being paid to manage was an uncomfortable one. The commitment was made begrudgingly. The work took several days to complete across the full estate, in part because there was no consistent baseline to correct from. Each site required individual examination and remediation. The joint session had effectively produced the first agreed configuration standard for this function that the vendor had ever documented and committed to.

Once complete, the SLA violations stopped. The service credit accrual stopped. The client retained the credits that had accumulated during the period of non-compliance, which had provided a meaningful benefit to their operational budget.

The organisation stopped paying for a compliance failure that had been caused by incorrect configuration on infrastructure it was paying the connectivity vendor to manage correctly.

The tool vendor was vindicated. The platform that had been procured under pressure, against internal resistance, and at personal professional risk, had done exactly what it was supposed to do. It had measured accurately. It had reported truthfully. The decision to implement it had been correct.

The connectivity vendor’s regional MD left the meeting without speaking further. He resigned from his position not long afterwards. Whether the events of that meeting were a contributing factor is not known. What is known is that the data did not lie, the configuration was wrong, and the evidence for both was produced in front of witnesses in a room he had agreed to attend.

Sometimes the person who was wrong would rather leave than acknowledge it. That is their choice to make. The work still needed to be done.

What Was Actually at Stake

The IP SLA investigation was not, in isolation, a story about router configuration. It was one chapter in a larger effort to bring a service delivery operation into compliance with what had been contractually promised, commitments that had been signed without verifying they could be met.

At the point this work began, the organisation was meeting approximately 35% of its contractual obligations. A contract at that level of non-compliance does not survive. It terminates.

The work of the following years, addressing each gap systematically, building the reporting infrastructure, making the difficult decisions about tools and vendors and configurations, turned that picture around. The contract not only survived. Over the following seven years, its value more than doubled.

That outcome did not happen because of clever commercial strategy or relationship management. The contract grew in part because the client grew, and a larger client naturally requires more service. But growth requires trust, and trust requires a foundation. The work of bringing the operation into compliance, making the decisions necessary to meet what had been contractually committed, and holding those decisions under sustained pressure, built that foundation. Without it, the relationship would not have survived long enough for growth to become possible.

The most valuable thing an external advisor can bring is the willingness to do what is right for the engagement regardless of what is comfortable for the institution.

The Leadership Dimension

Reading what others had not read

The compliance gap existed because nobody had mapped the contractual commitments against the actual delivery capability before signing. That mapping was not a technical task. It was a leadership one. It required someone to treat the contract as a document that meant something, rather than a formality that had been filed and forgotten.

Making a decision with no safety net

Choosing a non-standard platform on a contract basis, against executive preference, with the knowledge that the decision would be questioned for years, required a specific kind of conviction. Not the conviction that you will be proved right immediately, but the conviction that the right answer is worth the discomfort of finding it.

Holding the line on the contract

When the pressure came to remove the clause rather than fix the problem, the easier path was clear. Remove the obligation, eliminate the violation, stop the service credit liability. The harder path was to hold the contractual commitment and fix the underlying cause. The harder path was the right one.

Managing a room that wanted to fight

The joint session worked because the approach removed the fight from the room before the evidence was presented. Getting all parties to agree on what correct looked like, before anyone looked at anything, changed the nature of the conversation from adversarial to evidentiary. The data did not need to be argued. It needed to be read.

Holding the finding through months of resistance

The connectivity vendor’s position did not waver until the evidence was in front of them in a room they could not leave. Maintaining the investigation, continuing to validate the tool, refusing to accept the vendor’s assertion without proof, required sustained conviction over an extended period.

Knowing when the argument needs a room

The remote dispute was never going to resolve. Some problems can only be closed in person, with shared access to evidence, in conditions where deflection is no longer available. Recognising that moment and creating the conditions for it to happen is itself a leadership decision.

What This Investigation Teaches

CONFIGURATION IS NOT THE SAME AS COMPLIANCE

A vendor can believe their configuration is correct and still be wrong. Belief is not evidence. The investigation required moving from assertion to demonstration, and the demonstration required shared access to the same evidence in the same room at the same time.

NO STANDARD IS WORSE THAN A WRONG STANDARD

A systematic error applied consistently can be identified and corrected. An absence of any standard, where every engineer applies their own interpretation, is far harder to find and far harder to fix. Governance of configuration is not optional. It is the baseline from which everything else is measured.

THE TOOL THAT SHOWS THE PROBLEM IS NOT THE PROBLEM

The approved platform could not produce the required data. The alternative platform could, and did, and was accurate throughout. Resistance to the tool that reveals an uncomfortable truth is a common pattern. It is always worth examining whether the resistance is about the tool’s accuracy or about what the tool is showing.

AGREE ON WHAT CORRECT LOOKS LIKE BEFORE LOOKING AT THE EVIDENCE

The methodology that unlocked the joint session was simple and repeatable. Define the standard first. Then measure against it. Once the standard is agreed, the evidence either meets it or it does not. The vendor cannot argue about the standard they have just endorsed.

COMPLIANCE GAPS COMPOUND QUIETLY

The configuration errors had likely existed since deployment. The service credits had been accruing for months. The cost was real and ongoing. Problems that are not measured do not get fixed. They get absorbed into operational reality until something forces them into view.

THE EXTERNAL ADVISOR’S VALUE IS THE WILLINGNESS TO ACT

An internal team carries institutional relationships, career dependencies, and historical context that make certain decisions politically difficult. An external advisor who is genuinely committed to fixing what is broken, regardless of who it implicates, can go where internal teams cannot. That willingness is the service. The technical capability is the means.

About BIAMIC

BIAMIC is a consulting and leadership advisory practice working with executives and senior leaders on complex organisational and technology problems. Our approach is diagnostic first, solution second. We follow the data, and we hold the finding.

biamic.com | ronnie@biamic.com | +27 82 825 8601