A true-up thrill ride

If you hang around government software procurement spaces long enough, you'll start to hear horror stories about "true-ups."

Here's a rough sketch of how they go:

1. The government buys some software and the seller sets a unit price based on how much the government expects to use the software. This can be things like "per-user" or "per-device" or something like that.
2. There will be an attempt to track the amount of usage.[1]
3. At some point before the end of the contract period, there will be an audit of the usage. The audit can be formal or informal, depending on the stakes. And it can be internal or external.
4. Inevitably, one of two things happens:
   1. Dramatic under-utilization, in which case the government is mad because it wasted money and vows to never do that again. There will be a proliferation of spreadsheets and chase lists, but not much else.
   2. Rampant over-usage of the software, in which case the company is outraged and demands that the government cover the cost of all of the licenses that were used (the "true-up").
5. Internally, after the lawyers say "yep, you screwed up" and the budget analysts' blood pressure comes back down, the government will pay the extra cost.
6. The government will then create various policy documents aiming to implement some process to make sure that this never happens again.
7. It happens again.

Now that you have the idea of how it works, here is an absolute banger of a true-up horror story from the Court of Federal Claims — 4DD Holdings LLC v. United States — that came out last week.

Content warning: the following contains quotes from the court that may cause federal employees some big emotions.

Let's set the stage. 4DD is a company you've likely never heard of. And it sold a product — Tetra Healthcare Federator — you've definitely never used. Don't feel bad that you never used the software or heard of the company. Even though Tetra Healthcare Federator was described as "commercial" software, no one had ever used it before the DOD purchased licenses in the early 2010s. In fact, according to the court, "4DD has never sold its Tetra Healthcare product to any entity other than the government." The product has had "only one customer and effectively one sale."

Because it had never been sold before, there was no established commercial price for Tetra Healthcare Federator.[2] Still, even though Tetra didn't have a commercial price, the government had a budget: "roughly $1 million—the amount of other-direct-cost dollars remaining on [the system integrator's] contract." Accordingly, 4DD and the government backed into a licensing arrangement where Tetra Healthcare Federator was licensed at $10,000 per core and $3,000 per seat.

Now, in step 2 of the rough sketch above, I outlined that folks will attempt to track the amount of usage of software. And, well, here's what the court had to say about how that played out:

Licensing agreements often require a method for monitoring license usage, and software companies like 4DD normally design their software to alert them when a copy of their software is activated. That feature could not be used here, however, because it presented security risks to government networks. As a result, the responsibility to track license usage fell on the government, and 4DD had to “rely on [its] honesty.”
***
The license tracking portal created by 4DD had limited enforcement value. Although it recorded Tetra installations and many other details, it suffered from serious weaknesses. First, the portal required the government to voluntarily supply information, and so dishonesty or negligence would nullify its effectiveness. Second, the portal only tracked Tetra downloads directly from 4DD; it did not contemplate other Tetra copies created by the government after the government downloaded and installed the software. Perhaps the portal’s biggest flaw, however, was that [the Contracting Officer Representative] never looked at it. Instead, her license tracking method essentially boiled down to one question for [the chief engineer]: “Do you need more licenses?” She never asked [him] how many licenses were installed because she “just stupidly assumed he was under [the limit].”

Ouch. If you have a tracking system based entirely on trusting the government to monitor its own usage and updating data in a portal, you might have a problem.

Which brings us to steps 3 (the audit) and 4 (the overuse):

Using the license tracking portal—which, again, only contained the government’s self-reported instances—4DD eventually determined that the government exceeded its license by at least 68 computer cores. It did not immediately alert the government, however, because it claimed that it wanted to support the project.... After 4DD notified [the COR] of the problem, she contacted... the contracting officer for the Tetra licensing agreement. [The CO] directed [the COR] to initiate a “true-up” negotiation that would locate and pay for all the Tetra copies.

During the true-up negotiations, [the COR] independently worked with [4DD and the system integrator] “to make sure that the government had data from every place that Tetra could be loaded..." That process mainly entailed a convoluted exchange of spreadsheets in which the parties quarreled about how many Tetra copies existed.

Step 5. Eventually, the parties agreed that the government exceeded the license by 168 cores, and they worked to negotiate a price. They ultimately agreed to the same price per core: $10,000. Which meant that 4DD received about another $1.7 million due to the excess usage! No small sum for a true up. A good day for 4DD.

A while later, the government scrapped the use of Tetra in favor of a larger contract to Leidos. Presumably step 6 happened, but who knows for sure. And that might have been the end of the story.

Except.

Brace yourself.

Unfortunately, the government’s left hand did not know what its right hand was doing. While [the COR] worked to find Tetra copies, [the Chief Engineer]—without telling anyone—simultaneously ordered that the copies ... be deleted... [T]hese deleted copies were never acknowledged during the true-up.

Next, after several months of exchanging spreadsheets, [the COR and chief engineer] “verified” that the government had installed Tetra on 64 computer cores in the DTC. That was not true, however, as [he] had never looked for Tetra copies in the DTC. Instead, [the COR] invented that number as a “placeholder” and conceded at trial that it was not “based on any data from the DTC or SMS." Neither of them shared this knowledge with 4DD during the true up.

Look, this is not legal advice or contracting advice. But while "the government’s left hand did not know what its right hand was doing" feels like a Tuesday in the federal government, destroying evidence and making up random numbers and not saying anything about it is pretty damning stuff.

And, as it turns out, due to a boring quirk about how computers work, the government didn't just use 168 extra cores. No, in fact, the government ended up using 290,334 cores! That's not a typo. At $10,000 per core, that would amount to almost $3 billion in extra cost! Hahaha.

Surely, you might thinking, that's ridiculous. No one would actually ask for $3 billion for software that wasn't used by anyone. Well, 4DD swung for the fences:

In 4DD’s view, the Copyright Act entitles it to anywhere from $3 to $5 billion as compensation for the government’s infringement. It arrives at this range by adopting Mr. Myers’s computer core count of 290,334 and pricing them as high as $17,000 per core (its volume discounted SEWP price).

Although the court found that 4DD was entitled to compensation for the unlicensed usage, it the court dunked on that pricing approach:

An “established royalty rate” does not spring into existence any time
ink hits the page in a licensing agreement, however. Among other things,
it requires “general acquiescence” by a significant “number of persons.” Here, 4DD has never sold its Tetra Healthcare product to any entity other than the government, and so with only one customer and effectively one sale, it can hardly claim that it has an established royalty rate that entitles it to $5 billion.

In the end, the court ended up engaging in some "hypothetical negotiation" and found that the government owed 4DD more than $11 million for its use of Tetra Healthcare Federator.

All said and done, 4DD is going to walk away with almost $15 million for software that never saw the light of day.

Not bad work, if you can get it.

Glad it won't happen again. It won't, right?

---

[1] I put this in the passive voice because sometimes it's the government's job to track licenses and sometimes it's the vendor's job.

[2] In one of the mysteries of government contracting, despite having no other customers, Tetra Healthcare Federator was listed on NASA SEWP with a price list.