So you're telling me there's a chance...
It's awkward to say this but, in a normal distribution, half of the population is below average. This is a mathematical fact; but, socially, it's uncomfortable. There's even a cognitive bias—the Lake Wobegon effect—that protects people from confronting this fact; we like to pretend that no one is below average.[1]
A way this shows up in federal contracting is that, to select winners in a procurement, the government will often assign adjectival ratings to companies' proposals. Usually, at the top of the adjectival rating scales, the government will assign superlatives like "Excellent" or "Outstanding." At the bottom of the scale, the government will have use adjectives like "Poor" or "Unacceptable." In the middle, where most proposals will lie, the government will have a baseline, like "Acceptable."
In part because of the Lake Wobegon effect, however, most proposals will have at least an acceptable rating, and a smaller percentage will enjoy top ratings. Basically, no proposal will be unacceptable. No, in govcon, all proposals are above average.[2]
Typically, this isn't that big of a deal. People understand that grade inflation is a Thing and that companies will therefore strive to get top marks. Cs may get degrees, but they generally don't get you onto prized IDIQs.
Sometimes, though, there are situations where the Lake Wobegon effect can affect an acquisition. And the US Court of Appeals for the Federal Circuit's recent opinion in Rev, LLC v. United States is one such case.
Rev involved the VA's onramp of 9 vendors onto the Transformation Twenty-One Total Technology-Next Generation (“T4NG”) contract.[3] According to the solicitation, the VA planned a two-phase evaluation process, and would select vendors based on multiple factors, with a "Technical Factor" being "significantly more important" than the other factors. The technical factor itself would be evaluated on multiple subfactors.
For those factors and subfactors, the VA used adjectival ratings, working from Outstanding, Good, Acceptable, Susceptible to Being Made Acceptable,[4] and Unacceptable. Crucially, during the second phase of the competition, the solicitation indicated that the government planned to do two things: (1) eliminate vendors with technical ratings below "acceptable"; and (2) make awards to 7 vendors. Of course, the solicitation reserved the right to adjust if it was "in the best interest of the Government."
After the VA eliminated most of the offerors in phase one,[5] 33 offerors made it to phase two. The VA then evaluated those offerors' proposals. One vendor received an "outstanding" rating, 8 vendors received a "good" rating, and the other 24 received an "acceptable" rating. Because of the Lake Wobegon effect, none of the remaining vendors were below acceptable (though one offeror did have one subfactor with an unacceptable rating).
Given that 9 vendors were above the acceptable rating and every other vendor was merely acceptable, the government ended up deviating from its plan by (1) drawing the cut line at "good" (not acceptable); and (2) making awards to 9 vendors (not 7 vendors).
On the surface, this logic makes imminent sense. If 73% of the vendors are an undifferentiated mass of "acceptable," putting the cut line at "good" feels more defensible than trying to draw a distinction between 2 of the 8 good vendors.
Rev, however, saw it differently. Rev was one of the 24 vendors with an acceptable technical rating and, arguably, Rev was next in line for an award because of its ratings on the technical subfactors. Rev protested, and argued that 6 of the 9 winners should have been eliminated from competition for various reasons. With only 3 winners left, the argument went, Rev would have been next in line for an award.
The Court of Federal Claims ruled on Rev's protest and concluded that Rev lacked standing to challenge the 9 awards because Rev was not "prejudiced by a significant error in the procurement process." To meet that legal standard of prejudice, Rev needed to demonstrate that "there was a 'substantial chance' it would have received the contract award but for the alleged error in the procurement process."
And the Court of Federal Claims explained that Rev did not show prejudice because "there was no guaranteed number of awards, and the exclusion of one offeror from the competitive range did not necessarily ensure the inclusion of another." In other words, even if 6 awardees ahead of Rev were knocked out, it didn't guarantee Rev a spot on the contract.
On appeal, however, the Federal Circuit held otherwise and reversed:
REV has shown a substantial chance that if the number of offerors who were rated “Good” or “Outstanding” was only three – as it would have been if the six bidders REV attacks actually suffered from the defects REV alleges – then the VA might very well have lowered the minimum rating for being added to the pool to “Acceptable.” In that scenario, REV would have been in the top seven and would, hence, have been in the competitive range at Step 2. This logic leads, again, to the conclusion that REV has shown a not insubstantial chance of being added to the T4NG pool.
In other words, even though Rev was only acceptable, if only three vendors were above acceptable, the VA might have kept its standards lower and awarded Rev a spot on the contract.
To the Federal Circuit, the emphasis was less on whether Rev's chances were substantial, and more on whether Rev's chances were not insubstantial. Classic double negative! Read broadly, however, the legal standard in Rev is basically a version of the classic scene from Dumb & Dumber:
Lloyd: "Hit me with it. Just give it to me straight. I came a long way just to see you, Mary. The least you can do is level with me. What are my chances?"
Mary: [Pause] "Not good."
Lloyd: "You mean 'not good' like… one out of a hundred?"
Mary: "I'd say... more like one out of a million."
Lloyd: (slowly reacts) "So you're telling me there's a chance? … YEAH!!!"
The court felt Rev had a greater than insubstantial chance, and the court let the case move forward.
Cool cool. I mean, I can see where Lloyd and the Federal Circuit are coming from. If someone told me that my chances are "not good," I would definitely assume that my chances are less than 50%. But I also would assume that "not good" means my chances are more than 1%. If, however, "not good" is anything in the range of 1% and a 0.00001% chance, a "greater than insubstantial chance" is also somewhere within that range.
Simply put, Mary should have used harsher words. A "not good chance" apparently isn't descriptive enough. Similarly, if the government says that a vendor is "acceptable" when the government really means to say that a vendor is a worst-case scenario, vendors are going to hang around in hopes for that worst case.
Post-Rev, the incentives are pretty tricky for the government. In general, you can imagine that the Rev decision will open up more opportunity for protests on multiple-award contracts to dispute the competitive range. So, what will agencies do to avoid protests?
One strategy might be for agencies to keep their standards lower and default to allowing more vendors on to contracts. The VA took that approach when it picked 9 vendors instead of 7, but maybe a post-Rev VA will be even less selective? If you lower what you consider average, more companies can be above average!
Another strategy, though, could be to address the Lake Wobegon effect head on. Instead of having a 5-point scale where everyone gets at least a 3, the government could draw more distinctions among vendors and cut more deeply? If the goal is to separate the wheat from the chaff, maybe the lesson is that the government should always expect more chaff?[6]
Time will tell how the government reacts to Rev and whether agencies do anything to adjust their approaches to multiple-award contracts. But if the government starts telling more vendors that they don't have a chance instead of leading them on, industry might just have Rev to blame.
[1] The Lake Wobegon effect is named after Garrison Keillor's fictional town of Lake Wobegon and the town's catch phrase that concludes "... and all the children are above average."
[2] This has been my experience. And yet, it's entirely possible that my core thesis here is totally wrong! If you have been in situations where the government grades harshly, email me. If need be, put me under NDA and tell me how it went! Seriously!
[3] A thing that's kind of wild? Last year, the VA made initial awards on the second iteration of T4NG (natch called T4NG2). Which means that by the time the Federal Circuit ruled on the standing question of the first iteration of T4NG, the government had already moved on to the next-gen next-gen vehicle. It's possible, though, that T4NG2 protests will be going on until T4NG3? I have nothing else to add here other than a sense of amazement.
[4] Y'all. I love this rating so much. Instead of just having the binary of "Acceptable" and "Unacceptable," the government created a euphemistic middle category?! Susceptible to Being Made Acceptable?! In what other context would this work? "Hey, do you think this tie goes with this shirt?" "Well, Dave, it's susceptible to being made acceptable!" I mean, if an adjective is stated in the passive voice, you know you're gonna have problems... Just such a sick burn.
[5] Technically, the opinion doesn't say how many offerors there were in phase one and I am making an assumption that most offerors were cut in phase one. Based on a chart in the opinion, though, it looks like there were at least 96 offerors. So I feel comfortable with my assumption.
[6] There's actually a third strategy, which is for the government to think in terms of intervals instead of in absolute numbers. Instead of saying "we are planning to pick 7 vendors," the VA could have said "we are planning to pick between 4-8 vendors." That might seem like a silly distinction, but my hunch is that the VA would have had an easier legal argument. See, reading the footnotes can be helpful!