Hello GAO? About that F-35 ‘Report’

Next time maybe you could bring some experts in something other than ‘bean counting’.

It appears that the GAO attempted to structure their latest F-35 ‘Report’ “F-35 SUSTAINMENT: Need for Affordable Strategy, Greater Attention to Risks, and Improved Cost Estimates”, to deliver a certain message, a certain way, and with a certain flourish. But with only the most cursory professional eyeballing, gaps in the report’s observations, argumentation, and conclusions quickly appear. Viewed more closely and put into perspective with past F-35 reports, the gaps become gaping chasms and the document devolves to merely another example of the GAO ‘defense’ report archetype: a rather subjective, deeply flawed and--of course--superficial GAO product.

As with most GAO reports of this genre, this particular GAO product, ‘Incompetence’ comes to mind more than ‘malevolence’. Ultimately we cannot fault the GAO for attempting to audit and write about that which they know little, they do what they are told to do. But I do hold them responsible for their total pretension of authority and the Hybris they display in asserting their ignorance as authoritative.

We place a spotlight on the GAO’s most damning sins committed within this report. It will become readily apparent why this GAO product gained very little traction in the popular media once it was actually released: It confirms the beliefs of those who want the reports ‘findings’ to be true so they don’t really bother to critically review it, and the rest who care about the issue enough to take the time to really read and understand the content will just dismiss it. The majority of people simply remain disinterested. I happily observe that to date, with sporadic interest and promotion even by the lowest forms of media habitués (example: the Puffington Ho’s of this world), the report has not gathered much audience or furor.

There’s quite a bit the GAO missed in that ‘Comparison’ (P.2 and P. 12)

The following graphic appears twice in the report. Once in the front matter and once within the body proper. I’d say that makes the subject of the relative O&S costs of legacy aircraft to the F-35 one of their major ‘points’. I further assert they are completely bonkers if they think their reasoning and evidence supports, in any manner, the point they attempted to make—the point they expect others to believe.

GAO 2014 Comparison of Apples and Oranges

To paraphrase myself elsewhere:

Q1: Notice any aircraft type and associated costs missing from the list of aircraft being ‘replaced’?

Hint for those who need help: It starts with an "A" and ends with "-10".

So even IF this were an ‘apples-to-apples’ comparison, the GAO did not count the right apples, and it looks like there are not even the right number of ‘legacy’ apples to be counted within the equation. I should not have to point out to anyone, that even if the GAO substituted the same number of aircraft, since they were not the same type and number of aircraft with the same mission lists in the same relative proportions, that the F-35 is replacing, any such ‘adjustment’ only introduces error.

Q2: Notice the different cost number sources for the F-35 compared to the 'others'?

The legacy aircraft listed have had a theoretical inflation added to their 2010 O&S costs and applied for the years shown (2036-2040—when peak numbers of F-35s will be operating) in the comparison. 

BUT on the other side of the comparison, the F-35 cost estimates are known--from previous reports and other publications (see page ‘XX’ footnote here for an example)--to include not only inflation, but to also include estimated Cost Growth ABOVE Inflation (CGAI). CGAI is another ‘guess’ that gets compounded (not just added) on top of the ‘guesstimated’ inflation, and can be expected to be part of F-35 program cost accounting until at least the baseline Block 3F configuration aircraft designs are finalized and/or fielded.

Given the crudeness of the GAO comparison, I seriously doubt the GAO was sufficiently thorough to also include CGAI in the legacy aircraft cost estimate. But even IF they did, they are still not comparing ‘apples-to-apples’. I’m not even the first one to observe this discrepancy BTW, for within the first leaked news of this document we find:

“A source close to the program pointed to this comparison as one example of how GAO was “comparing apples and oranges.” 

Quite true. TOO true.

More Content: More Missing Oranges.

This is not even real news. But it has been observed that whenever these sort of lame comparisons are made, those making the cost comparisons are particularly thorough in ignoring the fact that whatever F-35 costs are, they include the cost of more ‘content’ than legacy aircraft costs. For example, in testimony before the Senate just this year (pg 5) , it came out that for F-18 legacy aircraft, the Department of the Navy will need to keep buying some of the capabilities that are similar (but still less advanced) to the capabilities the F-35 already has just to keep F-18A-Ds relevant into the near future: 

In order to maintain warfighting relevancy in a changing threat environment, we will continue to procure and install advanced systems such as Joint Helmet-Mounted Cueing Systems (JHMCS), High Order Language (HOL) Mission Computers, ALR-67v3, ALQ-214v5, Multi-Function Information Distribution System (MIDS), APG-73 radar enhancements, Advanced Targeting FLIR (ATFLIR) upgrades, and LITENING for the Marine Corps on selected F/A-18A-D aircraft.

We observe now that for the F-35, there are several counterparts of the items listed in the testimony above, such as the Helmet Mounted Display System , as well as all the necessary sensors and targeting systems that are accounted for not only in the F-35 production costs, but also in the costs for the upkeep of same--captured as part of the F-35’s sustainment costs. For legacy aircraft, these sustainment costs for all the additional systems they need to operate, along with the external fuel tanks I might add, are relatively ‘hidden’: they are  accounted for on accounting ledgers that are separate from the host aircraft’s. These ‘hidden’ costs are not trivial.

Finally, the entire F-35 Autonomic Logistics Information System is in the F-35’s cost estimate as well. This system is a streamlined logistics support network, global in scope. Imagine the network support costs alone! NONE of the legacy systems’ support infrastructure is included in their O&S costs.

We could go further and expand the scope of comparison to total force employment costs. We could discuss how the F-35 is expected to require fewer mission support assets (transports, jamming aircraft, aerial refueling aircraft, etc) or talk about how the value of the F-35’s ‘stealth’ features certainly far outweigh the additional system support costs not found with non-low observable aircraft. But why bother? The cost comparison the GAO attempted fails within its own set boundaries.

GAO Shows Us Some Goodness…Was this some kind of a slip-up? (PP. 17-18)

Excerpt:

• Mean Flight Hours between Failures (Design Controllable) is the average amount of flight hours achieved before a design-controllable failure occurs. As of March 2014, this metric was progressing in that the number of flight hours before a failure occurs was increasing for all three variants. For example, the average flight hours between failures for the F-35A—the variant with the most flight hours to date—was 5.2 in March 2014, surpassing the expectation at its current flight hours by about 1.2 and growing toward its requirement at maturity of 6.0. Moreover, this was an increase of about 1.8 average flight hours between failures since September 2013, as reported by GAO.

I find how MFHBF(DC) was framed this time around ‘interesting’. This data is absolute “goodness”, but it is delivered so deadpan you’d hardly know it without reading it twice. The graphic the GAO used two years ago (Figure 9, pg 30) to then illustrate the F-35 being ‘behind’ in MFHBF (the ‘DC’ was left off in 2012) looked like this:

2012 GAO Report Chart of MFHBF (Really MFHBF(DC))

In the 2012 report, the GAO was all over this metric for all the variants as ‘lagging’. I noted at the time that it was a premature analysis.

This year, the positive performance for the same metric just gets a ‘paragraph’. Here’s the same MFHBF(DC) info from this report in a graph similar to back in 2012 when the GAO was more interested in selling a negative story. 

2014 GAO MFHB(DC) Data in Chart Format

Even though the metric indicates good reliability performance (ahead of where it is supposed to be at the point in time it was measured), it shouldn’t be taken any more seriously than in the past without more perspective and data behind it including knowledge of the long term trend. But guess what?

MFHBF(DC) 2012 & 2014 Data Merged 

Putting the 2012 and 2014 report data we get enough data to call it ‘better’ and to note the trend is—encouragingly--positive, so we can call this unquestionably good news. Keep the trend line (it is calculated and not simply drawn by the way) in mind for later use--when the GAO whips out a fallacious Appeal to Authority on us.

But we observe again that it is still ‘early’ in the grading period. The trend could reverse if ‘something’ pops up, though the longer the F-35 goes with a positive trend, the less likely something will pop up until it starts reaching its end of life at the other end of the ‘bathtub’ curve. We must remember that in June 2014, the fleet was only .085 (8.5%) complete towards maturation to be measured/graded at 200,000 flight hours. The data stops in March in this report, so it is based upon probably no more than 7% of flying hours towards maturity. 

Still, it is interesting -- mostly as an illustration to contrast how the GAO delivers information that supports their message and how they deliver that which doesn’t support the message.

In passing, I also add that in light of the positive news above, we should remember the manner in which the report was ‘leaked’ to gather negative headlines before the release, to support the mystery leaker’s purposes. That too is interesting.

Now try and remember what we’ve just covered and use it to judge the importance of what the GAO has to say about the next two metrics. It is particularly important to remember in light of what the GAO says about MFHBCF, which for the most part can be considered a SUBSET of the MFHBF and MFHBF(DC). I say ‘for the most part’ because MFHBCF can contain the results of induced (think self-inflicted) failures that are not "design controlled".

Enjoyed The Goodness?… It’s followed by drivel (PP. 17-18)

Excerpt:

•Mean Flight Hours between Critical Failures is the average amount of flight hours achieved before a failure occurs that results in the loss of a capability to perform a mission-essential function. As of March 2014, this metric was lagging well below its requirements at maturity, meeting an average of 42 percent of those requirements across all three variants.

“…this metric was lagging well below its requirements at maturity”? That is an interesting turn of the phrase there: as if it should be significant that it is not near full expected value, with only about 93% of the fleet flying hours missing and need to be flown before we are to actually ‘grade’ the performance. I view this as a logical parallel to berating a 10-year old for having not yet finished college.

So where is the MTBCF relative to where it was planned to be AT THIS EARLY POINT on the road to maturity? This info is curiously absent, given how past GAO reports made a big deal of being ‘below the curve’ when the data was even less meaningful. I strongly suspect it is no longer below the ‘should be’ curve because, among other reasons, the GAO conveniently leaves this information out.  In addition, there is absolutely ZERO significance to throwing out an ‘average’ MTBCF value that runs across all variants—each at different stages of maturity in their development. What are the values for each variant? Why did the GAO not tell us? Was the GAO trying to make a point without revealing something about a sensitive or classified program spec? I doubt it.

The relevant information we need, and yet the GAO does not provide (if they even know) is:

1. Where is the value compared to where it was predicted to be at this time? 
2. What is the trend?
3. Is the reason the value is ‘what it is’ understood?
4. Is there an adverse impact that needs to be eliminated/mitigated to meet the spec at ‘maturity’? 

The GAO has actually been boneheaded enough in the past to also raise a question in my mind as to even the significance of how they use the term MFHBCF itself. I question it, because as I noted above, MFHBCF is often a catchall that includes induced failures. It can also contain the effects of misdiagnosis of non-existent faults using not-yet-mature tech data: also known as normal ‘growing pains’ experienced in learning to operate and maintain the system.

What percentage of MFHBCF hits are the result of Airman Doofus breaking something or Lance Corporal Slacker not following repair direction correctly, or of either misinterpreting the troubleshooting directions? How about the aircrew botching his evolving, maturing checklists? A better measures to judge the reliability progress would be MFHBF(DC) as we saw above. As an aside, I suspect what the GAO means by MFHBCF is really “Mean Flying Hours Between Operational Mission Failure” (MFHBOMF). 

And AGAIN, we note (can’t repeat it enough) the F-35 fleet had accrued only about 7% of the total flying hours needed (where the reliability targets are supposed to be met) at the time the report’s data was collated, yet the GAO still has the temerity to assert that it was ‘only’ operating at an ‘average’ of 42% of the ‘maturity’ spec is (somehow) 'significant'? 

If the F-35s were really ‘breaking bad’ we’d hear about it from the pilots screaming about ‘availability’ and the maintainers about too many maintenance man-hours per flying hour (MMH/FH). We hear nothing about these performance metrics from the GAO. Surprise.

The GAO then brings up MTTR:

Mean Time to Repair is the average time it takes a maintainer to repair a failed component or device. Currently, this metric is not improving in that as flight hours increase, it is taking maintainers longer to repair failed components for the F-35A and F-35C, and the amount of time it takes to repair failed components for the F-35B remains unchanged. Specifically, GAO reviewed R+M growth curves provided by DOD showing the historical growth of this metric from 2009 for the F-35B and 2010 for the F-35A and F-35C to March 2014, and we observed that the metric is trending in the opposite direction of its predicted path for the F-35A and F-35C, and the metric is remaining steady, without improvement, for the F-35B.

I have suspected for some time higher MTTR was due to Outer Mold Line (OML) restoration cure times. If so, I would also suspect a good portion of that is driven by For Other Maintenance (FOM) actions. I still suspect it is a factor, but perhaps not as significant as I previously believed. What is nagging me is the missing commentary on Maintenance Man Hour Per Flight Hour (MMH/FH) numbers.

OML restoration would probably have a significant (but lesser) impact on the MMH/FH metric however. But with the GAO the norm is ‘no news is good news’. I therefore see missing complaints about MMH/FH as indicative of relatively lower OML restoration activity than I previously believed. The longer things go without us hearing about MMH/FH, the less I think OML restoration has a major impact. This has caused me to turn my thinking to new directions as to why the F-35 is experiencing higher than expected MTTR.  It now seems there are a couple of really interesting unknowns that could better explain the higher than desired MTTRs.

First there is the distribution of actual task repair times. IF the F-35 designers did a good job driving out all the typical ticky-tacky maintenance actions of the past, there would be fewer short duration maintenance events than seen on legacy systems, this would skew the average (Mean) repair time to the high side very quickly, not because more tasks were taking longer, but because longer tasks are harder to drive out of the equation. The MFHBF(DC) trend and the lack of details concerning MFHBCF would support this possibility. 

Second, I would be very interested in the internals of the data to see if there were discriminants by operating location and LRIP lot buys for each variant. I would expect Edwards, Eglin and Pax River to have more MTTR hours because they have more early jets ( how much planned mod activity is impacting this metric?) and/or are more aggressively trying to break the plane. But inversely, I would expect Yuma and Luke to have better numbers with the newer jets and more ‘operational-like’ environments than Eglin, Edwards or Pax River. Further, I would expect the Operational Test F-35s at any location to have even better numbers due to their flying of later LRIP jets and flying programs with a greater emphasis on more ‘operational’ maintenance paradigms.

There are also any number of reasons that the high MTTR could be deceptive or even unimportant. For all we know, it could even have nothing to do with the jet: some of this might be merely an issue with the maintenance process closeout and ALIS development/maturation (yes, I’ve done maintenance data collection: NO system is perfect). There is also the question of what percentage of this metric is also due to maintainers working on the plane when there is ‘No Fault Found’ or work caused by ‘Induced’ actions? We need the GAO to tease out the relevant bits if only to improve their own sorry analyses. But if they are going to highlight something as a problem in a public report, they need to let the public know the ‘whys’ behind the MTTR and MMH/FH data to justify their claims. That is, unless the purpose of the report is to just dupe and spin up the rubes. 

In passing, I note once again, knowledge of crew sizes would also be helpful. I suspect they too are meeting the F-35’s needs. 

If there were valid reasons to be concerned about the MTTR at this time, it would show up in the other data the GAO isn’t showing. What data the GAO does give us sufficient information on, doesn’t support the idea anyone should be concerned: the missing data needed to further evaluate the seriousness of the high MTTR hints at the MTTR as not being a serious problem at all. When the GAO is involved, I’ve consistently found that ‘absence of evidence IS indicative of ‘evidence of absence’. 

As it stands, even IF the F-35 does takes longer to fix than spec (and it very well could be even the spec was overly ambitious) then it only becomes important if it is failing too often (availability issue), or taking more manpower to fix (cost issue). Without knowing the relationship, which lets us determine actual availability and cost impacts, our knowing the MFHBCF and MTTR is pretty meaningless. It is however, something for the GAO to air out just so others can b*tch about it.

Does the GAO even know what ‘troubleshooting’ or ‘learning curve’ mean? (Pgs. 18-19)

For that matter, do they understand how ‘engineering’ works? As a long-time ‘tester’ this section cracks me up:

To identify some software issues as they arise, users in the field use an internal system to submit requests to the contractor, but these requests are submitted on an individual basis and may not always be addressed immediately as it takes time to determine whether the issue is related to hardware or software. For example, officials told us that the Electrical Optical Targeting System, which is used to track a target, continues to fail. In this instance, testers reported the problem, and officials attempted to improve the capability with hardware changes. However, not all issues with the Electrical Optical Targeting System were fixed with the hardware changes, and officials have decided to also try to address the issue with software changes, causing users to identify workarounds in the meantime. 

This is either illustrative of the GAO's general unfamiliarity with how troubleshooting and engineering is done OR how the GAO is oriented to bring up issues without any consideration or understanding as to the relevance of the problems they highlight. The GAO reports this bit as if the F-35 program was taking some kind of a 'hobby-shop' approach to solving the issue mentioned. If experience is any guide (and it usually is) the F-35 engineers probably 'racked and stacked' options to remedy the problem as quickly and cost-effectively as possible. In analyzing the options, it was probably determined that the hardware changes would be the most beneficial and easiest to employ, knowing all along that there was either a possibility OR certainty they would need to implement the software changes afterwards. Stating "officials have decided to also try to address the issue with software changes" makes employing standard engineering management tools sound like ad hoc (“try”) guesswork, and also tells me the GAO really has no idea how engineering works. Does the GAO understand that maturing fault isolation software involves learning what the faults look like and how they manifest in an operational system? Does the GAO realize that until the system is fielded, the knowledge will always be imperfect beforehand?
The GAO continued:

As another example, officials discussed instances in which the diagnostics system signals to a maintainer that the landing gear failed, but it was actually a sensor near the landing gear that failed. Because software for isolating these types of failures is not yet mature, operators and maintainers on the ground may continue to check the landing gear without discovering the sensor issue and reporting it. Finally, officials stated that with the release of the next increment of software in 2015, a number of new issues may arise, and these issues may be related to software because the new software includes new processors that can affect mission systems on the aircraft.

What the GAO is describing in the second half of the section concerning the ‘landing gear’ anecdote is what is usually thought of as a 'nuisance' software problem. It is standard (and sound) program management to triage your design (hardware or software) development issues so that the most important problems (those impacting flight safety are highest priority) are dealt with first. Without knowing how well the F-35 program as a whole is managing all the issues, singling out something like this is pretty pointless: it has no programmatic utility. It is, however a good scare story.

The GAO ‘Need More Control’ Over Tech Data Rant (Pgs 22-23)

It would be too much reading for most people, so I’m not going to post all the GAO’s ramblings lamenting the F-35 program tech data strategy. My response as to ‘why’ the lamentation is pernicious and only an invitation to trouble and more costs-- is long enough.

Bottom line: The F-35 program was undertaken using one strategy for tech data. In the last few years (and well after the F-35 program was started) the DoD has changed their preferred strategy. 

Now the GAO is b*tching about the F-35 program not being in ‘compliance’ with a newer strategy. The original F-35 strategy was to only pay for the tech data the F-35 operators and maintainers would actually use. Pretty smart huh? Unless you are working in certain parts of the USG and believe EVERYTHING should be under the direct control of the USG.

When the government decided to buy only the tech data they needed, it does not mean the rest of the tech data they desire is just sitting there for the asking. I doubt in most cases it even exists, much less is already in hands of the suppliers ready to go. And it most probably does not exist (yet) for the simple reason that no company these days can afford to expend effort on tasks that for which they are not contracted to deliver. The proprietary KNOWLEDGE needed to create the data exists in the hands of the suppliers, but there probably is no USG-grade (deliverable) tech data extant beyond what the F-35 Program is already paying for. 

On acquisition contract programs, the government: 

1. By law owns the data it pays to own under the contracts, 
2. Has limited rights to related contractor data that allows the USG to use that data for the DoD’s own purposes but cannot be shared with a third party, and 
3. May include a mechanism reserving rights to buy more data from the contractors that contains the tribal knowledge they want…but they still have to pay for it to be developed and delivered. 

To change the strategy NOW to buy more data would cost bigger bucks than anyone involved would ever be willing to talk about.  The GAO may get to whine about this for decades.  

If the F-35 program, GAO, or other agency managed to get a wild hair up the ‘nethers’ and insist the contractors produce data over and above that already agreed to be delivered, and then turn it over to the Government for free, It would cost the taxpayers even MORE money to settle the lawsuits (perhaps dozens that could last decades) over what would, in essence, be the USG compelling the contractors to turn over their property (proprietary data) for use by competitors. This information would not only allow those competitors to compete against the incumbent contractors using the incumbent’s own trade secrets on the F-35 program but also on any number of future competitions as well.  

If this GAO ‘whine’ gains political traction, it will be ‘Stupid’ on steroids.

Life Cycle O&S Cost B.S. (P. 24)

The most important thing to take away from this section is that the ‘huge’ O&S cost numbers being contrasted are:

1.  Costs over a “56 year operational life” and 
2.  Based upon premature and incorrect judgments being made as to the relevance of immature data. 

Let that sink in for a moment. What could go wrong in relying on ‘estimates’ ONLY a half-century into the future? (/sarc)

Could you see anyone in 1943 sitting down with a rational expectation they could characterize a reasonable (or even remotely credible) O&S cost estimate for the P-51 Mustang through to the year 19969? In decrying and calling for ‘resolving’ the 56 year O&S cost estimate ambiguity, the GAO is apparently seeking more certainty in their uncertainty. They aren’t looking for better information. They are looking for information they will feel better about.

As 'Neptune’ at F-16.net summarized:

GAO was mandated to review DOD's F-35 sustainment planning efforts... 

GAO recommends that DOD develop better informed affordability constraints… 

DOD concurred with all... 

and partially concurred with the recommendation… 

GAO continues to believe that the recommended analysis would provide a more comprehensive sense of the uncertainty in the estimates... 

It never hurts to have someone, else, to look over your "most favored" program…Glad to hear they can provide a more comprehensive sense of uncertainty…

Here’s my take on "Certainty".

If the GAO wants “certainty”, tell them to first have someone build them a time machine (they can write ‘reports’ about it too!), and then get back to us with their numbers when they return. Until the GAO can get that time machine fielded, then EACH, ANY and EVERY “56 YEAR” cost estimate that is produced isn’t just ‘uncertain’. It is a SWAG.

 

Are there no aviators in the GAO? Pg 26

In this particular report the GAO treats Fuel Burn Rates in cartoonish fashion, and from the first sentence it is clear they are working back from a conclusion to justify their 'uncertainty'.

•Fuel burn rates: The JPO and CAPE estimates do not use a reasonable assumption for the Marine Corps fuel burn rate across the life cycle of the aircraft. Both estimates use a fuel burn assumption of 1,493 gallons per flying hour for the Marine Corps variant. While the rate used in the estimates was calculated based on service-planned missions at maturity and DOD-validated physics-based models, this burn rate is an almost exact match of the burn rate being observed at the operational sites. This may be a reasonable assumption for the short term, but it is likely that the fuel burn rates will increase in the long term. The burn rate of 1,493 gallons per flying hour used in the JPO and CAPE cost estimates reflects fuel burn data from aircraft flown using limited capabilities—at slower speeds and at lower altitudes than the F-35 will eventually fly. After 2015, the Marine Corps is planning on using the aircraft’s increased capability, which will likely result in more fuel being used. In addition, shortly after 2015, the Marine Corps is planning on deploying the F-35 to ships, where the aircraft’s fuel-intensive vertical landing capability will be used more frequently, which will likely increase fuel usage. Consequently, the use of lower fuel burn rate across the entire life cycle of the Marine Corps variant is not a reasonable assumption.

Perhaps if the GAO took a broader view as to the possible factors affecting fuel burn rates?
Afterburning turbofan engines LIKE to fly faster and higher. Up to ~35-40K feet and about Mach .95, they tend to get more efficient the higher and faster they fly. The F-35, like all aircraft, is a design that is optimized to perform best in a bounded range of speeds and altitudes. If the F-35 is flying lower and slower now than it will be flying later, this suggests speed and altitude as drivers for LOWER fuel consumption in the future. I find it interesting that the GAO thinks it is worthy of observation that the DOD-validated physics models closely track actual data but then it discounts the model and methodology for future fuel consumption predictions. Did the GAO look at the internals of the model being used?
We should probably also note here, that as the F-35 F135 engine has a rather high thrust rating at military power (no-afterburner) relative to its thrust rating with afterburner, and the F-35 has been acknowledged as being able to ‘supercruise’ (by definition ‘without’ afterburner) some distance at around M1.2. Therefore, the F-35 fleet may spend less net flight time in afterburner than the legacy aircraft. It would also take very few seconds of reduced afterburner time to save more fuel than that burned at 'high' throttle settings (without afterburner) in STOVL mode. Maybe the F-35 will spend more time in AB if the pilots find it advantageous and eat up the fuel savings. Who knows until they get their hands on the jet in numbers?
While the GAO is specific as to one change that may increase fuel consumption: the short mission segment involving (sometimes) vertical landings. The GAOers are evidently oblivious to the relative effects of being able to spend significant percentages of flight hours at more economic fuel burn settings on the other side of the balance of things. Consequently, the GAO's dismissal of the possibility of lower fuel burn rates in the future was not a reasonable assumption. I think the GAO may have been alluding to more afterburner time in the future when they talk about speed, but the GAO did not specify afterburner time as the discriminant, and GAO’s crude approach to comparing fuel costs does not permit the needed granularity to compare costs anyway.

More Fuel Fun (Pg 27)

The GAO wasn't done with this bone yet:

Similarly, the JPO estimate lowered its fuel burn rate assumption to 1,480 gallons per flight hour from 1,558 gallons per flight hour for the Air Force variant. However, the Air Force stated that the more conservative assumption of 1,558 gallons per flight hour should be used across the life cycle because the F-35 has yet to use its full flight capabilities, weapons, or mission systems, which will likely increase the fuel burn rates in the long term. The higher fuel burn rate assumption for the Air Force would represent a $4.0 billion cost increase in base year 2012 dollars across the life cycle of the aircraft.

This comes closer to making sense, if the implication is that the AF's F-35s will be flying at higher weights and yanking-and-banking than it currently does. But even IF the USAF's ~5.27% higher than the program's estimate of total fuel consumption is a ‘better’ estimate, it may not be ‘right’ either. It all depends upon how closely the internal model ground rules and assumptions will match future use. There is also a VERY good chance both models make incorrect (and high) assumptions as to afterburner use. 
Finally, the amount of fuel burned isn’t nearly as important as the cost of fuel burned, and not all fuel ‘costs’ the same; cost depends largely on delivery method and location. With typical planned F-35 payloads, carried internally, the range without refueling is comparatively longer than the legacy aircraft it is replacing for all the variants. This indicates less fuel will be delivered by mid-air refueling than for legacy aircraft the F-35 is replacing. Fuel delivered by aerial refueling was shown (circa 2001) to cost about 13.8 TIMES the cost of refueling via ground refueling (Pg. 8).

I imagine fuel delivery via ship lies somewhere in between ground and air delivery, and probably closer to ground refueling since it is moved and delivered in greater bulk than by air. But my point is made: it’s not the number of gallons that is important. What is important is the cost of the gallons delivered. Just something for the GAO to consider for their next SWAG.

The absolute bottom line on the fuel story is that the GAO did not present proper justification for their assertion that “The JPO and CAPE estimates do not use a reasonable assumption for the Marine Corps fuel burn rate across the life cycle of the aircraft”. Additionally, the GAO including the statement of “ the Air Force stated that the more conservative assumption of 1,558 gallons per flight hour should be used across the life cycle because the F-35 has yet to use its full flight capabilities, weapons, or mission systems, which will likely increase the fuel burn rates in the long term” should be a clear indication that there is great ambiguity in any long-term estimate of future fuel use. The AF position makes sense, if the implication is that the AF's F-35 s will be flying at higher weights than it currently does and nothing else changes. But even if the USAF's ~5.27% higher than DoD's estimate of total fuel consumption is a better estimate, it may not be right either depending upon internal model ground rules and assumptions. There is a VERY good chance both models make incorrect (on the high side) assumptions as to afterburner use. 
However, it appears the main objectives of the GAO to include this point it is to use it to question the USMC’s fuel estimates and to warn “The higher fuel burn rate assumption for the Air Force would represent a $4.0 billion cost increase in base year 2012 dollars across the life cycle of the aircraft.” I particularly enjoy the GAO’s penchant for obfuscating cost impacts by rolling them up into the largest possible numbers over the longest possible times. The “$4.0 billion” figure is a good example. $4.0 billion over 56 years is a very small number when thought in terms of cost per aircraft per day. I expect it to be even smaller than the savings to be had from the fuel consumption reduction that will come from the progressive engine improvements already in planning.


Why do I expect it?
In July, A Pratt and Whitney official was reported on as saying some very interesting things:

Mr. Croswell said Pratt & Whitney also is starting to explore how to upgrade the F135 powerplant in coming years. "We see real opportunities to continue to improve the F135 over time," he said. Fuel consumption could be cut 7% by around 2020 and a combined 15% to 20% around five years later, he said. The life of the engine could be increased around 50% within the next decade, he said. Those effort could be a key element in the wider push within the F-35 program to cut the long-term costs of the Pentagon's most expensive weapons program. (link)

Yeah, that probably explains why the program's internal fuel estimates aren't as high as the others, but don’t tell the GAO. To them the world is static. Planned and programmed change  just sounds like “crazy talk” as far as they’re concerned.

Part Replacement: The GAO really goes off the rails (Pg.28)

This is where the GAO really displays a lack of knowledge, And they do it SO effusively...

Part replacement:The JPO estimate does not include reasonable assumptions for part replacement. Based on data from the Air Force and Marine Corps F-35 variants at testing and operational sites, parts are being replaced, on average, 15 to 16 times more frequently than the assumptions used.

The GAO report makes no case for the first sentence, and the second sentence is Garbage Out as the result of Garbage In. Their characterization of the data they present is so awful I find it hard to believe any ill intent, but I also find it not hard at all to observe that analytically, they are WAY out of their depth. What the GAO shows us immediately after the passage above is that they improperly correlate ‘part removals’ to ‘failure rate’ Nothing could be further from the truth.
The GAO confounds Mean Time Between Removals (MTBR) with time between failures. They presume that all ‘part removal’ was due to a failure (reliability issue) and then it was replaced with a different part. The table the GAO showed provides some insight as to what the list is really about, and it isn’t just about failure rates.

Sorry, Removal Rates Do Not equal Failure Rates

‘Removal rates’ include a variety of ‘causes’ that have nothing to do with reliability or failures.
The GAO ignores that possibility (more like probability in some cases) that the same removed part was reinstalled for any of several reasons. Most likely among those reasons are to get at another part (For Other Maintenance’ or FOM) or to perform a precautionary inspection or correct a miss-installation (Ejection Seat Module?). Since the program is actively bringing early LRIP jets into baseline Block 3 configurations, it should be no surprise if this list contains parts removed to be upgraded and replaced by a previously upgraded part (HMD system?).
I would also note here that the ‘multiples of times than expected’ figures themselves are almost certainly misleading for at least some components. The given time frame for the removals was from March 2013 to March 2014. As the removal rates are in flight hours, using the F-35’s ‘Fast Facts’ updates for source data (March 2013 Here and February 2014 Here, both in PDF) we see the entire fleet flight hours for that timeframe was about 6000 or so flight hours. The F-35 A and B model flight hours would be some subset of the 6000+ hours. If any of those components listed has a high predicted MFHBR rate and only handful of them are removed for any reason, the ‘multiples of times’ could be very high though the number of actual removals for were small.       

Even IF the GAO’s list of removals actually reflected failure data (which it doesn’t) they still wouldn’t be correct in assuming them as representative of the entire F-35 system reliability. They would be basing their conclusions on just data for the current high drivers, and extrapolating it to effects on the entire aircraft. It is one thing to pick a small sample, it is another thing to pick a pseudo-random sample, and yet another thing altogether to pick ALL items intentionally only because they are the ‘high drivers’.
In doing so the GAO effectively ignores the removal rate of perhaps several hundred (at least and perhaps a thousand at most) components when they admit that they looked at fewer than 200 of the ‘high drivers’ for each of the F-35A and F-35B models. As far as overall reliability, the impact of the reliability of the components that are NOT failing at ‘higher multiples’ could easily outweigh the impact accounted for in evaluating the high drivers the GAO cherry-picked for the report.

Ummm...About those component ‘costs’

The ‘costs’ presented caught my eye first when looking at the table above. They are clearly unit costs, but when a part is actually failed, then unit cost for replacement is only suitable when the component is non-reparable. Looking at the list, and based upon unit cost, I would estimate only one, perhaps two of the components on this list would normally be non-reparable. The number of spares required to cover the repair items in the pipeline would cost as much as the numbers provided but the number of spares would be a very small number compared to the number of actual repairs over the same timeframe and is dependent upon repair turnaround time and transport/processing time. Some small percentages of failures would require scrapping and replacement, but the rest would have repair costs that would be only fractions of the replacement costs. So even if all the parts removed were repairs, the implication that the costs the GAO presented were relevant to their costs conclusions is hogwash.

The Curious Case of the Fallacious Appeal to Authority (P. 29)

The GAO attempts to wrap this section up in some semblance of authority but fails miserably:

However, according to officials from the Institute for Defense Analysis, who conducted a study of the F-35’s R+M for DOT&E, the F-35 program would have to achieve a higher reliability-growth improvement rate than has been observed in almost all other aircraft in order to meet the anticipated reliability by 2020. As a result, it is likely that the depot maintenance hourly cost used in the JPO’s 2013 estimate is not a reasonable assumption. As previously stated, reliability improvement efforts are under way that could reduce these costs, but it is unlikely that these efforts will bring significant results in the near term because the current F-35 fleet must be modified into the configuration necessary for reliability improvement.

Treating a DOT&E sponsored ‘analysis’ by IDA (by the way I’m familiar with a bomber ‘study’ IDA did once that stunk so bad it got buried before the ink was dry) as somehow more authoritative is a form of fallacious appeal to authority. The GAO’s preferred source is in disagreement with the JSFPO’s source. this is an area where experts can be expected to disagree. So What? It doesn’t make the GAO’s pet ‘authority’ any more authoritative than the JSFPO’s sources, but it sure does make the GAO’s favored pet farther away from the data, experience and knowledge base needed to understand it. The trend line we noted above for MFHBF(DC) certainly doesn't support the GAO's Doubting Thomases. Therefore, we can conclude disagreement between the two sources certainly doesn’t warrant the GAO’s specious conclusion: 

As a result, for the next 10 years, the F-35 fleet will not represent the configurations necessary for reliability-growth improvement.

One of These is Not Like the Others (Pg. 30)

The GAO now declares:

To develop its hourly cost, the JPO used F-16 data from Lockheed Martin contractors that had been adjusted for the F-35. According to the JPO’s current assumptions, materiel costs would be 30 percent of the labor costs, but data for the AV-8 and F/A-18 depot inductions used in the previous year’s estimate indicate that depot induction materiel costs have historically been closer to 45 percent of labor costs.

Here the GAO reveals its ignorance about what aircraft depots do and how they work. All depots are not all the same. The 'Depot' type and activities are determined by the aircraft requirements. The first thing that comes to mind in reading this passage is to wonder if the GAO is aware as to how material-intensive the F-18 and AV-8 depot operations are here at the end of their service lives? Think structural repairs for starters, and obsolescing components follow in thought. Is the GAO aware of the different types of Depot maintenance? That is to say: The F-18A-D and AV-8B have entirely different depot constructs than the F-35, but the F-35 has the same depot construct as the F-16.

Fortunately, I did an in-depth aircraft depot maintenance analysis a couple of years ago and so I DO know the differences (see figure below) and can explain them to you.

Know Your Depot Types and Activities

There are currently three major types of U.S. fighter aircraft ‘depot’ operations. Depot activity may be conducted at one of the Military Service Depots, or they may occur ‘in the field’. The Department of the Navy conducts Periodic Maintenance Intervals (PMIs) as shown. A simple explanation as to what occurs during a PMI is that ‘discovery’ repairs are completed when discrepancies are found during the inspections, aircraft are upgraded to the latest configurations and structural durability (life) modifications are performed. In addition, selective preventative maintenance is performed to ensure reliability (Reliability Centered Maintenance). It can easily be shown that the F-18A-D has lately required increased maintenance attention in the field and up to the depot level, and the GAO itself knows it is no different for the AV-8B.

As the PMIs of the Navy Fighters are ‘maintenance’ centered, so is the F-15’s Programmed Depot Maintenance. There is a lot of programmed maintenance for the F-15, and I suspect it is getting even more attention these days.

These other depots are designed to perform regular maintenance actions. Programmed maintenance  however, IS NOT part and parcel with depot work for the F-16 paradigm, which is the Modification or 'Mod' Depot. By ‘chance’ (I joke) would have it, the Mod Depot is the construct for the F-35. That’s right, the F-16 and F-35 Depots are for performing modifications to the aircraft, because the airplanes themselves are designed such that regular PDM/PMI is not necessary. The Mod Depots still perform the extraordinary and discovery repairs as needed, but these are O&S costs over and above the ‘depot’ costs. They’re still accounted for, but in a different ‘pot’ of money. The GAO should have at least known how depots differ if they were going to attempt to draw a credible line around depot costs.

I will offer one 'depot' consideration for the GAO to factor into their next quest for “certainty in their uncertainty”. And that is they should keep a watchful eye out for changes to the F-35B and C depot constructs to more closely fit the Navy depot mold versus the Navy adapting their depots to the planes they work on. I say this because, 1) it is the Navy we’re talking about, and 2) the overarching driver in timing and durations of PMI activity for naval aircraft is the timing and workup to full readiness in order to support planned ship deployments rather than to meet the aircraft requirements themselves. It is a Navy-unique ‘cost of doing business’ .

Therefore, when the GAO concludes…

“As a result, it is likely that the depot maintenance hourly cost used in the JPO’s 2013 estimate is not a reasonable assumption.”

…simply because they prefer F-18 and AV-8 depot cost numbers over the F-16 numbers as a planning basis, even though the F-16 depot construct is closer to the F-35's construct, they are most certainly talking out of their a**.

We left some small stuff on the table, but I think we've now adequately covered the GAO's flailing about in their desire to 'feel good about their uncertainties'. Good night!

Bonus Fun

If you want to hear the GAO bray about their ‘watchdog’ role, catch this short podcast on this report. The transcript is there, but in the podcast you get to hear the Valley Girl (Imitation?)interviewer grilling her words away as she questions the interviewee-- a seriously earnest, if hapless, protector of the American taxpayer.
It's a hoot.

Note: minor changes made 5 Nov. for clarity, readability, and in defiance of Bill Gates; spellchecker.