We all know that for a waste-basket fire extinguished on a marble floor inside a sprinklered mall one can't claim with a straight face that the entire $100 million mall was saved unless there were some highly unusual circumstances. But put the waste basket in the corner of someone's living room, with flames impinging on drapes or an upholstered chair, and one can credibly claim that extinguishing the fire at that point would save the entire house and possibly the people in it. If the fire were in one of several connected row houses, one might even claim further value saved, depending on the details of the fire and the structures.

Loss averted or dollars saved is the true measure of firefighting effectiveness, but it has been hard to measure directly. To this end, the U.S. Navy Fire and Emergency Services program has pioneered a credible new approach to estimating the dollars saved by its fire units. The methods can be used by civilian fire departments, airport fire departments and other military fire services.

The first three years of data showed that the Navy's fire operations saved more than its cost. In fact, it saved about double its cost. This was the first time the Navy fire service had the data to show the value of its operations side quantitatively.

The concept of computing the dollars and people saved by fire operations isn't new. However, most previous estimates were overstated and lacked credibility. The total value of the property in which a fire occurred minus the estimated dollar loss often was claimed as the dollars saved, regardless of the scenario and likelihood of fire spread if the fire service had not arrived. The number of people in the structure were sometimes all claimed as saved, even if they self-egressed.

The key to estimating dollars saved (or loss averted) is to answer how far would the fire have extended if the fire service had not arrived when it did. Providing a credible answer to that question was at the heart of the Navy's technique. Because budgeteers tend to callously focus more on dollars saved than people, the methodology focused mostly on property saves, but was extended to life saves as well.

Fire models could be used to run a fire scenario and scientifically estimate how far a fire in a structure would have extended if there were no fire department intervention, though the models rarely are used for that purpose. Fire modeling has been used to forensically analyze major fires and for the design of structures, aircraft, ships and vehicles. The model must have as input the geometry of the structure, its fire loading, fire rating, materials, fire protection features (active and passive), windows open or closed and other factors. Different models are needed for different types of structures, aircraft and ships.

This rigorous approach seemed too difficult to individually model a large number of fires, and would take much time and engineering expertise with the present state-of-the-art of modeling.

Ways to model efficiently might be developed in the future for the purpose of approximating saves. For example, one might develop a set of scenarios that can be matched to the most typical fires fought by a particular fire department. (One could even use such models to estimate the effectiveness of different response times in terms of damage averted by the fire service.) But this technique used a simpler approach that relied on fire service and engineering judgment for making a first approximation of saves.

From war movies and novels or their own military service, many readers are familiar with the concept of a fighter pilot or bomber crew being debriefed on their claim to have shot down an enemy aircraft. They might declare a “confirmed” kill (saw the pilot bailout or the plane explode), a “probable” kill (saw the plane hit and smoking but did not see it crash or crew bailout), or a “possible” kill (some evidence of hitting it). The Navy flipped this concept around to categorize firefighting saves as being highly probable, probable, possible, or unlikely.

More specifically, it wanted to get an estimate of the loss averted in the set of fires fought in a year. To do this we need to estimate the likelihood that the fires would have done more damage had the fire service not stopped the fires where it did, and the value of the property to which the fire might have extended and destroyed. If a particular fire scenario is rerun multiple times, there is some statistical variation in the outcomes. Sometimes the fire might spread much farther because there are variations in the details of materials and structures that seem on the surface to be identical. However, it is quite difficult for a fire officer (or anyone else) to estimate the numeric probability that a fire would extend throughout a fire compartment or entire building or aircraft.

Knowing all of the relevant details reduces the uncertainty of the outcome, but full knowledge of the details rarely exists and takes a long time to determine. Also, most fire officers don't think in probability terms; even if they did, it seemed too difficult and arbitrary to force them to record a probability number that has little basis. This method acknowledges the variability of fire outcomes and the lack of detailed knowledge of the scenario, both combined in one estimation of the likelihood the fire would have extended.

To make the estimation process easier for a first approximation, the Navy used the following definitions for the likelihood of a save:

• Highly probable, 90% or higher likelihood that the fire would consume the area or object claimed to be saved if not stopped where it was. The fire almost surely would have spread farther given the scenario.

• Probable, 60% or higher likelihood of spread. There is a better-than 50/50 chance of spreading farther, but not so sure as to call it “highly probable.”

• Possible, 25% or higher likelihood that the fire might have spread, but it probably would not have spread three out of four times.

• Unlikely/low possibility of spread, only about 10% likelihood of spread. In most cases, the fire would not spread even without fire service intervention. In fact, if a fire officer thought the spread was unlikely, it should not be recorded as a save.

The likelihood of spread also may be thought of as reflecting how often the scenario would have resulted in the spread of the fire if repeated over and over. For example, a kitchen fire involving food on a stove might spread beyond the stove only one in 10 times, such as when the food being cooked flared up or gave off sparks that ignited nearby food boxes. But a kitchen fire that already had spread to the walls and cabinets may continue to spread to the entire home nine out of 10 times if the home was of wood-construction, had no credible compartmentation or sprinklering, and was not stopped by the fire service.

The system uses the lower bound in each likelihood category (0.9, 0.6, 0.25, and 0.1 respectively) as “discount factors” that are multiplied by the claimed value of the property saved to estimate the dollars saved. The value of the property saved is the value of the structure and contents that probably would have been damaged, which is not necessarily the total value of the structure and contents. The value saved could be only a fraction of the remaining property if, for example, part was sprinklered. Technical judgment was used to estimate how far the fire was likely to progress if not stopped. This judgment requires a detailed fire scenario and knowledge of the property at risk. This looks at the order of magnitude saved, not precise amounts. That is, did the fire department stop a $20,000 or $200,000 or $2 million loss in a particular fire?

To make the judgment about likely spread, a fire officer needs to think about where the fire was when stopped, what the avenues of spread were for further damage, what active and passive fire protection systems were present and whether they functioned properly. Would the fire have stayed in the fire compartment and destroyed everything in it? Could it get out of the compartment to the rest of the floor or building? It may be difficult to be definitive, but an educated prediction can be made — and can be documented in the narrative to claim a save. One needs to report the likely path of spread. (As a side benefit, the process of thinking about paths of spread also sharpens thinking about fire protection.)

A claim of a save might be given in two or three parts. For example, there could be a highly probable save of a room and contents, a probable save of the rest of the building, and a possible save of the adjacent building. This may happen, for example, if the fire area were not sprinklered, the door to the adjacent area was closed but not well fitted, and the exposure building was sprinklered but could have its exterior ignited by radiant heat if the fire department had not intervened.

To keep the procedure simple, the property values used were the Navy's book values of the property plus estimated value of contents. For civilian departments, the estimated current value, insured value or purchase cost (if relatively new) could be used. The estimated dollar loss from the fire is subtracted from the property value before applying the discount factor. So if a property was valued at $1 million and suffered an estimated $100,000 loss, but the rest of the property was almost surely saved, this estimated the save would be ($1 million - $100,000) × 0.9 = $810,000.

Navy's Estimated Saves (right) shows the estimated property value saved by the Navy in one year, using the methodology described above. There were 20 fires in which spread seemed highly probable without the fire service stop. The value at risk in those structures less fire loss was $15,286,800, which was multiplied by the discount factor of 0.9 to get a more conservative estimate of $13,754,000 saved. There also were six fires where unburned property valued at $22,613,000 probably was saved. That property value less the loss was multiplied by 0.6 to reflect the less than certain likelihood of spread. Another group of four fires were judged to have possible saves, and their property value at risk less the loss was multiplied by 0.25 to estimate dollars saved. The estimated total dollar value saved in structures, after discounting, was $34 million in round numbers. The same process was repeated for aircraft, ships, other vehicles and outside fires.

Of course there is much judgment involved, some arbitrary, in estimating how far a fire would have spread without intervention and choosing the likelihood category, but it does represent at least a start on getting a reasonable approximation and is certainly more accurate than simply counting the total value of the property in which the fire occurred as the value saved, regardless of circumstances.

One can see from the estimates that in the Navy, much higher property values were at stake in aircraft fires and ship fires than in structures. And structures tend to have more value at risk than vehicle and outside fires. But there are exceptions. One warehouse fire had $90 million in contents at risk, higher than the value of many aircraft and higher than the value of the structure itself.

The property at risk from further fire damage totaled $741 million in 2005, which after discounting by the likelihood of spread yielded an estimated $354 million in saves. Only about half of Navy installations reported saves in 2005, so this estimate probably is conservative.

The Navy focused on estimating dollars saved only from “significant saves,” defined as any fire in which at least $100,000 was claimed to have been saved or a life or serious injury averted. A significant save can result from a small-loss fire that is stopped at the incipient fire level. A significant save also can result from a large-loss fire if an even larger loss was averted. For example, the Navy sometimes extinguishes an aircraft engine fire on the ground, resulting in a million dollar engine loss but saving the rest of a $60 million aircraft, for a potential save of $59 million. A fire that destroys one house in a fourplex may result in a loss of $150,000 but a save of $450,000 (3 × $150,000).

There is no reason why the methodology couldn't be used for all fires, but the majority of the dollars saved come from a small number of fires. It would take more effort than seemed worthwhile to make estimates for saves from every fire, and the Navy wanted to minimize the extra paperwork for fire officers from this new data requirement.

The number of Navy fires in which there were significant property saves in 2005-2007 is shown at the right. The raw dollar value of the property saved by fire services as reported by the Navy regions was $5 billion for the three years, which included saves of $3.2 billion for two nuclear submarines in dry dock. Without the two submarine fires, the raw property value saved was about $1.95 billion. After discounting, the three-year save estimate was still $1.46 billion. The annual saves averaged double the fire service budget, which included the cost of prevention, and the number of saves were under-reported, so the cost-effectiveness of fire operations is even higher than the data indicated.

A completed National Fire Incident Reporting System incident report with an excellent narrative would suffice for estimating saves. But because data collection is not perfect and narratives rarely have the details needed to estimate potential spread, the Navy asked its fire installations to report some specific data for each significant save that was claimed. The fire officers reporting the save were asked to describe the scenario of the fire and the path for its spread. An electronic data entry form was used to collect the extra data elements. The detailed scenario of the fire must say how the reporting fire officer thought the fire would have extended, to what areas of the structure, aircraft or vehicle it would have extended, and the potential extensions beyond the building or vehicle of origin.

The special data collected included information on the units, methods and extinguishing agents used to extinguish the fire, to obtain more details on the way the save was achieved. This data allows judgments to be made about the effectiveness of various firefighting resources. For example, aircraft saves often resulted from rapid use of a flight line extinguisher by a flight crew or ground crew member who was trained by the fire service, and provided with extinguishers inspected by the fire service. Bumper nozzles and single handlines also were commonly used, but rarely more than two hoselines for an aircraft save. This had implications for staffing and equipping ARFF units.

Independent validity checks were made on all significant saves claimed by Navy firefighters. Experienced third-party fire experts reviewed each claim. The validation used four approaches:

1. Expert review of the narrative of spread as to its plausibility.

2. Review of the estimated value at risk to make sure it was plausible and consistent with the values used in other incidents.

3. Where data was incomplete, phone interviews with the incident commander or another fire officer who was on the scene. (Significant saves and losses tend to be memorable, and many important details came from these interviews.)

4. For a small sample of saves, visits to the scene to view the structure, aircraft or ship, or if that was not feasible, a visit to the same class of aircraft or ship for which the save was reported. Special visits were reserved for very large save claims (over $100 million). Some visits to fire scenes were made incidental to visiting an installation for other purposes.

The validation process changed some estimates of dollars saved and the likelihoods of saves. But the overall finding from the validation checks was that the saves data appeared to have quite reasonable validity, with only a few exceptions. Surprisingly, the fire service installations more often were underreporting saves, not exaggerating them. The value of the contents of the fire building often was not included in the initial estimate of dollars saved, because the fire officer could readily look up the value of the structure but not the contents. Potential exposure fires also were not always included.

Some ground rules need to be established on what to include when computing dollars and lives saved. The Navy estimates used the following rules:

1. Include the value of contents saved.

2. Include potential damage by smoke or other damage agents.

3. Include saves from hazmat, technical-rescue and EMS incidents.

4. Do not include saves made solely by outside mutual aid.

5. Include but report separately the saves made by the fire service out of its jurisdiction; that is, from mutual aid given off the installation. While the Navy must justify its deployment and fire service resources primarily in terms of its own needs, it often saves the tax money by providing mutual aid outside of base installations. The Navy fire service also wins precious goodwill overseas when it saves homes or businesses off the base.

6. Exclude saves from fires extinguished on an aircraft in the air or on a ship at sea. The fires had to have been fought by Navy fire and emergency services based on installations. This included aircraft on the ground and ships in port if the land-based crews helped extinguish a fire.

7. For saves of the environment, if making a dollar estimate is difficult without a special study, simply describe the type and amount of environment saved, such as “500 acres of protected habitat” or “several mining shacks on the historical register.”

8. Include saves made by ground crews trained by the fire service using equipment provided or inspected by the fire service.

9. Include saves by automatic suppression systems if they were supplemented by fire and emergency services. If not supplemented — that is, extinguished solely by the systems themselves — report that separately.

10. Fires fought on crashes where nothing was saved are not to be included unless the fire operating stopped spread to other structures, vehicles or lands.

The number of people saved from dying or serious harm also were recorded, but with no discounting. Instead, the scenarios were reviewed to determine if the saves were credible. Besides screening incident reports, the Navy's file of certificates of life saving were reviewed. The certifications are awarded to acknowledge people credited with a life save, and require a written citation much as for a medal. The National Fire Protection Association has long recorded life saves from its Learn Not to Burn and Risk Watch programs, based on documentation such as letters from school officials as well as fire incident reports. The recording of documented saves for the Navy was similar to that in principle. Again, the key for credibility is to explain for each save claimed how the person would have been injured more severely or possibly killed if the fire service had not intervened.

As with any new method, problems occurred in the initial use of this save methodology. One of the largest problems was illustrated by two small fires in nuclear submarines, which cost $1.6 billion each. The submarines were in dry dock with hatches open. Fires were started by workers who were making repairs that required hotworks inside the submarines. There was professional disagreement hampered by classified information as to how far the small fires might have spread if not extinguished, and the dollar value of the potential damage. Because of the potential high property value at risk, damages to even a small fraction of a submarine can result in huge dollar loss. Because the estimated spread was difficult to determine, the annual saves estimates were presented two ways, with and without the submarine fires. Any fire large enough to distort a single year's data should be investigated with a special study, or report the save separately so it does not distort the annual data.

Another problem was getting fire officers to report details on the potential avenue of spread of the fires they stopped. They were used to discussing cause and origin and how the fire got to the point at which they stopped it, but were not used to thinking about spread beyond that point. Forensic engineers and lawyers often spend much time trying to prove how fire operations were conducted poorly, to assist lawsuits. It's best if the same forensic engineering approach to be used more positively, to estimate the losses averted, which is a new way of thinking about fires for most people.

As practice in making saves estimates improves, the fire services will be able to credibly measure the true effectiveness of suppression for the first time — not just the dollar losses that occurred, but the losses averted.

Philip Schaenman is the president of the TriData Division of System Planning Corp. He was the lead author of the report Best Practices in Fire Prevention for Fire and Emergency Services on U.S. Navy Installations, on which his previous article “First Class” was based. Schaenman previously was associate administrator of the U.S. Fire Administration in charge of the National Fire Data Center and the Fire Technology Program. He holds advanced engineering degrees from Stanford University and Columbia University.

Navy's Estimated Saves

	Number of incidents	Property Value at Risk	Discount Factor	Estimated Value Saved
STRUCTURE FIRES
Highly Probable Saves	20	$15,286,800	0.9	$13,754,400
Probable Saves	6	$22,613,000	0.6	$13,567,920
Possible Saves	4	$27,970,000	0.25	$6,992,500
Subtotal	28	$65,869,800		$34,314,020
AIRCRAFT FIRES
Highly Probable Saves	4	$169,000,000	0.9	$152,100,000
Probable Saves	2	$58,000,000	0.6	$34,800,000
Possible Saves	1	$280,000,000	0.25	$70,000,000
Subtotal	8	$507,000,000		$256,900,000
SHIP FIRES
Highly Probable Saves	2	$200,000	0.9	$180,000
Probable Saves	1	$1,000,000	0.6	$600,000
Possible Saves	2	$23,000,000	0.25	$13,800,000
Unlikely Saves	1	$100,000,000	0.1	$10,000,000
Subtotal	4*	$124,200,000		$24,580,000
VEHICLE FIRES
Highly Probable Saves	1	$39,600,000	0.9	$35,640,000
OUTSIDE FIRES
Highly Probable Saves	1	$2,500,000	0.9	$2,250,000
Probable Saves	2	$600,000	0.6	$360,000
Possible Saves	2	$1,100,000	0.25	$275,000
Subtotal	5	$4,200,000		$2,885,000
GRAND TOTAL	46	$740,900,000		$354,000,000

Significant Saves, 2003-2005 ($ in millions)

TYPE OF PROPERTY	2005	2004	2003	Total
Structure	28	23	16	67
Aircraft	8	13	7	28
Ship	4	1	-	5
Vehicle	1	1	1	3
Outside	5	1	1	7
TOTAL	46	29	25	100
Property Value at Risk	$741	$878 ($2,478)*	$333 ($2,210)*	= $1,952 ($5,020)
Discounted Save Estimate	$354	$964**	$139**	= $1,457
* With $1.6 billion for submarine fire ** With small percentage from submarine fire