Certain tasks make fire chiefs cringe, and budgeting is one of them. One of the more difficult tasks in budgeting is either reporting your agency's performance for the previous year or justifying budget requests by comparing your agency's performance to that of comparable departments.

Measuring your agency's performance or comparing it against other agencies can be risky for several reasons. First, the media love to compare one fire department to another, which often puts one in a less-than-favorable light. Second, when seeking additional resources to get the job done, decision-makers often want to compare your performance to that of similar agencies. If your performance is good, then your budget requests face an uphill battle — why provide more funding if things are going well? If your agency compares poorly, you must defend why.

Measuring performance and accountability in budgeting have been hot topics for the past decade. While some may see it as a fad, the truth is that these issues are important for public agencies. The theory of performance measurement is fine, but implementation can be full of problems. Understanding these problems is critical if you want to be more successful in the budget process.

Definitional ambiguity

Budget folks often want to know basic performance measures for your agency: average response time, number of calls during the year, dollar losses and the like. They also will want to benchmark your organization so they may provide key decision-makers with comparisons. Typical questions they ask that allow for comparisons include number of uniformed personnel, protected population, staffing levels and budget size.

During budget season it seems to be a weekly ritual for fax machines to spit forth two- or three-page surveys, each asking similar questions and each promising to share their results with you. (While I often ask for the results, I rarely receive them.) The problem with these surveys, and the benchmark comparisons made from them, is that they suffer from definitional ambiguity, or the use of different definitions for important terms. Let's look at a couple of easy performance measures often used to make comparisons between fire agencies.

Number of calls

Chief officers often ask their colleagues how many calls they run each year, but rarely do they know how the other chief counts his or her calls. Some agencies count each emergency incident as a single event. Others assign a different call number for each piece of equipment that responds, a practice that often results in a 30% increase in the number of calls reported. There are some agencies that count both the number of apparatus responding, with each assigned a separate call number, and the number of patients transported in each vehicle, with each assigned a separate patient number as well. The method of counting both apparatus and patients can result in an agency that responds to 10,000 emergency events each year reporting a total of 14,000 or more “calls” for the same period.

Response time

Fire departments, politicians and the media often use this term, but rarely does anyone know what it means. There are many different definitions of response time. In a recent survey in Florida, fire-rescue agencies reported at least nine different definitions of response time. Almost 50% of the agencies use a definition starting when dispatch answers a 911 call until the emergency vehicle arrives on scene. Another 33% use a definition starting at the time dispatch sends a unit until arrival on scene. There are various other definitions used as well.

The National Fire Incident Reporting System simply uses the term “alarm time,” a phrase that agencies apply differently to identify the point at which the response time clock starts. Both the federal government and IAFC have used contradictory definitions of response time in the past. Consider the following:

• The U.S. Department of Transportation recommends a definition of response time that starts at the time of alarm until the emergency unit arrives on scene. The Department of Health and Human Services recommends a definition that starts at the time of dispatch until the arrival on scene. These competing definitions can make a difference of 1H minutes or more.
• The IAFC endorses a response time definition through its accreditation process that starts at the time of alarm and stops at the arrival of the emergency unit on scene. However, they also support NFPA 1710 and its definition of response time that encompasses only travel time, or the time starting when the responding unit goes en route until arrival on scene. The two definitions, if applied to the same data, provide significantly different reported response times.

There is no one correct answer. Look at the information in Table 1, below, based on actual incident data and calculated from a random sample of 50 emergency responses. Departments involved in this study were large metropolitan departments with similar characteristics, which make them appropriate to benchmark against each other.

Which response time looks better? If you think C, think again. The labels A, B and C are not different departments; they represent different definitions of response time applied to the exact same data. Definition A uses the definition starting at the 911 answer until arrival on scene. The time from dispatch to arrival on scene is definition B and C is the time from unit en route until unit arrival.

I used this approach a number of years ago to explain to elected officials the games that could be, and were being, played by different agencies with respect to response times. The differing response time definitions used here, all examining the exact same data, resulted in response time differences of over 2H minutes. If you're attempting to meet a goal of 6 minutes 90% of the time, you can see the effect of using one definition over another.

Conscious errors

It may appear the only problem is too many agencies using too many definitions. It seems that all the fire service needs to do is develop a common set of definitions for every department to use.

However, it's not that simple; there are reasons that different agencies use different definitions. If your dispatch operation is under control of a different organization, such as the police department, you may argue you have no control over their performance. Therefore, your agency's response time should not include the call-processing component. However, an adjoining community may have their own dispatch center and measure the total time from call receipt until unit arrival on scene. Who is right?

The fire service has advocated measuring performance since the mid-1970s. Many of the early advocates of measuring performance cautioned about potential abuses of these systems. Conscious error is a polite term for distorting the facts.

For example, if somebody asks the number of calls your agency handles each year, you have several choices on how to respond. You could ask them to clarify what they mean by number of calls. You could provide them with your definition of calls and explain that the answer you provide is based only on your definition. A third option is to provide information without clarification.

If taking this third approach, the opportunity exists that the recipient may be using a completely different definition than you employed. If you suspect that different definitions are being used and you don't attempt to clarify your response, there may be conscious errors at work.

In a recent study conducted in Florida, we asked fire chiefs how they defined response time and how they believed others defined it. Not surprisingly, it's clear that fire chiefs generally use a significantly different definition from what they believe politicians and citizens use. More importantly, the definitions used by most fire chiefs make their performance look better than may otherwise be justified. However, do not think the fire service is alone in experiencing conscious errors.

For example, after the Gulf War, even though military officials claimed their weapons systems could strike with tremendous accuracy, the General Accounting Office found many of these claims “were overstated, misleading, inconsistent with the best available data or unverifiable.”

In another case, a study of physicians performing emergency airway management found significant discrepancies between self-reported performance and actual video analysis of the procedures. Researchers believed that medical providers were reluctant to report their actual performance because of embarrassment or fear of retribution.

Individuals and organizations are self-interested, and each has a desire to survive and appear in a positive light. However, when forced to put a number on performance, departments risk looking worse than other departments. In such circumstances, there's a risk that a department may take intentional advantage of the definitional ambiguity to make it look more favorable.

Make sure you measure up

While you shouldn't become paranoid, there are actions you can take to help eliminate the embarrassment should your performance make the front page in a less-than-favorable light.

If asked to respond to a survey where the questions are ambiguous, either get clarification of what the survey is asking or make sure you define what you mean in your response; write your definition next to your answer. Also be sure to keep a copy of what you send back. Nothing is more embarrassing than having your performance published in a local newspaper or other community's budget document with an assertion they got the numbers directly from you. If in doubt, simply do not respond to the survey.

Know how other agencies define and calculate their key performance indicators. The information in Table 2, page 89, provides an example of how other agencies define response time. The survey asked when agencies started the response time clock and when they stopped the response time clock.

You should also have an idea of how your department's performance compares to others. Is there a difference in the definition of key terms, or is your agency actually performing below average? When you publish or report your performance, be sure to clearly define your statistics. This will help decision-makers know what information they're really looking at and how to use it properly.

Every agency wants to appear in a positive light. While the temptation may be great, fire officers must remain ethical in their approach to reporting performance. Expect honest differences, but “cooking the books” will eventually call into question the integrity of the organization and make future budget requests receive unwanted scrutiny. Being clear in what you mean by key performance measures is one way to stay clear of ethical concerns. Make sure headings, footnotes or parenthetical notes clarify the reports.

While others may perform a critical review of your agency's performance only during the budget process, the increasing desire to establish standards within the fire service means fire agencies must become more proficient at performance measurement throughout the year. Understanding the definition of key terms and being clear on what you report to others will help make your performance measurement activities meaningful.

---

Bruce Moeller, Ph.D., is chief of the Sunrise (Fla.) Fire Department. A member of the Institute of Fire Engineers, Moeller holds a bachelor's degree in fire administration and both a master's degree and doctoral degree in public administration. He has taught undergraduate and graduate courses in labor relations, organizational behavior and management and is an adjunct professor at Florida Atlantic University.

Table 1
Differences in response time methods

	Average response time (min: sec)	Responses within 6 minutes or less (%)
A	7:09	39.8%
B	5:49	59.2%
C	4:26	79.9%
A, B and C are not different departments; they represent different definitions of response time applied to the exact same data. Definition A starts at the 911 answer until arrival on scene. Definition B is the time from dispatch to arrival on scene and C is the time from unit en route until unit arrival.

Table 2
Response times

Response time starts

911 answered
47.9%
911 call completed, entered into CAD
14.5%
Emergency unit dispatched
33.3%
Emergency unit en route
1.7%

Response time stops

Arrival on scene
85.1%
Patient contact/begin operations
14.9%

This survey, which shows how other agencies define response time, asked when agencies started and stopped the response time clock.

For further info

Federal Emergency Management Agency. Fire Data Analysis Handbook. Washington, D.C.. International City/County Management Association. Benchmarking: Achieving Superior Performance in Fire and Emergency Medical Services. Management Information Service, Volume 25, Number 2. Washington, D.C. 1993 Research Triangle Institute. Evaluating the Organization of Service Delivery: Fire. Research Triangle Park, N.C. 1977.