Projects
The Community Capacity Development Office Weed and Seed Indicator Project: Considerations for Implementing an Enhanced Performance Measure System
Size 284.5 kB - File type application/pdf
Full screen
D
The Community Capacity
Development Office Weed
and Seed Indicator Project:
Considerations for
Implementing an Enhanced
Performance Measure
System
Laura Winterfield
Mark Coggeshall
Terence Dunworth
RESEA
RCH R
E
PORT
January 20
06
URBAN INSTITUTE
Justice Policy Center
URBAN INSTITUTE
Justice Policy Center
2100 M Street NW
Washington, DC 20037
www.urban.org
© 2005 Urban Institute
This report was prepared under subcontract to the Justice Research and Statistics Association
(JRSA) and the U.S. Department of Justice, the Community Capacity Development Office
(CCDO). Opinions expressed in this document are those of the authors and do not necessarily
represent the official position or policies of JRSA or the U.S. Department of Justice, the Urban
Institute, its trustees, or its funders.
Contents
Enhanced Performance Measure for the Community Capacity Development Office
i
Acronyms Used in This Report
Community Capacity Development Office
CCDO
Executive Office for Weed and Seed
EOWS
Federal Bureau of Investigation
FBI
Government
Accountability
Office
GAO
Government Performance and Results Act of 1993
GPRA
National Incident-Based Reporting System
NIBRS
Office of Management and Budget
OMB
Performance Measurement Indicators Project
PMIP
Performance Measurement Indicator System
PMIS
President’s
Management
Agenda
PMA
Program Assessment Rating Tool
PART
Uniform
Crime
Reports UCR
Urban
Institute UI
ACKNOWLEDGMENTS
The authors would like to thank the many individuals who have supported the first phase of the
Weed and Seed Performance Measurement Indicators Project (PMIP). At the Urban Institute, we
thank Avi Bhati, Caterina Gouvis Roman, Gretchen Moore, and Sinead Keegan (formerly of UI),
and Kevin Roland and Michelle Scott, who supported and conducted much of the necessary data
preparation and analysis; Elizabeth Cincotta and Deborah Chester, who provided editorial
assistance; and Harry Hatry, Adele Harrell, and John Roman, who reviewed drafts of this report
to provide feedback about the proposed measure and alternative measures. Joan Weiss, Jim
Zepp, Matt Perkins, and Jason Trask of the Justice Research and Statistics Association provided
input on the proposed Performance Measurement Indicators System and provided data from the
Weed and Seed Data Center. The support of the former acting director of the Community
Capacity Development Office, Bob Samuels, was immeasurable. The PMIP would not be
possible without the participation of many willing Weed and Seed sites across the country and
their local police departments, which generously supplied the PMIP team with an abundance of
detailed local crime data. Thanks to all.
Enhanced Performance Measure for the Community Capacity Development Office
ii
Introduction
OVERVIEW AND PROJECT GENESIS
In 2003, the Community Capacity Development Office (CCDO, formerly known as the
Executive Office for Weed and Seed), the Justice Research and Statistics Association (JRSA),
and the Urban Institute (UI) initiated a review of the CCDO approach to performance
measurement. That performance measurement system was based on a single indicator: year-to-
year changes in homicide counts. The key consideration was to determine if enhancements could
be made that would move beyond the simple calculation of year-to-year changes in homicides
without unduly increasing the reporting burden for the sites. This review, then, provided an
opportunity to think broadly about how best to measure crime control strategies that would
account for variation in both crime prevalence and population affected, an interest that goes
beyond CCDO to any agency or policymaker seeking to understand the effect of crime control
programs.
In 2003, UI began developing an enhanced performance measure that would be less
volatile than the number of homicides. In this report, we describe the current performance
measurement and performance management system in place within the executive branch. Then,
we describe our examination of alternative performance measures for the Weed and Seed
program. We conclude that crime rate, rather than counts of homicides alone, would better serve
the need of the CCDO to set performance targets for both individual sites and the Weed and Seed
program as a whole. In the final sections, we recommend changes to the existing Weed and Seed
performance measurement system as well as an approach to integrating the revised system we
propose into the management of the individual sites and of the entire Weed and Seed program.
COMMUNITY CAPACITY DEVELOPMENT OFFICE AND PERFORMANCE INDICATORS
Performance Measures in the Federal Government
As part of the budgeting process for fiscal year (FY) 2004, the Office of Management and
Budget (OMB) issued the following statement:
Federal programs should receive taxpayer dollars only when they prove they
achieve results…Good government—a government responsible to the people
whose dollars it takes to fund its operations—must have as its core purpose the
achievement of results…In a results-oriented government, the burden of proof
rests on each federal program and its advocates to prove that the program is
getting results…
This budget [FY 2004] makes an unprecedented effort to assess the
effectiveness of specific federal programs. It introduces a new rating tool
[Program Assessment Rating Tool, or PART] to hold agencies accountable for
accomplishing results. Programs are rated from Effective to Ineffective, and the
ratings and specific findings produced will be used to make decisions regarding
budgets and policy. The tool assumes that a program that cannot demonstrate
positive results is no more entitled to funding, let alone an increase, than a
program that is clearly failing…
Enhanced Performance Measure for the Community Capacity Development Office
3
Taken seriously, rigorous performance assessment will boost the quality of
federal programs, and taxpayers will see more of the results they were
promised. What works is what matters, and achievement should determine
which programs survive, and which do not. The public must finally be able to
hold managers and policymakers accountable for results, and improve the
performance of federal programs in the way that Presidents and Congresses
have sought for decades. Whether resources shift over time from deficient to
good programs will be the test the PART itself must pass.
Even without consideration of the government’s desire to assess individual agency
performance, a performance measurement indicator system (PMIS) is simply good practice.
Compared with a scientifically sound evaluation, a PMIS is a more cost-effective way of
monitoring activities and expected outcomes. A PMIS can also help isolate approaches and
outcomes that appear worthy of further exploration through hypothesis testing (e.g., a program
that focuses on quality-of-life crimes showed a decline in drug arrests over time) and can help
guide the formulation of a research approach (e.g., the program was in a small rather than a large
jurisdiction). Finally, a PMIS can aid in program management: communities can monitor the
PMIS in other jurisdictions to identify best practices, and managers can use changes in measures
over time to determine whether the programmatic approaches have had the intended effects. If
they have not, then some changes are likely in order.
Local Program Management
Developing the measures is only one component of performance management.
Developing performance measures forces program managers to conceptualize the specific inputs
and activities that they believe are necessary to accomplish the desired outcomes (performance
measures). Further, the specific contextual factors that may influence program performance must
also be understood and addressed. Then, over time, the measures should be used as criteria to
determine whether a specific approach has met its stated goals. If the goals have not been met,
program managers can then review whether the necessary activities were conducted and whether
the appropriate inputs were available. This review can help specify where programmatic changes
might be necessary.
For example, suppose that a specific crime reduction program had as its performance goal
the reduction of robberies. It set as its target a 10 percent reduction in robberies for each of three
consecutive years. Period 1 served as a baseline, during which there were 150 robberies. During
period 2 there were 120 robberies (a 20 percent reduction from period 1), but during period 3
there were 160 robberies (a 33 percent increase from period 2). In a performance management
system, program staff managers would examine the degree to which their targets were met. For
period 2 the targets were met, but for period 3 they were not. Managers would then examine
whether the program’s activities had been adequately conducted (e.g., perhaps there had been a
staffing shortage) or whether there were changes in inputs (e.g., perhaps overall funding had
been cut). If nothing had changed in either the programmatic inputs or activities, perhaps there
1
Office of Management and Budget. 2003. “Rating the Performance of Federal Programs.” in Budget of the United States Government, Fiscal
Year 2004. pp. 47-53. http://www.gpoaccess.gov/usbudget/fy04/pdf/budget/performance.pdf.
Enhanced Performance Measure for the Community Capacity Development Office
4
had been contextual changes (e.g., an adaptive change in the behavior of heroin traffickers).
Performance management incorporates performance measures as guides for ongoing program
administration.
National Program Management
At the national level (e.g., within the Community Capacity Development Office at the
Office of Justice Programs), incorporating performance measures into program management
entails different managerial decisions. Primarily, it can be used by CCDO to assess individual
sites’ performance as one criterion for continued funding. It can also be used to monitor the
relationship between various programmatic activities and outcomes, allowing for a preliminary
understanding of promising program practices. These promising practices, then, are the most
likely candidates for evaluation.
Establishing the Counterfactual
Nonetheless, even with full incorporation of performance measures into program
management (either at the site or the national level), the degree to which observed changes can
be directly attributed to the specific programmatic intervention will not be known if performance
measures are collected only from the Weed and Seed sites themselves during the period of
program activity. Without establishing some type of comparison, either to the site itself between
the pre– and post–Weed and Seed periods or to some other comparison area, the degree to which
the outcome (i.e., performance measure) occurred elsewhere without programmatic intervention
will remain unmeasured.
The only way that causal claims can be made is through the computation and use of
reliable counterfactuals.
This would involve not only setting a specific performance measure or
target for upcoming years, but also developing a methodology to examine the degree to which a
change in the measure for a given site is different from comparison areas or different from
expectations based on observations from past years. Designs for a reporting approach could use
either of the following comparisons:
A comparison of the change for a given measure in the Weed and Seed site with that of a
similar, nearby jurisdiction
A comparison of trends in the Weed and Seed site from before program implementation to
site trends after program implementation (using as many years of data as feasible) to
determine if the trend shifts in the expected direction after Weed and Seed implementation
While these kinds of analyses go beyond the scope of the typical OMB or Government
Performance and Results Act of 1993 (GPRA) reporting, they should be considered in some set
of sites to establish the degree to which the performance goals, as developed by a given federal
agency (here, the CCDO), continue to be logically linked to programmatic efforts and are
attainable. A performance management system can be used to determine which sites to choose.
2
If there were no discernable changes in any of the factors that might be expected to affect the measures, then an examination of the measures
themselves is in order. Perhaps the expectation that a reduction will be achieved each year is inappropriate; one could conceive of a measure that
would show an initial decrease and then stability. The issues of how best to measure change and set performance targets over time are not easy for
any program and require serious consideration.
3
The term “counterfactual” is used to represent the expected value of the outcome in the absence of the intervention.
Enhanced Performance Measure for the Community Capacity Development Office
5
For example, those sites that meet their targets over time might be the best candidates for a full
evaluation.
The CCDO Performance Measure—
Crime Rate
PERFORMANCE MEASUREMENT INDICATORS PROJECT (PMIP) CONTEXT
In FY 2005, the Weed and Seed program received an annual appropriation of
approximately $60 million. Jurisdictions seeking Weed and Seed funding compete annually by
submitting applications for review by the CCDO. Successful sites each receive the same base
award of federal funds. The amount—ranging from $175,000 to $225,000 per annum with
possible renewal for up to four years—is modest compared with most federal programs that
provide direct support to state and local agencies. The award can be supplemented to a modest
degree through special emphasis programs that the CCDO operates. Not all sites receive
supplementary funds, and those that do rarely get more than an additional $50,000. In 2005, 340
sites had federal funding.
Sites have two primary mandates. The first mandate is to reduce and control crime, with a
particular emphasis on violent and drug crime (i.e., “weeding” activities). The second is to
implement social programs that will make the Weed and Seed area more resistant to crime (i.e.,
“seeding” activities). Sites are instructed to devote approximately equal parts of their federal
funding to weeding and seeding activities. Sites are also required to develop ways of sustaining
the initiatives they have launched after their federal support ends.
Sites are required to report annually on their performance toward these goals, and it is the
responsibility of the CCDO to use this information to manage the overall Weed and Seed
program. Continuation of federal support to each site is contingent upon satisfactory progress,
which the CCDO determines. Further, the CCDO has its own performance reporting
requirements to the OMB and the Justice Department. The CCDO must demonstrate to OMB
and the Justice Department the progress of the entire Weed and Seed strategy toward its national
goals.
It is therefore necessary to establish an approach to performance measurement that,
ideally, will allow sites to report accurately on their individual situation and will also give the
CCDO a way to manage the sites through aggregating site-specific information to produce a
national portrait. These objectives need to be met on both the enforcement and social program
aspects of the Weed and Seed program, but because the activities between the components differ
too greatly, no single performance measurement system will be suitable for both components. In
this report, we focus on the crime reduction and control mandate, not the social program
mandate.
There are two conceptual elements to a performance measurement system: the statement
of a measurable outcome and change in this measure over time. Performance measurement
4
This is the number of active sites listed on the web site of the Weed and Seed Data Center (http://www.weedandseeddatacenter.org/map.aspx)
on October 6, 2005.
Enhanced Performance Measure for the Community Capacity Development Office
6
entails the a priori setting of a numeric goal and the specification of a measurement period
during which change in the goal would be measured. Note that the period can be any length—
one month, one year, three years, and so on. A site would be judged as a success or failure based
upon the achievement of the measure from period 1 to period 2.
In criminal justice, the most common performance measure for documenting changes in
crime is unadjusted counts of incidents. For example, if, in a given site, reported incidents of
robberies were 50 in period 1 and 40 in period 2, assuming the two periods are the same length
and the counts are derived from the same geographic area, the unadjusted count approach
indicates that there has been a decline of 10 incidents. If the goal for the site had been a period-
to-period decline of five incidents, it was met. If the goal had been a decline of 20 incidents, it
was not met. More problematic, of course, would be the situation in which reported robberies
were higher in period 2 than in period 1, or in which there is no stated goal. Presumably,
however, consistent failure to meet a goal would be considered poor performance and might
result in termination of federal funding.
In the past, the Community Capacity Development Office used changes in counts of
homicides to assess the performance of Weed and Seed sites. As a performance measure, using
change in homicides has two primary benefits. First, homicide is the most validly reported type
of crime. Relative to other types of crime, few homicides go undetected or unreported for long.
Second, homicide is uniformly recorded. However, for CCDO, this measure has several pitfalls.
First, many crime control initiatives are likely to have only a modest or indirect effect on
homicide. Although CCDO’s focus is on reducing violent crime, drug abuse, and gang activities,
sites frequently begin with such issues as quality-of-life crimes (e.g., vagrancy, loitering,
aggressive panhandling) that have little direct relationship to homicide. A superior performance
measure would allow for a larger set of crimes to be examined. In addition, the typically low
incidence of homicides, even in urban environments, means that it is not uncommon to observe
zero homicides in a community in a given period. Although it is desirable for a community to
have zero homicides, it makes homicide a problematic performance measure because the number
of homicides cannot drop below zero. A more useful performance measure would be sufficiently
common that it would rarely reach its truncation point.
These drawbacks to the use of homicide counts provided the impetus for UI, in
conjunction with CCDO and JRSA, to develop an improved performance measure, one that
would account for differences between sites in population and crime prevalence. For example,
given that Weed and Seed sites all receive similar funding levels yet vary widely in population, it
would be beneficial if the performance measure differentiated between sites that have a few
thousand residents and those that have tens of thousands.
Additionally, if a site with a lower
prevalence of crime achieves a reduction of 10 crimes between periods, its accomplishments are
more dramatic than when a larger site with a higher prevalence of crime achieves the same
reduction of 10 crimes.
5
The population range across sites is remarkable. In October 2005, the JRSA web site (http://www.weedandseeddatacenter.org/) listed 52 sites
with fewer than 5,000 residents, 70 sites with more than 20,000 residents. Eleven sites have more than 50,000 residents. Regardless of size, each
site has a similar federal support level from Weed and Seed.
Enhanced Performance Measure for the Community Capacity Development Office
7
REQUIREMENTS OF THE ENHANCED PERFORMANCE INDICATOR SYSTEM
An ideal performance measurement system would mimic a full-scale outcome evaluation
and capture all the activities, outputs, and outcomes of the Weed and Seed strategy. In addition,
it would account for the counterfactual (i.e., what changes would have been observed in the
outcomes if the Weed and Seed strategy had not been in operation in that site). However, such a
comprehensive approach is not practical, except perhaps in a very limited number of sites,
because of issues surrounding study design and implementation. A practical reporting approach
should be easy to operate and update and should be able to be maintained independently by
Weed and Seed sites. To be practical, the performance indicator should be all of the following:
Easy to understand. The performance indicator should be easily grasped and should not
require complicated, burdensome computation.
Valid and credible. The performance indicator must be reliable and worthy of confidence.
Otherwise, the measure cannot be used to assess operations objectively.
Inexpensive. Weed and Seed sites rely on limited funds for programming and are unable to
support elaborate and expensive analyses for site evaluation. A performance indicator needs
to be effective while not imposing a high cost on the local sites.
Easy to update. Once the performance indicator is in place, sites should be able to input
current information with ease, thereby allowing for ongoing progress reports. Sites would
then be able to more readily identify where programs were effective and also where resources
should be diverted.
Standardized across all sites. The performance indicator should allow a Weed and Seed site
to measure its own performance while supporting national assessments of the Weed and Seed
program as a whole. Consequently, cross-site standardization is essential.
ALTERNATIVE PERFORMANCE MEASURES CONSIDERED BY THE URBAN INSTITUTE
After observing that the current measure of homicide counts—or any measure based on
simple counts—would not account for population, UI has, to date, developed and examined three
measures: a weighted crime count approach, a social benefit index, and simple crime rate. All of
these approaches account for variation in crime prevalence and population over time.
UI initially developed an approach wherein crime rates for specific types of offenses
(e.g., homicide, robbery, assault) in each site were weighted to reflect the prevalence of crime in
the site, whether the type of offense is a CCDO focus area, whether the type of offense was a
focus area for the site, and the contribution of the site to the collective residential population of
all the sites. This weighted approach had two failings. First, it was multilayered and complex,
which rendered it difficult to understand. Second, the population weight was restricted to those
sites included in the performance measurement system.
The second measure UI developed, the social benefit index, began with the inverse of the
crime rate (i.e., the number of persons per crime in a given period), called the absorption rate.
6
It is possible that not all Weed and Seed sites could be included in whatever system is implemented. For example, program managers may
decide to only include those sites that have been officially recognized or those sites that have been operational for at least two years.
Enhanced Performance Measure for the Community Capacity Development Office
8
The social benefit index was the difference between absorption rates in period 1 and absorption
rates in period 2 (i.e., the number of persons per crime in period 2 minus the number of persons
per crime in period 1).
We proposed the social benefit index because we saw two shortcomings with using
changes in the crime rate. First, reductions in crime rates in larger sites protected more people
than the same reductions in sites with smaller populations. That is, more people avoided probable
victimization in the former than in the latter from a similar drop in crime rates. Second, it was
easier to reduce crime in sites where initial prevalence was high than when it was low. The social
benefit index accounted for both the number of people affected and the size of the crime
problem. Nonetheless, because many people found the explanation of the social benefit index
difficult to grasp, we decided that this measure is not appropriate for policy use.
We come, then, to the crime rate. This measure accounts for variations in the two
essential components—the number of crimes and the population being served—combining them
in an easily understood manner. It has the advantage of being easy to calculate, and it is
standardized so that it can be used to compute metrics to compare sites or to assess the
performance of the Weed and Seed program as a whole.
ELEMENTS OF THE URBAN INSTITUTE PERFORMANCE INDICATOR SYSTEM
This section summarizes our thinking about what an ideal performance measurement
system for the Weed and Seed program might look like. When we began our work, we were
interested in grouping offense types conceptually (e.g., violence, property, disorder). However,
because the sites will be relying on reports of crime data from local police departments, UI’s
recommendations include adjusting the GPRA reporting form to incorporate Part I offenses of
the Uniform Crime Reports (UCR) system. Since many police departments already report counts
of Part I offenses to the Federal Bureau of Investigation (FBI), it would be possible to build an
enhanced performance measurement system for Weed and Seed based on the UCR offense
categories without significantly increasing the reporting burden on the sites (see
“Recommendation,” below for details about the changes we propose).
Disaggregated Crime Types
The ideal performance measure should be based on counts of overall crime, which must
be developed from consistently defined categories and subcategories. For this discussion, the
categories for this work were structured as shown in table 1:
Table 1. Crime Categories for the UI Performance Measurement Indicator System
Category Subcategory
Murder, rape, robbery, aggravated assault, simple assault
Violent
7
What is referred to here as the absorption rate has been referred to elsewhere as the “safety index” (Gyimah-Brempong, Kwabena 1989.
“Production of Public Safety: Are Socioeconomic Characteristics of Local Communities Important Factors?” Journal of Applied Econometrics
4(1): 57-71.).
8
When we began the performance measure work, we were not only developing crime categories for the PMIP, but for other evaluation projects as
well. This gave rise to the categories presented here. However, these specific crime categories would impose significant new data collection
burdens on the sites, and we believe that use of UCR Part I offense categories would not.
Enhanced Performance Measure for the Community Capacity Development Office
9
Property Burglary,
theft, car theft, arson
Disorder Includes
disorderly
conduct, public intoxication, prostitution, others
Includes distribution, possession, and use
Drug
Includes illegal possession and use
Weapons
The performance measure calculation could be computed for any subset of these
categories (or subcategories)or for overall crime (i.e., all categories combined). The indicator
presented here, crime rate, is calculated from counts of crimes and an estimate of site population.
Change Over Time
The next concept in developing the performance measure is the change from time 1 (T
1
)
to time 2 (T
2
) in the counts for each of the categories (or subcategories) of crime. These change
calculations form the building blocks of the performance indicator. The periods can be user-
defined, although the duration of each period must be the same across sites. In other words, the
measure is less sensitive to the actual start or end date of the intervals over which the change is
measured than it is to the duration of the interval itself.
Site Population
The third concept in the calculation of the performance measure is the site population at
the beginning of the measurement period (T
1
);
this initial value will be used for subsequent
periods (T
2
, T
3
). Using a single population estimate for relatively small measurement periods (in
this example, for a three-year period) is appropriate because the purpose of dividing the crime
counts by population to produce a rate is to have a standardized measure (i.e., crimes per person)
suitable for aggregation across sites. Since population estimates are error-prone, especially those
projected from the decennial U.S. Census estimates, using a time series of population estimates
as the denominator to compute site-level crime rates is likely to increase the amount of error in
the calculations. We believe, therefore, that the population should be used as a standardization
factor to compute a crime rate for performance measurement. It needs to be only approximately
correct, and its imprecision should be acknowledged. Any variability found in the rate over time
would then be confined to the directly observed counts of crime in the numerator.
Site-Specific Crime Rate
The crime rate measures the approximate risk of victimization an average individual
faces.
Crime rate is computed as the number of reported crimes divided by the number of
persons residing in the jurisdiction in question. Crime rates could be computed from individual
crime types, such as the seven UCR Part I offense categories included in UI’s recommended
performance measurement system (see “Recommendation,” below). In addition, by summing
9
Although some sites are counties, many are not necessarily contiguous with political boundaries but rather reflect neighborhood boundaries.
10
The crime rate approximates this risk because it does not adjust for (1) multiple victimizations of individual persons, (2) the victimization of
nonresidents within the boundaries of a site, or (3) the victimization residents of a site may experience in other jurisdictions.
Enhanced Performance Measure for the Community Capacity Development Office
10
offenses across categories, it is possible to compute an aggregated crime rate (e.g., a Part I crime
rate or a Part I violent crime rate).
Calculation of a National Score
The crime rate can also be computed across various jurisdictions to produce an overall
national-level performance indicator for the program as a whole. This indicator, unlike the
disaggregated crime rate reductions, is not a simple summation of all the individual jurisdiction
crime rate reductions. Rather, it is a weighted sum of all the jurisdiction-specific crime rate
reductions where the weights are proportional to the population of a specific jurisdiction.
Because reductions in the individual site-specific crime rates and the national-level crime
rate are measured on the same scale (e.g., number of crimes reduced per 1,000 residents), they
may be compared directly or displayed on the same figure. This ability provides a convenient
means of displaying various features of performance. For example, it is easy to identify sites that
performed the best or worst in a given year.
Demonstration of Measuring Program
Performance for 16 Pilot Sites
Throughout 2003 and 2004, UI corresponded with more than 50 Weed and Seed sites to
generate interest in the Performance Measurement Indicators Project and to identify sites willing
to participate. More than 30 jurisdictions initially agreed, and UI has received data from 19
jurisdictions to date. Within those jurisdictions, UI designated 16 as pilot sites because these sites
supplied all the necessary underlying data. Data from 16 pilot sites have been analyzed for this
report, and detailed figures comparing sites and presenting trends are provided below. Table 2
displays the participating sites.
Table 2. Weed and Seed Sites Involved in the PMIP
Site city
State
UI received data Multiple sites PMIP pilot site
Baltimore
MD
X
Battle Creek
MI
Boston
MA
Camden NJ
X
Dallas TX
X
Delray Beach
FL
X X
Evansville
IN
Flint
MI
Fresno CA
X
Hamilton OH
X
Houston
TX
Indianapolis IN
X X X
LaFourche Parish
LA
Las Vegas
NV
X
Enhanced Performance Measure for the Community Capacity Development Office
11
Site city
State
UI received data Multiple sites PMIP pilot site
Lowell
MA
X
Maine ME
X X
Miami FL
X X X
Mobile
AL
X
Nassau County
NY
X
New York
NY
North Miami Beach
FL
X
Ocala
FL
X
Ogden City
UT
Philadelphia PA
X X X
Phoenix
AZ
Pittsburgh PA
X X X
Portland OR
X
Sacramento
CA
X X
Saint Louis
MO
San Antonio
TX
X
San Francisco
CA
Seattle WA
X X X
Suffolk County
NY
X
Toledo OH
X X
Washington
DC
X
Village of Hempstead
NY
X
UI = Urban Institute; PMIP = Performance Measurement Indicators Project
Note: An ‘X’ in the ‘Multiple sites’ column indicates that there were multiple Weed and Seed target areas within the
jurisdiction.
The first task was to examine the differences in site performance that accrued dependent
on the type of measure used (i.e., homicide counts as compared with crime rates). For the pilot
sites noted above, we calculated changes from 1999 to 2000, from 2000 to 2001, and then from
2001 to 2002 in both the number of homicides and the overall and individual category crime
rates.
We assigned each of the 16 sites one ordinal ranking for each of the performance
measures being compared. For each performance measure, we assigned lower-ranking values to
sites with the greatest observed decreases in the measure. That is, the site with the greatest
reduction in the measure was ranked first; the site with the least reduction (or largest increase as
was often the case) was ranked sixteenth. Lower-ranking values correspond to greater salutary
changes in the performance measure. We further subdivided the ranking values into quartiles,
grouping the top four performing sites into the first quartile, the next four best sites into the
second quartile, and so on, to further simplify the comparison of the performance measures.
Enhanced Performance Measure for the Community Capacity Development Office
12
Figure 1 compares the site rankings that result from using violent crime rates to those using
homicide counts.
The sites are numbered to avoid identifying the jurisdictions.
The site rankings for relative change from 2001 to 2002 vary rather dramatically between
the rank that results from changes in simple homicide counts and the rank that results from
changes in the violent crime rate. For the sixteen sites for which changes in both homicide counts
and the violent crime rate were calculated, two sites were in the top quartile on both measures.
Five sites stayed in the same quartile, four sites were ranked higher on change in violent crime
than on change in homicide counts, and seven sites were ranked lower on change in violent
crime rates than on change in homicide counts.
Figure 1. Comparison of Site Rankings on Changes in Homicide Counts and Violent Crime Rates, 2001–2002
Site
number
Change in
homicides
Change in
violent crime
rate (per
1,000)
Rank:
Change in
homicides
Rank:
Change in
violent crime
rate
Quartile:
Change in
homicides
Quartile:
Change in
violent crime
rate
1
4
-1.68
12
4
3
1
2
-9
-2.80
1
3
1
1
3
5
-0.36
16
8
4
2
4
3
-0.86
11
7
3
2
5
-6
-5.18
2
2
1
1
6
4
-0.05
12
10
3
3
7
0
2.80
9
13
3
4
8
-1
-9.20
6
1
2
1
9
-6
-1.58
2
5
1
2
10
-6
-0.33
2
9
1
3
11
4
2.17
12
12
3
3
12
0
18.02
9
14
3
4
13
4
19.59
12
15
3
4
14
-2
38.38
5
16
2
4
15
-1
-1.06
6
6
2
2
16
-1
0.83
6
11
2
3
Although not directly included in the OMB PART reporting schema (which simply
requires the specification of a single overall target for crime reduction going forward), a
comparative examination of individual sites both to each other and to the overall national
average will enable CCDO managers to develop a more complete understanding of the effects of
the Weed and Seed program. Figure 2 provides an example with the changes in the rates of
violent crimes between 2001 and 2002. This graph suggests the following information:
More than half of the sites reduced their rates of violent crime.
Although no sites reduced crime rates by more than 10 crimes per 1,000 residents, three
sites experienced an increase of more than 10 crimes per 1,000 residents.
11
To provide a crime rate for a type of crime most similar to homicide, we restrict this example to the violent crime rate and homicide counts.
Enhanced Performance Measure for the Community Capacity Development Office
13
Most sites saw violent crime rates fall, but the collective population of all sites experienced
an increase in violent crime rates
This increase was driven largely by a few sites and was not a general pattern of increasing
violence.
Being able to understand the distribution surrounding a single number (required by the PART
form) will enable CCDO to provide valuable contextual discussion for each year’s examination
of the degree to which a target was (or was not) met.
Similar graphs can be created and analyzed for various other forms of disaggregated or
aggregated crime categories.
Figure 2. Performance of Individual Sites Compared with Overall Program Performance:
Changes in Violent Crime Rate, 2001-2002
-20
-10
0
10
20
30
40
50
Site
1
4
Site
1
3
Site
1
2
Site
7
Site
1
1
Site
1
6
Site
6
Site
1
0
Site
3
Site
4
Site
1
5
Site
9
Site
1
Site
2
Site
5
Site
8
Change i
n
number of vi
ol
ent cri
m
es per 1,
000 resi
dents
Median site
Note: The brown shaded area (above the horizontal axis) and green shaded area (below the horizontal axis) represent,
respectively, the magnitude of the increase or decrease in violent crime rate in each site from 2001 to 2002.
CONCEPTUAL INTEGRATION WITH ANNUAL PERFORMANCE PLANNING
This report presents an indicator—changes in the crime rate over time—that has been
designed to become part of the CCDO’s annual performance plans and progress reports. Because
this measure incorporates a variety of crime types and can be either aggregated or disaggregated
as desired, it meets the site need for a measure that is broader and more linked to site activities
Enhanced Performance Measure for the Community Capacity Development Office
14
than homicide. Because it uses site population in its calculus, it accounts for the variation in site
size. It also permits cross-site comparisons as well as the ability to aggregate to the national
level. We have recommended that CCDO begin using the crime rate as part of its set of formal
performance measures. We now discuss what new data would need to be collected and the issues
that will need to be addressed for CCDO to use crime rate for performance measurement.
Recommendation
At present, Weed and Seed sites ask the local police departments to report the number of
homicides that occurred within the site boundaries during each calendar year of the grant. The
sites, in turn, report this annual homicide count to CCDO using GPRA reporting forms provided
by JRSA. In short, the sites report only one performance measurement number per year to the
CCDO. This keeps the reporting burden for the Weed and Seed program agreeably low, but it
also restricts the ability of the CCDO to assess the relative performance of the sites.
UI recommends that the Weed and Seed sites be required to report monthly counts of
several categories of serious offenses, using the same definitions and reporting forms used by the
approximately 17,000 police agencies participating in the FBI’s UCR crime reporting program.
Since the police departments are already categorizing, counting, and reporting crimes for the
FBI, the CCDO can significantly increase the quantity and richness of the performance
measurement data available from the Weed and Seed sites without significantly increasing the
reporting burden imposed on the sites.
To realize the cross-site comparability of the performance measures that the CCDO
desires, police departments in the sites must categorize and report similar criminal incidents in
similar fashion. The challenge is to develop a set of offense categories and operational
definitions that can be uniformly applied in jurisdictions nationwide. Fortunately, the UCR
system has already accomplished this formidable task. Most police departments in the United
States are familiar with the UCR offense categories and definitions, and approximately 17,000
departments already routinely categorize and report incidents to the FBI’s UCR system. For
those police departments that are not familiar with the UCR, the definitions and guidelines for
reporting complex cases (e.g., those involving multiple types of offenses or multiple law
enforcement agencies) are readily accessible.
UI recommends that the CCDO revise question 3 of Section C of its GPRA reporting
form. Currently, that question requires Weed and Seed sites to report the number of homicides
that occurred during the three preceding calendar years within the boundaries of the Weed and
Seed site and within the surrounding jurisdiction.
UI recommends that sites be required to
report the number of actual (i.e., not unfounded) Part I offenses (i.e., criminal homicides,
forcible rapes, robberies, aggravated assaults, burglaries, larceny-thefts, and motor vehicle thefts)
12
Federal Bureau of Investigation. Uniform Crime Reporting Handbook. Revised 2004. Washington, DC: U.S. Department of Justice.
http://www.fbi.gov/ucr/handbook/ucrhandbook04.pdf.
13
Some sites may find it difficult to identify boundaries of a surrounding jurisdiction for reporting. The GPRA reporting form currently in use
requires reports of crime data from a surrounding jurisdiction, however, and we believe that this information would be useful for the performance
measurement system UI recommends. For example, crime data from surrounding jurisdictions could be compared with the crime data from the
site to examine whether crime trends within the site diverged—after the start of Weed and Seed activities—from what would have been expected.
Enhanced Performance Measure for the Community Capacity Development Office
15
during each month of the three preceding calendar years.
The sites would report counts of each
of these seven types of offenses for each of two geographic areas: the Weed and Seed site itself
and the surrounding jurisdiction as a whole. The instructions for the revised question 3 should
state that all of the offense counts should be developed in accordance with UCR guidelines and,
wherever possible, should be taken from column 4 of the UCR Return A reporting forms
submitted by the police agencies in the area.
RATIONALE
If this recommendation were adopted, in each annual reporting cycle, CCDO would
receive up to 36 offense counts for each of seven types of offense in each of two geographic
areas. The first report from each Weed and Seed site would include at least 24 months of data
from before the grant was awarded. Furthermore, since the current GPRA reporting form already
asks for population estimates for each site, the monthly offense counts could be easily converted
to a crime rate to compare changes across sites or to assess the collective performance of the
Weed and Seed sites in satisfaction of PART requirements.
The additional performance measurement data that UI recommends would yield
significant gains in the capability of CCDO to monitor the performance of the Weed and Seed
sites.
Many Weed and Seed sites experience periods of quiescence in their suppression and
prevention activities, and levels of crime may rise and fall in response. Based on annual
homicide counts alone, a pattern of ebb and flow in activity that is paralleled in the crime
patterns could be misinterpreted as a failed program model. With monthly counts of more
common crimes available, CCDO and the site coordinator could correctly identify the
service delivery problem rather than blaming the design of the prevention or suppression
activities.
Some Weed and Seed sites begin their activities midway through a calendar year. Having
monthly, rather than annual, offense data would permit CCDO to more precisely mark the
beginning of activities in the trend lines.
Weed and Seed sites sometimes select a mix a program activities that may be expected to
have a beneficial effect on the reduction of offenses other than homicide. Collecting
performance data on a wider variety of offenses would permit CCDO to set targets that are
more closely matched to each site’s activities.
With the cooperation of the FBI, the CCDO could validate the offense counts it receives on
the GPRA form against the UCR data from the jurisdictions.
Data received on offenses that occurred prior to the beginning of activities could be used to
establish the baseline level of crime in the site. This baseline would be useful for setting targets
and establishing the counterfactual. For example, the baseline data could be transformed using a
common smoothing technique (e.g., moving averages or exponential smoothing) to predict what
crime levels might be expected in the first 6–12 months of the grant period if no weeding or
seeding activities occurred. Those predicted crime levels would represent the counterfactual. A
target crime level could be set at some level below the predicted level.
14
Although arson is a UCR Part I offense, we have excluded it from our recommendation. Few Weed and Seed sites focus their prevention or
suppression efforts on arson.
Enhanced Performance Measure for the Community Capacity Development Office
16
Site-Level Implementation Issues
If UI’s recommendation were proposed for adoption, most of the site-level
implementation issues would focus on the reporting burden and the implications of the revised
PMIP for decisions about the renewal of funding.
RESTRICTING OFFENSE REPORTING TO THE SITE BOUNDARIES
Many large, urban police departments already use Geographic Information Systems
(GIS) software to map their incidents. A growing number of medium-sized police departments
are also finding GIS software to be a useful tool. Smaller police departments may find that
officers can easily identify the relevant incidents by manually reviewing the incident locations.
The problem of how to limit offense reporting to the site boundaries is an issue in the current
GPRA reporting form as well. If UI’s recommendation were adopted the issue would become
more formidable since a jurisdiction with a dozen homicides a month is likely to have a few
hundred thefts, several dozen burglaries, and so on. CCDO should not mandate the approach
sites should use to restrict their reporting to incidents within their boundaries. CCDO may be
able to offer recommendations or referrals to sites in need of a solution to this problem, however.
MULTIPLE OR OVERLAPPING POLICE JURISDICTIONS
For Weed and Seed sites that span jurisdictional boundaries, the site coordinator should
collect offense counts from each police department that polices part of the site. The coordinator
would then aggregate these counts to prepare the GPRA reporting form. The UCR reporting
system has guidelines for ensuring that police departments with overlapping jurisdictions do not
report the same incident.
The site coordinator would be responsible for ensuring that the police
departments operating within the site boundaries followed those guidelines to avoid double
counting. This potential for double counting is a concern on the current GPRA reporting form as
well. UI’s recommendation to expand offense reporting to include more common types of
offenses (e.g., larceny) makes the potential for double counting a larger issue.
REPORTING INCIDENTS INVOLVING MULTIPLE OFFENSE TYPES
The UCR reporting system developed a hierarchy rule for use in multiple-offense
incidents. Under the rule, each incident is recorded only once regardless of the number of
offenses involved, and the offense type recorded is that of the most serious offense committed
during the course of the incident.
REPORTING INCIDENTS INVOLVING MULTIPLE VICTIMS
Some criminal incidents involve more than one victim. The UCR system has a long-
standing rule about how many offenses should be counted in multiple-victim incidents. The rule
is based on a distinction between crimes against persons and crimes against property:
In the UCR Program, the offenses of criminal homicide, forcible rape, and aggravated
assault are crimes against the person. For these crimes, one offense is counted for each
victim. Robbery, burglary, larceny-theft, motor vehicle theft, and arson are crimes against
15
FBI. Uniform Crime Reporting Handbook, 8.
16
For more information on the hierarchy rule, see FBI. Uniform Crime Reporting Handbook, 10.
Enhanced Performance Measure for the Community Capacity Development Office
17
property. For these crimes, one offense is counted for each distinct operation or attempt,
except in the case of motor vehicle theft for which one offense is counted for each stolen
vehicle and one offense for each attempt to steal a motor vehicle.
Weed and Seed sites should observe this rule when preparing their offense counts.
ESTABLISHING THE STARTING DATE FOR SETTING PERFORMANCE TARGETS
Some Weed and Seed sites receive funding in one month but are granted “Official
Recognition” (OR) status one or more months later. The question is whether the performance
targets should be established relative to the months prior to the receipt of funding or relative to
the months prior to when the site reached OR status. Weeding and seeding activities cannot be
expected to reduce crime until after they actually begin, of course, and this would seem to
suggest setting the targets based on the date sites achieve OR status. On the other hand, CCDO
may wish to create an incentive for sites to minimize the length of the start-up period between
the receipt of funding and the beginning of program activities. Setting targets relative to funding
dates would create such an incentive.
If UI’s recommendation to collect monthly offense counts were adopted, the issue of the
reference date for the performance targets would be less acute. Using monthly, rather than
annual, performance measurement data makes it possible to set the reference date closer to the
actual date on which the site began its activities.
SETTING PERFORMANCE TARGETS FOR SITES FOCUSED ON LESS SERIOUS OFFENSES
The UCR Part I offenses included in UI’s recommendation include many of the most
serious types of offenses. Some Weed and Seed sites focus their attention on less serious
offenses, especially quality-of-life crimes (e.g., vagrancy, loitering, aggressive panhandling) . In
UI’s assessment, including direct measures of these less serious types of offenses in a
performance measurement system would be problematic for two reasons. First, for the less
serious Part II offenses the UCR program collects counts of arrests rather than incidents. Second,
arrest counts are problematic performance indicators because changes in police manpower or
arrest practices may influence them, even in the absence of changes in the underlying crime rate.
The UCR program collects arrest counts on both Part I and Part II offenses, but incident
(or offense) counts are requested only for Part I crimes. There are many more Part II incidents
than Part I incidents in a given jurisdiction, so reporting counts of Part II incidents would be a
significantly more burdensome task than reporting counts of Part I incidents alone. In addition,
many less serious Part II offenses are not reported to police. For many types of crime, the
common understanding is that the validity of an offense count is inversely proportional to the
seriousness of the offense as perceived by the public.
Arrest counts are problematic performance indicators because it is difficult to draw
inferences about site performance from changes in arrests over time. An increase in arrests may
indicate an increase in the vigor of suppression activity or a weakening of prevention activities or
community capacity. Conversely, a decrease in arrests may indicate an increase in the site’s
resistance to crime or a decline in the effectiveness of suppression activities.
17
Ibid., 41.
Enhanced Performance Measure for the Community Capacity Development Office
18
For these reasons, including Part II offenses in the performance measurement system
would mean either substantially increasing the reporting burden imposed on Weed and Seed sites
or incorporating the less interpretable arrest data into the performance measurement system. At
this time, UI’s view is that only the Part I offense counts should be collected for use in
performance measurement. Sites that focus on reducing less serious types of offenses may object
that the goals of their sites will not be addressed by examination of Part I offenses alone.
However, the theory of change that underlies a focus on quality-of-life offenses holds that the
prevalence of less serious offending establishes conditions favorable for more serious offending,
including Part I crimes. In this sense, the Part I offenses are defensible indirect measures of the
performance of all Weed and Seed sites, including those sites that have chosen to focus on
quality-of-life offenses.
INTEGRATION INTO SITE-LEVEL MANAGEMENT
CCDO should require Weed and Seed sites to set performance targets in each application
for funding. The process of setting targets is likely to be an iterative one. Some sites would
propose targets that are unrealistic or too modest, and CCDO would suggest revisions based on
its experience with other sites. At the conclusion of each year, site-level managers would review
their crime rate reduction targets to determine if the targets had been met. If the targets were not
met, the managers would review the programmatic activities and inputs to determine what might
need to be addressed to achieve the targets. If the managers were unable to identify an internal
programmatic explanation for the lack of achievement, they would examine contextual factors
that might account for the failure to meet the targets. In their application for continued funding,
mangers of sites that fell short of their targets would explain what remedial action they would
take to meet their targets in the following year. Reviewers at CCDO would consider site
performance, targets, and (for sites that did not meet their targets) remedial action plans in
making funding decisions.
National-Level Implementation Issues
There are three levels in which performance measurement should be integrated into
national program management. The first is in the annual site-funding process conducted by
CCDO, the second is overall programmatic monitoring, and the third is the PART reporting from
CCDO to OMB for the national budgeting process. Each of these is discussed in turn.
INTEGRATION INTO ONGOING SITE-LEVEL FUNDING
To receive continued funding, or to apply for new funding, each site applies to CCDO
annually. The program narrative section of the application requires that sites provide information
on their current levels of drug and violent crime and specify their strategy for both weeding and
seeding activities. Each strategy section specifies goals, activities, and outcome measures.
The application process, then, provides an opportunity for sites to develop program logic
models linking proposed weeding and seeding activities to performance measures and targets.
CCDO program managers could assess the internal validity of the applications, that is, the extent
to which the specified activities are likely to have the expected effects on the desired outcomes.
For example, an application with face validity would be one where an increase in bike patrols by
police (the activity) is expected to yield a 5 percent reduction in the robbery rate in a city park
Enhanced Performance Measure for the Community Capacity Development Office
19
(the performance measurement target). CCDO could use its experience to advise sites about
setting appropriate targets. This level of review should become part of the review process for
both initial and continuation funding.
COMPUTATION OF NATIONAL SCORES
To quantify the crime situation of a particular site in manner that permits comparison with other
sites, we have suggested computing a crime rate. For a site k at time t the crime rate, CR
kt
, may
be expressed as:
kt
kt
kt
P
C
CR
=
(1)
where
C
kt
is a count of crimes in site
k at time t and P
kt
is the residential population of site
k at
time
t. Equation 1 will yield the number of crimes per resident. The result may be multiplied by a
constant (e.g., 1,000) to rescale the values as desired as we did in our examples. If a measure of
within-site change is desired, the following formula will yield the change in the crime rate from
time
t
0
to time
t
1
:
0
0
1
kt
kt
kt
k
P
C
C
CR
−
=
Δ
(2)
which, like equation 1, could be multiplied by a constant to rescale the result. Equation 2 may be
applied to each of a number of sites to compute standardized estimates of the magnitude and
direction of crime change, which would be useful for comparing crime changes between sites.
Note that in the interest of simplicity, equation 2 utilizes a single population estimate at time
t
0
,
implicitly assuming that the population change between
t
0
and
t
1
is trivial. This assumption is
more plausible the shorter the interval between
t
0
and
t
1
.
A somewhat different approach is required to create a performance indicator for the Weed and
Seed program a whole, which reflects the contributions of all the sites. A number of measures
could reasonably be used, but the one we prefer reflects the mean percentage change in the crime
rate from time
t
0
to
t
1
(
μ
PCR
), which is equal to the mean of
Δ
CR
k
/CR
kt0
across
k sites. The
formula may be simplified through transformation as follows:
∑
∑
∑
=
=
=
⎥
⎥
⎦
⎤
⎢
⎢
⎣
⎡
⎟
⎠
⎞
⎜
⎝
⎛
⎟
⎟
⎠
⎞
⎜
⎜
⎝
⎛
−
=
⎥
⎥
⎦
⎤
⎢
⎢
⎣
⎡
⎟
⎠
⎞
⎜
⎝
⎛
⎟
⎟
⎠
⎞
⎜
⎜
⎝
⎛
⎟
⎟
⎠
⎞
⎜
⎜
⎝
⎛
−
=
⎥
⎥
⎥
⎥
⎥
⎦
⎤
⎢
⎢
⎢
⎢
⎢
⎣
⎡
⎟
⎠
⎞
⎜
⎝
⎛
⎟
⎟
⎟
⎟
⎟
⎟
⎠
⎞
⎜
⎜
⎜
⎜
⎜
⎜
⎝
⎛
⎟
⎟
⎠
⎞
⎜
⎜
⎝
⎛
−
=
k
k
kt
kt
kt
k
k
kt
kt
kt
kt
kt
k
k
kt
kt
kt
kt
kt
PCR
k
C
C
C
k
C
P
P
C
C
k
P
C
P
C
C
1
1
1
1
1
1
0
0
1
0
0
0
0
1
0
0
0
0
1
μ
(3)
Equation 3 weights the contribution of each site equally regardless of population. As a
performance measure,
μ
PCR
reflects the crime change experienced in the typical
site. In many
Enhanced Performance Measure for the Community Capacity Development Office
20
cases, this will be an acceptable program-level performance measure. Since it is standardized as
a percentage change, it may used to make comparisons between groups of sites (e.g., by
computing
μ
PCR
for sites in the west and comparing with
μ
PCR
for sites in the east) or between all
sites over time (e.g., by comparing
μ
PCR
for sites active in 2003 with
μ
PCR
for sites active in
2004).
For some purposes, however, it may be desirable to compute a program-level performance
measure that takes into account some of the important differences between sites. In that case, a
weighted mean percentage change in crime rate (
w
μ
PCR
) could be computed. For example, the
following formula would compute
w
μ
PCR
so that each site contributed to the mean in proportion
to its residential population:
∑
∑
=
=
⎥
⎥
⎦
⎤
⎢
⎢
⎣
⎡
⎟
⎟
⎠
⎞
⎜
⎜
⎝
⎛
−
=
1
1
0
0
0
0
1
k
k
kt
k
k
kt
kt
kt
kt
PCR
P
P
C
C
C
w
μ
(4)
The resulting value of
w
μ
PCR
would reflect the crime change experienced by a typical
resident of
a Weed and Seed site. This measure might, therefore, be more reflective than
μ
PCR
of changes in
public safety across all Weed and Seed sites between times
t
0
and
t
1
.
INTEGRATION INTO NATIONAL PROGRAM MANAGEMENT
UI recommends that, in addition to reviewing each site’s performance data as part of the
funding process, CCDO program managers should also review annual graphs of all sites’
performance relative to the performance of the program as a whole (e.g., by using the mean
percentage change in crime rate [
μ
PCR
] as a program-level performance measure. This
information would allow CCDO to determine whether the majority of sites are outperforming the
program as a whole. If so, then CCDO program managers should feel relatively confident that
the overall programmatic approach is performing as desired. However, if a majority of sites were
underperforming the program as a whole, then it would behoove CCDO managers to reconsider
various aspects of the program’s approach. Perhaps certain approaches should not be left to site
discretion or sites should receive additional technical assistance.
INTEGRATION INTO PART REPORTING
As stated in PART training performed by OMB staff, PART provides a framework for
targeting and designing program improvements by linking program actions to intended
outcomes. PART also identifies program strengths and weaknesses by highlighting both
successful and unsuccessful links between activities and outcomes. The reporting logic OMB
outlines parallels the programmatic logic model that is in place for the individual sites when
developing their funding request, where the performance measures are connected to the goals
and activities of the specific program.
For PART reporting, CCDO should consider two additional issues:
1.
What is the best approach to set specific PART targets?
2.
What is a realistic target over a longer period of time (e.g., 5 years)?
Enhanced Performance Measure for the Community Capacity Development Office
21
The setting of programmatic targets should not be done in isolation and without regard to
extant crime trends. Weed and Seed program managers at CCDO should examine national trend
lines from UCR data in developing their PART targets. For example, if the rate of property crime
has been increasing rapidly over each of the past several years, the Weed and Seed program
would be unlikely to realize a 20 percent reduction for the next budget year. Without an
understanding of the context within which an expected outcome is to occur, unrealistic PART
targets may be set, and the program will not be able to meet them.
The second issue—the degree to which reductions in crime rate can be achieved year
after year—also must be considered. Crimes will always occur, regardless of the degree to which
programmatic interventions are put into place. Is it realistic to expect CCDO to demonstrate a
continued percentage decrease change over time, or is there a natural bottom? CCDO managers
need to consider whether it is appropriate to set targets that are percentage reductions year after
year. A more realistic approach might be to develop asymptotic targets that represent a
stabilization of the crime rate at a low level relative to the recent past. These targets could then
be linked to programmatic activities as appropriate.
ESTABLISHING THE COUNTERFACTUAL: TARGETED RESEARCH STUDIES
The only way that causal claims can be made is through the computation and use of
reliable counterfactuals. This can only be addressed by widening the information-gathering
design. Using the reduction in crime rates, then, complete reporting would involve not only
setting a specific performance target for upcoming years but also developing a methodology to
permit the examination of the degree to which the reduction in crime rate in a site is different
from comparison areas or different from that which was expected in the site itself based on the
experience of past years. Designs for a reporting approach could use the changes in crime rate
for either of the following comparisons:
A cross-sectional comparison of the changes in crime rates in the Weed and Seed site to
changes in the surrounding jurisdiction;
A longitudinal comparison of changes in crime rate in the Weed and Seed site (using as
many years of data as feasible), using interrupted time series to determine if the trend after
Weed and Seed implementation diverges from what would have been expected given
historical trends.
While these kinds of analyses go beyond the scope of the typical OMB or GPRA reporting, they
should be considered in some set of sites to establish the degree to which the performance goals,
as developed by a given federal agency (e.g., CCDO), make conceptual sense and are attainable.
Once the performance measurement system is instituted into the site-level GPRA
reporting system, examination of these reports can help to identify those specific Weed and Seed
sites that may merit closer analysis or even formal evaluation. For example, CCDO could
highlight those sites with an unexpected decrease in a specific type of crime and examine the
specific programmatic activities that had been put into place for that year. At that point, further
analysis might compare the site’s crime reduction to crime trends in the larger jurisdiction or
place the initiation of program activities and the crime reduction in the context of a longer trend
within the site. This would help to determine if the performance outcome found in a given site
were likely linked to the programmatic activities. While this type of exploration requires
additional funds, it is only through such analysis that the value of performance measures—that of
Enhanced Performance Measure for the Community Capacity Development Office
22
being able to determine which among a host of intervention options is truly worth replicating—
can be realized.
Summary of Recommendations
The CCDO has a functional performance measurement system that permits it to monitor
the hundreds of Weed and Seed sites and comply with OMB PART requirements. In this report,
we have made several recommendations about how this performance measurement system could
be improved and expanded without imposing significant new burdens on the Weed and Seed
sites. UI recommends that CCDO do the following:
Require sites to report counts of each of seven types of UCR Part I incidents, not just
homicides as the GPRA form currently requires. These incident counts can be taken
directly from the UCR Return A forms completed monthly by thousands of police
agencies.
Advise sites to observe the rules of the UCR system for categorizing offenses, identifying
the most serious of several offenses in a single incident, counting incidents, and resolving
overlapping police jurisdictions.
Require sites to report monthly incident counts for the two years preceding their awards
and for each year of Weed and Seed funding.
Require sites to report counts of incidents and estimates of the residential population for
both the Weed and Seed site and the surrounding jurisdiction, just as the current GPRA
form requires. Having these population estimates would permit CCDO to compute crime
rates from the Part I incident counts to compute performance measures for the entire Weed
and Seed program that would be suitable for PART reporting.
Require site managers to set performance targets in each annual application for Weed and
Seed funding.
Require site mangers to review their performance vis-à-vis their targets at the end of each
funding year and report to CCDO on what, if any, adjustments they propose to make in
their application for continued Weed and Seed funding.
Incorporate a review of the targets, performance measures, and proposed adjustments from
each site into the process of making funding decisions.
Consider whether the mean percentage change in crime rate (mu_(PCR)) is a suitable
program-level performance measure.
Incorporate review of longer-term national crime trends into the development of OMB
PART targets.
Consider whether Weed and Seed PART targets should become asymptotic as crime rates
hover near historic lows rather than continuing to be a set percentage of the previous year’s
level.
Mine the richer performance measurement data the adoption of these recommendations
would yield to identify Weed and Seed sites that are consistently strong performers that
may merit formal evaluation.
Enhanced Performance Measure for the Community Capacity Development Office
23
By collecting monthly counts of Part I incidents and holding sites accountable for meeting their
performance targets (or making reasonable adjustments), CCDO can improve the performance of
its sites and the performance of the Weed and Seed program as a whole in the PART system.
Enhanced Performance Measure for the Community Capacity Development Office
24
File contents
Considerations for
Performance Measure
System
Mark Coggeshall
Terence Dunworth
January 20
2100 M Street NW
Washington, DC 20037
www.urban.org
© 2005 Urban Institute
This report was prepared under subcontract to the Justice Research and Statistics Association
(JRSA) and the U.S. Department of Justice, the Community Capacity Development Office
(CCDO). Opinions expressed in this document are those of the authors and do not necessarily
represent the official position or policies of JRSA or the U.S. Department of Justice, the Urban
Institute, its trustees, or its funders.
ACKNOWLEDGMENTS
The authors would like to thank the many individuals who have supported the first phase of the
Weed and Seed Performance Measurement Indicators Project (PMIP). At the Urban Institute, we
thank Avi Bhati, Caterina Gouvis Roman, Gretchen Moore, and Sinead Keegan (formerly of UI),
and Kevin Roland and Michelle Scott, who supported and conducted much of the necessary data
preparation and analysis; Elizabeth Cincotta and Deborah Chester, who provided editorial
assistance; and Harry Hatry, Adele Harrell, and John Roman, who reviewed drafts of this report
to provide feedback about the proposed measure and alternative measures. Joan Weiss, Jim
Zepp, Matt Perkins, and Jason Trask of the Justice Research and Statistics Association provided
input on the proposed Performance Measurement Indicators System and provided data from the
Weed and Seed Data Center. The support of the former acting director of the Community
Capacity Development Office, Bob Samuels, was immeasurable. The PMIP would not be
possible without the participation of many willing Weed and Seed sites across the country and
their local police departments, which generously supplied the PMIP team with an abundance of
detailed local crime data. Thanks to all.
and the Urban Institute (UI) initiated a review of the CCDO approach to performance
measurement. That performance measurement system was based on a single indicator: year-to-
year changes in homicide counts. The key consideration was to determine if enhancements could
be made that would move beyond the simple calculation of year-to-year changes in homicides
without unduly increasing the reporting burden for the sites. This review, then, provided an
opportunity to think broadly about how best to measure crime control strategies that would
account for variation in both crime prevalence and population affected, an interest that goes
beyond CCDO to any agency or policymaker seeking to understand the effect of crime control
programs.
measurement and performance management system in place within the executive branch. Then,
we describe our examination of alternative performance measures for the Weed and Seed
program. We conclude that crime rate, rather than counts of homicides alone, would better serve
the need of the CCDO to set performance targets for both individual sites and the Weed and Seed
program as a whole. In the final sections, we recommend changes to the existing Weed and Seed
performance measurement system as well as an approach to integrating the revised system we
propose into the management of the individual sites and of the entire Weed and Seed program.
achieve results…Good government—a government responsible to the people
whose dollars it takes to fund its operations—must have as its core purpose the
achievement of results…In a results-oriented government, the burden of proof
rests on each federal program and its advocates to prove that the program is
getting results…
[Program Assessment Rating Tool, or PART] to hold agencies accountable for
accomplishing results. Programs are rated from Effective to Ineffective, and the
ratings and specific findings produced will be used to make decisions regarding
budgets and policy. The tool assumes that a program that cannot demonstrate
positive results is no more entitled to funding, let alone an increase, than a
program that is clearly failing…
promised. What works is what matters, and achievement should determine
which programs survive, and which do not. The public must finally be able to
hold managers and policymakers accountable for results, and improve the
performance of federal programs in the way that Presidents and Congresses
have sought for decades. Whether resources shift over time from deficient to
good programs will be the test the PART itself must pass.
Compared with a scientifically sound evaluation, a PMIS is a more cost-effective way of
monitoring activities and expected outcomes. A PMIS can also help isolate approaches and
outcomes that appear worthy of further exploration through hypothesis testing (e.g., a program
that focuses on quality-of-life crimes showed a decline in drug arrests over time) and can help
guide the formulation of a research approach (e.g., the program was in a small rather than a large
jurisdiction). Finally, a PMIS can aid in program management: communities can monitor the
PMIS in other jurisdictions to identify best practices, and managers can use changes in measures
over time to determine whether the programmatic approaches have had the intended effects. If
they have not, then some changes are likely in order.
and activities that they believe are necessary to accomplish the desired outcomes (performance
measures). Further, the specific contextual factors that may influence program performance must
also be understood and addressed. Then, over time, the measures should be used as criteria to
determine whether a specific approach has met its stated goals. If the goals have not been met,
program managers can then review whether the necessary activities were conducted and whether
the appropriate inputs were available. This review can help specify where programmatic changes
might be necessary.
consecutive years. Period 1 served as a baseline, during which there were 150 robberies. During
period 2 there were 120 robberies (a 20 percent reduction from period 1), but during period 3
there were 160 robberies (a 33 percent increase from period 2). In a performance management
system, program staff managers would examine the degree to which their targets were met. For
period 2 the targets were met, but for period 3 they were not. Managers would then examine
whether the program’s activities had been adequately conducted (e.g., perhaps there had been a
staffing shortage) or whether there were changes in inputs (e.g., perhaps overall funding had
been cut). If nothing had changed in either the programmatic inputs or activities, perhaps there
administration.
entails different managerial decisions. Primarily, it can be used by CCDO to assess individual
sites’ performance as one criterion for continued funding. It can also be used to monitor the
relationship between various programmatic activities and outcomes, allowing for a preliminary
understanding of promising program practices. These promising practices, then, are the most
likely candidates for evaluation.
be directly attributed to the specific programmatic intervention will not be known if performance
measures are collected only from the Weed and Seed sites themselves during the period of
program activity. Without establishing some type of comparison, either to the site itself between
the pre– and post–Weed and Seed periods or to some other comparison area, the degree to which
the outcome (i.e., performance measure) occurred elsewhere without programmatic intervention
will remain unmeasured.
change in the measure for a given site is different from comparison areas or different from
expectations based on observations from past years. Designs for a reporting approach could use
either of the following comparisons:
determine if the trend shifts in the expected direction after Weed and Seed implementation
of sites to establish the degree to which the performance goals, as developed by a given federal
agency (here, the CCDO), continue to be logically linked to programmatic efforts and are
attainable. A performance management system can be used to determine which sites to choose.
would show an initial decrease and then stability. The issues of how best to measure change and set performance targets over time are not easy for
any program and require serious consideration.
evaluation.
Crime Rate
submitting applications for review by the CCDO. Successful sites each receive the same base
award of federal funds. The amount—ranging from $175,000 to $225,000 per annum with
possible renewal for up to four years—is modest compared with most federal programs that
provide direct support to state and local agencies. The award can be supplemented to a modest
degree through special emphasis programs that the CCDO operates. Not all sites receive
supplementary funds, and those that do rarely get more than an additional $50,000. In 2005, 340
sites had federal funding.
implement social programs that will make the Weed and Seed area more resistant to crime (i.e.,
“seeding” activities). Sites are instructed to devote approximately equal parts of their federal
funding to weeding and seeding activities. Sites are also required to develop ways of sustaining
the initiatives they have launched after their federal support ends.
program. Continuation of federal support to each site is contingent upon satisfactory progress,
which the CCDO determines. Further, the CCDO has its own performance reporting
requirements to the OMB and the Justice Department. The CCDO must demonstrate to OMB
and the Justice Department the progress of the entire Weed and Seed strategy toward its national
goals.
CCDO a way to manage the sites through aggregating site-specific information to produce a
national portrait. These objectives need to be met on both the enforcement and social program
aspects of the Weed and Seed program, but because the activities between the components differ
too greatly, no single performance measurement system will be suitable for both components. In
this report, we focus on the crime reduction and control mandate, not the social program
mandate.
during which change in the goal would be measured. Note that the period can be any length—
one month, one year, three years, and so on. A site would be judged as a success or failure based
upon the achievement of the measure from period 1 to period 2.
robberies were 50 in period 1 and 40 in period 2, assuming the two periods are the same length
and the counts are derived from the same geographic area, the unadjusted count approach
indicates that there has been a decline of 10 incidents. If the goal for the site had been a period-
to-period decline of five incidents, it was met. If the goal had been a decline of 20 incidents, it
was not met. More problematic, of course, would be the situation in which reported robberies
were higher in period 2 than in period 1, or in which there is no stated goal. Presumably,
however, consistent failure to meet a goal would be considered poor performance and might
result in termination of federal funding.
change in homicides has two primary benefits. First, homicide is the most validly reported type
of crime. Relative to other types of crime, few homicides go undetected or unreported for long.
Second, homicide is uniformly recorded. However, for CCDO, this measure has several pitfalls.
sites frequently begin with such issues as quality-of-life crimes (e.g., vagrancy, loitering,
aggressive panhandling) that have little direct relationship to homicide. A superior performance
measure would allow for a larger set of crimes to be examined. In addition, the typically low
incidence of homicides, even in urban environments, means that it is not uncommon to observe
zero homicides in a community in a given period. Although it is desirable for a community to
have zero homicides, it makes homicide a problematic performance measure because the number
of homicides cannot drop below zero. A more useful performance measure would be sufficiently
common that it would rarely reach its truncation point.
would account for differences between sites in population and crime prevalence. For example,
given that Weed and Seed sites all receive similar funding levels yet vary widely in population, it
would be beneficial if the performance measure differentiated between sites that have a few
thousand residents and those that have tens of thousands.
more dramatic than when a larger site with a higher prevalence of crime achieves the same
reduction of 10 crimes.
site has a similar federal support level from Weed and Seed.
it would account for the counterfactual (i.e., what changes would have been observed in the
outcomes if the Weed and Seed strategy had not been in operation in that site). However, such a
comprehensive approach is not practical, except perhaps in a very limited number of sites,
because of issues surrounding study design and implementation. A practical reporting approach
should be easy to operate and update and should be able to be maintained independently by
Weed and Seed sites. To be practical, the performance indicator should be all of the following:
require complicated, burdensome computation.
Otherwise, the measure cannot be used to assess operations objectively.
support elaborate and expensive analyses for site evaluation. A performance indicator needs
to be effective while not imposing a high cost on the local sites.
current information with ease, thereby allowing for ongoing progress reports. Sites would
then be able to more readily identify where programs were effective and also where resources
should be diverted.
to measure its own performance while supporting national assessments of the Weed and Seed
program as a whole. Consequently, cross-site standardization is essential.
measures: a weighted crime count approach, a social benefit index, and simple crime rate. All of
these approaches account for variation in crime prevalence and population over time.
the site, whether the type of offense is a CCDO focus area, whether the type of offense was a
focus area for the site, and the contribution of the site to the collective residential population of
all the sites. This weighted approach had two failings. First, it was multilayered and complex,
which rendered it difficult to understand. Second, the population weight was restricted to those
sites included in the performance measurement system.
rates in period 2 (i.e., the number of persons per crime in period 2 minus the number of persons
per crime in period 1).
than the same reductions in sites with smaller populations. That is, more people avoided probable
victimization in the former than in the latter from a similar drop in crime rates. Second, it was
easier to reduce crime in sites where initial prevalence was high than when it was low. The social
benefit index accounted for both the number of people affected and the size of the crime
problem. Nonetheless, because many people found the explanation of the social benefit index
difficult to grasp, we decided that this measure is not appropriate for policy use.
in an easily understood manner. It has the advantage of being easy to calculate, and it is
standardized so that it can be used to compute metrics to compare sites or to assess the
performance of the Weed and Seed program as a whole.
interested in grouping offense types conceptually (e.g., violence, property, disorder). However,
because the sites will be relying on reports of crime data from local police departments, UI’s
recommendations include adjusting the GPRA reporting form to incorporate Part I offenses of
the Uniform Crime Reports (UCR) system. Since many police departments already report counts
of Part I offenses to the Federal Bureau of Investigation (FBI), it would be possible to build an
enhanced performance measurement system for Weed and Seed based on the UCR offense
categories without significantly increasing the reporting burden on the sites (see
“Recommendation,” below for details about the changes we propose).
categories for this work were structured as shown in table 1:
4(1): 57-71.).
burdens on the sites, and we believe that use of UCR Part I offense categories would not.
presented here, crime rate, is calculated from counts of crimes and an estimate of site population.
defined, although the duration of each period must be the same across sites. In other words, the
measure is less sensitive to the actual start or end date of the intervals over which the change is
measured than it is to the duration of the interval itself.
counts by population to produce a rate is to have a standardized measure (i.e., crimes per person)
suitable for aggregation across sites. Since population estimates are error-prone, especially those
projected from the decennial U.S. Census estimates, using a time series of population estimates
as the denominator to compute site-level crime rates is likely to increase the amount of error in
the calculations. We believe, therefore, that the population should be used as a standardization
factor to compute a crime rate for performance measurement. It needs to be only approximately
correct, and its imprecision should be acknowledged. Any variability found in the rate over time
would then be confined to the directly observed counts of crime in the numerator.
crime types, such as the seven UCR Part I offense categories included in UI’s recommended
performance measurement system (see “Recommendation,” below). In addition, by summing
rate or a Part I violent crime rate).
disaggregated crime rate reductions, is not a simple summation of all the individual jurisdiction
crime rate reductions. Rather, it is a weighted sum of all the jurisdiction-specific crime rate
reductions where the weights are proportional to the population of a specific jurisdiction.
may be compared directly or displayed on the same figure. This ability provides a convenient
means of displaying various features of performance. For example, it is easy to identify sites that
performed the best or worst in a given year.
Performance for 16 Pilot Sites
to participate. More than 30 jurisdictions initially agreed, and UI has received data from 19
jurisdictions to date. Within those jurisdictions, UI designated 16 as pilot sites because these sites
supplied all the necessary underlying data. Data from 16 pilot sites have been analyzed for this
report, and detailed figures comparing sites and presenting trends are provided below. Table 2
displays the participating sites.
Note: An ‘X’ in the ‘Multiple sites’ column indicates that there were multiple Weed and Seed target areas within the
jurisdiction.
sites noted above, we calculated changes from 1999 to 2000, from 2000 to 2001, and then from
2001 to 2002 in both the number of homicides and the overall and individual category crime
rates.
sites with the greatest observed decreases in the measure. That is, the site with the greatest
reduction in the measure was ranked first; the site with the least reduction (or largest increase as
was often the case) was ranked sixteenth. Lower-ranking values correspond to greater salutary
changes in the performance measure. We further subdivided the ranking values into quartiles,
grouping the top four performing sites into the first quartile, the next four best sites into the
second quartile, and so on, to further simplify the comparison of the performance measures.
homicide counts.
changes in the violent crime rate. For the sixteen sites for which changes in both homicide counts
and the violent crime rate were calculated, two sites were in the top quartile on both measures.
Five sites stayed in the same quartile, four sites were ranked higher on change in violent crime
than on change in homicide counts, and seven sites were ranked lower on change in violent
crime rates than on change in homicide counts.
number
homicides
violent crime
rate (per
1,000)
Change in
homicides
Change in
violent crime
rate
Change in
homicides
Change in
violent crime
rate
comparative examination of individual sites both to each other and to the overall national
average will enable CCDO managers to develop a more complete understanding of the effects of
the Weed and Seed program. Figure 2 provides an example with the changes in the rates of
violent crimes between 2001 and 2002. This graph suggests the following information:
form) will enable CCDO to provide valuable contextual discussion for each year’s examination
of the degree to which a target was (or was not) met.
respectively, the magnitude of the increase or decrease in violent crime rate in each site from 2001 to 2002.
this measure incorporates a variety of crime types and can be either aggregated or disaggregated
as desired, it meets the site need for a measure that is broader and more linked to site activities
size. It also permits cross-site comparisons as well as the ability to aggregate to the national
level. We have recommended that CCDO begin using the crime rate as part of its set of formal
performance measures. We now discuss what new data would need to be collected and the issues
that will need to be addressed for CCDO to use crime rate for performance measurement.
sites, in turn, report this annual homicide count to CCDO using GPRA reporting forms provided
by JRSA. In short, the sites report only one performance measurement number per year to the
CCDO. This keeps the reporting burden for the Weed and Seed program agreeably low, but it
also restricts the ability of the CCDO to assess the relative performance of the sites.
approximately 17,000 police agencies participating in the FBI’s UCR crime reporting program.
Since the police departments are already categorizing, counting, and reporting crimes for the
FBI, the CCDO can significantly increase the quantity and richness of the performance
measurement data available from the Weed and Seed sites without significantly increasing the
reporting burden imposed on the sites.
similar fashion. The challenge is to develop a set of offense categories and operational
definitions that can be uniformly applied in jurisdictions nationwide. Fortunately, the UCR
system has already accomplished this formidable task. Most police departments in the United
States are familiar with the UCR offense categories and definitions, and approximately 17,000
departments already routinely categorize and report incidents to the FBI’s UCR system. For
those police departments that are not familiar with the UCR, the definitions and guidelines for
reporting complex cases (e.g., those involving multiple types of offenses or multiple law
enforcement agencies) are readily accessible.
that occurred during the three preceding calendar years within the boundaries of the Weed and
Seed site and within the surrounding jurisdiction.
forcible rapes, robberies, aggravated assaults, burglaries, larceny-thefts, and motor vehicle thefts)
measurement system UI recommends. For example, crime data from surrounding jurisdictions could be compared with the crime data from the
site to examine whether crime trends within the site diverged—after the start of Weed and Seed activities—from what would have been expected.
and the surrounding jurisdiction as a whole. The instructions for the revised question 3 should
state that all of the offense counts should be developed in accordance with UCR guidelines and,
wherever possible, should be taken from column 4 of the UCR Return A reporting forms
submitted by the police agencies in the area.
areas. The first report from each Weed and Seed site would include at least 24 months of data
from before the grant was awarded. Furthermore, since the current GPRA reporting form already
asks for population estimates for each site, the monthly offense counts could be easily converted
to a crime rate to compare changes across sites or to assess the collective performance of the
Weed and Seed sites in satisfaction of PART requirements.
sites.
homicide counts alone, a pattern of ebb and flow in activity that is paralleled in the crime
patterns could be misinterpreted as a failed program model. With monthly counts of more
common crimes available, CCDO and the site coordinator could correctly identify the
service delivery problem rather than blaming the design of the prevention or suppression
activities.
beginning of activities in the trend lines.
performance data on a wider variety of offenses would permit CCDO to set targets that are
more closely matched to each site’s activities.
establish the baseline level of crime in the site. This baseline would be useful for setting targets
and establishing the counterfactual. For example, the baseline data could be transformed using a
common smoothing technique (e.g., moving averages or exponential smoothing) to predict what
crime levels might be expected in the first 6–12 months of the grant period if no weeding or
seeding activities occurred. Those predicted crime levels would represent the counterfactual. A
target crime level could be set at some level below the predicted level.
PMIP for decisions about the renewal of funding.
are also finding GIS software to be a useful tool. Smaller police departments may find that
officers can easily identify the relevant incidents by manually reviewing the incident locations.
The problem of how to limit offense reporting to the site boundaries is an issue in the current
GPRA reporting form as well. If UI’s recommendation were adopted the issue would become
more formidable since a jurisdiction with a dozen homicides a month is likely to have a few
hundred thefts, several dozen burglaries, and so on. CCDO should not mandate the approach
sites should use to restrict their reporting to incidents within their boundaries. CCDO may be
able to offer recommendations or referrals to sites in need of a solution to this problem, however.
would then aggregate these counts to prepare the GPRA reporting form. The UCR reporting
system has guidelines for ensuring that police departments with overlapping jurisdictions do not
report the same incident.
counting. This potential for double counting is a concern on the current GPRA reporting form as
well. UI’s recommendation to expand offense reporting to include more common types of
offenses (e.g., larceny) makes the potential for double counting a larger issue.
offenses involved, and the offense type recorded is that of the most serious offense committed
during the course of the incident.
is based on a distinction between crimes against persons and crimes against property:
assault are crimes against the person. For these crimes, one offense is counted for each
victim. Robbery, burglary, larceny-theft, motor vehicle theft, and arson are crimes against
except in the case of motor vehicle theft for which one offense is counted for each stolen
vehicle and one offense for each attempt to steal a motor vehicle.
targets should be established relative to the months prior to the receipt of funding or relative to
the months prior to when the site reached OR status. Weeding and seeding activities cannot be
expected to reduce crime until after they actually begin, of course, and this would seem to
suggest setting the targets based on the date sites achieve OR status. On the other hand, CCDO
may wish to create an incentive for sites to minimize the length of the start-up period between
the receipt of funding and the beginning of program activities. Setting targets relative to funding
dates would create such an incentive.
annual, performance measurement data makes it possible to set the reference date closer to the
actual date on which the site began its activities.
offenses, especially quality-of-life crimes (e.g., vagrancy, loitering, aggressive panhandling) . In
UI’s assessment, including direct measures of these less serious types of offenses in a
performance measurement system would be problematic for two reasons. First, for the less
serious Part II offenses the UCR program collects counts of arrests rather than incidents. Second,
arrest counts are problematic performance indicators because changes in police manpower or
arrest practices may influence them, even in the absence of changes in the underlying crime rate.
than Part I incidents in a given jurisdiction, so reporting counts of Part II incidents would be a
significantly more burdensome task than reporting counts of Part I incidents alone. In addition,
many less serious Part II offenses are not reported to police. For many types of crime, the
common understanding is that the validity of an offense count is inversely proportional to the
seriousness of the offense as perceived by the public.
indicate an increase in the vigor of suppression activity or a weakening of prevention activities or
community capacity. Conversely, a decrease in arrests may indicate an increase in the site’s
resistance to crime or a decline in the effectiveness of suppression activities.
or incorporating the less interpretable arrest data into the performance measurement system. At
this time, UI’s view is that only the Part I offense counts should be collected for use in
performance measurement. Sites that focus on reducing less serious types of offenses may object
that the goals of their sites will not be addressed by examination of Part I offenses alone.
However, the theory of change that underlies a focus on quality-of-life offenses holds that the
prevalence of less serious offending establishes conditions favorable for more serious offending,
including Part I crimes. In this sense, the Part I offenses are defensible indirect measures of the
performance of all Weed and Seed sites, including those sites that have chosen to focus on
quality-of-life offenses.
propose targets that are unrealistic or too modest, and CCDO would suggest revisions based on
its experience with other sites. At the conclusion of each year, site-level managers would review
their crime rate reduction targets to determine if the targets had been met. If the targets were not
met, the managers would review the programmatic activities and inputs to determine what might
need to be addressed to achieve the targets. If the managers were unable to identify an internal
programmatic explanation for the lack of achievement, they would examine contextual factors
that might account for the failure to meet the targets. In their application for continued funding,
mangers of sites that fell short of their targets would explain what remedial action they would
take to meet their targets in the following year. Reviewers at CCDO would consider site
performance, targets, and (for sites that did not meet their targets) remedial action plans in
making funding decisions.
CCDO, the second is overall programmatic monitoring, and the third is the PART reporting from
CCDO to OMB for the national budgeting process. Each of these is discussed in turn.
on their current levels of drug and violent crime and specify their strategy for both weeding and
seeding activities. Each strategy section specifies goals, activities, and outcome measures.
CCDO program managers could assess the internal validity of the applications, that is, the extent
to which the specified activities are likely to have the expected effects on the desired outcomes.
For example, an application with face validity would be one where an increase in bike patrols by
police (the activity) is expected to yield a 5 percent reduction in the robbery rate in a city park
setting appropriate targets. This level of review should become part of the review process for
both initial and continuation funding.
sites, we have suggested computing a crime rate. For a site k at time t the crime rate, CR
within-site change is desired, the following formula will yield the change in the crime rate from
time
applied to each of a number of sites to compute standardized estimates of the magnitude and
direction of crime change, which would be useful for comparing crime changes between sites.
Note that in the interest of simplicity, equation 2 utilizes a single population estimate at time
Seed program a whole, which reflects the contributions of all the sites. A number of measures
could reasonably be used, but the one we prefer reflects the mean percentage change in the crime
rate from time
⎦
⎣
⎠
⎝
⎦
⎣
⎠
⎝
⎠
⎝
⎟
⎜
⎠
⎝
Equation 3 weights the contribution of each site equally regardless of population. As a
performance measure,
a percentage change, it may used to make comparisons between groups of sites (e.g., by
computing
For some purposes, however, it may be desirable to compute a program-level performance
measure that takes into account some of the important differences between sites. In that case, a
weighted mean percentage change in crime rate (
⎦
⎣
⎠
⎝
performance relative to the performance of the program as a whole (e.g., by using the mean
percentage change in crime rate [
program as a whole. If so, then CCDO program managers should feel relatively confident that
the overall programmatic approach is performing as desired. However, if a majority of sites were
underperforming the program as a whole, then it would behoove CCDO managers to reconsider
various aspects of the program’s approach. Perhaps certain approaches should not be left to site
discretion or sites should receive additional technical assistance.
outcomes. PART also identifies program strengths and weaknesses by highlighting both
successful and unsuccessful links between activities and outcomes. The reporting logic OMB
outlines parallels the programmatic logic model that is in place for the individual sites when
developing their funding request, where the performance measures are connected to the goals
and activities of the specific program.
lines from UCR data in developing their PART targets. For example, if the rate of property crime
has been increasing rapidly over each of the past several years, the Weed and Seed program
would be unlikely to realize a 20 percent reduction for the next budget year. Without an
understanding of the context within which an expected outcome is to occur, unrealistic PART
targets may be set, and the program will not be able to meet them.
programmatic interventions are put into place. Is it realistic to expect CCDO to demonstrate a
continued percentage decrease change over time, or is there a natural bottom? CCDO managers
need to consider whether it is appropriate to set targets that are percentage reductions year after
year. A more realistic approach might be to develop asymptotic targets that represent a
stabilization of the crime rate at a low level relative to the recent past. These targets could then
be linked to programmatic activities as appropriate.
design. Using the reduction in crime rates, then, complete reporting would involve not only
setting a specific performance target for upcoming years but also developing a methodology to
permit the examination of the degree to which the reduction in crime rate in a site is different
from comparison areas or different from that which was expected in the site itself based on the
experience of past years. Designs for a reporting approach could use the changes in crime rate
for either of the following comparisons:
Weed and Seed implementation diverges from what would have been expected given
historical trends.
should be considered in some set of sites to establish the degree to which the performance goals,
as developed by a given federal agency (e.g., CCDO), make conceptual sense and are attainable.
sites that may merit closer analysis or even formal evaluation. For example, CCDO could
highlight those sites with an unexpected decrease in a specific type of crime and examine the
specific programmatic activities that had been put into place for that year. At that point, further
analysis might compare the site’s crime reduction to crime trends in the larger jurisdiction or
place the initiation of program activities and the crime reduction in the context of a longer trend
within the site. This would help to determine if the performance outcome found in a given site
were likely linked to the programmatic activities. While this type of exploration requires
additional funds, it is only through such analysis that the value of performance measures—that of
can be realized.
we have made several recommendations about how this performance measurement system could
be improved and expanded without imposing significant new burdens on the Weed and Seed
sites. UI recommends that CCDO do the following:
directly from the UCR Return A forms completed monthly by thousands of police
agencies.
overlapping police jurisdictions.
form requires. Having these population estimates would permit CCDO to compute crime
rates from the Part I incident counts to compute performance measures for the entire Weed
and Seed program that would be suitable for PART reporting.
their application for continued Weed and Seed funding.
level.
may merit formal evaluation.
performance targets (or making reasonable adjustments), CCDO can improve the performance of
its sites and the performance of the Weed and Seed program as a whole in the PART system.
Document Outline
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
- ��
