Abstract
Environmental and natural resource litigation increasingly relies on economic valuations derived from benefit transfer and meta-analysis techniques. These methods are often considered legitimate shortcuts to estimating environmental economic damages when site-specific studies or data are unavailable; however, this acceptance may be misplaced. Benefit transfer and meta-analysis techniques are typically not sufficiently reliable to the extent that litigation demands. The fundamental mismatch is between the circumstances surrounding the source study and the case-specific facts to which the source study is being applied. The source locations for data points used as unit values rarely align with the destination site with sufficient comparability to support reliable damage estimates. This article examines the technical and legal dimensions of that mismatch and argues that courts should scrutinize these methods in environmental litigations.
What Is Benefit Transfer?
Benefit transfer was developed as a practical concession to the cost of scholarship. Conducting an original, site-specific valuation study is expensive, time-consuming, and requires specialized expertise. Analysts need ways to estimate non-market values without funding original research for every analysis. Benefit transfer attempts to fill that gap by allowing analysts to borrow estimates from existing studies conducted at other locations (the “study site” or “source site”) and apply them, perhaps with adjustments, to new analytical contexts and geographies (the “policy site” or “destination site”).
The technique arises in several forms (Desvousges et al. 1998; Vetter et al. 2013). Unit-value transfer or benefit-value transfer applies a point estimate from a source study directly to the case site, sometimes adjusting for site-specific idiosyncrasies (e.g., scaling by household income). Functional transfer (or benefit-function transfer) offers opportunities for adjustment of study contexts. Instead of borrowing a single number, analysts employ the entire valuation function, which includes the effects of a variety of site-specific characteristics of the study site and plugs in the corresponding characteristics of the policy site to generate a predicted value.
What Are Meta-Analysis Techniques in Benefit Transfer?
Consider the following definitions for the use of meta-analysis techniques in benefit transfer:
- Meta-analysis is an umbrella term capturing a set of tools to synthesize results from multiple individual studies identified during a systematic literature review.
- Meta-regression is a meta-analysis that uses regression analysis to aggregate estimates from multiple source studies into a statistical model and uses that aggregate model to generate a predicted value for the policy site.
Some argue that meta-analysis improves upon unit or function transfers because it draws on a broader empirical base, reduces dependence on any single study’s idiosyncrasies, and allows analysts to statistically control for differences across studies. Meta-regressions weight studies by sample size, giving greater influence to larger studies, but those conclusions might be misleading or biased if underlying methodologies are not accurate. These approaches have some validity in regulatory contexts where the goal is a reasonable approximation of aggregate social benefits across a large population or region.
Meta-analyses should be interpreted cautiously. In some cases, meta-analyses may attenuate outlier values into an expected, converged value that may not be representative of the case site. Meta-analyses, in most cases, create an illusion of rigor by aggregating what may be flawed studies with poorly specified control variables; but no statistical law of large numbers may rescue an aggregated set of studies built on unstable foundations.
The Correspondence Problem
The validity of any benefit transfer rests on the deceptively simple assumption that the welfare change measured at the source site is sufficiently similar to the welfare change at the policy site that the values can be reasonably extrapolated. More technically, this is the requirement that the two sites share similar aggregate demand functions for the environmental good or service in question. When that assumption holds, a transfer may produce acceptable results. When it does not hold, the transfer produces numbers that appear valid but are unreliable in fact.
The dimensions along which source and destination sites must correspond are numerous They include the physical and ecological characteristics of the resource, the nature and severity of the injury, the availability of substitutes for the resource, the baseline conditions against which the change is measured, and the numbers of affected individuals and their beliefs and preferences. A mistake on any of these dimensions and the transferred value may be systematically biased, either overstating or understating the actual welfare loss.
In practice, analysts relying on benefit transfer for litigation rarely find source studies that match on all these site-specific dimensions simultaneously. What they find instead is a collection of studies conducted on resources in different regions at different times that measure different kinds of changes using different methods for populations with different characteristics. The analysts then either select the closest available match(es), which may still be quite distant, or estimate a meta-regression that averages across a heterogeneous pool of studies that may not be appropriate, hoping the statistical model will control for the relevant differences. Neither approach solves the fundamental correspondence problem; both often merely obscure it behind a perceived layer of quantitative sophistication. For example, consider studies concerned with the valuation of freshwater fishing trips in Texas (Berkman et al. 2018). An expert could rely on a single study published in 1997 that finds a trip value of approximately $1. Another expert could rely on a study from 2001 that finds a value of $288. If a third expert were to combine the Texas-based fishing studies via a regression, estimated trip value would fall somewhere between the two depending on what is controlled for in the analysis.
As this example shows, excluding just one study from the meta-analysis can have substantial impacts on trip value estimates. Reliance on a single study or small pool of studies can produce crucial differences in damages assessments. It begs the question of what characteristics of the studies result in this enormous difference in the results. Without a standard protocol for the selection of papers to be used in a benefits transfer approach, the incentives exist for certain studies to be preferred or cherry-picked by either party.
The Specific Failures of Meta-Analysis in a Litigation Context
Meta-analysis can compound the correspondence problem in ways that deserve additional attention. A meta-regression of non-market valuation studies is not an analysis of the resource at issue; it is an analysis of how past researchers measured willingness to pay for what may or may not be similar resources under similar conditions. The dependent variable in the meta-regression is a willingness-to-pay estimate from a prior study, not an observed or estimated economic willingness-to-pay value. This means that the model is a model of prior modeling, and the systematic biases embedded in each constituent study may not be minimized through aggregation. They accumulate in ways that may be obscured by the apparent precision of the aggregate estimate.
Several problems arise from this structure. First, publication bias is pervasive in the valuation literature. Studies that find statistically significant, positive values are more likely to be published than studies that find small, zero, or negative values. A meta-regression that draws predominantly from the published literature will thus systematically overstate willingness to pay because the distribution of published estimates is not a random sample of estimates (Hoehn 2014).
Second, the studies included in a meta-analysis are often not produced under a common analytical approach or set of assumptions. This feature can be either a strength or a weakness, depending on the studies used and what the meta-analysis reveals.
Convergence of estimated values in the source studies is not always the case. Contingent valuation surveys[1] differ in how they describe the good being valued, how they frame the payment vehicle, whether they use open-ended questions, and numerous other design features known to affect elicited values. Travel cost models[2] differ in how they define the recreational site, how they value travel time, and how they handle multi-destination trips. Hedonic models[3] differ in the housing market data used, the spatial extent of the analysis, and the functional form being used. When these methodologically diverse studies are pooled into a meta-regression, the estimated coefficients reflect not only the genuine variation in the underlying resource values but also the variation in researcher choices.
Meta-regression is subject to the risk of ecological fallacy as the analyst tries to make inferences about individual locations using study-level information from other locations. Meta-regression is designed to address this kind of heterogeneity, and under certain conditions, it addresses it relatively well. The difficulty arises when the sources of methodological variation are not fully observed or cannot be adequately expressed as explanatory variables, which is common in environmental valuation contexts where study design choices are numerous, interdependent, and inconsistently reported. In those circumstances, the estimated coefficients conflate substantive and methodological variation and transferring the resulting function to a new policy site inherits those confounded estimates without any reliable means of detecting or correcting for the conflation.
Finally, and perhaps most directly relevant to the correspondence problem, the policy site in litigation is almost always unique in ways that are not captured by the parameters available in a meta-regression. The meta-regression can control for broad features, but it cannot capture the specific cultural and historical significance of the affected resource to the local community, the specific substitution patterns available to local recreationists, or the specific baseline conditions that existed before the defendant’s actions. These site–specific features are precisely what drives the welfare loss experienced by the affected population, and they are what the meta-regression is unable to accommodate.
Legal Standards and the Daubert Problem
In federal courts and state courts, expert testimony must satisfy reliability standards. Under these standards, a court must assess whether the methodology underlying an expert’s opinion is scientifically valid and whether it has been reliably applied. Benefit transfer, as typically deployed in litigation, usually struggles to satisfy both elements of this inquiry.
Benefit transfer and meta-analysis are not controlled experiments. They result in inference from imperfectly matched analogies, and the uncertainty in that inference is not fully captured by the confidence intervals that experts typically present. Those intervals reflect statistical uncertainty within the model, but they do not capture the structural uncertainty arising from the choice of source studies, the choice of model specification, and the assumption that the policy site’s demand function resembles those of the source sites. The presented precision is therefore often illusory.
On the question of reliable application, experts routinely make consequential choices that are not dictated by any established protocol and that can dramatically affect the final estimate. While NOAA (2021) and OECD (2018) have published guidelines on how to conduct a rigorous and defensible approach to benefit transfer, differences still remain in how experts build their bundle of comparable studies (Desvousges et al. 1998). Which studies are included in the meta-regression? What exclusion criteria are applied? What functional form is chosen for the regression? How are outliers addressed? Each of these choices can alter the final estimate by substantial amounts, and there is rarely a principled basis for preferring one set of choices over another. The result is that two experts using the same meta-analysis technique with the same literature can produce estimates that differ significantly, which undermines the claim that the method produces replicable, reliable results.
The apparent rigor of regression output can be persuasive to individuals who may lack the analytical background to evaluate what the numbers actually represent. Deferring to cross-examination as the corrective mechanism assumes that opposing counsel has the resources and expertise to conduct an effective and speedy methodological critique, which is an assumption that is frequently unwarranted.
Example Scenarios
Consider a suit arising from a chemical spill that kills fish in a stretch of a mid-Atlantic river. The plaintiff’s expert conducts a meta-analysis of willingness-to-pay studies for avoiding fish mortality events and applies the resulting function to estimate damages. The source studies in the meta-regression include estimates from Pacific salmon rivers in the Pacific Northwest, warmwater fisheries in the Gulf Coast, and Great Lakes tributaries. These are resources that differ from the injured site in species composition, cultural importance, regional recreational demand, and baseline ecological productivity, among other dimensions. The expert may control statistically for “river type” and “species type” using categorical variables, but these crude controls do not capture the idiosyncratic features that make one angler’s willingness to pay for a brown trout fishery in the mid-Atlantic different from another angler’s willingness to pay for a salmon fishery in Oregon.
Alternatively, consider a coastal wetland case in which the plaintiff seeks damages for the loss of ecosystem services. The expert imports unit values from a widely cited study of wetland ecosystem services conducted in the Gulf of Mexico and applies them to a New England salt marsh. The Gulf study reflects the values of a population with different income levels, different cultural relationships to wetland resources, and access to different substitute sites. The wetland types differ in their hydrological function and their recreational uses. The expert adjusts for income differences between the two regions but makes no adjustment for any of the other relevant dimensions because the source study does not provide the data necessary to make such adjustments. The transferred value is thus a number that bears uncertain and unquantified resemblance to the actual welfare loss experienced by the affected New England population.
These scenarios are not hypothetical. They describe the ordinary practice of benefit transfer as it appears in environmental litigation, and they illustrate why the correspondence problem is not a technical quibble but a fundamental weakness to the validity of the damage estimates being presented to courts. Wetland valuation studies illustrate the challenge with clarity. Wetlands are not a homogeneous ecosystem service, and they vary dramatically by location, species, and water type, among many other features.
Using a recent case example, an economics expert transferred willingness-to-pay values for wetland contribution to oyster production in Virginia to an area on the Gulf Coast. The underlying data source used in Virginia (Batie and Wilson 1978) was itself taken from a global meta-analysis used to determine coastal wetland habitat values. These wetlands from across the globe differ significantly from coastal wetlands on the Gulf Coast. The assumption that any of those values are immediately applicable to a specific Gulf Coast site with its own distinct characteristics (e.g., salinity gradients) is not a minor assumption. It represents a chain of transfers, each of which may be unverified and the errors of which would compound across transfers. The global meta-analysis produced a value, which was used in Batie and Wilson (1978) in Virginia, which was then transferred to the Gulf Coast more than 45 years later. At which point in the transfer chain was the correspondence accuracy tested and verified to be sufficiently similar to the Gulf Coast?
What Should Replace These Methods?
The critique presented here does not suggest that non-market environmental damages are not compensable. It is an argument about the appropriate standard of rigor. Where time and resources permit, original site-specific studies provide a far more defensible basis for damages testimony than a benefit transfer.
Where original studies are genuinely infeasible, experts relying on benefit transfer should be required to demonstrate, rather than merely assert, that source studies are sufficiently similar to the policy site to support the transfer. That demonstration should include explicit discussion of the ways in which source and destination differ, acknowledgment of the uncertainty those differences introduce, and sensitivity analyses showing how the estimate changes when different source studies or model specifications are used. Confidence intervals should be presented that reflect total uncertainty rather than only the uncertainty that the model can quantify.
Courts, in turn, should be more willing to exclude benefit-transfer testimony that cannot meet this standard, or at minimum should instruct juries on the specific limitations of transferred values and the ways in which they may diverge from actual site-specific damages. Benefit-transfer methods are useful tools when precision is less important than timeliness and when the consequences of error are distributed across a large population or geography. In litigation, the objective is different—it is to establish specific damages with sufficient reliability that a court can hold a specific party accountable. The methodological requirements of that purpose are not routinely satisfied by benefit-transfer practice as it currently appears in courtrooms. Until that changes, courts should regard damages testimony utilizing benefit-transfer methods with the skepticism the method’s limitations demand.
References
Batie, S.S., and J.R. Wilson. 1978. Economic values attributable to Virginia’s coastal wetlands as inputs in oyster production. J. Agric. Appl. Econ. 10(1):111–118.
Berkman, M., M Rogers, and G. Waterfield. 2018. The accuracy of benefits transfer for environmental valuation. Trends Jul./Aug.:15. https://www.americanbar.org/content/dam/aba/publications/trends/2018/tr_vol49n06_issue.pdf
Desvousges, W.H., F. Reed Johnson, and H. Spencer Banzhaf. 1998. Environmental Policy Analysis with Limited Information: Principles and Applications of the Transfer Method. Edward Elgar Publishing, Cheltenham, UK and Northampton, MA. 256 pp.
Hoehn, J. 2014. Water demand elasticities: Selecting effects in meta-analysis. Selected paper, Agricultural and Applied Economics Association Annual Meeting, Minneapolis, MN. https://doi:10.22004/ag.econ.169291.
NOAA. 2021. Benefits valuation method: Benefit transfer. National Oceanic and Atmospheric Administration, Office for Coastal Management. Digital Coast Academy. Washington DC. June. https://coast.noaa.gov/data/digitalcoast/pdf/econguide-benefit-transfer.pdf
OECD. 2018. Cost-benefit analysis and the environment: Further developments and policy use. OECD Publishing, Paris. http://dx.doi.org/10.1787/9789264085169-en
Vetter, D., G. Rücker, and I. Storch. 2013. Meta-analysis: A need for well-defined usage in ecology and conservation biology. Ecosphere 4(6):74. http://dx.doi.org/10.1890/ES13-00062.1
[1] A contingent valuation survey is a method used to estimate the economic value of non-market resources by asking individuals how much they would be willing to pay for specific environmental services or how much compensation they would accept for their loss.
[2] A travel cost model is an economic method used to estimate the value of recreational sites or ecosystems based on the costs people incur to visit them.
[3] A hedonic model is a statistical approach used to determine how different attributes of a product (often a single-family home) contribute to its market value.