Assessing the Effect of Calibration on Nonresponse Bias in the 2005 ARMS Phase III Sample Using 2002 Census of Agriculture Data

摘要

Phase III of the Agricultural Resource Management Survey (ARMS) is one of the most complex and detailed sample survey data collections conducted by the United States Department of Agricultures (USDA) National Agricultural Statistics Service (NASS). For this survey, NASS collects highly detailed, calendar year economic data from agricultural producers nationwide. The USDA uses ARMS data to evaluate the financial performance of farms and ranches, which influence agricultural policy decisions. The Department also uses Phase III data for objective evaluation of critical issues related to agriculture and the rural economy. In September 2006 the Executive Office of the President released the Office of Management and Budget Standards and Guidelines for Statistical Surveys based on the recommendations of the Federal Committee on Statistical Methodologys (FCSM) Subcommittee on Standards for Statistical Surveys. The Office of Management and Budgets (OMB) new standards and guidelines for statistical surveys addressed a number of federal statistical agency issues, specifically response rates below 80 percent and analysis of unit nonresponse. The 2005 Agricultural Resource Management Survey (ARMS) Phase III Survey Administration Analysis (Hopper, 2006) reported a response rate of 70.5 percent, and thus NASS was required by OMB to conduct an analysis of nonresponse bias. This report focuses on fulfilling OMBs directive, by assessing the effect of nonresponse bias as well as the utility of the NASS calibration process in adjusting for such biases. Records sampled for the 2005 ARMS Phase III were matched with those from the 2002 Census of Agriculture, and means of census data were calculated for matching records which had also provided 2002 expenditure data for the Census. Nonresponse bias in ARMS data was assessed, using census data as a proxy, in terms of the degree to which the mean based on all sample cases versus respondent cases differed. Three means were computed and compared across 20 regions in order to assess relative bias: 1) the mean of all matching cases using base sampling weights, 2) the mean for only matching ARMS respondents using the same base sampling weights, and 3) the mean for matching ARMS respondents using the sampling weights as adjusted through calibration. Using 17 study variables, relative bias of the mean was assessed using a variation of the formula provided by OMB in Guideline 3.2.9. Although significant biases were exhibited in 9 of 17 variables using the 2005 ARMS Phase III base sampling weights, the 2005 ARMS Phase III calibration weights were able to reduce the bias so that it was no longer significantly different from zero (p