Appendix F. Survey Methodology
Interval: Interval measures are standard units, such as cups of milk. When survey responses include standard units (e.g., ¼ cup, ½ cup, 1 cup), it is preferred to use paired or matched statistical tests to determine whether there are changes in mean (average) scores before and after the program. Use paired or matched statistical tests, such as a t–test, to determine whether the changes are statistically significant.
Ordinal: Assessments of attitudes or agreement with statements using a Likert-type rating scale use ordinal measures. For ordinal levels of measurement, the simplest approach is to compare percentage distributions of responses before and after the program. For instance, before the program, a certain percentage of participants may strongly agree with a statement; at follow-up, a different percentage may strongly agree. Calculate the percentage change from before to after the program. Unlike interval data, calculating means is not appropriate for ordinal responses. However, comparing the median (middle) or mode (most frequent) response before and after the program can be appropriate. The Wilcoxon Signed-Rank statistical test will identify the level of statistical significance.
Nominal: When an outcome measure is nominal (e.g., names of fruit or answers to “yes or no” questions), these are categorical responses. For nominal data, the simplest approach is to compare percentage distributions of responses before and after the program. For instance, before the program, a certain percentage of participants may drink low-fat milk; at follow-up, a different percentage may drink low-fat milk. Calculate the percentage change from before to after the program. The McNemar’s statistical test will identify the level of statistical significance.
Open-ended versus closed-ended questions: A key part of creating an excellent survey or questionnaire is the appropriate use of open-ended and closed-ended questions. An example of the difference between closed-ended and open-ended questions would be the offer of fish or meat for dinner (closed-ended), as opposed to asking, “What would you like for dinner?” (open-ended). Questions that are closed-ended are conclusive in nature as they are designed to create data that are easily quantifiable. The fact that questions of this type are easy to code makes them particularly useful when trying to determine statistical significance. The information gained by closed-ended questions allows evaluators to categorize respondents into groups based on the options. One drawback to the use of closed-ended questions is the possibility that the response options may not be comprehensive enough to reflect the respondent’s true response. For example, a question may ask if the participant takes the bus, bikes or takes the metro to work, but doesn’t include car-pooling as a response option. Open-ended questions are exploratory in nature. Open-ended questions provide rich qualitative data because the respondent can provide any answer. Since questions that are open-ended ask for critical thinking they are ideal for gaining information from specialists in a field, small groups of people, and preliminary research. Although respondents’ answers are rich in information, it takes great effort to distill the information provided.
Pre- and post-tests: Pre- and post-tests are used to measure changes as a result of an intervention. The pre-test is a set of questions given to participants before the training or activity, and the post-test is administered after the training or activity and contains the same questions as the pre-test. Comparing participants’ post-tests and pre-tests enables the evaluator to assess changes in specific outcomes. In the evaluation framework, healthy eating, physical activity, and food security goals, intentions, and behaviors are the main indicators for which pre- and post-tests are recommended. Evaluators should use a unique, anonymous identifier to facilitate matching of pre-post pairs (e.g., MM/DD/YYYY birthdate).
Limitations of surveys: When in the planning stages of creating and implementing a survey, evaluators should keep in mind survey limitations. Some of these limitations include limited access to the population of interest, compressed or limited time schedule to conduct the survey, and lack of funding.
Bias in survey sampling: Bias refers to a sample statistic that either over- or under-estimates the populations’ parameters. This means that the results from the sample of participants that is drawn either over- or under-represent the true population parameter (such as the true population mean). Bias often occurs when the survey sample does not accurately represent the population.
The bias that results from an unrepresentative sample is called selection bias. Some common examples include: nonresponse bias, undercoverage, and voluntary response bias. This type of bias can be reduced through random sampling. There can also be bias that is introduced through problems in the measurement process such as asking leading questions and respondent social desirability.
Statistical Significance (T-tests): With a t-test, statistical significance indicates that the difference between two groups’ averages most likely reflects an actual difference in the populations from which the groups were sampled. A statistically significant t-test means that the difference between two groups is unlikely to have occurred because the sample happened to be atypical. Statistical significance is determined by the size of the difference between the group averages, the sample size, and the standard deviations of the groups.