1 THE ART AND SCIENCE OF FORMULATING SURVEY QUESTIONS
2 Principal Sources of Error in Survey Results Sampling error The sample of interviewees is not representative of the universe from which it is drawn; it is skewed in various ways. (“Sample of the whole” eliminates bias) Respondent error (can be reduced by skillful questionnaire construction) Response is not appropriate/relevant to the question, due to misunderstanding (= poorly worded question?) Response is deliberately misleading (= lying) Faulty recall (retrospective questions) Social desirability response set: Response is determined by what interviewee believes is socially acceptable behavior or attitude Acquiescence response set: Respondent agrees with whatever statement is presented to him/her, for fear of offending interviewer Interviewer error Question not asked (skipped), question was asked of inappropriate respondent, response categories were read (directing respondent), response was inaccurately recorded, etc. Data processing error “GIGO”: garbage in, garbage out. Errors in transferring data from questionnaire into data base. (Electronic data capture eliminates such errors)
3 From research objective to survey question: An example Objective: Document and explain change in rate of settlement in U.S. (vs. circular migration). Explanatory hypotheses: Such changes may be a function of: -- Continued tightening of border controls, increasing coyote fees, indebtedness, and physical risk -- Family reunification migration | (fewer relatives to maintain in hometown) -- Maturation of social networks -- Less seasonal work in U.S., more employment in “permanent” year-round jobs -- “Anchoring” investments in U.S. (home ownership, small businesses) -- Life events (marriage, children reaching school age, etc.) -- Changing gender roles in migrant families (greater female labor force participation, etc.) -- Decisions to naturalize (gain U.S. citizenship), for “defensive” purposes, employment eligibility, etc. Empirical tests/indicators? Survey questions?
4 Filter questions: 1.¿Es Ud. de Tlacuitapa? (si NO es de Tlacuitapa, agradecerle a la persona y busca otra persona) 2.¿Cuántos años tiene Usted? (si tiene menos de 15 años o más de 65 años, pedir hablar con otra persona) The answers to these questions are not marked on the cuestionario. Types of Survey Questions
5 Standardized Question, Closed Response
6 Non-standardized Questions, Closed Response Códigos: Documentos 1= Cuidadano 2 = Residente Legal 3 = Contratado (H2A/H2B) 4 = Contratado (Bracero) 5 = Turista 6 = Tarjeta cruce local 7 = Visa de estudiante 8 = Indocumentado 8888 = N/A 9999 = NS/NQR ViajeLugar de Destino (Ciudad, Estado) Año en que llegó Tiempo que duró este viaje en EEUU (meses) DOC (usar códigos) 1er
7 Standardized Question, Open Response (avoid as much as possible) (Si no usó coyote:) ¿Cómo logró cruzar la frontera, sin usar coyote? (Anotar respuesta completa) ___________________________________________ _________________________ ___________________________________________ _________________________
8 Open Question, Open Response Ya no es encuesta. It’s a qualitative, life-history interview. Questions soliciting answers that will involve a lot of complexity, nuance, and detail should be in a semi- structured interview schedule. Qualitative interviews are also generally better for getting at emotion, meaning, and narrative. E.g.: ¿Qué es los que más/menos le gusta de los EE.UU.?
9 Questions with a numeric response: Different Types of Closed Responses
10 A poorly constructed question: A better question with a simple numerical response: ¿Cuántas veces a la semana fuma Ud.? ________ (Specify a time frame and a frequency of behavior. Answer may be “0.”)
11 Questions with a dichotomous yes/no response:
12 Questions with scaled responses:
13 “Thermometer” scaled response ( MOSTRAR TARJETA ) Le voy a preguntar su opinión acerca de algunas personas e instituciones. En una escala de 0 a 10, donde cero significa que su opinión es muy mala y el 10 que su opinión es muy buena, ¿cuál es su opinión acerca de….? (LEER Y ROTAR) a.Seguro Social b.Programa “Oportunidades” c. El Papa Benedicto Dieciséis d. Barack Obama, presidente de Estados Unidos e. Enrique Pena Nieto, presidente de Mexico
14
15 Dichotomous vs Scaled Responses: 55. Actualmente, si uno no tiene papeles, ¿es muy peligroso cruzar la frontera? Si 1 No 2 N/A8888 NS/NQR9999 55. Actualmente, ¿qué tan peligroso es cruzar la frontera, sin papeles? Muy peligroso1 Algo peligroso2 Poco peligroso3 Nada peligroso4 N/A8888 NS/NQR9999 Better:
16 Questions with unscaled categorical responses:
17 Comparative opinion question (paired alternatives): 91. Algunas personas dicen que los jovenes nacidos en Tlacuitapa pueden progresar en la vida sin salir del pueblo. Otras personas dicen que para superarse, los jovenes nacidos en Tlacuitapa tienen que salir. ¿Qué diria Ud.? puede superarse sin salir 1 tiene que salir2 N/A8888 NS/NQR9999
18 Questions asking respondent for a rank ordering:
19 OTHER ISSUES IN SURVEY DESIGN Define complex concepts precisely ¿Cuántos viajes a hecho a los Estados Unidos, para trabajar o vivir? Definición de ‘un viaje’: Un viaje es una estancia en los EE.UU. que dura por lo menos 1 mes y cuyo motivo principal es trabajar o vivir. No incluye vacaciones ni turismo. Si el entrevistado regresa a Tlacuitapa por menos de 1 mes y luego regresa a los EE.UU., consideramos que es el mismo viaje a los EE.UU.
20 Always specify the time frame for reporting 21. En los últimos 5 años, ¿dónde ha pasado más tiempo, en Tlacuitapa o en los EEUU? San Miguel Tlacotepec1 Otra localidad en México2 EEUU3 N/A8888 NS/NQR9999 22. En los últimos 5 años, ¿cuántas veces regresó Ud. a Tlacuitapa ? Nº de veces:_________ N/A8888 NS/NQR9999
21 Avoid questions that don’t elicit sufficiently detailed responses 55. Actualmente, si uno no tiene papeles, ¿es muy peligroso cruzar la frontera? Si 1 No 2 N/A8888 NS/NQR9999 55. Actualmente, ¿qué tan peligroso es cruzar la frontera, sin papeles? Muy peligroso1 Algo peligroso2 Poco peligroso3 Nada peligroso4 N/A8888 NS/NQR9999 Better:
22 Softening the question/reducing sensitivity Ahora quiero hacerle algunas preguntas sobre su último intento o éxito en cruzar la frontera. 30.¿Lo hizo con papeles, o tuvo que entrar sin papeles? Con papeles legales1 Con papeles prestados2 Con papeles falsos o chuecos3 Sin papeles4 N/A8888 NS/NQR9999
23 Organization of questionnaire/sequence of items Time elapsed in interview Sensitivity of items 0
24 The art (and a little of the science) of field interviewing
25 Interview each other to gain confidence
26 Get your bearings
27 Build rapport with the entire community
28 Try to convince Profe that you’re not just having a pachanga
29 Key issues/challenges Introducing yourself How to explain what you are doing to the potential interviewee. “How will I – or my community – benefit?” (Para que sirve todo esto?) Hints: Have an honest “sales pitch.” We are writing the “history” of the town and its migration. Our UCSD group was here three years ago. “Confident assertiveness” – Fowler We’re not from the US or Mexican government. Information is anonymous. Participation is voluntary.
30 What are the benefits for Tlacuitapa? Don’t promise anything we can’t deliver. Community leaders are interested in high-quality data about the health of residents and migrants Our research findings can be used to educate U.S. public and “funcionarios” who make U.S. immigration policy – para promover la reforma migratoria. Smaller expressions of gratitude: passing out free books from previous Tlacuitapa studies (selectively!), offering to take family photos to migrants in the U.S., etc… Benefits tend to be at the community-level for the town & binational society rather than for specific individuals.
31 Managing social distance issues: You are an “elite” college student: perceived as upper- class, highly educated, culturally different. How to make yourself less threatening? Hints: Allow the interviewee to ask you questions about yourself, to reduce social distance and establish trust. Trade on your student status. You are there to learn. Emphasize our team’s previous experience in Tunkas and southern California; common social ties. We are an “equipo binacional.” Draw on the gender/ ethnic/ national diversity of the team to manage difficult interview situations.
32 How to manage time during the interview Need to keep to a schedule, to complete interview in a reasonable amount of time, while maintaining the flexibility needed to build rapport and stimulate informative exchanges of information with interviewee. Hints: -- allow interviewee to tell you (some) of his own stories; -- use pauses, allowing enough time for a response (don’t interrupt silence); -- internalize the questionnaire (memorize it!) so you don’t waste time looking for the next relevant question)
33 Read the question the first time EXACTLY as it is written in the cuestionario. This is especially important for attitudinal questions. If the respondent doesn’t understand the question, you may explain it in your own words. Probing: How deep to probe, to get a response that’s more detailed, more complete, more reliable, more relevant to the question? Caution: Danger is that, with extensive probing, interviewee may “invent” an opinion that in reality he doesn’t have, to be courteous or to avoid the shame of not having an opinion.
34 Detecting and correcting errors of inconsistency in responses Hints: “Active listening” is important. Pay very close attention to responses. Think about the response content. Repeat the respondent’s own words to signal your understanding of him, and to set up follow-up question to clarify a contradiction or inconsistency.
35 Reducing respondent error (“reporting error”) Interviewee doesn’t understand question, gives inappropriate response. Interviewee gives incomplete answers (not telling the whole truth) Interviewee gives deliberately misleading answers (untruthful responses) Interviewee has recall problems (poorly recalled facts, e.g., in response to retrospective questions about migration & employment history) Hint: Prompt respondent with questions about major life events like birth dates of kids, marriage, to help establish a timeline for other events
36 Other sources of respondent error Social desirability response set (bias): Interviewee tells you what he thinks YOU want him to say, or what is most socially acceptable in his community, in the U.S. Absurdly high values: For example, 40-year-old man tells you that he has had 10 wives or has been to the U.S. 100+ times, to impress you, or to signal he’s not taking interview seriously. Non-response: Interviewee refuses to answer certain questions at all (due to fear, shame, desire to conceal income, etc.) Hint: Try to convince respondent of the non-threatening nature of the study. Try to get answers for all questions, but if the respondent refuses, move on to other questions.
37 Still more challenges How to record responses (note-taking) without interrupting flow of the interview? How to assure confidentiality? How to respond to interviewee requests, for money, transportation, introductions, information about “El Norte,” etc.? What “body language” should you use to build rapport and elicit more useful responses?