D.6 Step 6: Choice of data collection method

There are two ways through which respondents can be contacted in an online survey: Either a link to a questionnaire is sent via e-mail (e-mail survey), or such a link is based on a website (web/ internet based survey). In online surveys, no interviewers are involved. Hence, the questionnaire needs to be self-explanatory and to include very clear instructions, questions and response categories. A lot of attention should be paid to an attractive and easy-to-fill-out questionnaire format.

A great disadvantage of online surveys is that the response rates are usually very low (mostly < 5%). This can lead to biased results as those people who do respond are normally more interested in the topic than the nonrespondents. Additionally, an online survey is only accessible by people who own a computer and know how to use it. Another disadvantage of online surveys is that you cannot be completely sure who is actually filling in the questionnaire. Furthermore, technical problems need special attention to avoid data loss.

However, in some cases the advantages of online surveys can outweigh the problems connected with them. First of all, as no interviewers are involved, this is one less source of error. Second, data from online surveys are immediately available in an electronic format. This saves time and money for manually entering data, and helps to avoid another source of error. Additionally, the extent of the questionnaire (and thus the number of questions that need to be read by each respondent) can be reduced by the introduction of automatic filters (see the next part on questionnaire design for more information on filter questions). Answer inconsistencies can be detected through automatic plausibility checks. Online surveys are quite cheap to conduct as you do not have to pay interviewers, travel costs or costs for stamps etc. (such as in mail surveys).

If your population of interest has access to computers and knows how to use them, and if there is experience with online surveys in your organisation, they can be a reasonable way to collect certain types of market data. In order to increase response rates, it can be a good option to combine an online survey with follow-up telephone calls to remind the participants of their participation (Lozar et al.  2008) and/or to offer participants incentives. For a survey on organic product prices among farmers, you could for example offer to send the survey results to those people who answered the complete questionnaire. Knowledge about these prices will provide a benefit for farmers, so this incentive could motivate them to participate in the survey.

In mail surveys, a (printed) questionnaire is sent to people of a certain population via ‘ordinary’ mail (not e-mail). The envelope in which it is sent should further contain a return envelope for sending the filled-in questionnaire back, with the return address and sufficient postage (freepost) provided.  With mail surveys, you can also access people who do not have a computer or internet access. Otherwise, in many ways mail surveys are very similarly to online surveys, and you have to consider the same issues in their design.

The response rate of mail surveys, though often higher than in e-mail or online surveys, is usually still quite low, very often between 5 and 10% only. Furthermore, you obviously need to know the postal addresses of all elements of your population of interest. Even if you have to pay for printing costs as well as for envelopes and postage, mail surveys are still a cheaper option than personally conducted interviews. Participants of both mail and online surveys can fill in the questionnaires whenever it suits them, so the time pressure is reduced.  In addition, respondents have more privacy and there is no interviewer bias, so that more honest answers can be expected (de Leeuw & Hox, 2008).

As telephone surveys are conducted by interviewers, they have the potential problem of interviewer bias (the influence on a respondent through an interviewer). Hence, it is very important that not only the questions, but also the interviewing procedure are standardized and precisely followed by all interviewers. Telephone surveys should also be rather short, and the questions need to be very clear.

Response rates for telephone surveys are also quite low, but usually slightly higher than for online or mail surveys. It might be difficult to reach people of the population of interest, either because they are not available (nonresponse) or because not all telephone numbers are available (undercoverage). Mailing out a letter informing about the survey in advance of contacting by phone may improve response rates. Telephone surveys, however, have lower costs and less interviewer bias than face-to-face interviews. Compared to mail and online surveys, respondents can ask for clarification in telephone surveys. Telephone surveys can be carried out with the assistance of computers (so-called computer assisted telephone interviews, CATI). One basic function of CATI is that during the interview the interviewer directly enters the respondent’s answers into an electronic format (Steeh, 2008). Specialised marketing agencies further use computers for the automatic creation of telephone numbers (as address books are normally incomplete) and for automatic dialling. In this case, contacted numbers are only connected with an interviewer if the phone call is answered by a person and not by an answering machine.

Face-to-face surveys are rarely used in market data collection, as it might not be feasible to approach all respondents personally; especially if they are widespread in larger geographical areas. In face-to-face surveys the respondents are directly questioned by the interviewer, so that interviewer bias can be problematic. Furthermore, face-to-face surveys are very costly and time-consuming. Their major advantage is that the response rates are relatively high. In addition, the questionnaire can be longer and more complex. Face-to-face surveys can also be conducted in a computer-assisted manner, so that the structure and interview procedure is provided on the screen. If the interviewer guides through the survey and uses the computer for assistance, these interviews are referred to as CAPI (computer assisted personal interviews). If interviewees conduct the survey themselves, following the instructions on the screen, the interviews are called CASI (computer assisted self-interviews). In CASI the respondents fill in the questionnaire on the screen without any assistance and guidance through an interviewer. This way, it is possible to reduce the interviewer bias (Loosveldt, 2008). Data are available in an electronic format immediately after the survey.

Mixed-mode surveys refer to surveys, in which two data collection methods are combined. There are two different options. On the one hand groups of participants can be contacted in different ways (parallel data collection) and on the other hand groups of participants can be approached with two different types of surveys – one after the other (sequential).  Depending on the accessibility of respondents, the combination of two types of surveys might increase the response rate.

The checklist contains the relevant questions for your choice of the right research design.