Michalis A. Michael
Managing Director Western Europe
MASMI Research Group
London, January 2010
Table of Contents
1. Executive Summary
2. Introduction
3. Recruitment of Panelists
a. Panel Representivity
b. The Issue With Professional Respondents
c. Ways To Recruit Panelists
4. Panel Management
a. Panel Types
b. The Use of Panels
c. Treat Them Like Customers
d. Panel Quality
e. Online Communities
f. Panelist Utilization
5. Sampling
a. “Friendly” Algorithms
b. Rates That Matter
c. Quota Management
6. Technology
a. Infrastructure (Load balancing, risk mitigation)
b. Panel Management System
c. Sampling System
d. Data Collection Platform and Tools
e. Digital Fingerprinting
f. Broadcasting (ISP relations, multilingual)
g. Multimedia
7. Financials
a. The Size & Volume Dilemma
b. Incentives
c. Pricing
d. Attrition
8. Data Collection Methods
a. Quantitative
b. Qualitative
9. Conclusions
1.Executive Summary
As this paper presents more of a “how to” (and “what not to”) approach to panels, its practical applications for MR agencies lie in the implementation and management of panels. From the MR users’ perspective, it is a guide that can help select an appropriate panel research provider.
Representivity is always a question for everyone involved and a view is provided in this paper on how to approach information collected from online panels. It is very important for the users of panel research to be aware of the pitfalls of cheap research that involves professional respondents who are described diligently in this document.
This paper can also be viewed as an expanded check-list (similar to the ESOMAR 26 questions) that can guide not only panel owners when they build and maintain their panels but also panel users when they choose a panel provider.
Finally the transparency provided in this paper about all aspects of opinion panels might be a unique source of information that was never available before for marketing students, market research trainees on the agency and client side alike, market research clients new to panels and online (e.g., in countries where internet penetration has not allowed this type of research up to now), or simply clients who are unfamiliar with this research method.
2. Introduction
People in the market research industry speak of the democratization of research. Creating and managing global panels or simply having access to them through panel brokers is now possible, for any small multinational or even single-country research agency with simply the ambition to offer global online research. This might be a naïve way of approaching global panels. Not only local knowledge but also local presence is beneficial in order to provide consistently high quality research and client service.
Mail Panels have existed in the US since 1941. When the internet became part of our lives it was natural to start moving to online data collection since it is cheaper and fieldwork can be carried out much faster. In most of the rest of the world, however, mail panels were not as popular, so things evolved the other way round. First came the internet and then panels as a good way to have access to email addresses and avoid spamming. This phenomenon can be seen as a “geographic paradox”!
Another reason why opinion panels are now so popular can be illustrated with a few statistics, again from the US: Fifteen years ago to complete one telephone interview, seven calls were needed, today we need to call 35 people or more to get one interview. If, however, we use an access panel to select respondents for a telephone survey we return to a seven-to-one ratio as it was 15 years ago using RDD (random digit dialing). The ratio gets even better if the panel is custom or client proprietary .
Panels also provide us with demographic and other variables on panel members, allowing us to conduct research for a specific target audience without the screening phase, which can be very frustrating for the panelists, time consuming and expensive, especially for minority incidence requirements.
3. Recruitment of Panelists
As a best practice panelists should opt-in at least twice to participate in a “respectable” panel, and typically they provide demographic information for themselves and the members of their households. Based on this information and additional screening information collected subsequent to recruitment or just before taking a survey (not ideal), panelists qualify to participate in specific surveys (see Figure 1.).
There are three main types of recruitment:
• Open Approach: open invitation on the web or print publications whereby panelists self-select.
• Closed Approach: panelists are recruited by invitation only through the use of email lists or RDD (random digit dialing).
• River Approach: panelists are intercepted on the web and they typically take one survey only; they are typically not asked to join a panel.
Panelist Profiles
Figure 1
There is a minimum age limit that needs to be observed when recruiting panelists which differs by country. Minors can be interviewed (or recruited) only after acquiring permission from at least one parent (panel member).
3.a. Panel Representivity
A panel is a sample of the population of all households or all people. A sample drawn from a panel is thus only representative of the panel; unless the panel itself is randomly selected from the population, then the sample can be representative of the population
Telephone contacts using Random Digit Dialing (RDD) can be a way to recruit a random panel – provided all members of the population have equal chance to be selected and in addition they agree to participate. Realistically a) this is not achievable since some people only have mobile phones, some do not have any landline phone (in both developed and emerging markets), and very few from those contacted will agree to become members of the panel; and b) in most developed countries it is much more expensive to recruit via phone versus online.
Consequently if a panel cannot be randomly selected, a randomly selected sample drawn from it cannot be considered representative. However, for business decision-making purposes, it has been proven through parallel studies that a balanced sample according to demographics and other criteria combined with post stratification, produces sufficiently comparable results to conventional surveys and therefore can be successfully used.
The real question is not how representative a sample is but how comfortable the decision-maker is with the overall results of the survey and the business decisions they imply/suggest.
3.b. The Issue of Professional Respondents
One of the most important issues that owner companies of access panels have to confront is that of "professional respondents". It is key to avoid recruiting them as much as possible by not recruiting from sites that offer incentive points or miles for various marketing related activities. Also it is better to avoid luring people to join a panel with the promise of money as the main attraction. Ideally people should join panels because they have an opinion and they want to be heard. "Incidentally" they will get some money or participate in sweepstakes, but it should be clear that they cannot become “rich” from such participation. If this sounds too idealistic, at a minimum financial incentives and “their opinion matters” should be presented as equally important. It is also essential to have a process in place that helps identify patterns that would indicate or even prove that certain panelists are professional respondents, in which case they should be purged from the panel. Another indication that a panelist is perhaps a professional respondent is when they participate in multiple panels (Overlap). According to an ARF report published in August 2009 based on a survey called Foundations of Quality , 16% of the panelists in 17 online panels in the US belong to more than one panels. The findings of the survey, however, indicate that “whether a respondent belonged to one or to many panels there was virtually no difference in survey results”. This is a situation that exists today and makes it difficult for a panel provider to know how many times a panelist has participated in surveys, unless appropriate processes such as digital profiling, re-profiling, algorithms that check consistency or quality of response, etc. are in place that attempt to limit, understand the impact of or prohibit this.
Ways to Identify Professional Respondents
Figure 2.
3.c. Ways to Recruit Panelists
There are a number of potentially effective methods to recruit panelists but, of course, like everything in life they have their pros and cons. They are classified as suitable for the open, closed or river approach. Here is an attempt to provide a comprehensive list of double (or triple) opt-in recruitment methods:
Telephone (Random Digit Dialing – RDD )- Closed
In the absence of mobile phones and in markets where the telephone penetration is 100%, this could have been the ideal way to recruit panelists. The reason is that it is the closest to “everyone having an equal chance to be selected”. But then again this is a very expensive way to recruit people to a panel – almost cost prohibitive in some countries. Another disadvantage of this approach. adding to the cost, is the so-called “catastrophic non-response rate”.
Email Campaigns - Closed
If one can get access to email lists that are not used for recruitment by other panels (and will not be used in the future for this purpose) then an email invitation can be sent that highlights the aspect of opinion sharing as opposed to earning money. Payment for the list can be based on CPM (cost per thousand emails), CPC (cost per click on the URL provided in the email), or CPA (cost per acquisition).
Card Direct Mail or Insert Campaigns - Closed
Similar to an email campaign but more expensive. A card invitation to join a panel is sent by mail to a physical address, or it is inserted in a newspaper or magazine that is delivered to a physical address.
Web Banners- Open or River
This is probably the most common way of recruitment. A panel owner has the option to contact websites directly, through affiliate networks to which those websites belong or through a media agency. Again here, the banner content should focus more on making opinions known and impacting the world around us as opposed to “becoming rich”! The higher the number of websites that participate in recruiting, the more representative the panel will be. Payment can be based similarly to email campaigns on CPM, CPC or CPA.
When it comes to the actual recruitment agreement (say of online panelists) between the panel owner and the website that has access to its visitors (or even better, subscribers) there are various ways to go about it. The panel owner can pay cash for exclusive recruitment based on CPM, CPC or ideally CPA. Another possibility is to provide research services in return for recruited panelists – a barter deal in other words. The recruitment could be for more than one purposes; i.e. co-registration.
Face-to-Face-Closed
This recruitment method can be used at events, shopping malls, door-to-door (during face-to-face or hall test surveys - in the absence of which this method would be very expensive). As with telephone recruitment , the prospective panelists need to take a note of the URL they have to manually type in their browser in order to go to the recruitment landing page. Alternatively they can simply provide their email on a paper form and then receive an email invite to register.
Advertisements in Publications Open or Closed depending on the publication
This is an ad in a newspaper or targeted magazines. For example, if a panel owner requires panelists that are very active with their garden, a gardening magazine would be an appropriate recruitment media vehicle (and this would be classified as a quasi closed approach).
With all the above methods an invitation is extended to click or insert a URL in one’s browser and visit the recruitment website. On this website the expected panelist activity is explained once again and the potential panelist needs to go through a short questionnaire and provide information about themselves and their household members (first opt-in). An email will follow from the panel company inviting the prospective panelists to create a user ID and a password to access the panel website (second opt-in).
Referrals (or snow-balling)
Generally referrals is not considered to be best practice unless it is kept within limits (say up to 3 referrals, per panel member, maximum) – and even then it is subject to the ratio of people recruited as a result of a referral compared to the total number of panelists. If, for example, one-third of the panel came through referrals then the representivity of the samples drawn from such a panel is questionable. An example of how not to do things is a specific panel company in the US that recruits only through referrals and the person who refers gets paid every time the panelists they recruited complete a survey. It goes even further – reminding us of a pyramid scheme – as part of the payment to the recruits of the recruits also goes to the original panelist who invited them to join the panel.
In terms of who exactly should be the panelist in a household there are 2 options: to recruit individuals only and not to be concerned about households, or to recruit representatives of households and use them to also invite other members of the household to participate in surveys. Ideally what research clients will be looking for is the relevant purchase decision-maker of their products or services.
One real issue with online panels is not knowing 100% that the registered person is who they say they are and their demographics and other details are reported truthfully. There are countries like South Korea where reporting a social security number is not a taboo like it is in the US. This way the true identity of a person is guaranteed. However, even then there is the odd chance that the young son will be filling in surveys for the mother who is shown to be registered as a panelist. The only way to pick this up is through validation by using post study analysis of completion patterns (see also section 3.d).
4. Panel Management
4.a. Panel Types
A general population/consumer opinion panel is the most generic form of a panel and the one mostly used by CPG companies as well as other industries. As the title of this section suggests there are many different types of panels.
General Population / Consumer Panels
General opinion and consumer panels are more or less the same and they can be recruited to represent the general online population, or they can be skewed towards demographics that are mostly used for CPG research, such as females 25-55 or mothers with young children .
Sub-panels
These are panels within the general population panels for which specific attributes are available that place them in a certain homogeneous group of panelists available for specific types of research. An example here would be an Automotive Panel. These are people from the general population panel from which we have information on car ownership, intention to buy, preferred makes, etc. This way research among low incidence target groups is made possible and is not cost prohibitive. Other examples of sub-panels are Patients, Mothers, Teachers, Farmers, Handicapped, and more.
Specialty Panels
A specialty panel is one that has members who would not necessarily become members of the general population panel (GPP) or at least not enough of them can be found in a GPP. Special targeted (expensive) recruitment methods need to be applied in order to find and recruit these profiles of panelists. Some examples are: Physicians, hunters, hobby gardeners, legal professionals, politicians, musicians, IT Professionals, CFOs, CEOs, HR Professionals, etc.
Business to Business (B2B)
This is a sub-set of the Specialty Panels. Most of the SPs will be used for business to business research but not all of them. To be more specific about B2B, IT Professionals, CFOs, CEOs and others are used to provide information about their business to other businesses.
Proprietary/Custom Panels
These are panels that belong to and are used by one company only for their own purposes. This can be managed by a research agency exclusively for the owner or the owner can opt to manage their panel themselves. Insurance companies, banks and mobile phone operators are companies which own large transactional client databases. To turn those into active panels would make a lot of sense from their point of view.
4.b. The Use of Panels
Panel research and online research (or otherwise known as Internet Research or Web Research or CAWI) seem to be used in an interchangeable way by some. Actually, the former is a sample source and the latter a data collection method, both absolutely necessary in order to implement any market research. There is online panel research, offline panel research, and even online non-panel research. Consequently opinion panels can be used as a sample source with any data collection method: face-to-face, CATI, CAPI, IVR, SMS, HAPI and CAWI. Panels can also be used to recruit participants for qualitative research (e.g., FGDs and IDIs). Mystery shoppers can also be recruited from panels provided special care is paid to product category exposure.
4.c. Treat Them Like Customers
A panel can literally be treated as a fixed asset in financial terms. Such a unique asset with a mind of its own needs to be nurtured and taken care of. It needs to be kept satisfied for at least 3 full years (asset amortization period in the US). Of course, there are reasons why a market research client would want a panel to be regularly refreshed. More details about panel attrition are outlined under paragraph 6.d.
In this paragraph we’ll concentrate on the need for panelists to be treated like customers. The goal is to achieve panelist loyalty, so they do not defect or become inactive, in which case the investment to recruit the panelists is lost. Another reason is that loyal and committed panelists are also more truthful and honest: they will not want to harm a website or an institution they like.
What ways are there to create loyal panelists?
• Give them choices such as: how many invitations to receive in a year, their preferred method of contact, etc.
• Give them engaging surveys and questionnaires designed in a user friendly way
• Through the website provide a feeling of belonging to the panelist community
• Provide rewards for participation – ideally creative rewards (e.g., gift certificates, contribution to charities)
• Provide useful content on the website as well entertainment (games, quizzes) where possible
Suggestions for Panel Relations Activities
Dedicated Staff
Panel owners need to have dedicated staff to assist panel members with questions, comments and profile changes. Inquiries are handled by native-speaker panel relations representatives, responding to requests via email, mail and telephone.
Panelist Website
Panelists can check out lottery winners, review the privacy statement, make changes to their profile, check rewards earned, refer a friend for panel membership and conduct many other activities through the panelist website.
Extensive Privacy Policy
The panel owner should make a strict commitment to protecting the privacy of its panelists. It makes sense for the panel owner to be a member of an independent, privacy watchdog program.
Frequency of Usage
Panel members should be used frequently enough to understand the research process and feel a personal bond with the panel owner (which is how response rates can be improved), but not so much that where they would feel inundated with research requests.
Figure 3.
4.d. Panel Quality
Many market research associations and in some cases agencies took panel quality assurance to heart and have developed tools to help the users choose wisely. For example, ESOMAR developed the infamous 26 questions which actually started off as the “ESOMAR 25” in 2006. There were 3 additional questions that the author of this paper recommended to ESOMAR to add to the 25 questions back in 2006 (and become The 28 Questions about online and panel research). These are:
26. What procedures do you apply to ensure a balanced sample of respondents?
27. What are your procedures for supplementing your sample where it is necessary to meet the client's target?
28. What is your panel turnover rate and is it sufficient to ensure that “internet newbies” are represented?
Other organizations that are active in panel quality matters are: the Panel Quality Summit, ARF and even Comscore and e-Rewards.
Inattentive, fraudulent, hyperactive and speeders are four types of panelists that every panel owner should be looking to find in every survey (with the use of traps) in order to red-flag for investigation or exclude from the results. Digital fingerprinting as described under 5.e. is another way to identify fraudulent or hyperactive panelists.
4.e. Online Communities
With the Web 2.0 being part of our lives online communities have become a hot topic for advertisers. Companies like Coca Cola and P&G have pioneered with the creation of themed online communities for marketing and market research purposes. Online panels could easily create multiple online communities if they had the technology platform. It would not be surprising if some of the panel operators decide to partner with companies like Facebook or Linkedin, which already have the required technology. By simply adapting the “skin” panel owners could find themselves generating marketing relevant content without having to ask their panelists any questions; i.e., self-generated content.
An online community should be a sub-set of a custom panel, according to Easyinsites. It is a step that should follow the creation of a custom panel, that agency says. Managing an online community is a much more complex exercise than utilizing a custom panel.
4.f. Panelist Utilization
This is quite a controversial subject. First of all there does not seem to be a consensus on how many times a panelist can participate in surveys during a year without being considered an “expert respondent”.
A strong distinction should be made between Professional Respondents and Expert Respondents. It seems that in many publications these two types of panelists have been confused. Professional Respondents participate in surveys just for the money; they tend to be in a hurry to fill in a questionnaire to cash the incentive and move on to the next one. Many of them are unemployed or they want to subsidize their income through participation in research and direct marketing activities. Through their urge to finish quickly, they tend to click the easy answer e.g. …“just the middle box” – generating invalid information. On top of this they might even lie in order to meet the survey criteria (e.g., certain brand users) and be selected. An Expert Respondent, on the other hand, is someone who participates in a great number of surveys or is a market research professional who knows how questions are asked and for what purpose, but is not necessarily dishonest. The downside here is when they are exposed to brands that they would not know otherwise and their responses in subsequent surveys are biased.
Many panel owners declare that they use their panelists only 12-14 times a year. Others quote number of invitations to participate as opposed to actual participation. One specific agency in Japan declares without hesitation that they send over 100 invitations to each panelist every year.
Revenue generation for panel owners can be limited by their ability to sell surveys directly to end clients or indirectly through panel brokers and/or research agencies or by the number of available panelists and the times they are allowed to participate in surveys. This is an interesting question for every panel owner to answer: “Based on my panelist utilization policy, how many surveys can I sell and by extension how much revenue can I generate if I could sell as much capacity as I have available?
PANEL MAINTENANCE PROCEDURES
Recruitment
Hundreds of affiliates and partners (portals, sites, networks) are used to recruit by Access Panels, avoiding bias from limited recruiting sources. Panelists should be recruited on a continuous basis, maintaining a fresh pool of research respondents.
Bad Addresses
‘Non-deliverable’ email addresses should be monitored and eliminated from sample selection.
Duplicate Records
Panel owners should conduct de-duplication when loading panelists onto their panel database. Once recruitment data is in-house, duplication within that data file and also against existing panelists should be checked. Ongoing checks should be conducted for panelist duplication - in case a respondent’s information has changed through the updating process.
Eliminations
Project eliminations should be done on a subject matter basis. Panel members should be excluded from being surveyed on the same topic/category for at least six weeks. For product tests, panelists should be excluded for at least six weeks over and above the product usage period.
Figure 4.
5. Sampling
Sampling is the process of drawing a sample from the panel that fulfills certain criteria, often referred to as quotas. Due to the large panel sizes required in order to provide a viable sample source even for low incidence populations, a highly sophisticated sampling approach is required. As mentioned under Panel Recruitment, panelists can be selected based on not only demographics but also brand and category usage, lifestyles and attitudes, ownership health conditions, and more.
5.a. “Friendly Algorithms”
Even for basic samples an algorithm is required for balancing samples according to the given criteria. The question many research buyers ask is: can a representative sample be drawn by a panel that is not representative? If yes, is selecting a stratified sample according to demographics enough or is there a need to select a sample according to additional criteria in order to be as close to a randomly selected sample as possible? Relevant criteria might include such things as lifestyles, attitudes, brand or category usage, etc.
Another opportunity to use an algorithm is in the area of probability sampling. This is a module of a sampling engine that selects panelists to be invited based on historical behavior in terms of survey participation. For example, people who rejected participation in pet surveys in the past might not be invited to participate in a dog-food survey.
A third example is an algorithm for balanced responses. See 4.c.
5.b. Rates That Matter
Clients often ask panel owners to provide an indication of the quality of their panel. What are good metrics in order to do this? Here is a selection of rates that are currently used:
Response rate: This is the number of people who took the first step in filling in a survey but did not necessarily complete it divided by the total number of people who were invited to participate.
Cooperation Rate: This is the number of people who completed a survey divided by the number of people who qualified from those invited.
Completion Rate: Same as Cooperation Rate
Drop-out Rate: This is the number of people who voluntarily discontinued filling in the survey divided by the total number of people who started the survey
Termination Rate: This is the number of people who were discontinued involuntarily as a result of a screener divided by the total number of people who started the survey
Incidence Rate: This is the number of people who fulfill the selection criteria to participate in a survey divided by the total number of available panelists. This rate requires some more attention since it is one of the biggest challenges to get right and is crucial at the proposal stage. Usually clients are much more optimistic than realistic in terms of incidence rate. Proposals tend to be based on higher incidence rates than what is reality, and then the result is cost disputes between the panel owner and the panel buyer.
The answer to the question asked at the beginning of section 4b would have to be: higher than average Response and Cooperation rates are positive indicators for the quality of a panel.
Sample Composition: The Benefit of Staged Invite Release
Figure 5.
5.c. Quota Management for Online Panel Projects
Quota management is – as strange as it may sound – linked somewhat to management of client expectations. It seems that both researchers and clients have different expectations depending on the geography they are in. In the US for many it is acceptable to balance outgoing invitations (outgoes) to panelists according to what is required and hope that it is close to the ideal requirement. Some clients in the US and elsewhere have no problem with post-weighing the results in order to reflect the required quotas. Others will not agree to do this.
In Japan, for example, and surely in some other countries in Asia and in Europe, many clients expect that the quotas will be met exactly as required (Figure 5). In other words, if a quota cell is full, we stop accepting responses (the guillotine approach) and if one is not full after we have sent all the outgoes initially planned, we keep sending email invitations until each and every cell is full. This is obviously a very unpredictable and expensive method.
There is another way to do this: “Close to ideal”. The aim is to achieve balanced responses as opposed to just sending balanced outgoes and seeing what happens. This can be achieved by sending the email invitations for participation in waves. Each wave of outgoes is adjusted according to the responses received up until the time of the next wave of outgoes. In other words, each wave “learns” from the results of the previous one in terms of returns and adjusts in order to come as close to the quotas as possible by the time the last invitation is sent (according to an initial estimation based on incidence and cooperation rates). This approach works best when the fieldwork time is longer than 6 days, since the average response time for all respondents invited drops if we keep sending emails until the last day. However, if the multi-wave approach is utilized for the first half of the fieldwork time only, the negative effect of including mainly fast responders is reduced.
6. Technology
Multiple systems are required in order to run research using panels such as:
• Returns system
• Broadcast engine
• Panel management system
• Sampling system
6.a. Infrastructure
Referring to the online data collection capability, whether it is infrastructure for a single country, multiple countries or global, the number of server locations is dictated by the need to mitigate risk (at least two locations required) and not so much by the internet speed (available bandwidth between countries and continents), or the need for every location to have their own servers to manage online surveys.
Even for offline research, servers and specialized applications are required for the sampling activity. A few conditions need to be fulfilled: there should be an automated load balancing mechanism for all available servers, and fast scalability of each one of the two locations in order to cover the workload in case one facility is destroyed or is out of order.
6.b Panel Management System (PMS)
This is the website and the back-end application required to recruit and store data of panelists who have access to the internet. The panelist website is used as a way to communicate, to do marketing, to reward, to entertain and to create loyalty between the panel members and their panel (or the panel owner).
It is advisable for the panel recruitment website to be totally different from the public website of the agency that owns the panel. There are a few good reasons for this advice: one does not want to make it easy for competitors to register on one’s panel and have access to information that one would not necessarily want to share. Also competitors of clients might do the same in order to spy on the products their competitors test.
A world-class Panel Management System should be able to handle extremely large and diversified global panels that can deliver global samples out of the available panel members. It should support the panel owner to attain highest retention rate, and by extension lowest possible attrition rate, through highly satisfied panel members. Finally it should support the optimized usage of panelists and high quality of information from them.
6.c. Sampling System (SMS)
This is a system that could cost from US$20,000 to over US$1,000,000 to develop, depending on functionality (such as sample selection, elimination, user interface, returns, probability sampling, balanced responses etc.). The latter cost reflects more of a world class sampling system in terms of range of functionality, speed of service and quality of sample. It is a system that is able to draw a project-ready, consistent sample at the first attempt within minutes from submission into the sampling system.
6.d. Online Data Collection Platform and Tools
Since online is the most widely used data collection method with panels, and growing in importance, the technology platform used in conjunction with the other applications is of paramount importance.
Some examples of platforms widely used are MR Interview from the SPSS Dimensions suite of research applications, Confirmit, Kinesis, CfMC, Voxco, Global Park, Nebu, and Toluna.
SPSS Dimensions is one of the most popular open architecture platforms that allows companies to connect applications other than the ones provided from SPPS. For example a company could use MR Interview and MR Tables for online data collection and reporting, but use NIPO for CATI and CAPI and a proprietary HAPI and IVR application, all connected through an interface: Data Source Components (DSCs) to the SPSS Dimensions platform.
Confirmit seems to also be one of the most popular online data collection and reporting engines at least among the top 10 global market research agencies, and GMI used to be mostly used by smaller companies as an Application Service Provider but at some point they decided that they preferred to be a panel company and not so much a technology company anymore. This is a limited selection of platforms, used just as an example.
6.e. Digital Fingerprinting
This new technology can be used to avoid using the same panelist twice for the same survey. Without this technology a respondent may register more than one times on the same panel with different name and email address .
A company that pioneered this technology is Peanut Labs who license their technology to panel owners. Optimus™ by Peanut Labs is a new digital fingerprinting technology that improves data quality by accurately identifying and flagging suspect respondents as defined by panel owners in real-time from any sample source across the market research industry. Optimus™ can work with any existing survey tool or methodology. It uses patent-pending algorithms to filter out suspect survey respondents that have been shown to significantly bias research results, without altering the respondent experience in any way. Many of the top market research agencies are testing or deploying this technology to improve data validity and demonstrate a commitment to quality through the Optimus™-certified Seal
6.f. Broadcasting
An application is required to manage the vast amounts of email traffic from and to the online data collection servers. What is key here is for the application to be able to handle multiple languages when it comes to global panels (including double byte languages). Also the broadcasting system should be able to balance broadcasting in order to take into account own system load limitations (i.e., what the servers can handle in terms of traffic) and also ISP limitations (i.e., what number of simultaneously sent emails might trigger a spam red flag that could lead to the black listing of the panel owner). For every panel company it is of ultimate importance to stay on the white lists of the ISPs, otherwise 2 things can happen:
a) certain ISPs might slow down the traffic coming from a website that is suspicious (because of the volumes) and
b) a panel URL might be moved to the black list if there is a suspicion that spamming is taking place.
It is advisable for panel owners to have direct relationships with all the ISPs they depend on and also to observe load/traffic limitations that are imposed by the ISPs or to negotiate separate treatment after proving that no spamming will ever incur through the panel owner’s URL/IP addresses. ISPs should be a phone call away so that an agency can return quickly to being white-listed in case of an automated black-listing that can jeopardize a survey or more.
6.g. Multimedia
With the internet being totally part of our lives, new capabilities are made available to research agencies. All online questionnaires can take full advantage of the world of multimedia and use photos, film and sound all together or separately. This was also possible with CAPI but the difference is that it is real time and centrally managed – no need to worry about different versions of stimuli for example (the only worry is different versions of internet browsers). The downside is that for highly confidential materials there is no real guarantee that they cannot be copied. Even though there is special security software that prevents a panelist from downloading the photo of a new product or a new TV advertising spot, a panelist can always use a camera to film the screen or take a photo of the screen. It is good advice that if something is really confidential and should under no circumstances leak to competition, then it should not be shown on the Internet.
7. Financials
7.a. The Size & Volume Dilemma
Owning panels is a great way to increase profitability for a market research agency. A precondition for this is that a critical mass of business goes through those panels, especially if we utilize them through online research. When converting offline research (say telephone) to online and panel research, clients have the expectation that the price will go down. This is a justified expectation as long as it is not over 20%. Based on experiential data from one company, no more than a 15% reduction in price is realistic over a wide range of study types with different deliverables ranging from just clean data (ASCII file) to a presentation of results with recommendations and action planning. This price reduction ideally should be coupled with a gross margin improvement of over 20% thus achieving an overall profitability increase for the agency.
This is a definitive win-win situation since from the client’s perspective the deliverable is faster and cheaper. Even though better is the third element of the clients’ wish list, it is not always easy to convince clients that all three requirements are met when converting from offline to online. In fact, a valid position of the MR industry, or any other services industry, could be that only two of the three requirements can realistically be offered. A client can either expect a service to be provided faster and better at a higher price, or cheaper and better but it should take longer time to deliver, or lower quality service delivered faster and cheaper. For anyone to want to get a service faster, cheaper and better at the same time is in most cases an unrealistic expectation.
Recruiting a panel is a substantial investment for the panel owner. The good thing about this investment from an accounting point of view is that it can be viewed as capital expenditure and can be amortized over a period of 3 years (in certain countries). The bad news is that only the incremental panel recruitment can be considered as capital expenditure, not the recruitment that aims to replace panelists that defect or are purged from the panel.
Now the dilemma referred to in the heading of this section is about how large the panel should be (at the beginning) in order to accommodate the volume of business and be profitable for the panel owner. There are a few issues here to be taken into account:
- What is the profile of the respondents required in the majority of the research projects to be sold initially? What is the incidence range of the samples used most often?
- What participation limitations are imposed on the panelists in terms of how many times per month/year they are allowed to participate in surveys?
- Is the limitation the ability of the MR agency to sell or will it be the number of available panelists?
- At what percentage of the cases does it stop making sense to top-up the proprietary panel with third party panels?
- What is the maximum possible revenue per panelist and how to make sure that this is met before a further increase of the panel size is initiated?
- In order to avoid this dilemma is it possible to recruit panelists without paying, and then pay the website owners from where the panelists originated on a per usage basis?
- In how many months is it acceptable to return the investment in recruiting the panelists?
Once all the above questions are answered, then hopefully a formula can be created that will suggest the ideal panel size for a given revenue projection at a given margin.
7.b. Incentives
There are different ways to incentivise panelists to participate in surveys. The guiding principle should always be that whatever the remuneration, it should not be enough to make a panelist “rich” (in other words to be a major source of income) so that professional respondents will not be encouraged to join.
A very popular way is to use redeemable points that are earned every time a panelist fills in a survey. The number of points should depend on number of minutes and incidence of respondent.
The points can be redeemed in various ways: cash by check or bank transfer, gift certificates or other forms of vouchers, multiple participation in sweepstakes depending on number of points, and purchase of products or services through specific websites. One issue here when it is left up to the panelists to decide when to redeem, is how to treat the unredeemed points at the end of a year for accounting purposes. Also when a panelist leaves the panel voluntarily or involuntarily and points are unredeemed, a way to deal with this “unexpected gain” in accounting should be identified in advance.
7.c. Pricing
The pricing of fieldwork using panels can be broken down in the following parts:
- Panel Usage: usually based on a combination of questionnaire length and respondent incidence.
- Labor: this is the cost to draw the sample from the panel and when it comes to online research to execute the project
- Hosting: this is the cost for acquiring and maintaining the infrastructure
7.d. Attrition
Attrition is a double edged sword! If the attrition is low, say below 20% per year, the panel owner will be very pleased because of having to recruit fewer panelists to maintain the panel size. So from a purely financial point of view, low attrition (ideally zero) is sought after. However, if you are a client of panel research you would want the panel you use to be refreshed frequently (say 40%-50% per year) so as to avoid the issue of the professional/expert respondents. Attrition over 40% could easily render a panel to be an uneconomical proposition. Again like most things in life the solution must be somewhere in the middle, probably between 0-40%.
8. Data Collection Methods
8.a. Quantitative
This paper would not have been complete if we did not address the ways data can be collected from the panels. Of course the dominating data collection method is online and as time goes by its dominance will increase. However, in markets like the US one cannot ignore the mail survey - still a major data collection method utilizing panels. Other methods are telephone (CATI to landline or mobile), face-to-face (PAPI, CAPI, HAPI) and text messaging. Many traditional full service companies, especially in the US, now generate more than half their revenue from online surveys.
Using email lists instead of panels is also a possible way to conduct online research; however
it is not as efficient, and generally a lost opportunity to get consent for an ongoing participation in surveys.
8.b. Qualitative
Panels can also be used to recruit participants to qualitative research (face-to-face and online) such as Focus Group Discussions, In-Depth-Interviews, online bulletin boards, chat rooms, or online mystery shopping.
9. Conclusions
It is quite obvious that owning and operating panels is a multifaceted and complex business. It is challenging but at the same time it can be very rewarding if a panel owner manages to strike the optimum price versus quality ratio. This ratio, along with the ability of the panel owner to sell, will define the revenues, profit and by extension the success of this business.
A constant debate will be about how high can one “afford” the quality to be. The truth is that this depends on the MR clients’ willingness to accept having to pay a premium for high quality standards. As long as clients will go after low price solutions they encourage low quality panel operators and make it less of a dilemma for the rest of the operators as to whether they should keep “crusading” for high quality panels.
Certain conditions need to be defined and accepted by the MR industry and related bodies and maybe even take it a step further by giving panel operators an accreditation after a thorough audit. This way a minimum quality standard will be set that will be accepted by both research agencies and research buyers.
The objective of this paper was to provide a holistic and honest view about today’s status of access panels globally. They will undoubtedly evolve further from today’s state to something bigger and different in the future. It is important for all concerned and involved to continue sharing views, debating issues and sharing best practice in order for the evolution to be guided towards the right direction. But then again, who is willing to define “right”?
References
1-Easyinsites.com
2-MRSI Online Give and Take – White Paper
3-MASMI Russia Panel Operations Guidelines
4-ARF’s Foundations of Quality survey report
5-Michalis Michael Presentation Europe Open Days 2005
6-Michalis Michael Answers to 26 ESOMAR questions
7-MRSI Online Give and Take - White Paper
8-Easyinsites.com
9-Michalis Michael Answers to 26 ESOMAR questions
10-Michalis Michael Presentation Europe Open Days 2005
11-Peanut Labs
12Peanut Labs
Contributing Reviewers
Tony Rhodes, MASMI Research Group
Leonid Voronin, MASMI Research Group
Adam Berman, CatalystMR
John Surrey, Veloce! Management & Marketing Insights