WO2004092910A2 - Test design - Google Patents

Test design Download PDF

Info

Publication number
WO2004092910A2
WO2004092910A2 PCT/US2004/011453 US2004011453W WO2004092910A2 WO 2004092910 A2 WO2004092910 A2 WO 2004092910A2 US 2004011453 W US2004011453 W US 2004011453W WO 2004092910 A2 WO2004092910 A2 WO 2004092910A2
Authority
WO
WIPO (PCT)
Prior art keywords
data
respondent
relevant
question
questions
Prior art date
Application number
PCT/US2004/011453
Other languages
French (fr)
Other versions
WO2004092910A3 (en
Inventor
Shail Patel
Original Assignee
Unilever N.V.
Unilever Plc
Hindustan Lever Limited
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority to EP03252372.2 priority Critical
Priority to EP03252372 priority
Application filed by Unilever N.V., Unilever Plc, Hindustan Lever Limited filed Critical Unilever N.V.
Publication of WO2004092910A2 publication Critical patent/WO2004092910A2/en
Publication of WO2004092910A3 publication Critical patent/WO2004092910A3/en

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06QDATA PROCESSING SYSTEMS OR METHODS, SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL, SUPERVISORY OR FORECASTING PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL, SUPERVISORY OR FORECASTING PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00Commerce, e.g. shopping or e-commerce
    • G06Q30/02Marketing, e.g. market research and analysis, surveying, promotions, advertising, buyer profiling, customer management or rewards; Price estimation or determination
    • GPHYSICS
    • G09EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09BEDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B5/00Electrically-operated educational appliances
    • GPHYSICS
    • G09EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09BEDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B7/00Electrically-operated teaching apparatus or devices working with questions and answers

Abstract

The present invention provides a method for adaptive generation of questions, comprising the steps of (a) collating for a respondent the existing data stored in one or more electronic data-sources and relating to said respondent; (b) determining data gaps in the existing data; (c) providing at least one investigation parameter; (d) determining the data gap which is most relevant to the investigation parameter; (e) generating at least one most relevant question for reducing the size of said most relevant data gap. Preferably, the method is carried out real time while contact is ongoing with the respondent.

Description

Test Design

Field of the Invention

The present invention relates to a method for designing tests in particular a method for the generation of an interactive test for example a question or a sequence of questions for use in consumer testing or market research.

Background of the Invention Questions are a well-known tool for testing. For example questions are often posed to respondents to identify preferences of the respondents . These pre erences can then for example be used for targeted supplier- client interactions for example via targeted mailing or advertising.

WO 01/46891 describes a computer implemented method comprising the distribution of a first survey, receiving responses, analysing the responses and obtaining a second survey. US4,603,232 discloses a method for dissemination, deployment (hardware/infrastructure) and collation of personalised surveys. US5,842,195 discloses a method for authoring/storing of survey documents. US5,893,098 discloses a method for deployment (hardware) of branched surveys and obtaining responses. US6, 009,420 discloses an expert system for interactive task completion for decision support.

A problem with the posing of question is that the quality and reliability of the answers often is not optimal. Reasons for this can be that the respondent is not particularly interested in answering or the respondents find the sequence of questions too long and hence too time consuming. Another problem is that response rates are frequently poor, resulting in limited quantities of data which may be biased. Previous solutions to these issues have proposed splitting the complete questionnaire into a number of sub-sample components and randomly assigning each respondent to receive two of these sub-samples or eliciting a random sample of questions from each individual .

The present invention aims at providing an adaptive questionnaire system whereby on the one hand reliable results can be obtained and on the other hand the time necessary for completing the questionnaire can be minimised.

The present invention also aims at providing an adaptive questionnaire design that can advantageously work in the situation wherein the respondent initiated contact. Furthermore the system allows the opportunistic generation of questions or sequences of questions whereby for each new question the utility of the question can be maximised therewith reducing the above described negative effects.

Definition of the invention

Accordingly the present invention provides a method for adaptive generation of questions, comprising the steps of

(a) collating for a respondent all existing data stored in one or more electronic data-sources and relating to said respondent ;

(b) determining data gaps in the existing data;

(c) providing at least one investigation parameter; (d) determining the data gap which is most relevant to the investigation parameter; (e) generating at least one most relevant question for reducing the size of said most relevant data gap. Preferably, one or more of the above steps (b) - (e) is carried out using electronic means. More particularly, the at least one most relevant question may be suitably generated using a computer network which contains information on investigation parameters. Alternatively, said most relevant question may be automatically raised on a Web-site.

Detailed description of the invention

Preferably the method of the invention is carried out as soon as contact with the respondent is established. Preferably, this contact is initiated by the respondent. As soon as the most relevant question is generated this is posed to the respondent and the answer by the respondent is received.

After receipt of the answer by the respondent advantageously the answer can be added to the electronic data-sources and the method of the invention can be re-iterated based on the newly available set of existing data.

The end-point may be determined in several ways . For example it may be a pre-determined number of iterations, for example n=l to n=100, or when some data-based measure has been achieved. Desirably, n is lower than 10. Examples of data based measures include the standard error of the mean for one or more key variables being less than an acceptable amount, a statistically significant difference being detected for a key variable and/or variables or enough data having been collected to correctly categorise a large pre-determined proportion of respondents.

The invention will now be explained in more detail. Information about consumers is collected from a variety of storage means (1) . Said storage means comprise a number of details relating to (potential) respondents Rx. Said storage means may be any suitable data storage means, preferably for electronic data storage for example computer memory, tapes, CD ' s , dvd .

At a desired moment, for example as soon as contact is established with the respondent the data is accessed from the data storage means through connection means known in the art. Subsequently, all data concerning the specific respondent are combined, and gaps in knowledge are determined for example by evaluating the certainty of the data collected, and ranking accordingly.

The system then determines an investigation parameter. This can be separately inputted or alternatively be calculated. For example, a suitable investigation parameter may be to determine whether a respondent belongs to the group of most favoured customers for a certain product or service. Alternatively, the investigation parameter may be to determine whether the respondent would most likely buy a new product to be launched. This is e.g. done by putting a weighting (a value) to each of the variables for which data are collected.

Subsequently, the data, data gaps and investigation parameter are combined and based on these the system determines what the one or more most relevant questions are to minimise the size of the knowledge gaps relating to said investigation parameter.

Suitably, the most relevant questions can be selected from a pre-determined list of question stored in the system. Alternatively, the system may generate the question for example by selecting a number of textual sequences and combining these into a newly generated question. For example the first name of the respondent (for example "Linda") may be combined with a textual sequence "how are you" to generate a question "How are you Linda?". The selected question can be determined by various methods , for example known in the art .

The question generation is advantageously automated but if desired the selection of the appropriate question can be (partly) based on external input e.g. from an operator.

The selected most relevant questions are then transferred to an interaction means for posing these selected questions Ql-Qn to the respondent and receiving the response.

The interaction means may involve any suitable method but may for example be a computer terminal at which the questions are shown to the respondents and at which the respondents may input their responses. Other suitable interaction means may for example be access means to connected communication systems, for example questions may be posed to the respondents by electronic means such as computer networks, internet, or other data- communication systems for example via telecommunication such as telephones or mobile phones. Otherwise questions may be printed to paper, completed by the respondents and read back into electronic storage, either through manual entry or automatically, for example by scanning.

The response of the respondent is then collected in output receiving means. Again this can be any suitable data storage means as described above . The testing routine may be repeated for a number of times n, where after the end results may advantageously be stored or shown in data storage and/or display means.

The testing routine may be performed on any appropriate time scale. For example the filtering, interaction and collection of responses may occur in real time as the test is unfolding or in batches over a longer time period. In a first preferred embodiment the various iterations n may advantageously follow each other in time (iteration n is followed by iteration n+1 etc) . However for some applications it may also be advantageous to carry out multiple iterations at the same time e.g. iterations n, n+1 ...., n+p are carried out simultaneously.

The above-described system may advantageously be used for the design and implementation of tests, in particular the design and implementation of questionnaires.

A key element of the system of the invention is an adaptive process to come to the best sequence of questions. As described the system will generally use n steps or loops.

The number of steps or loops n can be any value from 1 to a reasonable number of questions for example 1 to 100, for example 2 to 20, generally from 2 to 10.

The filtering step for determining after each step which questions can best be used in the next step can be any suitable means for selection. For example calculations can be done to select the most important questions. Suitable methods to do this are for example Correlations, Partial correlations, Principal Component Analysis, Factor Analysis, Multiple Regression, Stepwise Regression, Neural Networks, Automatic Relevance Determination, Decision Trees, Neurofuzzy Methods.

If desired the system of the invention can include one or more validation checks whereby selection of questions are validated e.g. by checking back with a previous group of respondents.

The invention will now be illustrated in the following examples .

Example 1

A consumer X telephones the help line for a personal wash bar product, with queries about the availability of said product in the local supermarket and about the ingredients of said product. As the phone operator types the caller's name and address into the Consumer services database, this information is sent (via software agent) to the integrated consumer database. This database pulls out everything that is known about consumer X, specifically: - the recent questionnaire responses by consumer X through the Home Basics magazine (in numeric form) ; the contents of his last conversations with the help line (in analysed text form) ; his interactions with the online chat bot on the product concerned.

Optionally, the system uses this information and the information relating to people like consumer X, in order to form an approximate characterisation or profile of the person. The system then decides, or is told, what type of information is of most value. In other words, the system decides or is told which investigation parameter is used, e.g.:

= it may be desired to fill in important general gaps in the integrated consumer database;

= it may be desired to determine whether consumer X is a most valued consumer (MVC) for the personal wash bar product; = it may be desired to know better whether consumer X is likely to respond to particular marketing levers, such as promotions, discounts and free samples;

= it may be desired to reduce the uncertainty in the assessment of whether consumer X is a MVC or may respond to a particular product promotion.

Given the result of these assessments, the system decides which question will give an answer that satisfies the relevant criteria. This question is then automatically highlighted on the screen of the telephone operator, and the operator asks this question in a standard format, viz.: "Excuse me Sir, would you mind if we asked you a question? We would be happy to send you a promotion of product ABC if you would give us the answer to this question." This sentence is then followed by the specific question concerned. When answered, the answer is then duly entered into the appropriate screen and placed in the integrated consumer database. Example 2

A consumer Y goes online to one of the online magazines, say pourtoutvousdire.com, perhaps looking for a recipe or a cleaning tip. As soon as the consumer is online, the system sends his e-mail address to the integrated consumer database, which pulls out everything that is known about this consumer Y.

A procedure similar to that of example 1 is then carried out. At the end of this procedure, the selected question is automatically raised on the web-site. Subsequently, the answer to said question is analysed and the result placed in the database.

Claims

Claims
1. A method for adaptive generation of questions, comprising the steps of
(a) collating for a respondent the existing data stored in one or more electronic data-sources and relating to said respondent ;
(b) determining data gaps in the existing data;
(c) providing at least one investigation parameter;
(d) determining the data gap which is most relevant to the investigation parameter;
(e) generating at least one most relevant question for reducing the size of said most relevant data gap.
2. A method according to claim 1, which is carried out real time while contact is ongoing with the respondent.
3. A method according to claim 1, wherein the most relevant question is posed to the respondent and the answer is added to the data-sources.
4. A method according to claim 1, wherein the method is repeated for a further 1 to 100 times, preferably 1 to 10 times .
5. A method according to claims 1, wherein one or more of steps (b) - (e) is carried out using an electronic means.
PCT/US2004/011453 2003-04-15 2004-04-14 Test design WO2004092910A2 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP03252372.2 2003-04-15
EP03252372 2003-04-15

Publications (2)

Publication Number Publication Date
WO2004092910A2 true WO2004092910A2 (en) 2004-10-28
WO2004092910A3 WO2004092910A3 (en) 2005-03-31

Family

ID=33185967

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2004/011453 WO2004092910A2 (en) 2003-04-15 2004-04-14 Test design

Country Status (1)

Country Link
WO (1) WO2004092910A2 (en)

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5893098A (en) * 1994-09-14 1999-04-06 Dolphin Software Pty Ltd System and method for obtaining and collating survey information from a plurality of computer users

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5893098A (en) * 1994-09-14 1999-04-06 Dolphin Software Pty Ltd System and method for obtaining and collating survey information from a plurality of computer users

Also Published As

Publication number Publication date
WO2004092910A3 (en) 2005-03-31

Similar Documents

Publication Publication Date Title
Yousafzai et al. Technology acceptance: a meta-analysis of the TAM: Part 1
Fombrun et al. Who's tops in corporate reputation?
Drollinger et al. Development and validation of the active empathetic listening scale
Perkins et al. The role of experience in information use and decision making by marketing managers
Putsis Jr et al. Mixing behavior in cross-country diffusion
US7606726B2 (en) Interactive survey and data management method and apparatus
Roy et al. A framework for investigating habits,“The Hand of the Past,” and heterogeneity in dynamic brand choice
Wang et al. The conceptualization and measurement of m-commerce user satisfaction
Floropoulos et al. Measuring the success of the Greek taxation information system
Deutskens et al. An assessment of equivalence between online and mail surveys in service research
Baxendale et al. The impact of different touchpoints on brand consideration
Crabbe et al. An adoption model for mobile banking in Ghana
US7379537B2 (en) Method and system for automating the creation of customer-centric interfaces
Leimeister et al. Success factors of virtual communities from the perspective of members and operators: An empirical study
Wang et al. An instrument for measuring customer satisfaction toward web sites that market digital products and services.
Kuusik Affecting customer loyalty: Do different factors have various influences in different loyalty levels?
Robin et al. The perceived importance of an ethical issue as an influence on the ethical decision-making of ad managers
Ahire et al. Quality management in TQM versus non-TQM firms: an empirical investigation
den Ouden et al. Quality and reliability problems from a consumer's perspective: An increasing problem overlooked by businesses?
AU2011205137B2 (en) Social media variable analytical system
US7383200B1 (en) Method and apparatus for collecting and categorizing data at a terminal
Pablo Maicas Lopez et al. The impact of customer relationship characteristics on customer switching behavior: Differences between switchers and stayers
Day Collecting comprehensive consumer compliant data by survey research
US20120290357A1 (en) Identifying industry segments with highest potential for new customers or new spending for current customers
US8332232B2 (en) System and method for mobile interaction

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A2

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BW BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE EG ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NA NI NO NZ OM PG PH PL PT RO RU SC SD SE SG SK SL SY TJ TM TN TR TT TZ UA UG US UZ VC VN YU ZA ZM ZW

AL Designated countries for regional patents

Kind code of ref document: A2

Designated state(s): BW GH GM KE LS MW MZ SD SL SZ TZ UG ZM ZW AM AZ BY KG KZ MD RU TJ TM AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IT LU MC NL PL PT RO SE SI SK TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG

121 Ep: the epo has been informed by wipo that ep was designated in this application
DPEN Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed from 20040101)
122 Ep: pct application non-entry in european phase