JP2011233044A - Device, method and program for subjective evaluation - Google Patents

Abstract

PROBLEM TO BE SOLVED: To compare search results by different respondents and different evaluation objects by the same standard.SOLUTION: This device includes: an evaluation object database storing evaluation object data; a determining part for one evaluation object presentation order by generating a first presentation order of a plurality of evaluation object IDs each having a reference evaluation value, and a second presentation order of a plurality of evaluation object IDs each having no reference evaluation value and integrating the first and second presentation orders; a transmission and reception processing part for transmitting the evaluation object data to a respondent terminal according to the evaluation object presentation order, and receiving a response value to the evaluation object data from a respondent identified by the a respondent ID; a respondent characteristic estimating part for estimating a characteristic value of the respondent identified by the respondent ID on the basis of the reference evaluation value and the response value; and an evaluation value estimating part for calculating an estimation value of an evaluation value to an evaluation object having no reference evaluation value on the basis of the response value and the characteristic value of the respondent.

Description

  Embodiments described herein relate generally to a subjective evaluation apparatus, method, and program suitable for a questionnaire survey or the like.

  In the fields of marketing, opinion polls, or Kansei engineering, questionnaire surveys are often carried out to quantitatively grasp the tendency of human subjective judgments and attitudes. Examples of questionnaire surveys include those in which respondents respond in two categories of “preferred” and “not preferable” as a result of subjective evaluation of a certain evaluation target, or from “preferred” to “not preferable” There are those that answer in the five categories. Such a subjective evaluation method is shown, for example, in Non-Patent Document 1 below.

  In subjective evaluation, there are problems specific to human subjective judgments and answers. For example, among respondents, respondents who lack knowledge about the subject to be evaluated and are unable to make an appropriate assessment of the subject, There are respondents who are non-cooperating and respond appropriately to the survey. In order to deal with this problem, it is necessary for the survey operator to visually check the response results of each respondent and omit the responses of respondents who seem to be making serious responses from the tabulation. However, it is very difficult for a person to visually confirm the answer results for a large amount of data.

  For example, Patent Document 1 below discloses a method in which the answer reliability is calculated based on objective facts about an individual respondent and only the answer results of respondents whose reliability is a certain value or more are totaled. However, this method has a problem that information on objective facts about each individual respondent is required. In addition, there is a problem that it is not possible to clearly set a standard for determining whether or not to exclude respondents from the aggregation.

  For example, the following Patent Document 2 describes a method of measuring the time required for responding to each question of the respondent and performing aggregation after assuming that the required time is long, so that the response is considered to be low. . However, this method has a problem that, for example, if a malicious respondent responds randomly, the required time does not necessarily increase, and even if the required time is long, If it is an answer, there is a problem that the reliability is not always low.

  Even if different respondents feel that they are subjectively good to the same extent, the rigor of answers varies among respondents. Therefore, there is a problem that the evaluation results of different respondents and the average values of the evaluation results of different respondent groups cannot be compared on the same scale.

  Even if the respondents are the same, it is known that the answer value has a bias depending on the combination of objects to be evaluated (for example, see Non-Patent Document 2 below). Therefore, in order to be able to compare the evaluation results of different respondents and combinations of different evaluation targets on the same scale, experienced questionnaire survey operators must select the characteristics of respondents and the combination of evaluation targets. It was necessary to conduct a questionnaire with careful consideration.

JP 2004-13277 A JP 2001-350877 A

Japanese Industrial Standards JIS Z9080: 2004 "Sensory Evaluation Analysis-Method" Laming, The Measurement of Sensation, 1997

  As described above, in order to be able to compare evaluation results for different respondents and combinations of different evaluation targets on the same scale, experienced questionnaire surveyors must be able to It is necessary to conduct a questionnaire survey with careful consideration of the target combination. Even if the survey practitioner omits the man-hours related to the above procedure, it is desirable to obtain a survey result equivalent to the survey performed according to the above procedure.

  An object of this invention is to provide the subjective evaluation apparatus, method, and program which can compare the survey result by a different respondent and a different evaluation object on the same scale.

According to the embodiment, for each of the evaluation object ID having the reference evaluation value and the evaluation object ID not having the reference evaluation value, the evaluation object database storing the evaluation object data, and the plurality of evaluation object IDs each having the reference evaluation value And the second presentation order of the plurality of evaluation object IDs each having no reference evaluation value, and integrating the first and second presentation orders, A determination unit that determines an evaluation target presentation order, and transmits the evaluation target data to a respondent terminal according to the evaluation target presentation order, and receives a response value for the evaluation target data from a respondent identified by a respondent ID. A transmission / reception processing unit, an answer result database storing the answer value in association with the evaluation target ID and the respondent ID, and the characteristic value of the answerer as the reference An answerer characteristic estimation unit that estimates based on the value and the answer value; an answerer characteristic database that stores the answerer's characteristic value in association with the answerer ID; and the answer value and the answerer's characteristic value. An evaluation value estimation unit that calculates an estimated value of an evaluation value for an evaluation object that is presented to the respondent and does not have a reference evaluation value; and an evaluation result database that stores the estimated value of the evaluation value for each evaluation object ID And an evaluation result output unit that outputs an estimated value of the evaluation value for each evaluation target ID, and the evaluation value estimation unit uses the characteristic value of the respondent stored in the respondent characteristic database. Then, when the reference evaluation value is Y, the probability that the answer value of the respondent ID _i is X _i is P (X _i | Y), and the probability distribution of Y is arbitrary P (Y), Having the reference evaluation value The estimated value of the evaluation value with respect to have evaluated,

There is provided a subjective evaluation apparatus characterized by estimating as Y that maximizes the value of.

1 is a block diagram of a system according to a first embodiment. It is a figure which shows the example of the recording content of evaluation object DB. It is a figure which shows the example of the recording content of reply result DB. It is a figure which shows the example of the recorded content of an answerer characteristic DB. It is a figure which shows the example of the content of recording of evaluation result DB. It is a flowchart showing the process of the whole system which concerns on 1st Embodiment. It is a flowchart which shows a questionnaire allocation process. It is a flowchart which shows a questionnaire presentation order determination process. It is a figure for demonstrating a questionnaire presentation order integration process. It is a flowchart which shows an answer acquisition process. It is a flowchart which shows a reply acceptance notification reception process. It is a flowchart which shows an answer result reception process. It is a figure which shows the example of the questionnaire screen in a multistep scale. It is a figure which shows the example of the questionnaire screen in a non-step scale. It is a block diagram of the system concerning a 2nd embodiment. It is a flowchart showing the process of the whole system which concerns on 2nd Embodiment. It is a graph which shows the relationship between an estimated value and an error. It is a figure which shows the example of the content of recording of evaluation result DB in 2nd Embodiment. It is a flowchart showing the process of the whole system which concerns on the modification of 2nd Embodiment. It is a flowchart which shows the questionnaire allocation process which concerns on the modification of 2nd Embodiment. It is a figure which shows the example of the screen for performing the presentation of an evaluation object, and the input of a reply.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram of a questionnaire survey implementation system. The questionnaire processing server 1 is an embodiment of the subjective evaluation apparatus, and is connected to the respondent terminals 2 a to 2 c via the network 3. The questionnaire processing server 1 includes an evaluation target input unit 4, an evaluation target DB 5, an answer result DB 6, an answerer characteristic DB 7, an evaluation result DB 8, a determination unit 9, a transmission / reception processing unit 10, an answerer characteristic estimation unit 11, an evaluator characteristic estimation unit. 12. An evaluation result output unit 13 is provided.

The respondent terminal 2 includes a transmission / reception processing unit 14, an evaluation target presentation unit 15, and a response input unit 16. The total number of respondent terminals may be an arbitrary number of one or more. The transmission / reception processing unit 14 of the respondent terminal 2 receives the evaluation target information from the questionnaire processing server 1. The information received here is presented by the evaluation object presentation unit 15, and the respondent can input an answer to the evaluation object by the answer input unit 16. FIG. 21 is an example of a screen for presenting an evaluation target and inputting an answer. In this embodiment, the respondent is “preferred (1)” or “not preferable (0)” for the presented evaluation target. ". This answer result is transmitted to the questionnaire processing server 1 by the transmission / reception processing unit 14.

  In the system configuration shown in FIG. 1, the questionnaire processing server 1 may be realized as a program executed by a computer. The computer has, for example, an input interface, a CPU, a memory, a hard disk, an output interface, etc., which are connected to a bus. The hard disk of the computer is an example of a recording medium for storing a program for realizing a questionnaire processing server, and the program is read from the hard disk to a memory via a bus and executed by the CPU.

  The evaluation target input unit 4 inputs data of an evaluation target (that is, a questionnaire) to the questionnaire processing server 1. There are two types of evaluation target data input here: data for which a standard evaluation value is assigned in advance, and data that is not. The evaluation target data may be in any format that can be presented to the respondent at the respondent terminal 2, such as a document file, a video file, and an audio file, in addition to the data provided by the image file. The reference evaluation value is a real value representing the degree of preference of each evaluation object, for example, measured in advance along a certain scale.

  The evaluation target DB 5 holds information on the evaluation target input by the evaluation target input unit 4. FIG. 2 shows an example of a list stored in the evaluation target DB 5. In this list, an evaluation object file name and a reference evaluation value are associated with an evaluation object ID unique to each of a plurality of evaluation objects. However, the reference evaluation value is stored when a value is assigned in advance, but is stored as NA (Not Available) when the value is not assigned.

  The determination unit 9 refers to the contents of the evaluation target DB 5, assigns the respondent ID to the evaluation target ID, and determines the presentation order of the evaluation targets.

  The transmission / reception processing unit 10 transmits the evaluation target file corresponding to the evaluation target ID and the information on the presentation order thereof to the respondent terminal 2 via the network 3. Further, the transmission / reception processing unit 10 receives the response result transmitted from the respondent terminal 2 via the network 3 and writes it in the response result DB 6.

  The answer result DB 6 stores the answer value in association with the evaluation target ID and the respondent ID. The list of stored data has a structure as shown in FIG. 3, for example. The answer value may be categorical data instead of real value data as shown in FIG.

  The respondent characteristic estimation unit 11 estimates the characteristics of each of the plurality of respondents. Specifically, the respondent characteristic estimation unit 11 includes the reference evaluation value of the evaluation object having the reference evaluation value in the evaluation object DB 5 and the response person DB existing in the answer result DB 6 for the evaluation object having the reference evaluation value. Based on the answer value, first and second parameters representing the characteristics of the respondent are estimated. The first and second parameters representing the estimated answerer characteristics are written to the answerer characteristics DB 7 for each answerer ID.

  The respondent characteristic DB 7 stores one or more parameters representing the characteristics of each respondent for each respondent ID. An example of the data structure of the respondent characteristic DB 7 is shown in FIG. Here, the characteristics of the respondent are represented by a total of L parameters from parameter 1 to parameter L. The number L of parameters depends on the assumed respondent characteristics. In this embodiment, L = 2.

  The evaluation value estimation unit 12 refers to the contents of the answer result DB 6 and the respondent characteristic DB 7 and estimates an evaluation value for each evaluation object. The estimated evaluation value is written in the evaluation result DB 8 together with the corresponding evaluation target ID.

  The evaluation result DB 8 stores the evaluation value for each evaluation object in association with the evaluation object ID. An example of the data structure of the evaluation result DB 8 is shown in FIG.

  The evaluation result output unit 13 refers to the contents of the evaluation result DB 8 and outputs an evaluation value for each evaluation target in a format that can be used by a questionnaire survey operator.

  The operation of the questionnaire survey implementation system configured as described above will be described in detail with reference to the flowcharts of FIGS. 6 to 8 and FIGS.

  In step M <b> 1 of FIG. 6, the questionnaire survey operator inputs information to be evaluated to the evaluation target input unit 4. Moreover, the evaluation target input unit 4 stores information input by the questionnaire survey operator in the evaluation target DB 5. Next, in step M2, the determination unit 9 refers to the content of the evaluation target DB 5 and assigns the evaluation target to the respondent.

  FIG. 7 is a flowchart showing details of the questionnaire assignment process in step M2 of FIG. First, in step D1, necessary numerical values are initialized. Specifically, the lower limit of the number of respondents assigned per evaluation object is Nmin, and the upper limit of the number of evaluation objects assigned per respondent is Mmax. These means that at least Nmin respondents are required for each evaluation object, and the number of evaluation objects assigned to each respondent is at most Mmax.

  Next, in step D2, the ID of the evaluation object that does not have a reference evaluation value and has not yet been assigned by the respondent among the plurality of evaluation objects is set as an evaluation object to be assigned the respondent. In subsequent steps D3 to D5, the ID of the respondent who evaluates the evaluation object set in step D2 is determined.

  First, in step D3, one of a plurality of respondent IDs is selected by a random number. Next, in step D4, the number of evaluation objects already assigned to the respondent ID selected in step D3 is compared with the upper limit value Mmax. If the number of evaluation targets is less than the upper limit value, the process proceeds to step D5, and if it is equal to or greater than the upper limit value, the process returns to step D3.

  In step D5, assignment of the respondent ID randomly selected in step D3 to the evaluation target ID set in step D2 is confirmed. Step D6 is a step of determining whether the number of respondent IDs assigned to the evaluation target ID set in step D2 has reached the required number. If the number of respondent IDs has reached the required number Nmin, the process proceeds to step D7. If the number of respondent IDs has not reached the required number Nmin, the process returns to step D3, and an additional respondent ID is assigned.

  Step D7 is a step of determining whether or not the respondent assignment has been performed for all evaluation targets. If the assignment of respondents has not been completed for all evaluation targets, the process returns to step D2, and if the assignment has been completed, the process ends.

  In step M3, the order of presentation to each respondent to be evaluated is determined. Here, the presentation order of the evaluation objects assigned to each respondent determined in step M2 is determined. FIG. 8 is a flowchart showing details of the questionnaire presentation order determination process in step M3 of FIG. In step C1, respondents who have not yet set the evaluation target presentation order are selected from all respondents in subsequent steps C2 to C6.

  In step C2, the presentation order of the evaluation objects having the reference evaluation value is generated according to the random number. In step C3, a correlation coefficient between the presentation order generated in step C2 and the reference evaluation value is calculated. Next, in step C4, it is determined whether the absolute value of the correlation coefficient between the reference evaluation value and the presentation order is less than a preset allowable range. If the absolute value of the correlation coefficient is greater than or equal to the set numerical value, the process returns to step C2 and the presentation order is generated again. If the absolute value of the correlation coefficient is less than the set numerical value, the presentation order generated in step C2 is adopted, and the process proceeds to step C5. As a result, when there is a correlation between the presentation order and the reference evaluation value, it is impossible to separate the change in the answer value due to the difference in the evaluation target and the change in the answer value due to the presentation order (a phenomenon called order effect). Can be prevented.

  Next, in step C5, for the respondent selected in step C1, the order of presentation of evaluation targets without reference evaluation values is generated according to random numbers.

  In step C6, the presentation order of the evaluation objects for which the reference evaluation value does not exist and the presentation order of the evaluation objects for which the reference evaluation value exists are integrated. For example, as shown in FIG. 9, an evaluation object presentation order 92 is generated by integrating an evaluation object presentation order 90 having a reference evaluation value and an evaluation object presentation order 91 having no reference evaluation value. In this case, it inserts in the row | line | column of evaluation object presentation without a reference | standard evaluation value so that the evaluation object with a reference | standard evaluation value may become equal intervals.

  In Step C7, it is determined whether or not the evaluation target presentation order has been set for all the respondents. If the setting of the evaluation object presentation order has not been completed for all the respondent IDs, the process returns to step C1, and the process is performed for the respondent ID for which the evaluation object presentation order has not been set. When the evaluation target presentation order is completed for all respondents, the process is terminated.

  When the process of step M3 in FIG. 6 is completed, the setting of the respondent IDs to be evaluated is completed for all the evaluation target IDs, and what evaluation target (questionnaire) is set for each respondent ID. This completes the setting of the order of presentation.

  Instead of generating the presentation order using random numbers as described above, the determination unit 9 may hold a presentation order table in advance and assign it.

  Next, in step M4, the transmission / reception processing unit 10 transmits a question to the respondent terminal 2 via the network 3 based on the respondent ID, the evaluation target ID, and the questionnaire presentation order set in the immediately preceding steps M2 and M3. The respondent terminal 2 presents a question to the respondent, and transmits the answer result from each respondent to the questionnaire processing server 1 via the network 3. The transmission / reception processing unit 10 receives the response result from the respondent terminal and writes it in the respondent DB 6. Details of the processing here will be described with reference to the flowcharts of FIGS. 10, 11, and 12.

  FIG. 10 is a flowchart showing details of the answer acquisition process in step M4. FIG. 11 and FIG. 12 show the contents of processing executed when the answer acceptance notification or answer result transmitted from the respondent terminal is received during the answer acquisition process shown in FIG.

  In step A <b> 1 of FIG. 10, the transmission / reception processing unit 10 transmits a response to the questionnaire to the respondent terminal 2. The response request may be notified to the registered respondent panel by e-mail or the like, or may be in the form of recruiting an unspecified number of respondents on the Web page. In step A <b> 2, the questionnaire processing server 1 transitions to a state of waiting for a response from the respondent terminal 2. When a reply acceptance notification is received from the respondent terminal 2 in a state of waiting for reply reception, a reply acceptance notification reception process shown in FIG. 11 is executed. Similarly, when an answer result is received, an answer result receiving process shown in FIG. 12 is executed.

  Next, in step A3, the passage of time is counted, and the processing after step A4 is executed at regular time intervals.

  In step A4, it is determined whether or not the answer reception waiting state is set. If the answer reception waiting state is set, the process proceeds to step A5, and if the answer reception waiting state is released, the process ends. In step A5, it is determined whether there is a respondent terminal that has received a reply acceptance notice and has assigned a respondent ID but has not received a reply result after a certain period of time. If there is a corresponding respondent terminal, the process proceeds to step A6, and if not, the process returns to step A3. In step A6, among the respondent terminals 2 to which the respondent ID has been assigned, the respondent terminals that have passed a certain time but have not received the answer result are set as time-out terminals, and the assignment of the respondent ID is cancelled.

  The processing when the answer acceptance notice is received in the answer reception waiting state in step A2 will be described with reference to the flowchart of FIG. First, in step J1, the identification information of the respondent terminal 2 that has transmitted the answer acceptance notice is referred to, and it is determined whether the answer result from the respondent identification terminal 2 has already been received. If the answer result has been received, the process proceeds to step J4. If the answer result has not been received, the process proceeds to step J2.

  In step J2, a respondent ID that has not been assigned to the respondent terminal 2 is selected from the list of respondent IDs, and the same ID is assigned to the respondent terminal 2. Next, in Step J3, the question set in Steps M1 and M2 and including the information about the evaluation target corresponding to the respondent ID and the presentation order thereof is transmitted to the respondent terminal 2 via the network 3.

  Step J4 is executed when it is determined in step J1 that the answer result from the respondent terminal 2 has already been received. The respondent terminal 2 is notified via the network 3 that the answer cannot be accepted because it has already been answered.

  Next, processing when an answer result is received in the answer reception waiting state in step A2 will be described with reference to the flowchart of FIG. First, in step K1, the answer result for each evaluation object received by the transmission / reception processing unit 10 is written in the answer result DB 6 in association with the respondent ID and the evaluation object ID.

  Next, in step K2, it is determined whether or not the answer results from all the respondent IDs assigned the evaluation target ID have been received. If received, the process proceeds to step K3, and if not received, the process returns from the answer result receiving process. Step K3 is executed when it is determined in step K2 that the answer results from all the respondent IDs have been received, and the answer reception waiting state of the questionnaire processing server 2 is canceled.

  When the above-described answer acquisition process in step M4 is completed, answer values corresponding to all evaluation target IDs and respondent IDs are recorded in the answer result DB6.

  Next, in step M5, the respondent characteristic estimation unit 11 refers to the contents of the response result DB 6 and the evaluation target DB 5, and estimates the responder characteristics corresponding to each respondent ID.

  The method for estimating the respondent characteristics will be described in detail.

The value of the answer result of a certain respondent ID _i is X _i = x _ij (the subscript _j corresponds to the evaluation target ID _j ). Further, the value of the reference evaluation value corresponding to the evaluation object ID _j where the reference evaluation value exists is Y = y _j . In this embodiment, the value that X _i can take is binary (“0” or “1”), the value that Y can take is an arbitrary real value, and the respondent characteristics are two parameters of item response theory A case corresponding to a logistic regression model will be described. The item response theory is described in, for example, the reference “Science of Test and Measurement—Item Response Theory [Introduction], Hideki Toyoda, 2002”.

The following regression equation is obtained from the values of Y and X _i corresponding to the evaluation object ID _j where the reference evaluation value exists.

The parameters obtained here are P _1i and P _2i , for example,

Then, the values of P _1i and P _2i that minimize E can be obtained by a general numerical optimization method such as Newton's method.

The first parameter P _1i thus determined corresponds to the discriminating power of the 2-parameter model in the item response theory, and the second parameter P _2i corresponds to the difficulty level. Usually, in item response theory, these parameters are set for questions. However, in the present embodiment, these parameters are set for the respondent. In this case, the first parameter is a parameter that indicates how appropriate the respondent is performing the evaluation, and the second parameter is a parameter that indicates how severe the respondent is performing the evaluation. .

Then, the regression equation (1) can be interpreted as “the probability P (X _i = 1 | Y) that the respondent i answers“ 1 ”when the value corresponding to the reference evaluation value is Y”). . Further, since the probability of answering “0” can be expressed by 1-P (X _i = 1 | Y), P (X _i | Y) is expressed by the following equation (3).

In this way, the first parameter P _1i and the second parameter P _2i representing the respondent characteristics are obtained for all the respondent IDs, and stored in the respondent characteristics DB 7 in association with the respondent IDs.

  Next, in step M6 of FIG. 6, the evaluation value estimation unit 12 refers to the contents of the answer result DB 6 and the respondent characteristic DB 7, and estimates an evaluation value for an evaluation object for which no reference evaluation value exists. This estimation value estimation method will be described in detail.

Let Y _j (value to be obtained) be an evaluation value for an evaluation target ID _j for which no reference evaluation value exists. When the value corresponding to the reference evaluation value is Y, the probability that the answer value of the respondent i is “1” or “0” is expressed by Expression (3). Therefore, when there are answer results for the evaluation target ID _j of a plurality of respondents i, the probability of obtaining the answer result is represented by the product of the respective probabilities.

Represented by Thus, ^ _{y j} estimate _{y j,} when put the probability distribution of Y and P (Y), from Bayes' theorem

It can ask for. This can be obtained by a general numerical optimization method such as Newton's method because the function has only to be maximized with respect to Y. Note that an arbitrary probability distribution including a uniform distribution can be used for P (Y).

  Here, the expression (5) includes a first parameter and a second parameter representing the characteristics of each respondent. Therefore, by estimating the evaluation value using the equation (5), it is possible to estimate the evaluation value in consideration of the appropriateness of each respondent's answer and the severity of the evaluation. This value can be handled on the same scale as the reference evaluation value.

After obtaining an evaluation value according to the expression (5) for the evaluation object ID _j for which no reference evaluation value exists, the value is associated with the evaluation object ID and written in the evaluation result DB 8.

  Finally, in Step M7 of FIG. 6, the evaluation result output unit 13 outputs the evaluation value of the evaluation target obtained up to Step M6 in a format that can be used by the questionnaire survey operator.

  Through the above process, even if the severity and appropriateness of evaluation differ from respondent to respondent, experienced questionnaire surveyors carefully design questionnaire surveys with highly accurate results that can be compared on the same scale. You can get without doing.

(Modification)
[Respondent characteristics replacement example 1 (multi-stage scale)]
The respondent characteristics estimated by the respondent characteristic estimation unit 11 and used by the evaluation value estimation unit 12 for estimation of the evaluation value may be replaced with the following.

  FIG. 13 is an example of a questionnaire survey screen based on a multi-stage scale, and the number of categories that can be selected at the time of answering is 3 or more. In this case, the contents of the process in step M5 in FIG. 6 are replaced as follows.

It is assumed that the number of categories that can be selected is C, and the possible values of the answer value X are 0, 1,. When _{| (Y X i = c)} , the probability that _{X i ≧ c P (X i} ≧ c | evaluation value corresponding to the reference evaluation value is the probability that _X i = c _P when was Y Y )Using,

It can be expressed as. Here, P (X _i ≧ 0 | Y) = 1 and P (X _i ≧ C | Y) = 0. For 1 ≦ c ≦ C−1, for example, assuming a 2-parameter logistic model,

It can be expressed as. These parameters P _1i and P _2ic (c = 1,..., C−1) are parameters representing the respondent characteristics.

As described above, P _1i and P _2ik that minimize E can be obtained, and can be solved by a general numerical optimization method such as Newton's method. However,

It is. When solving this optimization, a constraint condition of P _2i1 <P _2i2 <... <P _2iC-1 may be imposed.

In this modification, the parameter P _{1i is} a parameter indicating how appropriate the respondent is performing the evaluation, and the parameter P _2ic is how severely the respondent performs the evaluation at each stage. It becomes a parameter indicating whether or not.

In the process of step M6, the evaluation value may be estimated by replacing P (X _i | Y) with equation (6).

  By replacing the respondent characteristics as described above, even if there are 3 or more options in the response and the severity and appropriateness of evaluation differ from respondent to respondent, the accuracy can be compared on the same scale. Results can be obtained by experienced survey practitioners without carefully designing the survey.

[Respondent characteristics replacement example 2 (stepless scale)]
The respondent characteristics estimated by the respondent characteristic estimation unit 11 and used by the evaluation value estimation unit 12 for estimation of the evaluation value may be replaced with the following.

  FIG. 14 is an example of a questionnaire survey screen based on a stepless scale, and the answer value is a real value. In this case, the process in step M5 in FIG. 6 is replaced with the following.

First, when the reference evaluation value is Y, the regression equation of the response value X _i of the respondent ID _i is set to a linear form such as the following equation (10).

, P _1i and P _2i are obtained so as to minimize E.

And

To obtain P _3i . Here, J represents the number of data used when calculating Expression (11).

Then, the probability P (X _i | Y) that the answer value of the respondent ID _i is X _i when the reference evaluation value is Y is assumed to have a normal distribution in error.

It becomes. Parameters representing the characteristics of the respondent i are P _1i , P _2i , and P _3i . In this modification, the severity and appropriateness of each respondent's answer can be expressed by the three parameters P _1i , P _2i , and P _3i .

  Although an example of a linear linear regression equation is shown here, an arbitrary regression equation can be used. Further, the result of the step response value can be regarded as a numerical value, and this processing can be applied.

Moreover, in the process of step M6, P (X _i | Y) may be replaced with equation (13) to estimate the evaluation value.

  According to the replacement of respondent characteristics as described above, even if the response value of the stepless scale is adopted and the severity and appropriateness of evaluation differ from respondent to respondent, the same scale is used for comparison. Highly accurate results that are possible will be obtained by experienced questionnaire practitioners without careful questionnaire design.

[Respondent characteristics replacement example 3 (consideration of presentation order)]
The respondent characteristics estimated by the respondent characteristic estimation unit 11 and used by the evaluation value estimation unit 12 for estimation of the evaluation value may be replaced with the following. Here, an example of replacement when the respondent characteristics change will be described.

In the above-described embodiment, it is assumed that the respondent characteristics are always constant, but there is a possibility that the characteristics of the respondents fluctuate with time (answer order). Thus, for example, P _1i and P _2i in equation (1) are

In this way, the time can be set as a parameter. Here, T represents the presentation order of evaluation objects. The order of presentation of the evaluation object ID _j to the respondent i is set as T = t _ij ,

If the parameters P _{1i_1} , P1 _{i_2} , P _{2i_1} , and P _{2i_2} are determined so as to minimize E, it is possible to express how the respondent characteristics change with time. The parameter can be obtained by a general numerical optimization method.

  In addition, the linear linear expression expressed by the formula (14) or the formula (15) is used as an example for expressing the change of the respondent characteristic parameter, but the function for expressing the change is not limited to this. .

  According to this modification, it is possible to express the respondent characteristics when the respondent characteristics change with time. In particular, this modification is suitable when there are a large number of evaluation targets to be evaluated for each respondent.

(Second Embodiment)
Hereinafter, the second embodiment will be described. FIG. 15 is a block diagram of a system according to the second embodiment. Description of the blocks having the same processing contents as those in the first embodiment is omitted.

  The evaluation value error estimation unit 17 estimates the evaluation value error value to be evaluated based on the evaluation value estimated value by the evaluation value estimation unit 12 and notifies the evaluation value estimation unit 12 of the estimated value. The evaluation value estimation unit 12 records the estimated value of the evaluation value error in the evaluation result DB 8 in association with the estimated value of the evaluation value and the evaluation target ID. The evaluation result output unit 13 refers to the contents of the evaluation result DB 8 and outputs an evaluation value and an evaluation value error for each evaluation object in a format that can be used by a questionnaire survey operator.

  The flow of processing in this embodiment will be described in detail with reference to the flowchart of FIG. Except for the processing of step R7, the processing is the same as the processing in the first embodiment described with reference to FIG.

  In step R7, an evaluation value is estimated for an evaluation object for which no reference evaluation value exists, and its error is also estimated at the same time. A method for estimating the error of the evaluation value will be described in detail.

An error from the actual value Y exists in Y ^ when the evaluation value is estimated. FIG. 17 is a graph showing the shape of the following equation.

If the total area under the graph curve is A and the area from Y ^-[epsilon] to Y ^ + [epsilon] is B, the probability that Y falls within the range of Y ^ ± ε is B / A. For example, assuming a normal distribution for the shape of the graph, the error is less than ± ε with 95% probability.

Is true.

  Therefore, in step R7, the evaluation value error estimator 17 calculates the value of ε for which the equal sign of Expression (17) holds based on the evaluation value estimated for each evaluation object. This can be obtained by a general numerical calculation such as Newton's method. The estimated error value thus obtained is recorded as error information in the evaluation result DB 8. In the present embodiment, the contents recorded in the evaluation result DB 8 have a structure as shown in FIG.

  Next, in step R8, the obtained evaluation value of the evaluation object and the estimated value of the evaluation value error are output in a format that can be used by the questionnaire survey operator. According to this embodiment, it is possible to estimate the error of the evaluation value together with the evaluation value of the evaluation target, and the survey operator can know the accuracy of the questionnaire survey result.

(Modification of the second embodiment)
Subsequently, a modified example of the second embodiment will be described.

  In this modification, the determination unit 9 refers to the evaluation value error information in the evaluation result DB 8 and performs additional assignment of respondents to the evaluation target. The flow of processing in this modification will be described with reference to the flowcharts of FIGS. 19 and 20. In the processing of this modification, description of the same processing as that of the second embodiment described above will be omitted.

  FIG. 19 is a flowchart showing the overall processing flow in this modification. In step E8, the error information corresponding to each evaluation object recorded in the evaluation result DB 8 is compared with a preset threshold value. If Expression (17) holds for all the evaluation objects, the process proceeds to Step E9. If there is an evaluation object that does not hold (an evaluation object whose evaluation value does not reach the required accuracy), the process proceeds to Step E2.

  When step E2 is reached via step E8, a new respondent is additionally assigned to the evaluation target. Note that the evaluation target to which the respondent is newly assigned may be limited to the evaluation target determined in step E8 that the evaluation value has not reached the required accuracy.

  The process at step E2 will be described with reference to the flowchart of FIG. Note that description of processing that is the same as that described above with reference to FIG. 7 is omitted. In step K3, the evaluation value error (estimated value) corresponding to the evaluation object ID set in step K2 is compared with a predetermined specified value. If the evaluation value error is greater than or equal to the specified value, the process proceeds to step K4. If the evaluation value error is less than or equal to the specified value, the process proceeds to step K2 without assigning a new respondent to the evaluation target.

  By adding the above processing, it is possible to assign a new respondent to the evaluation object whose evaluation value error is equal to or greater than the specified value, and perform the questionnaire survey again.

  According to the present modification, when the estimated value of the evaluation value to be evaluated does not reach the required accuracy in the first estimation, it is possible to newly assign a respondent and estimate the evaluation value in advance, An evaluation value that guarantees the set accuracy can be estimated.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

DESCRIPTION OF SYMBOLS 1 ... Questionnaire processing server 2a-2c ... Respondent terminal 3 ... Network 4 ... Evaluation object input part 5 ... Evaluation object database 6 ... Answer result database 7 ... Respondent characteristic database 8 ... Evaluation result database 9 ... Evaluation object selection control part 10 ... Transmission / reception processor (server side)
DESCRIPTION OF SYMBOLS 11 ... Respondent characteristic estimation part 12 ... Evaluation value estimation part 13 ... Evaluation result output part 14 ... Transmission / reception processing part (respondent terminal side)
15 ... Evaluation target presentation unit 16 ... Answer input unit 17 ... Evaluation value error estimation unit

Claims (9)

An evaluation object database that stores evaluation object data for each of an evaluation object ID having a reference evaluation value and an evaluation object ID having no reference evaluation value;
Generating a first presentation order of a plurality of evaluation object IDs each having a reference evaluation value and a second presentation order of a plurality of evaluation object IDs each having no reference evaluation value; A determination unit that determines the presentation order of one evaluation object by integrating the presentation order of
A transmission / reception processor that transmits the evaluation target data to the respondent terminal according to the evaluation target presentation order, and receives a response value for the evaluation target data from the respondent identified by the respondent ID;
An answer result database for storing the answer value in association with the evaluation target ID and the answerer ID;
A respondent characteristic estimation unit that estimates the characteristic value of the respondent based on the reference evaluation value and the response value;
A respondent characteristic database for storing the characteristic value of the respondent in association with the respondent ID;
Based on the answer value and the characteristic value of the respondent, an evaluation value estimation unit that calculates an estimated value of an evaluation value that is presented to the respondent and does not have a reference evaluation value;
An evaluation result database for storing the estimated value of the evaluation value for each evaluation target ID;
An evaluation result output unit that outputs an estimated value of an evaluation value for each evaluation target ID,
The evaluation value estimation unit uses the characteristic value of the respondent stored in the respondent characteristic database, and when the reference evaluation value is Y, the answer value of the respondent ID _i is X _i The probability P (X _i | Y), the probability distribution of Y as an arbitrary P (Y), and the estimated value of the evaluation value for the evaluation object not having the reference evaluation value,

A subjective evaluation apparatus characterized by estimating as Y that maximizes the value of. The possible value of the answer value X _{i is} a binary value of 1 or 0, the reference evaluation value of the evaluation object ID _j having the reference evaluation value is y _j , and the answer value of the respondent ID _i for the evaluation object ID _j is X _{When i} = x _ij

The respondent characteristic estimation unit obtains parameters P _1i and P _2i that minimize E as the characteristic value of the respondent,
The evaluation value estimation unit calculates the P (X _i | Y) using the parameters P _1i and P _2i .

The apparatus according to claim 1, obtained by: The possible values of the answer value X _i are C discrete values (X _i = c = {0, 1,..., C−1}, C ≧ 3), and an evaluation target ID having a reference evaluation value _When the reference evaluation value of _j is y _j and the answer value of the respondent ID _i is x _ij ,

Toki,

The respondent characteristic estimation unit obtains parameters P _1i and P _2ic (c = 1,..., C-1) that minimize E as the characteristic value of the respondent,
The evaluation value estimation unit uses the parameters P _1i and P _2ic.

And P (X _i | Y) is

The apparatus according to claim 1, obtained by: When the reference evaluation value of the evaluation target ID _j having the reference evaluation value is y _j and the answer value of the respondent ID _i is x _ij ,

The respondent characteristic estimation unit obtains a parameter P _1i and a parameter P _2i that minimize E as the characteristic value of the respondent,

To obtain the parameter P _3i (where J represents the number of data used in calculating equation (8)),
The evaluation value estimation unit calculates the P (X _i | Y) using the parameters P _1i , P _2i and P _3i ,

The apparatus according to claim 1, obtained by:   The apparatus according to claim 1, wherein the characteristic value of the respondent is a function of the evaluation object presentation order.   The apparatus according to claim 1, further comprising an evaluation value error estimator that estimates an error of the estimated value of the evaluation value.   The apparatus according to claim 1, wherein when the error is larger than a specified value, another respondent is added to an evaluation target ID that does not have the reference evaluation value. The evaluation object database stores evaluation object data for each of an evaluation object ID having a reference evaluation value and an evaluation object ID having no reference evaluation value;
The determining unit generates a first presentation order of a plurality of evaluation object IDs each having a reference evaluation value and a second presentation order of a plurality of evaluation object IDs each having no reference evaluation value, Determining one evaluation target presentation order by integrating the first and second presentation orders;
A transmission / reception processing unit transmitting the evaluation target data to the respondent terminal according to the evaluation target presentation order, and receiving a response value for the evaluation target data from the respondent identified by the respondent ID;
An answer result database storing the answer value in association with the evaluation target ID and the respondent ID;
A step of estimating a characteristic value of a respondent identified by the respondent ID based on the reference evaluation value and the response value;
A respondent characteristic database storing the characteristic value of the respondent in association with the respondent ID;
An evaluation value estimating unit, based on the answer value and the characteristic value of the respondent, calculating an estimated value of an evaluation value that is presented to the respondent and does not have a reference evaluation value;
An evaluation result database storing an estimated value of the evaluation value for each evaluation target ID; and
An evaluation result output unit that outputs an estimated value of an evaluation value for each evaluation target ID; and
The evaluation value estimation unit uses the characteristic value of the respondent stored in the respondent characteristic database, and when the reference evaluation value is Y, the response value of the respondent ID _i is X _i The probability P (X _i | Y), the probability distribution of Y as an arbitrary P (Y), and the estimated value of the evaluation value for the evaluation object not having the reference evaluation value,

A subjective evaluation method characterized by estimating as Y that maximizes the value of. Computer
An evaluation object database that stores evaluation object data for each of an evaluation object ID having a reference evaluation value and an evaluation object ID having no reference evaluation value;
Generating a first presentation order of a plurality of evaluation object IDs each having a reference evaluation value and a second presentation order of a plurality of evaluation object IDs each having no reference evaluation value; A determining unit that determines the presentation order of one evaluation object by integrating the presentation order of
A transmission / reception processor that transmits the evaluation target data to the respondent terminal according to the evaluation target presentation order, and receives a response value for the evaluation target data from the respondent identified by the respondent ID;
An answer result database for storing the answer value in association with the evaluation target ID and the respondent ID;
A respondent characteristic estimation unit that estimates a characteristic value of a respondent identified by the respondent ID based on the reference evaluation value and the response value;
A respondent characteristic database for storing the characteristic value of the respondent in association with the respondent ID;
An evaluation value estimation unit that calculates an estimated value of an evaluation value for an evaluation object that is presented to the respondent and does not have a reference evaluation value based on the answer value and the characteristic value of the respondent;
An evaluation result database for storing the estimated value of the evaluation value for each evaluation target ID;
An evaluation result output unit that outputs an estimated value of the evaluation value for each evaluation target ID;
Function as
The probability P (X _i | Y) that the answer value of the respondent ID _i is X _i when the reference evaluation value is Y, using the characteristic value of the respondent stored in the respondent characteristic database. ), The probability distribution of Y as an arbitrary P (Y), and an estimated value of an evaluation value for an evaluation object that does not have the reference evaluation value,

A program for causing a function to be estimated as Y that maximizes the value of.

Images