KR101953839B1 - Method for estimating updated multiple ranking using pairwise comparison data to additional queries - Google Patents

Abstract

According to an embodiment of the present invention, there is provided a method of estimating multiple rankings using pairwise comparison data, comprising: inputting information on pairwise comparison data as first input data; Calculating multiple ranking data based on the input data; Selecting an additional query based on the calculated multiple ranking data; Receiving second input data relating to pairwise comparison data for the further query; And updating the multi-ranking data based on the first and second input data, wherein at least one of the first and second input data includes a set of a plurality of respondents, a set of a plurality of items, And a plurality of pairs of comparison data for a plurality of items, wherein each pair of comparison data of the plurality of pairs of comparison data includes data for determining a degree of preference between two items according to one of the plurality of determination criteria The multi-rank estimating method according to the present invention.

Description

[0001] The present invention relates to an updated multiple ranking estimation method using pairwise comparison data for an additional query,

The present invention relates to a ranking estimation method and apparatus, and more particularly, to a method for estimating updated multiple rankings by considering pairwise comparison results for an additional query.

Ranking learning is a method of learning a ranking model by using a given set of items and learning data and finding the rankings (preference rank) of items by using the learned results. Recently, this method has been widely used in information retrieval and recommendation systems such as finding the document most similar to a given document.

There are various types of learning data that can be used for ranking learning, among which there is a ranking learning method that utilizes the results of pairwise comparisons (comparing ranking among two items). For example, suppose that there are 4 items (A, B, C, D), and the results of five pairs of comparisons are given as in FIG. "B <A" means that B is higher than A. In this case, the number of rankings for all items is 4! = 24. It is possible to determine which ranking is more accurate by using how well the result of the pair comparison is reflected in the number of these 24 cases as the evaluation scale. In the case of FIG. 1, since the two results of B <A <C <D and A <B <C <D best reflect the results of five pairs of comparison, the number of these two cases can be used as the actual ranking.

In the above-described ranking learning, it is assumed that the ranking among all the items is determined according to only one criterion. However, the multiple ranking problem assumes that the ranking among all the items may exist differently according to a plurality of judgment criteria (also referred to as " dimension ") and aims at obtaining all the rankings according to each judgment criterion.

For example, as shown in FIG. 2, it is assumed that the hotel's room rate and the distance to the center are given. In this example, the criterion (dimension) is the price and the distance. If the higher the ranking is, the higher the ranking is, the higher the ranking is, the higher the ranking is. If the ranking is higher, the rankings for the accommodation fees are A <B <D <C and the ranking for the distances is C <B <A <D.

In the above example, it is assumed that the judgment criterion such as 'accommodation fee' and 'distance' is already known. However, in the actual multi-ranking problem, it is impossible to know which criterion of each pair is made, In many cases, it is necessary to determine the ranking according to the potential judgment criteria. The actual pairwise comparisons are conducted by most people, and people are more likely to be more practical than the existing single-rank learning problem because they rank items according to specific criteria according to individual tendencies.

For example, when the list of FIG. 2 exists, it is assumed that the result data of the pair comparison as shown in FIG. 3 is obtained from the four respondents (u ₁ to u ₄ ). In this case, it is assumed that each respondent does not know how to make a pairwise comparison based on a certain criterion, but only a pairwise comparison result is given as shown in FIG. As shown in FIG. 3, the result of the comparison between the respondent (u ₁ ) and the respondent (u ₂ ) is very different. It can be inferred that respondents (u ₁ ) comparisons were based on the rate of accommodation, while respondents (u ₂ ) comparisons were based on distance. Since the number of pairwise comparisons that are inconsistent when data of pairwise comparison results are given in this way may be very large, it is difficult to obtain an accurate ranking through the existing single ranking learning. The algorithm that infer the inter-item ranking using the results of the existing pairwise comparison is useful when only one dimension value is used. However, when applied to the recommendation system, the actual users judge the ranking according to various judgment criteria, There is a limit that can not be obtained.

On the other hand, when executing an algorithm for estimating a ranking according to various judgment criteria based on the result of pairwise comparison, the data of all pairs collected so far may not provide sufficient information for calculating the pairwise comparison result. For example, when the comparison data between certain comparison objects (items) is insufficient or missing, the accuracy of the comparison results of the pair will not be high.

Therefore, if the initial input data is insufficient, it may be necessary to further collect pairwise comparison data. In the prior art, it is necessary to collect a comparison result of a certain query to a respondent so as to appropriately determine whether the accuracy of the pair- .

Also, when the new pairwise comparison data is added to the existing pairwise comparison result calculated from the initial input data, the algorithm for calculating the pairwise comparison result must be executed again from the beginning by inputting the existing initial input data and the additional input data again Therefore, there is a problem that each time the additional comparison data is acquired, the result calculation time of the multiple ranking data is consumed.

Korean Patent No. 10-1605654 (issued April 4, 2016) U.S. Patent No. 7,617,164 (published on November 10, 2009)

According to an embodiment of the present invention, there is provided a multiple ranking estimation method capable of estimating a ranking of items according to a plurality of judgment criteria when a plurality of judgment criteria exist in a given pairwise comparison result.

According to an embodiment of the present invention, there is provided a multiple ranking estimation method that can present an optimal query to an optimal respondent, obtain additional input data from the optimal query, and reflect the input data to the multiple ranking algorithm to increase the accuracy of the pair comparison result.

The method of estimating multiple rankings according to an embodiment of the present invention can be applied to a crowdsourcing system. Existing crowd sourcing systems can not select respondents or select questions according to requester's preference, but the multi-rank estimation method of an embodiment of the present invention is based on the data collected so far and / or the calculated multiple ranking data The present invention provides a multiple ranking estimation method capable of enhancing the accuracy of multi-rank estimation because it collects additional input data by selecting a target of a query and a query.

According to an embodiment of the present invention, there is provided a method of estimating multiple rankings using pairwise comparison data, comprising: inputting information on pairwise comparison data as first input data; Calculating multiple ranking data based on the input data; Selecting an additional query based on the calculated multiple ranking data; Receiving second input data relating to pairwise comparison data for the further query; And updating the multi-ranking data based on the first and second input data, wherein at least one of the first and second input data includes a set of a plurality of respondents, a set of a plurality of items, And a plurality of pairs of comparison data for a plurality of items, wherein each pair of comparison data of the plurality of pairs of comparison data includes data for determining a degree of preference between two items according to one of the plurality of determination criteria The multi-rank estimating method according to the present invention.

According to another aspect of the present invention, there is provided a computer-readable recording medium having recorded thereon a program for causing the computer to execute the multiple rank estimation method.

According to an embodiment of the present invention, when there are a plurality of judgment criteria in a given pairwise comparison result, it is possible to estimate the ranking of items according to a plurality of judgment criteria, and furthermore, It is advantageous to provide it.

According to an embodiment of the present invention, there is an advantage that the optimal query can be presented to the optimum respondent, the additional input data can be obtained therefrom, and the resultant data can be reflected to the multi-ranking algorithm.

1 to 3 are diagrams for explaining a conventional ranking learning method,
FIG. 4 is a flow chart for explaining a multiple rank estimation method using pair comparison data according to an embodiment of the present invention; FIG.
5 is a flow chart illustrating an exemplary method of a multiple ranking learning algorithm,
FIG. 6A is a diagram for explaining a probability distribution of determination criterion according to an exemplary embodiment; FIG.
FIG. 6B is a view for explaining an item score according to an embodiment,
6C is a diagram for describing a response accuracy probability distribution according to an embodiment,
7 shows a pseudo code that briefly shows how to update multiple ranking data,
8 is a diagram for explaining experimental results of a multiple rank estimation method according to an embodiment,
FIG. 9 is a block diagram illustrating an exemplary apparatus configuration for implementing a multiple rank estimation method according to an embodiment.

BRIEF DESCRIPTION OF THE DRAWINGS The above and other objects, features, and advantages of the present invention will become more readily apparent from the following description of preferred embodiments with reference to the accompanying drawings. However, the present invention is not limited to the embodiments described herein but may be embodied in other forms. Rather, the embodiments disclosed herein are provided so that the disclosure can be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

Where the terms first, second, etc. are used herein to describe components, these components should not be limited by such terms. These terms have only been used to distinguish one component from another. The embodiments described and exemplified herein also include their complementary embodiments.

In the present specification, the singular form includes plural forms unless otherwise specified in the specification. The terms "comprise" and / or "comprising" used in the specification do not exclude the presence or addition of one or more other elements.

Hereinafter, the present invention will be described in detail with reference to the drawings. In describing the specific embodiments below, various specific details have been set forth in order to explain the invention in greater detail and to assist in understanding it. However, it will be appreciated by those skilled in the art that the present invention may be understood by those skilled in the art without departing from such specific details. In some cases, it should be mentioned in advance that it is common knowledge in describing an invention that parts not significantly related to the invention are not described in order to avoid confusion in explaining the present invention.

The present invention models the process by which respondents respond to a pairwise comparison between two items into a probability model. In one embodiment, the multiple ranking data used for modeling this probability model is used to determine the probability that a person will give a correct answer for each criterion, the weight of each person's preferred criterion, and the rank- Values, and the like. In one embodiment, a likelihood function may be obtained that indicates the probability that the results of the actual pairwise comparisons will be obtained according to this multiple ranking data value. The multi-ranking data for maximizing the likelihood function value is calculated by using an expected value maximization algorithm (EM algorithm), and the ranking among the items for each determination criterion is calculated based on the ranking score for each item of the multiple ranking data do.

FIG. 4 is a flowchart illustrating a method for estimating multiple rankings using pair-by-pair comparison data according to an embodiment of the present invention.

In one embodiment, a method of estimating multiple rankings using pairwise comparison data may be performed in a multiple ranking learning algorithm 100. In one embodiment, the multi-rank learning algorithm 100 may include a multiple ranking data calculation unit 110, a query selection unit 120, and an update unit 130, Hardware, or a combination thereof.

The method for estimating multiple rankings according to an embodiment includes the steps of inputting information on pair comparison data as input data to a multiple ranking data calculation unit 110 and a step of calculating multiple ranking data by multiply ranking data calculation unit 110 based on the input data And generating ranking data.

In this case, the input data input to the multi-ranking learning algorithm may include, for example, a set of a plurality of responders, a set of a plurality of items, and a set of a plurality of pairs of comparison data for a plurality of items. Each pair-by-pair comparison data of a plurality of pairs of comparison data is data that determines a preference between two items according to an arbitrary one of a plurality of determination criteria.

The multi-ranking data generated by the multi-ranking data calculation unit 110 includes, for example, a probability distribution ("judgment criterion probability probability distribution") of probability that the respondent prefers an arbitrary judgment criterion, ("Item score"), and a probability distribution ("response accuracy probability distribution") of the probability that the responder correctly answered for each criterion. Particularly, ranking results among the items according to judgment criteria can be derived from item scores among the multiple ranking data generated at this time.

In the illustrated embodiment, the query selection unit 120 selects further questions to be queried based on the generated multiple ranking data. Based on the existing input data and / or the generated multiple ranking data, the query selection unit 120 determines which query is the most effective to inquire to which respondent the query is to acquire additional input data from.

A predetermined number of queries selected by the query selection unit 120 are given to a predetermined number of respondents, and receive new additional input data related to the pair comparison result. The updating unit 130 may then update the multiple ranking data based on the additional input data and the existing input data.

Hereinafter, one embodiment of an exemplary learning method according to a multiple rank learning algorithm executed in the multiple ranking data calculation unit 110 will be described in detail with reference to FIG. 5 and FIG.

5 is a flow diagram illustrating an exemplary method of a multiple ranking learning algorithm.

Referring to the drawings, a method for generating multiple ranking data by a multiple ranking learning algorithm includes calculating a probability of a pairwise comparison result for any two items based on input data and prior multiple ranking data Calculating a likelihood function based on the probability (S120), and calculating (S130) posterior multiple ranking data for maximizing the likelihood function.

In step S110, the probability of the pair comparison result is calculated using the input data and the dictionary multi-ranking data. The dictionary multi-ranking data used at this time is a prior decision criterion preference probability distribution, a dictionary item score, and a pre-response accuracy probability distribution. The pre-multilingual data may be an arbitrarily set estimation value or a value calculated based on actual input data. The multiple ranking data used in step S110 is the estimated probability distribution and the estimated item score and then the posterior multiple ranking data (i.e., , Posterior criterion preference probability distribution, post item score, and post-response accuracy probability distribution).

That is, in a preferred embodiment, the present invention uses Bayes' Theorem to derive posterior probability information using a prior probability and an observation value (i.e., input data) The posterior (actual) probability distribution and the post item score of the multiple ranking data are obtained through the prior probability distribution (judgment criterion probability distribution and response correctness probability distribution) of the ranking data, the dictionary item score and the observation value (input data).

Input data

In step S110, the probability of the pair comparison result is calculated using the input data and the dictionary multiple ranking data. Herein, 'input data' is data necessary for generating post-ranking multi-ranking data, and is a known value that has been collected or determined in advance, and is a set of a plurality of respondents, a set of a plurality of items, And may include a plurality of pairs of comparison data.

: 쌍별 비교를 수행한 응답자들의 집합이다. L명(단, L은 2이상의 정수)의 복수의 응답자가 쌍별 비교를 하였다고 가정하며, 각각의 응답자를 "u"로 표현하기로 한다.

: A set of respondents who performed pairwise comparisons. It is assumed that a plurality of responders of L (where L is an integer of 2 or more) compares pairs, and each responder is represented by " u &quot;.

: 쌍별 비교의 대상이 되는 아이템들의 집합이다. M개(단, M은 2이상의 정수)의 복수개의 아이템이 있다고 가정하며, 각 아이템을 "o_i", "o_j" 등으로 표현하기로 한다. 쌍별 비교에서 아이템(o_i)이 아이템(o_j)보다 우선순위가 높다는 쌍별 비교결과를 "o_i<o_j"로 표현하기로 한다.

: A set of items that are subject to pairwise comparison. It is assumed that there are a plurality of M items (where M is an integer of 2 or more), and each item is represented by "o _i ", "o _j ", and the like. The result of the pairwise comparison that the item (o _i ) has a higher priority than the item (o _j ) in the pairwise comparison is expressed as "o _i <o _j ".

C _u : It is a set of all pairwise comparisons that each respondent u presents.

Multiple ranking  data

The multi-ranking data includes prior data input to a multi-ranking algorithm or posterior data generated as a result of calculation of a multi-rank algorithm, including a decision criterion preference probability distribution, an item score, and a response accuracy probability distribution as follows can do.

Judgment criteria preference probability distribution (θ _{u, m} ): Probability distribution of probability that respondent (u) selects m-th judgment criterion.

Item score (π _i (m)): The score of each item, which means the score according to the mth judgment criterion of item i.

Response Accuracy Probability Distribution (η _m ): Probability distribution of the probability that the responder correctly responded to each criterion.

In this regard, FIGS. 6A-6C illustrate exemplary values for each of the decision criterion preference probability distribution, item score, and response accuracy probability distribution for ease of understanding.

6A is a probability distribution of preference probability (θ _{u, m} ) according to an exemplary embodiment _, and shows, as a probability, a specific criterion to be selected by each respondent when the number of respondents (u) is L and the number of judgment criteria is d. Referring to the table, the responder (u ₁₎ is determined based on a (m ₁₎ preferred by ssangbyeol comparison, this probability of selecting the criteria (m ₁₎ 0.15 a, the probability of selecting in favor of the criteria (m ₂₎ is 0.10 and, the thus responder (u ₁₎ is likely to prefer each criterion (m ₁ to m _d) being displayed, the responder is a first adding the affinity probabilities for all criteria for a (u _1). Similarly, for each of the respondents (u ₂ ) to (u _L ), the likelihood of each judgment criterion is displayed, and it can be seen that the likelihood of each judgment criterion is 1 for each responder.

로 표현할 수 있으며, 이하에서는 임의의 응답자(u)의 m번째 판단기준에 대한 판단기준 선호도 확률분포를 θ_u,m로 나타내기로 한다. The decision criterion preference probability distribution is a probability that each respondent (u) who presents the result of the pair comparison preferring one criterion among d different criterion criteria,

Hereinafter, the probability distribution of the criterion for the mth judgment criterion of an arbitrary responder (u) is denoted by θ _{u, m} .

가 디리클레 분포

를 따른다고 가정하며, 여기서

는 외부에서 미리 주어지는 인자이다.The prior decision preference probability distribution? _{U, m} input to the multiple ranking algorithm in step S110 may have a predetermined or estimated value. In one embodiment, it can be assumed that such a pre-decision criterion probability probability distribution follows an arbitrary probability distribution, and in the embodiment described below, it follows that it follows a Dirichlet distribution as an example. That is,

Dirichlet distribution

, Where &lt; RTI ID = 0.0 &gt;

Is an externally given argument.

Figure 6b is an item score (π _i (m)) in accordance with one embodiment, the item (o) when M is numbered, it is determined based on d dog is present represents the score of each item. Referring to the table, each item (o) has a score according to d different criteria and can be expressed, for example, as an arbitrary real number from 0 to 10. It is assumed that the lower the score, the higher the priority.

However, since the pairwise comparison indicates the relative preference between the two items, it is not necessary that the item score necessarily have a value in a certain range (e.g., between 0 and 10), and in an alternative embodiment, You can also score item points. Also, in an alternative embodiment, it may be assumed that the higher the score, the more preferred the item.

와 같이 표기할 수 있다. 또한 모든 아이템의 m번째 판단기준의 점수 값을 모은 벡터를

로 표기하기로 한다. Hereinafter, the score value of an arbitrary item (o _i ) in the mth judgment criterion is denoted by pi _i (m). Mathematically, the score on all criteria of the item is converted to a d-dimensional vector

As shown in Fig. Also, the vector that collects the score of the m-th judgment criterion of all items

.

6C is a response accuracy probability distribution? _M according to an exemplary embodiment, and indicates the probability that the responder himself / herself will perform the correct pairwise comparison for each judgment criterion when there are d judgment criteria. Hereinafter, the probability that the respondent correctly answers the m-th judgment criterion, that is, the probability of response accuracy, is denoted by η _m .

는 상술한 α와 마찬가지로 외부에서 미리 주어지는 인자이다. The pre-response correctness probability distribution? _M input to the multi-ranking algorithm in step S110 may have a predetermined or estimated value. In one embodiment, it can be assumed that such a prior-response accuracy probability distribution follows an arbitrary probability distribution, and in the following embodiments it is assumed that it follows a beta distribution as an example. here

Is a factor given in advance from the outside as in the above-mentioned?.

Calculate the probability of the result of pairwise comparison

)을 산출한다. 여기서 확률(

)은 임의의 응답자(u)가 임의의 제1 아이템(o_i)을 임의의 제2 아이템(o_j)보다 선호하는 비교결과를 내놓을 확률을 의미한다. Referring again to FIG. 5, using the above-described input data and dictionary multilingual data, the probability of the pair comparison result (step S110)

). Here, the probability (

) Means the probability that any responder u will give a comparison result that prefers any first item o _i to any second item o _j .

)은 다음과 같이 수학식1로서 표현될 수 있다. We think that the process of creating each pairwise comparison is independent of each other. Assuming that the respondent (u) has a plurality of pairwise comparison questions, the t-th question asks the pairwise comparison between the two items o _i and o _j .

) Can be expressed as Equation (1) as follows.

는 응답자(u)가 m번째 판단기준을 이용하여 두 아이템을 비교할 것인지를 나타내는 확률이다. 이는

에 따른 다항분포 Multinomial

를 이용하여 임의의 판단기준을 선택함으로써 정해진다. 여기서 s_u,t는 전체 d개의 판단기준 중 어느 판단기준을 선택했는지 나타내는 변수로서, 1에서 d 사이의 임의의 정수값이다. 예컨대

이면, 첫번째 판단기준(m₁)을 이용해서 두 아이템간의 쌍별 비교를 하였을 확률이다. m번째 판단기준이 선택될 확률, 즉

는 다항분포의 정의에 의해 판단기준 선호도 확률분포(θ_u,m)가 된다. here,

Is a probability that the respondent (u) will compare two items using the mth judgment criterion. this is

Multinomial distribution according to

By selecting an arbitrary criterion. Here, s _{u, t} is a variable indicating which one of all d judgment criteria is selected, and is an arbitrary integer value between 1 and d. for example

, It is the probability that a pairwise comparison between two items is made using the first judgment criterion (m ₁ ). The probability that the mth determination criterion is selected, that is,

Is a decision criterion preference probability distribution (? _{U, m} ) by the definition of a polynomial distribution.

는, 응답자(u)가 비교하는 과정에서 실수가 있지 않았는지를 고려할 때, 다음과 같이 표현된다. In Equation (1)

Is expressed as follows when considering whether the respondent (u) did not make a mistake in the comparison process.

In the above equation 2 p _{u, t} is assumed to respondents (u) is a variable that refers to that mistakes were in the comparison process ssangbyeol, according to the binomial distribution Binomial (η _m) of the η _m said to have a value of 1 or 0, . If p _{u, t} = 1, it means that the respondent gave the correct pairwise comparison result, and p _{u, t} = 0, it means that the respondent mistakenly gave the opposite result.

는 m번째 판단기준이 선택된 후 이 판단기준에 따라 응답자(u)가 올바르게 응답하였을 확률로서 응답정확성 확률분포(η_m)가 되고,

는 m번째 판단기준이 선택된 후 이 판단기준에 따라 응답자(u)가 실수로 반대로 응답하였을 확률이고, (1-η_m)로 표현할 수 있다. 즉 다음과 같이 표현 가능하다. therefore

The response probability probability distribution? _M is a probability that the respondent u has correctly responded according to the determination criterion after the mth determination criterion is selected,

Is the probability that the responder (u) reacted inversely to the real number after the mth judgment criterion was selected, and can be expressed as (1 -? _M ). That is, it can be expressed as follows.

는 응답자가 올바로 응답하였을 때의 제1 아이템(o_i)을 제2 아이템(o_j)보다 선호하는 비교결과의 확률이고, In Equation (2)

And the probability of the result of the comparison that the respondents preferred the first item _(i o) of the second item (o _j) at the time when the response correctly,

는 응답자가 실수로 잘못 응답하였을 때의 제2 아이템(o_j)을 제1 아이템(o_i)보다 선호하는 비교결과의 확률이다.

Is the probability of a comparison result in which the second item o _j when the respondent mistakenly responds incorrectly is preferred to the first item o _i .

The probability of the comparison result favoring the first item o _i over the second item o _j and the probability of the comparison result favoring the second item o _j over the first item o _i can be arbitrarily set It can be expressed as a mathematical model. In the following embodiments, it is assumed that the probabilities of these comparison results follow the Bradley-Terry model, respectively.

The Bradley-Terry model is a model proposed to explain the preference between two items of respondents. In this model, there is a score π _i for each item (o _i ), and the lower the score, It is assumed that the probability of doing so increases. According to this model, given the two items (o _i , o _j ), the probability that a respondent will prefer o _i to o _j , ie, o _i <o _j ,

은 아래와 같다.Accordingly, the probability of the comparison result that the first item o _i when the responder correctly responded is preferable to the second item o _j , that is,

Is as follows.

은 아래와 같다. And the second item o _j when the respondent erroneously responded incorrectly to the probability of the comparison result favoring the first item o _i ,

Is as follows.

Assuming that Equation (2) through Equation (4), Equation (6) and Equation (7) are substituted into Equation (1) and all the pairs of items asking the responder (u) The probability (Pr _u ) of the result of pairwise comparison on responder (u) can be expressed as follows.

Likelihood function  Calculation

Referring to FIG. 5, after calculating the probability of the pair comparison result in step S110 as described above, a likelihood function is calculated on the basis of the probability in step S120.

)는 아래 수학식9와 같이 계산된다. In one embodiment, given multi-ranking data, given a decision criterion preference probability distribution (? _{U, m} ), an item score (? _I (m)) and a response accuracy probability distribution (? _M )

) Is calculated as shown in Equation (9) below.

Maximize the likelihood function Multiple ranking  Data output

) 값이 최대가 되도록 하는 판단기준 선호도 확률분포(θ_u,m), 아이템 점수(π_i(m)), 및 응답정확성 확률분포(η_m)를 계산한다. If the likelihood function is calculated as in Equation (9) in step S120, posterior multiple ranking data for maximizing the likelihood function is calculated in step S130. That is,

(? _{U, m} ), the item score (? _I (m)), and the response accuracy probability distribution (? _M ).

This process can use a known method known as an Expectation Maximization (EM) algorithm. The EM algorithm is described in, for example, AP Dempster, NM Laird, and DB Rubin, "Maximum likelihood from incomplete data via EM algorithm" (Journal of Royal Statist. Soc., 39: 1-38, 1977), and it is a method to obtain a probability distribution value as a solution when the likelihood value does not increase any more while performing E-step and M-step repeatedly. Since the EM algorithm does not always guarantee that it finds the optimal value, it is desirable to run the EM algorithm multiple times to make the probability distribution with the greatest likelihood value the best.

Meanwhile, when applying the EM algorithm to an embodiment of the present invention, the probability (Pr _u) calculation, two items (that is, Equation 8) _(i o, _j o) aberrations (π _j (m) between the _i -π (m)) it is used because it is a factor of the exponential function, the EM algorithm does not know the item scores (π _i (m)) difference between the direct score value _{(π j (m) -π i} (m)) Only. The posterior criterion preference probability distribution (θ _{u, m} ), the post-response accuracy probability distribution (η _m ), and the score difference between items π _j (m) - π _i (m) are derived by the EM algorithm, For example, the score (? _I (m)) of each item can be calculated by the Least Square Method (LSM).

As described above, steps S110 to S130 may be performed to obtain post-multiple ranking data. It is possible to obtain the item score (? _I (m)) as one of the post-ranking multiple ranking data, to provide the ranking to each user according to the judgment criterion of each item according to the item score, Can be utilized. Also, a determination criterion preference probability distribution ([theta] _{u, m} ) is obtained as one of post-multiple ranking data, and the preference of each user for a plurality of judgment criteria can be estimated from this data and utilized.

In addition _, it can be seen that s _{u, t} and p _{u, t} are also obtained as potential variables when calculating multiple ranking data as above. That is, the random variable (s _{u, t} ) and the respondent (u) indicating which one of the judgment criteria were selected as the comparison result for each pair comparison (o _i , o _j ) (P _{u, t} ) may be respectively calculated and stored in a manner that matches the comparison result of each pair.

Additional query selection - Active running

As described with reference to FIG. 4, the query selection unit 120 selects questions to be further inquired based on the generated multiple ranking data. The query selection unit 120 may use a method of uncertainty sampling in one embodiment to determine which query is the most effective to query the respondent and obtain additional input data therefrom.

The uncertainty sampling method is a method in which a learning system based on existing response data among various queries selects a query with the highest uncertainty in predicting an answer and presents the query to the respondent as an additional question. In one embodiment, the uncertainty in answering a query can be quantified using entropy based on probability theory.

In one embodiment, the following variables are defined for entropy calculation.

X _{u, i, j} : a random variable of whether respondent u answered item (o _i ) or item (o _j )

Y _u : random variable for the preference rank for the mth judgment criterion of respondent (u), and has one of the values from 1 to M (the number of judgment criteria). For example, if Y _u = 1, this means that the respondent has selected the first of the M decision criteria.

? _{u, m} : a probability distribution with the above-mentioned criterion of preference, that is, a probability distribution of the probability that the responder (u) selects the mth judgment criterion, and may be expressed as " Pr ( _Yu = m) &quot;.

In one embodiment, the uncertainty for the answer can be expressed as entropy, and since the higher the entropy of the question is, the higher the uncertainty is, the higher the entropy, the better the question is to choose.

Item (o _i, o _j) when the pair is given to respondents (u), X _{u, i, j,} in response to said responder (u) The item (o _i) the preferred or preferred the item (o _j) And Y _u is a categorical random variable indicating the index of the mth potential decision criterion selected by the responder (u), and is called a prior random variable representing the Bernoulli distribution at time _t , , The entropy of the joint distribution of X _{u, i, j} and Y _u can be calculated as:

In the above equation (10) in the first term of the right side of the formula H [X _{u, i, j} | Y _u = m] is, the responder (u) is based on the criterion to select the m-th criteria two items (o _i , o _j ) is the conditional entropy of the answer.

를 응답자(u)가 m번째 판단기준에 따라 (o_i<o_j)라고 판단할 확률이라고 하면, 지금까지 수집된 쌍별 비교 결과들에 기초한

의 베르누이 분산의 불편 추정량(unbiased estimation)은 다음과 같이 정의될 수 있다. To compute the conditional entropy H [X _{u, i, j} | Y _u = m], it is necessary to define the distribution of X _{i, j} .

A responder (u) according to the m-th criteria Speaking (o _i <o _j) determining said probability, based on the comparison result ssangbyeol collected so far

Can be defined as follows: &lt; EMI ID = 1.0 &gt;

는 m번째 판단기준에 기초하여 응답자들이 (o_i<o_j)라고 쌍별 비교를 한 횟수를 나타내는 "사전 믿음"(prior belief)이다. 사전 믿음은 베르누이 분포의 불확실성을 측정할 수 있다. 베르누이 분포의 불확실성을 계산하기 위해, 베타 분포

및 균등 분포 Beta(1,1) 사이의 쿨백-라이블러(KL) 분산(Kullback-Leibler divergence)이 사용될 수 있으며, 이 KL 분산은 또한 미분 엔트로피(differential entropy)라고 칭하기도 한다. 예를 들어, Beta(1,1), Beta(8,2), Beta(40,10), 및 Beta(80,20) 분포의 미분 엔트로피는 각각 0, -0.79, -1.48, 및 -1.81 이다. 엔트로피가 더 작을수록 불확실성이 작음을 의미한다. In the above equation

Is subject to the respondents (o _i <o _j) is called "pre-faith" (prior belief) that indicates the number of times the comparison ssangbyeol based on the m-th criteria. Pre-belief can measure the uncertainty of Bernoulli's distribution. To calculate the uncertainty of the Bernoulli distribution, the beta distribution

And a Kullback-Leibler divergence between equidistant Beta (1,1) may be used, which is also referred to as differential entropy. For example, the differential entropy of the Beta (1,1), Beta (8,2), Beta (40,10), and Beta (80,20) distributions is 0, -0.79, -1.48, and -1.81, respectively . The smaller the entropy, the smaller the uncertainty.

)와 유사한 선호도 (

)를 갖는 다른 응답자(v)가 (o_i<o_j)라고 답한 경우, 이것은 수학식 10에서의 미분 엔트로피 H[X_u,i,j|Y_u=m]를 증가시킨다. 따라서 아이템(o_i, o_j)에 대해, 사전 믿음(

)은 다음과 같이 계산될 수 있다. Since each responder does not repeatedly respond to the same query, there is a problem with respect to responder (u) that the number of the results of pairwise comparison that responder (u) responded to is set as the prior belief of Equation (11) . Thus, in one embodiment, it may be desirable to use pairwise comparisons that other respondents with similar preferences answer based on potential criteria. That is, if the preference vector of the responder (u)

) And similar preference (

If) answered by another responder _{(v) (o i <o} j) having, this differential entropy H in Equation 10 | increases the _{[X u, i, j Y} u = m]. Thus, for item (o _i , o _j )

) Can be calculated as follows.

는 모든 사용자에 대해 m번째 판단기준에 따라 아이템(o_i)을 아이템(o_j) 보다 선호한다고 대답한 횟수를 의미하며,

는 사용자(v)가 m번째 판단기준에 따라 아이템(o_i)을 아이템(o_j) 보다 선호한다고 대답한 횟수의 기대값이다. In the above equation

Means the number of times the item (o _i ) is answered to be preferred to the item (o _j ) according to the mth judgment criterion for all users,

Is an expected value of the number of times that the user v has answered that the item (o _i ) is preferred to the item (o _j ) according to the mth judgment criterion.

는, 응답자가 임의의 판단기준을 선택할 확률의 확률분포(θ_u,m)와 응답자가 그 판단기준에 따라 아이템(o_i)을 아이템(o_j) 보다 선호한다고 대답한 횟수의 기대값의 곱을 모든 응답자에 대해 더함으로써 계산할 수 있다. According to the above equation,

, The probability distribution of the probability of respondents to select any criterion (θ _{u, m)} and respondents item according to the criteria (o _i) an item (o _j) than the product of the expected value of the number of times indicated you prefer It can be calculated by adding for all respondents.

는 다음과 같이 계산될 수 있다. Using the Bradley-Terry model described with reference to equation (5), in one embodiment

Can be calculated as follows.

는 1이고

는 0이며, 제2 아이템(o_j)을 제1 아이템(o_i) 보다 선호한다는 대답이 있었다면 그 반대의 값을 각각 가진다. In this equation, I _(condition) is a function with a value of 1 or 0 and has a value of 1 if it satisfies a given condition ("condition") or a value of 0 otherwise. For example, if there is a result of pairwise comparison of two items (o _i , o _j ) and there is an answer that the first item o _i prefers the second item o _j

Is 1

Has a value of 0, and has the opposite value if there is an answer that the second item o _j is preferred to the first item o _i .

는 따라서 다음과 같이 계산할 수 있다.Given the condition that the criterion on which the answer is based is the mth criterion, the conditional entropy

Can be calculated as follows.

를 구할 수 있고, 이 값을 수학식 10에 대입하여 엔트로피 H(X_u,i,j,Y_u)를 구할 수 있다. By using Equation (13) to calculate Equation (12) and applying Equation (12) to Equation (14), conditional entropy

And the entropy H (X _{u, i, j} , Y _u ) can be obtained by substituting this value into the equation (10).

At this time, the entropy is calculated for all combinations of all persons and all item pairs. (X _{u, i, j} ) for which respondents prefer items and a combined distribution of random variables (Y _u ) for preference ranks for respondents' criteria (for all respondents and all judgment criteria) joint distribution), and then a predetermined number of queries with the highest entropy value can be selected as an additional query.

When choosing an additional query, for example, a total of 100 questions, you could ask 100 respondents for a question or 10 questions for 10 respondents. In one embodiment of the present invention, the following three steps are performed to select questions and respondents.

가 큰 순서대로 소정 개수 선택한다. 즉 각 사람마다, 엔트로피가 큰 차례대로 소정 개수의 질문을 선택하며, 이 때 소정 개수는 미리 설정된 값일 수 있다. 이 단계에서, 각 사람마다 이 소정 개수씩의 질문을 부여받지만 질문 내용을 각기 다를 것이다. As a first step, candidates for a query to be requested are selected for each responder in descending order of the entropy value. For example, a candidate for a question to ask for each respondent

Are selected in ascending order. That is, for each person, a predetermined number of questions are selected in descending order of entropy, and the predetermined number may be a preset value. At this stage, each person will be given a certain number of questions, but the contents of the questions will be different.

Next, entropy values for the predetermined number of queries are added to each responder to calculate an entropy sum value. That is, all the entropy values of the predetermined number of questions given to the respective respondents are all added, and one entropy sum value is calculated for each responder.

And then selects a predetermined number of respondents in descending order of the entropy sum value. In other words, since the entropy sum value is derived for each responder in the second step, a predetermined number of respondents are selected in descending order of the entropy sum value. At this time, the predetermined number of people may be a predetermined value.

With additional input data Multiple ranking  Of data update

As described above, the pair comparison data collected by a predetermined number of questions to a predetermined number of respondents is called " additional input data ".

When the additional input data is collected, the multi-ranking data, that is, the updated decision criterion preference probability distribution (? _{U, m} ), the item score (? It updates the π _i (m)), and response accuracy probability distribution (η _m).

In one embodiment, an Incremental Gibbs sampling algorithm may be used as a method for updating multiple ranking data.

According to this algorithm, when k pairs of comparison results of responders (u) are collected as additional input data, (o _i <o _j ) _t ∈ C _u (ie, For each pairwise comparison in a pairwise comparison set (C _u )), the random variables s _{u, t} and p _{u, t} are sampled again, and then for each new pairwise comparison of the additional input data the random variable s _{u, t} and p _{u, t} .

Let n _{u, m ,} 를 be the number of pairwise comparisons of responders (u) answered according to the mth judgment criterion, and let n _{, m,} , l = 1) or wrongly (i.e., l = 0). Then the index of the random variable s _{u, t} used for (o _i <o _j ) _t ∈ C _u can be sampled as follows.

의 세트를 나타내며,

는 m번째 판단기준에 따라 제1 아이템(o_i)을 제2 아이템(o_j) 보다 선호할 확률로서 브래들리-테리 확률을 나타낸다고 가정한다. 또한 n^{-( Su,t )}는 전체 쌍별 비교의 개수들(예컨대 n_{u,m ,ㆍ} 또는 n_{ㆍ, m,ℓ}) 중 t번째 쌍별 비교를 제외한 개수를 의미한다. In the above equation, s _{( u, t )} is a set of random variables excluding the specific t-th judgment criterion of any particular responder (u) among all s _{u, t}

&Lt; / RTI &gt;

Bradley is a probability of a first preferred to item _(i o) of the second item (o _j) in accordance with the m-th criteria - is assumed to represent Terry probability. Also, n ^{- ( Su, t )} means the number of comparisons among all pairs (for example, nu _{, m , ㆍ} or n _{ㆍ, m, ℓ} ) excluding the t-th pair comparison.

Similarly, a random variable p _{u, t,} which indicates whether the pairwise comparisons of (o _i <o _j ) _t ∈ C _u have been correctly answered _, can be sampled as follows.

를 계산할 수 있다. Figure 7 shows the pseudo-code for this incremental Gibbs sampling algorithm. By performing the Gibbs sampling according to the pseudo code for new pairwise comparisons of additional input data and setting the probability that the first item is preferred to be equal to the expected value obtained from the prior belief by using the least square method, score

Can be calculated.

In the above equation, c _{m, (i, j)} is a prior belief indicating the number of times that it is determined to prefer the first item (i.e., (o _i o _j )) based on the mth determination criterion. Also, M _{m, (i, j)} represents the score difference between two items as defined in the following equation.

를 얻을 수 있다. If the equation is transformed such that only M _{m, (i, j)} is left on the left side of the equation (17 _), the following equation is obtained. If the least squares method is applied to this equation,

Can be obtained.

도 구할 수 있다. 또한 이 때 랜덤 변수 s_u,t는 판단기준 선호로 확률분포(θ_u,m)와 관련된 변수이고 랜던 변수 p_u,t는 응답정확성 확률분포(η_m)에 관련된 변수이므로, 결과적으로 업데이트된 판단기준 선호도 확률분포(θ_u,m), 아이템 점수(π_i(m)), 및 응답정확성 확률분포(η_m)를 얻을 수 있다. As described above, according to the multi-ranking data updating method of the present invention, new random variables s _{u, t} and p _{u, t} are calculated for each pairwise comparison by considering additional input data and existing input data, score

Can be obtained. In this case, the random variable s _{u, t} is a variable related to the probability distribution (θ _{u, m} ) in preference to the determination criterion and the random variable p _{u, t} is a variable related to the response accuracy probability distribution (η _m ) (Θ _{u, m} ), item score (π _i (m)), and response accuracy probability distribution (η _m ).

Experiment result

FIG. 8 is a diagram for explaining experimental results of a multiple rank estimation method according to an embodiment.

In an experiment according to an embodiment, experiments were conducted on both synthetic data and real life data. The composite data is generated by generating a response probability distribution (? _M ) and a decision criterion probability distribution (? _{U, m} ) by performing the probability model of the present invention as it is. The score (π _i (m)) for each item was selected randomly between 0 and 10, and the result of pairwise comparison was also generated using the generation model.

The real-life data used MovieLens-100k data, which is a collection of ratings of each person's movies. A pair of all movies rated by each person is created so that if the ratings of the two movies are different, then the movie with the higher score will be prioritized The result of

Kendall's tau was used as a measure to evaluate the experimental results. This value has a value of -1 to 1, and the closer to 1, reflects the actual ranking ranking well, and the closer to -1, the more the actual ranking ranking is reflected.

Referring to FIG. 8, the X axis is the number of pairs of comparisons, and the Y axis is the Kendall's tau value. CrowdRank-ACT " among the three graphs in FIG. 8 is a result value when updating multiple ranking data by incremental Gibbs sampling according to the present invention, and " CrowdRank-RND " And collecting additional input data. Also, "CrowdBT-ACT" is a result of applying the active running method to the single ranking algorithm.

As can be seen from the graphs of the drawings, the three methods show similar performance until about the first 5000 comparison questions. However, as the number of questions increases, the accuracy of the method according to the present invention increases gradually, It can be seen that it is improved to 0.87.

FIG. 9 is a block diagram illustrating an exemplary apparatus configuration for implementing a multiple rank estimation method according to an embodiment.

9, an apparatus for implementing a multiple rank estimation method according to an embodiment may include a server 30, an input data DB 50, and a multiple ranking data DB 60, To communicate with a plurality of user terminals 10 via a network.

The user terminal 10 may be, for example, a portable mobile terminal such as a smart phone, a tablet PC, a notebook computer, or a non-portable terminal such as a desktop computer.

The network 20 is any type of wired and / or wireless network that provides a data transmission / reception path between the portable terminal 10 and the server 30 and may be a LAN, a WAN, an Internet network, and / .

The server 30 may be any computing device as a service server that provides the multi-ranking data extracted by the multi-rank estimation method to the user terminal 10, and in one embodiment, performs the multi-ranking estimation method The multi-rank learning algorithm 100, which may be a multi-rank learning algorithm, may include, for example, the multiple ranking data calculation unit 110, the query selection unit 120, and the update unit 130 described with reference to FIG. 4, 120, and 130 may be implemented in software and / or hardware. The server 30 may include a processor, a memory, a storage unit, a communication unit, and the like. The multi-rank learning algorithm 100 may be stored in a storage unit and loaded into a memory under the control of the processor.

In the illustrated embodiment, the server 30 is communicatively coupled with the input data DB 50 and the multiple ranking data DB 60. In an alternate embodiment, the server 30 may include at least one of an input data DB 50 and a multi-ranking data DB 60. The input data DB 50 includes input data (for example, a set of a plurality of responders, a set of a plurality of items, and a set of a plurality of pairs of comparison data for a plurality of items) necessary for generating the multiple ranking data DB, And the multi-ranking data DB 60 may store pre-multi-ranking data before being input to the learning algorithm 40 and / or post-multi-ranking data generated after the learning algorithm 100 is performed. The query selecting unit 120 may further include a query database (DB) not shown in the figure, for example, in which query contents are stored, and the query selecting unit 120 may select queries to be further inquired from the query DB. In addition, a database for storing various information such as a responder database and a statistical database may be additionally provided, and these databases may be included in the input data DB 50 or the multiple ranking data DB 60 shown in the figure.

As described above, although the present invention has been described with reference to the limited embodiments and drawings, the present invention is not limited to the above embodiments. It will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the present invention as defined by the appended claims. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the equivalents of the claims, as well as the claims.

10: User terminal
20: Network
30: Server
50: input data DB
60: Multiple Ranking Data DB
100: Multiple Ranking Learning Algorithm

Claims (11)

A method of estimating multiple rankings based on pairwise comparison data using a computing device,
The computing device receiving information on the pair comparison data as first input data;
The computing device calculating multiple ranking data based on the input data;
The computing device selecting an additional query based on the calculated multiple ranking data;
The computing device receiving second input data relating to pairwise comparison data for the additional query; And
The computing device updating the multi-ranking data based on the first and second input data,
Wherein at least one of the first and second input data includes a set of a plurality of responders, a set of a plurality of items, and a set of a plurality of pairs of comparison data for a plurality of items,
Wherein each pair of comparison data of the plurality of pairs of comparison data is data for determining a degree of preference between two items according to one of the plurality of determination criteria,
Wherein updating the multi-ranking data comprises:
(S _{u, t} ) indicating which decision criterion has been selected among all decision criteria and a random variable (s _{u, t} ) indicating whether the respondent (u) has a mistake in the pair comparison process, based on the incremental Gibbs sampling algorithm Sampling a variable (p _{u, t} ); And
Wherein the computing device is configured to set a probability of a comparison result in which a first item among the plurality of items is preferred to a second item and an expected value obtained from a pre-trust to be equal,

Calculating a plurality of ranking estimates for each of the plurality of rankings. 2. The method of claim 1, wherein the multi-
A decision criterion preference probability distribution (? _{U, m} ) which is a probability distribution of probability that a respondent selects an arbitrary criterion;
Item score (π _i (m)), which is the score assigned to each item per judgment criterion; And
And a response accuracy probability distribution (? _M ), which is a probability distribution of the probability that the responder correctly answered every judgment criterion. 3. The method of claim 2, wherein the calculating multi-
Calculating (S110) a probability Pr of pairwise comparison results for any two items based on the input data, the first multiple ranking data;
Calculating a likelihood function based on the calculated probability Pr (S120); And
And calculating second multi-ranking data for maximizing the likelihood function (S130). The method of claim 3,
Wherein the first multiple ranking data includes a prior decision likelihood probability distribution, a dictionary item score, and a prior response correctness probability distribution,
Wherein the second multi-ranking data includes a posterior criterion preference probability distribution, a post item score, and a post-response accuracy probability distribution. 5. The method of claim 4,
The probability Pr of the result of the pair comparison is a probability of a comparison result in which the first item is preferred to the second item,
The probability (Pr) of the result of the pair comparison is, for each determination criterion,
A probability of a comparison result in which the first item is more favored than the second item when the respondent correctly answered; And
And a probability of a comparison result in which the second item is more favorable than the first item when the respondent responds incorrectly. 4. The method according to claim 3, wherein the step of calculating second multiple ranking data for maximizing the likelihood function comprises:
Calculating a score difference between a posterior criterion preference probability distribution, a postresponse correctness probability distribution, and a post item score of each of the first item and the second item, by an expected value maximization algorithm; And
And calculating a post-score of each item by a least squares method. The method according to claim 1,
Wherein the step of selecting an additional query based on the calculated multiple ranking data comprises:
For all respondents and all decision criteria, a random variable (X _{u, i, j} ) for which item the responder prefers and a random variable (Y _u ) for the preference rank for the responder's criteria calculating the entropy of each distribution; And
And selecting a predetermined number of queries having the highest entropy value as an additional query. 8. The method of claim 7,
Wherein the step of selecting the predetermined number of queries as an additional query comprises:
Selecting candidates for a query to be requested for each responder in a descending order of the entropy values;
Adding an entropy value of the predetermined number of queries to each responder to calculate an entropy sum value; And
And selecting a predetermined number of respondents in descending order of the sum of entropy values. delete The method according to claim 1,
The item score

Calculating an item score using the least squares algorithm. &Lt; RTI ID = 0.0 &gt; [10] &lt; / RTI &gt; A computer-readable recording medium having recorded thereon a program for causing a computer to execute the method according to any one of claims 1 to 8 and 10.

Images