JP6701026B2 - Judgment device, judgment method, and judgment program - Google Patents

Description

  The present invention relates to a determination device, a determination method, and a determination program.

  Conventionally, a technique for determining the significance of the evaluation result by the user is known. As an example of such a technique, by using an evaluation result when a user evaluates a certain web content and an evaluation result performed in the past with respect to the same web content, a variation in the average value of the evaluation values is obtained. There is known a technique such as CUPED (Controlled-experiment Using Pre-Experiment Data) that reduces the error rate and improves the accuracy of significance determination.

JP, 2007-052601, A JP, 2011-043911, A

“Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data”, Alex Deng, Ya Xu, Ron Kohavi, Toby Walker, WSDM′13, February 4-8, 2013, Rome, Italy.

  However, in the above-described conventional technique, when the user who newly evaluates and the user who previously evaluates do not overlap, the dispersion of the average value increases, and the accuracy of significance detection may deteriorate. is there.

  The present application has been made in view of the above, and an object thereof is to improve the accuracy of detecting the significance of the evaluation result.

  The determination apparatus according to the present application is estimated to be used for the predetermined content based on a first index value indicating the behavior of the user regarding the predetermined content and a second index value based on the attribute of the user. A calculation unit that calculates an estimated index value indicating the behavior of a person for each sample related to the predetermined content, and a determination unit that determines a significant difference between samples based on the estimated index value calculated for each sample. And having.

  According to the aspect of the embodiment, it is possible to improve the accuracy of detecting the significance of the evaluation result.

FIG. 1 is a diagram illustrating an example of a determination process executed by the determination device according to the embodiment. FIG. 2 is a diagram illustrating a configuration example of the determination device according to the embodiment. FIG. 3 is a diagram showing an example of information registered in the action history database according to the embodiment. FIG. 4 is a diagram illustrating an example of information registered in the user database according to the embodiment. FIG. 5 is a diagram illustrating an example of a result screen generated by the determination device according to the embodiment. FIG. 6 is a flowchart showing an example of the flow of the determination process executed by the determination device according to the embodiment. FIG. 7 is a diagram illustrating an example of the hardware configuration.

  Hereinafter, modes (hereinafter, referred to as “embodiments”) for carrying out the determination device, the determination method, and the determination program according to the present application will be described in detail with reference to the drawings. The determination device, the determination method, and the determination program according to the present application are not limited to this embodiment. Also, in each of the following embodiments, the same parts are designated by the same reference numerals, and duplicate description will be omitted.

[Embodiment]
[1-1. Example of determination device]
First, an example of the determination process executed by the determination device will be described with reference to FIG. FIG. 1 is a diagram illustrating an example of a determination process executed by the determination device according to the embodiment. In FIG. 1, the determination device 10 can communicate with a content server 50 that delivers predetermined content and a manager terminal 200 via a predetermined network N such as the Internet. The determination device 10 may be capable of communicating with the terminal device 101 used by the user U01 and the terminal device 102 used by the user U02 via the network N.

  The terminal devices 101 and 102 are smart devices such as smartphones and tablets, and communicate with an arbitrary server device such as the content server 50 via a wireless communication network such as 3G (3rd Generation) or LTE (Long Term Evolution). It is a portable terminal device that can be. The terminal devices 101 and 102 are not limited to smart devices and may be information processing devices such as desktop PCs and notebook PCs. In the following description, the terminal devices 101 and 102 used by the users U01 and U02 may be collectively referred to as the terminal device 100. Further, in the following description, it is assumed that, in addition to the users U01 and U02, any number of terminal devices 100 used by any number of users can communicate with the content server 50 and the like.

  The content server 50 is a server device that delivers various web contents (hereinafter referred to as “contents”) such as web pages, widgets, still images, moving images, and games to the terminal device 100. For example, the content server 50 transmits the predetermined content included in the predetermined domain to the terminal device 100.

  The administrator terminal 200 is a terminal device used by an administrator who manages the determination device 10, and is realized by, for example, a PC or a server device. Note that the administrator terminal 200 is a terminal device used by any user as long as it is a terminal device used not only by the administrator of the determination device 10 but also by a user who desires the result of the determination processing by the determination device 10. You can

  The determination device 10 is a determination device that determines a significant difference between samples included in the same event, and is realized by, for example, a server device or a cloud system. To describe a more specific example, the determination device 10 extracts a plurality of samples by randomly selecting a plurality of information from the information indicating the behavior of the user in the predetermined content distributed by the content server 50, and extracts the sample. It is determined whether or not there is a statistically significant difference between the prepared samples. More specifically, the determination device 10 is a determination device that determines a significant difference between samples showing user behavior in predetermined content by an arbitrary test method such as t-test or Z-test. In the following description, an example of performing a test using the Z test will be described, but the determination device 10 may apply the determination method described below to any test method.

[1-2. About control variations]
Here, in a general test method, an average value is calculated for each sample for which a significant difference is to be determined, and whether the significant difference exists is determined using the calculated average value. For example, in the method called the Z test, a sample Y ^(t) and a sample Y ^{(c) in} which one or more pieces of information are randomly extracted from a population of information are generated, and the value of the following formula (1) is predetermined. If it is larger than the threshold value of, it is determined that there is a significant difference between the sample Y ^(t) and the sample Y ^(c) . In addition, in formula (1), the overlined Y ^(t) represents the average value of the values extracted as the sample Y ^(t) , and the overlined Y ^(c) is the sample Y ^(c). The average value of the values extracted as is shown. In the following description, an overlined symbol indicates the average value of the values indicated by the symbol.

  Here, when the sample Y includes n values Xi, the average value of the sample Y can be expressed by the following equation (2).

  Here, when the expression (2) is used, the variance of the average value may not necessarily be small depending on the selection of the sample and the content of the event, that is, the information on the behavior of the user in the predetermined content. However, when the variance of the average value of the sample increases, the accuracy of detecting the significant difference may deteriorate.

Therefore, a predetermined estimated amount (hereinafter referred to as “estimation index value”) is calculated using an expression called Control Variates, and each sample is replaced with the average value shown in Expression (1). There is known a method called CUPED for evaluating a significant difference using the estimated index value of. For example, in CUPED, the estimated index value shown by the following equation (3) is calculated for each sample. Here, in Expression (3), the estimated index value is represented by Y _cv . Further, X shown in the equation (3) is a value selected independently of the sample Y, and θ is a coefficient whose optimum value is determined according to the value of X. Further, EX shown in Expression (3) indicates an expected value of X.

  Here, when the sample size is brought close to infinity, the equation (3) becomes the same as the equation (2). Therefore, it is considered that the determination device 10 can appropriately determine the significant difference even if the significant difference is determined from the equation (1) using the estimated index value shown in the equation (3). Furthermore, the following equation (4) can be obtained by calculating the variance of the estimated index value shown in equation (3).

Here, the right side of Expression (4) can be expressed by the product of the variance of the average value of the sample Y and (1-ρ ² ), but (1-ρ ² ) takes a value smaller than 1. That is, the variance of the estimated index value for a sample Y is smaller than that of the average value. Therefore, when the determination device 10 determines the significant difference from the equation (1) using the estimated index value shown in the equation (3), the determination device 10 can accurately calculate the significant difference.

[1-3. Judgment processing]
Here, an arbitrary value can be adopted as the value of X when calculating the estimated index value. On the other hand, the power of detecting a significant difference increases when the coverage of the adopted information is increased or when the correlation between the adopted information is strengthened.

  Therefore, the determination device 10 executes the following determination process. First, the determination device 10 determines the user's estimated for the predetermined content based on the first index value indicating the behavior of the user regarding the predetermined content and the second index value based on the attribute of the user. The estimated index value indicating the behavior is calculated for each sample regarding the predetermined content. Further, the determination device 10 determines a significant difference between each sample based on the estimated index value calculated for each sample. Then, the determination device 10 determines a significant difference between the samples based on the estimated index value calculated for each sample, and outputs the determination result.

  For example, the determination device 10 determines whether the link clicked by the user while browsing the predetermined content, the value of the query input by the user, whether the click or the query is input, or the like when browsing the predetermined content. A value indicating a user-executable action is selected as the first index value. For example, the determination device 10 assigns the first index value “1” to the user who inputs the query, and assigns the first index value “0” to the user who did not input the query.

Subsequently, the determination device 10 extracts a first sample and a second sample in which a predetermined number of values are randomly extracted from the values indicating the behavior of the user while browsing the predetermined content. To do. Then, the determination apparatus 10 calculates the estimated index value by executing the following process for each sample. First, the determination device 10 sets, as the first index value, a value indicating the action taken by the user while browsing the predetermined content, as the Y value shown in the formula (2) or the formula (3). More specifically, the determination apparatus 10 sets the value extracted as the first sample as Y ^(t) and the value extracted as the second sample as Y ^(c) . Then, the determination device 10 calculates the average value of Y ^(t) and Y ^(c) , respectively.

  Subsequently, the determination device 10 divides each user who browses the predetermined content into a plurality of groups based on the attribute information, and calculates a value based on the result of the division as the second index value. For example, the determination device 10 determines whether the gender of the user is male or female, and when the gender of the user is male, the value “0” is set as the second index value, and When the gender is female, the value "1" is used as the second index value. Then, the terminal device 10 sets the second index value as the value of X in Expression (3).

For example, the determination device 10 identifies the user who has performed the action indicated by the value included in the first sample, and collects the second index value based on the attribute of the identified user. Then, the determination device 10 sets the collected value as X ^(t) . Similarly, the determination device 10 identifies the user who has performed the action indicated by the value included in the second sample, and collects the second index value based on the attribute of the identified user. Then, the determination device 10 sets the collected value as X ^(c) .

Then, the determination device 10 subtracts the product of the predetermined coefficient θ and the average value of the values of X ^(t) from the average value of the values of Y ^(t) , and further, the predetermined coefficient θ and X ^(t). The value obtained by adding the product of the value of and the expected value is set as the estimated index value Y ^(t) _{cv of} the first sample. Further, the determination device 10 subtracts the product of the predetermined coefficient θ and the average value of the values of X ^(c) from the average value of the values of Y ^(c) , and further determines the predetermined coefficient θ and X ^(c). The value obtained by adding the product of the value of and the expected value is set as the value of the estimated index value Y ^(c) _cv of the second sample. That is, the determination device 10 calculates, for each sample, an estimated index value indicating the behavior of the user estimated with respect to the predetermined content.

Then, the determination device 10 calculates the value of the above-described formula (1) using Y ^(t) _cv as the average value of Y ^(t) and Y ^(c) _cv as the average value of Y ^(c) , and calculates the value. The significant difference between the first sample and the second sample is determined based on whether or not the calculated value exceeds a predetermined threshold value.

  In this way, the determination device 10 determines not only the index value indicating the behavior of the user regarding the predetermined content but also the attribute of the user when determining the significant difference between the samples of the behavior of the user regarding the predetermined content. An estimated index value is calculated for each sample using the second index value according to the information, and a significant difference between samples is determined based on the calculated estimated index value. That is, the determination device 10 sets the information on the user clustering related to the first index value as the second index value. As a result, the determination device 10 can improve the coverage of the sample, and thus can improve the accuracy of determining the significant difference. Further, since the determination device 10 identifies the user whose first index value is included in the first sample and the second sample and uses the second index value based on the attribute of the identified user, X As a result of improving the correlation with Y, the accuracy of determining a significant difference can be improved.

  Here, in order to improve the coverage, as the second index value, the index value based on the activity of the user regarding the predetermined content, which is based on the activity acquired earlier than the activity that is the basis of the first index value. A method using is conceivable. However, in such a method, when the predetermined content is a new content, or when a new user exists at the delivery destination of the predetermined content, there is no information output in the past, so it is estimated The dispersion of index values may increase.

  However, the determination device 10 calculates the value of the estimated index value by using the index value based on the attribute of the user as the second index value, not the index value based on the behavior acquired in the past. Therefore, even when the predetermined content is delivered to the predetermined user for the first time, the determination device 10 sets the value based on the attribute information of the user as the second index value, so that the estimated index value Dispersion can be reduced. As a result, the determination device 10 can improve the determination accuracy of the significant difference.

[1-4. About attribute information]
Here, the determination device 10 may use a value based on an arbitrary attribute as the second index value as long as it is a value based on the attribute of the user, and a value generated from the attribute of the user based on an arbitrary rule. May be used as the second index value.

  For example, the determination device 10 may use information such as sex, age, age, address, hobbies, etc., as the attributes of the user. Further, the determination device 10 may use, as the attribute of the user, information indicating the type of the terminal device 100 used by the user and the position when the user browses the content. Further, the determination device 10 has, as the attributes of the user, a history of products and services that the user has purchased in the online shopping mall in the past, an exhibition history and a successful bid history at an auction site, etc., a history of web pages browsed by the user, and the like. , Various types of history may be used as user attributes.

  Then, the determination device 10 classifies the users into a plurality of groups according to the attributes of the users, and sets a value corresponding to the classification result as the second index value. For example, the determination device 10 sets a plurality of groups to which predetermined numbers are given in advance, and classifies the users into any of the groups. Then, the determination device 10 may use the number set in the group into which the users are classified as the second index value.

  Further, for example, the determination device 10 may determine a value according to the attribute of each user and calculate the second index value based on the determined value. That is, the determination device 10 may calculate an arbitrary value as the second index value as long as the value indicates the attribute of the user.

[1-5. About grouping]
Here, the determination device 10 classifies the users by using the model that classifies the users according to the attributes of the users, and adopts a value according to the classification result as the second index value. Good. For example, the determination device 10 may classify the users using a model in which the attributes of the users are learned by an arbitrary learning method such as SVM (Support Vector Machine), neural network, and deep learning.

  In addition, the determination device 10 may use a model that has been learned so that the determination accuracy of the significant difference is improved when the second index value according to the user's classification result is used. That is, the determination device 10 may use a model learned to classify the users so that the determination accuracy of the significant difference is improved. For example, the determination device 10 acquires, as correct answer data, the first sample and the second sample for which a significant difference has been calculated in advance. Further, the determination device 10 classifies the users from the correct answer data and calculates the second index value based on the classification result. Then, the determination device 10 may correct the model so that the significant difference calculated when the calculated second index value is used approaches the significant difference calculated in advance.

  The above-described model may be learned by the determination device 10 or may be one that is previously learned by a learning device different from the determination device 10.

[1-6. About content]
Here, the determination device 10 may employ any content as the predetermined content. For example, the determination device 10 may be a portal site where a predetermined content accepts a web search, an auction, a route search, an online shopping mall, a game, an accommodation reservation, finance, a weather forecast, a restaurant information site, a real estate site, an automobile-related site, The predetermined content may be various bulletin boards, blogs, websites that provide recipes, and the like. Further, the determination device 10 may set not only the website but also an application or a widget related to a predetermined content as the predetermined content, and a value indicating the behavior of the user in the content as the first index value.

[1-7. Regarding commonality of users]
The determination device 10 may add not only the value based on the attribute of the user who performed the action indicated by the first index value but also the value based on the attribute of another user to the second index value. That is, the determination device 10 does not need to completely make the user associated with the first index value and the user associated with the second index value the same as the user who performed the action indicated by the first index value. The second index value may be acquired such that the second index value is partly common with the user indicating the attribute. That is, the determination device 10 determines the first index value indicating the behavior of a plurality of users with respect to the predetermined content and the attributes of the plurality of users including a predetermined number of users among the plurality of users. The estimated index value may be calculated using the second index value based on.

[1-8. Regarding the behavior indicated by the index value)
Here, the determination device 10 uses, for example, a value indicating the presence/absence of a click made by the user while browsing the predetermined content, a link indicating the clicked content, or the like as an index value indicating the behavior of the user regarding the predetermined content. Use as index value. However, the embodiment is not limited to this. For example, the determination device 10 may use, as an index value, for example, the content of the query input when performing the web search or the number of times the query is input, other than the click. That is, the determination device 10 may use a value indicating an arbitrary action as the index value.

  For example, the determination device 10 may use, as the first index value, a value indicating the user's evaluation of the predetermined content or the degree of interest in the predetermined content input by the user. That is, if the first index value and the second index value are KPIs (Key Performance Indicators) related to the user in which the first index value and the second index value are common to some extent, the determination device 10 sets any index value as the first index value. You may adopt it.

[1-9. Regarding an example of determination processing]
Next, an example of the process executed by the determination device 10 will be described with reference to FIG. In the following description, an example of a process will be described in which the information indicating the behavior of the user regarding the predetermined content is used as the first index value and the information indicating the attribute of the user is used as the second index value. Further, in the example shown in FIG. 1, an example in which the attribute information of the user U01 is “attribute #1” and the attribute information of the user U02 is “attribute #2” will be described.

  First, the content server 50 delivers predetermined content to the terminal device 100 according to a request from the terminal device 100 (step S1). Then, the content server 50, along with the click history indicating the details of the clicks made by the users U01 and U02 while browsing the predetermined content, the attribute information “attribute #1”, “attribute #1” indicating the attributes of the users U01 and U02, Attribute #2" is collected (step S2). The content server 50 may collect the attribute information of the users U01 and U02 from an external server such as a management server (not shown) that manages the attribute information of the users U01 and U02.

  Then, the determination device 10 collects the click history and the attribute information regarding the predetermined content from the content server 50 (step S3). Then, the determination device 10 sets the content of the click indicated by each click history as the action content, and registers the action content and the user ID of the user who performed the click in the action history database 31 in association with each other.

  The determination device 10 also groups the users U01 and U02 based on the collected attribute information (step S4). More specifically, when the attribute information is input, the determination device 10 calculates a value indicating the attribute information of the user by using a model that outputs a value indicating the content of the attribute for each attribute type.

  For example, the model shown in FIG. 1 outputs the value “1” corresponding to the gender when the gender is “male”, and outputs the value “0” corresponding to the gender when the gender is “female”. It shall be output. Further, for example, the model shown in FIG. 1 outputs a value “2” corresponding to the age when the age is “20s”, and corresponds to the age when the age is “30s”. The value "3" to be output is output. Further, for example, the model shown in FIG. 1 outputs a value “4” corresponding to the address when the address is “resident in Kanagawa” and corresponds to the address when the address is “resident in Tokyo”. The value "5" to be output is output.

  For example, in the example shown in FIG. 1, it is assumed that the content of the attribute information “attribute #1” of the user U01 is gender “male”, age “30s”, and address “resident in Tokyo”. When such an attribute is input, the model shown in FIG. 1 outputs the values “1”, “3”, and “5” indicating the attribute information of the user U01. Then, the determination device 10 sets the three-digit value “135” in which the values output by the model are arranged, as the value indicating the group that is the classification destination of the user U01, that is, the second index value. Then, the determination device 10 registers the user ID, the user attribute information, and the second index value in the user database 32 in association with each other.

  Note that, for example, when the content of the attribute information “attribute #2” of the user U02 is gender “female”, age “20s”, and address “Kanagawa resident”, the model shown in FIG. "0", "2", and "4" are output as the values indicating the attribute information. Therefore, the determination device 10 sets the three-digit value “024” in which the values output by the model are arranged, as the value indicating the group to which the user U02 is classified, that is, the second index value.

Further, when determining the significant difference between the samples regarding the predetermined content, the determination device 10 calculates the first index value for each sample from the click history of the predetermined content (step S5). For example, the determination apparatus 10 randomly selects a predetermined number of click histories as the first sample from the click histories of the predetermined content, and sets a set of values indicating the contents of the click history selected as the first sample as the first sample. The index value Y ^(t) is 1. Similarly, the determination device 10 randomly selects a predetermined number of click histories as the second sample from the click histories of the predetermined contents, and sets a set of values indicating the contents of the click history selected as the second sample. Let it be the first index value Y ^(c) .

Further, the determination device 10 calculates the second index value for each sample according to the user's grouping result (step S6). For example, the determination device 10 identifies the user who clicked the click history selected as the first sample, and extracts from the user database 32 a value indicating a group into which the identified user is classified. Then, the determination device 10 sets the set of extracted values as the second index value X ^(t) . Similarly, the determination device 10 identifies the user who performed the click indicated by the click history selected as the second sample, extracts the value indicating the group into which the identified user is classified, from the user database 32, The set of extracted values is set as the second index value X ^(c) .

Then, the determination device 10 calculates, for each sample, a value indicating an average user behavior estimated for a predetermined content, that is, an estimated index value (step S7). For example, the determination device 10 calculates the value of Y ^(t) _cv from the equation (3) using the first index value Y ^(t) and the second index value X ^(t) , Using the index value Y ^(c) and the second index value X ^(c) , the value of Y ^(c) _cv is calculated from the equation (3). Then, the determination device 10 uses the calculated estimated index value to determine a significant difference between the samples (step S8). For example, the determination device 10 performs a Z test using the value of Y ^(t) _cv as the average value of Y ^{(t) and} the value of Y ^(c) _cv as the average value of Y ^(c) , and there is a significant difference. It is determined whether to do. Then, the determination device 10 provides the test result to the administrator terminal 200 (step S9).

[2. Configuration of determination device]
Hereinafter, an example of a functional configuration of the determination device 10 that realizes the determination process described above will be described. FIG. 2 is a diagram illustrating a configuration example of the determination device according to the embodiment. As shown in FIG. 2, the determination device 10 includes a communication unit 20, a storage unit 30, and a control unit 40.

  The communication unit 20 is realized by, for example, a NIC (Network Interface Card) or the like. The communication unit 20 is connected to the network N in a wired or wireless manner, and transmits/receives information to/from the terminal device 100, the administrator terminal 200, and the content server 50.

  The storage unit 30 is realized by, for example, a semiconductor memory device such as a RAM (Random Access Memory) or a flash memory (Flash Memory), or a storage device such as a hard disk or an optical disk. The storage unit 30 also stores an action history database 31, a user database 32, and classification model data 33.

  The action history database 31 stores information indicating a user's action regarding a predetermined content, that is, an action history. For example, FIG. 3 is a diagram showing an example of information registered in the action history database according to the embodiment. As shown in FIG. 3, the action history database 31 is registered with information having items such as “content ID”, “user ID”, and “contents of action”. Note that, in the action history database 31, any information other than the information shown in FIG. 3 may be registered.

  Here, the "content ID" is an identifier of the content. The "user ID" is an identifier of a user who has performed the action indicated by the associated action history. The “action content” is information indicating the action of the user, for example, information indicating the content of the click. For example, in the example shown in FIG. 3, the content ID “content #1”, the user ID “User#1”, and the action content “Click#1” are registered in association with each other. Such information indicates that the user indicated by the user ID “User#1” clicks the action indicated by the action content “Click#1” while browsing the content indicated by the content ID “content #1”. It is information to show. The action content may be not only what kind of click has been made, but also information indicating the presence or absence of a click.

  Returning to FIG. 2, the explanation will be continued. In the user database 32, a user ID that is a user identifier and user attribute information are registered in association with each item of attribute information. For example, FIG. 4 is a diagram illustrating an example of information registered in the user database according to the embodiment. For example, in the example shown in FIG. 4, the user database 32 has a plurality of pieces of information having items such as “user ID”, “attribute item”, “attribute information”, and “group” for each “user ID”. It is registered. The “attribute item” is information indicating an item of attribute information, and for example, information such as “sex”, “age”, “address”, etc. is registered. The “attribute information” is the attribute information of the user indicated by the associated user ID, and is the attribute information of the content indicated by the associated attribute item. The "group" is a value indicating a group into which the user indicated by the associated user ID is classified, that is, a value indicating attribute information of the user.

  For example, in the example shown in FIG. 4, the attribute item "sex" and the attribute information "male" are registered in association with the user ID "User#1", and the attribute item "age" ("age" may be used). ) And attribute information “30s” are registered in association with each other, attribute item “address” and attribute information “A city” are registered in association with each other, and group “135” is registered in association with each other. Such information indicates, for example, that the gender of the user indicated by the user ID “User#1” is male, the age is in the age group of thirties, and the address is A city. Further, such information indicates that the group to which the user indicated by the user ID "User#1" is to be classified is "135". That is, such information indicates that the value indicating the user attribute information indicated by the user ID “User#1” is “135”.

  In addition to the attribute items and attribute information shown in FIG. 4, arbitrary attribute items and attribute information may be registered in the user database 32. Further, the user database 32 does not necessarily need to register the attribute information for all the attribute items, but only the attribute information registered by the user or estimated from the user's action history is registered. Good.

  Returning to FIG. 2, the description will be continued. The classification model data 33 is a model for classifying users based on the attribute information of the users. For example, the classification model data 33 is a model that, when the attribute information of the user is input, calculates a value indicating the attribute of the user for each type of attribute information.

  The control unit 40 is a controller, and various programs stored in a storage device inside the determination device 10 include a RAM and the like by a processor such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit). It is realized by being executed as a work area. The control unit 40 is a controller, and may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

  As shown in FIG. 2, the control unit 40 includes a collection unit 41, a selection unit 42, a calculation unit 43, a determination unit 44, an output unit 45, and a learning unit 46. The collection unit 41 collects, from the content server 50, a behavior history indicating the behavior of the user regarding predetermined content. More specifically, the collection unit 41 has a user ID of a user who has performed an action, an action content that indicates the content of the action performed by the user, and a content ID that indicates the content that the user has browsed. The information associated with is collected from the content server 50 as an action history. Then, the collection unit 41 registers the collected action history in the action history database 31. That is, the collecting unit 41 collects the action history, collects the action history for each content browsed by the user, and registers the action history in the action history database 31.

  Further, the collection unit 41 collects user attribute information from the content server 50. Further, the collection unit 41 inputs the collected attribute information into the classification model data 33, and outputs the value output from the classification model data 33, that is, the value serving as the second index value, to the user attribute information and the user. It is registered in the user database 32 in association with the ID.

  The selection unit 42 selects the first sample and the second sample from the information registered in the action history database 31 and the user database 32. For example, when the selection unit 42 receives a request from the administrator terminal 200 to determine a significant difference between samples related to a predetermined content, the selection unit 42 assigns the content ID indicating the predetermined content (for example, the content ID “content #1”). The action history including the action history database 31 is specified.

  Specifically, the selection unit 42 randomly selects, as the first index value included in the first sample, a predetermined number of sets of user IDs and action history from the identified action history. Subsequently, the selection unit 42 selects, from the user database 32, the value of “group” associated with the same user ID as the user ID included in the set selected as the first index value, as the second index value. To do. Note that the selection unit 42 selects a predetermined number of user IDs from the user IDs included in the set selected as the first index value, and sets the selected value to the group value associated with the selected user ID. In addition, the value of the randomly selected group may be used as the second index value. Similarly, the selection unit 42 randomly selects, as the first index value included in the second sample, a set of a predetermined number of user IDs and action history from the identified action history, and selects the selected user. The value of the “group” associated with the user ID corresponding to the ID is set as the second index value included in the second sample.

  The calculation unit 43 is estimated for the predetermined content based on the first index value indicating the behavior of the users U01 and U02 regarding the predetermined content and the second index value based on the attributes of the users U01 and U02. The estimated index value indicating the behavior of the users U01 and U02 is calculated for each sample regarding a predetermined content.

  Specifically, the calculation unit 43 calculates the estimated index value as the first index value from the average value of the values indicating the contents of the selection operation performed while browsing the predetermined content. Further, the calculation unit 43 may calculate the estimated index value as the second index value from the average value of the values indicating the contents of the selection operation performed while browsing the second content. Further, the calculation unit 43 classifies the users by using the model that classifies the users according to the attributes of the users, and sets the value corresponding to the classification result as the second index value. Then, the calculation unit 43 subtracts the product of the predetermined coefficient and the average value of the second index values from the average value of the first index values to obtain the predetermined coefficient and the expected value of the second index value. The value obtained by adding the products of is taken as the estimated index value.

For example, the calculation unit 43 digitizes each of the plurality of action histories included in the first sample selected by the selection unit 42, and sets the numerical values as the first index value Y ^(t) . In addition, the calculation unit 43 digitizes the plurality of action histories included in the second sample, and sets them as the first index value Y ^(c) . Subsequently, the calculation unit 43 sets the value of the “group” included in the first sample selected by the selection unit 42 as the second index value X ^(t) . Further, the calculation unit 43 sets the value of the “group” included in the second sample selected by the selection unit 42 as the second index value X ^(c) .

Then, the calculation unit 43 uses the first index value Y ^(t) and the second index value X ^(t) to calculate the value of the estimated index value Y ^(t) _cv related to the first sample by the formula ( 3) and using the first index value Y ^(c) and the second index value X ^(c) , the value of the estimated index value Y ^(c) _cv relating to the second sample is ^calculated by the formula (3 ).

The determination unit 44 determines a significant difference between each sample based on the estimated index value calculated for each sample. Specifically, the determination unit 44 uses the estimated index values Y ^(t) _cv and Y ^(c) _cv calculated by the calculation unit 43 to calculate a value indicating a significant difference between the samples from Expression (1). , It is determined whether the calculated value is larger than a predetermined threshold value. Then, the determination unit 44 determines that there is a significant difference when the calculated value is larger than the predetermined threshold value, and determines that there is no significant difference when the calculated value is less than or equal to the predetermined threshold value. To do.

  The output unit 45 outputs the determination result of the determination unit 44 to the administrator terminal 200. For example, the output unit 45 generates a result screen C10 indicating the determination result based on the determination result executed by the determination unit 44, and distributes the generated result screen C10 to the administrator terminal 200.

  For example, FIG. 5 is a diagram illustrating an example of a result screen generated by the determination device according to the embodiment. For example, the determination apparatus 10 extracts a plurality of sets of a first sample and a second sample that are randomly selected from the user's action history regarding predetermined content, and executes the above-described determination process for each extracted set. By doing so, a value indicating a significant difference is calculated. Then, the determination device 10 generates a result screen C10 as shown in FIG. 5 and causes the administrator terminal 200 to display the generated result screen C10.

  For example, as illustrated in FIG. 5, the determination device 10 calculates a value indicating a significant difference for each set of samples, and uses the calculation result to show the set of samples in the horizontal axis direction and the significant difference in the vertical axis direction. Generate a graph with values set. Then, the determination device 10 indicates, in the graph, a range in which the value of the significant difference exceeds a predetermined threshold value by a thick line, and a range in which the value of the significant difference is equal to or less than the predetermined threshold value by a dotted line. Further, when the administrator of the administrator terminal 200 selects a position on the graph, as shown by a black circle in the graph shown in FIG. 5, the determination device 10 sets a set of samples corresponding to the selected position. A result screen C10 for displaying the action history included in each is generated.

  Returning to FIG. 2, the description will be continued. The learning unit 46 learns the classification model data 33. For example, the learning unit 46 sets the first sample and the second sample for which a significant difference has been calculated in advance as correct answer data, classifies users from correct answer data, and calculates a second index value based on the classification result. To do. Then, the learning unit 46 corrects the classification model data 33 so that the significant difference calculated when using the calculated second index value approaches the significant difference calculated in advance. The learning unit 46 may learn a model that groups the users U01 and U02 from the attributes of the users U01 and U02 by using any learning method other than the above-described learning process.

[3. Example of flow of determination process]
Subsequently, a flow of processing executed by the determination device 10 will be described with reference to FIG. FIG. 6 is a flowchart showing an example of the flow of the determination process executed by the determination device according to the embodiment. For example, the determination device 10 collects the action history and attribute information of each user (step S101). Then, the determination device 10 groups users based on the collected attribute information (step S102). Subsequently, the determination apparatus 10 calculates a first index value for each sample for each sample (step S103), and obtains a second index value based on the grouping result of the same users as the users associated with each sample. , For each sample (step S104).

  Then, the determination device 10 calculates an estimated index value for each sample based on the equation (3), that is, an equation called control variation (step S105), and uses the estimated index value to determine the significant difference between the samples. The determination is made (step S106). Then, the determination device 10 outputs the determination result (step S107) and ends the process.

[4. Modification example)
In the above, an example of the determination process by the determination device 10 has been described. However, the embodiment is not limited to this. Hereinafter, variations of the determination process executed by the determination device 10 will be described.

[4-1. About control variations]
Here, the determination device 10 determines not only the first index value regarding the predetermined content but also the second index value based on the attribute of the user to determine the significant difference between the index values regarding the predetermined content. For example, an expression other than the expression shown as the expression (3) may be used. Further, the determination device 10 may obtain the value of θ or the value of EX from the first index value and the second index value by an arbitrary method.

[4-2. Regarding the second index value]
Here, in the control variation formula shown as the formula (3), arbitrary information can be applied to X. Therefore, the determination device 10 determines a value indicating the behavior of the user regarding a web page different from a predetermined web page, such as a web page having a cross-domain relationship with the web page related to the behavior indicated by the first index value. It may be the second index value.

  For example, the determination device 10 sets the value indicating the behavior of the user regarding the predetermined content as the first index value when determining the significant difference between the samples of the behavior of the user regarding the predetermined content. Further, the determination device 10 sets the value indicating the action taken by the user while browsing the second content having a cross domain relationship with the predetermined content as the second index value. Then, the determination device 10 may calculate an estimated index value for each sample and determine a significant difference between samples based on the calculated estimated index value.

  Note that the determination device 10 may use, as the second index value, a value corresponding to the action performed by the user while browsing the second content and the attribute of the user. For example, the determination device 10 may use, as the second index value, a value obtained by arranging a value indicating the behavior of the user while browsing the second content and a value indicating the attribute information of the user.

  Further, in another example, the determination device 10 performs clustering in advance with all the users who have collected the action history as the target of clustering. Further, the determination apparatus 10 selects the first sample and the second sample from the action history regarding the predetermined content. In addition, the determination device 10 determines, for each sample, a first index value indicating the behavior of the user while browsing the predetermined content, the behavior of the user while browsing the second content, and the utilization thereof. A second index value indicating a combination of groups in which the person is clustered. Then, the determination device 10 may calculate an estimated index value for each sample from the first index value and the second index value, and determine a significant difference between each sample based on the estimated index value. Note that the determination device 10 does not need to target all users who have collected the action history as clustering targets, and may target only users whose user IDs are included in the first sample as clustering targets, for example.

[4-3. About judgment target and judgment contents]
Here, the determination apparatus 10 calculates the estimated index value using the first index value indicating the behavior of the user regarding the predetermined content, and determines the significant difference between the samples using the estimated index value. That is, the determination device 10 determines the significant difference in the information indicating the behavior of the user regarding the predetermined content. Here, if the determination device 10 indicates the behavior of the user regarding the predetermined content, the determination device 10 may determine the significant difference in the information by using the information indicating the arbitrary behavior as the first index value. For example, the determination device 10 uses the content of the evaluation performed on the predetermined content by the user as the first index, such as the result of the AB test, the evaluation of the product or any post, the evaluation of various contents such as music and movies, and the like. It may be a value, or may be a significant difference determination target.

  Further, the determination apparatus 10 may use the above-mentioned estimated index value to determine a significant difference between each sample by using any statistical test method such as t-test in addition to the Z test. ..

[4-4. Device configuration〕
The determination device 10 may be realized by a front-end server that exchanges information with the terminal device 100 or the administrator terminal 200, and a back-end server that exchanges information with the content server 50 and executes the determination process described above. .. In such a case, the output unit 45 shown in FIG. 2 is arranged in the front-end server, and the collection unit 41, the selection unit 42, the calculation unit 43, the determination unit 44, the action history database 31, and the back-end server. The user database 32 will be arranged. The databases 31 and 32 and the classification model data 33 stored in the storage unit 30 may be stored not in the determination device 10 but in an external storage server or the like, for example.

[4-5. Other]
Further, of the processes described in the above embodiment, all or part of the processes described as being automatically performed may be manually performed, or the processes described as being manually performed may be performed. All or part of the process can be automatically performed by a known method. In addition, the processing procedures, specific names, information including various data and parameters shown in the above-mentioned documents and drawings can be arbitrarily changed unless otherwise specified. For example, the various kinds of information shown in each drawing are not limited to the illustrated information.

  Further, each constituent element of each illustrated device is functionally conceptual, and does not necessarily have to be physically configured as illustrated. That is, the specific form of distribution/integration of each device is not limited to that shown in the figure, and all or a part of the device may be functionally or physically distributed/arranged in arbitrary units according to various loads or usage conditions. It can be integrated and configured.

  Further, the respective embodiments described above can be appropriately combined within a range in which the processing content is not inconsistent.

[5. program〕
Further, the determination device 10 according to the above-described embodiment is realized by, for example, a computer 1000 configured as shown in FIG. 7. FIG. 7 is a diagram illustrating an example of the hardware configuration. The computer 1000 is connected to an output device 1010 and an input device 1020, and an arithmetic device 1030, a primary storage device 1040, a secondary storage device 1050, an output IF (Interface) 1060, an input IF 1070, and a network IF 1080 are connected by a bus 1090. Have.

  The arithmetic unit 1030 operates based on a program stored in the primary storage device 1040 or the secondary storage device 1050, a program read from the input device 1020, or the like, and executes various processes. The primary storage device 1040 is a memory device such as a RAM that temporarily stores data used by the arithmetic device 1030 for various calculations. The secondary storage device 1050 is a storage device in which data used by the arithmetic device 1030 for various calculations and various databases are registered, and is realized by a ROM (Read Only Memory), an HDD, a flash memory, or the like.

  The output IF 1060 is an interface for transmitting information to be output to an output device 1010 that outputs various kinds of information such as a monitor and a printer. For example, a USB (Universal Serial Bus) or a DVI (Digital Visual Interface), It is realized by a connector of a standard such as HDMI (registered trademark) (High Definition Multimedia Interface). The input IF 1070 is an interface for receiving information from various input devices 1020 such as a mouse, a keyboard, and a scanner, and is realized by, for example, USB.

  The input device 1020 is, for example, an optical recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), or a tape. It may be a device that reads information from a medium, a magnetic recording medium, a semiconductor memory, or the like. The input device 1020 may also be an external storage medium such as a USB memory.

  The network IF 1080 receives data from another device via the network N and sends the data to the arithmetic device 1030, and also transmits data generated by the arithmetic device 1030 via the network N to another device.

  The arithmetic device 1030 controls the output device 1010 and the input device 1020 via the output IF 1060 and the input IF 1070. For example, the arithmetic device 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040, and executes the loaded program.

  For example, when the computer 1000 functions as the determination device 10, the arithmetic device 1030 of the computer 1000 implements the function of the control unit 40 by executing the program loaded on the primary storage device 1040.

[6. effect〕
As described above, the determination device 10 sets the predetermined index value based on the first index value indicating the behavior of the users U01 and U02 regarding the predetermined content and the second index value based on the attributes of the users U01 and U02. The estimated index value indicating the behavior of the user estimated with respect to the content is calculated for each sample regarding the predetermined content. Then, the determination device 10 determines a significant difference between each sample based on the estimated index value calculated for each sample. As a result, the determination device 10 can improve the coverage without deteriorating the correlation of the information used for the determination of the significant difference, so that the variance of the estimated index value is reduced and the detection accuracy of the significant difference is improved. be able to.

  The determination device 10 also classifies the users U01 and U02 using a model that classifies according to the attributes of the users U01 and U02, and sets the value according to the classification result as the second index value. Therefore, the determination device 10 can reduce the variance of the estimated index value and improve the detection accuracy of the significant difference with a simple process.

  Further, the determination device 10 calculates the estimated index value as the first index value from the average value of the values indicating the contents of the selection operation performed while browsing the predetermined content. Therefore, the determination device 10 can appropriately calculate the estimated index value.

  Further, the determination device 10 subtracts the product of the predetermined coefficient and the average value of the second index values from the average value of the first index values to obtain the predetermined coefficient and the expected value of the second index value. The value obtained by adding the products of is taken as the estimated index value. Therefore, the determination device 10 can reduce the variance of the estimated index values below the normal average value, and thus can improve the accuracy of detecting a significant difference.

  In addition, the determination device 10 determines whether a predetermined number of the first index value indicating an action performed by a plurality of users with respect to a predetermined content and a plurality of users performing an action indicated by the first index value. The estimated index value is calculated using the second index value based on the attributes of the plurality of users including the user. Therefore, the determination device 10 can appropriately determine the significant difference.

  The determination device 10 also outputs the determination result. Therefore, the determination device 10 can provide the determination result to the administrator or the like.

  As described above, some of the embodiments of the present application have been described in detail based on the drawings, but these are examples, and various modifications based on the knowledge of those skilled in the art, including the modes described in the section of the disclosure of the invention, It is possible to implement the present invention in other forms with improvements.

  Further, the above-mentioned "section (module, unit)" can be read as "means" or "circuit". For example, the calculation unit can be read as a calculation unit or a calculation circuit.

10 determination device 20 communication unit 30 storage unit 31 action history database 32 user database 33 classification model data 33
40 control unit 41 collection unit 42 selection unit 43 calculation unit 44 determination unit 45 output unit 46 learning unit 50 content server 100 to 102 terminal device 200 administrator terminal

Claims (8)

A user belonging to the first sample and the user belonging to the second sample are randomly selected, and for each sample, a set of values indicating the behavior of the user belonging to the sample and indicating the behavior of each user regarding a predetermined content. in the first and the index value, based on the second index value based on the set of values indicating the attributes of each user, a value that indicates the behavior of the average user to be estimated for the predetermined content based on a calculation unit for de San some estimate index value,
A determination unit that determines a significant difference between each sample based on the estimated index value calculated for each sample. The calculation unit classifies the users by using a model that performs classification according to the attributes of the users, and sets a set of values according to the classification result as the second index value. The determination device according to claim 1. The calculation unit, judgment unit according to the mean value before Symbol value indicating the predetermined content selection operation made during viewing of contents to claim 1 or 2, characterized in that said first index value .. The calculation unit calculates a first index value that is an average of values that indicate the behavior of each user regarding the predetermined content and a second index value that is an average of values that indicate the attributes of each user. , said first index value or, et al., subtracts the product of the a predetermined coefficient the second index value, a value obtained by adding the product of the expected value of the with the predetermined coefficient a second index value, The determination device according to claim 1, wherein the determination index value is used as the estimated index value. The calculation unit determines a first index value based on a value indicating an action performed by a plurality of users with respect to the predetermined content, and a predetermined index among a plurality of users who performed the action indicated by the first index value. The estimated index value is calculated using a second index value based on a value indicating an attribute of a plurality of users including the number of users. Judgment device. The determination device according to claim 1, further comprising: an output unit that outputs a determination result by the determination unit. A determination method executed by the determination device,
A user belonging to the first sample and the user belonging to the second sample are randomly selected, and for each sample, a set of values indicating the behavior of the user belonging to the sample and indicating the behavior of each user regarding a predetermined content. in the first and the index value, based on the second index value based on the set of values indicating the attributes of each user, a value that indicates the behavior of the average user to be estimated for the predetermined content based on a calculation step of de San some estimate index value,
A determination step of determining a significant difference between each sample based on the estimated index value calculated for each sample. A user belonging to the first sample and the user belonging to the second sample are randomly selected, and for each sample, a set of values indicating the behavior of the user belonging to the sample and indicating the behavior of each user regarding a predetermined content. in the first and the index value, based on the second index value based on the set of values indicating the attributes of each user, a value that indicates the behavior of the average user to be estimated for the predetermined content based on a calculation procedure for de San certain estimated index value,
A determination program for causing a computer to execute a determination procedure for determining a significant difference between samples based on the estimated index value calculated for each sample.

Images