JP2019053620A - Generation apparatus, generation method, and generation program - Google Patents

Abstract

To provide a generation apparatus for properly generating a model corresponding to an index for behavior of a user, a generation method, and a generation program.SOLUTION: A generation apparatus includes an acquisition unit and a generation unit. The acquisition unit acquires content information on a content. The generation unit generates evaluation information on a model, on the basis of an output value output from the model by inputting the content information acquired by the acquisition unit to the model which outputs a score corresponding to an index for user behavior relating to the content, and prediction information predicted from information different from the model.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a generation device, a generation method, and a generation program.

  In recent years, with the rapid spread of the Internet, sales via the Internet by electronic commerce services and the like have become popular. For example, a technique for providing bid advice regarding the purchase of similar products from a plurality of different types of online auctions is provided.

JP 2003-281410 A

  However, in the above-described conventional technology, it may be difficult to appropriately generate a model corresponding to an index related to user behavior. For example, when there is little log information related to a certain index, it is difficult to appropriately generate a model corresponding to that index because the amount of learning data used for learning is insufficient.

  The present application has been made in view of the above, and an object thereof is to provide a generation device, a generation method, and a generation program that can appropriately generate a model corresponding to an index related to a user's behavior.

  The generation apparatus according to the present application inputs an acquisition unit that acquires content information related to content, and the content information acquired by the acquisition unit into a model that outputs a score corresponding to an index related to the user's behavior regarding the content. And a generation unit configured to generate evaluation information related to the model based on an output value output from the model and prediction information predicted from information different from the model.

  According to one aspect of the embodiment, there is an effect that it is possible to appropriately generate a model corresponding to an index related to user behavior.

FIG. 1 is a diagram illustrating an example of a generation process according to the embodiment. FIG. 2 is a diagram illustrating an example of distribution processing according to the embodiment. FIG. 3 is a diagram illustrating a configuration example of a generation system according to the embodiment. FIG. 4 is a diagram illustrating a configuration example of the generation apparatus according to the embodiment. FIG. 5 is a diagram illustrating an example of the first learning data storage unit according to the embodiment. FIG. 6 is a diagram illustrating an example of the prediction information storage unit according to the embodiment. FIG. 7 is a diagram illustrating an example of the second learning data storage unit according to the embodiment. FIG. 8 is a diagram illustrating an example of a model information storage unit according to the embodiment. FIG. 9 is a diagram illustrating an example of a user information storage unit according to the embodiment. FIG. 10 is a diagram illustrating an example of the log information storage unit according to the embodiment. FIG. 11 is a diagram illustrating an example of a product information storage unit according to the embodiment. FIG. 12 is a flowchart illustrating an example of the generation process according to the embodiment. FIG. 13 is a flowchart illustrating an example of distribution processing according to the embodiment. FIG. 14 is a diagram illustrating a change in the magnification based on the required time according to the embodiment. FIG. 15 is a hardware configuration diagram illustrating an example of a computer that realizes the function of the generation apparatus.

  Hereinafter, a generation apparatus, a generation method, and a mode for executing a generation program (hereinafter referred to as “embodiment”) according to the present application will be described in detail with reference to the drawings. Note that the generation device, the generation method, and the generation program according to the present application are not limited by this embodiment. In the following embodiments, the same portions are denoted by the same reference numerals, and redundant description is omitted.

(Embodiment)
[1. Generation process)
First, an example of the generation process according to the embodiment will be described with reference to FIG. FIG. 1 is a diagram illustrating an example of a generation process according to the embodiment. In FIG. 1, an output value output from a model when the generation apparatus 100 inputs a score corresponding to an index related to a user's behavior related to content, and prediction information predicted from information different from the model. The case where the evaluation information regarding a model is produced | generated based on is shown. In the example of FIG. 1, as an example, an example of an index indicating whether or not a product corresponding to distributed content can be sold quickly is shown as an index related to user behavior regarding the content. It is assumed that a product corresponding to the distributed content is sold quickly when purchased by the user within a predetermined period (for example, within 24 hours) from the start of content distribution. In addition, the index used as the object which produces | generates evaluation information is not restricted to "early sale", A various index may be sufficient. For example, the indices targeted by the model may be various indices such as “click through rate (CTR)”, “conversion rate (CVR)”, and “selling high”.

[Configuration of generation system]
First, the generation system 1 shown in FIG. 3 will be described prior to the description of FIG. As illustrated in FIG. 3, the generation system 1 includes a terminal device 10 and a generation device 100. The terminal device 10 and the generation device 100 are connected via a predetermined network N so as to be communicable by wire or wirelessly. FIG. 3 is a diagram illustrating a configuration example of a generation system according to the embodiment. Note that the generation system 1 illustrated in FIG. 3 may include a plurality of terminal devices 10 and a plurality of generation devices 100.

  The terminal device 10 is an information processing device used by a user. The terminal device 10 is realized by, for example, a smartphone, a tablet terminal, a notebook PC (Personal Computer), a desktop PC, a mobile phone, a PDA (Personal Digital Assistant), or the like. In the example shown in FIG. 1, the case where the terminal device 10 is a smartphone used by the user is shown. Hereinafter, the terminal device 10 may be referred to as a user. That is, hereinafter, the user can be read as the terminal device 10. For example, FIG. 2 illustrates a case where the terminal device 10 is a smartphone used by a user identified by the user ID “U101” (hereinafter, may be referred to as “user U101”). Note that, as described above, “user U * (* is an arbitrary numerical value)” indicates that the user is a user identified by the user ID “U *”. For example, when “user U101” is described, the user is a user identified by the user ID “U101”.

  The generation device 100 is an information processing device that generates a model that outputs a score indicating an evaluation based on an index corresponding to a user's behavior related to content using content information related to content. Further, the generation device 100 generates prediction information from information different from the model. In the example of FIG. 1, the generation device 100 generates prediction information for predicting a score from statistical information related to content. Further, the generation apparatus 100 generates evaluation information related to the model based on the output value (score) output from the model by inputting the content information and the prediction information.

  Further, in the generation system 1, the generation apparatus 100 provides an electronic commerce service that enables purchase of products provided by a plurality of vendors. That is, in the generation system 1, an electronic commerce service is provided that can purchase commodities provided by a plurality of vendors.

  The generation apparatus 100 may provide various services such as an electronic commerce service and a search service. Further, when an external information processing apparatus provides a search service or an electronic commerce service, the generation apparatus 100 may not provide a search service or an electronic commerce service. In this case, the generation apparatus 100 acquires various types of information from an external information processing apparatus that provides a search service or an electronic commerce service, and performs generation processing or information provision to the external information processing apparatus based on the acquired various types of information. You may go.

  In the example of FIG. 1, the generation apparatus 100 distributes content to a large number of users such as the users U1 to U5 using a model M1 corresponding to the index “Click Through Rate (CTR)”. Shall. That is, in the example of FIG. 1, it is assumed that the generation apparatus 100 has acquired sufficient information for generating the model M1. For example, the generation device 100 generates a model using the user's behavior related to the index “CTR” as correct answer information. Hereinafter, the target information associated with the correct answer information is also referred to as “learning data”.

  For example, the generation device 100 collects information for generating correct answer information. In the example of FIG. 1, the generation apparatus 100 distributes content such as a product GD101 (A product) to a product GD105 (E product) to the user, and acquires information (behavior information) related to the user's behavior with respect to the distributed content.

  As described above, when “product GD * (* is an arbitrary numerical value)” is described, it indicates that the product is a product identified by the product ID “GD *”. For example, when “product GD1” is described, the product is a product identified by the product ID “GD1”. Hereinafter, the product information of “product GD * (* is an arbitrary numerical value)” may be expressed as “product GD * (* is an arbitrary numerical value)”. For example, hereinafter, the product GD1 can be read as product information (content) of the product GD1. In addition, although the five users U1-U5 are illustrated in the 1st learning data storage part 121 in FIG. 1, the production | generation apparatus 100 is not restricted to the users U1-U5, but many users (for example, 1 million users). And action information on content distributed by 10 million users). In the example of FIG. 1, the generation device 100 adds data DT101 to DT105 and the like corresponding to each of the products GD101 to GD105 to which the content is distributed to the first learning data storage unit 121.

  As described above, when “data DT * (* is an arbitrary numerical value)” is described, it indicates that the data is data identified by the data ID “DT *”. For example, when “data DT1” is described, the data is data identified by the data ID “DT1”.

  “Data ID” shown in the first learning data storage unit 121 in FIG. 1 indicates identification information for identifying data. The “target product” shown in the first learning data storage unit 121 in FIG. 1 indicates a target product in the data identified by the data ID.

  The “correct information” shown in the first learning data storage unit 121 in FIG. 1 indicates correct information corresponding to the data identified by the data ID. “Click” in “correct answer information” shown in the first learning data storage unit 121 in FIG. 1 indicates whether or not the content of the corresponding target product has been clicked by the user of the distribution destination. For example, when “click” is “1”, it indicates that the content of the corresponding target product has been clicked by the distribution destination user. For example, when “click” is “0”, it indicates that the content of the corresponding target product has not been clicked by the distribution destination user.

  “Sell early” in the “prediction information” shown in the first learning data storage unit 121 in FIG. 1 indicates that the target product may be purchased early by the user who is the distribution destination of the content of the corresponding target product. Is indicated by a score, details will be described later. “Content information” shown in the first learning data storage unit 121 in FIG. 1 indicates information related to the content corresponding to the data identified by the data ID. “Content information” indicates information related to distribution of content corresponding to data identified by the data ID.

  “Content data” in “content information” shown in the first learning data storage unit 121 in FIG. 1 indicates content data provided to the user in the distribution corresponding to the data identified by the data ID. FIG. 5 shows an example in which conceptual information such as “CDT101” is stored in “content data”, but in practice, various information related to content or a file path name indicating the storage location is stored. The For example, the “content data” includes various types of information such as product names, categories, images, titles, and text information about the corresponding products. For example, “content data” may be static information that does not vary depending on a user who is a distribution destination or a distribution date and time, among information regarding the content. For example, the “content data” may include a price at which a contract in a free market format is established and a price (prompt decision price) at which a contract is established even within an auction period in an auction format. Hereinafter, the flea market format may be referred to as “free market format”, and the auction format in which the prompt decision price is set may be referred to as “prompt decision auction format”.

  “Log data” in “content information” shown in the first learning data storage unit 121 in FIG. 1 indicates history information in content distribution of the target product corresponding to the data identified by the data ID. The “log data” includes various users at the time of distribution and the user who is the distribution destination of the content distribution of the target product corresponding to the data identified by the data ID. FIG. 5 shows an example in which conceptual information such as “LDT101” is stored in “log data”, but in practice, various information related to content distribution or a file path name indicating the storage location is stored. Is done. For example, the “log data” may include information regarding a user who is a distribution destination of content distribution of the target product corresponding to the data identified by the data ID.

  Further, for example, the “log data” may include dynamic information that dynamically varies depending on a user's action (bid) regarding the target product, and dynamic information at the time of content distribution. For example, the “log data” may include various information such as the current price, the number of bids, the number of bidders, and the remaining time at the time of content distribution. Note that the above is an example, and the log data is not limited to the above and may include various information. Note that “content data” and “log data” may include overlapping information as long as they are information about content.

  For example, in the example illustrated in FIG. 1, data (data DT101) identified by the data ID “DT101” indicates that the target product is the product GD101 (A product). Further, the data DT101 indicates that “click” is “1” in the correct answer information. Data DT101 indicates that “content data” is “CDT101” and “log data” is “LDT101 (U1 etc.)” in the content information.

  That is, the data DT101 indicates that it is data relating to the distribution of the content of the product GD101 (A product) to the user U1. Data DT101 indicates that the user U1 has clicked on the content of the distributed product GD101.

  In addition, although illustration is abbreviate | omitted in the 1st learning data memory | storage part 121 in FIG. 1, the item of index "it sells early" is contained in correct answer information. For example, in the first learning data storage unit 121 in FIG. 1, for a very small number of contents (for example, less than 1% of the whole contents), the correct answer information is acquired for the index “selling quickly”. To do. For example, in the first learning data storage unit 121 in FIG. 1, it is assumed that correct information about the index “sell early” is also acquired for content related to products sold in a flea market format or a prompt auction format.

  For example, “sell early” in the correct information in the first learning data storage unit 121 in FIG. 1 indicates whether or not the user who is the distribution destination of the content of the corresponding target product has purchased the target product. For example, when “sell early” is “1”, it indicates that the user has purchased the content within a predetermined period after the distribution of the content of the corresponding target product is started. For example, when “selling early” is “1”, it indicates that the user has purchased the content within 24 hours after the distribution of the content of the corresponding target product is started. In addition, the time indicated when “sell early” is “1” indicates the time elapsed from the start of the distribution of the content of the corresponding target product to the purchase by the user. Further, for example, when “selling early” is “0”, it indicates that the user has not purchased the content within a predetermined period after the distribution of the content of the corresponding target product is started.

  For example, in the example of FIG. 1, the first learning data storage unit 121 indicates that the user U501 has purchased the product GD501 one hour after the start of the distribution of the content of the product GD501 in the flea format, although it is a small number. Learning data shall be included. In the example of FIG. 1, the generation device 100 generates a model M2 from information in the first learning data storage unit 121 using information that can be used to generate the model M2.

  In the example of FIG. 1, it is assumed that the generation device 100 has already generated each model based on the learning data. For example, the generating apparatus 100 generates a model by performing learning using the data DT101 to DT105 and the like in the first learning data storage unit 121 as learning data (teacher data). For example, the generating apparatus 100 generates the model M1 by performing learning using the data DT101 to DT105 in the first learning data storage unit 121 as learning data (teacher data).

  Further, for example, the generation device 100 learns the data DT101 to DT105 in the first learning data storage unit 121 using data that can be used to generate the model M2 among the learning data (teacher data). Thus, the model M2 is generated. For example, the generating apparatus 100 uses the data DT101 to DT105 and the like in the first learning data storage unit 121 using the data associated with the correct answer information corresponding to the index “sell early” among the learning data (teacher data). The model M2 is generated by performing learning. As described above, in the example of FIG. 1, the generation apparatus 100 extracts data that can be used to generate the model M2 from the data in the first learning data storage unit 121, and uses the extracted data to extract the model M2. Generate. Therefore, in the example of FIG. 1, the generation apparatus 100 generates the model M2 using data with a data amount smaller than the data amount available for generating the model M1.

  For example, the generation device 100 generates a model using a feature amount related to content. For example, the generation apparatus 100 generates a model using a feature amount related to the content content. For example, the generation apparatus 100 generates a model using a feature amount related to a product corresponding to the content. For example, the generating apparatus 100 generates a model using a feature amount related to a product category or sales mode corresponding to the content. For example, the generation apparatus 100 generates a model using a feature amount related to a user who is a content distribution destination. In the example of FIG. 1, the generation apparatus 100 learns characteristics regarding a product or the like corresponding to the content included in the content information, and generates a model that predicts whether or not the product corresponding to the content is purchased by the user. The features included in the content information learned by the generation apparatus 100 may be input to the generation apparatus 100 by a person such as an administrator of the generation apparatus 100 or may be automatically learned (extracted) by the generation apparatus 100. Also good. Note that the generation apparatus 100 may generate the model M2 using the same feature amount (feature) as the model M1.

  For example, the generating apparatus 100 generates the model M2 using data indicating that the user U501 has purchased the product GD501. For example, the generation apparatus 100 inputs the content information corresponding to the data (hereinafter, also referred to as “early sale data”) into the model M2 for the data whose correct information is the index “early sell” is “1”. Learning process is performed so that the score output by the model M2 approaches “1”. For example, when content data, log data, or the like of early selling data is input to the model M2, the generation device 100 performs a learning process so that the score output by the model M2 approaches “1”. Note that the generation apparatus 100 may change the weighting for the index “sell early” based on the time from the start of sales to purchase, and details of this point will be described later.

  Further, for example, for the data whose correct information is the index “early sell” is “0”, the generation apparatus 100 stores content information corresponding to the data (hereinafter also referred to as “non-early sell data”) in the model M2. Is input, the learning process is performed so that the score output by the model M2 approaches “0”. For example, when content data, log data, or the like of non-early sale data is input to the model M2, the generation device 100 performs a learning process so that the score output by the model M2 approaches “1”.

  Note that the model learning technique is not limited to the technique described above, and any known technique can be applied. Each model may be generated using various conventional techniques relating to machine learning as appropriate. For example, the model generation may be performed using a technique related to machine learning of supervised learning such as SVM (Support Vector Machine). Further, for example, the model generation may be performed using a technique related to machine learning of unsupervised learning. For example, the generation of the model may be performed using a deep learning technique. For example, the model generation may be performed by appropriately using various deep learning techniques such as DNN (Deep Neural Network), RNN (Recurrent Neural Network), and CNN (Convolutional Neural Network). The description relating to the generation of the model is merely an example, and the generation of the model may be performed by a learning method appropriately selected according to information that can be acquired. That is, when the content information included in the learning data is input, the generation apparatus 100 can generate the model M2 by any method as long as the model M2 can be learned so as to output a score corresponding to the correct answer information. May be performed.

  In the example of FIG. 1, the generation apparatus 100 generates a model (model M2) identified by the model ID “M2” as shown in the model information storage unit 124 by the processing as described above. As described above, when “model M * (* is an arbitrary numerical value)” is described, it indicates that the model is a model identified by the model ID “M *”. For example, when “model M2” is described, the model is a model identified by the model ID “M2”. Further, as shown in the model information storage unit 124 in FIG. 1, the model M2 is a prediction target “sell early” indicating an index to be predicted, that is, a product that is a target of content delivered to the user is sooner to the user. This is a model used for predicting whether or not the product will be purchased, and indicates that the specific model data is “model data MDT2”. For example, the generating apparatus 100 inputs content information to the model M2, and causes the model M2 to output a score indicating whether a product corresponding to the input content information can be sold quickly. And the production | generation apparatus 100 estimates evaluation of the content corresponding to the parameter | index corresponding to the user's action regarding a content based on the score which the model M2 outputs. Specifically, the generating apparatus 100 predicts the evaluation of the content corresponding to the index “sell early” indicating whether the product corresponding to the content sells quickly based on the score output by the model M2.

  Here, as described above, in content distribution using the model M1 corresponding to the different index “CTR”, the generation apparatus 100 has sufficient correct answer information to generate the model M2 corresponding to the index “selling fast”. There are many cases that cannot be obtained. Therefore, the generation apparatus 100 needs to evaluate whether or not the model M2 can accurately predict the index “sell early”. Therefore, in the example of FIG. 1, the generation apparatus 100 generates evaluation information in order to evaluate whether the model M2 can accurately predict the index “sell early”.

  First, the generation apparatus 100 acquires prediction information corresponding to the target index “selling early” (step S11). In the example of FIG. 1, the generation apparatus 100 stores prediction information corresponding to the index “sell early” from the prediction information storage unit 122.

  The “target ID” shown in the prediction information storage unit 122 in FIG. 1 indicates identification information for identifying a target. The “target” shown in the prediction information storage unit 122 in FIG. 1 indicates a target identified by the corresponding target ID. The “element” shown in the prediction information storage unit 122 in FIG. 1 includes items such as “category” and “keyword”. Further, “content” shown in the prediction information storage unit 122 in FIG. 1 includes items such as “trend” and “score”. For example, “trend” indicates a statistical tendency of each element. For example, “score” indicates a statistical tendency of each element as a score.

  The “category” shown in the prediction information storage unit 122 in FIG. 1 indicates a statistical tendency for each category related to the corresponding target. “Category” includes items such as “A category”, “B category”, and “C category”. Further, for example, the category such as “A category” may be “fashion” or “food”. Further, for example, the category such as “A category” may be “shoes” or “drinking water”. For example, the “category” may include categories of various levels.

  Further, “keyword” shown in the prediction information storage unit 122 in FIG. 1 indicates a statistical tendency for each keyword related to the corresponding target. The “keyword” includes items such as “keyword A” and “keyword B”. For example, a keyword such as “keyword A” is not limited to a specific product name or the like, but may be an abbreviation or popular name thereof. A keyword such as “keyword A” may be “re-arrival” or “popular product”. ”,“ Limited to the actual product ”,“ same day delivery ”, or the like.

  For example, in the example illustrated in FIG. 1, the target (target TG1) identified by the target ID “TG1” is the target “sell early”, and the product that is the target of the content distributed to the user is purchased by the user earlier. Indicates that

  Further, the target TG1 indicates that the tendency of the A category is “medium (average A time)” and the score is “0.2”. For example, in the case where the target TG1 has a product category A, the product is purchased in an average A time, has a moderate tendency to sell quickly, and the score indicating the evaluation is “0.2”. It shows that.

  For example, the closer the score is to “1”, the higher the possibility that a product belonging to that category will be sold earlier. For example, the score of each category may be closer to “1” as the average time until products of the category are purchased is shorter than the average time until products of all categories are purchased. Note that the above is an example, and the score of each category may be determined using various information as appropriate. For example, the generating apparatus 100 may determine the score of each category using various statistical information such as AUC (Area Under the Curve).

  Further, the target TG1 indicates that the tendency of the B category is “high (average B time)” and the score is “0.5”. For example, when the category of the product is the B category, the target TG1 indicates that the product is purchased in an average B time, tends to sell quickly, and the score indicating the evaluation is “0.5”. .

  Further, the target TG1 indicates that the tendency of the keyword A is “low (average D time)” and the score is “0.05”. For example, if the target TG1 includes the keyword A in the content information of the product, the product is purchased with an average D time, the tendency to sell quickly is low, and the score indicating the evaluation is “0.05”. It shows that.

  For example, the closer the score is to “1”, the higher the probability that a product whose keyword is included in the content information will be sold earlier. For example, the score of each keyword is closer to “1” as the average time until the product including the keyword is included in the content information is shorter than the average time until the product of all keywords is purchased. It may be. Note that the above is an example, and the score of each keyword may be determined using various information as appropriate.

  Further, the target TG1 indicates that the tendency of the keyword B is “high (average E time)” and the score is “0.6”. For example, if the target content TG1 includes the keyword B in the content information of the product, the product is purchased with an average E time, tends to sell quickly, and the score indicating the evaluation is “0.6”. It shows that.

  In the example of FIG. 1, it is assumed that the generation apparatus 100 has acquired prediction information as shown in the prediction information storage unit 122. For example, the generation apparatus 100 may generate prediction information as shown in the prediction information storage unit 122 based on purchase information of products in past content distribution or may be acquired from an external apparatus.

  Further, the generation device 100 generates prediction information corresponding to the index “selling early” (step S12). In the example of FIG. 1, as illustrated in the first learning data storage unit 121, the generation apparatus 100 generates prediction information corresponding to the index “sell early” for the data DT101 to DT105 and the like. For example, for the data DT101 to DT105, the generation device 100 generates prediction information corresponding to the index “sell early” based on the product category corresponding to each data, the keyword included in the content data, and the like. For example, the generation device 100 generates prediction information corresponding to the index “sell early” based on the ratio of the number of distributions sold quickly for each category, the similarity of words (keywords) common to the products sold early, and the like. To do.

  For example, the generating apparatus 100 calculates a score corresponding to the index “sell early” of the data DT101 based on the score of the category to which the product GD101 of the data DT101 belongs and the score of each keyword included in the content data CDT101. For example, the generation apparatus 100 may calculate the score of the category to which the product GD101 of the data DT101 belongs and the average score of each keyword included in the content data CDT101 as the score corresponding to the index “sell early” of the data DT101. Good. Note that the above is an example, and the generation apparatus 100 may calculate the prediction information (score) corresponding to the index “sell early” of the data DT101 using various information as appropriate.

  In the example of FIG. 1, the generation apparatus 100 calculates the prediction information of the index “selling early” as the score “0.5” for the data DT101 (A product). Further, the generation device 100 calculates the prediction information of the index “sell early” as the score “0.75” for the data DT102 (B product).

  And the production | generation apparatus 100 selects the model used as evaluation object (step S13). In the example of FIG. 1, the generation device 100 selects the model M2 corresponding to the index “sell early” to be evaluated.

  And the production | generation apparatus 100 outputs a score to the model M2 by inputting content information into the model M2. In the example of FIG. 1, the generation device 100 inputs content data, log data, and the like such as the products GD101 to GD105 stored in the first learning data storage unit 121 to the model M2.

  In the example of FIG. 1, the generation apparatus 100 calculates a score indicating the possibility that the product GD101 (A product) can be quickly sold to the user U1 as a distribution destination by the processing as shown in the processing group PS12-1. The generating apparatus 100 inputs the content information DT101 to the model M2 (Step S14-1). Specifically, the generation apparatus 100 inputs data DT101 including the content data CDT101 of the A product, information about the user U1, and the like to the model M2. The model M2 to which the data DT1 is input outputs a score (step S15-1). In the example of FIG. 1, the model M2 to which the data DT101 is input outputs a score “0.55” as indicated by the score SC1-1.

  Further, the generation apparatus 100 calculates a score indicating the possibility that the product GD102 (B product) can be sold to the user U2 as a delivery destination quickly by the processing as shown in the processing group PS12-2. The generation device 100 inputs the content information DT102 to the model M2 (Step S14-2). Specifically, the generation apparatus 100 inputs data DT102 including content data CDT102 of the B product, information on the user U2, and the like to the model M2. The model M2 to which the data DT102 is input outputs a score (step S15-2). In the example of FIG. 1, the model M2 to which the data DT2 is input outputs a score “0.5” as indicated by the score SC1-2.

  In the example of FIG. 1, the generation device 100 generates a score list including scores corresponding to the products GD101 to GD105 and the like by processing as shown in the processing groups PS12-1 and PS12-2 (step S16). In the example of FIG. 1, the generation device 100 predicts a score corresponding to each of the products GD101 to GD105 (A to E products), as shown in the score list SL1.

  And the production | generation apparatus 100 produces | generates evaluation information based on the output value of the model M2, and prediction information (step S17). In the example of FIG. 1, the generation apparatus 100 has a score corresponding to each of the products GD101 to GD105 (A to E products) as shown in the score list SL1 and an index “sold early” shown in the first learning data storage unit 121. Evaluation information is generated based on the prediction information corresponding to.

  For example, the generation apparatus 100 predicts corresponding to the scores corresponding to the products GD101 to GD105 (AE products) as shown in the score list SL1 and the index “sell early” shown in the first learning data storage unit 121. Evaluation information is generated based on the comparison with the information (score).

  For example, the generation device 100 generates evaluation information according to a difference between a score (output score) of each product output by the model M2 and a score (prediction score) that is prediction information. For example, the generating apparatus 100 generates evaluation information according to a comparison between an average (error average) of a difference between an output score of each product and a predicted score of each product and a predetermined threshold value.

  For example, when the error average of the model to be evaluated is less than the first threshold, the generation apparatus 100 generates evaluation information that sets the accuracy of the model to “low”. Further, for example, when the error average of the model to be evaluated is equal to or greater than the first threshold and less than the second threshold greater than the first threshold, the generation apparatus 100 sets evaluation information that sets the accuracy of the model to “medium”. Generate. Further, for example, when the error average of the model to be evaluated is equal to or greater than the second threshold value, the generation apparatus 100 generates evaluation information that sets the accuracy of the model to “high”.

  In the example of FIG. 1, the generation device 100 generates evaluation information RP11. For example, the generation device 100 calculates an error average of the model M2 to be evaluated as an error ER1 that is less than the first threshold. Then, the generation device 100 generates evaluation information RP11 in which the accuracy of the model M2 to be evaluated is “low”. In this way, in the example of FIG. 1, the generation apparatus 100 performs evaluation using other information on a model that targets an index “selling quickly” for which data that can be used for generation of the model is not sufficient. The accuracy of the model can be evaluated even when there is not enough data to generate Further, when the accuracy of the model is low, the generation apparatus 100 can be used even when the data is not sufficient by updating the model based on the evaluation information so as to improve the accuracy of the model. A model can be generated and the accuracy of the model can be gradually improved. For example, the generation apparatus 100 updates the model so that the score output by the model approaches the value (score) of the prediction information, thereby improving the accuracy of the model even when the data is insufficient. Is possible.

  As described above, the generation apparatus 100 inputs prediction to a model that outputs a score corresponding to an index related to the user's behavior related to content, and prediction information predicted from information output from the model and information different from the model Based on the above, evaluation information about the model can be generated. Therefore, the generating apparatus 100 can make it possible to evaluate the accuracy of the model by generating the evaluation information for evaluating the model as described above. Therefore, the generation device 100 can appropriately generate a model corresponding to an index related to user behavior. In addition, the generation apparatus 100 evaluates the model using the history information distributed in the past, and does not perform processing for acquiring information for evaluating the model by random distribution or the like. It is possible to evaluate models that do not have enough data to perform Specifically, the generation apparatus 100 evaluates a model using information indicating a statistical tendency, and without performing a process (live test) that actually collects a history using the model, The model can be evaluated based on the already acquired information (offline data). That is, the generation apparatus 100 can evaluate a model based on already acquired information (offline data) even if the logic (model) cannot be evaluated by a log. In the example of FIG. 1, the generation apparatus 100 can evaluate a model M2 that is difficult to evaluate with a log based on CTR and that sells quickly, even in a state where there are few logs corresponding to selling quickly.

[2. (Delivery process)
An example of distribution processing according to the embodiment will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of distribution processing according to the embodiment. FIG. 2 illustrates a case where the generation apparatus 100 distributes content based on a predicted score when a distribution request from a user is acquired. In the example of FIG. 2, the generation apparatus 100 updates the model M2 using information regarding the content distributed using the model M2.

  First, the generation apparatus 100 acquires a content distribution request (step S21). In the example of FIG. 2, the generation device 100 acquires a content distribution request from the terminal device 10 used by the user U101.

  The generation apparatus 100 that has acquired the content distribution request predicts the score by inputting the content information into the model. For example, the generating apparatus 100 inputs content data such as each of the products GD1 to GD5 and GD101 to GD105 stored in the product information storage unit 127 and information about the user U101 serving as a distribution destination into the model M2.

  In the example of FIG. 2, the generation apparatus 100 calculates a score indicating the possibility that the product GD1 (V product) can be quickly sold to the user U101 serving as the distribution destination by the processing as illustrated in the processing group PS22-1. The generating apparatus 100 inputs the content information DT1 to the model M2 (Step S22-1). Specifically, the generation device 100 inputs data DT1 including information regarding the content data CDT1 of the V product and the user U101 to the model M2. The model M2 to which the data DT1 is input outputs a score (step S23-1). In the example of FIG. 2, the model M2 to which the data DT1 is input outputs a score “0.56” as indicated by the score SC11-1.

  Further, the generation apparatus 100 calculates a score indicating the possibility that the product GD2 (W product) can be quickly sold to the user U101 serving as the distribution destination by the processing as shown in the processing group PS22-2. The generating apparatus 100 inputs the content information DT2 to the model M2 (Step S22-2). Specifically, the generation apparatus 100 inputs data DT2 including content data CDT2 of the W product, information about the user U101, and the like to the model M2. The model M2 to which the data DT2 is input outputs a score (step S23-2). In the example of FIG. 2, the model M2 to which the data DT2 is input outputs a score “0.58” as indicated by the score SC11-2.

  In the example of FIG. 2, the generation apparatus 100 predicts scores corresponding to the products GD1 to GD5, GD101 to GD105, and the like by processing as shown in the processing groups PS22-1 and PS22-2 (step S24). In the example of FIG. 2, the generation device 100 predicts scores corresponding to the products GD1 to GD5, GD101 to GD105, and the like as shown in the score list SL11. Moreover, the production | generation apparatus 100 estimates ranking of each goods GD1-GD5, GD101-GD105 grade | etc., Based on the score corresponding to each goods GD1-GD5, GD101-GD105 grade | etc.,. In the example of FIG. 2, the generation device 100 predicts the rankings of the products GD1 to GD5, GD101 to GD105, etc. in order from the highest score.

  Thereafter, the generation device 100 performs content distribution based on the prediction result (step S25). For example, the generation device 100 distributes content in order from the highest score. In the example of FIG. 2, the generation apparatus 100 includes a terminal device 10 in which a user U101 uses content in the order of a product GD5 (Z product), a product GD102 (B product), a product GD2 (W product), and a product GD1 (V product). Deliver to. For example, the generating apparatus 100 distributes the content displayed in a list in the order of the product GD5 (Z product), the product GD102 (B product), the product GD2 (W product), and the product GD1 (V product) to the terminal device 10.

  Thereafter, the generation device 100 acquires purchase information of a product corresponding to the content distributed from the user U101 (step S26). In the example of FIG. 2, the generation device 100 acquires behavior information indicating that the user U101 has purchased a Z product from the terminal device 10. Note that the generation device 100 may acquire action information indicating that the user U101 has purchased a Z product from an external device other than the terminal device 10. In the example of FIG. 2, it is assumed that the time when the user U101 purchases the Z product is the time when 3 hours have elapsed since the start of the distribution of the content of the Z product.

  And the production | generation apparatus 100 adds learning data based on the information acquired in step S26 (step S27). For example, the generation device 100 adds, as learning data, a combination of information related to the distributed content and correct information on the user's behavior regarding the distributed content. Specifically, the generation device 100 adds data DT201 and the like corresponding to the product GD5 that has distributed the content to the second learning data storage unit 123. For the sake of explanation, the second learning data storage unit 123 and the first learning data storage unit 121 are illustrated separately, but the second learning data storage unit 123 and the first learning data storage unit 121 are the same storage unit. It may be.

  “Data ID” shown in the second learning data storage unit 123 in FIG. 2 indicates identification information for identifying data. The “target product” shown in the second learning data storage unit 123 in FIG. 2 indicates the target product in the data identified by the data ID.

  “Sell early” in the “correct answer information” shown in the second learning data storage unit 123 in FIG. 2 indicates whether or not the user who is the distribution destination of the content of the corresponding target product has purchased the target product. . For example, when “sell early” is “1”, it indicates that the user has purchased the content within a predetermined period after the distribution of the content of the corresponding target product is started. For example, when “selling early” is “1”, it indicates that the user has purchased the content within 24 hours after the distribution of the content of the corresponding target product is started. In addition, the time indicated when “sell early” is “1” indicates the time elapsed from the start of the distribution of the content of the corresponding target product to the purchase by the user. Further, for example, when “selling early” is “0”, it indicates that the user has not purchased the content within a predetermined period after the distribution of the content of the corresponding target product is started.

  “Content information” shown in the second learning data storage unit 123 in FIG. 2 indicates information regarding the content corresponding to the data identified by the data ID. “Content information” indicates information related to distribution of content corresponding to data identified by the data ID.

  The “content information” shown in the second learning data storage unit 123 in FIG. 2 is the same as the “content data” in the first learning data storage unit 121 in FIG.

  For example, in the example illustrated in FIG. 2, the data (data DT201) identified by the data ID “DT201” indicates that the target product is the product GD5 (Z product). Further, the data DT201 indicates that “sell early” is “1 (3 hours)” in the correct answer information. Data DT201 indicates that “content data” is “CDT5” and “log data” is “LDT5 (U101, etc.)” in the content information.

  That is, the data DT201 indicates that the data is related to the distribution of the content of the product GD5 (Z product) to the user U101. Data DT201 indicates that the user U101 has purchased the distributed product GD5. The data DT201 indicates that the purchase of the product GD5 by the user U101 is three hours after the start of distribution of the content of the product GD5.

  And the production | generation apparatus 100 updates a model based on learning data (step S28). For example, the generating apparatus 100 updates the model M2 using the data DT201 and the like in the second learning data storage unit 123 as learning data (teacher data). For example, for the data DT201 whose index “sell early” is “1” as the correct answer information, the generation device 100 outputs the score output by the model M2 when the content information corresponding to the data DT201 is input to the model M2. The model M2 is updated so as to approach “1”.

  As described above, the generation apparatus 100 predicts (generates) a score indicating a degree of whether or not a product corresponding to the content is quickly purchased by the user by using the model. For example, in the example of FIG. 2, the generation apparatus 100 inputs the data DT1 including the content data CDT1 of the V product and the information about the user U101 to the model M2, so that the V product is quickly purchased by the user U101 in the model M2. Output a score indicating the degree of In the example of FIG. 2, the generation apparatus 100 causes the model M2 to output a score indicating the degree of early purchase of each product to the user U101, similarly to the other products. In the example of FIG. 2, the generation device 100 distributes the content to the terminal device 10 of the user U101 so that the content of the product with a high score is preferentially viewed by the user. Thereby, the generation apparatus 100 can preferentially provide the user with the content for which the product is purchased.

  Further, the generation apparatus 100 updates the model M2 using information related to the content distributed using the model M2. Thereby, the generation device 100 can further improve the prediction accuracy of the model M2. As described above, in the example of FIG. 2, the generation apparatus 100 uses the added data even after generation for a model that targets an index “selling quickly” for which data available for generation of the model is not sufficient, By updating the model, the accuracy of the model can be improved. That is, the generation apparatus 100 evaluates the model using information other than the information used for model generation, and updates the generated model according to the addition of data, so that even if the data is not sufficient, It is possible to generate an available model and gradually improve the accuracy of the model.

(2-1. Target of weight prediction based on time to sell)
In the example of FIG. 1, in order to simplify the explanation, the case where it is sold early and the same score is shown regardless of the elapsed time is shown. However, the generation apparatus 100 weights according to the time until it is sold. A score may be set. This point will be described with reference to FIG. FIG. 14 is a diagram illustrating a change in the magnification based on the required time according to the embodiment.

  As illustrated in FIG. 14, the generation device 100 may increase the weight as the elapsed time from the start of sales is shorter even if the product is purchased by the user within a predetermined period from the start of sales. For example, the generation apparatus 100 may generate a model so that a product with a higher score is obtained as the elapsed time from the start of sales is shorter even if the product is purchased by the user within a predetermined period from the start of sales.

  The second magnification α2 is a magnification in which the difference between the weight at the start of sales and the weight at the threshold is smaller than the first magnification α1. That is, the second magnification α2 is a magnification in which the rate of weight decrease with the passage of time is smaller than the first magnification α1. The third magnification α3 is a magnification in which the difference between the weight at the start of sales and the weight at the threshold is smaller than the second magnification α2. In other words, the third magnification α3 is a magnification in which the rate of weight decrease with the passage of time is smaller than that of the second magnification α2. The generation apparatus 100 may generate a model using a plurality of magnifications with different weights.

  For example, the generation apparatus 100 may generate a model using a magnification with the highest prediction accuracy among various magnifications such as the first magnification α1, the second magnification α2, and the third magnification α3. For example, the generation apparatus 100 generates a plurality of models based on each of various magnifications such as the first magnification α1, the second magnification α2, the third magnification α3, and the most accurate prediction among the generated models. Content distribution or the like may be performed using a model with a high value.

[3. Configuration of the generator
Next, the configuration of the generation apparatus 100 according to the embodiment will be described with reference to FIG. FIG. 4 is a diagram illustrating a configuration example of the generation apparatus according to the embodiment. As illustrated in FIG. 4, the generation apparatus 100 includes a communication unit 110, a storage unit 120, and a control unit 130. The generation device 100 includes an input unit (for example, a keyboard and a mouse) that receives various operations from an administrator of the generation device 100 and a display unit (for example, a liquid crystal display) for displaying various types of information. May be.

(Communication unit 110)
The communication unit 110 is realized by, for example, a NIC (Network Interface Card). The communication unit 110 is connected to the network N by wire or wirelessly and transmits / receives information to / from the terminal device 10.

(Storage unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory device such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 4, the storage unit 120 according to the embodiment includes a first learning data storage unit 121, a prediction information storage unit 122, a second learning data storage unit 123, a model information storage unit 124, and a user. An information storage unit 125, a log information storage unit 126, and a product information storage unit 127 are included.

(First learning data storage unit 121)
The 1st learning data storage part 121 concerning an embodiment memorizes various information about learning data. FIG. 5 is a diagram illustrating an example of the first learning data storage unit according to the embodiment. For example, the first learning data storage unit 121 stores teacher data used for generating a model. The first learning data storage unit 121 illustrated in FIG. 5 includes items such as “data ID”, “target product”, “correct information”, “prediction information”, and “content information”. The “correct answer information” includes an item such as “click”. “Prediction information” includes items such as “selling early”. The “content information” includes items such as “content data” and “log data”.

  “Data ID” indicates identification information for identifying data. For example, the data identified by the data ID “DT101” corresponds to the data DT101 shown in the example of FIG. “Target product” indicates the target product in the data identified by the data ID.

  “Correct answer information” indicates correct information corresponding to data identified by the data ID. “Click” in “correct answer information” indicates whether or not the content of the corresponding target product has been clicked by the user of the distribution destination. For example, when “click” is “1”, it indicates that the content of the corresponding target product has been clicked by the distribution destination user. For example, when “click” is “0”, it indicates that the content of the corresponding target product has not been clicked by the distribution destination user.

  “Sell early” in the “prediction information” indicates, as a score, the possibility that the target product will be purchased early by the user who is the distribution destination of the content of the corresponding target product. For example, “sell early” indicates that the higher the score, the higher the possibility of purchase by the user. For example, the closer to “1”, the higher the possibility that the user will purchase the content within a predetermined period after the distribution of the content of the corresponding target product is started. For example, the closer to “1”, the higher the possibility that the user will purchase the content within 24 hours after the distribution of the content of the corresponding target product is started. In addition, for example, the closer to “0”, the lower the possibility that the user will purchase the content within a predetermined period after the distribution of the content of the corresponding target product is started.

  “Content information” indicates information related to the content corresponding to the data identified by the data ID. “Content information” indicates information related to distribution of content corresponding to data identified by the data ID.

  “Content data” in the “content information” indicates content data provided to the user in distribution corresponding to the data identified by the data ID. FIG. 5 shows an example in which conceptual information such as “CDT101” is stored in “content data”, but in practice, various information related to content or a file path name indicating the storage location is stored. The For example, the “content data” includes various types of information such as product names, categories, images, titles, and text information about the corresponding products. For example, “content data” may be static information that does not vary depending on a user who is a distribution destination or a distribution date and time, among information regarding the content. For example, the “content data” may include a price at which a contract in a free market format is established and a price (prompt decision price) at which a contract is established even within an auction period in an auction format.

  “Log data” in “content information” indicates history information in content distribution of the target product corresponding to the data identified by the data ID. The “log data” includes various users at the time of distribution and the user who is the distribution destination of the content distribution of the target product corresponding to the data identified by the data ID. FIG. 5 shows an example in which conceptual information such as “LDT101” is stored in “log data”, but in practice, various information related to content distribution or a file path name indicating the storage location is stored. Is done. For example, the “log data” may include information regarding a user who is a distribution destination of content distribution of the target product corresponding to the data identified by the data ID.

  Further, for example, the “log data” may include dynamic information that dynamically varies depending on a user's action (bid) regarding the target product, and dynamic information at the time of content distribution. For example, the “log data” may include various information such as the current price, the number of bids, the number of bidders, and the remaining time at the time of content distribution. Note that the above is an example, and the log data is not limited to the above and may include various information. Note that “content data” and “log data” may include overlapping information as long as they are information about content.

  For example, in the example illustrated in FIG. 5, the data (data DT101) identified by the data ID “DT101” indicates that the target product is the product GD101 (A product). Further, the data DT101 indicates that “click” of the correct answer information is “1”. Further, the data DT101 indicates that “selling early” in the prediction information is “0.5”. Data DT101 indicates that “content data” is “CDT101” and “log data” is “LDT101 (U1 etc.)” in the content information.

  That is, the data DT101 indicates that it is data relating to the distribution of the content of the product GD101 (A product) to the user U1. Data DT101 indicates that the user U1 has clicked on the content of the distributed product GD101. Further, the data DT101 indicates that the possibility that the user U1 will purchase the delivered product GD101 early is moderate (for example, 50%).

  In addition, the 1st learning data memory | storage part 121 may memorize | store not only the above but various information according to the objective. For example, the first learning data storage unit 121 may store information related to the date and time when learning data was added. For example, the 1st learning data storage part 121 may memorize the information which shows what kind of judgment processing each learning data was added. For example, the first learning data storage unit 121 may store information indicating what information is used for generating the prediction information.

(Prediction information storage unit 122)
The prediction information storage unit 122 according to the embodiment stores various types of information used for generating prediction information. FIG. 6 is a diagram illustrating an example of the prediction information storage unit according to the embodiment. For example, the prediction information storage unit 122 stores information indicating a statistical tendency for generating prediction information. The prediction information storage unit 122 illustrated in FIG. 6 includes items such as “target ID”, “target”, “category”, and “keyword”.

  “Target ID” indicates identification information for identifying a target. For example, “target ID” indicates identification information for identifying a target for which prediction information is generated. “Target” indicates a target identified by a corresponding target ID. The “element” includes items such as “category” and “keyword”. The “content” includes items such as “trend” and “score”. For example, “trend” indicates a statistical tendency of each element. For example, “score” indicates a statistical tendency of each element as a score.

  “Category” indicates a statistical tendency for each category related to the corresponding object. “Category” includes items such as “A category”, “B category”, and “C category”. In the example illustrated in FIG. 6, “category” indicates an abstract code such as “A category”, but may be information indicating a specific category name or the like. Further, for example, the category such as “A category” may be “fashion” or “food”. Further, for example, the category such as “A category” may be “shoes” or “drinking water”. For example, the “category” may include categories of various levels.

  “Keyword” indicates a statistical tendency for each keyword related to the corresponding target. The “keyword” includes items such as “keyword A” and “keyword B”. In the example shown in FIG. 6, “keyword” shows an abstract code such as “keyword A”, but it may be information such as a character string such as a specific product name. For example, a keyword such as “keyword A” is not limited to a specific product name or the like, but may be an abbreviation or popular name thereof. A keyword such as “keyword A” may be “re-arrival” or “popular product”. ”,“ Limited to the actual product ”,“ same day delivery ”, or the like.

  For example, in the example illustrated in FIG. 6, the target (target TG1) identified by the target ID “TG1” is the target “sell early”, and the product that is the target of the content distributed to the user is purchased by the user earlier. Indicates that

  Further, the target TG1 indicates that the tendency of the A category is “medium (average A time)” and the score is “0.2”. For example, in the case where the target TG1 has a product category A, the product is purchased in an average A time, has a moderate tendency to sell quickly, and the score indicating the evaluation is “0.2”. It shows that.

  Further, the target TG1 indicates that the tendency of the B category is “high (average B time)” and the score is “0.5”. For example, when the category of the product is the B category, the target TG1 indicates that the product is purchased in an average B time, tends to sell quickly, and the score indicating the evaluation is “0.5”. .

  Further, the target TG1 indicates that the tendency of the keyword A is “low (average D time)” and the score is “0.05”. For example, if the target TG1 includes the keyword A in the content information of the product, the product is purchased with an average D time, the tendency to sell quickly is low, and the score indicating the evaluation is “0.05”. It shows that.

  Further, the target TG1 indicates that the tendency of the keyword B is “high (average E time)” and the score is “0.6”. For example, if the target content TG1 includes the keyword B in the content information of the product, the product is purchased with an average E time, tends to sell quickly, and the score indicating the evaluation is “0.6”. It shows that.

  The prediction information storage unit 122 is not limited to the above, and may store various model information according to the purpose.

(Second learning data storage unit 123)
The 2nd learning data storage part 123 concerning an embodiment memorizes various information about learning data. FIG. 7 is a diagram illustrating an example of the second learning data storage unit according to the embodiment. For example, the second learning data storage unit 123 stores teacher data used for model generation. The second learning data storage unit 123 illustrated in FIG. 7 includes items such as “data ID”, “target product”, “correct information”, and “content information”. The “correct answer information” includes items such as “selling early”. The “content information” includes items such as “content data” and “log data”.

  “Data ID” indicates identification information for identifying data. For example, the data identified by the data ID “DT201” corresponds to the data DT201 shown in the example of FIG. “Target product” indicates the target product in the data identified by the data ID.

  “Correct answer information” indicates correct information corresponding to data identified by the data ID. “Sell early” in the “correct answer information” indicates whether or not the user who is the distribution destination of the content of the corresponding target product has purchased the target product. For example, when “sell early” is “1”, it indicates that the user has purchased the content within a predetermined period after the distribution of the content of the corresponding target product is started. For example, when “selling early” is “1”, it indicates that the user has purchased the content within 24 hours after the distribution of the content of the corresponding target product is started. In addition, the time indicated when “sell early” is “1” indicates the time elapsed from the start of the distribution of the content of the corresponding target product to the purchase by the user. Further, for example, when “selling early” is “0”, it indicates that the user has not purchased the content within a predetermined period after the distribution of the content of the corresponding target product is started.

  “Content information” indicates information related to the content corresponding to the data identified by the data ID. “Content information” indicates information related to distribution of content corresponding to data identified by the data ID.

  “Content data” in the “content information” indicates content data provided to the user in distribution corresponding to the data identified by the data ID. FIG. 7 shows an example in which conceptual information such as “CDT5” is stored in “content data”, but in reality, various information related to content or a file path name indicating the storage location is stored. The For example, the “content data” includes various types of information such as product names, categories, images, titles, and text information about the corresponding products. For example, “content data” may be static information that does not vary depending on a user who is a distribution destination or a distribution date and time, among information regarding the content. For example, the “content data” may include a price at which a contract in a free market format is established and a price (prompt decision price) at which a contract is established even within an auction period in an auction format.

  “Log data” in “content information” indicates history information in content distribution of the target product corresponding to the data identified by the data ID. The “log data” includes various users at the time of distribution and the user who is the distribution destination of the content distribution of the target product corresponding to the data identified by the data ID. Although FIG. 7 shows an example in which conceptual information such as “LDT5” is stored in “log data”, various information related to content distribution or a file path name indicating the storage location is actually stored. Is done. For example, the “log data” may include information regarding a user who is a distribution destination of content distribution of the target product corresponding to the data identified by the data ID.

  Further, for example, the “log data” may include dynamic information that dynamically varies depending on a user's action (bid) regarding the target product, and dynamic information at the time of content distribution. For example, the “log data” may include various information such as the current price, the number of bids, the number of bidders, and the remaining time at the time of content distribution. Note that the above is an example, and the log data is not limited to the above and may include various information. Note that “content data” and “log data” may include overlapping information as long as they are information about content.

  For example, in the example illustrated in FIG. 7, the data (data DT201) identified by the data ID “DT201” indicates that the target product is the product GD5 (Z product). Further, the data DT201 indicates that “sell early” in the correct answer information is “1 (3 hours)”. Data DT201 indicates that “content data” is “CDT5” and “log data” is “LDT5 (U101, etc.)” in the content information.

  That is, the data DT201 indicates that the data is related to the distribution of the content of the product GD5 (Z product) to the user U101. Data DT201 indicates that the user U101 has purchased the distributed product GD5. The data DT201 indicates that the purchase of the product GD5 by the user U101 is 3 hours after the start of distribution of the content of the product GD5.

  In addition, the 2nd learning data memory | storage part 123 may memorize | store not only the above but various information according to the objective. For example, the second learning data storage unit 123 may store information related to the date and time when learning data was added. For example, the 2nd learning data storage part 123 may memorize | store the information which shows what kind of determination process each learning data was added. For example, the second learning data storage unit 123 may store information indicating whether each learning data is determined by an administrator's selection.

(Model information storage unit 124)
The model information storage unit 124 according to the embodiment stores information about the model. For example, the model information storage unit 124 stores model information (model data) generated by the generation process. FIG. 8 is a diagram illustrating an example of a model information storage unit according to the embodiment. The model information storage unit 124 illustrated in FIG. 8 includes items such as “model ID”, “prediction target”, and “model data”. In FIG. 8, only the models M1 and M2 are illustrated, but a large number of model information such as M3 and M4 may be stored according to each prediction target (prediction target).

  “Model ID” indicates identification information for identifying a model. For example, the model identified by the model ID “M2” corresponds to the model M2 illustrated in the example of FIG. “Prediction target” indicates the prediction target of the corresponding model. “Model data” indicates data of a corresponding model associated with the model data. For example, “model data” includes information including nodes in each layer, functions adopted by the nodes, connection relationships between the nodes, and connection coefficients set for connections between the nodes.

  For example, in the example shown in FIG. 8, the model (model M1) identified by the model ID “M1” has a prediction target “CTR”, and predicts whether the content delivered to the user is clicked by the user. Indicates that it will be used. The model data of the model M1 is model data MDT1.

  For example, in the example shown in FIG. 8, the model (model M2) identified by the model ID “M2” has a prediction target “selling early”, and a product that is a target of content delivered to the user is quick to the user. Indicates that it will be used to predict whether it will be purchased. Further, the model data of the model M2 indicates that it is model data MDT2.

  For example, the model M2 (model data MDT2) is a first layer belonging to any layer other than the output layer, which is an input layer to which content information related to content is input, an output layer, and an input layer to an output layer. A second element whose value is calculated based on the first element and the weight of the first element, and each element belonging to each layer other than the output layer for content information input to the input layer As a first element, by performing an operation based on the first element and the weight of the first element, the computer is caused to function so as to output a score value corresponding to an index related to the user's behavior regarding the content from the output layer. It is a model.

Here, it is assumed that the models M1, M2, etc. are realized by a regression model represented by “y = a ₁ * x ₁ + a ₂ * x ₂ +... + A _i * x _i ”. In this case, for example, the first element included in the model M2 corresponds to input data (xi) such as x1 or x2. The weight of the first element corresponds to the coefficient ai corresponding to xi. Here, the regression model can be regarded as a simple perceptron having an input layer and an output layer. When each model is regarded as a simple perceptron, the first element can correspond to any node of the input layer, and the second element can be regarded as a node of the output layer.

  Further, it is assumed that the models M1, M2, etc. are realized by a neural network having one or a plurality of intermediate layers, such as DNN. In this case, for example, the first element included in the model M2 corresponds to any node of the input layer or the intermediate layer. The second element corresponds to the next node, which is a node to which a value is transmitted from the node corresponding to the first element. The weight of the first element corresponds to a connection coefficient that is a weight considered for a value transmitted from a node corresponding to the first element to a node corresponding to the second element.

  The model information storage unit 124 is not limited to the above, and may store various model information according to the purpose.

(User information storage unit 125)
The user information storage unit 125 according to the embodiment stores various types of information regarding the user. For example, the user information storage unit 125 stores information on a plurality of workers (users) who perform tasks such as crowdsourcing. FIG. 9 is a diagram illustrating an example of a user information storage unit according to the embodiment. The user information storage unit 125 illustrated in FIG. 9 includes items such as “user ID”, “age”, “sex”, “home”, “work location”, and “interest”.

  “User ID” indicates identification information for identifying a user. For example, the user identified by the user ID “U1” corresponds to the user U1 illustrated in the example of FIG. “Age” indicates the age of the user identified by the user ID. The “age” may be a specific age of the user identified by the user ID, such as 35 years old. “Gender” indicates the gender of the user identified by the user ID.

  “Home” indicates location information of the user's home identified by the user ID. In the example illustrated in FIG. 9, “home” indicates an abstract code such as “LC11”, but may be information indicating latitude and longitude. For example, “home” may be a region name or an address.

  “Work location” indicates position information of the user's work location identified by the user ID. In the example illustrated in FIG. 9, “work location” illustrates an abstract code such as “LC12”, but may be information indicating latitude and longitude. Further, for example, the “work location” may be an area name or an address.

  “Interest” indicates the interest of the user identified by the user ID. That is, “interest” indicates an object that is highly interested by the user identified by the user ID. In the example shown in FIG. 9, one “interest” is illustrated for each user, but may be plural.

  For example, in the example illustrated in FIG. 9, the age of the user identified by the user ID “U1” is “20s”, and the gender is “male”. For example, the user identified by the user ID “U1” indicates that the home is “LC11”. For example, the user identified by the user ID “U1” indicates that the work location is “LC12”. For example, the user identified by the user ID “U1” indicates that he / she is interested in “sports”.

  Note that the user information storage unit 125 is not limited to the above, and may store various types of information according to the purpose. For example, the user information storage unit 125 may store information on demographic attributes of users and information on psychographic attributes. For example, the user information storage unit 125 may store information such as name, family structure, income, interest, and lifestyle.

(Log information storage unit 126)
The log information storage unit 126 according to the embodiment stores information related to various histories such as user actions. FIG. 10 is a diagram illustrating an example of the log information storage unit according to the embodiment. For example, the log information storage unit 126 stores various behavior information including user behavior information for content distributed to the terminal device 10 of each user. The log information storage unit 126 illustrated in FIG. 10 includes items such as “user ID”, “history ID”, “type”, “content”, and “date and time”.

  “User ID” indicates identification information for identifying a user. For example, the user identified by the user ID “U1” corresponds to the user U1 illustrated in the example of FIG. The “history ID” indicates information for identifying the user's action. The “type” indicates information related to the type of action of the corresponding user. Further, “content” indicates content that is targeted in the corresponding user's action. “Date and time” indicates the date and time when the corresponding user action was performed. “Date and time” is abstractly illustrated as “dt11” or the like, but a specific date and time such as “August 6, 2017 19:31:52” may be stored.

  For example, in the example illustrated in FIG. 10, the user (user U1) identified by the user ID “U1” is the history identified by the history ID “LG11” (history LG11) or the history identified by the history ID “LG12”. This indicates that an action corresponding to (History LG12) or the like has been performed. Specifically, in the example illustrated in FIG. 10, the user U1 indicates that a content distribution request (corresponding to the history LG11) has been made at the date dt11. For example, the history LG11 indicates that content such as A product is distributed to the terminal device 10 used by the user U1 at the date dt11. In FIG. 10, the history LG11 indicates that content such as A products has been distributed using the model M1 that predicts CTR of each content. That is, the history LG11 indicates that the content has been distributed using the ease of clicking each content as a criterion for evaluation. In the example illustrated in FIG. 10, the user U1 indicates that the user has clicked on the content of the A product at the date dt12 after the date dt11. In addition, in the example illustrated in FIG. 10, the user U1 indicates that the A product is purchased at the date dt13 after the date dt12.

  For example, in the example shown in FIG. 10, the user (user U2) identified by the user ID “U2” is the history identified by the history ID “LG21” (history LG21) and the history identified by the history ID “LG22”. It shows that the action corresponding to (history LG22) etc. was performed. Specifically, in the example illustrated in FIG. 10, the user U2 indicates that a content distribution request (corresponding to the history LG21) has been made at the date dt21. For example, the history LG21 indicates that content such as B product is distributed to the terminal device 10 used by the user U2 at the date dt21. In FIG. 10, the history LG21 indicates that the content such as the B product has been distributed using the model M1 that predicts the CTR of each content. That is, the history LG21 indicates that the content has been distributed using the ease of clicking each content as a criterion for evaluation. In addition, in the example illustrated in FIG. 10, the user U2 indicates that the content of the B product is clicked on the date dt22 after the date dt21. Further, in the example illustrated in FIG. 10, the user U <b> 2 indicates that B product has not been purchased.

  Note that the log information storage unit 126 is not limited to the above, and may store various information according to the purpose. Moreover, although the case where action information was memorize | stored in the log information storage part 126 for every user ID was shown in FIG. 10, action information may be memorize | stored not only for every user ID but in order of time, for example.

(Product information storage unit 127)
The product information storage unit 127 according to the embodiment stores information related to products. For example, the product information storage unit 127 stores detailed information on products that have been exhibited in the product. FIG. 11 shows an example of information related to products stored in the product information storage unit. As illustrated in FIG. 11, the product information storage unit 127 includes items such as “product ID”, “product name”, “sales mode”, and “content data” as information about the product.

  “Product ID” indicates identification information for identifying a product. “Product name” indicates the name of the product identified by the product ID. The “sales mode” indicates the sales mode of the corresponding product. For example, “sales mode” indicates an item display format such as an auction format or a flea market format. For example, when the “sales mode” is “furima”, the sales form of the product is a flea market form. For example, when the “sales mode” is “auction”, it indicates that the sales form of the product is an auction form. For example, when the “sales mode” is “auction (prompt decision)”, the sale form of the product is an auction form, and the price (prompt decision price) for establishing a contract is set even within the auction period. It shows that there is.

  “Content data” indicates content data corresponding to the product identified by the product ID. FIG. 11 shows an example in which conceptual information such as “CDT1” is stored in “content data”, but in practice, various information related to content or a file path name indicating the storage location is stored. The For example, the “content data” includes various types of information such as character information such as categories, images, titles, and explanations about the corresponding product. For example, the “content data” in FIG. 11 may be static information that does not vary depending on the user who is the distribution destination and the distribution date and time among the information regarding the content. For example, the “content data” may include a price at which a contract in a free market format is established and a price (prompt decision price) at which a contract is established even within an auction period in an auction format. In the example of FIG. 11, the item “product name” is shown in order to illustrate the product name, but the information of “product name” may be included in the “content data”.

  For example, in the example illustrated in FIG. 11, the product (product GD1) identified by the product ID “GD1” indicates that the product name is “V product”. Further, the product GD1 indicates that the sales mode is “free” and the content data is “CDT1”.

  The product information storage unit 127 is not limited to the above, and may store various information according to the purpose.

(Control unit 130)
Returning to the description of FIG. 4, the control unit 130 is a controller and is stored in a storage device inside the generation apparatus 100 by, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like. Various programs (corresponding to an example of a generation program) are implemented by using the RAM as a work area. The control unit 130 is a controller, and is realized by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). The control unit 130 is one of an input layer, an output layer, and an input layer to an output layer to which content information related to content is input by information processing according to the model M2 or the like stored in the model information storage unit 124. Input to the input layer, including a first element belonging to a layer other than the output layer and a second element whose value is calculated based on the first element and the weight of the first element A value of a score corresponding to an index related to a user's behavior related to content by performing calculation based on the first element and the weight of the first element with each element belonging to each layer other than the output layer as a first element for information Is output from the output layer.

  As illustrated in FIG. 4, the control unit 130 includes an acquisition unit 131, a generation unit 132, a prediction unit 133, and a distribution unit 134, and realizes or executes information processing functions and operations described below. . Note that the internal configuration of the control unit 130 is not limited to the configuration illustrated in FIG. 4, and may be another configuration as long as information processing described later is performed. The control unit 130, by information processing according to the model M2 (model data MDT2) stored in the storage unit 120, a second element whose value is calculated based on the first element and the weight of the first element, Content information input to the input layer, and by performing an operation based on the first element and the weight of the first element with each element belonging to each layer other than the output layer as the first element, The score value corresponding to the index related to the behavior of the is output from the output layer.

(Acquisition part 131)
The acquisition unit 131 acquires various types of information. For example, the acquisition unit 131 includes a first learning data storage unit 121, a prediction information storage unit 122, a second learning data storage unit 123, a model information storage unit 124, a user information storage unit 125, and a log information storage. Various information is acquired from the unit 126, the product information storage unit 127, and the like. The acquisition unit 131 may acquire various types of information from an external information processing apparatus. The acquisition unit 131 may acquire various types of information from the terminal device 10 or the like.

  For example, the acquisition unit 131 acquires content information regarding content. For example, the acquisition unit 131 acquires content information including log information regarding distribution of content to a terminal device used by a user. For example, the acquisition unit 131 acquires content information corresponding to a target that is exhibited in a manner in which a transaction is established in accordance with an approval for the amount set for the content-related target. For example, the acquisition unit 131 acquires content information corresponding to a target that is exhibited in a manner in which a transaction is established in accordance with a bid for the target within a period set as a target related to content.

  In the example of FIG. 1, the acquisition unit 131 collects information for generating correct answer information. For example, the acquisition unit 131 distributes content such as the product GD101 (A product) to the product GD105 (E product) to the user, and acquires information (behavior information) related to the user's behavior with respect to the distributed content.

  In the example of FIG. 1, the acquisition unit 131 acquires prediction information corresponding to the target index “selling early”. For example, the acquisition unit 131 acquires prediction information corresponding to the index “selling early” from the prediction information storage unit 122.

  For example, the acquisition unit 131 acquires a content distribution request. In the example of FIG. 2, the acquisition unit 131 acquires a content distribution request from the terminal device 10 used by the user U101.

(Generator 132)
The generation unit 132 generates various types of information. For example, the generation unit 132 generates a model as shown in the model information storage unit 124 using the learning data stored in the first learning data storage unit 121. For example, the generation unit 132 uses the content information acquired by the acquisition unit 131 to generate a model that outputs a score indicating an evaluation based on an index corresponding to the user's behavior regarding the content. Moreover, the production | generation part 132 produces | generates a model by making the user's action information regarding a content into correct information.

  In addition, the generation unit 132 inputs the content information acquired by the acquisition unit 131 into a model that outputs a score corresponding to an index related to the user's behavior regarding the content, and the output value output from the model and the model are: Evaluation information about the model is generated based on prediction information predicted from different information.

  For example, the generation unit 132 generates model evaluation information based on prediction information predicted from elements related to the index. For example, the generation unit 132 inputs the content information to a model that outputs a score indicating an evaluation based on an index related to whether or not the object is purchased by the user, and then evaluates the model based on the output value output from the model. Generate information. For example, the generation unit 132 relates to the model based on the output value output from the model by inputting the score indicating the evaluation based on the index related to whether or not the target is purchased within a predetermined period from the start of the content distribution. Generate evaluation information.

  For example, the generation unit 132 generates model evaluation information based on prediction information predicted from elements related to categories and keywords corresponding to content-related objects. For example, the generation unit 132 generates model evaluation information based on prediction information predicted from statistical information related to content. For example, the generation unit 132 generates a model based on content information and correct information based on prediction information. For example, the generation unit 132 generates a model based on content information in which a predetermined action corresponding to the index is performed by a predetermined user and correct information indicating that the predetermined user has performed the predetermined action. .

  For example, the generation unit 132 generates a model so that a higher score is output as the time from the start of content distribution until the target is purchased is shorter. For example, the generation unit 132 generates a model by increasing the weight of behavior information having a shorter time from the start of content distribution until the target is purchased.

  For example, the generation unit 132 generates models M1, M2, etc., and stores the generated models M1, M2, etc. in the model information storage unit 124. The generation unit 132 may generate the models M1, M2, and the like using any learning algorithm. For example, the generation unit 132 generates models M1, M2, and the like using a learning algorithm such as a neural network, a support vector machine (SVM), clustering, and reinforcement learning. As an example, when the generation unit 132 generates models M1, M2, etc. using a neural network, the models M1, M2, etc. include an input layer including one or more neurons, an intermediate layer including one or more neurons, and one And an output layer including the above neurons.

  The generation unit 132 generates a model and stores the generated model in the model information storage unit 124. Specifically, the generation unit 132 is a first element belonging to any layer other than the output layer, which is any layer from the input layer, the output layer, and the input layer to the output layer to which content information regarding the content is input. And a second element whose value is calculated based on the first element and the weight of the first element, and for each piece of element belonging to each layer other than the output layer to the content information input to the input layer, As one element, a calculation is performed based on the first element and the weight of the first element, thereby generating a model that outputs from the output layer a score value corresponding to an index related to the user's behavior regarding the content.

  In the example of FIG. 1, the generation unit 132 generates a model based on the learning data. For example, the generation unit 132 generates a model using the user's behavior regarding the index “CTR” as correct answer information.

  In the example of FIG. 1, the generation unit 132 generates a model by performing learning using the data DT101 to DT105 in the first learning data storage unit 121 as learning data (teacher data). For example, the generation unit 132 generates the model M1 by performing learning using the data DT101 to DT105 in the first learning data storage unit 121 as learning data (teacher data).

  Further, for example, the generation unit 132 learns the data DT101 to DT105 in the first learning data storage unit 121 using data that can be used to generate the model M2 among learning data (teacher data). Thus, the model M2 is generated. For example, the generation unit 132 uses the data DT101 to DT105 in the first learning data storage unit 121 and the like, among the learning data (teacher data), the data associated with the correct answer information corresponding to the index “sell early”. The model M2 is generated by performing learning.

  For example, the generation unit 132 generates a model using a feature amount related to content. For example, the generation unit 132 generates a model using a feature amount related to the content content. For example, the generation unit 132 generates a model using a feature amount related to a product corresponding to the content. For example, the generation unit 132 generates a model using the feature amount related to the product category or sales mode corresponding to the content. For example, the generation unit 132 generates a model using a feature amount related to a user who is a content distribution destination. In the example of FIG. 1, the generation unit 132 learns characteristics regarding a product or the like corresponding to the content included in the content information, and generates a model that predicts whether or not the product corresponding to the content is purchased by the user. The features included in the content information learned by the generation unit 132 may be input to the generation unit 132 by a person such as an administrator of the generation unit 132 or may be automatically learned (extracted) by the generation unit 132. Also good.

  For example, the generating unit 132 generates the model M2 using data indicating that the user U501 has purchased the product GD501. For example, for the data whose index “correctly sells” is “1” as the correct answer information, the generation unit 132 receives the model M2 when content information corresponding to the data (early sell data) is input to the model M2. The learning process is performed so that the score output by “1” approaches “1”. For example, when content data, log data, or the like of early-sale data is input to the model M2, the generation unit 132 performs a learning process so that the score output from the model M2 approaches “1”. Note that the generation unit 132 may change the weighting for the index “selling early” based on the time from the start of sales until the purchase, but details of this point will be described later.

  Further, for example, for the data whose index “correctly sell” is “0” as the correct answer information, the generation unit 132 receives content information corresponding to the data (non-early sell data) in the model M2. The learning process is performed so that the score output from the model M2 approaches “0”. For example, when content data or log data of non-early sale data is input to the model M2, the generation unit 132 performs a learning process so that the score output by the model M2 approaches “1”.

  In the example of FIG. 1, the generation unit 132 generates prediction information corresponding to the index “selling early”. For example, as illustrated in the first learning data storage unit 121, the generation unit 132 generates prediction information corresponding to the index “sell early” for the data DT101 to DT105 and the like. For example, for the data DT101 to DT105, the generation unit 132 generates prediction information corresponding to the index “sell early” based on the product category corresponding to each data, the keyword included in the content data, and the like.

  In the example of FIG. 1, the generation unit 132 generates evaluation information based on the output value of the model M2 and the prediction information. For example, the generation unit 132 causes the generation apparatus 100 to sell a score corresponding to each of the products GD101 to GD105 (A to E products) as shown in the score list SL1 and an index “sold early” in the first learning data storage unit 121. The evaluation information is generated based on the prediction information corresponding to “”. For example, the generation unit 132 has a score corresponding to each of the products GD101 to GD105 (A to E products) as shown in the score list SL1 and a prediction corresponding to the index “sell early” shown in the first learning data storage unit 121. Evaluation information is generated based on the comparison with the information (score). For example, the generation unit 132 generates evaluation information according to the difference between the score (output score) of each product output by the model M2 and the score (prediction score) that is prediction information. For example, the generation unit 132 generates evaluation information according to a comparison between an average (error average) of a difference between an output score of each product and a predicted score of each product and a predetermined threshold.

  For example, when the error average of the model to be evaluated is less than the first threshold, the generation unit 132 generates evaluation information that sets the accuracy of the model to “low”. For example, when the error average of the model to be evaluated is greater than or equal to the first threshold and less than the second threshold greater than the first threshold, the generation unit 132 generates evaluation information that sets the accuracy of the model to “medium”. . For example, when the error average of the model to be evaluated is greater than or equal to the second threshold, the generation unit 132 generates evaluation information that sets the accuracy of the model to “high”.

  In the example of FIG. 1, the generation unit 132 generates evaluation information RP11. For example, the generation unit 132 calculates the error average of the model M2 to be evaluated as the error ER1 that is less than the first threshold. Then, the generation device 100 generates evaluation information RP11 in which the accuracy of the model M2 to be evaluated is “low”.

  In the example of FIG. 2, the generation unit 132 updates the model based on the learning data. For example, the generation unit 132 updates the model M2 using the data DT201 and the like in the second learning data storage unit 123 as learning data (teacher data). For example, for the data DT201 whose index “sell early” is “1” as the correct answer information, the generation unit 132 outputs the score output by the model M2 when the content information corresponding to the data DT201 is input to the model M2. The model M2 is updated so as to approach “1”.

(Prediction unit 133)
The prediction unit 133 predicts various types of information. The prediction unit 133 includes a first learning data storage unit 121, a prediction information storage unit 122, a second learning data storage unit 123, a model information storage unit 124, a user information storage unit 125, and a log information storage unit 126. And various information is predicted using the information memorize | stored in the merchandise information memory | storage part 127 grade | etc.,. For example, the prediction unit 133 predicts various types of information based on various types of information acquired by the acquisition unit 131.

  For example, the prediction unit 133 predicts the score of the content using the model. For example, the prediction unit 133 predicts a content score by inputting content information regarding the content into the model. For example, the predicting unit 133 predicts the evaluation of the content corresponding to the index corresponding to the user behavior regarding the content based on the score output from the model M2. For example, the prediction unit 133 predicts the evaluation of the content corresponding to the index “sell early” indicating whether the product corresponding to the content sells quickly based on the score output from the model M2.

  In the example of FIG. 1, the predicting unit 133 generates a score list including scores corresponding to the products GD101 to GD105 and the like by processing as illustrated in the processing groups PS12-1 and PS12-2 and the like. As shown in the score list SL1, the prediction unit 133 predicts scores corresponding to the products GD101 to GD105 (AE products) and the like. Based on the score output by the prediction unit 133 model M2, the evaluation of the content corresponding to the index corresponding to the user's behavior regarding the content is predicted. Specifically, based on the score output by the prediction unit 133 model M2, the evaluation of the content corresponding to the index “sell early” indicating whether the product corresponding to the content sells quickly is predicted.

  In the example of FIG. 2, the prediction unit 133 predicts scores corresponding to the products GD1 to GD5, GD101 to GD105, and the like by processing as shown in the processing groups PS22-1 and PS22-2. For example, the prediction unit 133 predicts scores corresponding to the products GD1 to GD5, GD101 to GD105, and the like as shown in the score list SL11. For example, the prediction unit 133 predicts rankings of the products GD1 to GD5, GD101 to GD105, and the like based on scores corresponding to the products GD1 to GD5, GD101 to GD105, and the like. For example, the predicting unit 133 predicts the rankings of the products GD1 to GD5, GD101 to GD105, etc. in order from the highest score.

  For example, the prediction unit 133 calculates a score using a model having an arbitrary structure such as the above-described regression model or neural network. Specifically, the model M2 is a value obtained by quantifying the prediction of a predetermined target (that is, a plurality of indices) when content information related to the content (that is, each element used for calculating the score described above) is input. The coefficient is set so as to output a score indicating whether or not the action corresponding to each of the user is likely to be performed by the user. The prediction unit 133 calculates a score related to each exhibition target using such a model M2.

  In the above example, an example in which the model M2 is a model that outputs a value obtained by quantifying evaluation based on a plurality of indexes corresponding to the user's behavior related to the content when content information related to the content is input. . However, the model (model X) according to the embodiment may be a model generated based on a result obtained by repeatedly inputting / outputting data to / from the model M2. For example, the model X may be a model (model Y) that has been learned so that content information regarding the content is input and the score output by the model M2 is output. Alternatively, the model M2 may be a model that has been learned so that content information regarding the content is input and the output value of the model Y is output. When the prediction unit 133 performs a prediction process using GAN (Generative Adversarial Networks), the model M2 may be a model that constitutes a part of the GAN.

(Distribution unit 134)
The distribution unit 134 provides various types of information. For example, the distribution unit 134 provides various types of information to the terminal device 10. The distribution unit 134 provides a service based on the score predicted by the prediction unit 133. For example, the distribution unit 134 provides a content distribution service based on the score predicted by the prediction unit 133. Further, for example, the distribution unit 134 may provide information related to the model generated by the generation unit 132 to an external information processing apparatus. For example, the distribution unit 134 may provide information output from the model to an external information processing apparatus.

  The distribution unit 134 distributes content based on the score output by the model. For example, the distribution unit 134 distributes content based on a score output from the model among content groups including content related to an object traded in an auction format and content related to an object traded in a free market format.

  In the example of FIG. 1, the distribution unit 134 distributes content to the terminal devices 10-1 to 10-5 used by the users U <b> 1 to U <b> 5. In the example of FIG. 2, the distribution unit 134 performs content distribution based on the prediction result by the prediction unit 133. For example, the distribution unit 134 distributes content in order from the highest score. For example, the distribution unit 134 distributes the content to the terminal device 10 used by the user U101 in the order of the product GD5 (Z product), the product GD102 (B product), the product GD2 (W product), and the product GD1 (V product). For example, the distribution unit 134 distributes content displayed in a list in the order of the product GD5 (Z product), the product GD102 (B product), the product GD2 (W product), and the product GD1 (V product) to the terminal device 10.

[4. Generation process flow)
Next, a generation process procedure performed by the generation system 1 according to the embodiment will be described with reference to FIG. FIG. 12 is a flowchart illustrating an example of the generation process according to the embodiment.

  As illustrated in FIG. 12, the generation device 100 acquires first learning data (step S101). For example, the generation device 100 acquires learning data from the first learning data storage unit 121.

  Thereafter, the generation device 100 generates a model based on the first learning data (step S102). In the example of FIG. 1, the generation device 100 generates a model M2 using the first learning data stored in the first learning data storage unit 121.

  Thereafter, the generation device 100 generates prediction information from information other than the model (step S103). In the example of FIG. 1, the generation device 100 generates prediction information from statistical information stored in the prediction information storage unit 122.

  Thereafter, the generation apparatus 100 generates evaluation information about the model based on the output value output from the model and the prediction information (step S104). In the example of FIG. 1, the generation device 100 generates evaluation information RP11 related to the model M2.

[5. (Delivery process flow)
Next, the procedure of distribution processing by the generation system 1 according to the embodiment will be described with reference to FIG. FIG. 13 is a flowchart illustrating an example of distribution processing according to the embodiment.

  As illustrated in FIG. 13, the generation device 100 acquires a distribution request (step S201). In the example of FIG. 2, the generation device 100 acquires a distribution request from the terminal device 10 used by the user U101.

  Moreover, the production | generation apparatus 100 produces | generates the score of each goods using a model (step S202). In the example of FIG. 2, the generation apparatus 100 predicts the score of content (product information) such as the products GD1 to GD5 and GD101 to GD105 using the model M2.

  Moreover, the production | generation apparatus 100 provides the content regarding goods based on the produced | generated score (step S203). In the example of FIG. 2, the generation device 100 distributes content in order from the highest score, such as the product GD5 (Z product), the product GD102 (B product), the product GD2 (W product), the product GD1 (V product), and the like. .

[6. effect〕
As described above, the generation device 100 according to the embodiment includes the acquisition unit 131 and the generation unit 132. The acquisition unit 131 acquires content information regarding content. In addition, the generation unit 132 inputs the content information acquired by the acquisition unit 131 into a model that outputs a score corresponding to an index related to the user's behavior regarding the content, and the output value output from the model and the model are: Evaluation information about the model is generated based on prediction information predicted from different information.

  As described above, the generation apparatus 100 according to the embodiment differs from the model in the output value output from the model by inputting the content information to the model that outputs the score corresponding to the index related to the user's behavior related to the content. By generating the evaluation information regarding the model based on the prediction information predicted from the information, it is possible to appropriately generate the model corresponding to the index regarding the user's behavior.

  Further, in the generation device 100 according to the embodiment, the generation unit 132 generates model evaluation information based on prediction information predicted from an element related to an index.

  As described above, the generation apparatus 100 according to the embodiment appropriately generates a model corresponding to an index related to a user's behavior by generating model evaluation information based on prediction information predicted from elements related to the index. Can be generated.

  In the generation device 100 according to the embodiment, the acquisition unit 131 acquires content information including log information regarding distribution of content to the terminal device 10 used by the user.

  As described above, the generation apparatus 100 according to the embodiment appropriately acquires a model corresponding to an index related to a user's behavior by acquiring content information including log information related to distribution of content to the terminal device 10 used by the user. Can be generated.

  Further, in the generation device 100 according to the embodiment, the generation unit 132 outputs content information from a model by inputting the content information to a model that outputs a score indicating an evaluation based on an index related to whether or not the object is purchased by the user. Evaluation information about the model is generated based on the output value.

  As described above, the generation apparatus 100 according to the embodiment inputs content information to an output value output from the model by inputting the score indicating the evaluation based on the index related to whether or not the object is purchased by the user. Based on this, by generating the evaluation information regarding the model, it is possible to appropriately generate the model corresponding to the index regarding the user's behavior.

  In addition, in the generation device 100 according to the embodiment, the generation unit 132 is output from the model by inputting the score indicating the evaluation based on the index regarding the purchase / non-purchase of the target within a predetermined period from the start of content distribution. Evaluation information about the model is generated based on the output value.

  As described above, the generation apparatus 100 according to the embodiment outputs an output value output from a model by inputting the score indicating an evaluation based on an index related to whether or not the target is purchased within a predetermined period from the start of content distribution. By generating the evaluation information related to the model based on the above, it is possible to appropriately generate a model corresponding to the index related to the user's behavior.

  Further, in the generation device 100 according to the embodiment, the generation unit 132 generates model evaluation information based on prediction information predicted from a category corresponding to a content-related target or an element related to a keyword.

  As described above, the generating apparatus 100 according to the embodiment generates the evaluation information of the model based on the prediction information predicted from the elements related to the category and the keyword corresponding to the target related to the content, and thus becomes an index related to the user's behavior. Corresponding models can be generated appropriately.

  In the generation device 100 according to the embodiment, the generation unit 132 generates model evaluation information based on prediction information predicted from statistical information related to content.

  As described above, the generation apparatus 100 according to the embodiment generates a model corresponding to an index related to a user's behavior by generating model evaluation information based on prediction information predicted from statistical information related to content. It can be generated appropriately.

  In the generation device 100 according to the embodiment, the generation unit 132 generates a model based on the content information and the correct answer information based on the prediction information.

  As described above, the generation apparatus 100 according to the embodiment can appropriately generate a model corresponding to an index related to user behavior by generating a model based on content information and correct information based on prediction information. can do.

  Further, in the generation device 100 according to the embodiment, the generation unit 132 includes content information in which a predetermined action corresponding to the index is performed by a predetermined user, and correct information indicating that the predetermined user has performed the predetermined action. Based on the above, a model is generated.

  As described above, the generating apparatus 100 according to the embodiment is based on the content information in which the predetermined action corresponding to the index is performed by the predetermined user and the correct answer information indicating that the predetermined user has performed the predetermined action. Thus, by generating a model, it is possible to appropriately generate a model corresponding to an index related to user behavior.

  In addition, the generation device 100 according to the embodiment includes a distribution unit 134. The distribution unit 134 distributes content based on the score output by the model.

  As described above, the generation apparatus 100 according to the embodiment can provide information according to the prediction evaluation of the index related to the user's behavior related to the content by distributing the content based on the score output by the model.

[7. Hardware configuration)
The generation apparatus 100 according to the embodiment described above is realized by a computer 1000 configured as shown in FIG. 15, for example. FIG. 15 is a hardware configuration diagram illustrating an example of a computer that realizes the function of the generation apparatus. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, an HDD (Hard Disk Drive) 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F). ) 1700.

  The CPU 1100 operates based on a program stored in the ROM 1300 or the HDD 1400 and controls each unit. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started up, a program depending on the hardware of the computer 1000, and the like.

  The HDD 1400 stores programs executed by the CPU 1100, data used by the programs, and the like. The communication interface 1500 receives data from other devices via the network N and sends the data to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the network N.

  The CPU 1100 controls an output device such as a display and a printer and an input device such as a keyboard and a mouse via the input / output interface 1600. The CPU 1100 acquires data from the input device via the input / output interface 1600. In addition, the CPU 1100 outputs the generated data to the output device via the input / output interface 1600.

  The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads such a program from the recording medium 1800 onto the RAM 1200 via the media interface 1700, and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase change rewritable disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. Etc.

  For example, when the computer 1000 functions as the generation apparatus 100 according to the embodiment, the CPU 1100 of the computer 1000 executes control by executing a program or data (for example, model M2 (model data MDT2)) loaded on the RAM 1200. The function of the unit 130 is realized. The CPU 1100 of the computer 1000 reads and executes these programs or data (for example, model M2 (model data MDT2)) from the recording medium 1800. As another example, these programs or data are transmitted from other devices via the network N. May be obtained.

  As described above, some of the embodiments and modifications of the present application have been described in detail with reference to the drawings. It is possible to carry out the present invention in other forms that have been modified and improved.

[8. Others]
In addition, among the processes described in the above-described embodiments and modifications, all or a part of the processes described as being automatically performed can be manually performed, or are described as being performed manually. All or part of the processing can be automatically performed by a known method. In addition, the processing procedures, specific names, and information including various data and parameters shown in the document and drawings can be arbitrarily changed unless otherwise specified. For example, the various types of information illustrated in each drawing is not limited to the illustrated information.

  Further, each component of each illustrated apparatus is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution / integration of each device is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed or arbitrarily distributed in arbitrary units according to various loads or usage conditions. Can be integrated and configured.

  In addition, the above-described embodiments and modifications can be combined as appropriate within a range that does not contradict processing contents.

  In addition, the “section (module, unit)” described above can be read as “means” or “circuit”. For example, the acquisition unit can be read as acquisition means or an acquisition circuit.

DESCRIPTION OF SYMBOLS 1 Generation system 100 Generation apparatus 121 1st learning data storage part 122 Information storage part for prediction 123 2nd learning data storage part 124 Model information storage part 125 User information storage part 126 Log information storage part 127 Product information storage part 130 Control part 131 Acquisition unit 132 Generation unit 133 Prediction unit 134 Distribution unit 10 Terminal device N Network

Claims (12)

An acquisition unit for acquiring content information about the content;
The content information acquired by the acquisition unit is input to a model that outputs a score corresponding to an index related to the user's behavior related to the content, and an output value output from the model is predicted from information different from the model A generating unit that generates evaluation information on the model based on the prediction information to be generated;
A generating apparatus comprising: The generator is
The generation apparatus according to claim 1, wherein the evaluation information of the model is generated based on prediction information predicted from an element related to the index. The acquisition unit
The generation apparatus according to claim 1, wherein the content information including log information related to distribution of the content to a terminal device used by the user is acquired. The generator is
By inputting the content information into a model that outputs a score indicating an evaluation based on an index related to whether or not the target related to the content is purchased by the user, evaluation information about the model is obtained based on an output value output from the model. It produces | generates. The production | generation apparatus of any one of Claims 1-3 characterized by the above-mentioned. The generator is
Evaluation information about the model is input based on an output value output from the model by inputting the score indicating an evaluation based on an index related to whether or not the target is purchased within a predetermined period from the start of content distribution. The generation device according to claim 4, wherein the generation device is a generation device. The generator is
The generation apparatus according to claim 1, wherein the evaluation information of the model is generated based on prediction information predicted from an element related to a category or a keyword corresponding to an object related to the content. . The generator is
The generation apparatus according to claim 1, wherein the model evaluation information is generated based on prediction information predicted from statistical information related to the content. The generator is
The generation apparatus according to claim 1, wherein the model is generated based on the content information and correct information based on the prediction information. The generator is
The model is generated based on content information in which a predetermined action corresponding to the index is performed by a predetermined user and correct information indicating that the predetermined user has performed the predetermined action. The generation device according to any one of claims 1 to 8. A delivery unit for delivering content based on the score output by the model;
The generation device according to claim 1, further comprising: A generation method executed by a computer,
An acquisition process for acquiring content information about the content;
The content information acquired in the acquisition step is input to a model that outputs a score corresponding to an index related to the user's behavior related to the content, and an output value output from the model is predicted from information different from the model A generating step for generating evaluation information on the model based on the prediction information to be generated;
A generation method comprising: An acquisition procedure for acquiring content information about the content;
The content information acquired by the acquisition procedure is input to a model that outputs a score corresponding to an index related to the user's behavior related to the content, and an output value output from the model is predicted from information different from the model A generation procedure for generating evaluation information on the model based on the predicted information to be generated;
A program for causing a computer to execute.

Images