JP2023043103A - Information processing device, information processing method, and information processing program - Google Patents

Abstract

To allow for checking reliability of generated information.SOLUTION: An information processing device disclosed herein comprises a selection unit, a generation unit, and an evaluation unit. The selection unit selects a standard model for estimating a second event having association with a first event estimated by a determination target model, the standard model having estimation accuracy satisfying given criteria. The generation unit generates first list information enumerating ranked users on the basis of values indicative of association between each user and the first event as obtained from the determination target model and second list information enumerating ranked users on the basis of, values indicative of association between each user and the second event as obtained from the standard model. The evaluation unit compares the first list information with the second list information to evaluate reliability of the determination target model on the basis of, whether content of the first and second list information corresponds to association between the first event and the second event.SELECTED DRAWING: Figure 8

Description

The present invention relates to an information processing device, an information processing method, and an information processing program.

Conventionally, a technique for predicting user behavior on the Internet is known. For example, in Patent Document 1, for the purpose of predicting user behavior with higher accuracy than in the past, the tendency of search queries of users on the Internet is learned, and the learned tendency of search queries and subsequent user behavior are disclosed. A technique has been proposed in which a relevance score is calculated to generate a model, and the model is used to predict behavior of other users.

JP 2016-177377 A

However, with the techniques described above, it is difficult to obtain confirmation as to whether or not the generated information can be trusted. For example, in hindsight, there is a possibility that a model that produces effective results is included, and even if an AB test is conducted, an inappropriate model may be adopted depending on the content of the test.

The present application has been made in view of the above, and an object thereof is to provide an information processing apparatus, an information processing method, and an information processing program capable of confirming the reliability of generated information.

An information processing apparatus according to the present application includes a selection unit, a generation unit, and an evaluation unit. The selection unit selects a reference model, which is a model for estimating a second event that is related to the first event estimated by the determination target model, and whose estimation accuracy satisfies a predetermined standard. The generation unit generates first list information in which each user is ranked and listed based on a value indicating the relationship between each user and the first event obtained from the determination target model, and each user obtained from the reference model. Second list information is generated in which each user is ranked and listed based on the value indicating the relationship with the second event. The evaluation unit compares the first list information and the second list information, and determines whether the contents of the first list information and the second list information correspond to the relationship between the first event and the second event. Evaluate the reliability of the model to be judged based on whether.

According to one aspect of the embodiments, the reliability of the generated information can be verified.

FIG. 1 is a diagram illustrating an example of an information processing system according to an embodiment. FIG. 2 is a diagram illustrating an overview (Part 1) of a reliability evaluation method of a determination target model according to the embodiment. FIG. 3 is a diagram illustrating an outline (part 2) of a reliability evaluation method of a determination target model according to the embodiment. FIG. 4 is a diagram illustrating an outline (part 3) of a reliability evaluation method of a determination target model according to the embodiment. FIG. 5 is a diagram illustrating an outline (part 4) of a reliability evaluation method of a determination target model according to the embodiment. FIG. 6 is a diagram illustrating an overview (No. 5) of a reliability evaluation method for a determination target model according to the embodiment. FIG. 7 is a diagram illustrating an overview (Part 6) of a reliability evaluation method of a determination target model according to the embodiment. FIG. 8 is a diagram illustrating a configuration example of an information processing apparatus according to the embodiment; FIG. 9 is a diagram showing an overview of user information according to the embodiment. FIG. 10 is a diagram illustrating an overview of estimation model information according to the embodiment; 11 is a flowchart illustrating an example of a processing procedure performed by the information processing apparatus according to the embodiment; FIG. FIG. 12 is a hardware configuration diagram showing an example of a computer that implements the functions of the information processing apparatus according to the embodiment.

Hereinafter, modes for implementing an information processing apparatus, an information processing method, and an information processing program (hereinafter referred to as "embodiments") according to the present application will be described in detail with reference to the drawings. The information processing apparatus, information processing method, and information processing program according to the present application are not limited to the embodiments described below. Further, the embodiments described below can be appropriately combined within a range that does not contradict the processing contents. Also, in the embodiments described below, the same parts are denoted by the same reference numerals, and overlapping descriptions are omitted.

Also, in the following description, the terminal device 10 may be referred to as a user U in some cases. That is, the user U can be read as the terminal device 10 .

[1. Information processing system]
FIG. 1 is a diagram illustrating an example of an information processing system according to an embodiment. As shown in FIG. 1 , the information processing system SYS according to the embodiment has a terminal device 10 and an information processing device 100 . The information processing system SYS is not limited to the example shown in FIG. 1, and may have multiple terminal devices 10 and multiple information processing devices 100.

The terminal device 10 and the information processing device 100 are each connected to a network N (see FIG. 8, for example) by wire or wirelessly. The network N is a communication network such as a LAN (Local Area Network), a WAN (Wide Area Network), a telephone network (mobile telephone network, fixed telephone network, etc.), a local IP (Internet Protocol) network, or the Internet. . The network N may include a wired network or a wireless network. The terminal device 10 and the information processing device 100 can communicate with each other through the network N. FIG.

A terminal device 10 shown in FIG. 1 is an information processing device used by a user U who uses various services provided by an information processing device 100 or the like. For example, the terminal device 10 is implemented by a smart phone, a tablet terminal, a notebook PC (Personal Computer), a desktop PC, a PDA (Personal Digital Assistant), or the like.

In addition, the terminal device 10 is compatible with wireless communication networks such as LTE (Long Term Evolution), 4G (4th Generation), 5G (5th Generation: fifth generation mobile communication system), Bluetooth (registered trademark), wireless LAN (Local Area Network) can be connected to the network N via short-range wireless communication.

In addition, the terminal device 10 can display information for using various services using a web browser or an application. When the terminal device 10 receives control information for implementing information display processing from the information processing device 100 or the like, the terminal device 10 implements the display processing according to the control information.

Also, the user U of the terminal device 10 provides user information to the information processing device 100 through the use of various services. The user information includes a search history including a search query input by each user U, a purchase history of each user U, a history of transaction targets browsed on an electronic shopping mall, a purchase history of transaction targets purchased, and so on. , browsing history, which is the history of content such as browsed news; posting history, which is the history of texts and images posted on SNS (Social Networking Service); and location history. Further, the user information includes, for example, a QR code (registered trademark) installed in a store using a smartphone owned by each user U, and a usage history of a settlement service realized using the reading result, and A QR code may be displayed on a smartphone owned, and the usage history of a payment service (for example, date and time of purchase, product purchased, purchase amount, etc.) performed by reading the QR code with a store terminal may be included. The user information may also include a credit card usage history and a bank account usage history owned by each user U.

User information may also include information estimated based on collected information. For example, from the purchase history, if the average monthly purchase amount of food items is 100,000 yen or more, for example, the user information includes income such as a household monthly income or personal monthly income of 300,000 yen or more, or a family of three members. It may also include information that presumes a family composition such as , or a cohabitant composition. Also, the user information may include information estimated by estimating the articles owned by each user U, the place of residence and workplace of each user U, etc., based on the location history and purchase history, for example.

The user information may also include information obtained by acquiring a service usage history from a server device that provides various services other than the information processing device 100, for example. Further, the user information may include various information estimated from a service usage history by a server device that provides such a service or by a single server device. The user information may also include information indicating psychographic attributes such as various geographic attributes registered by each user U or estimated for each user U, and hobbies and preferences.

The user information may also include sensor information acquired by various sensors mounted on the terminal device 10 . For example, the sensor information includes position information obtained by a position sensor, acceleration information detected by an acceleration sensor, angular velocity information detected by a gyro sensor, operation information obtained by a touch panel, and information obtained by an illuminance sensor. illuminance information acquired by an atmospheric pressure sensor, sound information acquired by a microphone, and the like.

The information processing apparatus 100 shown in FIG. 1 is an information processing apparatus that provides various services to each user U who is a service user. Various services provided by the information processing apparatus 100 may include API (Application Programming Interface) services corresponding to various applications and various online services. Online services include Internet connection, search service, SNS (Social Networking Service), electronic commerce service, electronic payment service, online game, online banking service, online trading service, accommodation reservation service, Ticket reservation service, video distribution service, music distribution service, news distribution service, map information service, route search service, route guidance service, route information service, operation information service, weather information service, etc. service may apply.

Further, the information processing apparatus 100 is an information processing apparatus that executes information processing according to the embodiment using user information collected through provision of various services. The information processing device 100 is typically a server device, but may be realized by a mainframe, workstation, or the like.

When the information processing apparatus 100 is realized by a server device, it may be realized by a single server, or may be realized by a cloud system or the like in which a plurality of server devices and a plurality of storage devices operate in cooperation. The information processing device 100 may function as a distribution device that distributes control information to the terminal device 100 that each user U uses. Here, the control information is described in, for example, a script language such as JavaScript (registered trademark) or a style sheet language such as CSS (Cascading Style Sheets). Note that the application itself distributed from the information processing apparatus 100 may be regarded as control information.

[2. information processing]
An example of information processing in the information processing system SYS will be described below. In the example shown in FIG. 1, the information processing device 100 stores user information collected from each user U. In the example shown in FIG. The information processing apparatus 100 then uses the user information to generate a plurality of estimation models and manages the generated estimation models. The estimation model generated by the information processing apparatus 100 is a model for estimating the relevance to events such as the user's U behavior and interests and preferences based on the user information of each user.

For example, the information processing apparatus 100 collects each user U who satisfies the condition (title) “I bought XX” or “I am interested in XX” as a seed user, and uses the search query input by the seed user. Generate a score list with scores based on commonality. The information processing apparatus 100 uses this score list as an estimation model. Specifically, the seed users “wearing glasses” were collected, and the search query: “glasses” had “70 points”, the search query: “wiping glasses” had “50 points”, and the search query: “apple”. is given a score such as "-50 points" to create a score list that is a combination of the search query and the score.

The seed user identification described above can be performed using any method. For example, the information processing apparatus 100 may estimate a user who has performed an assumed action based on various types of information such as purchase history, usage history, and location history, and identify the user as a seed user. Specifically, the seed user may be identified by entering the name of a given store as a search query and then estimating the visitors who actually visited the given store. As another example, the seed user may be identified by entering the name of the event as a search query and then estimating the participants who actually attended the event. As yet another example, the seed user may be specified by inputting the name of a product as a search query and then estimating the purchaser who actually purchased the product with that name. Further, the information processing apparatus 100 may identify seed users by estimating users who have actually used a predetermined store (for example, a store corresponding to a distributed advertisement). The information processing apparatus 100 may also identify seed users by estimating viewers who have viewed predetermined advertisements, web pages, content such as movies and music.

Further, the information processing apparatus 100 can obtain the score of the search query in the score list described above from the behavior model generated by the following machine learning. For example, the information processing apparatus 100 treats the search query input by the seed user as a positive example, and the search query input by another randomly selected user other than the seed user as a negative example. The learning model is trained so that a higher value is output when a negative example is input, and a lower value is output when a negative example is input. At this time, any method can be adopted for learning. For example, when a neural network is adopted as a model, it can be realized by back propagation or the like.

In addition, the information processing apparatus 100 may learn a model so that higher numerical values are output for search queries commonly input by seed users. For example, the model may be trained to output a higher value as more seed users enter positive search queries. Also, the model may be learned so that the larger the number of other users U who have input search queries that are negative examples, the smaller the value (negative value) that is output. Then, the information processing apparatus 100 generates a list of pairs of each search query and a score output when the search query is input to such a learning model as a score list. That is, the information processing apparatus 100 receives each search query input by a seed user who has performed a predetermined action, and the score corresponding to each of the search queries (the score indicating the relationship between the search query and the predetermined action). ) as a score list. The information processing apparatus 100 manages the generated score list in association with the event estimated from the score list. The information processing apparatus 100 may adopt the conditions at the time of seed user collection (for example, "wearing glasses") as events as they are, or information based on these conditions (for example, "wearing glasses" or "owning glasses"). ”) may be newly labeled as an event associated with the score list.

Note that the information processing apparatus 100 can obtain the degree of relevance to a predetermined event by using the score list described above. For example, the information processing apparatus 100 acquires scores associated with each of a plurality of search queries arbitrarily selected by the user U from the score list, and calculates the total value of the acquired scores. The information processing apparatus 100 then determines that the event corresponding to the score list has a high degree of relevance when the total value of the calculated scores exceeds a predetermined threshold value.

The information processing apparatus 100 that generates and manages a plurality of estimation models (score list) as described above selects a reference model from among the plurality of estimation models (step S1). The reference model is a model for estimating a second event having relevance to the first event estimated by the determination target model among a plurality of estimation models managed by the information processing apparatus 100, and has a predetermined estimation accuracy. It is a model that meets the criteria of

For example, the information processing apparatus 100 identifies a first event estimated by a determination target model selected from among a plurality of estimation models, and identifies a second event related to the identified first event. Specifically, the information processing apparatus 100 uses a natural language processing technique such as Word2Vec to derive a feature amount of a character string (label) indicating an event estimated by each estimation model. Then, the information processing apparatus 100 identifies a second event whose feature amount is similar to the first event estimated by the determination target model based on the derived feature amount. Alternatively, the information processing apparatus 100 identifies a second event whose feature amount is not similar to the first event estimated by the determination target model based on the derived feature amount. Thus, the information processing apparatus 100 can identify the second event that has some relationship with the first event. That is, when the feature amount of the first event estimated by the determination target model and the feature amount of the second event estimated by the reference model are similar, the information processing apparatus 100 determines that the first event and the second event It is assumed that there is a positive correlation between Alternatively, when the feature amount of the first event estimated by the determination target model and the feature amount of the second event estimated by the reference model are not similar, the information processing apparatus 100 determines that the first event and the second event It is estimated that there is a negative correlation between

After identifying the second event, the information processing apparatus 100 acquires a model whose estimation accuracy satisfies a predetermined standard from the estimation models (score list) for estimating the identified second event. For example, the information processing apparatus 100 generates a user list, which will be described later, based on an estimation model for estimating the specified second event. Subsequently, the information processing apparatus 100 distributes an advertisement relating to the second event corresponding to the user list using the generated user list, and measures the effect of the advertisement distribution. Then, the information processing apparatus 100 selects, as a reference model, the estimation model (score list) that is the basis of the user list whose measurement result satisfies a predetermined criterion. Note that the determination target model can be arbitrarily selected by the operator of the information processing apparatus 100 .

Next, the information processing apparatus 100 generates first list information (user list ) is generated (step S2).

For example, the information processing apparatus 100 quantifies the relationship between the first event and each user U based on the search query input by each user U related to the first event and the score list. First list information (user list) in which each user U is ranked can be generated based on the value indicating the relationship.

Specifically, the information processing apparatus 100 selects each user related to the first event, and identifies the score associated with the search query input by each selected user U from the score list. Note that the information processing apparatus 100 may select each user U at random, or may select based on a predetermined condition. Then, the information processing apparatus 100 derives a user list indicating the relationship between the first event and each user U from the specified total score. For example, in the user list, a user U who has a high relationship with the first event has a high score, and a user U who has a low relationship with the predetermined behavior has a low score.

An example of a method for generating the first list information (user list) by the information processing apparatus 100 will be described below. The information processing apparatus 100 acquires scores associated with each of the plurality of search queries input by each user U including the seed user from the score list, and calculates the total value of the acquired scores. In this way, since the relationship between a plurality of search queries input by each user U and a predetermined action can be represented as a numerical value, the reliability is higher than when a single search query is quantified. Quantification can be realized.

Next, after calculating the total score value for each user U, the information processing apparatus 100 ranks each user U in descending order of the score, and associates the search queries input by each user U in the order of ranking. Generate a user list. This user list functions as a list indicating what kind of search queries the users U who have entered have a strong relevance to the first event. This user list is used for analyzing the relationship between the first event and the search query, analyzing the user U, and the like.

The information processing apparatus 100 also provides second list information (user list) in which each user U is ranked and listed based on the value indicating the relationship between each user U and the second event obtained from the reference model. is generated (step S3).

For example, the information processing device 100 quantifies the relationship between the second event and each user U based on the search query input by each user U related to the second event and the score list, and digitizes the relationship. Second list information (user list) in which each user U is ranked can be generated based on the value indicating the relationship. Note that the method of generating the second list information is the same as the method of generating the first list information described above.

Then, the information processing apparatus 100 determines the reliability of the determination target model based on whether the contents of the first list information and the second list information correspond to the relationship between the first event and the second event. Evaluate the nature (step S4). A method for evaluating the reliability of the determination target model will be described below with reference to FIGS. 2 to 7. FIG.

FIG. 2 is a diagram illustrating an overview (Part 1) of a reliability evaluation method of a determination target model according to the embodiment. In addition, in FIGS. 2 to 4 shown below, an example of an evaluation method when it is estimated that the first phenomenon "shabu-shabu group" and the second phenomenon "yakiniku group" have a positive correlation is shown. The term “shabu-shabu group” refers to users U who tend to prefer shabu-shabu among meat dishes, and the term “yakiniku group” refers to users U who tend to prefer yakiniku among meat dishes. there is

The information processing apparatus 100 generates a user list L1-aa, which is first list information, based on the estimated model (score list) of the “shabu-shabu school” selected as the determination target model. Further, the information processing apparatus 100 generates a user list L2-a, which is second list information, based on the estimated model (score list) of "Yakiniku-ha" selected as the reference model.

Then, as shown in FIG. 2, the information processing apparatus 100 compares the user list L1-aa and the user list L2-a, and determines that the contents of the user list L1-aa and the user list L2-a are the first event. It is determined whether or not the contents correspond to the relationship of the second event. For example, in the example shown in FIG. 2, it is estimated that there is a positive correlation between the first event, "shabu-shabu" and the second event, "yakiniku". In such a case, the information processing apparatus 100 can evaluate the reliability of the determination target model based on whether the contents of the first list information and the second list information are similar.

Specifically, the information processing apparatus 100 determines that the matching rate of the user IDs included in the user list L1-aa and the user list L2-a is equal to or higher than the threshold TH-1, and the user list L1-aa and the user list L2-a It is determined whether or not the condition that the match rate of the search query contained in a is equal to or greater than the threshold TH-2 is satisfied. Then, when the information processing apparatus 100 determines that the matching rate of the user ID is equal to or higher than the threshold TH-1 and that the matching rate of the search query is equal to or higher than the threshold TH-2, the information processing apparatus 100 selects "shabu-shabu school" as the determination target model. ” estimation model (score list) is highly reliable.

When determining the matching rate of user IDs and the matching rate of search queries, information processing apparatus 100 may determine the matching rates of user IDs and search queries ranked high, such as within the top 100. good.

FIG. 3 is a diagram illustrating an outline (part 2) of a reliability evaluation method of a determination target model according to the embodiment. Similar to the example shown in FIG. 2, FIG. 3 shows the evaluation method when it is estimated that the first phenomenon “shabu-shabu group” and the second phenomenon “yakiniku group” have a positive correlation. Although an example is shown, the content of the user list L1-ab, which is the first list information, is different from the example shown in FIG.

For example, information processing apparatus 100 determines that the match rate of user IDs included in user list L1-ab and user list L2-a is equal to or greater than threshold TH-1, and user list L1-ab and user list L2-a include It determines whether or not the condition that the match rate of the retrieved search query is equal to or greater than the threshold TH-2 is satisfied. Then, when the information processing apparatus 100 determines that the above-described conditions are not satisfied, that is, as shown in FIG. If it is determined that the score is less than the threshold TH-2, the reliability of the estimation model (score list) of the "shabu-shabu school" selected as the determination target model is evaluated as low.

Further, when it is estimated that the first event and the second event have a positive correlation, the information processing apparatus 100 determines whether the time-series fluctuation contents of the first list information and the second list information are similar. The reliability of the determination target model may be evaluated based on whether. FIG. 4 is a diagram illustrating an outline (part 3) of a reliability evaluation method of a determination target model according to the embodiment.

First, the information processing apparatus 100 generates a user list L1-aa, which is first list information, based on the estimated model (score list) of the “shabu-shabu school” selected as the determination target model. Further, the information processing apparatus 100 generates a user list L1-b, which is the first list information, based on the estimated model (score list) of the “shabu-shabu school” after the time T has passed.

Further, the information processing apparatus 100 generates a user list L2-a, which is second list information, based on the estimated model (score list) of "Yakiniku-ha" selected as the reference model. Further, the information processing apparatus 100 generates a user list L2-b, which is the first list information, based on the estimated model (score list) of the “yakiniku group” after the time T has passed.

Then, as shown in FIG. 4, the information processing apparatus 100 compares the user list L1-aa and the user list L2-a, and determines that the contents of the user list L1-aa and the user list L2-a are the first event. It is determined whether or not the contents correspond to the relationship of the second event. Furthermore, the information processing apparatus 100 compares the user list L1-b and the user list L2-b, and determines whether the contents of the user list L1-b and the user list L2-b are related to the first event and the second event. It is determined whether the content is in accordance with the

For example, in the example shown in FIG. 4, it is estimated that the first event "shabu-shabu" and the second event "yakiniku" have a positive correlation. Therefore, information processing apparatus 100 determines that the match rate of user IDs included in user list L1-aa and user list L2-a is equal to or greater than threshold TH-1, and user list L1-aa and user list L2-a include It determines whether or not the condition that the match rate of the retrieved search query is equal to or greater than the threshold TH-2 is satisfied. Further, the information processing apparatus 100 determines that the match rate of the user IDs included in the user list L1-b and the user list L2-b is equal to or higher than the threshold TH-1, and the user IDs included in the user list L1-b and the user list L2-b It determines whether or not the condition that the match rate of the retrieved search query is equal to or greater than the threshold TH-2 is satisfied.

Then, the information processing apparatus 100 satisfies the condition that the matching rate of the user ID is equal to or higher than the threshold TH-1 and the matching rate of the search query is equal to or higher than the threshold TH-2 before and after the time T elapses. If it is judged as, the reliability of the estimation model (score list) of "shabu-shabu school" selected as the judgment target model is evaluated as high.

FIG. 5 is a diagram illustrating an outline (part 4) of a reliability evaluation method of a determination target model according to the embodiment. 5 to 7 shown below are an example of an evaluation method when it is estimated that the first event, “eating out” and the second event, “cooking at home” have a negative correlation. is shown. The term “eating out” refers to users U who tend to eat out, and the term “self-cooking” refers to users U who tend to prefer to cook their own meals.

The information processing apparatus 100 generates a user list L11-aa, which is first list information, based on the estimated model (score list) of the “eating out” type selected as the determination target model. Further, the information processing apparatus 100 generates a user list L22-a, which is second list information, based on the estimated model (score list) of the “self-cooking group” selected as the reference model.

Then, as shown in FIG. 5, the information processing apparatus 100 compares the user list L11-aa and the user list L22-a, and determines that the contents of the user list L11-aa and the user list L22-a are the first event. It is determined whether or not the contents correspond to the relationship of the second event. For example, in the example shown in FIG. 5, it is estimated that there is a negative correlation between the first event, “eating out” and the second event, “cooking at home”. In such a case, the information processing apparatus 100 can evaluate the reliability of the determination target model based on whether the contents of the first list information and the second list information are dissimilar (dissimilar). .

Specifically, the information processing apparatus 100 determines that the matching rate of the user IDs included in the user list L11-aa and the user list L22-a is less than the threshold TH-1, and the user list L11-aa and the user list L22-a It is determined whether or not the condition that the matching rate of the search query included in a is less than the threshold TH-2 is satisfied. When the information processing apparatus 100 determines that the matching rate of the user ID is less than the threshold TH-1 and that the matching rate of the search query is less than the threshold TH-2, the information processing apparatus 100 selects the "restaurant group" as the determination target model. ” estimation model (score list) is highly reliable.

FIG. 6 is a diagram illustrating an overview (No. 5) of a reliability evaluation method for a determination target model according to the embodiment. Similar to the example shown in FIG. 5, FIG. 6 shows the evaluation method when it is estimated that the first event, “eating out” and the second event, “cooking at home” have a negative correlation. Although an example is shown, the content of the user list L11-ab, which is the first list information, is different from the example shown in FIG.

For example, information processing apparatus 100 determines that the match rate of user IDs included in user list L11-ab and user list L22-a is less than threshold TH-1, and user list L11-ab and user list L22-a include It determines whether or not the condition that the match rate of the retrieved search query is less than the threshold TH-2 is satisfied. Then, when the information processing apparatus 100 determines that the above-described conditions are not satisfied, that is, as shown in FIG. When it is determined that the score is equal to or greater than the threshold TH-2, the reliability of the estimation model (score list) of the "eating out type" selected as the determination target model is evaluated as low.

Further, when the first event and the second event are estimated to have a negative correlation, the information processing apparatus 100 determines that the time-series variation contents of the first list information and the second list information are dissimilar. The reliability of the determination target model may be evaluated based on whether or not. FIG. 7 is a diagram illustrating an overview (Part 6) of a reliability evaluation method of a determination target model according to the embodiment.

First, the information processing apparatus 100 generates a user list L11-aa, which is first list information, based on the estimated model (score list) of the “eating out” type selected as the determination target model. Further, the information processing apparatus 100 generates a user list L11-b, which is the first list information, based on the estimated model (score list) of "eating out" after the time T has passed.

Further, the information processing apparatus 100 generates a user list L22-a, which is second list information, based on the estimated model (score list) of the “self-cooking group” selected as the reference model. Further, the information processing apparatus 100 generates a user list L22-b, which is the first list information, based on the estimated model (score list) of the “self-cooking group” after the time T has passed.

Then, as shown in FIG. 7, the information processing apparatus 100 compares the user list L11-aa and the user list L22-a, and determines that the contents of the user list L11-aa and the user list L22-a are the first event. It is determined whether or not the contents correspond to the relationship of the second event. Furthermore, the information processing apparatus 100 compares the user list L11-b and the user list L22-b, and determines whether the contents of the user list L11-b and the user list L22-b are related to the first event and the second event. It is determined whether the content is in accordance with the

For example, in the example shown in FIG. 7, it is estimated that there is a negative correlation between the first event, “eating out” and the second event, “cooking at home”. Therefore, the information processing apparatus 100 determines that the matching rate of the user IDs included in the user list L11-aa and the user list L22-a is less than the threshold TH-1 and It determines whether or not the condition that the match rate of the retrieved search query is less than the threshold TH-2 is satisfied. Further, the information processing apparatus 100 determines that the matching rate of the user IDs included in the user list L11-b and the user list L22-b is less than the threshold TH-1 and It determines whether or not the condition that the match rate of the retrieved search query is less than the threshold TH-2 is satisfied.

Then, the information processing apparatus 100 satisfies the condition that the matching rate of the user ID is less than the threshold TH-1 and the matching rate of the search query is less than the threshold TH-2 before and after the time T has passed. If it is judged as such, the reliability of the estimation model (score list) of the "eating out group" selected as the judgment target model is evaluated as high.

As described above, the information processing apparatus 100 according to the embodiment uses the second list information having a predetermined reliability, so that the content of the first list information is changed according to the relationship between the first event and the second event. It is possible to rationally confirm whether the content is correct, and indirectly evaluate the reliability of the determination target model that is the source of the first list information. As such, the information processing apparatus 100 according to the embodiment can confirm the reliability of the determination target model (score list) that is the generated information.

[3. Configuration of information processing device]
The configuration of the information processing apparatus 100 according to the embodiment will be described using FIG. FIG. 8 is a diagram illustrating a configuration example of an information processing apparatus according to the embodiment;

As shown in FIG. 8, the information processing apparatus 100 includes a communication section 110, a storage section 120, and a control section . Note that FIG. 8 shows a configuration example of the information processing apparatus 100, and is not limited to the form shown in FIG.

(Communication unit 110)
The communication unit 110 is connected to the network N by wire or wirelessly, for example, and transmits and receives information to and from another device such as the terminal device 10 via the network N. The communication unit 110 is implemented by, for example, a NIC (Network Interface Card), an antenna, or the like. The network N is a communication network such as a LAN (Local Area Network), a WAN (Wide Area Network), a telephone network (mobile telephone network, fixed telephone network, etc.), a regional IP (Internet Protocol) network, or the Internet. . The network N may include wired networks or wireless networks.

The communication unit 110 receives information such as captured images from the terminal device 10 . Also, the communication unit 110 transmits the proposal information to the terminal device 10 .

(storage unit 120)
The storage unit 120 is realized by, for example, a RAM (Random Access Memory), a semiconductor memory device such as a flash memory, or a storage device such as a hard disk or an optical disk. Storage unit 120 stores programs and data for realizing information processing executed by each unit of control unit 130 .

As shown in FIG. 8 , storage unit 120 has user information storage unit 121 and estimation model storage unit 122 .

(User information storage unit 121)
User information provided by each user is stored in the user information storage unit 121 in an arbitrary format. FIG. 9 is a diagram showing an overview of user information according to the embodiment.

As shown in FIG. 9, the user information includes items such as "user ID", "search history", and "purchase history". These items in the estimation model information are associated with each other.

The “user ID” item stores identification information that is individually given to each user by the information processing apparatus 100 in order to identify users who use various services when various services are provided.

The "search history" item stores information about the search history provided by each user as user information. In addition, in the item "Purchase History", information related to the purchase history provided by each user as user information is stored.

(Estimated model storage unit 122)
The estimation model storage unit 122 stores estimation model information about the estimation model in an arbitrary format. FIG. 10 is a diagram illustrating an overview of estimation model information according to the embodiment;

As shown in FIG. 5, the estimated model information has a "model ID" item, an "event" item, and a "data" item. These items of the estimated model information are associated with each other.

The “model ID” item stores identification information that is individually assigned to each estimation model to identify the estimation model.

"Event" stores information indicating the contents of the user's behavior, interests, and preferences estimated by the associated estimation model.

The "data" item stores the data of the associated estimation model. Note that FIG. 10 shows an example in which conceptual information such as "data A" and "data B" is stored in the "data" item. configuration) and information about parameters, and various other information that constitutes the corresponding estimation model. For example, the individual pieces of information that make up the estimation model include information including nodes in each layer of the network, functions adopted by each node, connection relations between nodes, and connection coefficients set for connections between nodes. can be included.

(control unit 130)
A control unit 130 shown in FIG. 8 is a controller that controls the information processing apparatus 100 . The control unit 130 executes various programs (for example, an information processing program) stored in a storage device inside the information processing apparatus 100 by a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), or the like. is executed using RAM as a work area. Also, the control unit 130 may be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

As shown in FIG. 8 , the control unit 130 has a selection unit 131 , a generation unit 132 and an evaluation unit 133 . Control unit 130 implements or executes the functions and actions of information processing described below by means of these units. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 8, and may be another configuration as long as it performs information processing described later. Moreover, the connection relationship of each part of the control unit 130 is not limited to the connection relationship shown in FIG. 8, and may be another connection relationship.

(Selection unit 131)
The selection unit 131 specifies a reference model, which is a model for estimating a second event that is related to the first event estimated by the determination target model, and whose estimation accuracy satisfies a predetermined standard. The reference model is a model for estimating a second event having relevance to the first event estimated by the determination target model among a plurality of estimation models managed by the information processing apparatus 100, and has a predetermined estimation accuracy. It is a model that meets the criteria of

For example, the selection unit 131 identifies a first event estimated by a determination target model selected from among a plurality of estimation models, and identifies a second event related to the identified first event. Specifically, the selection unit 131 uses a natural language processing technique such as Word2Vec to derive a feature quantity of a character string (label) indicating an event estimated by each estimation model. Then, the information processing apparatus 100 identifies a second event whose feature amount is similar to the first event estimated by the determination target model based on the derived feature amount. Alternatively, the information processing apparatus 100 identifies a second event whose feature amount is not similar to the first event estimated by the determination target model based on the derived feature amount. Thus, the selection unit 131 can identify a second event that has some relationship with the first event. That is, when the feature amount of the first event estimated by the determination target model and the feature amount of the second event estimated by the reference model are similar, the information processing apparatus 100 determines that the first event and the second event It is assumed that there is a positive correlation between Alternatively, if the feature amount of the first event estimated by the determination target model and the feature amount of the second event estimated by the reference model are not similar, the selection unit 131 selects the first event and the second event. Assume that there is a negative correlation.

After specifying the second event, the selection unit 131 acquires a model whose estimation accuracy satisfies a predetermined standard from the estimation models (score list) for estimating the specified second event. For example, the selection unit 131 generates a user list, which will be described later, based on an estimation model that estimates the specified second event. Next, using the generated user list, the selection unit 131 distributes an advertisement regarding the second event corresponding to the user list, and measures the effect of the advertisement distribution. Then, the information processing apparatus 100 selects, as a reference model, the estimation model (score list) that is the basis of the user list whose measurement result satisfies a predetermined criterion. Note that the determination target model can be arbitrarily selected by the operator of the information processing apparatus 100 .

(Generating unit 132)
The generation unit 132 generates first list information in which each user is ranked and listed based on the value indicating the relationship between each user and the first event obtained from the determination target model, and each user obtained from the reference model. and second list information in which each user is ranked and listed based on the value indicating the relationship with the second event.

For example, the generation unit 132 quantifies and quantifies the relationship between the first event and each user U based on the search query input by each user U related to the first event and the score list. First list information (user list) in which each user U is ranked based on the value indicating the relationship can be generated.

Specifically, the generation unit 132 selects each user related to the first event, and identifies the score associated with the search query input by each selected user U from the score list. Note that the information processing apparatus 100 may select each user U at random, or may select based on a predetermined condition. Then, the generation unit 132 acquires scores associated with each of the plurality of search queries input by each user including the seed user from the score list, and calculates the total value of the acquired scores. In this way, the relationship between a plurality of search queries input by each user and a predetermined action can be expressed as a numerical value. can be realized.

Next, after calculating the total score for each user U, the generation unit 132 ranks each user U in descending order of the score, and associates the search queries input by each user U in the order of rank to obtain the first A user list, which is one list information, is generated. This user list functions as a list indicating what kind of search queries the users U who have entered have a strong relevance to the first event. This user list is used for analysis of relevance between the first event and search queries, analysis of users, and the like. Note that the generation unit 132 can generate the user list, which is the second list information, by the same method as the method for generating the first list information described above.

(Evaluation unit 133)
The evaluation unit 133 compares the first list information and the second list information, and the contents of the first list information and the second list information correspond to the relationship between the first event and the second event. Evaluate the reliability of the model to be judged based on whether or not there is.

For example, when it is estimated that the first event and the second event have a positive correlation, the evaluation unit 133 determines whether the contents of the first list information and the second list information are similar. Based on, the reliability of the model to be judged can be evaluated (see, for example, FIGS. 2 and 3). Further, for example, when the first event and the second event are estimated to have a positive correlation, the evaluation unit 133 determines that the time-series fluctuation contents of the first list information and the second list information are similar. The reliability of the model to be judged may be evaluated based on whether or not there is (for example, see FIG. 4).

In addition, when it is estimated that the first event and the second event have a negative correlation, the evaluation unit 133 determines whether the contents of the first list information and the second list information are dissimilar. Based on what, the reliability of the model to be determined can be evaluated (see, eg, FIGS. 5 and 6). In addition, when the first event and the second event are estimated to have a negative correlation, the evaluation unit 133 determines whether the time-series fluctuation contents of the first list information and the second list information are dissimilar. The reliability of the model to be determined may be evaluated based on whether (see, for example, FIG. 7).

[4. Processing procedure]
A processing procedure performed by the information processing apparatus 100 according to the embodiment will be described below with reference to FIG. 11 . 11 is a flowchart illustrating an example of a processing procedure performed by the information processing apparatus according to the embodiment; FIG. The processing procedure shown in FIG. 11 is executed by the control unit 130 of the information processing device 100 . The processing procedure shown in FIG. 11 is repeatedly executed while the information processing apparatus 100 is in operation.

As shown in FIG. 11, the selection unit 131 is a reference model for estimating a second event that is related to a first event estimated by the determination target model and whose estimation accuracy satisfies a predetermined standard. is selected (step S101).

In addition, the generation unit 132 generates first list information in which each user is ranked and listed based on the value indicating the relationship between each user and the first event obtained from the determination target model (step S102). .

In addition, the generating unit 132 generates second list information in which users are ranked and listed based on the value indicating the relationship between each user and the second event obtained from the reference model (step S103). .

Then, the evaluation unit 133 compares the first list information and the second list information, and determines the contents of the first list information and the second list information according to the relationship between the first event and the second event. The reliability of the determination target model is evaluated based on whether or not it is (step S104), and the processing procedure shown in FIG. 11 is terminated.

[5. Modification]
The information processing apparatus 100 described above may be implemented in various different forms other than the embodiments described above. So, below, the modification of embodiment is demonstrated.

(5-1. Modification of judgment target model)
In the above-described embodiment, the information processing apparatus 100 may correct the determination target model based on the reference model when the reliability of the determination target model is evaluated as low. For example, the information processing apparatus 100 corrects the determination target model by adjusting the score of search queries common to the reference model among the search queries included in the determination target model to increase.

(5-2. Regarding the provision of information on the evaluation of the judgment target model)
In the above-described embodiment, the information processing apparatus 100 may provide the operator of the information processing apparatus 100 with information regarding the evaluation of the determination target model. For example, the information processing apparatus 100 can provide information on the reference model used for evaluation of the determination target model, the match rate of the user ID, the match rate of the search query, and the like.

[6. Hardware configuration]
The information processing apparatus 100 according to the embodiment is implemented by, for example, a computer 1000 configured as shown in FIG. FIG. 12 is a hardware configuration diagram showing an example of a computer that implements the functions of the information processing apparatus according to the embodiment.

A computer 1000 is connected to an output device 1010 and an input device 1020, and an arithmetic device 1030, a primary storage device 1040, a secondary storage device 1050, an output IF (Interface) 1060, an input IF 1070, and a network IF 1080 are connected via a bus 1090. have

The arithmetic device 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, programs read from the input device 1020, and the like, and executes various processes. The primary storage device 1040 is a memory device such as a RAM that temporarily stores data used by the arithmetic device 1030 for various calculations. The secondary storage device 1050 is a storage device in which data used for various calculations by the arithmetic device 1030 and various databases are registered, and is implemented by a ROM (Read Only Memory), HDD, flash memory, or the like.

The output IF 1060 is an interface for transmitting information to be output to the output device 1010 that outputs various types of information such as a monitor and a printer. It is realized by a connector conforming to a standard such as HDMI (registered trademark) (High Definition Multimedia Interface). Also, the input IF 1070 is an interface for receiving information from various input devices 1020 such as a mouse, keyboard, scanner, etc., and is realized by, for example, USB.

Note that the input device 1020 includes, for example, optical recording media such as CDs (Compact Discs), DVDs (Digital Versatile Discs), PDs (Phase change rewritable discs), magneto-optical recording media such as MOs (Magneto-Optical discs), and tapes. It may be a device that reads information from a medium, a magnetic recording medium, a semiconductor memory, or the like. Also, the input device 1020 may be an external storage medium such as a USB memory.

Network IF 1080 receives data from other devices via network N and sends the data to arithmetic device 1030, and also transmits data generated by arithmetic device 1030 via network N to other devices.

The arithmetic device 1030 controls the output device 1010 and the input device 1020 via the output IF 1060 and the input IF 1070 . For example, arithmetic device 1030 loads a program from input device 1020 or secondary storage device 1050 onto primary storage device 1040 and executes the loaded program.

For example, when the computer 1000 functions as the information processing device 100, the arithmetic device 1030 of the computer 1000 executes a program (for example, an information processing program) loaded on the primary storage device 1040 to perform the same functions as the control unit 130. to realize the function of That is, the arithmetic device 1030 realizes processing by the information processing device 100 according to the embodiment in cooperation with a program (for example, an information processing program) loaded on the primary storage device 1040 .

[7. others]
Of the processes described in the above embodiments, all or part of the processes described as being automatically performed can be manually performed, or all of the processes described as being manually performed. Alternatively, some can be done automatically by known methods. In addition, information including processing procedures, specific names, various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified.

In the above-described embodiment, in order to realize the information processing method (see, for example, FIG. 11) by the information processing apparatus 100, each unit (the selection unit 131, the generation unit 132, and the evaluation unit) of the control unit 130 included in the information processing apparatus 100 133) may be implemented as an add-on to an information processing program pre-installed in the information processing apparatus 100, or may be flexibly described as a dedicated providing program using a lightweight programming language or the like. It may be realized by

Also, each component of each device illustrated is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to the one shown in the figure, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. Can be integrated and configured. For example, the selection unit 131 and the generation unit 132 of the control unit 130 may be functionally integrated.

Further, the above-described embodiment and modifications can be appropriately combined within a range that does not contradict the processing content.

[8. effect]
The information processing apparatus 100 according to the above embodiment includes a selection unit 131 , a generation unit 132 and an evaluation unit 133 . The selection unit 131 selects a reference model, which is a model for estimating a second event related to the first event estimated by the determination target model and whose estimation accuracy satisfies a predetermined standard. The generation unit 132 generates first list information in which each user is ranked and listed based on the value indicating the relationship between each user and the first event obtained from the determination target model, and each user obtained from the reference model. and second list information in which each user is ranked and listed based on the value indicating the relationship with the second event. The evaluation unit 133 compares the first list information and the second list information, and the contents of the first list information and the second list information correspond to the relationship between the first event and the second event. Evaluate the reliability of the model to be judged based on whether or not there is.

For this reason, the information processing apparatus 100 according to the embodiment indirectly evaluates the reliability of the determination target model that is the source of the first list information by using the second list information having predetermined reliability. can. Further, when evaluating the determination target model, the information processing apparatus 100 according to the embodiment confirms whether the content of the first list information corresponds to the relationship between the first event and the second event. , the reliability of the model to be determined can be evaluated appropriately. As such, the information processing apparatus 100 according to the embodiment can confirm the reliability of the determination target model, which is the generated information.

Further, in the information processing apparatus 100 according to the embodiment, when the first event and the second event are estimated to have a positive correlation, the evaluation unit 133 determines the contents of the first list information and the second list information Evaluate the reliability of the model to be determined based on whether the are similar to each other.

Further, in the information processing apparatus 100 according to the embodiment, the evaluation unit 133 evaluates the reliability of the judgment model based on whether the time-series fluctuation contents of the first list information and the second list information are similar. do.

As a result, the information processing apparatus 100 according to the embodiment is a determination target model for estimating a first event having a positive correlation with a second event by simply comparing the contents of the first list information and the second list information. evaluation can be easily realized.

Further, in the information processing apparatus 100 according to the embodiment, when the first event and the second event have a negative correlation, the evaluation unit 133 determines that the contents of the first list information and the second list information are dissimilar. Evaluate the reliability of the model to be determined based on whether there is.

Further, in the information processing apparatus 100 according to the embodiment, the evaluation unit 133 evaluates the reliability of the determination target model based on whether or not the time-series variation contents of the first list information and the second list information are dissimilar. evaluate.

As a result, the information processing apparatus 100 according to the embodiment is a determination target model that estimates a first event having a negative correlation with a second event by simply comparing the contents of the first list information and the second list information. evaluation can be easily realized.

As described above, the embodiments of the present application have been described in detail based on several drawings, but these are examples, and various modifications and It is possible to carry out the invention in other forms with modifications.

Also, the above-mentioned "section, module, unit" can be read as "means" or "circuit". For example, the control unit can be read as control means or a control circuit.

10 terminal device 100 information processing device 110 communication unit 120 storage unit 121 user information storage unit 122 estimation model storage unit 130 control unit 131 selection unit 132 generation unit 133 evaluation unit

Claims (7)

a selection unit that selects a reference model that is a model that estimates a second event that is related to the first event that is estimated by the determination target model and whose estimation accuracy satisfies a predetermined standard;
First list information listing each user in order based on a value indicating the relationship between each user and the first event obtained from the determination target model, and each user obtained from the reference model a generation unit that generates second list information in which the users are ranked and listed based on the value indicating the relationship with the second event;
The first list information and the second list information are compared, and the contents of the first list information and the second list information are changed according to the relationship between the first event and the second event. and an evaluation unit that evaluates the reliability of the determination target model based on whether or not the determination target model is an information processing apparatus. The evaluation unit
When the first event and the second event are estimated to have a positive correlation, based on whether the contents of the first list information and the second list information are similar, the determination target The information processing apparatus according to claim 1, wherein the reliability of the model is evaluated. The evaluation unit
3. The information processing according to claim 2, wherein the reliability of the determination target model is evaluated based on whether the time-series fluctuation contents of the first list information and the second list information are similar. Device. The evaluation unit
When the first event and the second event have a negative correlation, the reliability of the determination target model based on whether the contents of the first list information and the second list information are dissimilar 2. The information processing apparatus according to claim 1, wherein the property is evaluated. The evaluation unit
2. The information according to claim 1, wherein the reliability of the determination target model is evaluated based on whether the time-series fluctuation contents of the first list information and the second list information are dissimilar. processing equipment. A computer-executed information processing method comprising:
a selection step of selecting a reference model, which is a model for estimating a second event related to the first event estimated by the determination target model and whose estimation accuracy satisfies a predetermined standard;
First list information listing each user in order based on a value indicating the relationship between each user and the first event obtained from the determination target model, and each user obtained from the reference model a generating step of generating second list information in which the users are ranked and listed based on the value indicating the relationship with the second event;
The first list information and the second list information are compared, and the contents of the first list information and the second list information are changed according to the relationship between the first event and the second event. and an evaluation step of evaluating the reliability of the model to be determined based on whether or not it is an information processing method. to the computer,
a selection procedure for selecting a reference model, which is a model for estimating a second event related to the first event estimated by the determination target model and whose estimation accuracy satisfies a predetermined standard;
First list information listing each user in order based on a value indicating the relationship between each user and the first event obtained from the determination target model, and each user obtained from the reference model a generation procedure for generating second list information in which the users are ranked and listed based on the value indicating the relationship with the second event;
The first list information and the second list information are compared, and the contents of the first list information and the second list information are changed according to the relationship between the first event and the second event. and an evaluation procedure for evaluating the reliability of the determination target model based on whether or not the model is determined to be an information processing program.

Images