JP2018156323A - Extraction device, extraction method, and extraction program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an extraction device, an extraction method, and an extraction program that extract users having histories inappropriate for learning a predictive model.SOLUTION: Disclosed is an information providing device representing an extraction device, and that includes: an identification part for identifying a period in which a selection ratio of a content decreases; and an extraction part for, from users to which the content is distributed within the period identified by the identification part, extracting the users satisfying a prescribed condition.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an extraction apparatus, an extraction method, and an extraction program.

  2. Description of the Related Art Conventionally, a technique for predicting whether or not content for a user is selected by the user is known. As an example of such a technique, based on a log indicating whether or not the distributed content has been selected by the user, a prediction model that learns the user attribute and whether or not to select the content is generated. Techniques to do this have been proposed.

JP 2014-174753 A

  However, with the above-described conventional technology, there is a possibility that a prediction model cannot be generated appropriately.

  For example, it is considered that content distribution destinations include users who do not select content such as advertisements and patrol programs (so-called bots) in addition to users who appropriately select content such as advertisements. . If the log contains a history of delivering content to such users and bots, it will include a history of ads not being selected regardless of the user's attributes, which reduces the prediction accuracy of the prediction model. End up.

  The present application has been made in view of the above, and an object thereof is to extract a history user who is not suitable for learning a prediction model.

  The extraction apparatus according to the present application extracts a specifying unit that specifies a period during which a content selection ratio decreases, and a user who satisfies a predetermined condition from users to whom the content has been distributed during the period specified by the specifying unit And an extraction unit that performs the above-described process.

  According to an aspect of the embodiment, it is possible to extract a history user who is not suitable for learning a prediction model.

FIG. 1 is a diagram illustrating an example of an extraction process executed by the information providing apparatus according to the embodiment. FIG. 2 is a diagram illustrating an example of a process in which the information providing apparatus according to the embodiment learns the characteristics of an excluded user. FIG. 3 is a diagram illustrating a configuration example of the information providing apparatus according to the embodiment. FIG. 4 is a diagram illustrating an example of information registered in the log database according to the embodiment. FIG. 5 is a diagram illustrating an example of information registered in the user database according to the embodiment. FIG. 6 is a diagram illustrating an example of processing in which the information providing apparatus according to the embodiment specifies a specific period. FIG. 7 is a diagram for describing processing in which the information providing apparatus according to the embodiment extracts excluded users. FIG. 8 is a flowchart illustrating an example of a flow of extraction processing executed by the information providing apparatus according to the embodiment. FIG. 9 is a diagram illustrating an example of a hardware configuration.

  Hereinafter, an embodiment (hereinafter referred to as “embodiment”) for carrying out an extraction apparatus, an extraction method, and an extraction program according to the present application will be described in detail with reference to the drawings. In addition, the extraction apparatus, the extraction method, and the extraction program which concern on this application are not limited by this embodiment. In the following embodiments, the same parts are denoted by the same reference numerals, and redundant description is omitted.

[Embodiment]
[1. About the processing displayed by the information providing device)
First, an example of an extraction process executed by an information providing apparatus as an example of an extraction apparatus will be described with reference to FIG. FIG. 1 is a diagram illustrating an example of an extraction process executed by the information providing apparatus according to the embodiment.

  In the following description, as the extraction process executed by the information providing apparatus 10, the selected content and the usage in which the content is selected from the distribution history and selection history (hereinafter referred to as “log”) of the content related to the advertisement. Although an example of the process which extracts an appropriate log as learning data of the prediction model which learns the characteristic (for example, co-occurrence) between a person's characteristic is described, embodiment is not limited to this . The information providing apparatus 10 may extract an appropriate log as learning data from a distribution history of arbitrary content such as a news feed or an image.

  Further, in the following description, the advertisement is a concept including not only a commercial or non-profit advertisement but also a recruitment of volunteers, a public advertisement, a notification to the public, and other arbitrary contents. In addition, the advertisement is included not only in content including so-called advertisement-related information but also in information related to an arbitrary target or content related to an arbitrary target (for example, a landing page) that interests the user. An image, a moving image, text, a figure, a symbol, a hyperlink, and other arbitrary contents may be included with the text as long as the information is widely notified.

[1-1. Example of information providing device)
The information providing apparatus 10 is an information processing apparatus capable of communicating with the log server 100 and the distribution server 200 via a predetermined network N (for example, see FIG. 3) such as the Internet. For example, the information providing apparatus 10 is a server apparatus or a cloud system. Etc. The information providing apparatus 10 may be able to communicate with an arbitrary number of log servers 100 and distribution servers 200 via the network N. For example, the information providing apparatus 10 can communicate with various external servers not shown. Good.

  The log server 100 is a server device that manages logs, and is realized by, for example, a server device or a cloud system. For example, the log server 100 includes a website accessed by the user, web content distributed to the user, advertisement content distributed to the user, content selected by the user among the distributed content, Acquire and manage various log data associated with content distribution, such as demographic attributes and psychographic attributes. The function exhibited by the log server 100 is the same as that of a well-known log server, and detailed description thereof is omitted.

  The distribution server 200 is an information processing apparatus that distributes content to users, and is realized by, for example, a server apparatus or a cloud system. For example, the distribution server 200 selects content that is highly likely to be selected by the user based on various attributes of the user who has accessed the web content, and distributes the selected content. As a more specific example, the distribution server 200 performs an advertisement listing based on a user attribute and distributes an advertisement selected as a result of the listing.

[1-2. Example of extraction process)
Here, in order to deliver content that is highly likely to be selected by the user, the attribute of the user who selected the content, the attribute of the user who did not select the content, and the characteristics of the content are used as learning data. A mode is conceivable in which a prediction model that learns features (for example, co-occurrence) is generated, and listing is performed using the learning model. A technique is known that uses a log that is a content distribution history or a content selection history when generating such a prediction model.

  However, such a log may include inappropriate information as learning data. For example, there are cases where the users who are the distribution destinations of the contents include users who never select the contents regardless of the contents. In addition, the users who are the distribution destinations of the contents may include users who intentionally access in order to deteriorate the CTR (Click Through Rate) from the viewpoint of advertisement costs and distribution modes. In addition, a user serving as a content distribution destination may include a so-called traveling bot. When such a user distribution history or selection history is included in the learning data, there is a possibility that the relationship between the feature of the content and the attribute of the user who selects the content cannot be learned accurately.

  Therefore, the information providing apparatus 10 executes the following extraction process. First, the information providing apparatus 10 specifies a period during which the content selection ratio decreases. And the information provision apparatus 10 extracts the user who satisfy | fills a predetermined condition from the users by which the content was delivered in the specified period. Here, the users extracted by the information providing apparatus 10 are not only users who do not select content but users who make access for worsening CTR, as well as accesses that have made inappropriate access as learning data. It is a concept that shows the origin, and includes not only natural people but also programs such as traveling bots.

[1-3. (Specifying the period)
For example, the information providing apparatus 10 collects a distribution history (for example, a user who is a distribution destination and a distribution date / time) and a selection history (for example, the number of clicks and a date / time) about the content to be determined as a log, and The number of distributions (for example, the number of impressions) per time (for example, every minute) and the number of selections (for example, the number of clicks) are calculated. Then, the information providing apparatus 10 specifies the ratio of the number of selections with respect to the number of content distributions, that is, the period during which CTR decreases beyond a predetermined threshold as the selection ratio.

  Here, the number of selections when the number of distributions has increased in the ideal state for learning data acquisition, such as when the users to whom content is distributed are uniformly included in each attribute. Will also rise. However, a user who does not select content, a user who deteriorates CTR, a user who is inappropriate as an acquisition target of learning data, such as a traveling bot (hereinafter, may be referred to as an “excluded user”) may be included. In this case, even if the number of distributions increases, it is considered that the increase rate of the number of selections does not accompany the increase rate of the distribution number.

  Therefore, the information providing apparatus 10 calculates a CTR value per unit time, and sets a period during which the calculated CTR value is reduced as a specific period. It is considered that excluded users are included in the users whose contents are distributed by accessing a predetermined website during such a specific period. Therefore, the information providing apparatus 10 extracts, as excluded users, users who satisfy a predetermined condition from users accessed during a specific period. More specifically, the information providing apparatus 10 specifies a period in which the number of distributions of content per unit time increases and the content selection ratio decreases. For example, the information providing apparatus 10 identifies a period in which the number of selections is not increasing despite the increase in the number of distributions of content or a period in which the increase in the number of selections is lower than the increase in the number of distributions. To do.

  As a result of such processing, the information providing apparatus 10 sets a specific period as a period during which it is estimated that a relatively large number of excluded users are included in the users to whom the content to be determined is distributed. Then, the information providing apparatus 10 excludes a user who satisfies a predetermined condition from users (that is, users who have accessed a predetermined website) from which content to be determined is distributed during a specific period. Extract as

[1-4. About extraction of excluded users)
Here, the information providing apparatus 10 may exclude all users who have accessed in a specific period as excluded users, and exclude the data of excluded users from the log data. On the other hand, the information providing apparatus 10 extracts users who are highly likely to be excluded users, that is, users who are highly likely to be inappropriate as learning data acquisition targets, from among users who have accessed during a specific period. May be. For example, the information providing apparatus 10 may extract a user who has a high possibility of being an excluded user based on an access history of a user who has accessed during a specific period.

  For example, it is considered that a predetermined pattern is included in an action history on the web in association with a life pattern and an action pattern for a general user, that is, an appropriate user as an object of learning data. Therefore, the information providing apparatus 10 specifies the access pattern of the user from the log of the user who accessed during the specific period, that is, the user to whom the content to be determined was distributed during the specific period. For example, the information providing apparatus 10 specifies a date / time pattern for accessing a predetermined website or a date / time pattern for frequently distributing content as an access pattern.

  In addition, it is predicted that a user who is appropriate as an object of learning data matches a pattern for accessing a predetermined website with a pattern for selecting content. Therefore, the information providing apparatus 10 is a pattern in which a user selects content from a log of a user who has accessed during a specific period, that is, a user to whom content to be determined is distributed during the specific period. Specify the selection pattern. For example, the information providing apparatus 10 specifies the date and time pattern for selecting content based on the history of the date and time when the user selected content.

  And the information provision apparatus 10 extracts an exclusion user based on the specified access pattern and selection pattern. For example, the information providing apparatus 10 excludes a user who is unlikely to access a predetermined website in a specific period or a content is unlikely to be distributed, or a user who is unlikely to select a content in a specified period. Extract as a user. To give a more specific example, the information providing apparatus 10 may be a user who is unlikely to access for a specific period based on the access pattern and the selection pattern, or may click on content during the specific period. Identify low users. That is, the information providing apparatus 10 identifies a user who has a low possibility of accessing during a specific period, or a user who is accessing although there is a low possibility of clicking during a specific period. Then, the information providing apparatus 10 extracts the specified user as an excluded user.

  Note that the information providing apparatus 10 extracts, as excluded users, users who have a possibility of accessing a predetermined website in a specific period or a possibility of selecting content in a specific period being lower than a predetermined threshold. May be. In addition, the information providing apparatus 10 uses the possibility that a predetermined website is accessed in a specific period is lower than a predetermined threshold, and any of the possibilities to select content in the specific period is lower than the predetermined threshold. A person may be extracted as an excluded user.

  Here, when the excluded user is extracted, the information providing apparatus 10 generates log data excluding information on the excluded user as learning data. And the information provision apparatus 10 produces | generates the prediction model which learned the relationship between the characteristic of a content, and the characteristic of the attribute of the user who selected the content using the produced | generated learning data. That is, the information providing apparatus 10 can generate learning data suitable for generating a prediction model by excluding the distribution history and selection history of the excluded user from the distribution history and selection history.

[1-5. About an example of extraction processing)
Next, an example of extraction processing executed by the information providing apparatus 10 will be described with reference to FIG. First, the information providing apparatus 10 acquires various log data related to predetermined content from the log server 100 (step S1). In such a case, the information providing apparatus 10 specifies a period during which the CTR decreases while the number of impressions increases as a specific period (step S2). For example, the information providing apparatus 10 calculates the number of impressions and the number of clicks of content to be determined from log data for each unit time. Then, the information providing apparatus 10 calculates the ratio of the number of clicks with respect to the number of impressions, that is, the CTR for each unit time, and specifies the period in which the calculated CTR is decreasing and the number of impressions is increasing.

  For example, in the example shown in FIG. 1A, since the number of impressions of a predetermined content to be determined has increased while the number of clicks has not increased, the CTR value decreases at a certain timing. ing. In such a case, the information providing apparatus 10 may, for example, from the date and time when the CTR decrease rate exceeds a predetermined threshold to the date and time after a predetermined period or the date and time when the CTR decrease rate is equal to or less than the predetermined threshold. The period is specified as a specific period CP. On the other hand, in the example shown in FIG. 1B, since the number of clicks increases as the number of impressions of the predetermined content to be determined increases, the value of CTR becomes a substantially constant value. In such a case, the information providing apparatus 10 continues the process without specifying the specific period.

  Subsequently, the information providing apparatus 10 calculates the possibility that the user accesses a predetermined website during the specific period and selects content based on the behavior history of the user who accessed during the specific period (step) S3). For example, in the example illustrated in FIG. 1, the information providing device 10 uses the log data of the user # 1 to determine the day of the week or the time of day when the user # 1 is likely to access a predetermined website. 1 estimates the day of the week, the time zone, and the like that are likely to select content. For such estimation, various prediction processes and well-known pattern analysis techniques can be applied. Then, the information providing apparatus 10 calculates the possibility that the user # 1 accesses a predetermined website during a specific period and selects content (for example, “90%” in the example shown in FIG. 1). .

  Subsequently, the information providing apparatus 10 excludes a user who accesses a predetermined website during the specific period and selects a content that is lower than a predetermined threshold among the users who have accessed the specific period. The log data of the excluded user is excluded from the log data (step S4). For example, in the example illustrated in FIG. 1, for user # 1 to user # 3, the possibility of accessing a predetermined website and selecting content is “90%”, “20%”, and “80%”. Is calculated. In such a case, the information providing apparatus 10 extracts the user # 2 whose calculated possibility is lower than a predetermined threshold (for example, 70%) as an excluded user. Then, the information providing apparatus 10 excludes the log data # 2 of the user # 2 from the learning data among the log data # 1 to the log data # 3 of the user # 1 to the user # 3. More specifically, the information providing apparatus 10 determines the content distribution history and user for the user # 2 from the distribution history and selection history of the content to be determined, that is, the content to be learned by the prediction model. Learning data excluding the selection history of # 2 is generated.

  And the information provision apparatus 10 produces | generates the prediction model which learned the click tendency using learning data (step S5). For example, the information providing apparatus 10 uses the feature information indicating the features of the log data # 1 of the user # 1, the log data # 3 of the user # 3, and the content # 1, and how the content having a certain feature is processed. A prediction model in which a user having a specific attribute has selected, that is, a click tendency is generated is generated.

  Thereafter, the information providing apparatus 10 transmits the generated prediction model to the distribution server 200 (step S6). As a result, the information providing apparatus 10 can perform content distribution using the generated prediction model, that is, listing distribution using the generated prediction model (step S7).

[1-6. (Examples of excluded users)
Here, the information providing apparatus 10 generates the prediction model using the learning data excluding the log data of the excluded user. However, the embodiment is not limited to this. For example, the information providing apparatus 10 may learn a feature of an inappropriate user as a learning data acquisition target by learning the feature of an excluded user. And the information provision apparatus 10 may produce | generate the learning data suitable for the production | generation of a prediction model by excluding the information of the user who has the learned characteristic from log data.

  For example, FIG. 2 is a diagram illustrating an example of processing in which the information providing apparatus according to the embodiment learns the characteristics of an excluded user. For example, the information providing apparatus 10 illustrated in FIG. 2 acquires various log data related to predetermined content (step S1). In such a case, the information providing apparatus 10 extracts a user whose possibility of access and click is lower than a predetermined threshold among users in a specific period (step S2). For example, the information providing apparatus 10 extracts the user # 2.

  And the information provision apparatus 10 produces | generates the feature model which learned the feature of the specified user. For example, the information providing apparatus 10 causes the feature model to learn the number of times that the user # 2 has accessed a predetermined website as a feature of the user # 2. When such a feature is learned in the feature model, the information providing apparatus 10 can generate a specific model that estimates another user who has a similar number of accesses to the user # 2 and a predetermined website. . For example, when the user # 2 is a user who unjustly lowers the traveling bot or CTR, it is considered that the user frequently accesses a predetermined website. Therefore, the information providing apparatus 10 can generate a feature model that can estimate another user who has a similar number of accesses to the user # 2 by causing the feature model to learn the number of accesses of the user # 2. .

  In addition, the information providing apparatus 10 learns the co-occurrence between the search query whose attention degree has rapidly increased as a feature of the user # 2, that is, the occurrence of a buzzword and the access timing (step S3). For example, when a buzzword occurs, the number of accesses to related websites generally increases. However, a user who accesses only when such a buzzword occurs is considered to have a low content selection rate. Therefore, the information providing apparatus 10 can estimate other users whose access timing is similar to that of the user # 2 by causing the feature model to learn the co-occurrence of the access timing and buzzword of the user # 2. A model can be generated.

  Moreover, the information provision apparatus 10 is made to learn whether various push notifications regarding a predetermined website are set as a feature of the user # 2. For example, a user who has set various push notifications related to a predetermined website is a user who intentionally accesses the predetermined website, and is considered to be a user who is more suitable for learning the prediction model. . Therefore, the information providing apparatus 10 can estimate a user who accesses with the same intention as the user # 2 by causing the feature model to learn whether or not various push notifications related to a predetermined website are set. A unique feature model can be generated.

  And the information provision apparatus 10 excludes the log data of the user whose feature is similar to the excluded user using the feature model (step S4). For example, the information providing apparatus 10 uses the feature model to determine whether the features of the user # 1, the user # 3, and the user # 4 are similar to the features of the user # 2 who is an excluded user. To do. Then, for example, when the characteristics of the user # 2 and the characteristics of the user # 4 are similar, the information providing apparatus 10 logs the log data # 2 of the user # 2 who is an excluded user, the user # 2, Log data # 4 of user # 4 having similar characteristics is excluded from the log data.

  And the information provision apparatus 10 produces | generates the prediction model which learned the click tendency using log data (step S5). Thereafter, the information providing apparatus 10 transmits the generated prediction model to the distribution server 200 (step S6). As a result, the information providing apparatus 10 can perform content distribution using the generated prediction model, that is, listing distribution using the generated prediction model (step S7).

  Note that the information providing apparatus 10 may learn an arbitrary feature as long as it is a feature related to the user # 2. For example, the information providing apparatus 10 may cause the feature model to learn the access pattern, click pattern, demographic attribute, psychographic attribute, IP (Internet Protocol) address, etc. of the user # 2 as the attribute of the user # 2. . That is, the information providing apparatus 10 specifies a user whose access mode to a predetermined website is presumed that the content selection mode is similar to the user # 2 or that various attributes are similar to the user # 2. If the model is learned, any feature may be learned based on any information.

[2. Configuration of information providing device]
Hereinafter, an example of the functional configuration of the information providing apparatus 10 will be described. FIG. 3 is a diagram illustrating a configuration example of the information providing apparatus according to the embodiment. As illustrated in FIG. 3, the information providing apparatus 10 includes a communication unit 20, a storage unit 30, and a control unit 40.

  The communication unit 20 is realized by, for example, a NIC (Network Interface Card). The communication unit 20 is connected to the network N in a wired or wireless manner, and transmits / receives information to / from the log server 100 and the distribution server 200.

  The storage unit 30 is realized by, for example, a semiconductor memory device such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 30 stores a log database 31 and a user stores a database 32.

  In the log database 31, log data related to distribution of various contents is registered. For example, FIG. 4 is a diagram illustrating an example of information registered in the log database according to the embodiment. In the example illustrated in FIG. 4, information such as “content ID (Identifier)”, “feature information”, “distribution destination”, “distribution date / time”, “selection history”, “selection date / time”, and the like is registered in the log database 31. Has been.

  Here, “content ID” is an identifier for identifying content. The “feature information” is a feature of the appearance or content of the content indicated by the associated “content ID”, such as a vector or a distributed expression indicating the feature of the content indicated by the “content ID”. It is a multidimensional quantity. The “distribution destination” is a user ID for identifying the user who is the distribution destination of the content indicated by the associated “content ID”. The “distribution date / time” is information indicating the date / time when the content indicated by the associated “content ID” is delivered to the user indicated by the associated “delivery destination”. The “selection history” is information indicating whether or not the content indicated by the associated “content ID” has been selected by the user indicated by the associated “distribution destination”. The “selection date / time” is information indicating the date / time when the content indicated by the associated “content ID” is selected by the user indicated by the associated “distribution destination”.

  For example, in the example shown in FIG. 4, the log database 31 includes a content ID “content # 1”, feature information “feature # 1”, a distribution destination “user # 1”, a distribution date and time “distribution date and time # 1”, and a selection. The history “selected” and the selected date “selected date # 1” are registered in association with each other. Such information is, for example, the content indicated by the content ID “content # 1” having the characteristics indicated by the feature information “feature # 1”, and the information indicated by the distribution destination “user # 1”. The date and time of distribution “delivery date and time # 1” and “selected” by the user indicated by the distribution destination “user # 1” and the date and time “selected” are indicated by the selection date and time “selected date and time # 1”. Indicates the date and time.

  In the example illustrated in FIG. 4, conceptual values such as “content # 1”, “feature # 1”, “user # 1”, “delivery date / time # 1”, “selection date / time # 1”, and the like are described. Actually, in the log database 31, a character string for identifying the content and the user, a numerical value for indicating the delivery date and time, a multidimensional quantity indicating the feature, and the like are registered. In addition to the information shown in FIG. 4, the log database 31 stores various types of information related to content distribution, such as information indicating a website accessed by the user when the content is distributed. Also good.

  Returning to FIG. 2, the description will be continued. Various types of information related to users are registered in the user database 32. For example, FIG. 5 is a diagram illustrating an example of information registered in the user database according to the embodiment. In the example illustrated in FIG. 5, information such as “user ID”, “user characteristic information”, “access destination”, “access date”, “distributed content”, and the like is registered in the user database 32.

  Here, the “user ID” is an identifier for identifying a user. The “user characteristic information” is characteristics of various users used for content listing such as demographic attributes and psychographic attributes of users. The “access destination” is information indicating an access destination such as a website accessed by the user indicated by the associated “user ID”. The “access date / time” is information indicating the date / time when the user indicated by the associated “user ID” accesses the website indicated by the “access destination” associated with the user. The “distributed content” is the content ID of the content distributed when the user indicated by the associated “user ID” accesses the website indicated by the “access destination” associated with the user. .

  For example, in the example illustrated in FIG. 5, the user database 32 includes a user ID “user # 1”, user feature information “user feature # 1”, access destination “access destination # 1”, access date “ Access date / time # 1 ”and distribution content“ content # 1 ”are registered in association with each other. In such information, the user indicated by the user ID “user # 1” has the characteristics indicated by the user characteristic information “user characteristic # 1”, and the access destination “access destination # 1” The website indicated by the user accesses the date indicated by the access date “access date # 1” to indicate that the content indicated by the distribution content “content # 1” has been distributed.

  In the example illustrated in FIG. 5, conceptual values such as “access destination # 1”, “access date / time # 1”, “user feature # 1”, and the like are described. The URL (Uniform Resource Locator) of the access destination, the numerical value indicating the date and time of access, various information indicating the characteristics, and the like are registered. Further, in the user database 32, not only the access history of various users and the history of distributed contents but also various history information such as search query history and browsing history of various web contents are registered. May be.

  Returning to FIG. 3, the description will be continued. The control unit 40 is a controller. For example, various programs stored in a storage device inside the information providing apparatus 10 are stored in a RAM or the like by a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). Is implemented as a work area. The control unit 40 is a controller, and may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

  As illustrated in FIG. 3, the control unit 40 includes an acquisition unit 41, a specification unit 42, an extraction unit 43, a feature learning unit 44, a generation unit 45, and a tendency learning unit 46. The acquisition unit 41 acquires various log data including the distribution history and selection history of each content from the log server 100. In such a case, the acquisition unit 41 specifies a distribution destination, a distribution date / time, a selection history, a selection date / time, and the like for each content using a known analysis technique. Then, the acquisition unit 41 registers the identified information in the log database 31 in association with each other. In addition, the acquisition unit 41 specifies an access destination, an access date and time, and a distribution content distributed as a result of access for each user, and registers the specified information in the user database 32 in association with each other.

  The specifying unit 42 specifies a specific period during which the content selection ratio decreases. More specifically, the specifying unit 42 determines, as the selection ratio, a period in which the ratio of the number of selections with respect to the number of content distributions decreases beyond a predetermined threshold, that is, a period in which the CTR value decreases beyond a predetermined value. Is identified. Further, the specifying unit 42 specifies a period in which the number of distributions of content per unit time increases and the content selection ratio decreases.

  For example, the specifying unit 42 refers to the log database 31 and calculates the determination target, that is, the number of distributions and the number of selections of content to be learned at predetermined time intervals. Further, the specifying unit 42 calculates the value of CTR at every predetermined time interval from the calculated distribution count and selection count. And the specific | specification part 42 specifies the period when the value of CTR is falling, although the frequency | count of delivery is rising.

  For example, FIG. 6 is a diagram illustrating an example of a process in which the information providing apparatus according to the embodiment specifies a specific period. For example, in the example shown in FIG. 6A, the CTR decreases because the number of selections does not increase even though the number of content distributions increases. In such a case, the specifying unit 42 specifies a predetermined period as the specific period CP from the date and time when the CTR decreases. The specifying unit 42 may specify, for example, the date and time when a predetermined margin is taken into account at the date and time when the CTR reduction rate exceeds a predetermined threshold as the start point of the specific period. Further, the specifying unit 42 specifies the date and time when the rate of increase in the number of distributions becomes equal to or less than a predetermined threshold, the date and time when the information about the number of distributions stopped, the date and time when a predetermined period has elapsed from the viewpoint as the end point of the specific period. Also good.

  On the other hand, in the example shown in FIG. 6B, the CTR takes a substantially constant value because the number of selections increases with an increase in the number of content distributions. In such a case, since it is considered that each accessed use appropriately selects an advertisement, the specifying unit 42 does not specify a specific period.

  Further, in the example shown in FIG. 6C, there is a spike that returns to the original level after the number of content distributions rapidly increases. On the other hand, the number of times content is selected has also increased, but since the rate of increase is lower than the rate of increase in the number of distributions, the value of CTR has returned to the original level after rapidly decreasing. Even in such a case, it is considered that a user who has not properly selected the advertisement is included, and therefore the specifying unit 42 specifies the period during which the CTR has decreased as the specific period CP. For example, the specifying unit 42 may specify a period during which a spike in the number of distributions occurs, that is, a period until the CTR suddenly decreases and then returns to the original period as the specific period CP.

  On the other hand, in the example shown in FIG. 6D, a spike occurs in the number of content distributions, but the number of selections increases and decreases as the number of content distributions increases and decreases. Take a certain value. In such a case, since it is considered that each accessed use appropriately selects an advertisement, the specifying unit 42 does not specify a specific period.

  Returning to FIG. 3, the description will be continued. The extraction unit 43 extracts a user who satisfies a predetermined condition from the users to whom the content to be determined is distributed during the specific period. More specifically, as the predetermined condition, the extraction unit 43 excludes users who are unlikely to distribute content during a specific period or users who are unlikely to select content during a specific period as excluded users. Extract.

  For example, FIG. 7 is a diagram illustrating a process in which the information providing apparatus according to the embodiment extracts excluded users. For example, the extraction unit 43 refers to the user database 32 and identifies the user to whom the content to be determined is distributed in the specific period specified by the specifying unit 42. Subsequently, the extraction unit 43 acquires the access date / time and the selection date / time of the specified user from the log database 31 and the user database 32. In addition, the extraction unit 43 selects a time period for each day of the week from which the user is highly likely to access a predetermined website or select content, as shown in FIG. To calculate. For example, in the example shown in FIG. 7, an arrow indicates a time zone in which a user accesses a predetermined website and is likely to select content.

  Then, the extracting unit 43 determines whether or not a time period in which the user is likely to access or select content overlaps with the specific period. For example, in the example illustrated in FIG. 7, a period during which user # 1 is likely to access a predetermined website or select content overlaps with a specific period. In such a case, the user # 1 has a high possibility of accessing a website during a specific period and selecting content. Therefore, the extracting unit 43 employs the log of the user # 1 as learning data without extracting the user # 1 as an excluded user.

  On the other hand, in the example illustrated in FIG. 7, a period during which user # 2 is highly likely to access a predetermined website or select content does not overlap with the specific period. In such a case, it is considered that the user # 2 has accessed the website irregularly during a specific period. Therefore, the extraction unit 43 extracts the user # 2 as an excluded user.

  For a user who does not select content, there is no time period when the user is likely to access a predetermined website and select content. For this reason, the extraction unit 43 accesses a website and calculates a time zone in which there is a high possibility of selecting content, so that a user who wants to lower a traveling bot or CTR, or a user who does not select content. Can be appropriately extracted as an excluded user.

  Returning to FIG. 2, the description will be continued. The feature learning unit 44 learns the features of the users extracted as excluded users. For example, the feature learning unit 44 may determine, as the characteristics of the excluded user, the number of accesses to a predetermined access destination in a predetermined period, the co-occurrence of information determined to be noticed in the predetermined period and the number of accesses, or It learns at least one of whether or not the setting to push the predetermined information to the excluded user is made.

  For example, the feature learning unit 44 identifies content distribution history and selection history for excluded users, the number of accesses to a predetermined access destination in a predetermined period, user characteristic information, and the like from the log database 31 and the user database 32. In addition, the feature learning unit 44 identifies a period in which a so-called buzzword has occurred from various external servers not shown, and whether or not the excluded user has accessed a predetermined website during the identified period. Etc. are specified. Further, the feature learning unit 44 specifies whether or not the excluded user has set the push notification from an external server that provides various services via a predetermined website.

  And the feature learning part 44 produces | generates the feature model which learned the characteristic of the exclusion user by learning the various information specified as a feature. Such a feature model is realized by various arbitrary classifiers and learners such as DNN (Deep Neural Network) and SVM (Support Vector Machine), and whether the features of any user are similar to those of excluded users. It can be determined whether or not.

  The generation unit 45 generates information excluding the distribution history and selection history for the excluded user extracted from the distribution history and selection history of the content to be determined. For example, the generation unit 45 specifies from the log database 31 the distribution history and selection history of the content to be determined, that is, the content to be learned about the tendency of the user to select. Subsequently, the generation unit 45 learns the information obtained by deleting the distribution history indicating that the content is distributed to the excluded user and the selection history indicating that the content is selected by the excluded user from the content distribution history and the selection history. Generate as

  Further, the generation unit 45 generates information excluding the distribution history and selection history for the user having the characteristics learned by the feature model from the distribution history and selection history of the content. For example, the generation unit 45 uses the feature model learned by the feature learning unit 44 to specify a user whose feature is similar to that of the excluded user. As a more specific example, the generation unit 45 may share the attributes of each user, the number of accesses to a predetermined access destination in a predetermined period, the information determined to be noticed in the predetermined period, and the number of accesses. Various types of information learned as features in the feature model are acquired, such as whether or not the user is set to push notification of predetermined information to the excluded user. And the production | generation part 45 determines whether the feature of each user is similar to the feature of an excluded user using a feature model. Thereafter, the generation unit 45 generates, as learning data, information excluding the distribution history and selection history regarding the user whose characteristics are determined to be similar to the characteristics of the excluded user from the distribution history and selection history of the content.

  The tendency learning unit 46 learns the tendency of the user who selects the content using the generated learning data. More specifically, the trend learning unit 46 uses the learning data generated by the generation unit 45 to acquire the feature of the content to be learned from the log database 31. In addition, the trend learning unit 46 identifies the user who has selected the distributed content and the user who has not selected the distributed content from the learning data generated by the generation unit, and the characteristics of the specified user Is read from the user database 32.

  Then, the tendency learning unit 46 generates a prediction model in which the relationship between the feature of the content and the feature of the user who has selected the content is learned. For example, the tendency learning unit 46 uses the feature of the user who selected the content as a positive example and the feature of the user who did not select the content as a negative example, and the relationship between the feature of the content and the feature of each user By making the predictive model learn the sex, a predictive model in which the tendency of the characteristics of the user who selects the content is learned is generated. Then, the trend learning unit 46 provides the generated prediction model to the distribution server 200.

[3. Example of flow of processing executed by information providing apparatus]
Next, the flow of extraction processing executed by the information providing apparatus 10 will be described with reference to FIG. FIG. 8 is a flowchart illustrating an example of a flow of extraction processing executed by the information providing apparatus according to the embodiment.

  For example, the information providing apparatus 10 acquires various log data (step S101). And the information provision apparatus 10 specifies the specific period when CTR is falling based on the frequency | count of delivery and selection of the content used as determination object (step S102). Subsequently, the information providing apparatus 10 is a period in which there is a high possibility of accessing a predetermined site or selecting content (hereinafter referred to as “accessibility”) from a log of a user who has accessed during a specific period. A pattern is predicted (step S103).

  Then, the information providing apparatus 10 extracts a user whose access possibility is lower than a predetermined threshold during the specific period as an excluded user (step S104). Subsequently, the information providing apparatus 10 generates a feature model in which the features of the excluded user are learned (step S105), excludes data corresponding to the excluded user from the log data (step S106), and extracts the feature model. In step S107, log data of a user whose characteristics are similar to those of the excluded user are excluded. Then, the information providing apparatus 10 generates a prediction model using the learning data obtained by excluding the log data of the excluded user or the user having similar characteristics to the excluded user from the log data (Step S108), and ends the processing. To do.

[4. (Modification)
In the above, an example of the extraction process by the information providing apparatus 10 has been described. However, the embodiment is not limited to this. Hereinafter, the variation of the extraction process which the information provision apparatus 10 performs is demonstrated.

[4-1. About generation of learning data)
In the example described above, the information providing apparatus 10 generates learning data that excludes various histories related to excluded users and various histories related to users whose characteristics are similar to those of excluded users from the distribution history and selection history of content. . However, the embodiment is not limited to this. For example, the information providing apparatus 10 may generate learning data excluding various histories related to the excluded user from the content distribution history and the selection history, and generate the prediction model using the generated learning data. Further, the information providing apparatus 10 generates learning data excluding various histories related to users whose characteristics are similar to those of the excluded user from the distribution history and selection history of the content, and uses the generated learning data to generate a prediction model. Generation may be performed.

  That is, the information providing apparatus 10 may generate learning data that excludes various histories of the excluded user, and uses a feature model that has learned the characteristics of the excluded user, and has characteristics similar to those of the excluded user. Learning data excluding various histories related to users may be generated.

[4-2. (Specification of specific period)
In the example described above, the information providing apparatus 10 selects the period during which the CTR decreases as the number of content distributions increases as the specific period. However, the embodiment is not limited to this. For example, the information providing apparatus 10 may simply specify the period during which the CTR decreases as the specific period. In addition, if the information providing apparatus 10 selects a specific period based on the timing when the CTR decreases, the information providing apparatus 10 includes a specific period including a period when the CTR increases or a period when the CTR decreases and then returns. You may specify.

[4-3. Device configuration〕
The information providing apparatus 10 may be realized by a front end server and a back end server. In such a case, the acquisition unit 41, the identification unit 42, and the extraction unit 43 illustrated in FIG. 3 are arranged in the front-end server, and the feature learning unit 44, the generation unit 45, and the trend learning unit 46 are arranged in the back-end server. Is placed. The databases 31 and 32 registered in the storage unit 30 may be held in an external storage server.

[4-4. Others]
In addition, among the processes described in the above embodiment, all or part of the processes described as being automatically performed can be performed manually, and conversely, the processes described as being performed manually. All or a part of the above can be automatically performed by a known method. In addition, the processing procedures, specific names, and information including various data and parameters shown in the document and drawings can be arbitrarily changed unless otherwise specified. For example, the various types of information illustrated in each drawing is not limited to the illustrated information.

  Further, each component of each illustrated apparatus is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution / integration of each device is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed or arbitrarily distributed in arbitrary units according to various loads or usage conditions. Can be integrated and configured.

  In addition, the above-described embodiments can be appropriately combined within a range in which processing contents do not contradict each other.

[4-5. program〕
Further, the information providing apparatus 10 according to the above-described embodiment is realized by a computer 1000 having a configuration as shown in FIG. 9, for example. FIG. 9 is a diagram illustrating an example of a hardware configuration. The computer 1000 is connected to an output device 1010 and an input device 1020, and an arithmetic device 1030, a primary storage device 1040, a secondary storage device 1050, an output IF (Interface) 1060, an input IF 1070, and a network IF 1080 are connected via a bus 1090. Have

  The arithmetic device 1030 operates based on a program stored in the primary storage device 1040 and the secondary storage device 1050, a program read from the input device 1020, and the like, and executes various processes. The primary storage device 1040 is a memory device such as a RAM that temporarily stores data used by the arithmetic device 1030 for various arithmetic operations. The secondary storage device 1050 is a storage device in which data used by the arithmetic device 1030 for various calculations and various databases are registered, such as ROM (Read Only Memory), HDD (Hard Disk Drive), flash memory, and the like. It is realized by.

  The output IF 1060 is an interface for transmitting information to be output to an output device 1010 that outputs various types of information such as a monitor and a printer. For example, USB (Universal Serial Bus), DVI (Digital Visual Interface), This is realized by a standard connector such as HDMI (registered trademark) (High Definition Multimedia Interface). The input IF 1070 is an interface for receiving information from various input devices 1020 such as a mouse, a keyboard, and a scanner, and is realized by, for example, a USB.

  The input device 1020 includes, for example, an optical recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), and a PD (Phase change rewritable disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), and a tape. It may be a device that reads information from a medium, a magnetic recording medium, a semiconductor memory, or the like. The input device 1020 may be an external storage medium such as a USB memory.

  The network IF 1080 receives data from other devices via the network N and sends the data to the arithmetic device 1030, and transmits data generated by the arithmetic device 1030 to other devices via the network N.

  The arithmetic device 1030 controls the output device 1010 and the input device 1020 via the output IF 1060 and the input IF 1070. For example, the arithmetic device 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040, and executes the loaded program.

  For example, when the computer 1000 functions as the information providing device 10, the arithmetic device 1030 of the computer 1000 implements the function of the control unit 40 by executing a program loaded on the primary storage device 1040.

[5. effect〕
As described above, the information providing apparatus 10 specifies a period during which the content selection ratio decreases. And the information provision apparatus 10 extracts the user who satisfy | fills predetermined conditions from the users to whom the content was delivered in the specific period. For this reason, the information providing apparatus 10 can efficiently extract users who do not select content, patrol bots, and users who want to lower the CTR, so that it is possible to extract a history user that is not suitable for prediction model learning. it can.

  In addition, the information providing apparatus 10 specifies a period during which the ratio of the number of selections with respect to the number of distributions of content decreases beyond a predetermined threshold as the selection ratio. For example, the information providing apparatus 10 specifies a period in which the number of distributions of content per unit time increases and the content selection ratio decreases. For this reason, the information provision apparatus 10 can specify the period when many users of the history unsuitable for learning the prediction model are accessing as the specific period.

  Moreover, the information provision apparatus 10 extracts a user with low possibility that a content will be delivered in the specified period as predetermined conditions. Moreover, the information provision apparatus 10 extracts a user with low possibility of selecting a content in the specified period as predetermined conditions. For this reason, the information providing apparatus 10 can appropriately extract a history user who is not suitable for the prediction model learning.

  Further, the information providing apparatus 10 generates information excluding the distribution history and selection history for the extracted user from the content distribution history and selection history. For this reason, the information provision apparatus 10 can produce | generate the learning data which can learn a prediction model accurately.

  Further, the information providing apparatus 10 learns the extracted user characteristics, and generates information excluding the distribution history and the selection history for the user having the learned characteristics from the content distribution history and the selection history. For this reason, since the information providing apparatus 10 can estimate whether or not a user who does not have much history is suitable for learning the prediction model, it is possible to generate learning data that can accurately learn the prediction model.

  Further, the information providing apparatus 10 is characterized by the number of accesses to a predetermined access destination in a predetermined period, the co-occurrence of information determined to be noticed in the predetermined period and the number of accesses, Learning at least one of whether or not the user is set to push the predetermined information. For this reason, the information provision apparatus 10 can produce | generate the learning data which can learn a prediction model accurately.

  Moreover, the information provision apparatus 10 learns the tendency of the user who selects content using the produced | generated information. For this reason, the information provision apparatus 10 can improve the prediction accuracy of a prediction model.

  As described above, some of the embodiments of the present application have been described in detail with reference to the drawings. However, these are merely examples, and various modifications, including the aspects described in the disclosure section of the invention, based on the knowledge of those skilled in the art, It is possible to implement the present invention in other forms with improvements.

  Moreover, the above-mentioned “section (module, unit)” can be read as “means”, “circuit”, and the like. For example, the providing unit can be read as providing means or a providing circuit.

DESCRIPTION OF SYMBOLS 10 Information provision apparatus 20 Communication part 30 Storage part 31 Log database 32 User database 40 Control part 41 Acquisition part 42 Identification part 43 Extraction part 44 Feature learning part 45 Generation part 46 Trend learning part 100 Log server 200 Distribution server

Claims (11)

A specific part that identifies the period during which the content selection rate decreases,
An extraction device comprising: an extraction unit that extracts a user who satisfies a predetermined condition from among users to whom the content has been distributed during the period specified by the specifying unit. The extraction device according to claim 1, wherein the specifying unit specifies, as the selection ratio, a period during which a ratio of the number of selections with respect to the number of distributions of the content decreases beyond a predetermined threshold. The extraction device according to claim 1 or 2, wherein the specifying unit specifies a period in which the number of distributions of the content per unit time increases and a selection ratio of the content decreases. The said extraction part extracts the user with low possibility that the said content will be delivered in the specified period as said predetermined conditions. The Claim 1 characterized by the above-mentioned. Extraction device. The extraction unit according to any one of claims 1 to 4, wherein the extraction unit extracts, as the predetermined condition, a user who is unlikely to select the content during a specified period. apparatus. 6. The generator according to claim 1, further comprising: a generator that generates information excluding the distribution history and the selection history for the user extracted by the extraction unit from the distribution history and the selection history of the content. The extraction device according to one. A feature learning unit for learning the features of the user extracted by the extraction unit;
6. A generation unit configured to generate information excluding a distribution history and a selection history for a user having a feature learned by the feature learning unit from the distribution history and selection history of the content. The extraction device according to any one of the above. The feature learning unit, as the feature of the user, the number of accesses to a predetermined access destination in a predetermined period, the co-occurrence of information determined to be noticed in a predetermined period and the number of accesses, or the use The extraction device according to claim 7, wherein at least one of whether or not the user is set to push notification of predetermined information is learned. The extraction device according to any one of claims 6 to 8, further comprising a tendency learning unit that learns a tendency of a user who selects the content using information generated by the generation unit. . An extraction method executed by an extraction device,
A specific process for identifying the time period during which the percentage of content selection will decrease,
An extraction step of extracting a user who satisfies a predetermined condition from users to whom the content has been distributed in the period specified in the specifying step. A specific procedure for identifying the time period during which the percentage of content selection will decrease
An extraction program for causing a computer to execute an extraction procedure for extracting a user who satisfies a predetermined condition from users to whom the content has been distributed during the period specified by the specific procedure.

Images