JP2012203663A - Evaluation method of recommendation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To evaluate a method of recommending items such as a community or merchandise from the past action record of a user so as to compare the result and method itself.SOLUTION: For data lowly related to past attribute data, after the lapse of a fixed period, only data for which a user has actually taken action to the attribute data is handled as an unexpected object. Then, of all the recommended attribute data, a ratio occupied by new objects and a ratio occupied by the unexpected objects are obtained.

Description

  The present invention relates to a method for evaluating a method itself for various methods for selecting and recommending attribute data according to hobbies and preferences to a user.

  In the operation and marketing of social network services, the past behavior of users is analyzed, and a community related to the behavior is recommended, and products related to products purchased in the past are recommended. . If it is a social network service, the community or personal data accessed or registered by one user will be accessed or registered by another user who has accessed or registered the same community or personal data. And personal data are displayed on a screen used by the one user so as to be recommended to the one user. In marketing, for one user who purchased a product, another product purchased by another user who purchased the same product is displayed in the recommendation column of the marketing site in order to recommend it to the one user. Or listed in the recommendation column of an e-mail magazine.

  Such a recommendation method has a great effect on the expansion of the community in social network services and marketing, but if you continue to select and present a particularly large number of users with similar hobbies and preferences The tendency of the community and the product to be presented is biased, and there is a limit to the expansion of the community and the sales of the product, and it tends to stay within a certain range. If this happens, the community and products that can be presented will not be fresh to the user, and it will not be possible to provide a surprising experience, which in turn will reduce interest in the service. For this reason, it is not only recommended that users with similar hobbies and preferences have the highest number of cases to be acted on, but also recommends those with a high degree of similarity. There is a case where a recommendation method is recommended for recommending an unexpected object with overlapping.

  For example, the method described in Patent Literature 1 learns and stores a user's preference model based on a user's behavior record, calculates a selection probability based on the preference model for a plurality of recommended candidates, Create an evaluation list representing the probability that the user selects for each recommendation candidate, identify the user's habit from the information about the user's behavior record, and obtain a default prediction value for the plurality of recommendation candidates, The difference obtained by subtracting the default prediction value from the selection probability is obtained as an unexpected evaluation value, thereby evaluating the unexpectedness of the content and presenting highly unexpected content. In other words, data that deviates from the threshold value is presented as being unexpected without being discarded.

  On the other hand, there are cases where it is difficult to recommend only those that are surprising. For example, there is a case where rapid expansion of the community is not required, or when it is desired to continue stable sales to stable customers.

  Therefore, the recommendation method does not have the best solution, and it is necessary to consider various recommendation methods. Even with one recommendation method, the recommendation result may change greatly by changing the parameter.

JP 2008-262398 A

  However, it is difficult to easily grasp what kind of properties these recommendation methods have, and as a result, it is difficult to provide an index as to what recommendation method should be used.

  In addition, what is presented as being unexpected by the method of Patent Document 1 is merely presented as an unexpected thing that is less relevant. As a result, while some of the contents are unexpected and greeted with surprise, even content that is almost irrelevant and should not be of interest is classified as surprising and far from the substance. There was a problem of giving results.

  Therefore, the present invention compares the properties of various recommendation methods by evaluating the candidates presented by the recommendation method and the properties of the recommendation method based on the unexpectedness in accordance with the substance. The purpose is to make it possible to study the effects of parameters to be substituted.

  The present invention recommends new attribute data for a user as an evaluation of unexpectedness in accordance with the actual situation, and data that is less relevant to past attribute data is actually used before a certain period of time. Only those that have acted on the attribute data are treated as unexpected objects, and then all to one user, to one group, or to all users In the recommended attribute data, the ratio of the novel object and the ratio of the unexpected object were obtained, and the above problem was solved by the numerical values of these ratios. That is, it is characterized by not only collecting past attribute data but also collecting data after a certain period of time and determining whether or not it should be an unexpected object. This is because even if the recommendation is made, if the user does not show an action result for the recommendation, there is no point in recommending it.

  If this numerical value is calculated for the recommended content presented to one user, it can be used as an evaluation of novelty and unexpectedness of the presented content. In addition, after calculating the same numerical value for multiple users, it is recommended to calculate the statistical value such as mean, median, distribution, etc., and plot it on a graph. It is possible to grasp the tendency of the recommendation method and the evaluation of the results presented to the plurality of users by the method. Furthermore, for the same user, the above numerical values by different recommendation methods are calculated, an average value is obtained for each recommendation method, and the values for each recommendation method are compared. It is possible to grasp the nature, advantages, appropriateness, etc. of the recommendation method.

Schematic diagram of an example of how to use the recommendation method Conceptual diagram showing the timing of action recording Example of recommended community Conceptual diagram of similarity of recommended community shown to user 1 Process flow diagram of the first computing unit that calculates similarity Conceptual diagram of novelty and unexpected classification based on similarity Process flow diagram of the entire evaluation method Process flow diagram of the second calculation unit for calculating similarity

  The present invention will be described in detail below. The present invention provides a result of recommending attribute data having different relevance to a user based on action records performed by the user in the past by one or a plurality of recommendation methods or the recommendation method itself. It is a method for evaluating the properties of As an environment for executing this evaluation, there are mainly recommendations and evaluations of user behaviors on one or a plurality of WEB sites, but not limited to events on the network, the purchase of registered users at actual stores It can also be applied to real space events such as marketing using history.

  Here, the attribute data is data having physical or conceptual attributes reflecting a person's hobbies, preferences, personality, and the like, and has some relationship with other data. It may also include what is called an item or content. Specifically, there is no particular limitation as long as hobbies and preferences can be reflected, such as selection of merchandise, hobbies, communities, search words, input words, stopover locations in location information services, travel destinations, and travel routes.

  A specific example is given. For example, it is assumed that “basketball”, “basket shoes”, and “marathon shoes” are attribute data for products at a marketing site or a sporting goods store. Users who have purchased basketball are more likely to be interested in basketball shoes, but less likely to be interested in marathon shoes. “Basketball” and “basket shoes” are highly related, but “basketball” and “marathon shoes” are less related.

  Further, for example, a community in a social network service, such as “baseball”, “soccer”, “ice hockey”, “gardening”, and “real giant robot anime work”, is taken as an example of attribute data. Users who belong to both “baseball” and “soccer” communities are more likely to be interested in sports overall, and more likely to be interested in the “ice hockey” community. The possibility of being interested in communities such as “gardening” and “real giant robot anime works” is low. However, compared to “gardening”, which tends to be of interest to members of society who have single-family homes, “real-type giant robot anime works” that are easy to like for men in their teens and 20s who enjoy sports in general are somewhat related. Increases nature. For this reason, when “baseball” is used as a reference, the order from “relevant” to “soccer”, “ice hockey”, “real-type giant robot anime work”, “gardening” is the order. Note that the social network service (SNS) is a service that provides means for establishing an interpersonal relationship. A community is a function of the SNS. Users with the same direction who are interested in or agree with a certain event can be grouped by tagging or registering in a table. And can have a dedicated place where users can interact with each other.

  Further, for example, in the news site, keywords and tags included in news documents such as “Girls idol group (1)”, “Girls idol group (2)”, “College exam” and “30s male idol group (3)” Is taken as an example of attribute data. Users watching news about “Girl Idol Group (1)” are likely to be interested in news about other “Girl Idol Group (2)”, but they are interested in news about “College Exam” It is unlikely that they have an interest in the “30's male idol group (3)”. The relationship between “Girl idol group (1)” and “Girl idol group (2)” is high, and the relationship between “Girl idol group (1)” and “University examination” is lower than that. The relationship between the “group (1)” and the “30s male idol group (3)” is further reduced. Here, the contents are “girl idol group (1)” and “30's male idol group (3)”, which are similar to those related to idol groups. Men in their 30s are interested, and the latter has a strong tendency to be interested in women in their 40s and the browsing layer is completely different. For this reason, for example, the teenage male group is likely to be interested in both the “girl idol group (1)” and the “university examination”, so the “girl idol group (1)” is the “30s male idol group (3 ) ”Is more similar to“ university exam ”.

  There are several methods for modeling the relevance of attribute data, and in general, content-based filtering and collaborative filtering are included. The former selects information based on the contents of each attribute data, and the latter selects what is commonly used by those having the same preference in a web site or network. There are mainly three types of content filtering methods: rule-based methods, memory-based methods (concept-based methods), and model-based methods. As a collaborative filtering method, there are mainly three types: a user-based method (memory-based method), an item-based method (memory-based method), and a model-based method. Since filtering based on content processes attribute data as keywords and tags, it is suitable for recommending articles on news sites and search sites. Collaborative filtering is based on so-called item units for attribute data itself. Suitable for product recommendations.

  The user's past action records used in the present invention are classified into explicit (direct) methods such as questionnaires and page evaluations, and implicit (indirect) methods such as access analysis and browsing time evaluation. There is. Although the explicit method has high accuracy, the burden on the user is large. The implicit method has a small load so that the user is not aware that the recommendation method is being evaluated.

  The recommendation method targeted by the present invention is to extract the above-mentioned attribute data from the user's past action record and treat it as the basic past attribute data. Data (recommendation attribute data) can be presented and recommended. This action record includes not only the access history of community pages, product pages, news pages, etc., but also a list of registered communities, product purchase history, search history, input word history, RSS registration items, and the like. Basically, past attribute data and recommended attribute data that can be recommended for presentation are common. For example, a registered community that can be recommended from the list of registered communities is a registered community that is proposed to register next, and another news page that can be recommended from the access history of a news page. However, the past attribute data and the recommended attribute data may be different as long as substantial contents can be associated. For example, it is possible to present an NBA team uniform advertisement as recommended attribute data to a user who is registered in an NBA (National Basketball Association) fan club as a registered community.

  In the present invention, as the above-mentioned similarity, if a value that can numerically compare the degree of relevance for the user between the respective attribute data can be calculated, the calculation standard and calculation method are set. There is no particular limitation. It can be assumed that the higher the similarity value is, the higher the relevance is, and a jump in interest occurs depending on whether the similarity value is greater than a certain threshold value. I just need to be able to classify. In general, the basic value is calculated by calculating the degree to which users accessing, registering, and purchasing the attribute data match for all individual attribute data. Is possible, but not particularly limited thereto.

  For example, as the attribute data, there is a method of dividing the number of common users by the total number of users as a method for obtaining a similarity value between communities in the SNS. There is a community X and a community Y, and during a certain period, X is accessed by five users A, B, C, D, and E, and Y has users A, C, D, F, and G If 5 people are accessing and there are 7 people in total, A, C, and D who are accessing in common, the calculation is 3/7 = 43%, and the degree of similarity Can be 43%.

  Also, as a method for obtaining the similarity of news documents in a news site, there is a method of extracting a keyword as attribute data from a news document and obtaining a distance value between word vectors with a sibling concept between word vectors and a superordinate concept. . This is because individual keywords that can be calculated after collecting a large number of news documents, collecting the concept vectors of words using a similar appearance pattern of words having similar concepts, and creating a database. The distance between them is used as the similarity.

  When there are a plurality of the past attribute data, the past attribute data is regarded as one group (group), and the similarity with other attribute data is uniquely determined. In this regard, an example of calculation will be shown along with the above community example. It is assumed that six users A, B, C, G, H, and I are accessing the community Z. Here, the similarity between the community group of X and Y (hereinafter referred to as XY group) and Z is calculated. Users who commonly access X and Y are A, B, C, D, E, F, and G. Therefore, if the similarity between the XY group and Z is calculated by the same method as described above, out of a total of 9 people (ABCDEFGHI), 4 people (ABCCG) are accessing in common, so 4 / 9≈ 44%.

  However, it is not surprising that the similarity is low. If the recommended attribute data with a low similarity to the past attribute data is presented to the user having the past attribute data on the action record, and the user takes a predetermined action on the recommended attribute data, It means that there is unexpectedness, and if you do not take action, it means that you were not surprised and not related. On the other hand, even if attribute data that has already been an object of action is included in the recommended attribute data, it is excluded from the following novelty object and unexpected object.

  The degree of similarity is obtained for the presented recommended attribute data. That is, among the recommended attribute data that is not included in the past attribute data, the similarity with the past attribute data group of the user to be compared is obtained for each attribute data. A thing whose similarity is equal to or greater than a threshold value is a novelty object, that is, a novel object that is not unexpected but has not yet been visited, unvisited, unregistered, or unseen. On the other hand, if this similarity is less than the threshold, if it becomes the target of appropriate actions such as user registration, access, purchase, etc. between the start of presentation and the elapse of a certain period of time, In other words, even if the relevance is low, it is surprising that the user is interested in it. If it is not subject to the user's action by a certain period of time, the latitude and longitude attribute data was not used. To do. This threshold value is also a parameter that can be evaluated according to the present invention, and the evaluation content of the recommendation method changes as the threshold value changes.

  Then, the novelty target ratio that is the ratio of the novelty target and the unexpected target ratio that is the ratio of the unexpected target are obtained from the total number of recommended attribute data. Thereby, the breakdown of the recommendation attribute data presented to the user can be evaluated.

  The novelty target rate and the unexpected target rate obtained in this way are obtained for a plurality of users, and numerical evaluation is performed on those values. The plurality of users may be all users or extracted users. Although the credibility of the evaluation improves as the number of target users increases, the analysis takes time. Here, the numerical evaluation includes calculation of statistical values, plotting on a graph, or both. Examples of the statistical value include a general average value, an intermediate value, a minimum value, a maximum value, a mode value, a variance, a standard deviation, and the like. Thereby, the general tendency of the data presented by the recommendation method can be grasped. For example, if the average value of the novelty target rate among the target users is high and the average value of the unexpected target rate is low, the recommended attribute data presented is recommended to be solid and highly similar I understand that. Conversely, if the average value of the unexpectedness target rate is higher, it can be seen that the user's usage tends to be dramatically increased. In addition, when the average value of the novelty target rate and the unexpected target rate is low, it can be understood that the recommendation method has a problem in usefulness. It can also be seen that if the width between the minimum value and the maximum value is large or the variance is large, the recommended content is uneven, and if it is the other way around, the unevenness is small and the aim is easy to determine.

  This kind of statistical value calculation is performed on recommended attribute data obtained by multiple recommendation methods, and the value of each recommendation method is obtained and compared with each other. Therefore, it is possible to determine which recommendation method should be adopted under what circumstances. Here, a plurality of recommendation methods can be considered as a plurality of recommendation methods as long as different results can be obtained depending on the parameter even if the algorithm is the same. If you want to recommend a steady expansion route, select a recommendation method with a high novelty target rate, and if you want to constantly expand interest and attract use, select a recommendation method with a high unexpected target rate. You can choose

  The evaluation method as described above will be described with specific examples. The object for which the recommendation method is adopted is the recommendation of the community on the SNS site. Here, the community is a virtual place or page that has a bulletin board function or a new notification function for gathering persons in the same section about matters in the service and can exchange topics about the matters. A schematic diagram of the usage pattern of the recommendation method is shown in FIG. When a user accesses a plurality of communities using the site (S101), an access log as an action record is accumulated. This log mainly has data on which community each user has accessed. This log is accumulated over a certain period (S102). Next, a plurality of recommendation methods to be evaluated are implemented using the accumulated action records as past attribute data (S103). Thereby, recommendation attribute data which is a list of communities recommended for each user is obtained (S104).

  This recommended attribute data is presented to the user for a certain period. Specifically, by continuing to display on the start page or the like when the user uses the SNS site (S101), the user is prompted to access the recommended community. This is continued for a certain period, and an access log is collected (S102). After the evaluation period is over, the novelty and the unexpectedness are evaluated based on the contents of the recommended and displayed communities and whether or not they have been accessed (S111). That is, in the present invention, the user's behavior is recorded over at least two periods as shown in FIG. One is for obtaining past attribute data for deriving recommended attribute data, and the other is for recording user behavior results for recommended attribute data for use in evaluating unexpectedness and novelty. is there.

  FIG. 3 shows an example of a recommended community. Users are grouped by the number of recommended methods, and one recommended method is executed for each group to create a list of recommended communities. Here, it is assumed that the concept-based technology, the collaborative filtering technology, and the random selection are executed for each group divided into 500 people.

  Taking the user 1 who is one of the users as an example, the calculation of the similarity will be described. This conceptual diagram is shown in FIG. 4, and the flow of processing is shown in FIG. First, it is assumed that the user 1 accesses a community of “baseball”, “soccer”, and “basketball” during a period of acquiring the first log (past attribute data acquisition period). This fact can be grasped from the access log in the past attribute data acquisition period. Assuming this, it can be seen that among the communities displayed in the recommended list (S211), "baseball" and "soccer" have already been accessed by the user 1 (S212). In this case, “baseball” and “soccer” are urged to revisit and are neither new nor surprising. Therefore, it belongs to neither novelty object nor unexpected object (S213). On the other hand, it is assumed that “ice hockey”, “real robot”, and “gardening” are communities that the user 1 has not yet accessed (S212).

  Next, among the recommended communities that are the recommended attribute data presented, the access logs of all users or a certain number of extracted users are assumed for each of the three communities that were not accessed before recommendation. The similarity with the visited community group is calculated (S216). Here, there are three visited community groups, “baseball”, “soccer”, and “basketball”. These three communities are regarded as one past attribute data, and the similarity from this group is obtained to determine whether the community is an unaccessed community. Classify.

  A conceptual diagram of the classification is shown in FIG. First, for classification, a threshold is determined for the similarity. Of the range of similarity used here, a community having a relationship higher than the threshold value has a high similarity, and it is assumed that there is no surprise even if displayed. For example, although the threshold value is 0.7 here, the appropriate threshold value changes depending on the number of communities. Here, since the degree of similarity of “ice hockey” is 0.8, which is equal to or greater than the threshold (S217), it is classified as a novelty target (S218). On the other hand, “real robot” and “gardening” whose threshold is lower than 0.8 are classified according to whether or not the user 1 has accessed during the evaluation period (S102 to S104) (S219). The “real robot” accessed by the user 1 during the evaluation period is an unexpected object that the user 1 is newly interested in even though the similarity with the community accessed before the recommendation is low ( S220). On the other hand, the “gardening” that the user 1 did not access during the evaluation period has been unable to attract interest and is not applicable because it is neither a novelty object nor an unexpected object (S221). .

  Thus, the evaluation for all the recommended attribute data is completed (S214). Thereby, the tendency of recommendation attribute data can be evaluated. Here, the recommended number of attribute data is 5, of which “baseball”, “soccer” and “gardening” are “not applicable”, “ice hockey” is a novelty object, and “real robot” is an unexpected object. . Accordingly, in the recommended attribute data presented to the user 1, the novelty target ratio that is the ratio of the novelty target is 1/5 = 20 (%), and the unexpectedness target ratio that is the ratio of the unexpected target is 1/5 = 20 (%). As a result given to the user 1, it can be seen that this led to an increase in interest.

  Such an evaluation is performed for all users using each recommendation method, and further for all the recommendation methods, the results and trends can be compared with each other. That is, classification and statistical processing are performed on all recommended attribute data recommended to individual users using the recommendation methods 1, 2, 3,.

  An overall flow diagram for this purpose is shown in FIG. In the flow, the calculation unit 1 in the center corresponds to the classification of novelty object and unexpected object in FIG. First, with respect to all recommended communities (recommended attribute data) presented by the recommendation method 1, the processing of the calculation unit 1 is performed in order from the user 1 (S211 to S222). Then, an average value of users recommended by the recommendation method 1 (here, users 1 to 500) is calculated (S223). In the figure, the novelty target rate is abbreviated as “novelty”, and the unexpectedness target rate is abbreviated as “unexpectedness”. As a result, the numerical process and the novelty and unexpectedness of the community recommended by the recommendation method 1 were classified. This value indicates the tendency of the recommended attribute data output by the recommendation method 1 (S232).

  This process is also performed for the recommendation methods 2, 3... (S231 to S232), and the novelty object rate and the unexpected object rate are calculated by the respective recommendation methods. Thereby, as shown in FIG. 7, the novelty object rate and the unexpected object rate for each recommendation method can be compared. For example, when the result shown in FIG. 7 is obtained, the recommendation method 1 has a good balance between novelty and unexpectedness, the recommendation method 2 has a lot of novelty and is suitable for steady operation, and the recommendation method 3 is unexpected. It can be evaluated that it is suitable for when you want to spur the expansion trend.

  As described above, the novelty target rate and the unexpected target rate for each user (user) are obtained, and are numerically processed for each recommended method, and the recommended methods are evaluated by comparing the recommended methods. it can. If this overall structure is maintained, other points can be changed.

  For example, in the above judgment, past attribute data is collected for the accessed community, and the unexpectedness evaluation of the recommended attribute data is also performed based on the presence / absence of access. The same can be done for the unexpectedness evaluation.

  In addition, for example, in the above flow, the recommendation attribute data having a similarity equal to or greater than the threshold value is unconditionally regarded as a novelty target, but as in the case of the unexpectedness target, the data that was not actually accessed is “not applicable It is good also as. This is because, even if the degree of similarity is high, there is no point in recommending it unless it actually contributes to access. In this case, the flow of the calculation unit 1 is as shown in FIG. After extracting and quoting the maximum similarity for each recommended attribute data (S216), it is first determined whether or not there was access during the evaluation period (S219A). None ”(S221A). On the other hand, for the recommended attribute data (recommended community) accessed during the evaluation period, it is determined whether or not the similarity is equal to or higher than a threshold (S217A), and is a novelty target (S218A). It is determined whether there is (S220).

  Further, for example, in the above flow, the recommendation methods 1 to 3 are performed for different user groups. However, the recommendation methods 1 to 3 are performed for all users, and the recommended attribute data are sequentially displayed by shifting the provision period. May be. However, in this case, when the user who sees the recommended attribute data by the recommendation method 1 acts on the recommended attribute data by the next recommendation method 2, the recommended attribute data previously recommended is stored in the memory, and the action is performed. Depending on the order of provision, some error occurs.

Claims (4)

Based on one user's past action records, multiple recommendation attribute data presented to one user by one recommendation method,
Calculate the degree of similarity that is highly relevant to the historical attribute data group derived from the action records made by the user in the past,
Classifying the recommendation attribute data that is less than a predetermined threshold that has achieved an action on the recommendation attribute data until a predetermined time has elapsed from the start of presentation as a target of unexpectedness,
Classifying the recommended attribute data whose similarity is equal to or higher than a predetermined threshold, or the one that has achieved an action for the recommended attribute data from the start of presentation to a novelty target,
Finding the novelty target rate that is the ratio of the novelty target and the unexpectedness target rate that is the ratio of the unexpected target with respect to the total number of recommended attribute data,
A method for evaluating the presented content, which evaluates the degree of novelty and unexpectedness of the presented content.   A plurality of the past attribute data is regarded as one group, and the similarity to the group calculated for each of the recommended attribute data is classified into the novelty object and the unexpected object classification for the recommended attribute data. The presentation content evaluation method according to claim 1, wherein the presentation content is used as a similarity that is a criterion of the presentation. For a plurality of users, the presentation content evaluation method according to claim 1 or 2 is executed, and for each of the plurality of users, the novelty target rate and the unexpected target rate are obtained.
A method for evaluating a recommendation method in which the degree of novelty and unexpectedness of the one recommendation method is evaluated by numerical evaluation of those values.   A method for evaluating a plurality of recommended methods, wherein the method for evaluating a recommended method according to claim 3 is executed for a plurality of recommended methods, and numerical evaluations of the methods are compared.

Images