JP7187865B2 - Content evaluation device - Google Patents

Description

The present invention relates to a content evaluation device.

With the development of social media, a huge amount of information is exchanged on the Internet, and it is becoming more and more important to find useful information from this huge amount of information.

US Pat. No. 6,200,000 describes a method for determining influential/popular participants in a communication network. representing participants in the group as nodes and representing messages exchanged between the participants as links; analyzing the messages to generate message-related data for each node; and message-related data for each node. to other nodes in the network to generate a corresponding influence value for each node that indicates the relative size and strength of the social activity or connection of the participant corresponding to each node.

Patent Literature 2 describes a device capable of accurately predicting the future number of posts on a specific topic on an SNS or the like. The device has a learning text data input unit that acquires text data for learning from a specific website, and an influence on the number of comments for each group to which a node indicating a single specific user for the topic belongs, based on the number of comments for each topic. a node influence learning unit that calculates and stores this as learning data, a prediction text data input unit that acquires text data for prediction from a specific website after storing the learning data, the number of remarks by topic and learning and a future post number prediction unit that predicts the number of posts on the topic at a specific time in the future based on data and outputs the result.

Patent Literature 3 describes an apparatus capable of appropriately evaluating an arbitrary image in terms of whether or not it attracts the user's attention. The apparatus includes: first counting means for inputting text for a first image and counting the number of the texts; first feature amount acquiring means for acquiring information indicating the feature amount of the first image; Using learning data in which the feature amount acquired by the first feature amount acquisition means is information corresponding to the explanatory variable and information based on the number aggregated by the first aggregation means is information corresponding to the objective variable Learning model generation means for executing machine learning to generate a learning model that outputs a score with a feature amount of an arbitrary image as input; image acquisition means for acquiring a plurality of second images; and features of the second image. The feature quantity acquired by the second feature quantity acquisition means is input as an explanatory variable to the learning model generated by the second feature quantity acquisition means for acquiring information indicating the quantity and the learning model generation means, and the Based on the score calculation means for calculating the score of the second image from the output of the learning model, the text acquisition means for acquiring the text for the second image, the second image, and the text acquired by the text acquisition means Based on the scores calculated by the score calculating means for the similarity determining means for determining whether or not the second images are similar, and the plurality of second images determined to be similar by the similarity determining means, A representative image determination means is provided for determining a representative image representing the plurality of second images.

In Patent Document 4, it is possible to analyze the future interest trends of an unspecified number of users by predicting the number of comments in the future even for prediction target content such as general news articles. A server that can be used is described. The server comprises learning information storage means for pre-storing, as learning information, the transition state of the number of comments per unit time according to the passage of time for each content; Prediction target comment search means for counting the number of comments in each unit time according to the passage of time, and learning information storage means for storing contents of the transition state of the number of comments similar to the transition state of the counted number of comments in each unit time. and a comment number prediction means for deriving the transition state of the number of comments after the judgment time corresponding to the searched content as the future transition state of the number of comments in the prediction target content.

Patent No. 5035642 WO 13/073377 Patent No. 6033697 Patent No. 5952711

In order to extract useful information from a huge amount of information on social media, it is effective to evaluate user reaction (popularity) of messages and images posted on social media. However, since the response level of users to messages, images, etc. posted on social media is affected by the posted user information, posting time, posting speed, etc., what kind of content (content) is effective for response level It is difficult to tell.

SUMMARY OF THE INVENTION An object of the present invention is to provide a technique capable of evaluating the degree of reaction (popularity) of only content posted to social media.

The invention according to claim 1 comprises attribute feature extracting means for extracting attribute features from posts on social media, attribute score calculating means for calculating attribute scores from the attribute features, and content feature extracting content features from the posts. an extracting means, a content score calculating means for calculating a content score from the content features and outputting it independently of the attribute score, and calculating and outputting a reaction degree of the post using the attribute score and the content score. and a reactivity calculating means for learning the attribute feature extracting means and the content feature extracting means using a data set of posts to the social media and the reactivity to the posts during learning, and learning contents to be evaluated during evaluation. It is a content evaluation device that supplies the content feature extraction means that has been completed, calculates the content score, and outputs the content score .

The invention according to claim 2 is the content evaluation apparatus according to claim 1 , wherein the evaluation is performed using an annotated score of the post on the social media.

The invention according to claim 3 is the content evaluation apparatus according to any one of claims 1 and 2 , wherein the content feature extraction means extracts the content features using LSTM or FNN.

The invention according to claim 4 is the content evaluation apparatus according to any one of claims 1 to 3 , wherein the attribute feature extracting means extracts the attribute feature using a multi-layer perceptron.

The invention according to claim 5 is the content evaluation device according to any one of claims 1 to 4 , wherein the reaction degree calculation means calculates the reaction degree of the post by multiplying the attribute score and the content score. be.

In the invention according to claim 6 , the reaction degree calculation means calculates the reaction degree of the post by multiplying the attribute score and the content score and adding the attribute feature as a bias. 3. The content evaluation device according to any one of

The invention according to claim 7 comprises attribute score calculation means for calculating an attribute score from attributes of a post on social media, and content score calculation for calculating a content score from the content of the post independently of the attribute score calculation means. means, and an output means for outputting at least one of the content score and the post reaction level calculated using the attribute score and the content score.

According to the inventions described in claims 1 and 7 , it is possible to evaluate the degree of reaction (degree of popularity) of only the content posted to social media. Also, a content score can be calculated and output at the time of evaluation.

According to the second aspect of the invention, the evaluation can be further performed using an annotation score.

According to the third and fourth aspects of the invention, learning can be further performed using a neural network such as LSTM.

According to the inventions of claims 5 and 6, the post reactivity can be calculated by multiplying the independently calculated attribute score and content score.

1 is a configuration block diagram of an embodiment; FIG. FIG. 3 is a schematic diagram (part 1) showing the relationship between post attributes and content and reactivity; FIG. 10 is a schematic diagram (part 2) showing the relationship between the attribute and content of a post and the degree of reaction; It is a functional block diagram of an embodiment. It is a table which shows the accuracy of the annotation score of embodiment. 4 is a table showing the accuracy of KarmaScore of the embodiment. FIG. 4 is an explanatory diagram of content score output according to the embodiment; FIG. 4 is a schematic diagram (Part 1) showing the relationship between the attribute and content of posts and the degree of reactivity according to the embodiment; FIG. 2 is a schematic diagram (part 2) showing the relationship between post attributes and content and reactivity according to the embodiment; It is a processing flowchart of the embodiment. It is explanatory drawing of learning and evaluation of embodiment. FIG. 10 is a table (part 1) showing the correspondence between content and content scores according to the embodiment; FIG. 2 is a table (part 2) showing correspondence between content and content scores according to the embodiment; It is a functional block diagram of a modification.

An embodiment of the present invention will be described below with reference to the drawings.

FIG. 1 shows a configuration block diagram of a content evaluation device according to this embodiment. The content evaluation device comprises one or more processors 10 , program memory 12 , storage device 14 , input/output interface (I/F) 16 , communication I/F 18 , input device 20 and output device 22 . A content evaluation device may consist of a general-purpose computer with one or more processors and memory.

One or more processors perform various functions by reading and executing the processing programs stored in the program memory 12 . The one or more processors 10 include, as functional blocks, an attribute feature extraction unit 10a, an attribute score calculation unit 10b, a content feature extraction unit 10c, a content score calculation unit 10d, and a reactivity calculation unit 10e.

The attribute feature extraction unit 10a extracts attributes (context) features of messages and images (including still images and moving images) posted on social media.

The attribute score calculator 10b calculates attribute scores using the attribute features extracted by the attribute feature extractor 10a.

The content feature extraction unit 10c extracts content features of messages and images posted on social media.

The content score calculator 10d calculates a content score using the content features extracted by the content feature extractor 10c.

The reaction degree calculation unit 10e calculates the reaction degree of the posted message or image using the attribute score calculated by the attribute score calculation unit 10b and the content score calculated by the content score calculation unit 10d.

Program memory 12 stores processing programs to be executed by processor 10 . The program memory 12 also stores various parameters obtained by learning.

The storage device 14 stores posted messages, images, test data, and the like used for learning.

The input/output I/F 16 is connected to an input device 20 such as a keyboard and mouse, and an output device 22 such as a display.

Communication I/F 18 is connected to cloud 24 via the Internet. The cloud 24 implements social media, and messages and images posted there are stored in the storage device 14 via the communication I/F 18 and further supplied to the processor 10 for analysis.

The model implemented by the processor 10, specifically the attribute feature extraction unit 10a and the content feature extraction unit 10c, uses messages and images actually posted on social media and their reaction (popularity) be learned. The trained processor 10 analyzes posted messages and images for which reactivity is to be predicted, calculates and outputs attribute scores and content scores, and further calculates reactivity and outputs to the output device 22 . Since the processor 10 independently calculates the attribute score and the content score, these scores can be output to the output device 22 separately from the final reactivity. That is, the output of processor 10 is
- attribute score only - content score only - reactivity only - content score and reactivity - attribute score and reactivity - attribute score and content score and reactivity The content score calculated and output by the processor 10 corresponds to the reactivity of only the posted message or image content.

Here, the degree of reaction is an index that indicates the degree of reaction (popularity) to a post on social media.
Number of retweets (RTs) on Twitter (registered trademark)
Number of likes on Facebook Number of views on YouTube (registered trademark)
KarmaScore on Reddit
etc.

Among them, Reddit is often the subject of user reaction research because it is relatively large and the data is open to the public. For example, as a study using Reddit,
Hao Fang, Hao Cheng, Mari Ostendorf. Learning Latent Local Conversation Modes for Predicting Com-munity Endorsement in Online Discussions. ACL 2016.
Hao Cheng, Hao Fang, Mari Ostendorf. A Factored Neural Network Model for Characterizing Online Discussions in Vector Space. EMNLP 2017.
etc. are known.

On Reddit, texts are posted on a wide variety of topics. Reddit has subreddits that organize posts by topic, and each subreddit has a group of posts called threads. A user first makes a contribution on a new topic (submission), can reply to this submission, and can also reply to the reply. A submission and all posts below it form a single thread. Since each post can have multiple replies, the structure of the thread is a tree structure. Then, the user can evaluate whether the post is supported or not supported. KarmaScore is the number of support votes minus the number of disapproval votes. i.e.
KarmaScore = number of votes in favor - number of votes in favor.

In this embodiment, Reddit, which has a large scale and is open to the public, is exemplified as a social media, and a case where KarmaScore is used as the degree of reactivity will be described. However, the present invention is not limited to specific social media, and can be applied to messages and images posted on any social media.

2A and 2B show the general relationship between attributes and content of messages posted on social media and reactivity.

FIG. 2A shows a case where poster A posted a message "It's cold today" at the posting time of 22:00. In this case, the attributes of the message are
Posting time: 22:00 Posted by: A
and the content of the message is
"it is cold today"
is. Assume that the reaction level of the posted message, for example, Reddit's KarmaScore, is 1000.
On the other hand, FIG. 2B shows a case where different posters post the same message at different times. In this case, the attributes of the message are
Posting time: 10:00 Posted by: B
and the content of the message is
"it is cold today"
is. Assume that the posted message has a reaction level of 10, for example, a KarmaScore.

In this way, even if the content is the same, if the attributes are different, the degree of response may differ. In order to evaluate the degree of reactivity of

Therefore, in the present embodiment, when predicting the degree of reaction of a message or image posted on social media, a model is used in which the degree of reaction of attributes and the degree of reaction of content are calculated independently.

FIG. 3 shows the functional blocks of processor 10 in more detail. The functional blocks of processor 10 are separated into attribute blocks for calculating attribute scores and content blocks for calculating content scores.

The attribute block includes an attribute feature extraction unit 10a and an attribute score calculation unit 10b.

The attribute feature extraction unit 10a is composed of, for example, a multi-layer perceptron (MLP), and extracts attribute features of posted messages and images. Attributes are context, the context or background facts of a posted message. The attribute feature extraction unit 10a uses, as attributes, the poster, the depth of the thread, the time since the previous posting, the rank of the posting time during all replies to the previous post, the number of all replies to the previous post, and the like. be done. More specifically, the following may be used as attributes.
(1) Whether or not the poster is the same as the person who started the thread (2) Number of messages posted by the poster (3) Number of replies to posts (4) Number of posts prior to posting (5) Posts after posting number (6) number of sibling posts of a post (7) number of posts in the subtree derived from the post (8) degree of subtree derived from the post (9) depth of thread (10) posts from the first post Time (11) Post Time from Parent Post A multi-layer perceptron may have, for example, 3 layers, each layer having, for example, 64 dimensions. The attribute score calculator 10b calculates an attribute score as a scalar value by integrating parameter vectors of the attribute features extracted by the attribute feature extractor 10a.

The content block comprises a content feature extraction unit 10c and a content score calculation unit 10d. The content feature extraction unit 10c is composed of an LSTM (Long short-term memory) or FNN (Factored Neural Network), and extracts content features. LSTM is used as a basic language model and FNN is used as a language model to predict the next word of a comment and subsequent reply using an attention mechanism.

ここで、ｃｔは記憶素子、ｉｔは入力ゲート、ｆｔは忘却ゲート、ｏｔは出力ゲート、ｇｔは状態候補、

は要素積を示す。また、Ｗ，Ｕ，ｂはパラメータである。 LSTM is an extension of RNN (Recurrent Neural Network) and is a model for sequential data. The LSTN is realized by replacing the intermediate layer unit of the RNN with a block called an LSTM block, which has a memory and three gates. A text is divided into words and given an identifier (ID). Then we put these arrays into the word embedding layer to get an array of word vectors. Let the array of word vectors be X=x ₁ , x ₂ , . . . x _T
and where T is the sequence length. Then, these word vectors are time-series processed by LSTM. After inputting the word vectors x _i into the LSTM, the hidden layer states h _i are computed as follows.

where ct is a storage element, it is an input gate, ft is a forget gate, ot is an output gate, gt is a state candidate,

indicates the element product. Also, W, U, and b are parameters.

The LSTM can be for example one layer and the last hidden layer can be used as content feature extraction. The dimension of the hidden layer may be, for example, 64 and the dimension of the word embedding may be 256. Also, if the message word length is greater than 50, only the first 50 words may be used. The IDs are sorted into the top 32000 words in order of frequency. ５回未満の頻度の単語は、トークン<unk>に置換する。 Squared error can be used as the loss function, and Adam can be used for optimization. The minibatch size can be 64 and the dropout ratio can be 0.5.

Regarding FNN, for example, Hao Cheng Hao Fang Mari Ostendorf. , Denmark. This document describes the use of a language model that uses 13 attributes including the above attributes, expresses topics as a linear sum of multiple latent variable vectors, and sequentially inputs them to the LSTM. That is, a post (comment) is embedded in a local vector and a content vector, with the local vector indicating the context of the post. Local vectors are calculated from the global mode vectors. A content vector indicates the content characteristics of a post.

The content score calculation unit 10d calculates a content score as a scalar value by accumulating the parameter vectors of the content features extracted by the content feature extraction unit 10c.

The attribute feature extraction unit 10a and content feature extraction unit 10c shown in FIG. 3 are trained using Karmascore as a label. The content score calculated by the trained model is then evaluated using the annotated dataset.

Since the distribution of Karmascore, which is Reddit's reactivity, is known to follow Zipf's law, in the present embodiment, Karmascore is converted using the following formula to smooth such a biased distribution.
f(k)=log(k+1) (k≧0)
= 0 (k < 0)
where k is Karmascore.

For learning, we collected 420,598, 247,012, and 644,034 posts from the three Reddit subreddits, AskMen, AskWomen, and AskReddit, and evaluated them for learning at a 4:1 ratio. Let the data set for All posts were actually posted between June 1, 2016 and June 1, 2017.

Annotate posts when using them as test data for evaluation. Annotations can be crowdsourced, for example. An annotator selects three posts that are considered preferable from the ten posts. Each of the 10 posts is a reply to a certain submission. For submissions with 10 or more replies, the top 10 posts with the highest KarmaScore are selected.

For each subreddit, 130 subsets are annotated for a total of 1300 posts annotated. Since 10 annotators vote on 3 different posts, the number of votes will be 0 to 10 when focusing on a certain post. The number of votes in the range 0-10 is called the annotate score. In short, 10 annotators are given a task of selecting 3 favorable ones out of 10 replies to a certain post, and the total number of times the 10 annotators have chosen them is called an annotate score.

In such a task, attributes such as poster, posting time, and posting speed are excluded, so it can be used as an indicator of the reactivity of only the content from which the influence of the attributes has been removed. In the present embodiment, after learning using Karmascore as the degree of reactivity, the content score calculated by the content feature extraction unit 10c and the content score calculation unit 10d after learning is compared with the annotation score for evaluation. , it can be evaluated that the content score reflects the reactivity of the content itself, that is, the influence of attributes is eliminated.

FIG. 4 shows the result of comparison between the content score and the annotation score calculated by the trained device. A case where LSTM is used as the content feature extraction unit 10c is indicated as LSTM _Disjunctive , and a case where FNN is used is indicated as FNN _Disjunctive . MAP (Mean Average Precision), MRR (Mean Reciprocal Rank), and prec3 (precision@3) are calculated for each of the three subreddits, AskMen, AskWomen, and AskReddit. These indicators are detailed in Brian Mcfee and Gert Lanckriet.2010.“Metric learning to rank”.In Proceedings of the 27th annual International Conference on Mechine Learning. For comparison, LSTM _Concat and FNN _Concat are reactivity prediction models that are used by connecting attributes and content without separation, and LSTM _Text and FNN _Text are reactivity prediction models that use only content.

LSTM _Disjunctive and FNN _Disjunctive in this embodiment are evaluated by calculating the content score, but in the comparative models LSTM _{Concat and} FNN _Concat and LSTM _{Text and} FNN _Text , KarmaScore is calculated and evaluated. This is because, in these comparison models, unlike the model in which the attribute and the content are separated as in the present embodiment, since the attribute and the content are not separated, the content score indicating the reactivity of the content alone cannot be calculated in the first place. be.

As shown in FIG. 4, for example, in prec3 of AskMen, the LSTM model is LSTM _Concat (comparative example): 0.306
LSTM _Text (comparative example): 0.335
LSTM _Disjunctive (Embodiment): 0.348
and in the FNN model, FNN _Concat (comparative example): 0.292
FNN _Text (comparative example): 0.342
FNN _Disjunctive (Embodiment): 0.388
, and the model of the present embodiment obtains results with higher accuracy than the comparison model for both LSTM _Disjunctive and FNN _Disjunctive , and it can be seen that the reactivity of content alone can be predicted with high accuracy. The reason why the accuracy of LSTM _Text and FNN _Text is low is that KarmaScore is affected by attributes even for the same content. Also, the low accuracy of LSTM _Concat and FNN _Concat can be similarly attributed. The results in Figure 4 show that the KarmaScore is affected by attributes, while the annotate score excludes the influence of attributes, so the two are different. On the other hand, the model of this embodiment can be said to be highly accurate because it can calculate a value close to the annotation score as the content score.

FIG. 5 shows the KarmaScore results of the model of the present embodiment and the comparative model. That is, in the model of the present embodiment, the KarmaScore is calculated as the reactivity by multiplying the attribute score and the content score, and the result is compared with the actual KarmaScore.
As shown in FIG. 5, for example, in prec3 of AskMen, the LSTM model is LSTM _Concat (comparative example): 0.348
LSTM _Text (comparative example): 0.300
LSTM _Disjunctive (Embodiment): 0.346
and in the FNN model, FNN _Concat (comparative example): 0.453
FNN _Text (comparative example): 0.320
FNN _Disjunctive (Embodiment): 0.441
, and LSTM _Text and FNN _Text are less accurate than other models. This is based on the fact that LSTM _Text and FNN _Text , as described above, calculate the KarmaScore using only content without using attributes.

Also, the LSTMDisjunctive and FNNDisjunctive in which the attribute and the content are separated from each other according to the present embodiment achieve almost the same accuracy as the LSTM _Concat and FNN _Concat in which the attribute and the content are connected without being separated from each other. This means that even if the attribute and the content are separated independently as in this embodiment, the degree of reactivity can be evaluated with high accuracy. means

FIG. 6 schematically shows a case where a learned device outputs the reactivity of only content. Contents of messages and images posted on social media are supplied to the content feature extraction unit 10c.

The content feature extraction unit 10c extracts a content feature vector and outputs it to the content score calculation unit 10d.

The content score calculator 10d integrates feature vectors of content and outputs a content score.

It should be noted that in FIG. 6, the content score is output from the content only, and is output as an indicator of the reactivity of the content only.

7A and 7B schematically show a case where a trained device outputs the final reactivity.

FIG. 7A shows a case where poster A posted a message "It's cold today" at the posting time of 22:00. In this case, the attributes of the message are
Posting time: 22:00 Posted by: A
and the content of the message is
"it is cold today"
is.

The attributes of the message are supplied to the attribute feature extraction unit 10a. The attribute feature extraction unit 10a extracts attribute feature vectors and outputs them to the attribute score calculation unit 10b. The attribute score calculation unit 10b calculates an attribute score by integrating feature vectors of attributes. Assume that the attribute score is, for example, "100".

Also, the content of the message is supplied to the content feature extraction unit 10c. The content feature extraction unit 10c extracts a content feature vector and outputs it to the content score calculation unit 10d. The content score calculator 10d calculates a content score by integrating feature vectors of content. Assume that the content score is, for example, "10".

The reactivity calculator 10e calculates reactivity by multiplying the attribute score calculated by the attribute score calculator 10b by the content score calculated by the content score calculator 10d. That is, the reactivity is attribute score×content score=100×10=1000.

FIG. 7B shows a case in which contributor B has posted a message "It's cold today" at the posting time of 10:00. In this case, the attributes of the message are
Posting time: 10:00 Posted by: B
and the content of the message is
"it is cold today"
is.

The attributes of the message are supplied to the attribute feature extraction unit 10a. The attribute feature extraction unit 10a extracts attribute feature vectors and outputs them to the attribute score calculation unit 10b. The attribute score calculation unit 10b calculates an attribute score by integrating feature vectors of attributes. Assume that the attribute score is "1", for example.

Also, the content of the message is supplied to the content feature extraction unit 10c. The content feature extraction unit 10c extracts a content feature vector and outputs it to the content score calculation unit 10d. The content score calculator 10d calculates a content score by integrating feature vectors of content. Assume that the content score is, for example, "10".

The reactivity calculator 10e calculates reactivity by multiplying the attribute score calculated by the attribute score calculator 10b by the content score calculated by the content score calculator 10d. That is, the reactivity is attribute score×content score=1×10=10.

In this way, although the message attributes are different, the attribute scores are different, but since the message contents are the same, the content scores are the same, and the final reactivity is different.

FIG. 8 shows a processing flowchart of this embodiment.

First, data posted on social media is collected as data used for learning. The data consists of a set of messages (or images) and their reactivity. For example, from three subreddits of Reddit, posted messages and KarmaScore as an index indicating the degree of reaction are extracted (S101). Data used for learning is stored in the storage device 14 .

Next, a model in which attributes and contents are independent and separated as shown in FIG. For example, LSTM (or FNN) is used to learn using the data collected in S101 (S102). If the word length of the message is 50 or more, only the first 50 words are used, and the IDs are distributed to the top 32000 words in order of frequency. ５回未満の頻度の単語は、トークン<unk>に置換する。 Square error is used as the loss function, and Adam is used for optimization.

After learning, the content score and reactivity are calculated using the trained model (S103). The reactivity is
It is calculated as reactivity=attribute score×content score. The content score corresponds to the reactivity of content only. Reactivity is a value that considers attributes and content, and corresponds to Reddit's KarmaScore. In S103, only the content score may be calculated and output. It may be configured such that the user of the content evaluation device can appropriately select the score to be output.

An annotation score is then calculated for the posted message to evaluate the content score. That is, 10 replies to a post on Reddit are extracted (S104), and 3 of the extracted 10 replies are selected (S105). A total of 10 annotators execute the process of selecting 3 replies (S106). If there are 10 or more replies to a post, the top 10 posts with the highest KarmaScore are targeted. The processing of S104-S106 is executed for each subreddit.

Then, the number of times of selection by a total of 10 people is counted for every 10 replies (S107), and the counted number of times for each reply is used as the annotation score of each post (S108). A post that is not picked by all 10 people has an annotated score of 0, and a post that is picked by all 10 people has an annotated score of 10. Attributes such as posting time, posting speed, and poster are excluded by selection by 10 annotators.

After the annotation score is calculated as described above, the post for which the annotation score has been calculated is evaluated by comparing the content score calculated by the learned device and the annotation score calculated in S108 (S109). . If the content score matches the annotate score with a predetermined accuracy, by outputting the content score with a trained device, it is possible to calculate and output the reactivity of only the content excluding the influence of attributes. Become.

FIG. 9 shows the configuration of learning and evaluation in the device of this embodiment. Learning is performed using messages posted to social media and their reaction levels, for example, KarmaScore, and model parameters of the attribute feature extraction unit 10a and the content feature extraction unit 10c are adjusted. Also, evaluation is performed using messages annotated by a plurality of annotators and their reactivity, that is, annotation scores, and is evaluated by comparing content scores and annotation scores. That is, different functional blocks and different indices are used for learning and evaluation.

10A and 10B show specific examples of posted messages and calculated content scores. FIG. 9A is an example of a posted message with a relatively high content score, and FIG. 9B is an example of a posted message with a relatively low content score. for example,
"my neighbor on the other side if the house were having a fire pit. we heard a fox in the woods before we knew what it was we"
The content score for this message is calculated as "1.062". again,
"yeah but somebodys opinion could be that climate change isnt real . up-voting that gives visibility to misinformation that had the potential to hurt people . im sorry but if somebody is trying to tell me that china invented climate change im downvoting that because its factually incorrect and contributes”
The content score for this message is calculated as "-0.457". In general, messages that are contextually independent tend to have high content scores, while messages that are highly contextual tend to have low content scores.

In S102, LSTM and FNN were exemplified as models of the content feature extraction unit 10c, but in addition to this, BoW (Bag of Words), BoW weighted by tf-idf (Term Frequency - Inverse Document Frequency), Embedded words Linear addition, Maxpooling, etc. of vectors can be used.

Also, in the present embodiment, a message is exemplified as content, but in the case of an image, BoF (Bag of Feature), CNN (Convolutional Neural Network), or the like can be used.

Although the embodiment of the present invention has been described above, the present invention is not limited to this, and various modifications are possible. Modifications will be described below.

<Modification>
In an embodiment, as shown in FIG.
Although the reactivity is calculated as reactivity=attribute score×content score, the reactivity may be calculated by additionally applying the influence of attributes as a bias.

FIG. 11 shows a functional block diagram of processor 10 in a modification. In addition to the configuration of FIG. 3, the scalar value calculated from the attribute feature vector extracted by the attribute feature extraction unit 10a is added to the value calculated by the reactivity calculation unit 10e as a bias by the adder 10f. Calculate the appropriate reactivity. i.e.
Reactivity is calculated as reactivity=attribute bias+attribute score×content score. Also in the configuration of FIG. 11, the content score is calculated independently and separately from the attribute score. The configuration of FIG. 3 can be said to be a special case where the bias in FIG. 11 is always zero.

10 processor, 12 program memory, 14 storage device, 16 input/output interface (I/F), 18 communication interface (I/F), 20 input device, 22 output device.

Claims (7)

attribute feature extracting means for extracting attribute features from posts on social media;
attribute score calculation means for calculating an attribute score from the attribute features;
content feature extraction means for extracting content features from the post;
a content score calculation means for calculating a content score from the content features and outputting the content score independently of the attribute score;
a reaction degree calculation means for calculating and outputting the reaction degree of the post using the attribute score and the content score;
with
During learning, the attribute feature extraction means and the content feature extraction means are learned using a data set of posts on the social media and the degree of reaction to the posts,
At the time of evaluation, the content to be evaluated is supplied to the learned content feature extraction means, and the content score is calculated and output.
content evaluator. The content evaluation device according to claim 1 , wherein the evaluation is performed using an annotated score of a post on the social media. 3. The content evaluation device according to claim 1, wherein said content feature extraction means extracts said content features using LSTM or FNN. 4. The content evaluation device according to any one of claims 1 to 3, wherein said attribute feature extracting means extracts said attribute feature using a multi-layer perceptron. The content evaluation device according to any one of claims 1 to 4, wherein the reaction degree calculation means calculates the reaction degree of the post by multiplying the attribute score and the content score. The content evaluation device according to any one of claims 1 to 4, wherein the reaction degree calculation means calculates the reaction degree of the post by multiplying the attribute score and the content score and adding the attribute feature as a bias. . attribute score calculation means for calculating an attribute score from attributes of posts on social media;
content score calculation means for extracting content features from the posted content independently of the attribute score calculation means and calculating a content score from the content features;
output means for outputting at least one of the content score and the reaction level of the post calculated using the attribute score and the content score;
with
During learning, learning a content feature extraction means for extracting the content feature using a data set of posts on the social media and the degree of reaction to the posts,
At the time of evaluation, the content to be evaluated is supplied to the learned content feature extraction means to calculate the content score;
content evaluator.

Images