JP2019215705A - Information determination model learning device and program thereof - Google Patents

Abstract

To provide an information determination model learning device which additionally learns an information determination model for determining whether a submission sentence which is information acquired from social media is which type of information.SOLUTION: An information determination model learning device 1 comprises: determination means 12 for inputting a new submission sentence, determining the type, and calculating reliability for the type using a preliminarily learned information determination model; additional learning data collection means 13 for preferentially displaying submission sentences with low reliability in a display device 2 and collecting selected submission sentences as additional learning data for an operator; additional learning means 14 for generating an additional learning model which is an additionally learned information determination model using additional learning data; and model update means 15 for updating the information determination model with the additional learning model as a new model.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information determination model learning device for learning an information determination model for determining what type of information acquired from social media is information, and a program therefor.

  2. Description of the Related Art In recent years, the development of a social networking service (SNS) has made it possible for an individual to easily transmit information in real time. For example, people who happen to be present at the site of a fire, an accident, or the like often post sighting information or the like on the site to the SNS. By acquiring the first report of an incident / accident, it is possible to go to the report quickly and report quickly. In addition, these postings include photos and videos of the site, and are often used in news programs and the like as images and videos that capture the moment when a fire or an accident occurs. Therefore, broadcast stations and the like manually extract such information by monitoring SNS or the like.

  As a method of manually extracting necessary information from the SNS, a keyword search is often used. However, for example, in an SNS post, a line name such as "the XX line is late" is written, or a station name such as "the train is delayed due to an accident at the station △△" is written. There are some cases, and there are various methods of expressing SNS. Therefore, it is difficult to create a keyword that covers all of these expressions.

Therefore, in order to solve these problems, many post extraction methods using machine learning have been studied. For example, a method has been disclosed in which a recurrent neural network (RNN) determines whether a tweet (registered trademark) is information useful for news production (see Non-Patent Document 1).
Further, in the SNS, since information about a new event is posted every day, a method of additionally learning a learned neural network is disclosed (see Patent Document 1).

JP 2007-305072 A

Taro Miyazaki, Shin Toriumi, Yuka Takei, Ichiro Yamada, Jun Goto, "Automatic extraction of tweet useful for news production", Proceedings of the 23rd Annual Conference of the Association for Language Processing, pp.418-421, March 2017 Month

In order to determine information useful for news production using machine learning, a large amount of learning data (teacher data) is required. Further, since new events occur every day, it is necessary to collect comprehensive learning data from a large amount of information of the SNS and improve the neural network model.
However, in the conventional method, since it is necessary to manually extract information in order to collect learning data, labor is required, and it is difficult to collect a large amount of learning data for additional learning. Therefore, in order to improve the model, a method of efficiently collecting learning data is desired.

  Therefore, the present invention provides an information determination model learning device capable of efficiently additionally learning an information determination model for determining which type of information is a posted sentence obtained from social media. And to provide the program.

  In order to solve the above-described problem, an information determination model learning device according to the present invention includes a neural network for determining to which type a posted sentence that is information acquired from social media belongs by classifying the posted content of the posted sentence. An information judgment model learning apparatus for additionally learning an information judgment model of a network, comprising a judgment unit, an additional learning data collection unit, an additional learning unit, and a model updating unit.

In such a configuration, the information determination model learning device uses the information determination model learned in advance by the determination unit to input a new posted sentence, determine the type, and calculate the reliability for the type. Note that a probability value which is an output value of an output layer of the neural network can be used as the reliability.
Then, the information determination model learning device preferentially displays the post with low reliability on the display device by the additional learning data collecting means, and collects the post selected by the operator as additional learning data. Accordingly, the information determination model learning device preferentially presents, to the operator, a posted sentence having a low degree of reliability and a high possibility of an error in the determination result as a posted sentence with insufficient learning. Here, when the operator selects a correct posted sentence of the type determination result from the posted sentences with low reliability, the posted sentence becomes additional learning data.

Then, the information determination model learning device uses the additional learning data of the known type collected by the additional learning data collection unit by the additional learning unit to perform additional learning by using the model parameters of the information determination model as initial values. Generate a learning model.
Then, the information judgment model learning device updates the information judgment model by using the additional learning model as a new model by the model updating means.
Note that the information determination model learning device can cause a computer to operate with an information determination model learning program for causing the computer to function as each of the above-described units.

The present invention has the following excellent effects.
According to the present invention, from a large amount of information obtained from social media, information that is a candidate for learning data is presented to the operator according to the reliability, so that learning data is collected in a short time, and an information determination model is created. You can learn additionally.
As a result, the present invention can efficiently update the information determination model by reducing the amount of work of the operator, and accurately determine what type of information acquired from social media is the information. be able to.

1 is a block diagram illustrating a configuration of an information determination model learning device according to an embodiment of the present invention. It is an example of the information determination model which the information determination model learning apparatus which concerns on embodiment of this invention learns, and is a neural network block diagram using bidirectional LSTM for an intermediate layer. FIG. 4 is an explanatory diagram for describing a structure of an output layer of an information determination model. FIG. 9 is a screen diagram showing an example of a posted sentence displayed on a display device by an additional learning data collection unit. 5 is a flowchart illustrating an operation of the information determination model learning device according to the embodiment of the present invention. It is a neural network block diagram of the information determination model of a modification. FIG. 4 is a neural network configuration diagram of an information determination model according to a modification in which a feature vector is added to the information determination model of FIG. 2. FIG. 11 is a block diagram illustrating a configuration of an information determination model learning device according to an embodiment of a modified example of the present invention. FIG. 2 is a block diagram illustrating a configuration of an information determination device using an information determination model learned by an information determination model learning device according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[Configuration of Information Judgment Model Learning Apparatus]
First, the configuration of the information determination model learning device 1 according to the embodiment of the present invention will be described with reference to FIG.

In the information determination model learning device 1, information transmitted by SNS (such as tweet [registered trademark], which is text data in a posting unit, hereinafter referred to as a posting sentence) is information belonging to any predetermined type (category). This is to learn a model (information judgment model) for judging whether or not.
As shown in FIG. 1, the information determination model learning device 1 includes a control unit 10 and a storage unit 20.

The control means 10 controls the operation of the information determination model learning device 1.
The control unit 10 includes an initial learning unit 11, a determination unit 12, an additional learning data collection unit 13, an additional learning unit 14, and a model updating unit 15.

The initial learning means 11 learns (pre-learns) an initial model of an information determination model for determining to which type a posted sentence belongs based on learning data.
The initial learning unit 11 refers to learning data stored in advance in the learning data storage unit 21 to machine-learn an initial model of the information determination model.
The learning data stored in advance in the learning data storage unit 21 includes learning data of a positive example whose type is a post sentence belonging to the type to be determined and negative example of a post sentence not belonging to the type to be determined. The learning data is pre-classified teacher data.

  The learning data of the positive example is obtained by classifying the posted sentences belonging to the type to be determined into a plurality of classes for each type. In the learning data of this positive example, the types of posts to be determined, for example, the types of news such as "fire", "explosion", and "car accident" are known, Use what was done. Each class is associated with a label ("1: fire", "2: explosion", etc.) indicating the content of the type. Here, an example is shown in which the learning data of the positive example is classified into a plurality of types in advance, but may be one type, for example, only one class of “information useful for news”.

  The learning data of the negative example is a collection of postings that do not belong to the type to be determined in a single class. In order to collect many vocabularies and contexts, for example, a post sentence that is randomly extracted from all available post sentences and does not belong to the type to be determined is used as the learning data of the negative example.

  Specifically, the initial learning means 11 learns a model for determining which class (positive example: plural classes, negative example: single class) the learning data belongs to by a neural network. For example, the initial learning means 11 is a forward propagation neural network (FFNN) using a bidirectional long short-term memory (LSTM), which is a type of recurrent neural network (RNN). To learn the information determination model.

Here, a configuration example of the information determination model will be described with reference to FIG. FIG. 2 is a configuration diagram of the information determination model M using the bidirectional LSTM.
As shown in FIG. 2, the information determination model M is a neural network having an input layer L1, an intermediate layer L2, and an output layer L3.

  The input layer L1 inputs a contribution sentence for each character (including a start symbol <s> and a terminal symbol <e>). Here, the characters are expressed in a vector. The vector representation of characters is a "one-hot expression" in which the number of dimensions is the total number of character types, only the component of the character to be represented is "1", and the others are "0".

  The intermediate layer L2 includes a forward LSTM, a backward LSTM, and a forward propagation neural network (FFNN). In the forward LSTM, characters (character vectors) are input from the input layer L1 in order from the start symbol of the posted sentence, and the LSTM operation is repeated up to the end symbol. In the reverse LSTM, characters (character vectors) are sequentially input from the input layer L1 from the terminal symbol of the posted sentence, and the LSTM operation is repeated up to the start symbol. Then, a vector obtained by concatenating the respective vectors that are the operation results of the bidirectional LSTM is further set as an output vector (intermediate layer output V) via a plurality of FFNNs.

The output layer L3 calculates the ratio (probability value) at the output node as reliability by adding a weight to the value of each element of the output vector, which is the output V of the intermediate layer, and adding and normalizing the value. The class corresponding to the node having the maximum reliability is the determination result.
The output layer L3 calculates the reliability of each node by normalizing the value input for each node using a softmax function.
As shown in FIG. 3, the output layer L3 adds the number of dimensions (the number of nodes) to the number of positive example classes (n) and the number of negative example classes (here, “1”). The number of classes is (n + 1).

Since the type of the input sentence is known (teacher data) as the learning data, the initial learning means 11 applies the model parameter (LSTM) of the information determination model M to the output of the output layer L3 in a direction to reduce the error. , Weight matrices between FFNN nodes, etc.). In order to reduce the error, a general error back propagation method (Back Propagation) may be used.
The bidirectional LSTM of the information determination model M shown in FIG. 2 may use an attention mechanism as described in Non-Patent Document 1.

Returning to FIG. 1, the description of the configuration of the information determination model learning device 1 will be continued.
The initial learning unit 11 writes the learned information determination model (model parameter) into the information determination model storage unit 22 and stores the information determination model. Hereinafter, the information determination model stored in the information determination model storage unit 22 may be referred to as a model A.

The determination unit 12 determines, using the information determination model stored in the information determination model storage unit 22, to which type the posted message whose type is unknown belongs.
Specifically, the determination unit 12 converts a posted sentence obtained from social media such as SNS into a vector expression, and forwards and posts it to the input layer L1 of the information determination model M configured by the neural network shown in FIG. Character vectors are sequentially input from the opposite direction.
Then, in the intermediate layer L2, the determination unit 12 repeats the calculation of the LSTM in the forward LSTM and the LSTM in the reverse direction, respectively, using the parameters of the information determination model M (examples of weight rows). Thereafter, a vector obtained by connecting the outputs of the forward LSTM and the backward LSTM is sequentially calculated via the FFNN to generate an intermediate layer output V.
Then, as shown in FIG. 3, the determining means 12 adds a weight to the value of each element of the intermediate layer output V and propagates the value to the output layer L3, and calculates the probability value of each node as reliability.

  Here, if the class corresponding to the node having the highest reliability is a positive example class, the determination unit 12 adds a label corresponding to the type corresponding to the class together with the reliability as a determination result together with the additional learning data collection unit. 13 is output. If the class corresponding to the node having the highest reliability is a negative example class, the determination unit 12 sends a label determined in advance as a negative example class to the additional learning data collection unit 13 together with the reliability as a determination result. Output.

The additional learning data collecting means 13 collects additional learning data for additionally learning the information determination model from the post sentence whose type is determined by the determining means 12.
The additional learning data collection unit 13 includes a data selection unit 130 and a data extraction unit 131.

The data selecting unit 130 displays the posted text whose type is determined by the determining unit 12 on the display device 2 with priority given to the candidate to be additionally learned, and sets the posted text as additional learning data by the operator (annotator). The selection of whether or not is accepted.
For example, the data selection unit 130 displays the posted text whose type is determined by the determination unit 12 on the display device 2 with priority given to the one with lower reliability.
Here, when the reliability is low, there is a high possibility that the determination result by the information determination model (model A) is incorrect. Therefore, the operator determines whether or not the posted sentence with a low reliability of the determination result is a correct determination result, and if correct, selects the posted sentence as additional learning data. As a result, this posted sentence is selected as additional learning data (teacher data), and the accuracy of the information determination model can be increased by the additional learning.

FIG. 4 shows an example of a screen displayed on the display device 2 by the data selection unit 130.
For example, the data selecting unit 130 may display the determined type label L and the reliability R together with the posted text C and the posted date and time T on the screen shown in FIG. It is displayed on the display device 2. Here, the data selection unit 130 scrolls the posted message at a constant speed on the screen of the display device 2 and sequentially displays the posted message, so that the operator confirms the correspondence between the posted message C and the label L, If the response is correct, the post sentence C is selected by a selection means such as a mouse.
The data selection unit 130 stores the selected posting text together with the label in the additional learning data storage unit 23 as additional learning data. Further, the data selection unit 130 notifies the data extraction unit 131 of the selected posted text and label, and when the selection is completed, for example, when the end button E (see FIG. 4) on the screen is pressed. , Is notified to the data extracting means 131.

  It is not necessary for the data selection unit 130 to display all the posted sentences on the display device 2, and the degree of representing the feature of the type (class) of the posted sentence by a word in the posted sentence is larger than a predetermined threshold. Only sentences may be displayed. For example, a delta IDF (Inverse Document Frequency) can be used as a degree representing the characteristics of this class.

Specifically, first, the data selection unit 130 calculates the value of the delta IDF (Δidf _j ) of the word j in the posted sentence by the following equation (1):

Here, P indicates the number of positive examples of the class corresponding to the posted sentence in the learning data stored in the learning data storage unit 21, and N indicates the number of negative examples. P _j and N _j indicate the number of positive examples and the number of negative examples including the word j of the class corresponding to the posted sentence in the learning data stored in the learning data storage unit 21.
Then, the data selection unit 130 sets a posting sentence having a word having a delta IDF value larger than a predetermined threshold value as a display target.
As described above, the data selection unit 130 gives priority to a post with low reliability and displays a post having a characteristic word on the screen, so that the operator can efficiently select additional learning data. Can be.
Note that the operator does not need to check all the posted messages, but may select a posted message, for example, by defining a work time and a work content such as a fixed time and a fixed number.

The data extraction unit 131 uses the data selection unit 130 to select a post sentence (hereinafter, a non-selected post sentence) that is not selected by the data selection unit 130 among the post sentences determined to be a positive example class by the determination unit 12. A post sentence similar to the selected post sentence is further extracted as additional learning data.
For example, the data extraction unit 131 determines the similarity between the vector expression of the post sentence selected by the data selection unit 130 and the vector expression of the post sentence not selected by the data selection unit 130 (the angle formed by the vectors of the post sentences). (A cosine similarity indicating the closeness of a post) is calculated, and a class of a similar posted sentence is associated with a non-selected posted sentence having a similarity greater than a predetermined threshold value to obtain additional learning data.
Note that the vector expression of the post sentence selected by the data selection unit 130 and the vector expression of the post sentence not selected are output by the determination unit 12 to the output vector of the information determination model when determining the class (the output vector of the output layer). Output). Alternatively, an output vector subsequent to the LSTM of the intermediate layer (for example, an intermediate layer output V; see FIG. 2) may be used.

Further, the data extracting unit 131 calculates the similarity (cosine similarity) between the vector expression of the post sentence selected by the data selecting unit 130 and the vector expression of the posting sentence not selected by the data selecting unit 130. Alternatively, only the non-selected posted text whose posting date is the same as the selected posted text may be targeted. By targeting posts on the same day, a plurality of expressions can be used as additional learning data for the same event, and learning efficiency can be improved.
As described above, the data extracting unit 131 can also collect additional learning data from the postings not selected by the operator by the data selecting unit 130.
The additional learning data collection unit 13 stores the collected additional learning data (posted sentence, type [class]) in the additional learning data storage unit 23.

The additional learning unit 14 additionally learns the information determination model using the additional learning data. The additional learning unit 14 uses the additional learning data stored in the additional learning data storage unit 23 at the timing when the additional learning is instructed from the outside, and stores the information stored in the information determination model storage unit 22. Learn additional judgment models. Note that the learning by the additional learning unit 14 is the same as that of the initial learning unit 11, and a description thereof will be omitted.
The additional learning unit 14 stores, in the additional learning model storage unit 24, an additional learning model (model parameter) obtained by additionally learning the information determination model. Hereinafter, the additional learning model stored in the additional learning model storage unit 24 may be referred to as a model B.
The additional learning means 14 notifies the model updating means 15 of the completion of the learning when the additional learning is completed.

The model updating means 15 updates the information determination model.
The model updating unit 15 includes an evaluating unit 150 and an updating unit 151.

The evaluation unit 150 sends a post sentence for each of the information determination model (model A) stored in the information determination model storage unit 22 and the additional learning model (model B) stored in the additional learning model storage unit 24. This is to evaluate the accuracy of the classification.
Note that a general method can be used as the evaluation method in the evaluation means 150. For example, the evaluation unit 150 calculates an F value for the information determination model and the additional learning model using evaluation data prepared in advance, and sets the F value as an evaluation value. The evaluation data is a posting sentence whose class of the positive example and the negative example is known in advance, and is stored in the evaluation data storage unit 25 in advance.

The F value is a harmonic mean of the precision and the recall.
The relevance ratio is a ratio of data (posted sentence) determined to be a positive example, which is actually a positive example, regardless of the class.
The recall is the ratio of data (posted sentence) actually determined to be a positive example, regardless of the class.
The F value (F-measure) can be obtained by the following equation (2), where the precision is Precision and the recall is Recall.

  The evaluation unit 150 outputs the evaluation value (F value) calculated for each of the information determination model and the additional learning model to the updating unit 151.

The updating unit 151 updates the information determination model based on the evaluation value calculated by the evaluation unit 150, when the evaluation value of the additional learning model exceeds the evaluation value of the information determination model before the additional learning. is there.
Only when the evaluation value of the additional learning model exceeds the evaluation value of the information determination model before the additional learning, the updating unit 151 updates the additional learning model (model parameter) stored in the additional learning model storage unit 24 with the information determination model. The information is stored in the storage unit 22 and updated to a new information determination model. At this time, the updating unit 151 deletes the additional learning data stored in the additional learning data storage unit 23.

  On the other hand, when the evaluation value of the additional learning model is equal to or less than the evaluation value of the information determination model before the additional learning, the updating unit 151 does not update the information determination model. At this time, the updating unit 151 may not delete the additional learning data stored in the additional learning data storage unit 23. Thereby, at the time of the next additional learning, the data amount of the additional learning data can be increased and effective learning can be performed.

The storage unit 20 stores various data used or generated in the operation of the information determination model learning device 1. The storage unit 20 includes a learning data storage unit 21, an information determination model storage unit 22, an additional learning data storage unit 23, an additional learning model storage unit 24, and an evaluation data storage unit 25.
Each of these storage means can be configured by a general storage device such as a hard disk or a semiconductor memory. Here, in the storage means 20, each storage means is individually provided, but the storage area of one storage device may be divided into a plurality of storage areas and each storage means may be used. Further, the storage unit 20 may be omitted from the configuration of the information determination model learning device 1 as an external storage device.

The learning data storage unit 21 stores learning data for the initial learning unit 11 to learn the information determination model in advance.
The information determination model storage unit 22 stores the information determination model (model A) learned by the initial learning unit 11, more specifically, the model parameters. The information determination model (model parameter) stored in the information determination model storage unit 22 is read when the additional learning unit 14 performs additional learning. This information determination model is updated by the model updating unit 15 (updating unit 151).

  The additional learning data storage unit 23 stores additional learning data in order to perform learning by adding an information determination model. The additional learning data storage unit 23 stores the additional learning data collected by the additional learning data collection unit 13, and is read by the additional learning unit 14 when learning the additional learning model.

  The additional learning model storage unit 24 stores the model parameters of the additional learning model (model B) learned by the additional learning unit 14. The model parameters of the additional learning model storage unit 24 and the model parameters stored by the information determination model storage unit 22 are model parameters corresponding to the same neural network structure.

  The evaluation data storage unit 25 stores, in advance, posted sentences whose classes of the positive example and the negative example are known as evaluation data for evaluating a model (information determination model, additional learning model). Note that the evaluation data may have a smaller data amount than the learning data stored in the learning data storage unit 21. The evaluation data storage unit 25 is read by the model update unit 15 (evaluation unit 150).

As described above, the information determination model learning device 1 is configured to collect post messages suitable for learning from post messages whose type is unknown as additional learning data and perform additional learning.
Further, the information determination model learning device 1 is configured to preferentially present a posted sentence with low reliability of type determination to the operator and receive a selection of whether or not to perform additional learning. Therefore, the information determination model learning device 1 can efficiently perform additional learning.

In addition, the information determination model learning device 1 extracts a post sentence similar to the post sentence selected by the operator from the input posted sentence, and sets it as additional learning data. Therefore, a large amount of additional learning data can be efficiently collected with a small number of operators.
Note that the information determination model learning device 1 can be operated by a program (information determination model learning program) for causing a computer to function as the above-described units.

[Operation of information judgment model learning device]
Next, an operation of the information determination model learning device 1 according to the embodiment of the present invention will be described with reference to FIG. Note that the learning data storage unit 21 stores in advance the learning data of the positive example, which is a posted sentence belonging to the type to be determined, and the learning data of a negative example, which is a posted sentence not belonging to the type to be determined. It is assumed that Further, it is assumed that the evaluation data storage unit 25 stores the evaluation data of the positive example and the evaluation data of the negative example in advance similarly to the learning data storage unit 21.

  In step S1, the initial learning unit 11 uses the learning data stored in the learning data storage unit 21 to learn an information determination model (model A) for determining to which type a posted sentence belongs. Then, the initial learning unit 11 stores the information determination model in the information determination model storage unit 22 (not shown as a step).

In step S2, the determination unit 12 sequentially inputs post messages acquired from social media such as SNS.
In step S3, the determining unit 12 determines to which type the posted text input in step S2 belongs by using the information determination model stored in the information determination model storage unit 22.

In step S4, the data selection unit 130 of the additional learning data collection unit 13 displays the post sentence determined in step S3 on the display device 2 with priority given to the lower reliability. As the reliability, an output value of the information determination model used for the determination by the determination unit 12 can be used.
At this time, the data selection unit 130 may display only the posted text in which the degree of the class characteristic (delta IDF or the like) is greater than a predetermined threshold value in the words in the posted text.

  In step S5, the data extraction unit 131 accepts the operator's selection of a posted message from the posted messages displayed in step S4. At this time, the operator determines whether or not the posted sentence with low reliability is a correct determination result, and if so, selects the posted sentence as additional learning data. Then, the data extraction unit 131 stores the selected additional learning data in the additional learning data storage unit 23 (not shown as a step).

  In step S6, the data extraction unit 131 of the additional learning data collection unit 13 additionally learns a post sentence similar to the post sentence selected in step S5 from the post sentence not selected in step S5 (unselected post sentence). Extract as data. As the degree of similarity, for example, cosine similarity can be used. Then, the data extraction unit 131 stores the extracted additional learning data in the additional learning data storage unit 23 (not shown as a step).

  In step S7, the additional learning unit 14 uses the additional learning data stored in the additional learning data storage unit 23 to perform additional learning on the information determination model (model A), thereby generating the additional learning model (model B). Generate. Then, the additional learning unit 14 stores the additional learning model in the additional learning model storage unit 24 (not shown as steps).

  In step S8, the evaluation unit 150 of the model updating unit 15 uses the evaluation data stored in the evaluation data storage unit 25 for the information determination model (model A) and the additional learning model (model B). , Evaluate the accuracy of the classification of the posted text. For example, the model updating unit 15 calculates an evaluation value of accuracy for each of the models A and B based on the F value (F-measure).

In step S9, the updating unit 151 of the model updating unit 15 compares the evaluation value of the information determination model (model A) with the evaluation value of the additional learning model (model B).
Here, when the evaluation value of the additional learning model (model B) exceeds the evaluation value of the information determination model (model A) (Yes in step S9), in step S10, the updating unit 151 executes the additional learning model (model B). ) Is updated as a new information determination model (model A) and stored in the information determination model storage unit 22. Then, in step S11, the updating unit 151 deletes the additional learning data stored in the additional learning data storage unit 23.

On the other hand, when the evaluation value of the additional learning model (model B) is equal to or smaller than the evaluation value of the information determination model (model A) (No in step S9), the information determination model learning device 1 ends the operation.
After the operation is completed, the information determination model learning device 1 performs the operation after step S2 periodically, for example, every day.

Through the above operation, the information determination model learning device 1 can generate a highly accurate information determination model by additionally learning the latest information (posted sentence) while minimizing manual work.
The configuration and operation of the information determination model learning device 1 according to the embodiment of the present invention have been described above, but the present invention is not limited to this embodiment.

[Modification]
(Modification 1)
Here, as a model (information determination model and additional learning model) that the initial learning unit 11 and the additional learning unit 14 learn, a forward propagation neural network (FFNN) using a bidirectional LSTM as an intermediate layer is used. However, general FFNN may be used for these models.

  FIG. 6 shows an example of an information determination model (additional learning model) M2 configured by a general FFNN. As shown in FIG. 6, the information determination model M2 includes an input layer L1, an intermediate layer L2 including a plurality of layers, and an output layer L3. A distributed expression vector having a predetermined number of dimensions is input to the input layer L1 for each posted sentence, weighted, propagated through each layer, and output from the output layer L3.

  Note that the relationship between the output vector (intermediate layer output V) of the intermediate layer L2 and the output layer L3 is the same as the relationship described with reference to FIG. The number of nodes in the output layer L3 of the information determination model M2 is the number of classes (n + 1) obtained by adding the number of positive example classes (n) and the number of negative example classes ("1"), as in FIG. .

The distributed expression vector of the post sentence input to the information determination model M2 is, for example, a vector having a predetermined number of dimensions for each word generated by a general distributed expression vector generation method such as Word2Vec or GloVe. Are added up for the number of words included in, and averaged.
For example, the initial learning means 11 and the additional learning means 14 divide the posted sentence "fire engine fire but fire?" Into words such as "fire engine fire / fire / fire / fire /?" By morphological analysis.
Then, the initial learning unit 11 and the additional learning unit 14 convert each word into a distributed expression vector, add the words, and divide by the number of words to generate a distributed expression vector corresponding to the posted sentence.
As a result, the information determination model M2 can be learned using a distributed expression vector having the same dimension even if the posted sentences have different numbers of words.

(Modification 2)
The information determination model M shown in FIG. 2 may be a model that adds a feature vector representing the weight of a characteristic word in a posted sentence to an intermediate layer.
FIG. 7 shows an example of an information determination model (additional learning model) M3 to which a feature vector is added. As shown in FIG. 7, the information determination model M3 is basically the same as the information determination model M described in FIG. 2, but the feature vector V2 is input at least at the subsequent stage of the bidirectional LSTM.

The feature vector V2 is, for example, a vector having a dimension corresponding to the number of words, and is obtained by setting a word score of an input posted sentence to a vector element. As the score, the same value of the delta IDF as in the above equation (1) can be used. However, in the case of the score of the feature vector V2, P in the above equation (1) is the number of positive examples of all classes in the learning data stored in the learning data storage unit 21, and N is the number of negative examples. Is shown. Further, P _j and N _j indicate the number of positive examples and the number of negative examples including the word j of all classes in the learning data stored in the learning data storage unit 21.

Further, the feature vector V2 may be a predetermined number of dimensions (for example, 10 dimensions) instead of the number of dimension vectors for the number of words. In this case, the feature vector V2 is, for example, the word weight (delta IDF value) of the posted sentence is made to correspond to the element of the vector in a predetermined unit (for example, every 0.1), and the element corresponding to the weight is “1”. , A vector in which “0” is set to uncorresponding elements. Alternatively, the feature vector V2 may be obtained by accumulating the weights of the words corresponding to each element and normalizing the sum so that the sum of the accumulation results is “1”.
This feature vector V2 is generated by the initial learning means 11 and the additional learning means 14.
Thus, the information determination model M3 is a model capable of determining information in consideration of a characteristic word in a posted sentence.

(Modification 3)
Here, the information determination model learning device 1 learns and generates an information determination model using learning data.
However, the information determination model before additional learning does not necessarily need to be generated in the information determination model learning device 1. For example, an information determination model learned and generated outside in advance may be stored in the information determination model storage unit 22. In that case, the information determination model learning device 1 can omit the initial learning means 11 from the configuration. Further, if it is not necessary for the data selection unit 130 to limit the display target of the posted text using the delta IDF, the learning data storage unit 21 can be further omitted.

(Modification 4)
Here, the additional learning data collecting means 13 selects the additional learning data by the operator, and further collects the additional learning data from the unselected postings.
However, the additional learning data collecting means 13 may collect the additional learning data only by the selection of the additional learning data by the operator. In that case, the information determination model learning device 1 can omit the data extracting means 131 from the configuration.

(Modification 5)
Here, the information determination model learning device 1 is a device that additionally learns an information determination model.
However, the information determination model learning device 1 may determine, using the information determination model after the additional learning, what type of information the post sentence belongs to.

For example, FIG. 8 shows a configuration of an information determination model learning device 1B capable of determining a posted sentence.
The information determination model learning device 1B operates by switching between two operation modes: an additional learning mode and an information determination mode. In the additional learning mode, the information determination model learning device 1B additionally learns the information determination model with a new posted sentence. In the information determination mode, the information determination model learning device 1B determines the type of the input posted text.
As shown in FIG. 8, the information determination model learning device 1B is configured by adding a switching unit 16 to the configuration of the information determination model learning device 1. The configuration other than the switching unit 16 is the same as the configuration of the information determination model learning device 1 in FIG.

The switching unit 16 switches the output of the determination result by the determination unit 12 according to the specified operation mode.
The switching unit 16 switches the output of the determination unit 12 to either the additional learning data collection unit 13 or the outside (for example, the display device 2) according to an operation mode instruction from the outside.
When operating in the additional learning mode, the information determination model learning apparatus 1B uses the switching unit 16 to output the output of the determination unit 12 to the additional learning data collection unit 13, thereby performing additional learning of the information determination model based on the input sentence. I do.
In addition, in the operation of the information determination mode, the information determination model learning device 1B determines the type of the input posted text by setting the output of the determination unit 12 to the external output by the switching unit 16 and outputs the determination result to the external. Output to
When it is determined which type the post belongs to, the information determination device 3 shown in FIG. 9 may be used.
The information determination device 3 shown in FIG. 9 includes the determination unit 12 and the information determination model storage unit 22 of the information determination model learning device 1 described with reference to FIG. The information determination model storage unit 22 may store the information determination model additionally learned by the information determination model learning device 1.
As a result, the information determination model learned by the information determination model learning device 1 can be used by a plurality of information determination devices 3.

1, 1B information judgment model learning device 10 control means 11 initial learning means 12 judgment means 13 additional learning data collection means 130 data selection means 131 data extraction means 14 additional learning means 15 model updating means 16 switching means 150 evaluation means 151 updating means 20 Storage means 21 Learning data storage means 22 Information judgment model storage means 23 Additional learning data storage means 24 Additional learning model storage means 25 Evaluation data storage means 3 Information judgment device

Claims (7)

An information determination model learning device that additionally learns an information determination model of a neural network for determining to which type the posted content, which is information acquired from social media, belongs to the classified content of the posted content,
Using the pre-learned information determination model, input a new posted sentence, determine the type and determine reliability for the type,
Additional learning data collecting means for displaying on the display device by giving priority to the post with low reliability, and collecting the post selected by the operator as additional learning data,
Using the additional learning data, additional learning means for generating an additional learning model additionally learning the information determination model,
Model updating means for updating the information determination model with the additional learning model as a new model,
An information judgment model learning device comprising: The additional learning data collection means,
A data selecting unit that sets a post selected by the operator as the additional learning data among the new post;
From the post not selected by the data selection unit, a data extraction unit that extracts a post sentence similar to the additional learning data as additional learning data,
The information determination model learning device according to claim 1, further comprising:   The data extraction unit calculates a cosine similarity between a vector expression of a post sentence selected by the data selection unit and a vector expression of a post sentence not selected by the data selection unit, and calculates a cosine similarity from a predetermined threshold. The information determination model learning device according to claim 2, wherein a contribution sentence having a larger size is extracted as the additional learning data. The model updating means, using predetermined evaluation data, evaluating means for calculating an evaluation value of the information determination model and the additional learning model,
When the evaluation value of the additional learning model exceeds the evaluation value of the information determination model, an updating unit that updates the information determination model,
The information determination model learning device according to any one of claims 1 to 3, further comprising:   When the evaluation value of the additional learning model is equal to or less than the evaluation value of the information determination model, the additional learning data collected by the additional learning data collection unit is retained and used at the time of the additional learning using the next posted sentence. The information determination model learning device according to claim 4, wherein: Switching means for switching between two operation modes of an additional learning mode for performing additional learning and an information determination mode for determining the type of the posted sentence,
The switching means outputs a determination result of the determination means to the additional learning data collection means in the additional learning mode, and outputs a determination result of the determination means to the outside in the information determination mode. The information determination model learning device according to any one of claims 1 to 5.   An information determination model learning program for causing a computer to function as the information determination model learning device according to any one of claims 1 to 6.

Images