JP2018022290A - Information processing device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve the accuracy of at least one of training and identification in comparison with a configuration in which lacking information is not interpolated in the case of using information of a plurality of formats to perform machine learning.SOLUTION: A multi-modal identification device 300 includes: a relevant information selection part 312 and an information interpolation part 313 for interpolating information of a lacking format by relevant information related to information of a plurality of formats in the case that information of a portion of formats in the information of the plurality of formats; and a multi-modal identification part 314 for using the information of the plurality of formats obtained by interpolating the information of the lacking format to execute at least one of training and identification.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an information processing apparatus and a program.

For example, Patent Document 1 discloses a method for customizing a structured document that is provided with a display device and is electronically distributed to a consumer's consumer computer. Augment the section by segmenting the document into multiple variable content sections, each with multiple selectable content alternatives, and for each variable content section, evaluate the content alternatives against the consumer's consumer profile Techniques are disclosed for selecting one of the content alternatives, augmenting a section with the selected content alternative, and presenting the augmented structured document to the consumer on a display device.

Japanese translation of PCT publication No. 2002-520689

2. Description of the Related Art Conventionally, a machine learning technique is known in which a certain number of data is input and analyzed, and useful rules and criteria are derived from the data. In such machine learning, various types of data are input and training and identification processes are performed. For example, when information in a plurality of formats is used as input data, it is desirable that input information is not missing.
An object of the present invention is to improve the accuracy of at least one of training and identification in comparison with a configuration in which missing information is not interpolated when machine learning is performed using information in a plurality of formats.

According to the first aspect of the present invention, when a part of information in a plurality of formats is missing, interpolation is performed to interpolate the information in the missing format by using related information related to the information in the plurality of formats. An information processing apparatus comprising: means; and processing execution means for executing at least one of training and identification using the information of the plurality of formats in which the information of the missing format is interpolated.
According to a second aspect of the present invention, the interpolation means has the missing format as the related information, and a predetermined time from the time when the missing information is posted out of the information in the plurality of formats. The information processing apparatus according to claim 1, wherein information posted inside is acquired.
According to a third aspect of the present invention, the interpolation means has the missing format as the related information, and a predetermined range from a position where the missing information is posted among the information of the plurality of formats. The information processing apparatus according to claim 1, wherein information posted at a position in the server is acquired.
According to a fourth aspect of the present invention, when the interpolation unit lacks the first format information and the second format information is not missing, the second format is not missing. The information in the first format posted together with other information in the second format satisfying a predetermined degree of similarity with the information of the information is acquired as the related information. Information processing apparatus.
According to a fifth aspect of the present invention, when the second format information that is not missing is text format information, the interpolation means determines the missing format based on the similarity of the document vector calculated from the text format information. 5. The information in the first format posted together with other information in the text format satisfying a predetermined degree of similarity with the information in the second format that is not obtained is acquired as the related information. It is an information processing apparatus as described in.
In the invention described in claim 6, when a part of information in a plurality of formats is missing in the computer, the information in the missing format is obtained by the related information related to the information in the plurality of formats. A program for realizing an interpolating function and a function of executing at least one of training and identification processing using the information of the plurality of formats in which the information of the missing format is interpolated.

According to the first aspect of the present invention, when performing machine learning using information in a plurality of formats, the accuracy of at least one of training and identification is improved as compared with a configuration in which missing information is not interpolated. be able to.
According to the second aspect of the present invention, when a part of information in a plurality of formats is missing, it is possible to perform interpolation using information that is temporally close.
According to the third aspect of the present invention, when a part of information in a plurality of formats is missing, interpolation can be performed using geographically close information.
According to the fourth aspect of the present invention, when a part of information in a plurality of formats is missing, it is possible to perform interpolation based on the similarity to the information that is not missing.
According to the fifth aspect of the present invention, when information in a text format is included in many pieces of information in a plurality of formats, it becomes possible to interpolate a lot of information in a plurality of formats.
According to the sixth aspect of the present invention, when machine learning is performed using information in a plurality of formats, the accuracy of at least one of training and identification is improved as compared with a configuration in which missing information is not interpolated. The function can be realized by a computer.

It is a figure for demonstrating an example of machine learning. It is a figure showing the example of whole composition of the computer system to which this embodiment is applied. It is a figure which shows the hardware structural example of the multimodal identification device which concerns on this Embodiment. 3 is a block diagram illustrating an example of a functional configuration of a multimodal identification device according to Embodiment 1. FIG. (A), (b) is a figure for demonstrating an example of the selection process of the relevant information which concerns on Embodiment 1. FIG. (A) is the flowchart which showed the procedure of the process at the time of training by the multimodal identification device which concerns on Embodiment 1. FIG. (B) is the flowchart which showed the procedure of the process at the time of the identification by the multimodal identification apparatus which concerns on Embodiment 1. FIG. 6 is a block diagram illustrating an example of a functional configuration of a multimodal identification device according to a modification of the first embodiment. FIG. (A), (b) is the figure which showed an example of the similarity of text information. FIG. 6 is a block diagram illustrating a functional configuration example of a multimodal identification device according to a second embodiment. (A), (b) is a figure for demonstrating an example of the selection process of the relevant information which concerns on Embodiment 2. FIG. (A) is the flowchart which showed the procedure of the process at the time of training by the multimodal identification device which concerns on Embodiment 2. FIG. (B) is the flowchart which showed the procedure of the process at the time of the identification by the multimodal identification apparatus which concerns on Embodiment 2. FIG.

Embodiments of the present invention will be described below in detail with reference to the accompanying drawings.
<Background>
First, the background of the present embodiment will be described.
FIG. 1 is a diagram for explaining an example of machine learning. The example shown in FIG. 1 shows machine learning using a neural network model. Specifically, the input data (input data) passes through the input layer, passes through the intermediate layer and the output layer, and is processed to generate an output result (output data). In the training process, a plurality of data is input, and weighting adjustment is performed to connect the input layer, the intermediate layer, and the output layer. By performing such training and adjusting the weighting, a desired output result can be obtained when some data is input and identified.

  In general, in machine learning, supervised learning is known in which training is performed based on data for which judgment criteria have been previously shown by humans, that is, so-called teacher data, and a rule for appropriately outputting input data is derived. ing. In addition, unsupervised learning is also known in which only input data is given, training is performed based on the data, and the computer itself automatically assigns an output that is optimized based on a certain standard. .

  In this embodiment, machine learning is performed using multimodal information (hereinafter referred to as multimodal information), in other words, information including a plurality of communication modes. That is, in this embodiment, training is performed using multimodal information, and weighting is adjusted. Then, data is input to the model whose weight is adjusted, and the input data is identified.

Here, the multi-modal information includes, for example, information in a format (communication style) such as text, image, video, audio, and the like, and in other words, information in at least two or more formats is included. include. The text indicates a character string or a document data composed of character codes.
As such multi-modal information, for example, information posted to a user through a social networking service (SNS) such as Twitter (registered trademark) or Instagram (registered trademark) and published can be exemplified.

  For example, let us consider a case in which a text “cute” and an image of “dog” are posted together as posted contents. In such a case, it is a post that expresses “dog” as “cute” because it handles multi-modal information such as text and images, and for example, it can be predicted that the poster is a dog dog. On the other hand, for example, if only the text “cute” is present, the subject to be “cute” is unknown. For example, if only the image “dog” is used, it is determined whether the dog likes or dislikes. I can't. Thus, by using multimodal information, it becomes easier to predict the user's preferences, behaviors, and the like than when only one type of information is used. Information such as predicted user preferences and behavior can be used as marketing information such as advertisement distribution.

  By the way, for example, information posted by SNS such as Twitter and Instagram does not include information of two or more formats and is often not multimodal information. For example, Twitter is the main SNS for posting text, and there are many cases where there is text information but image information is missing. In addition, for example, Instagram is a main SNS for image posting, and text information is often missing while there is image information. Therefore, in the following, a procedure for interpolating a part of missing information in a plurality of formats in order to perform machine learning using multi-modal information will be described.

<Overall system configuration>
A computer system to which this embodiment is applied will be described. FIG. 2 is a diagram showing an example of the overall configuration of a computer system to which the present embodiment is applied. As shown in the figure, in this computer system, a client terminal 100 (client terminals 100a to 100c in the illustrated example), an information storage device 200, and a multimodal identification device 300 are connected to a network 400.

  The client terminal 100 is a terminal device used by a user, and can be exemplified by, for example, a portable information terminal (so-called smart phone or tablet terminal) or a PC (Personal Computer). Using this client terminal 100, the user posts text and images in services such as Twitter and Instagram, for example. In other words, the client terminal 100 transmits information posted by the user such as text and images (hereinafter referred to as posted information) to the information storage device 200 based on the user's operation input. In FIG. 2, only three client terminals 100 are shown. However, the number is not limited to three, and actually, for example, a plurality of client terminals using various services such as Twitter and Instagram. Is the target.

  The information storage device 200 is a computer device that stores post information acquired from the client terminal 100, and may be a server or the like. More specifically, the information storage device 200 is a server provided to provide various services such as Twitter and Instagram. In FIG. 2, only one information storage device 200 is shown. However, the information storage device 200 is not limited to one. Actually, the information storage device 200 is provided for each of various services. 200 is provided.

The multimodal identification device 300 is a computer device that performs machine learning using multimodal information. Examples of the multi-modal identification device 300 include a PC and a workstation.
More specifically, the multimodal identification device 300 acquires post information stored in the information storage device 200 and performs training based on the acquired post information. As will be described in detail later, in this training, when the acquired post information is not multimodal information and there is missing information, the multimodal identification device 300 performs training after interpolating the missing information.
In addition, the multimodal identification device 300 generates a training model by training, and then identifies post information acquired from the information storage device 200 using the generated training model. As will be described in detail later, in this identification, when the acquired post information is not multimodal information but there is missing information, the multimodal identification device 300 performs identification after interpolating the missing information.

  The network 400 is a communication unit used for information communication among the client terminal 100, the information storage device 200, and the multimodal identification device 300, and is, for example, the Internet.

  In the example illustrated in FIG. 2, the multi-modal identification device 300 acquires post information directly from the information storage device 200, but is not limited to such a configuration. For example, another storage device that stores the posting information of the information storage device 200 may be provided, and the multimodal identification device 300 may acquire the posting information from the other storage device.

<Hardware Configuration of Multimodal Identification Device 300>
Next, a hardware configuration of multimodal identification device 300 according to the present embodiment will be described. FIG. 3 is a diagram illustrating a hardware configuration example of the multimodal identification device 300 according to the present embodiment. As shown in the figure, the multimodal identification device 300 includes a CPU (Central Processing Unit) 301 that is a computing means, a main memory 302 that is a storage means, and a magnetic disk device 303.

Here, the CPU 301 executes various programs such as an OS (Operating System) and applications, and realizes various functions of the multimodal identification device 300. The main memory 302 is a storage area for storing various programs and data used for execution thereof. The magnetic disk device 303 is a storage area for storing input data for various programs, output data from various programs, and the like.
Furthermore, the multimodal identification device 300 includes a communication interface (communication I / F) 304 for performing communication with the outside, a display mechanism 305 including a video memory, a display, and the like, and an input device 306 such as a keyboard and a mouse. .

[Embodiment 1]
<Functional configuration of multi-modal identification device>
The first embodiment will be described.
FIG. 4 is a block diagram illustrating a functional configuration example of the multimodal identification device 300 according to the first embodiment. The multi-modal identification device 300 includes an information loss detection unit 311 that detects information loss in posted information, a related information selection unit 312 that selects related information that is information for interpolating the missing information, and missing information Are interpolated with related information, a multi-modal identification unit 314 that performs training and identification based on post information, and a training model storage unit 315 that stores a training model generated by training.

  The information loss detection unit 311 acquires post information used for training or identification from the information storage device 200. Then, the information loss detection unit 311 determines whether or not there is information loss in the acquired post information. More specifically, the information loss detection unit 311 determines whether the acquired post information satisfies a predetermined loss condition (hereinafter referred to as a loss detection condition). Here, the information loss detection unit 311 determines that the information is not missing when the loss detection condition is satisfied, and determines that the information is missing when the loss detection condition is not satisfied. This missing detection condition defines the format of information constituting the multimodal information.

  For example, the case where it is prescribed | regulated that multi-modal information contains text information and image information as a missing detection condition is demonstrated. In this case, the information loss detection unit 311 determines whether text information and image information are included in the posted information acquired from the information storage device 200. When both the text information and the image information are included in the acquired post information, the missing detection condition is satisfied, and the information missing detection unit 311 determines that no information is missing. On the other hand, when at least one of text information and image information is not included in the acquired posted information, the missing detection condition is not satisfied, and the information missing detection unit 311 determines that the information is missing.

  Note that the information loss detection unit 311 outputs the post information to the related information selection unit 312 and the information interpolation unit 313 when it is determined that the post information acquired from the information storage device 200 is missing. . On the other hand, when it is determined that the post information acquired from the information storage device 200 is not missing, the information loss detection unit 311 outputs the post information to the multi-modal identification unit 314 as it is because it is multi-modal information. To do.

The related information selection unit 312 selects related information that is information for interpolating the missing information when it is determined that the information is missing for the posted information. In other words, the related information can be regarded as information related to the posted information in which the lack of information is detected.
In the present embodiment, the related information selection unit 312 includes a candidate information extraction unit 312a and a candidate information selection unit 312b.

  The candidate information extraction unit 312a expands the extraction range of post information to past posts, and extracts information groups that are candidates for related information. More specifically, the candidate information extraction unit 312a acquires, from the information storage device 200, past posting information of the user who posted the posting information determined to be missing. The information group acquired here (that is, past posting information) is treated as a candidate for related information.

  The candidate information selection unit 312b selects related information from the information group extracted by the candidate information extraction unit 312a. In other words, the candidate information selection unit 312b selects related information from post information posted in the past from post information determined to be missing information. Details of the procedure for selecting the related information will be described later.

  The information interpolation unit 313 interpolates the missing information with the related information selected by the related information selection unit 312 when it is determined that information is missing from the information missing detection unit 311. More specifically, the information interpolating unit 313 associates the related information selected by the related information selecting unit 312 with the posted information determined to be missing by the information missing detecting unit 311. By associating the related information with the posted information, the missing information is interpolated, and the posted information can be handled as multimodal information.

  The multimodal identification unit 314 acquires post information, which is multimodal information in which no information is missing, from the information loss detection unit 311. In addition, the multimodal identification unit 314 acquires post information in which information missing from the information interpolation unit 313 is interpolated into multimodal information. And the multimodal identification part 314 performs training based on the acquired posting information.

  In this training, the multimodal identification unit 314 updates and generates a training model using post information that is multimodal information as a set of training. For example, in the case of supervised learning, the user gives a correct label (that is, a desired output result) to the multimodal information after interpolation, and training is performed. On the other hand, in the case of unsupervised learning, the multimodal identification device 300 automatically assigns a correct answer label to the interpolated multimodal information and performs training. After the training is performed in this way and the weighting is adjusted, the identification process is started. That is, the multimodal identification unit 314 performs identification based on the posted information acquired from the information loss detection unit 311 and the information interpolation unit 313.

  The training model storage unit 315 stores a training model that is updated and generated by the training by the multimodal identification unit 314.

Note that each functional unit constituting the multimodal identification device 300 shown in FIG. 4 is realized by cooperation of software and hardware resources. Specifically, when the multi-modal identification device 300 is realized by the hardware configuration shown in FIG. 3, the OS program and application program stored in the magnetic disk device 303 are read into the main memory 302 and the CPU 301. As a result, the functions of the information loss detection unit 311, the related information selection unit 312, the information interpolation unit 313, and the multimodal identification unit 314 are realized. The training model storage unit 315 is realized by storage means such as the main memory 302 and the magnetic disk device 303.
In the present embodiment, the related information selection unit 312 and the information interpolation unit 313 are used as an example of an interpolation unit. As an example of processing execution means, a multi-modal identification unit 314 is used.

<Related information selection process>
Next, processing for selecting related information that is information for interpolating missing information will be described in detail. FIGS. 5A and 5B are diagrams for explaining an example of related information selection processing according to the first embodiment. In the example shown in FIGS. 5A and 5B, post information posted by a certain user (referred to as user A in the illustrated example) is shown. For each piece of post information, the presence / absence of text information and image information is indicated.

  For example, “t = 0” indicates the current time. “T = −1” indicates the time when the post information immediately before the post information of “t = 0” is posted. Similarly, “t = −2” indicates the time when the post information immediately before the post information of “t = −1” is posted. “T = −3” indicates the time when the post information immediately before the post information of “t = −2” is posted.

  In the example shown in FIG. 5A, the posted information posted at the time “t = 0” includes text information but does not include image information. Here, when the information loss detection unit 311 detects the loss of image information for the post information of “t = 0”, the candidate information extraction unit 312a of the related information selection unit 312 preliminarily before the time of “t = 0”. Post information posted within a predetermined time is acquired from the information storage device 200. Then, the candidate information selection unit 312b selects related information from the acquired past posting information.

In the example illustrated in FIG. 5A, the candidate information extraction unit 312a includes “t = −1”, “t = −2” as post information within a predetermined time before the time “t = 0”. ”And“ t = −3 ”. Then, the candidate information selection unit 312b searches for information in a format missing at “t = 0”, that is, image information, from the acquired past posting information. Here, the posted information posted at the time “t = −1” includes image information. Therefore, the candidate information selection unit 312b selects the image information posted at “t = −1” as related information.
In this way, the image information of “t = −1” is interpolated with respect to the post information of “t = 0”. As a result, the text information of “t = 0” and the image information of “t = −1” are used as training and identification in the multimodal identification unit 314 as multimodal information.

  More specifically, the post information posted at the time “t = −3” includes image information, but the time “t = −1” is more than the time “t = −3”. It is close to the time “t = 0”. In general, the posted contents tend to be similar as the posted time is closer, and not similar as the posted time is separated. Therefore, the candidate information selection unit 312b selects the image information posted at “t = −1” as the image information posted at the closest time in the past from “t = 0”. When image information is not included in the past post information acquired by the candidate information extraction unit 312a, the post information is output to the multi-modal identification unit 314 without being interpolated and remaining missing.

  Further, in the example shown in FIG. 5B, the post information posted at the time “t = 0” includes image information but does not include text information. Here, when the missing information detector 311 detects missing text information for the posted information with “t = 0”, the candidate information extracting unit 312a detects “t = 0” as in the example illustrated in FIG. Post information posted within a predetermined time prior to the time is acquired from the information storage device 200. Then, the candidate information selection unit 312b selects related information from the acquired past posting information.

In the example illustrated in FIG. 5B, the candidate information extraction unit 312 a includes “t = −1”, “t = −2” as post information within a predetermined time before the time “t = 0”. ”And“ t = −3 ”. Then, the candidate information selection unit 312b searches for information in a format missing at “t = 0”, that is, text information, from the acquired past posting information. Here, the post information of “t = −1” does not include text information. On the other hand, the post information “t = −2” includes text information. Therefore, the candidate information selection unit 312b selects the text information posted at “t = −2” as related information.
In this way, the text information of “t = −2” is interpolated with respect to the post information of “t = 0”. As a result, the image information of “t = 0” and the text information of “t = −2” are used as multimodal information for training and identification in the multimodal identification unit 314.

  More specifically, the post information posted at the time “t = −3” includes text information, but the time “t = −2” is more than the time “t = −3”. It is close to the time “t = 0”. Therefore, the candidate information selection unit 312b selects text information posted at “t = −2” as text information posted at the closest time in the past from “t = 0”. In addition, when text information is not included in the past posting information acquired by the candidate information extraction unit 312a, interpolation is not performed, and the posting information is output to the multi-modal identification unit 314 without being interpolated.

  As described above, in the present embodiment, when the lack of information is detected in the posted information, the related information selection unit 312 selects from the posted information posted within a predetermined time in the past from the posted information. Select related information. In this selection, the candidate information selection unit 312b selects information that is in the same format as the missing information and that has been posted at the closest time in the past from the posted information that is determined to be missing. To do. By interpolating the missing information in this way, the possibility of performing interpolation based on the same (or similar) content / topic as post information is increased.

<Procedure for processing by multi-modal identification device>
Next, a processing procedure performed by multimodal identification apparatus 300 according to the present embodiment will be described. A description will be given by dividing into a process at the time of training by the multi-modal identification device 300 and a process at the time of identification. FIG. 6A is a flowchart showing a procedure of processing during training by the multimodal identification device 300 according to the first embodiment. FIG. 6B is a flowchart showing a procedure of processing at the time of identification by the multimodal identification device 300 according to the first embodiment.

First, the processing at the time of training by the multimodal identification device 300 will be described with reference to FIG.
The information loss detection unit 311 acquires post information used for training from the information storage device 200 (step 101). Here, posting information acquisition conditions may be set, and posting information satisfying the acquisition conditions may be acquired. As acquisition conditions, for example, it is conceivable to specify a period in which posted information is posted, or to specify a region in which posted information is posted and a user who has posted posted information.

  Next, the information loss detection unit 311 determines whether or not there is missing information in the acquired post information (step 102). Here, it is determined whether or not the acquired post information satisfies the missing detection condition. If the acquired post information satisfies the missing detection condition, it is determined that there is no missing information. On the other hand, when the acquired post information does not satisfy the missing detection condition, it is determined that there is missing information.

If it is determined in step 102 that there is no missing information in the posted information (No in step 102), the posted information is output to the multimodal identifying unit 314. Then, the process proceeds to step 106.
On the other hand, when it is determined in step 102 that there is missing information in the posted information (Yes in step 102), the posted information is output to the related information selection unit 312. Next, the candidate information extraction unit 312a of the related information selection unit 312 expands the post information extraction range to past posts, and acquires past post information that is candidates for related information (step 103). Next, the candidate information selection unit 312b selects related information from the past posting information acquired by the candidate information extraction unit 312a (step 104).

  Next, the information interpolating unit 313 associates the related information selected by the related information selecting unit 312 with the post information determined to be missing, and interpolates the missing information (step 105). . Post information in which the missing information is interpolated is output to the multi-modal identification unit 314. Then, the process proceeds to step 106.

  After a negative determination (No) is made in step 102 or after step 105, the multimodal identification unit 314 performs training based on the posted information acquired from the information missing detection unit 311 or the information interpolation unit 313 (step 106). ). Next, the information loss detection unit 311 determines whether or not all post information acquired from the multimodal identification device 300 in step 101 has been processed (step 107). If there is post information that has not been processed yet (No in Step 107), the process proceeds to Step 102. On the other hand, when all of the posted information has been processed (Yes in step 107), this processing flow ends. In addition, training is complete | finished by repeating this process flow once or several times, and the process of identification is started.

Next, with reference to FIG. 6B, a process at the time of identification by the multimodal identification device 300 will be described.
The information loss detection unit 311 acquires post information to be identified from the information storage device 200 (step 201). Here, as in step 101 in FIG. 6A, posting information acquisition conditions may be set. The processing of the next step 202 to step 205 is the same as the processing of step 102 to step 105 in FIG.

  After a negative determination (No) is made in step 202 or after step 205, the multi-modal identification unit 314 performs identification based on the posted information acquired from the information missing detection unit 311 or the information interpolation unit 313 (step 206). ). Next, the information loss detection unit 311 determines whether or not all post information acquired from the multimodal identification device 300 in step 201 has been processed (step 207). If there is post information that has not been processed yet (No in Step 207), the process proceeds to Step 202. On the other hand, when all of the posted information has been processed (Yes in step 207), this processing flow ends. As described above, the information obtained as a result of the identification is used as marketing information such as advertisement distribution.

In the present embodiment, in the information interpolation, the information posted at the nearest time in the past is interpolated from the posted information determined to be missing information. Interpolation may be performed based on the similarity.
For example, in the example illustrated in FIG. 5A, the image information “t = −1” is interpolated with respect to the post information “t = 0”. Here, the related information selection unit 312 determines the similarity between the text information “t = 0” and the text information “t = −1”, the text information “t = 0”, and the text “t = −3”. Assume that the similarity with information is calculated. As a result, when it is determined that the text information of “t = −3” is more similar to the text information of “t = 0” than the text information of “t = −1”, “t = − Instead of the image information of “1”, the image information of “t = −3” may be selected and interpolated.

<Modification of Embodiment 1>
Next, a modification of the first embodiment will be described. FIG. 7 is a block diagram illustrating a functional configuration example of the multimodal identification device 300 according to the modification of the first embodiment. In this modified example, the related information selecting unit 312 further includes a candidate range specifying unit 312c as compared with the configuration of the first embodiment shown in FIG. The functions of the information loss detection unit 311, the information interpolation unit 313, the multimodal identification unit 314, and the training model storage unit 315 are the same as in the case of FIG. 4. Therefore, the related information selection unit 312 will be described below as a difference from the configuration of the first embodiment shown in FIG.

  A candidate range specifying unit 312c newly provided in this modification specifies a range of information groups extracted by the candidate information extracting unit 312a. More specifically, the candidate range designating unit 312c obtains from the information storage device 200 the past posted information of the user who posted the posted information determined to be missing. Then, the candidate range specifying unit 312c calculates the similarity between the post information determined to be missing information and the past post information, and based on the calculated similarity, Specify.

  For example, it is assumed that post information posted at the time “t = 0” includes text information but does not include image information, and the information loss detection unit 311 detects the loss of image information. In this case, the candidate range specifying unit 312c acquires past posting information posted by the same user before the time “t = 0” from the information storage device 200. For example, it is assumed that these pieces of posted information are posted at times “t = −1”, “t = −2”, “t = −3”, and “t = −4”, respectively.

  Here, the candidate range designation unit 312c performs the text information of “t = 0” and each of “t = −1”, “t = −2”, “t = −3”, and “t = −4”. Calculate similarity with text information. The similarity may be calculated using a conventional procedure. For example, calculation by the vector space method is performed. The vector space method is a method of calculating similarity by expressing documents as vectors in a multidimensional space and comparing the vectors. More specifically, when calculating by the vector space method, each of the text information of “t = 0”, “t = −1”, “t = −2”, “t = −3”, “t = −4”. Is converted to a document vector. From these document vectors, for example, by calculating a cosine similarity for expressing the closeness of the angle formed by the vectors, the similarity of each document vector (that is, the similarity of text information) is calculated. In the present embodiment, image information and text information are used as examples of the first format information and the second format information. Note that the document vector represents an entire vector representing a document. Therefore, for example, the vector representing words representing the document is also included in the document vector.

FIGS. 8A and 8B are diagrams showing an example of the similarity of text information.
In the graph shown in FIG. 8A, “t = 0” is associated with each text information of “t = −1”, “t = −2”, “t = −3”, and “t = −4”. A similarity value with text information is shown. Here, the smaller the similarity value, the more similar to the text information of “t = 0”. As described above, usually, the posted contents are more similar as the posting time is closer (that is, the similarity value is smaller), and are not similar as the posted time is farther away (that is, the similarity value is larger). Tend to). Therefore, the candidate range specifying unit 312c sets a range in which the similarity value satisfies a predetermined condition, here, a range in which the similarity value is lower than a predetermined threshold TH as a range of related information candidates. specify.

  In the case shown in FIG. 8A, the similarity values of “t = −1”, “t = −2”, and “t = −3” are smaller than the threshold value TH. Therefore, the candidate information extraction unit 312a extracts post information posted at times “t = −1”, “t = −2”, and “t = −3” as related information candidates. Then, the candidate information selection unit 312b selects related information from the post information extracted by the candidate information extraction unit 312a.

  In the example shown in FIG. 8B, the text information “t = −1”, “t = −2”, “t = −3”, and “t = −4” is posted immediately before. The difference (absolute value) in the similarity (for example, cosine similarity) with the text information is shown. For example, in the case of the text information “t = −2”, the text information is compared with the text information “t = −1” posted immediately before. That is, the similarity between the text information “t = −1” and the text information “t = 0”, and the similarity between the text information “t = −2” and the text information “t = 0”. The difference is calculated. Usually, when the topic of posting changes, the similarity may change sharply. Accordingly, the candidate range designating unit 312c sets a range in which the similarity value satisfies a predetermined condition, in this case, a range in which the difference in similarity is below a predetermined threshold TH2 as a candidate for related information. specify.

  In the case shown in FIG. 8B, the difference in similarity between “t = −1” and “t = −2” is smaller than the threshold value TH2. Therefore, the candidate information extraction unit 312a extracts post information posted at times “t = −1” and “t = −2” as related information candidates. Then, the candidate information selection unit 312b selects related information from the post information extracted by the candidate information extraction unit 312a.

  As described above, in the present modification, the related information selection unit 312 calculates the degree of similarity with the posted information that is determined to be missing information, and specifies a range to be extracted as a candidate for related information. Therefore, for example, as compared with the case where the extraction range is not specified based on the similarity, it is suppressed that the interpolation is performed by information not related to the posting information.

  In this modification, the similarity of the text information is calculated, but information in a format different from the text information, for example, the similarity of the image information may be calculated. The similarity of image information is calculated by, for example, comparing pixel color values included in the image information.

[Embodiment 2]
<Functional configuration of multi-modal identification device>
Next, the second embodiment will be described.
In the first embodiment, the missing information is interpolated based on the past postings of the user who posted the posting information determined to be missing. On the other hand, in the present embodiment, the missing information is interpolated based on posts from locations that are geographically close to the position of the user who posted the posted information determined to be missing. To do.
In other words, for example, the information posted from a common place such as an amusement park or event venue is likely to have the same (or similar) content. Therefore, in the present embodiment, the missing information is interpolated based on the posted information posted from geographically close locations.

  FIG. 9 is a block diagram illustrating a functional configuration example of the multimodal identification device 300 according to the second embodiment. In the present embodiment, the configuration of the related information selection unit 312 is different from the configuration of the first embodiment shown in FIG. The functions of the information loss detection unit 311, the information interpolation unit 313, the multimodal identification unit 314, and the training model storage unit 315 are the same as in the case of FIG. 4. Therefore, the related information selection unit 312 will be described below as a difference from the configuration of the first embodiment shown in FIG.

In the present embodiment, the related information selection unit 312 includes a candidate position information extraction unit 312d and a candidate information selection unit 312b.
The candidate position information extraction unit 312d expands the extraction range of the posted information to a location that is geographically close, and extracts information groups that are candidates for related information. More specifically, the candidate position information extraction unit 312d acquires the position information of the user who posted the posting information determined to be missing (hereinafter referred to as the target user). Then, from the information storage device 200, post information of another user (hereinafter referred to as “proximity user”) posted from a location close to the position of the target user, in other words, within a predetermined range from the position of the target user. get. The information group acquired here (that is, posted information of the proximity user) is treated as a candidate for related information.

  In addition, the user's position information can exemplify, for example, geotag information given at the time of posting by the user. A geotag is numerical data indicating a position (latitude and longitude) on a map. The candidate position information extraction unit 312d according to the present embodiment specifies the position of the target user based on the information of the geotag included in the post information, or post information posted from a place close to the position of the target user. Can be identified.

  The candidate information selection unit 312b selects related information from the information group extracted by the candidate information extraction unit 312a. More specifically, the candidate information selection unit 312b selects related information from posted information posted from a location close to a position where posted information determined to lack information is posted.

<Related information selection process>
Next, processing for selecting related information will be described in detail. FIGS. 10A and 10B are diagrams for explaining an example of related information selection processing according to the second embodiment.

  In the example shown in FIG. 10A, it is assumed that the posting information posted by the target user is determined to be missing. In this case, the candidate position information extraction unit 312d specifies a user who exists within a predetermined range from the position of the target user as a proximity user. As the predetermined range, for example, a range within a radius Lm around the target user can be exemplified. As a result, the other user A and the other user B are specified as the proximity users. On the other hand, the other user C posts text information and image information, but is excluded from the proximity user because it exists outside the predetermined range.

  In addition, the example illustrated in FIG. 10B illustrates an example of post information of the target user and other users A to C. The posted information of the target user includes text information but does not include image information. Here, when the information loss detection unit 311 detects the loss of image information for the post information of the target user, the candidate position information extraction unit 312d is a proximity user (here, the target user who exists within a predetermined range from the target user position). The posting information of other users A and B) is acquired from the information storage device 200. Then, the candidate information selection unit 312b selects related information from the acquired post information of the proximity user.

  Here, the post information of the other user A does not include image information. On the other hand, the post information of the other user B includes image information. Therefore, the candidate information selection unit 312b selects the image information of the other user B as related information. In this way, the image information of the other user B is interpolated with respect to the posting information of the target user. As a result, the text information of the target user and the image information of the other user B are used as multimodal information for training and identification in the multimodal identification unit 314.

  Further, in the example shown in FIGS. 10A and 10B, there is only another user B who has posted the image information, but there may be cases where a plurality of neighboring users have posted the image information. In such a case, related information is selected from the image information of a plurality of neighboring users according to a predetermined condition.

For example, image information of a neighboring user who is closest to the target user is selected from among a plurality of neighboring users who have posted image information.
Further, for example, it may be determined which neighboring user's image information is selected based on the similarity calculated in the modification of the first embodiment. In this case, for example, the candidate position information extraction unit 312d calculates the similarity between the text information of the target user and the text information of each neighboring user. Then, based on the calculated similarity, image information of a neighboring user whose post content is most similar to the target user is selected.
Furthermore, for example, image information posted at a time closest to the posting time of the target user may be selected from among a plurality of neighboring users who have posted image information.

  In other words, in this embodiment, the distance between the target user and the neighboring user, the similarity between the posted content of the target user and the posted content of the neighboring user, the time difference between the posted time of the target user and the posted time of the neighboring user, etc. Based on the conditions, it is only necessary to determine which neighboring user's posted information of the plurality of neighboring users is used for interpolation.

  As described above, in the present embodiment, the related information selection unit 312 is posted from a location within a predetermined range from the position where the posted information is posted when the lack of information is detected in the posted information. Select related information from the posted information. By interpolating the missing information in this way, the possibility of performing interpolation based on the same (or similar) content / topic as post information is increased.

  Moreover, in this Embodiment, the proximity | contact user who exists within the predetermined range from the object user was specified, and the posting information of the specified proximity | contact user was acquired, For example, a user's contribution close | similar to an object user It is good also as acquiring in order from information. In this case, for example, if image information is not included in the posted information of the target user, the presence or absence of the image information is checked in order from the posted information of the user close to the target user, and the image information is interpolated.

<Procedure for processing by multi-modal identification device>
Next, a processing procedure performed by multimodal identification apparatus 300 according to the present embodiment will be described. A description will be given by dividing into a process at the time of training by the multi-modal identification device 300 and a process at the time of identification. FIG. 11A is a flowchart showing a procedure of processing during training by the multimodal identification apparatus 300 according to the second embodiment. FIG. 11B is a flowchart showing a procedure of processing at the time of identification by the multimodal identification device 300 according to the second embodiment.

First, the processing at the time of training by the multimodal identification device 300 will be described with reference to FIG.
The information loss detection unit 311 acquires post information used for training from the information storage device 200 (step 301). Here, as in step 101 of FIG. 6, posting information acquisition conditions may be set. Next, the information loss detection unit 311 determines whether or not there is missing information in the acquired post information (step 302). Here, as in step 102 of FIG. 6, it is determined whether or not the acquired post information satisfies the missing detection condition.

If it is determined in step 302 that there is no missing information in the posted information (No in step 302), the posted information is output to the multimodal identification unit 314. Then, the process proceeds to step 306.
On the other hand, when it is determined in step 302 that there is missing information in the posted information (Yes in step 302), the posted information is output to the related information selection unit 312. Next, the candidate position information extraction unit 312d of the related information selection unit 312 expands the post information extraction range to a location that is geographically close to obtain post information of other users who are candidates for the related information (step) 303). Next, the candidate information selection unit 312b selects related information from the post information of other users acquired by the candidate position information extraction unit 312d (step 304).

  Next, the information interpolation unit 313 associates the related information selected by the related information selection unit 312 with the post information determined to be missing information, and interpolates the missing information (step 305). . Post information in which the missing information is interpolated is output to the multi-modal identification unit 314. Then, the process proceeds to step 306.

  After a negative determination (No) is made in step 302 or after step 305, the multimodal identification unit 314 performs training based on the posted information acquired from the information missing detection unit 311 or the information interpolation unit 313 (step 306). ). Next, the information loss detection unit 311 determines whether or not all post information acquired from the multimodal identification device 300 in step 301 has been processed (step 307). If there is post information that has not been processed yet (No in Step 307), the process proceeds to Step 302. On the other hand, when all of the posted information has been processed (Yes in step 307), this processing flow ends. In addition, training is complete | finished by repeating this process flow once or several times, and the process of identification is started.

Next, with reference to FIG. 11B, a process at the time of identification by the multimodal identification device 300 will be described.
The information loss detection unit 311 acquires post information to be identified from the information storage device 200 (step 401). Here, as in the case of step 301 in FIG. 11A, a condition for obtaining posted information may be set. The processing of the next step 402 to step 405 is the same as the processing of step 302 to step 305 in FIG.

  After a negative determination (No) is made in step 402 or after step 405, the multimodal identification unit 314 performs identification based on the posted information acquired from the information missing detection unit 311 or the information interpolation unit 313 (step 406). ). Next, the information loss detection unit 311 determines whether or not all post information acquired from the multimodal identification device 300 in step 401 has been processed (step 407). If there is post information that has not been processed yet (No in Step 407), the process proceeds to Step 402. On the other hand, when all of the posted information has been processed (Yes in step 407), this processing flow ends. As described above, the information obtained as a result of the identification is used as marketing information such as advertisement distribution.

  In the first embodiment and the second embodiment, the multimodal identification unit 314 performs the training after performing the training. However, the multimodal identification unit 314 may perform the training and the identification in parallel. good. In this case, for example, the information loss detection unit 311 separately acquires post information for training and post information for identification, and detects a loss of information for each. Then, the multi-modal identification unit 314 updates the training model using the posting information for training, and in parallel with this, identifies the posting information for identification using the training model.

  Moreover, in Embodiment 1 and Embodiment 2, the case where omission has occurred in multimodal information mainly including information of two formats has been described, but multimodal information including information of three or more formats is included. The same processing is performed when there is a loss. For example, for multi-modal information including information of three or more formats, when at least one of the formats is missing, interpolation of the missing information is performed.

  Furthermore, in the first embodiment and the second embodiment, one multi-modal identification device 300 includes an information loss detection unit 311, a related information selection unit 312, an information interpolation unit 313, a multi-modal identification unit 314, and a training model storage unit 315. Although these functions are provided, these functions may be realized by a plurality of devices instead of a single device.

  The program for realizing the embodiment of the present invention can be provided not only by a communication means but also by storing it in a recording medium such as a CD-ROM.

  Although the embodiments of the present invention have been described above, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is clear from the description of the scope of the claims that various modifications or improvements added to the above embodiment are also included in the technical scope of the present invention.

300 ... multimodal identification device, 311 ... information loss detection unit, 312 ... related information selection unit, 312a ... candidate information extraction unit, 312b ... candidate information selection unit, 312c ... candidate range specification unit, 312d ... candidate position information extraction unit, 313 ... information interpolation unit, 314 ... multi-modal identification unit, 315 ... training model storage unit

Claims (6)

Interpolating means for interpolating the information of the missing format by the related information related to the information of the plurality of formats when the information of some formats is missing from the information of the plurality of formats;
An information processing apparatus comprising: a process execution unit that executes at least one of training and identification using the information of the plurality of formats in which the information of the missing format is interpolated. The interpolating unit acquires the information posted within a predetermined time from the time when the missing information has the posted format as the related information, and the information that is not missing among the information of the plurality of formats is posted. The information processing apparatus according to claim 1. The interpolation means has the missing format as the related information, and information posted at a position within a predetermined range from the location where the missing information is posted among the information of the plurality of formats. The information processing apparatus according to claim 1, further comprising: In the interpolation means, when the information of the first format is missing from the information of the plurality of formats and the information of the second format is not missing, the similarity with the information of the second format that is not missing is determined in advance. 4. The information processing apparatus according to claim 2, wherein the information in the first format posted together with other information in the second format that satisfies the specified condition is acquired as the related information. 5. When the information in the second format that is not missing is information in the text format, the interpolation means determines whether the information in the second format that is not missing is based on the similarity of the document vector calculated from the information in the text format. 5. The information processing apparatus according to claim 4, wherein the information in the first format posted together with other information in the text format that satisfies a predetermined degree of similarity is acquired as the related information. On the computer,
A function of interpolating the information of the missing format by the related information related to the information of the plurality of formats when the information of the format of the plurality of formats is missing,
A program for realizing a function of executing processing of at least one of training and identification using the information of the plurality of formats in which the information of the missing format is interpolated.

Images