JP2022013256A - Keyword extraction apparatus, keyword extraction method, and keyword extraction program - Google Patents

Abstract

To provide a keyword extraction apparatus configured to properly extract a keyword related to a video, a keyword extraction method, and a keyword extraction program.SOLUTION: A keyword extraction apparatus 1 includes: a keyword candidate word extraction unit 11 which extracts keyword candidate words from an input text in a video; a mask text generation unit 12 which generates a mask text formed by masking the keyword candidate words in the input text; a mask estimation unit 13 which calculates estimation values by estimating each of the masked keyword candidate words, in the mask text; a high-saliency object determination unit 16 which outputs a class name of an object detected from the video; a relevance score calculation unit 17 which calculates relevance scores between the class name and the keyword candidate words on the basis of the estimation values of each of the masked words; and a keyword output unit 18 which outputs a keyword candidate word having the highest relevance score, as a keyword related to the video.SELECTED DRAWING: Figure 1

Description

The present invention relates to a device, a method and a program for extracting keywords spoken by a robot.

Conventionally, technology has been studied in which a robot that watches TV or the like with a person speaks according to the contents of a program. Such a robot extracts keywords from program information according to a predetermined rule.

For example, Patent Document 1 proposes a method of extracting a keyword from a subtitle sentence of a program. Further, Non-Patent Document 1 proposes a method of detecting an object from an image and at the same time estimating the prominence of the detected object and extracting keywords according to the prominence of the object.
In addition to these methods, attempts are being made to extract appropriate keywords by using cloud services such as voice recognition, person recognition, object recognition, and character recognition in parallel.

Japanese Unexamined Patent Publication No. 2018-190077

Yuta Hagio et al., "Design and Prototyping of Communication Robots for Simultaneous Viewing Experiments of Humans and Robots", 2019 Annual Meeting of the Society of Video and Information Media.

However, it is difficult to determine which of the keywords extracted by a plurality of methods is suitable for the program content, and conventionally, it has been randomly used from the obtained keywords.
In addition, although it can be expected that an object suitable for the content of the program will be selected by using the saliency, the types of objects that can be detected are limited to about several hundred classes, and as a result, only the same keywords are used. Is extracted, and only the same utterance sentence is generated.

An object of the present invention is to provide a keyword extraction device, a keyword extraction method, and a keyword extraction program capable of appropriately extracting keywords related to video.

The keyword extraction device according to the present invention has a keyword candidate word extraction unit that extracts words registered in the keyword database as keyword candidate words from an input sentence accompanying a video, and a mask that masks the keyword candidate words in the input sentence. A mask sentence generation unit that generates a sentence, a mask estimation unit that calculates an estimated value for each word that estimates the keyword candidate word in the masked part by a learning model, and an object detected from the video. Based on the object determination unit that outputs the class name of the above and the estimated value for each word of the masked portion calculated by the mask estimation unit, the keyword candidate word for each of the class name and the masked portion. It is provided with a relevance score calculation unit for calculating the relevance score, and a keyword output unit for outputting the keyword candidate word having the highest relevance score as a keyword related to the video.

The mask estimation unit may calculate the estimated value in a sentence obtained by adding a past input sentence to the mask sentence.

The mask sentence generation unit generates mask sentences that mask only one of the keyword candidate words included in the input sentence, as many as the number of the keyword candidate words, and the mask estimation unit generates one mask sentence. In sentences containing only, the estimated value for each word may be calculated.

The mask sentence generation unit may generate the mask sentence from the latest input sentence at the timing when the object is detected from the video.

The keyword output unit does not have to output the keyword related to the video when the maximum value of the relevance score does not reach a predetermined threshold value.

The keyword extraction device includes a saliency estimation unit that gives a saliency score to each pixel of the video by a learning model, and the object determination unit is the object among a plurality of objects detected from the video. The class name of the object in the region with the highest evaluation based on the saliency score may be output.

The keyword extraction device collates a camera image to which the coordinates of the user's viewpoint position are added with the video, thereby imparting a viewpoint position estimation having a predetermined distribution to each pixel of the video. The object determination unit may output the class name of the object in the region having the highest evaluation based on the gazing point score among the plurality of objects detected from the video.

The relevance score calculation unit calculates the cosine similarity between the distributed expression vector for the class name and the distributed expression vector of each of a predetermined number of words whose estimated value is higher, and calculates the average value as the relevance score. You may.

The relevance score calculation unit may acquire the estimated value of the same word for each of a plurality of representative object names registered in advance corresponding to the class name, and calculate the average value as the relevance score. ..

The keyword extraction method according to the present invention includes a keyword candidate word extraction step of extracting a word registered in a keyword database as a keyword candidate word from an input sentence accompanying a video, and a mask masking the keyword candidate word in the input sentence. A mask sentence generation step for generating a sentence, a mask estimation step for calculating an estimated value for each word that estimates the keyword candidate word in the masked part by a learning model, and an object detected from the video. Based on the object determination step that outputs the class name of the above and the estimated value for each word of the masked part calculated by the mask estimation unit, the keyword candidate word for each of the class name and the masked part. The computer executes a relevance score calculation step for calculating the relevance score and a keyword output step for outputting the keyword candidate word having the highest relevance score as a keyword related to the video.

The keyword extraction program according to the present invention is for operating a computer as the keyword extraction device.

According to the present invention, keywords related to video can be appropriately extracted.

It is a figure which shows the functional structure of the keyword extraction apparatus in 1st Embodiment. It is a flowchart which illustrates the flow of the keyword extraction method in 1st Embodiment. It is a figure which illustrates the relationship between the keyword candidate word and the detected object in 1st Embodiment. It is a figure which illustrates the procedure of determining a keyword by the relevance score in 1st Embodiment. It is a flowchart which illustrates the flow of the keyword extraction method in 2nd Embodiment. It is a figure which shows the functional structure of the keyword extraction apparatus in 3rd Embodiment.

[First Embodiment]
Hereinafter, the first embodiment of the present invention will be described.
In the first embodiment, there is provided a keyword extraction device 1 that is incorporated into a robot or the like that watches a broadcast program accompanied by a video such as a television together with a human being and is used for utterance generation.
The keyword extraction device 1 extracts the most prominent object from the objects detected from the video of the broadcast program, and extracts the keyword candidate word from the voice or the subtitle, and the most prominent object and the keyword. By ranking the degree of relevance score with the candidate word, keywords related to the video are output.

FIG. 1 is a diagram showing a functional configuration of the keyword extraction device 1 in the present embodiment.
The keyword extraction device 1 is an information processing device provided with various interfaces in addition to a control unit and a storage unit, and the control unit executes software (keyword extraction program) stored in the storage unit to execute the present embodiment. Various functions are realized.

The control unit of the keyword extraction device 1 includes a keyword candidate word extraction unit 11, a mask sentence generation unit 12, a mask estimation unit 13, a prominence estimation unit 14, an object detection unit 15, and a highly prominent object determination unit 16. (Object determination unit), a relevance score calculation unit 17, and a keyword output unit 18 are provided.

In addition to the keyword extraction program, the storage unit of the keyword extraction device 1 includes a keyword database (DB) 21, an input sentence memory 22, a word estimation model 23, a saliency estimation model 24, and an object detection model 25. It also includes a distributed representation vector database (DB) 26.

The keyword candidate word extraction unit 11 divides an input sentence such as a subtitle sentence input along with the video or a recognition result of the television sound into words. After that, the keyword candidate word extraction unit 11 confirms whether or not a word registered in the keyword DB 21 prepared in advance exists in the divided word group, and if it exists, this word is referred to as "keyword candidate". It is extracted as a word and stored in the input sentence memory 22 in association with the input sentence.

The mask sentence generation unit 12 extracts a sentence including the latest input sentence from the input sentence memory 22 within the upper limit of the input to the mask estimation unit 13 in the subsequent stage (for example, 512 words), and determines a keyword candidate word in the latest input sentence. A masked mask statement is generated by replacing it with a character string of (for example, [MASK]).
It should be noted that this process is executed periodically (for example, at a cycle of about 5 seconds) at a timing synchronized with the process of the object detection unit 15.

At this time, the mask sentence generation unit 12 generates a mask sentence that masks only one of the keyword candidate words included in the latest input sentence, and if there are a plurality of keyword candidate words in the input sentence, the number of the keyword candidate words is the same. Converts the pattern and generates multiple mask statements.
After that, the mask sentence generation unit 12 provides a sentence composed of a plurality of input sentences, a keyword candidate word, and a mask sentence to the mask estimation unit 13.

The mask estimation unit 13 calculates the estimated value for each word in which the keyword candidate word of the masked portion is estimated in the sentence including the mask sentence by the learned word estimation model 23.
The word estimation model 23 may be, for example, a BERT model that has been pre-learned in advance, and the words in the masked sentence are estimated from the surrounding sentences by the pre-learning task “Masked LM”. The BERT model is proposed in the following document A, and the estimation result is an estimated value (value of 0 to 1) of each word (for example, about 30,000 words) included in the vocabulary of the model, and all words. The sum of the estimated values of is the list of 1).

Reference A: J. Devlin et al. , "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", NAACL-HLT 2019.

Here, when a plurality of mask sentences are input, the mask estimation unit 13 processes each mask sentence in order.
That is, the mask estimation unit 13 calculates an estimated value for each word in which the masked portion is estimated in a sentence containing only one mask sentence.

The saliency estimation unit 14 executes the saliency estimation process on the captured image of the input video by using the saliency estimation model 24 that has been learned in advance.
As the saliency estimation model 24, for example, a method of computer-implementing the physiological model proposed in Document B or a method using deep learning proposed in Document C can be applied, and the method is captured as an output. A saliency score estimated in the range of 0 to 1 is given to each pixel of the image.

Reference B: L. Itti et al. , "A model of salience-based visual attention for rapid sense analysis", PAMI 1998.
Reference C: Q. How et al. , “Deeply Supervised Salience Object Detection With Short Connections”, PAMI 2019.

The object detection unit 15 periodically (for example, at a cycle of about 5 seconds) executes an object detection process on the captured image of the input video by using the object detection model 25 that has been learned in advance.
For example, the learning method proposed in Documents D, E, and F can be applied to the object detection model 25, and as an output, the coordinate information (rectangular area) of each of the plurality of detected objects and the object category can be obtained. The class name to indicate (eg, human, dog, cake, etc.) is obtained.

Reference D: S. Ren et al. , "Faster R-CNN: Towers Real-Time Object Detection with Region Proposal Network", NIPS 2015.
Reference E: J. Redmon et al. , "You Only Look Object: Unified, Real-Time Object Detection", CVPR2016.
Reference F: W. Liu et al. , "SSD: Single Shot MultiBox Detector", ECCV2016.

The high-saliency object determination unit 16 uses the estimation result by the saliency estimation unit 14 and the detection result by the object detection unit 15, and the evaluation based on the saliency score is the most among the plurality of objects detected from the video. Determine the most prominent object in the high region.
Specifically, the high-saliency object determination unit 16 calculates the average value of the saliency scores of the pixels in the rectangular region of the detected object, and determines the object with the highest calculated value as the "high-saliency object". And output the class name of this object.

The relevance score calculation unit 17 is based on the word-by-word estimation value of the masked portion calculated by the mask estimation unit 13, and the class name and the masked portion output from the high-saliency object determination unit 16 are respectively. Calculate the relevance score with the keyword candidate word. That is, the relevance score calculation unit 17 calculates the relevance score for selecting the word most closely related to the highly prominent object from the keyword candidate words.

Specifically, the relevance score calculation unit 17 refers to the distributed expression vector DB 26 that has been learned in advance, and converts the class name of the highly prominent object into a distributed expression vector and a predetermined number having an estimated value higher (for example,). The cosine similarity with the distributed representation vector obtained by converting each of the words (about 10) is calculated. Then, the relevance score calculation unit 17 uses the calculated average value of the cosine similarity as the relevance score between the highly prominent object and the keyword candidate word.
As the distributed expression of words, existing methods such as Word2Vec or FastText can be used.

The keyword output unit 18 outputs the keyword candidate word having the highest relevance score calculated by the relevance score calculation unit 17 as a keyword related to the input video.
Here, the keyword output unit 18 may not output the keyword related to the video when the maximum value of the relevance score does not reach a predetermined threshold value.

FIG. 2 is a flowchart illustrating the flow of the keyword extraction method in the present embodiment.
In this example, the subtitles and the video constituting the television program and the like are input in parallel in steps S1 and S5, respectively.

In step S1, the control unit of the keyword extraction device 1 acquires the subtitle text of the program being played as an input text.

In step S2, the keyword candidate word extraction unit 11 divides the subtitle sentence into words.
In step S3, the keyword candidate word extraction unit 11 extracts the keyword candidate word from the divided words.
In step S4, the keyword candidate word extraction unit 11 stores the subtitle sentence and the keyword candidate word in the input sentence memory 22. After that, the process returns to step S1.

In step S5, the control unit in the keyword extraction device 1 acquires the video data of the program being played.
In step S6, the control unit captures an image from the acquired video data.

In step S7, the object detection unit 15 executes a process of detecting an object from the captured image.
In step S8, the control unit determines whether or not the object has been detected. If this determination is YES, steps S9 and S12 are executed in parallel, and if the determination is NO, the process returns to step S5.

In step S9, the mask sentence generation unit 12 takes out a subtitle sentence from the input sentence memory 22 and creates a list of a plurality of patterns of mask sentences in which each of the keyword candidate words is masked.
In step S10, the mask estimation unit 13 estimates the word masked in the mask sentence.
In step S11, the relevance score calculation unit 17 selects a predetermined number of words having a higher estimated value and converts these words into a distributed expression vector. After that, the process proceeds to step S15.

In step S12, the saliency estimation unit 14 performs saliency estimation processing for each pixel of the captured image and gives a saliency score.
In step S13, the highly prominent object determination unit 16 determines the highly prominent object having the highest prominence among the detected objects.
In step S14, the relevance score calculation unit 17 converts the determined class name of the highly prominent object into a distributed representation vector.

In step S15, the relevance score calculation unit 17 calculates the cosine similarity between the distributed expression vector of each estimated word and the distributed expression vector of the class name of the highly prominent object, and sets the average value as the relevance score. do.
In step S16, the keyword output unit 18 determines and outputs the keyword candidate word of the masked portion having the maximum relevance score as a keyword related to the highly prominent object.

FIG. 3 is a diagram illustrating the relationship between the keyword candidate word and the detected object in the present embodiment.
In this example, two objects are detected from the captured image of the program, and the class names "human" and "cake" are obtained, respectively.

Also, at this time, the input sentence "Mr. Tanaka's lunch today is French toast" is obtained from the subtitles of the program. Two keyword candidate words, "Tanaka" and "French toast", are extracted from this subtitle.

Here, the keyword candidate word obtained from the subtitle sentence is a specific name, but on the other hand, since the class name of the detected object is more abstract, the two names often do not match.
Therefore, when "cake" is a highly prominent object, whether the corresponding specific keyword is "Tanaka" or "French toast" is determined by the relevance score.

FIG. 4 is a diagram illustrating a procedure for determining a keyword based on the relevance score in the present embodiment.
A mask sentence masking "Tanaka", which is one of the keyword candidate words of the subtitle sentence "Mr. Tanaka's lunch today is French toast.", And a mask sentence masking "French toast" are generated, and each masked part. Word is estimated.

In the "Tanaka" section, for example, the estimated values of words such as "Sato", "Suzuki", and "Watanabe" are calculated high, and in the "French toast" section, for example, "bento", "rice ball", and "sandwich". The estimated value of such words is calculated high.
At this time, the highly prominent object "cake" is more similar to "lunch box", "rice ball", "sandwich", etc. than to "Sato", "Suzuki", "Watanabe", etc. Is high, so "French toast", which is a keyword candidate word for the corresponding masked part, is determined as a keyword related to the program video.

According to the present embodiment, the keyword extraction device 1 sets the class name of the object detected from the video and the keyword candidate word based on the estimated value of the word estimated when the keyword candidate word in the input sentence is masked. Calculate the relevance score.
As a result, the keyword extraction device 1 can appropriately extract important words related to the video as keywords from the input sentence such as the subtitle sentence.

The keyword extraction device 1 estimates the masked portion with a sentence obtained by adding a past input sentence to the latest mask sentence.
As a result, for example, in the mask sentence "I ordered [MASK] today.", It is difficult to estimate the mask location, but the input "I entered because there was a Chinese restaurant that looked delicious." By using the sentence with the sentence added, it can be accurately estimated that the masked part is food, and more specifically, it is a word of Chinese food.
In this way, the keyword extraction device 1 can improve the accuracy of the estimation result and can appropriately evaluate the degree of association with the detected object.

The keyword extraction device 1 can improve the accuracy of the estimation result for the masked part by estimating the word of the masked part in the sentence in which only one of the keyword candidate words is masked, and the degree of relevance to the detected object is appropriate. Can be evaluated.

The keyword extraction device 1 can appropriately extract highly relevant keywords by generating a mask sentence from the latest input sentence accompanying the video in synchronization with the timing when the object is detected.

By not outputting the keyword when the maximum value of the relevance score is less than the threshold value, the keyword extraction device 1 can suppress the output of an inappropriate keyword when the input sentence is not related to the video.

The keyword extraction device 1 assigns a saliency score to each pixel of the image, and determines an object in the region having the highest evaluation based on the saliency score as a high saliency object.
As a result, the keyword extraction device 1 can appropriately output important keywords related to the most prominent object in the video.

The keyword extraction device 1 calculates the relevance score by the cosine similarity by converting the class name of the object and the estimated word into a distributed expression vector.
As a result, the keyword extraction device 1 can efficiently calculate an appropriate relevance score.

[Second Embodiment]
Hereinafter, a second embodiment of the present invention will be described.
In the second embodiment, the function of the relevance score calculation unit 17 is different from that in the first embodiment, and the representative object database (DB) 27 is provided in place of the distributed representation vector DB 26.

For each class detected by the object detection unit 15, an object name belonging to this class is prepared in advance and registered in the representative object DB 27.
For example, a plurality of representative objects (for example, about 10) such as dogs, cats, horses, sheep, etc. are registered in the class "animal".

The relevance score calculation unit 17 acquires the estimated value of the same word by the mask estimation unit 13 for each of the plurality of representative object names belonging to the class selected as the high-saliency object, and sets the average value as the high-saliency object and the keyword. Calculated as a relevance score with the candidate word.

The relevance score calculation unit 17 ranks the relevance score for each keyword candidate word by executing this calculation for each keyword candidate word, and the output unit outputs the keyword candidate word having the maximum relevance score. do.

FIG. 5 is a flowchart illustrating the flow of the keyword extraction method in the present embodiment.
Steps S1 to S10 are the same as in the first embodiment (FIG. 2), and words for each masked keyword candidate word in the subtitle sentence are estimated according to the detection of the object from the video.
After step S10, the process proceeds to step S15a.

In step S12, the saliency estimation unit 14 performs saliency estimation processing for each pixel of the captured image and gives a saliency score.
In step S13, the highly prominent object determination unit 16 determines the highly prominent object having the highest prominence among the detected objects.
In step S14a, the relevance score calculation unit 17 extracts representative objects belonging to the determined class of highly prominent objects.

In step S15a, the relevance score calculation unit 17 extracts the estimated value of the same word as each representative object from the estimated value of the masked word, and sets the average value as the relevance score.
In step S16, the keyword output unit 18 determines and outputs the keyword candidate word of the masked portion having the maximum relevance score as a keyword related to the gaze object.

According to the present embodiment, the keyword extraction device 1 acquires an estimated value of the same word for each of a plurality of representative object names registered in advance corresponding to the class name of the detected object, and sets the average value as the degree of relevance. Calculated as a score.
Therefore, the keyword extraction device 1 can appropriately evaluate the degree of relevance using the estimated value of the word by pre-characterizing each class in the object detection by the representative object included in the vocabulary of the word estimation model.

[Third Embodiment]
Hereinafter, a third embodiment of the present invention will be described.
The keyword extraction device 1a of the third embodiment uses the user's gaze point instead of the saliency estimation method of the first embodiment. The saliency score is the result of estimating the point where the user's gaze is generally likely to be gathered, while the keyword extraction device 1a actually pays attention to the user by estimating the user's gaze point using the eye tracker. Extract keywords related to the object you are in.

FIG. 6 is a diagram showing a functional configuration of the keyword extraction device 1a according to the present embodiment.
In the third embodiment, the saliency estimation unit 14 of the first embodiment is the viewpoint detection unit 14a and the viewpoint position estimation unit 14b, and the highly prominent object determination unit 16 is the gaze object determination unit 16a (object determination unit), respectively. It has been replaced.

The viewpoint detection unit 14a detects the user's viewpoint position from the eye tracker worn by the user (program viewer). The viewpoint position can be detected by various conventional methods such as a method of photographing the eyeball with an infrared camera and estimating the viewpoint position from the movement thereof.
The viewpoint detection unit 14a provides the viewpoint position estimation unit 14b with data in which the coordinate information of the viewpoint position is added to the image of the camera mounted on the eye tracker as the detection result.

The viewpoint position estimation unit 14b collates the camera image to which the coordinates of the user's viewpoint position acquired from the viewpoint detection unit 14a are added with the program image, so that the note of a predetermined distribution is given to each pixel of the program image. Give a viewpoint score.

Here, since the camera image is an image close to the user's field of view, including outside the frame of the television on which the program image is presented, the user's viewpoint is located on the program image, or the program image is viewed. It is necessary to estimate whether or not it is.
First, the viewpoint position estimation unit 14b captures the program image and the camera image of the eye tracker, respectively, and estimates the area of the program image in the camera image of the eye tracker. For the estimation of the region, for example, an image feature amount corresponding to enlargement, reduction, and rotation of the image, such as the ORB feature amount proposed in Document G, is used.

Reference G: E. Ruby et al. , "ORB: An effective alternate to SIFT or SURF", ICCV2011.

When the program image exists in the camera image and the area of the program image can be detected, the viewpoint position estimation unit 14b converts the image of the detected area into a rectangle by homography conversion. Further, the viewpoint position estimation unit 14b also maps the coordinates of the viewpoint position on the rectangle after the homography transformation, and gives each pixel a gaze point score according to a normal distribution centered on this point.
As a result, the result in which the gazing point score of each pixel in the program video is estimated in the range of 0 to 1 can be obtained.

The gaze object determination unit 16a uses the estimation result by the viewpoint position estimation unit 14b and the detection result by the object detection unit 15, and has the highest evaluation based on the gaze point score among the plurality of objects detected from the video. Determines the object that is presumed to be being watched by the user.
Specifically, the gaze object determination unit 16a calculates and calculates the average value of the gaze point scores of the pixels in the rectangular region of the detected object, similarly to the high-saliency object determination unit 16 of the first embodiment. The object with the highest value is determined as the "gaze object", and the class name of this object is output.

The relevance score calculation unit 17 is a keyword candidate for each of the class name and the masked part output from the gaze object determination unit 16a based on the word-by-word estimated value of the masked part calculated by the mask estimation unit 13. Calculate the relevance score with the word. That is, the relevance score calculation unit 17 calculates the relevance score for selecting the word most closely related to the gaze object from the keyword candidate words.

As for the specific calculation method of the relevance score, either the method using the cosine similarity using the distributed representation vector DB 26 in the first embodiment or the method using the representative object using the representative object DB 27 in the second embodiment is applied. It is possible.

In the flow of the keyword extraction method in this embodiment, the prominence estimation in step S12 of the first embodiment (FIG. 2) or the second embodiment (FIG. 5) is used as the viewpoint position estimation, and the highly prominent object in step S13. Is replaced with the determination of the gaze object.

According to the present embodiment, the keyword extraction device 1a gives a gaze point score by estimating the viewpoint position of the user instead of the prominence score in the image.
As a result, the keyword extraction device 1 can appropriately output important keywords related to the object that the user is actually gazing at.

Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. Moreover, the effects described in the above-described embodiments are merely a list of the most preferable effects resulting from the present invention, and the effects according to the present invention are not limited to those described in the embodiments.

In the above-described embodiment, the keyword extraction device 1 has been described as being incorporated in the robot, but the present invention is not limited to this, and the keyword extraction device 1 is arranged outside the robot and is connected to the robot by wire or wirelessly or via a network. May be done.
Further, various databases, learning models, and the like are configured to be provided in the keyword extraction device 1, but the present invention is not limited to this, and they may be arranged in an external server such as a cloud.

In the present embodiment, the configuration and operation of the keyword extraction device 1 have been mainly described, but the present invention is not limited to this, and each component may be provided and configured as a method or a program for extracting keywords. ..

Further, even if a program for realizing the function of the keyword extraction device 1 is recorded on a computer-readable recording medium, the program recorded on the recording medium is read into a computer system and executed. good.

The term "computer system" as used herein includes hardware such as an OS and peripheral devices. Further, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, or a CD-ROM, and a storage device such as a hard disk built in a computer system.

Furthermore, a "computer-readable recording medium" is a communication line that dynamically holds a program for a short period of time, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. It may also include a program that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or a client in that case. Further, the above program may be for realizing a part of the above-mentioned functions, and may be further realized for realizing the above-mentioned functions in combination with a program already recorded in the computer system. ..

1, 1a Keyword extraction device 11 Keyword candidate word extraction unit 12 Mask sentence generation unit 13 Mask estimation unit 14 Severity estimation unit 14a Viewpoint detection unit 14b Viewpoint position estimation unit 15 Object detection unit 16 Highly prominent object determination unit 16a Gaze object determination Part 17 Relevance score calculation part 18 Keyword output part 21 Keyword database 22 Input sentence memory 23 Word estimation model 24 Severity estimation model 25 Object detection model 26 Distributed representation vector database 27 Representative object database

Claims (11)

A keyword candidate word extraction unit that extracts words registered in the keyword database as keyword candidate words from the input sentences that accompany the video,
A mask sentence generation unit that generates a mask sentence that masks the keyword candidate word in the input sentence,
In the mask sentence, a mask estimation unit that calculates an estimated value for each word that estimates the keyword candidate word in the masked part by a learning model, and a mask estimation unit.
An object determination unit that outputs the class name of the object detected from the video, and
Relevance score for calculating the relevance score between the class name and the keyword candidate word in each of the masked locations based on the estimated value for each word of the masked portion calculated by the mask estimation unit. Calculation unit and
A keyword extraction device including a keyword output unit that outputs the keyword candidate word having the highest relevance score as a keyword related to the video. The keyword extraction device according to claim 1, wherein the mask estimation unit calculates the estimated value in a sentence obtained by adding a past input sentence to the mask sentence. The mask sentence generation unit generates as many mask sentences as the number of the keyword candidate words by masking only one of the keyword candidate words included in the input sentence.
The keyword extraction device according to claim 1 or 2, wherein the mask estimation unit calculates an estimated value for each word in a sentence containing only one mask sentence. The keyword extraction device according to any one of claims 1 to 3, wherein the mask sentence generation unit generates the mask sentence from the latest input sentence at the timing when an object is detected from the video. The keyword extraction device according to any one of claims 1 to 4, wherein the keyword output unit does not output a keyword related to the video when the maximum value of the relevance score does not reach a predetermined threshold value. A saliency estimation unit that gives a saliency score to each pixel of the video by a learning model is provided.
The object determination unit according to any one of claims 1 to 5, wherein the object determination unit outputs the class name of the object in the region having the highest evaluation based on the saliency score among the plurality of objects detected from the video. Keyword extractor. It is provided with a viewpoint position estimation unit that gives a gaze point score of a predetermined distribution to each pixel of the image by collating the camera image to which the coordinates of the viewpoint position of the user are added with the image.
The object determination unit according to any one of claims 1 to 5, which outputs the class name of the object in the region having the highest evaluation based on the gazing point score among the plurality of objects detected from the video. Keyword extractor. The relevance score calculation unit calculates the cosine similarity between the distributed expression vector for the class name and the distributed expression vector of each of a predetermined number of words whose estimated value is higher, and calculates the average value as the relevance score. The keyword extraction device according to any one of claims 1 to 7. The relevance score calculation unit acquires the estimated value of the same word for each of a plurality of representative object names registered in advance corresponding to the class name, and calculates the average value as the relevance score. The keyword extraction device according to any one of claims 7. A keyword candidate word extraction step that extracts words registered in the keyword database as keyword candidate words from the input sentences that accompany the video,
A mask sentence generation step for generating a mask sentence that masks the keyword candidate word in the input sentence, and
In the mask sentence, a mask estimation step of calculating an estimated value for each word in which the keyword candidate word of the masked part is estimated by a learning model, and a mask estimation step.
An object determination step that outputs the class name of the object detected from the video, and
Relevance score for calculating the relevance score between the class name and the keyword candidate word in each of the masked locations based on the estimated value for each word of the masked portion calculated in the mask estimation step. Calculation steps and
A keyword extraction method in which a computer executes a keyword output step of outputting the keyword candidate word having the highest relevance score as a keyword related to the video. A keyword extraction program for operating a computer as the keyword extraction device according to any one of claims 1 to 9.

Images