JPWO2009113505A1 - Video segmentation apparatus, method and program - Google Patents

Abstract

Provided is a video dividing device capable of appropriately dividing video data into topics according to meaning contents. The video segmentation device refers to a text or a word string (hereinafter referred to as “words”) included in the text with reference to text associated with the video and provided with a playback position in the video. A subject determination unit that determines whether or not to represent a subject, a subject weighting unit that weights a word or the like that is determined to represent the subject, a weight that is greater than a weight for the other, a text based on the weight And a video dividing unit that divides the video by dividing the video (FIG. 1).

Description

[Description of related applications]
The present invention is based on the priority claim of Japanese Patent Application: Japanese Patent Application No. 2008-066221 (filed on March 14, 2008), the entire contents of which are incorporated herein by reference. Shall.

  The present invention relates to a video segmentation apparatus, method, and program, and more particularly, to a video segmentation apparatus, method, and program that divide video data into semantically organized units.

  In recent years, a large amount of video data has been distributed, and the technology for dividing video data into semantic units (hereinafter referred to as “topics”) is important as a technology for improving the listability and searchability of video data. Is increasing.

  As a typical method for dividing video data into topics, there is a method in which a text division technique is applied to text representing the content of an utterance included in video data (hereinafter referred to as “uttered text”). As the utterance text, text obtained by voice recognition of the utterance included in the video data can be used. In addition, when the video data is a television program, caption information (closed caption) may be used. Such utterance text is given reproduction position information in a video such as an elapsed time from the beginning of the video data. Therefore, the video data can be divided by dividing the text.

  In general, in the text segmentation technique, a word boundary that changes in meaning content in a text is obtained by analyzing a word or a word string (hereinafter referred to as “word or the like”) constituting an input text. When performing such processing, it is better to give a greater weight to words that are strongly related to the semantic content of each topic in the input text than to treat all words equally. Can be divided into topics more appropriately.

  A method using IDF (Inverse Document Frequency) is known as a method of assigning weights to important words in text processing. The IDF of the word Wi is obtained by collecting a large amount of documents in advance and then IDF (Wi) = log (total number of documents / number of documents including the word Wi). That is, the IDF increases as the word appears only in a small number of documents. IDF is a word weighting method based on the assumption that “a word that appears only in a small number of documents when a large number of documents are collected after a certain document unit is determined” is an important word.

  An example of word weighting by IDF in text division is described in Non-Patent Document 1. In Non-Patent Document 1, an analysis interval having a certain width is set for each part of the input text, and a topic vector having the importance of each word in the analysis interval as an element is obtained for each analysis interval. At this time, the importance of the word Wi in a certain analysis section is obtained by TF (Wi) × IDF (Wi) where the appearance frequency of the word Wi in the analysis section is TF (Wi). That is, the topic vector is a value obtained by correcting the appearance frequency distribution of words in the analysis section by IDF. After obtaining the topic vector in this way, a cosine similarity series between topic vectors in adjacent analysis sections is obtained, and the minimum point of the similarity is detected as a topic boundary point.

International Publication No. 2004/095374 Pamphlet Utsumi, Fujii, Tanaka, “Text Segmentation Method with Variable Analysis Interval Length”, Proc. Of the 12th Annual Conference of the Language Processing Society, p. 117-120, 2006. K. Kimura, I. et al. Yamada, H .; Sumiyoshi, N .; Yagi, “Automatic Generation of a Multimedia Encyclopedia from TV Program by Using Closed Captions and Detecting Principal Video Objects,” 873-880, 2006.

  The entire disclosure of the above patent documents and non-patent documents is incorporated herein by reference. The following analysis is given by the present invention.

  However, the above prior art has the following problems.

  That is, when video data is divided by dividing the text of the utterance text, it is not always possible to appropriately divide the video data into topics according to the semantic content simply by weighting words with IDF. There is. The reason is as follows.

  The IDF of a word is an index that represents the general importance of the word to the last, and does not necessarily represent the strength of relevance between the word and the semantic content of each topic. For example, the IDF has the same value regardless of the scene where the word “Month white butterfly” appears. However, in reality, the strength of the relevance between the white butterfly and the semantic content of the topic is considered to differ depending on the scene in which the word “Butterfly” appears. Therefore, in order to divide the topic according to the semantic content, the weight of the word “Monthrocho” is increased when the white butterfly is actually the subject of the topic; It is desirable to reduce the weight. However, IDF does not perform such weighting.

  Therefore, it is an object to provide a video dividing device, a video dividing method, and a video dividing program that can appropriately divide video data into topics according to meaning contents.

  The video segmentation device according to the first aspect of the present invention refers to a text or a word string (hereinafter referred to as “words”) included in the text with reference to text associated with the video and having a playback position attached to the video. And a subject determination unit that determines whether or not a subject included in the video is represented, and a weight that is determined to represent the subject among words or the like is larger than a weight for other subjects. A subject weighting unit; and a video dividing unit that divides the video by dividing text based on the weighting.

  The video segmentation method according to the second aspect of the present invention refers to a word or a word string (hereinafter referred to as “a word sequence”) included in a text by referring to text associated with the video and having a playback position attached to the video. A word determination process for determining whether or not a word or the like represents a subject included in the video, and a weight that is greater than a weight for other words determined to represent the subject. A subject weighting step for performing the above and a video dividing step for dividing the video by dividing the text based on the weighting.

  The program according to the third aspect of the present invention refers to a text or a word string (hereinafter referred to as “word or the like”) included in the text with reference to the text associated with the video and the playback position in the video. ) Represents a subject included in the video, and subject weighting processing for weighting a word or the like that is determined to represent the subject larger than the weights for other subjects. And a video dividing process for dividing the video by dividing the text based on the weighting.

  According to the video dividing apparatus according to the present invention, video data can be appropriately divided into topics according to the semantic content. The video dividing device determines whether or not a word or the like included in the text represents a subject shown in the video with reference to text associated with the video and provided with a playback position in the video. In addition, the video dividing device divides the video by increasing the weighting of a word or the like determined to represent the subject and dividing the text based on the weighting.

It is a block diagram which shows the structure of the 1st Example of this invention. It is a block diagram which shows the structure of the to-be-photographed object determination part in 1st Example of this invention. It is a flowchart which shows operation | movement of the 1st Example of this invention. It is a figure explaining the specific example of the text memory | storage part in 1st Example of this invention. It is a figure explaining the specific example of the to-be-recognized result memory | storage part in 1st Example of this invention. It is a figure explaining the specific example of the determination result of the to-be-photographed object determination part in 1st Example of this invention. It is a figure explaining the specific example of the weighted text memory | storage part in 1st Example of this invention. It is a figure explaining an example of operation | movement of the image | video division | segmentation part in 1st Example of this invention. It is a figure explaining the specific example of operation | movement of the 1st Example of this invention. It is a figure explaining the specific example of operation | movement of the 1st Example of this invention. It is a block diagram which shows the structure of the 2nd Example of this invention.

Explanation of symbols

11 Video Data Storage Unit 12 Text Storage Unit 13 Subject Determination Unit 14 Subject Weighting Unit 15 Weighted Text Storage Unit 16 Video Division Unit 17 Division Result Storage Unit 18 Video Viewing Unit 31 Video Division Program 32 Data Processing Device 33 Storage Device 130 Subject Recognition unit 131 Object recognition unit 132 Face image recognition unit 133 Character recognition unit 134 Subject extraction unit 135 Subject recognition result storage unit 136 Verification unit 331 Video data storage unit 332 Text storage unit 333 Weighted text storage unit 334 Division result storage unit 335 Subject Recognition result storage

  In the video dividing apparatus according to the first development form, the subject determination unit may determine whether or not a word or the like represents a subject that appears in a predetermined range of the video with respect to the playback position. preferable.

  In the video dividing apparatus according to the second development form, when the subject determination unit determines that the word or the like represents the subject shown in the video, the subject weighting unit calculates the reliability that the word or the like represents the subject. It is preferable to weight a word or the like with higher reliability.

  In the video dividing apparatus according to the third development form, when the subject determination unit determines that a word or the like represents a subject in the video, the importance level in the video of the subject is determined, and the subject weighting unit is important It is preferable to weight a word or the like with a higher degree.

  In the video dividing apparatus according to the fourth development form, it is preferable that the subject determination unit determines the importance of the subject according to the ratio of the subject to the video.

  In the video dividing apparatus according to the fifth embodiment, the subject determination unit compares the subject and the subject recognition result text with the subject recognition unit that recognizes the subject reflected in the video and outputs the subject recognition result text. It is preferable to further include a matching unit that determines whether or not a word or the like represents a subject that is shown in the video.

  In the video dividing apparatus according to the sixth development form, it is preferable that the subject includes an object, a face image, or a character.

  In the video dividing apparatus according to the seventh development form, the object recognition unit has an object recognition unit that recognizes an object, a face image recognition unit that recognizes a face image, a character recognition unit that recognizes a character, a word included in the text, and the like It is preferable to include at least one of a subject extracting unit that extracts a word representing a subject from the subject.

  In the eighth video splitting device, the playback position is preferably given in units of sentences or words included in the text.

  In the video dividing apparatus according to the ninth development form, it is preferable that the collation unit develops at least one of a word or the like and the subject recognition result text with a thesaurus, and collates the word or the like with the subject recognition result text.

  In the video dividing device according to the tenth development form, it is preferable that the text is a text representing the content of an utterance included in the video.

  In the video dividing method according to the eleventh development mode, in the subject determination step, it is determined whether or not a word or the like represents a subject appearing in a predetermined range with reference to the reproduction position in the video. preferable.

  In the video dividing method of the twelfth development mode, in the subject determination step, when it is determined that the word or the like represents the subject reflected in the video, the reliability that the word or the like represents the subject is calculated, and in the subject weighting step, It is preferable to weight a word or the like with higher reliability.

  The video dividing method of the thirteenth development mode determines the importance in the video of the subject when it is determined in the subject determination step that a word or the like represents the subject shown in the video, and the importance in the subject weighting step. It is preferable to weight a word or the like with a higher value.

  In the video dividing method according to the fourteenth development mode, it is preferable that the importance level of the subject is determined in accordance with the ratio of the subject to the video in the subject determination step.

  In the video dividing method according to the fifteenth development mode, the subject determination step compares the subject and the subject recognition result text with the subject recognition step of recognizing the subject reflected in the video and outputting it as the subject recognition result text. And a collation step of determining whether or not a word or the like represents a subject shown in the video.

  In the video dividing method according to the sixteenth development form, it is preferable that the subject includes an object, a face image, or a character.

  In the video dividing method according to the seventeenth development mode, the subject recognition step includes an object recognition step for recognizing an object, a face image recognition step for recognizing a face image, a character recognition step for recognizing characters, a word included in the text, etc. It is preferable to include at least one of subject extraction steps for extracting a word or the like representing a subject from the subject.

  In the video dividing method according to the eighteenth development form, it is preferable that in the collation step, at least one of the word and the subject recognition result text is developed by a thesaurus to collate the word and the subject recognition result text.

  In the video dividing method according to the nineteenth development mode, it is preferable that the text is a text representing the content of an utterance included in the video.

  The program according to the twentieth development form preferably determines whether or not a word or the like represents a subject appearing in a predetermined range with reference to the reproduction position in the video in the subject determination process.

  In the subject development process, the program of the twenty-first form calculates the reliability that the word or the like represents the subject when it is determined that the word or the like represents the subject that appears in the video. It is preferable to weight a word or the like with a higher value.

  The program in the twenty-second development mode determines the importance in the subject image when it is determined in the subject determination process that a word or the like represents the subject in the image, and the importance is high in the subject weighting process. It is preferable to give a greater weight to a word or the like.

  The program in the twenty-third development form preferably determines the importance of the subject according to the ratio of the subject to the video in the subject determination process.

  The program according to the twenty-fourth development mode includes a subject recognition process for recognizing a subject shown in a video and outputting it as a subject recognition result text in a subject determination process, a word or the like by collating a word or the like with the subject recognition result text, and the like. It is preferable to cause the computer to execute collation processing for determining whether or not represents a subject appearing in the video.

  In the program of the twenty-fifth development form, it is preferable that the subject includes an object, a face image, or a character.

  The program in the twenty-sixth form includes an object recognition process for recognizing an object, a face image recognition process for recognizing a face image, a character recognition process for recognizing a character, a word included in the text, and the like. It is preferable to cause the computer to execute at least one of the subject extraction processing for extracting a word or the like representing.

  It is preferable that the program in the twenty-seventh development mode collates the word and the subject recognition result text by developing at least one of the word and the subject recognition result text by a thesaurus in the collation processing.

  In the program in the twenty-eighth expanded form, the text is preferably a text representing the content of an utterance included in the video.

  Embodiments of the present invention will be described below with reference to the drawings.

  Referring to FIG. 1, the video segmentation apparatus according to the present embodiment includes a subject determination unit 13, a subject weighting unit 14, and a video segmentation unit 16. The subject determination unit 13 includes a word or a word string (hereinafter referred to as a “word or the like”) included in the text with reference to text associated with the video with a reproduction position in the video. It is determined whether or not the subject is represented. The subject weighting unit 14 weights a word or the like that is determined to represent the subject as a weight that is greater than the weighting for other words. The video dividing unit 16 divides the video by dividing the text based on the weighting.

  Next, a first embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the first embodiment of the present invention. The first embodiment of the present invention is a video dividing apparatus that divides a video into a plurality of sections. Referring to FIG. 1, the video dividing apparatus includes a video data storage unit 11, a text storage unit 12, a subject determination unit 13, a subject weighting unit 14, a weighted text storage unit 15, a video division unit 16, a division result storage unit 17, and A video viewing unit 18 is provided. Each of these units operates as follows.

  The video data storage unit 11 stores video data to be divided. The text storage unit 12 stores text related to the video data stored in the video data storage unit 11 and provided with reproduction position information in the video data.

  First, the subject determination unit 13 represents, for each word or the like included in the text stored in the text storage unit 12, a subject in which the word or the like is reflected in the video data stored in the video data storage unit 11. And the determination result is output to the subject weighting unit 14.

  Next, the subject weighting unit 14 weights each word included in the text stored in the text storage unit 12 so that the weight of the word determined to represent the subject by the subject determination unit 13 increases. And outputs the result to the weighted text storage unit 15.

  The video dividing unit 16 reads the text weighted with each word from the weighted text storage unit 15 and divides the video data into topics by dividing the text using the weights, and the result is divided into result storage units. 17 to output.

  The video viewing unit 18 retrieves and reproduces the video data stored in the video data storage unit 11 in units of the divided topics by reading the video data division result stored in the division result storage unit 17.

  Next, the overall operation of this embodiment will be described in detail with reference to the block diagrams of FIGS. 1 and 2, the flowchart of FIG. 3, and the explanatory diagrams of FIGS. 4 to 10.

  The video data storage unit 11 stores video data to be divided. As the video data, for example, various videos such as a TV program, a lecture video, and a home video can be considered.

  The text storage unit 12 stores text associated with video data stored in the video data storage unit 11 and having a playback position in the video data. As such text, for example, text representing the content of an utterance included in video data can be considered. Specifically, a speech recognition result text obtained by speech recognition of an utterance, subtitle information (closed caption) transmitted in a television program, or the like can be used. In addition, the text storage unit 12 may store comments, comments, summary text of each scene, and the like that are manually assigned to each scene in the video.

  The text stored in the text storage unit 12 in this embodiment is assumed to be written in units of words. In the case of a language in which text is not divided into word units such as Japanese, it is preferable to divide the text into word units in advance using a known morphological analysis technique.

  In order to indicate which section of the video data the text is associated with, the text needs to be provided with information indicating a reproduction position in the video data. As information representing the playback position, the start of video data, the elapsed time from a specific position in the video data, the number of image frames, and the like can be used. Information indicating the reproduction position may be given to each sentence included in the text, or may be given to each word included in the text. By dividing the text into topics by the video dividing unit 16, the video data can also be divided based on the reproduction position information given to the divided text.

  In the following, it is assumed that the text storage unit 12 stores, as an example, text representing the content of an utterance included in video data together with elapsed time information from the beginning of the video data. Of course, in the present invention, the text stored in the text storage unit 12 is not limited to the text representing the content of the utterance.

  FIG. 4 is an example of data stored in the text storage unit 12. Referring to FIG. 4, in the video data stored in the video data storage unit 11, “the competition between mobile phone manufacturers is intensifying between 102.0 to 105.0 seconds from the beginning of the video data. "The various functions are now installed in the mobile phone" during the elapsed time from the beginning 105.0 to 110.0 seconds. Each text is divided into words by a morphological analysis technique. As described above, such text can be obtained by voice recognition of an utterance or using subtitle information.

  When dividing the video data, first, the subject determination unit 13 displays each word or the like included in the text stored in the text storage unit 12 in the video data stored in the video data storage unit 11. It is determined whether or not the subject is represented (steps S21 to S23). The subject determination unit 13 outputs the determination result to the subject weighting unit 14. Note that the subject means any object, face image, character image, place, or the like reflected in the video data.

  FIG. 2 is a block diagram illustrating a configuration of the subject determination unit 13. Referring to FIG. 2, the subject determination unit 13 includes a subject recognition unit 130, a subject recognition result storage unit 135, and a collation unit 136. The subject recognition unit 130 includes an object recognition unit 131, a face image recognition unit 132, a character recognition unit 133, and a subject extraction unit 134.

  The subject determination unit 13 uses the subject recognition unit 130 to recognize a subject shown in the video data stored in the video data storage unit 11 and outputs text representing the subject to the subject recognition result storage unit 135 as subject recognition result text. (Step S21). Recognition of the subject shown in the video data is performed by the object recognition unit 131, the face image recognition unit 132, the character recognition unit 133, and the subject extraction unit 134.

  The object recognizing unit 131 reads video data from the video data storage unit 11, and for example, Patent Document 1 (the matters described in Patent Document 1 are incorporated herein by reference) This can be realized by applying the object recognition technology described in the above. Here, the object means a group of objects reflected in the video. According to Patent Document 1, in order to recognize an object, first, a partial video is extracted by dividing an image in video data (partial video extraction means in Patent Document 1), and a color layout, an edge histogram, etc. are extracted from the partial video. Is calculated (visual feature value setting means in Patent Document 1), and the similarity between the visual feature value of the partial video and the visual feature value of the candidate object that is a candidate for the subject is calculated (patent) Feature quantity comparison means in Document 1). If the similarity between the two exceeds the threshold value, it is determined that the candidate object appears in the video data. By performing this determination processing on various candidate objects, various objects reflected in the video data can be recognized as subjects. The object recognition unit 131 outputs the name of the object determined to be reflected in the video to the subject recognition result storage unit 135 as subject recognition result text.

  Such processing may be performed on all image frames in the video data. In addition, in order to reduce the processing amount, the object recognition process may be performed for each shot (a continuous video section without camera switching or the like). In order to perform such processing, known shot division may be applied to the video data in advance, and object recognition processing may be performed on the obtained representative image of each shot. Various methods can be used for selecting the representative image. For example, the representative image frame may be simply the head image frame of the shot.

  Here, the technique described in Patent Document 1 has been described as a technique for realizing the object recognition unit 131. Of course, the method for realizing the object recognition unit 131 in the present invention is not limited to the method described in Patent Document 1, and any method may be used as long as it recognizes an object shown in a video.

  The face image recognition unit 132 can be realized by reading video data from the video data storage unit 11 and applying a known face image recognition technique to the video data. The face image recognition unit 132 outputs the name or title of the person determined to be reflected in the video to the subject recognition result storage unit 135 as subject recognition result text.

  The character recognition unit 133 can be realized by reading video data from the video data storage unit 11 and applying a known character recognition technique to the video data. For example, the character recognition unit 133 outputs the character recognition result of the character image shown in the video as it is to the subject recognition result storage unit 135 as the subject recognition result text. Further, the character recognition unit 133 may extract a person name, a place name, general nouns, and the like from the character recognition result, and output them to the subject recognition result storage unit 135 as subject recognition result text.

  When the video data is a television program or the like, it is also conceivable that information relating to the contents of characters (telops) displayed in the video is distributed. In such a case, the character recognition unit 133 may output the subject recognition result text by regarding the distributed telop information as the character recognition result without performing the character recognition of the character string shown in the video.

  The subject extraction unit 134 reads text from the text storage unit 12 and extracts a word or the like representing the subject shown in the video from words or the like included in the text. Such processing is realized, for example, by using the technology described in Non-Patent Document 2 (note that the matters described in Non-Patent Document 2 are incorporated herein by reference). Can do. That is, whether or not a noun in the text is reflected in the video, such as "whether the noun is a verbal stop", "what is the type of noun phrase that includes the noun" Judged by production rules that combine features. According to the technique described in Non-Patent Document 2, for each shot in the video, a noun representing the main subject of the shot can be extracted from the text. The subject extraction unit 134 outputs words or the like determined to represent the subject shown in the video to the subject recognition result storage unit 135 as subject recognition result text.

  Here, the method described in Non-Patent Document 2 has been described as a method for realizing the subject extraction unit 134. Of course, the method of realizing the subject extraction unit 134 in the present invention is not limited to the method described in Non-Patent Document 2, and any method may be used as long as it is a method for extracting a word representing a subject from text. .

  The object recognition unit 131, face image recognition unit 132, character recognition unit 133, and subject extraction unit 134 described above may output the reliability of subject recognition in addition to the subject recognition result text. Here, the reliability of subject recognition refers to a value indicating how much the obtained subject recognition result is considered correct. That is, the subject recognition results output from the object recognition unit 131, the face image recognition unit 132, the character recognition unit 133, and the subject extraction unit 134 are automatic processes, and may be incorrect. However, the higher the reliability, the higher the possibility that the obtained subject recognition result text is actually reflected in the video data.

  For example, when the object recognizing unit 131 is realized by the method described in Patent Literature 1, the similarity between the visual feature amount extracted from the partial video in the video data and the visual feature amount of the candidate object is obtained as the subject recognition result. Reliability. This is because the higher the similarity between the two, the higher the possibility that the subject recognition result is actually reflected in the video data.

  When the subject extracting unit 134 is realized by the method described in Non-Patent Document 2, the prediction accuracy of the production rule applied when extracting the subject can be used as the reliability of the subject recognition result. This is because a subject recognition result obtained by a rule with high prediction accuracy is considered to be more likely to be actually reflected in video data.

  The face image recognition unit 132 and the character recognition unit 133 can also use the reliability calculated by a well-known method in the known face image recognition technology and character recognition technology. Note that these reliability values may be normalized to a range of 0 to 1 by an appropriate method.

  In addition to the subject recognition result text, the object recognition unit 131, the face image recognition unit 132, and the character recognition unit 133 may output the importance in the video data of the subject. Here, the importance in the video data of the subject is a value representing how much the subject is related to the semantic content of the video data.

  The importance of the subject is, for example, an area in which an object recognized by the object recognition unit 131, a face image recognized by the face image recognition unit 132, or a character image recognized by the character recognition unit 133 is shown in the video. It can be calculated as a percentage of the entire screen. This is because the larger the range in which the subject such as an object, face image, or character appears in the video, the more the subject is the main subject in the video and the higher the relevance to the semantic content of the video. is there. In addition, the length of time during which a subject such as an object, a face image, or a character appears in the video can be set as the importance level.

  In this manner, the object recognition unit 131, the face image recognition unit 132, the character recognition unit 133, and the subject extraction unit 134 recognize the subject shown in the video data stored in the video data storage unit 11, and the subject recognition result. The text is obtained. The objects, face images, and character images shown in the video are considered to be particularly highly related to the semantic content of the video data. At this time, each subject recognition result text can be provided with information on a reproduction position that indicates in which section of the video data the subject is reflected, for example, as an elapsed time from the start of the video data. . Further, as described above, the reliability of the subject recognition result or the importance in the video data of the subject can be given to each subject recognition result text. These subject recognition results are stored in the subject recognition result storage unit 135.

  FIG. 5 is an example of data stored in the subject recognition result storage unit 135. Referring to FIG. 5, the “mobile phone” is shown as the subject during the elapsed time from the beginning of the video data of 105.0 to 120.0 seconds, and the reliability of the subject recognition result is 0.8. The importance of the subject is 0.4. In addition, “PaPeRo” appears as a subject during the elapsed time from the beginning of the video data between 190.0 and 195.0 seconds, the reliability of the subject recognition result is 0.6, and the importance of the subject Is 0.2.

  The subject recognition unit 130 does not have to include all of the object recognition unit 131, the face image recognition unit 132, the character recognition unit 133, and the subject extraction unit 134, and includes at least one of these means. It only has to be.

  Next, the subject determination unit 13 determines, for each word included in the text stored in the text storage unit 12, by the matching unit 136 whether or not the word represents the subject shown in the video data. (Steps S22 and S23). The subject determination unit 13 outputs the determination result to the subject weighting unit 14. The collation unit 136 performs this determination by collating each word included in the text with the subject recognition result text stored in the subject recognition result storage unit 135.

  The operation of the verification unit 136 will be described. As an example, the text storage unit 12 stores the text shown in FIG. 4, and the subject recognition result storage unit 135 stores the subject recognition result shown in FIG. The collation unit 136 determines whether or not the word “mobile phone” included in the text represents the subject (step S22). This determination is performed by collating the “mobile phone” with the subject recognition result text stored in the subject recognition result storage unit 135. Now, since the subject recognition result text includes “mobile phone”, the word “mobile phone” is determined to represent the subject shown in the video data.

  Since the determination of all the words in the text stored in the text storage unit 12 has not been completed (No in step S23), the collation unit 136 determines that the word “maker”, which is the next word after “mobile phone”, is the subject. Is determined (step S22). As a result of collating “maker” with the subject recognition result text, “maker” does not exist in the subject recognition result text, so it is determined that the word “maker” does not represent the subject (step S22). Such a process is repeated until the determination of all the words in the text is completed (Yes in step S23). When making such a determination, the collating unit 136 may collate the word string and the subject recognition result text instead of the word in the text. For example, it may be verified whether the word string “mobile phone / maker”, which is a combination of two words “mobile phone” and “maker”, exists in the subject recognition result.

  Further, when it is determined that a certain word represents a subject, the matching unit 136 may include the reliability of the determination in the determination result and output it to the subject weighting unit 14. Here, the reliability of the determination refers to a value indicating how much the determination is considered correct.

  Further, when determining that a certain word represents a subject, the collation unit 136 may include the importance in the video data of the subject in the determination result and output it to the subject weighting unit 14. Here, the importance level is a value indicating how highly the subject is related to the semantic content of the video data.

  In order to output the reliability and importance, values stored in the subject recognition result storage unit 135 may be used. For example, referring to FIG. 5, since the reliability of “mobile phone” in the subject recognition result is 0.8 and the importance is 0.4, it is determined that the word “mobile phone” in the text represents the subject. Sometimes the reliability of the determination can be 0.8 and the importance of the subject can be 0.4. In the subject recognition result, “PaPeRo” appears in two places. In such a case, for example, the larger value may be adopted as the reliability and importance. As a result, when it is determined that the word “PaPeRo” in the text represents a subject, the reliability of the determination can be 0.8 and the importance of the subject can be 0.6. Further, the value of the subject recognition result of “PaPeRo” that is temporally closest to the reproduction position associated with the word “PaPeRo” in the text may be used.

  Further, when collating a certain word with the subject recognition result text, the collation unit 136 expands at least one of the word or the subject recognition result text with a thesaurus, and then collates the two. Also good. Here, the thesaurus refers to a dictionary that can obtain synonyms, broader terms, narrower terms, related terms, and the like of words.

  For example, by using a thesaurus, “mobile phone” in the subject recognition result text shown in FIG. 5 is expanded to the synonym “mobile phone” or the narrower term “N905i”, or “PaPeRo” is a broad term. The collation may be performed after expanding to a certain “robot”. By performing such processing, for example, when the word “mobile phone” is included in the text stored in the text storage unit 12, it can be determined that “mobile phone” represents the subject. In addition, when such an expansion can be used, various expansions such as expansion from a building name to a place name where the building exists can be considered. For example, expanding the “Eiffel Tower” to “Paris”. In this way, the collation unit 136 absorbs the difference in expression between the word included in the text stored in the text storage unit 12 and the subject recognition result text stored in the subject recognition result storage unit 135. Thus, the subject can be determined more appropriately.

  As described above, the subject determination unit 13 determines, for each word included in the text stored in the text storage unit 12, whether or not the word represents a subject reflected in the video data. The determination result is output to the subject weighting unit 14. At this time, as described above, when it is determined that a certain word or the like represents a subject, the reliability of the determination or the importance in the video data of the subject may be included in the determination result.

  FIG. 6 shows an example of a determination result output from the subject determination unit 13 to the subject weighting unit 14. The determination result includes, for each word included in the text stored in the text storage unit 12, a determination result as to whether or not the word represents a subject shown in the video data. Further, when it is determined that the word represents the subject, the determination result includes the reliability of the determination and the importance in the video data of the subject.

  In this embodiment, in the subject determination unit 13, the object recognition unit 131, the face image recognition unit 132, and the character recognition unit 133 recognize the subject appearing in the video without any limitation. However, the object recognizing unit 131, the face image recognizing unit 132, and the character recognizing unit 133 read the text from the text storage unit 12, and operate to recognize the subject by limiting the recognized subject to a word or the like included in the text. May be. For example, when the text shown in FIG. 4 is stored in the text storage unit 12, the object recognition unit 131 does not recognize all the objects shown in the video data, but instead recognizes candidate objects to be recognized as text. It is also possible to operate so as to recognize an object after limiting to “mobile phone”, “PaPeRo”, etc. Similarly, the face image recognition unit 132 may operate so as to recognize face images after limiting candidates to face images corresponding to person names included in the text. Similarly, the character recognition unit 133 may operate so as to recognize a character image after limiting candidates to words included in the text.

  Furthermore, in this embodiment, prior to the operation of the collating unit 136, the subject recognizing unit 130 recognizes a subject shown in the video data in advance. However, the subject recognizing unit 130 recognizes the subject after limiting the word or the like when the collating unit 136 collates the word included in the text with the subject recognition result text without recognizing the subject in advance. You may do it.

  Next, the subject weighting unit 14 weights each word included in the text stored in the text storage unit 12 so that the weight of the word determined to represent the subject by the subject determination unit 13 increases. (Step S24), and outputs the result to the weighted text storage unit 15.

  As an example, Sa = 10 may be given as a weight to a word determined to represent a subject, and Sb = 1 may be given as a weight to a word determined not to represent a subject. Of course, if the weight when determined to represent the subject is greater than the weight when determined not to represent the subject (if Sa> Sb), the specific values of these weights are other values. But you can. Here, by giving a minimum weight to a word determined not to represent a subject, the video dividing unit 16 can also consider a word that does not represent a subject.

  In addition, when the reliability of the determination is included in the word determined to represent the subject, the weight of the word or the like may be increased as the reliability increases. By doing so, a greater weight is given to those that are considered to be correct in representing the subject.

  In addition, when the importance of the subject is included in the word determined to represent the subject, the weight of the word or the like may be increased as the importance increases. By doing so, a greater weight is given to the subject so that the subject is considered highly related to the semantic content of the video data.

  In order to perform such weighting, for example, a value proportional to the product of reliability and importance may be used as the weight. That is, calculation is performed as weight = Sa × reliability × importance. As shown in FIG. 6, when the word “mobile phone” represents the subject, the reliability is 0.8, and the importance is 0.4, the weight is 10 × 0.8 × 0. 4 = 3.2 can be given. Here, 10 is given as the proportionality constant (Sa), but the value may be other values. In addition, the function for calculating the weight may be another function as long as the weight increases as the reliability or importance increases. At this time, it is preferable that the weight of the word determined to represent the subject is not smaller than the weight given to the word determined not to represent the subject. For example, when the reliability is 0.3 and the importance is 0.2, if the weight is calculated as described above, the weight becomes 10 × 0.3 × 0.2 = 0.6. If the weight to be given when it is determined not to be expressed is Sb = 1, it is preferable to give Sb = 1 instead of 0.6 as the weight.

  Further, when the part of speech of the word to be weighted is other than the noun and the verb, the weight may be set to a small value, for example, 0. Further, when the word to be weighted is an attached word or a function word, the weight may be a small value. By doing so, it is possible to reduce the weight of a word that is considered to be less relevant to the semantic content of the video data.

  7 is stored in the weighted text storage unit 15 when the text shown in FIG. 4 is stored in the text storage unit 12 and the text shown in FIG. 5 is stored in the subject recognition result storage unit 135. Data is shown. Here, the weight of the word determined to represent the subject is calculated by 10 × reliability × importance, the weight of the word determined not to represent the subject is set to 1, and the word weight other than the noun and the verb is set to 0. It was.

  Finally, the video dividing unit 16 reads the text in which the weights are assigned to the respective words and the like from the weighted text storage unit 15, divides the video data into topics by dividing the text using the weights, and determines the result. The result is output to the division result storage unit 17. That is, as described above, since the text is provided with information on the reproduction position indicating which section of the video data each text corresponds to, the video corresponding to the division point is divided by dividing the text. The data reproduction position can be obtained.

  Hereinafter, a method of dividing the text read from the weighted text storage unit 15 by the video dividing unit 16 into topics representing a semantic group will be described in detail. FIG. 8 is a diagram illustrating an example of processing in which the video dividing unit 16 divides text. First, an analysis interval having a certain width is set for each part of the text (FIG. 8A), and for each analysis interval, a distribution of weights given to each word in the analysis interval is obtained (FIG. 8). 8 (b)). Specifically, the number of types of words appearing in the entire text may be the number of dimensions, and a vector having the sum of the weights given to each word in the analysis section as an element may be used as the weight distribution. For example, when the text shown in FIG. 7 is stored in the weighted text storage unit 15 and the width of the analysis section is set to two sentences, the distribution of word weights in the analysis section composed of sentence ID1 and sentence ID2 Is “mobile phone” → 6.4, “maker” → 1, “between” → 1, “competition” → 1, “intensification” → 1, “function” → 1, “installation” → 1, other It is represented by a vector with word → 0 as an element. Next, the similarity of the weight distribution between the adjacent analysis sections is calculated, and the minimum point of the similarity is obtained (FIG. 8C). Since the minimum point of the similarity is a point where the distribution of the weight of the word changes, this is set as a division point of the text. In FIG. 8, since X and Y are local minimum points, these points are used as text dividing points. In addition, when calculating | requiring a minimum point, you may limit to the depth with a certain depth or more, and you may limit to a thing whose similarity is below a threshold value.

  In the video data, it is considered that the subject reflected in the video is particularly highly related to the semantic content of each topic included in the video data. This is because, when creating video data, it is a matter of course that an object, person, character, place, etc. relating to the content to be transmitted to the viewer is shown as a video. It is very unlikely that the subject shown in the video is irrelevant to the content that the video wants to convey.

  Therefore, by dividing the text appropriately reflecting the semantic content of the topic contained in the video data by giving a large weight to the word representing the subject shown in the video data and detecting the point where the weight distribution of the word changes. It becomes possible. That is, in the examples of FIGS. 4 to 7, “mobile phone” and “PaPeRo” are reflected in the video as subjects, and these are considered to be closely related to the subject matter of the topic. Therefore, giving a large weight to these words to find the change point of the weight distribution of the words means that the text is divided while making the topic theme more prominent, and the division accuracy into topics is improved.

  Also, even if the word “mobile phone” is the same, if “mobile phone” is not shown in the video as the subject, the weight for “mobile phone” is small, and the effect on text division is small. It becomes. Since “mobile phone” is not the subject, it is highly possible that “mobile phone” is not the subject of the topic in the video. Accordingly, even in such a case, the division accuracy into topics is improved. As described above, according to the present invention, even when the same word is used, the division accuracy into topics is improved by appropriately weighting the word according to whether or not the word represents a subject.

  FIG. 9 is a diagram for more specifically explaining the effect of the present invention. Here, the text is a text representing the utterance content included in the video data. FIG. 9 shows a case where the appearance frequency distribution of words in each analysis section is simply obtained when video data (FIG. 9A) composed of seven topics A to G is divided (FIG. 9 ( b)) and a case where the word weight distribution of each analysis section is obtained by giving a large weight to the word representing the subject according to the present invention (FIG. 9C), between the adjacent analysis sections. Comparison similarity series of distributions are shown. In the video section of Topic F, “activin” and “cell” are shown as subjects between times t1 and t2 (FIG. 9 (d)), and these “activin” and “cell” are the time in the text. It appears between t3 and t4 (FIG. 9 (e)), and some words such as the word “skin” that is not the subject appear between times t5 and t6 in the text (FIG. 9 ( f)). Topic F is a topic whose subject is the substance “activin”. Therefore, the word “activin” appears in sections t3 to t4 of topic F in the text.

  When the appearance frequency distribution of each analysis section is simply obtained (FIG. 9B), the similarity is not different from the surroundings at the boundary between the topic E and the topic F, and therefore, between the topic E and the topic F. Cannot be divided. This is because there are many words that exist across the topic E and the topic F such as “skin” and the similarity of the word distribution at the boundary between the topic E and the topic F is not reduced by these influences. is there.

  On the other hand, when a large weight is given to the word representing the subject according to the present invention and the distribution of the word weight in each analysis section is obtained (FIG. 9C), the degree of similarity at the boundary between the topic E and the topic F is calculated. A valley is obtained and can be correctly divided between topic E and topic F. Such a result is obtained by reducing the weight of words such as “skin” that are not included in the subject and increasing the weight of words such as “activin” and “cell” included in the subject. That is, a word that does not represent a subject may be less relevant to the topic subject. According to the present invention, the influence of such words can be removed. In addition, the word representing the subject is highly related to the subject of the topic, and such a word is emphasized by the present invention.

  Note that “activin” and “cells” are shown in the video at times t1 to t2, and are different from the topic F section, so that the section in which the subject is simply displayed can be used as the topic section as it is. It cannot be divided into various topics. This is because even if “activin” or “cell” is the subject of the topic, the subject is not always shown. However, in the utterances included in the video, words such as “activin” and “cell” that are the subject of the topic appear throughout the section of the topic F (time t3 to t4). it can. As described above, in order to further enhance the effect of the present invention, the text stored in the text storage unit 12 is preferably a text including words that are highly relevant to the semantic content of the video in each part of the video. For example, as the text stored in the text storage unit 12, it is preferable to use text or the like representing the content of the utterance included in the video.

  Note that the video dividing unit 16 may multiply the IDF of each word when calculating the weight distribution of the words in each analysis section. As described above, if the word is the same, the IDF has the same value regardless of the scene in which the word appears. However, since the IDF is an index indicating the general importance of the word, it is used in combination with the present invention. You can also

  In this embodiment, the text is divided by obtaining the minimum point of the similarity of the weight distribution between adjacent analysis sections after obtaining the distribution of word weights in each analysis section. However, the method for dividing text in the present invention is not limited to such a method. For example, using a text corpus that has been divided into topics such as newspaper articles in advance, prepare topic models for various topics, and check each topic model against the word weight distribution in each analysis section. May be divided. As the topic model, for example, a model in which word distribution such as the appearance frequency of words appearing in each topic is learned may be used. Here, by appropriately determining the likelihood of transition between topics, the topic model series that best matches the series of analysis sections can be obtained together with the position of the topic dividing point. Text segmentation methods using such a topic model are, for example, “JP Yaman, I. Carp, L. Gillick, S. Lowe, and P. van Mulbregt,“ A HIDDEN MARKOV MODEL APPROACH TO TEXT SEGMENTATION AND. TRACKING, “IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 333-336, 1998.”.

  As described above, the video is divided by the video dividing unit 16. The divided video is stored in the division result storage unit 17 as, for example, reproduction position information corresponding to the division point.

  The video viewing unit 18 reads the video data division result stored in the division result storage unit 17, thereby searching and reproducing the video data stored in the video data storage unit 11 in units of divided topics. Do. For example, a news program can be searched or played in units of news items, and lecture videos can be searched or played in units of learning items. Of course, these functions are merely examples of a method of using video data divided according to the present invention. The present invention can be applied to any application that utilizes video data in units of topics.

  In the present embodiment, the text stored in the text storage unit 12 is preliminarily written in units of words. However, the text stored in the text storage unit 12 may not be written in units of words. That is, when the subject determination unit 13 or the subject weighting unit 14 reads the text from the text storage unit 12 and operates to divide the text into words using a known morphological analysis technique, the text storage unit 12 The present invention can be applied even when the stored text is not previously written in units of words.

  In the present embodiment, the collation unit 136 included in the subject determination unit 13 determines whether or not each word included in the text stored in the text storage unit 12 represents the subject. When the word is matched with all the subject recognition result texts stored in the subject recognition result storage unit 135 and the word matches any of the subject recognition result texts, the word is the subject. Judged. At this time, when the collation unit 136 determines whether or not a certain word represents the subject, the word is reflected in a video section that is limited to a predetermined range in time from the reproduction position corresponding to the word. It may be determined whether or not the subject is represented.

  As an example, it is assumed that the text shown in FIG. 4 is stored in the text storage unit 12, and the data shown in FIG. 5 is stored in the subject recognition result storage unit 135. At this time, when it is determined whether or not the word “mobile phone” included in the sentence ID1 represents the subject, the reproduction position corresponding to the sentence ID1 is 102.0 to 105.0 seconds, for example, 10 seconds. It is also possible to determine whether or not the subject is shown in the video segment within the range, that is, the video segment limited to 92.0 to 115.0 seconds. For this purpose, when the “mobile phone” and the subject recognition result text stored in the subject recognition result storage unit 135 are collated, the subject recognition result in which the reflected time overlaps with 92.0 to 115.0 seconds. It is sufficient to collate by limiting to.

  FIG. 10 is a diagram for specifically explaining such processing. The video data shown in FIG. 10 includes a topic H whose theme is “flower”, a topic I whose theme is “honey bee”, and a topic J whose theme is “Monro butterfly” (FIG. 10B). )). In each topic, “flowers”, “bees”, and “white butterflies” are shown as subjects (FIG. 10A). Further, the text storage unit 12 stores the text shown in FIG.

  At this time, if it is determined whether or not the word included in the text represents a subject without imposing time constraints as described above, words such as “bee” and “Monro butterfly” included in the text of Topic H are also included. It is determined that the subject is represented, and a large weight is given. However, words such as “bee” and “Monro butterfly” appearing in the text of Topic H do not indicate “bee” appearing in the video in Topic I or “Monro butterfly” appearing in the video in Topic J. Absent. As described above, in the video that the word originally does not indicate, when the subject represented by the word happens to be reflected, the weight of the word should not be increased.

  Therefore, by imposing time constraints as described above and determining whether or not the word included in the text represents a subject, the words such as “bee” and “Monro butterfly” included in the topic H text are: Since it is at a playback position that is temporally separated from the video section in which the “bee” or “Monro butterfly” appears in the video, it is not determined to represent the subject and the weight does not increase. As a result, only the word “flower” is given a high weight in the topic H, and only a word highly related to the subject of the topic H can be given a high weight.

  The present invention can be applied to any video data. For video data that often explains the subject itself, such as news programs and educational videos, or video data that explains the details of the video by utterances in the video. Thus, the present invention exhibits a particularly high effect.

  Next, the effect of the present embodiment will be described. According to the present invention, it is possible to appropriately divide video data into topics according to semantic content. The reason for this is that in the present invention, after determining a subject that is considered to be particularly strongly related to the semantic content of each topic included in the video data, the weight of a word or the like representing the subject is increased, and the video data This is because the video data is divided by dividing the text related to the.

  Next, a second embodiment of the present invention will be described in detail with reference to the drawings.

  The second embodiment of the present invention is realized as a computer that operates according to a program when the first embodiment is configured as a program.

  Referring to FIG. 11, in the second embodiment of the present invention, a data processing device 32 including a CPU and the like, a storage device 33 including a magnetic disk, a semiconductor memory, and the like, and a video dividing program 31 are shown. It consists of.

  The storage device 33 is used as a video data storage unit 331, a text storage unit 332, a weighted text storage unit 333, a division result storage unit 334, a subject recognition result storage unit 335, and the like.

  The video segmentation program 31 is read by the data processing device 32 and controls the operation of the data processing device 32 to realize the functions of the first embodiment on the data processing device 32. That is, the data processing device 32 controls the subject division unit 13, the subject weighting unit 14, the video division unit 16, the video viewing unit 18, or the object recognition unit 131 in FIG. The processing of the face image recognition unit 132, the character recognition unit 133, the subject extraction unit 134, and the collation unit 136 is executed.

  INDUSTRIAL APPLICABILITY The present invention can be applied to uses such as an information browsing system for browsing video data in a state of being organized for each topic, and a program for realizing the information browsing system on a computer. Further, the present invention can be applied to uses such as an information retrieval system for retrieving video data related to a specific topic from a large amount of video data. Furthermore, the present invention can be applied to any application that utilizes video data in units of topics.

  It should be noted that the examples and the examples can be changed and adjusted within the scope of the entire disclosure (including claims) of the present invention and based on the basic technical concept. Various combinations and selections of various disclosed elements are possible within the scope of the claims of the present invention. That is, the present invention of course includes various variations and modifications that could be made by those skilled in the art according to the entire disclosure including the claims and the technical idea.

Claims (31)

A word or a word string (hereinafter referred to as “words”) included in the text indicates a subject included in the video with reference to a text associated with the video and having a playback position in the video. A subject determination unit that determines whether or not
A subject weighting unit that weights a word that is determined to represent the subject among the words or the like to a weight that is greater than a weight for the other ones;
And a video dividing unit that divides the video by dividing the text based on the weighting.   The subject determination unit determines whether or not the word or the like represents a subject appearing in a predetermined range of the video with respect to the playback position. The video dividing device described in 1. When the subject determination unit determines that the word or the like represents a subject reflected in the image, the subject determination unit calculates the reliability that the word or the like represents the subject,
The video dividing apparatus according to claim 1, wherein the subject weighting unit weights the word or the like more as the reliability is higher. When the subject determination unit determines that the word or the like represents a subject reflected in the video, the subject determination unit determines the importance of the subject in the video;
The video dividing apparatus according to claim 1, wherein the subject weighting unit weights the word or the like more as the importance is higher.   5. The video segmentation device according to claim 4, wherein the subject determination unit determines the importance of the subject according to a ratio of the subject to the video. The subject determination unit recognizes a subject reflected in the video and outputs it as a subject recognition result text;
A collation unit further comprising: a collation unit that collates the word or the like with the subject recognition result text to determine whether or not the word or the like represents a subject reflected in the video. 2. The video dividing device according to 1.   The video segmentation device according to claim 6, wherein the subject includes an object, a face image, or a character.   The object recognizing unit includes an object recognizing unit that recognizes the object, a face image recognizing unit that recognizes the face image, a character recognizing unit that recognizes the character, and a word that represents a subject from words included in the text. The video segmentation device according to claim 7, further comprising at least one of a subject extraction unit that extracts a video.   7. The collation unit according to claim 6, wherein the collation unit develops at least one of the word or the like and the subject recognition result text by a thesaurus, and collates the word or the like with the subject recognition result text. Video segmentation device.   The video segmentation apparatus according to claim 1, wherein the text is a text representing the content of an utterance included in the video.   11. The video dividing apparatus according to claim 1, wherein the reproduction position is given in units of sentences or words included in the text. By computer
A word or a word string (hereinafter referred to as “words”) included in the text indicates a subject included in the video with reference to a text associated with the video and having a playback position in the video. A subject determination step for determining whether or not
A subject weighting step of weighting a word that is determined to represent the subject among the words, etc., with a weight greater than a weighting for other ones;
And a video dividing step of dividing the video by dividing the text based on the weighting.   13. The subject determination step determines whether or not the word or the like represents a subject in a predetermined range of the video as a reference with respect to the playback position. The video dividing method described in 1. In the subject determination step, when it is determined that the word or the like represents a subject reflected in the video, the reliability that the word or the like represents the subject is calculated,
13. The video segmentation method according to claim 12, wherein, in the subject weighting step, the higher the reliability is, the greater weight is given to the word or the like. In the subject determination step, when it is determined that the word or the like represents a subject reflected in the video, the importance of the subject in the video is determined,
13. The video segmentation method according to claim 12, wherein, in the subject weighting step, the higher the importance, the greater weight is given to the word or the like.   16. The video segmentation method according to claim 15, wherein, in the subject determination step, importance of the subject is determined according to a ratio of the subject to the video. The subject determination step includes a subject recognition step of recognizing a subject reflected in the video and outputting the subject recognition result text;
The collating step of collating the said word etc. with the said subject recognition result text, and determining whether the said word etc. represents the to-be-photographed object reflected in the said image | video is included, The collating process characterized by the above-mentioned. Video segmentation method.   The video segmentation method according to claim 17, wherein the subject includes an object, a face image, or a character.   The subject recognition step includes an object recognition step for recognizing the object, a face image recognition step for recognizing the face image, a character recognition step for recognizing the characters, a word representing a subject from words included in the text, and the like The video segmentation method according to claim 18, further comprising at least one of a subject extraction step of extracting a subject.   18. The collation process according to claim 17, wherein in the collation step, at least one of the word or the like and the subject recognition result text is developed by a thesaurus, and the word or the like and the subject recognition result text are collated. Video segmentation method.   21. The video segmentation method according to claim 12, wherein the text is text representing the content of an utterance included in the video. A word or a word string (hereinafter referred to as “words”) included in the text indicates a subject included in the video with reference to a text associated with the video and having a playback position in the video. Subject determination processing for determining whether or not
A subject weighting process for weighting a word determined to represent the subject among the words or the like, with a weight greater than a weight for the other
A program that causes a computer to execute video division processing for dividing the video by dividing the text based on the weighting.   23. In the subject determination process, it is determined whether or not the word or the like represents a subject shown in a predetermined range with respect to the reproduction position in the video. The program described in. In the subject determination process, when it is determined that the word or the like represents the subject shown in the video, the reliability that the word or the like represents the subject is calculated,
23. The program according to claim 22, wherein, in the subject weighting process, the higher the reliability, the greater weighting is given to the word or the like. In the subject determination process, when it is determined that the word or the like represents a subject shown in the video, the importance of the subject in the video is determined,
23. The program according to claim 22, wherein, in the subject weighting process, the higher the importance, the greater weighting is given to the word or the like.   26. The program according to claim 25, wherein in the subject determination process, the importance level of the subject is determined according to a ratio of the subject to the video. In the subject determination processing, subject recognition processing for recognizing a subject reflected in the video and outputting as a subject recognition result text;
The computer is caused to execute collation processing for collating the word or the like with the subject recognition result text to determine whether the word or the like represents a subject shown in the video. 22. The program according to 22.   The program according to claim 27, wherein the subject includes an object, a face image, or a character.   In the subject recognition process, an object recognition process for recognizing the object, a face image recognition process for recognizing the face image, a character recognition process for recognizing the character, a word representing a subject from a word included in the text, etc. 30. The program according to claim 28, wherein at least one of the subject extraction processing for extracting a subject is executed by a computer.   28. The collation process according to claim 27, wherein in the collation process, at least one of the word or the like and the subject recognition result text is developed by a thesaurus, and the word or the like and the subject recognition result text are collated. program.   The program according to any one of claims 22 to 30, wherein the text is a text representing the content of an utterance included in the video.

Images