JP2019160071A - Summary creation system and summary creation method - Google Patents

Abstract

To provide a summary creation system and a summary creation method for creating a correct and optimal summary from video data including moving image data and voice data.SOLUTION: A summary creation system 10 comprises: text integration means for integrating an utterance text, a telop text, a background image text and a logo mark text which are created from content and outputting the integrated text; and summary creation means for outputting a summary text obtained by summarizing the integrated text. The text integration means includes text integration setting means which learns an optimal text integration setting using accumulated past input data and output data of text integration processing as teaching material. The summary creation means includes summary setting learning means which learns an optimal summary creation setting using accumulated past input data and output data of summary creation results as teaching material.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a summary creation system and a summary creation method for creating summary text of content from a video signal of the content.

  The importance of metadata related to TV broadcast programs and Internet distribution videos is increasing. Metadata refers to information related to the data, not the data itself. The metadata includes data creation date and time, creator, data format, title, annotation, and summary. This is important information for efficiently managing and retrieving data.

  The present patent applicant includes a recording unit having a recording file for recording a video, a character information acquiring unit for acquiring character information from the video recorded in the recording file, and the character information acquired by the character information acquiring unit. Text information means for gathering texts into text, and metadata for storing the text information text-written by the text information text means in a metadata storage file as video metadata recorded in the recording file A metadata generation system including storage means is proposed (see Patent Document 1).

JP 2018-33048 A

  However, in the technique described in Patent Document 1, text information displayed on video and text information from speech are collated and generated as metadata. However, the accuracy of metadata generation is further improved. There is a desire to want.

  SUMMARY An advantage of some aspects of the invention is that it provides a summary creation system and a summary creation method for generating an accurate and optimal summary from video data including moving image data and audio data.

  The invention according to claim 1, which solves the problem, describes utterance text describing a person's utterance content in the content extracted from the video signal about the content, and display contents of a telop in the content extracted from the video signal. Telop text, background image text that describes at least one of scenes, situations, articles, and events that appear in the video signal, logo that appears in the video signal, a logo that describes a mark that includes a trademark, symbol, or sign A text integration unit that integrates at least two texts of the mark text and outputs an integrated text; and a summary creation unit that outputs a summary text summarizing the integrated text, the text integration unit being stored Input data and output data for past text integration processing Text summarization setting means for learning the optimum text integration setting using the data as the teaching material, and the summary creation means sets the optimum summarization setting using the accumulated input data and output data of the past summary creation results as the teaching material. A summary creation system comprising summary setting learning means for learning.

  According to the present invention, at least two of the utterance text, the telop text, the background image text, and the logo mark text are integrated by the text integration means and output as the integrated text, and the integrated text is summarized by the summary creation means. Summary text is output. At this time, the text integration means includes text integration setting means, and learns the optimal text integration setting using the accumulated input data and output data of the past text integration processing as teaching materials. The summary creation means includes summary setting learning means for learning the optimum summary creation setting using the accumulated past summary creation result input data and output data as learning materials, and learns the optimum summary creation setting. . For this reason, it is set as the text integration setting and summary creation state that are optimal for the scene and situation of the content.

  Therefore, it is possible to create a summary text that is optimal for a scene or situation of a content, in which speech text, telop text, background image text, and logo mark text are integrated.

  The invention according to claim 2 also includes means for extracting utterance information from an audio signal of the video signal for the content, means for recognizing utterance contents from the utterance information, and utterance text by converting the utterance contents into text. A text-to-speech unit comprising: means for outputting telop; means for extracting telop information for displaying a telop from the video signal of the video signal; means for recognizing the telop content from the telop information; and A telop text converting unit comprising: a means for outputting a telop text, a means for extracting background image information from the video signal of the video signal, a means for recognizing the background image content from the background image information, and the background Means for converting the image content into text and outputting a background text; Logo mark comprising means for extracting logo mark image information from a video signal of a video signal, means for recognizing the logo mark content from the logo mark image information, and means for converting the logo mark content into text and outputting the logo mark text And a text forming unit, and at least two text converting units selected from the text forming unit.

  According to the present invention, the utterance content text conversion unit extracts the utterance information from the audio signal of the video signal, recognizes the utterance content, and generates the utterance text. Also, the telop text conversion unit extracts telop information from the video signal of the video signal, recognizes the display content of the telop, and generates telop text. The background image text converting unit extracts background image information from the video signal of the video signal, recognizes the content of the background image, and generates a background text. Further, the logo mark text converting unit recognizes the logo mark text information from the video signal of the video signal and generates the logo mark text.

  Therefore, text is generated from speech information, telop information, background image information, and logo mark information, and text is generated based on various information from the video signal of the content. Can be created.

  Similarly, the invention described in claim 3 is the summary creating system according to claim 2, wherein at least one of the utterance content text conversion unit, the telop text conversion unit, the background image text conversion unit, and the logo mark text conversion unit is provided. And a text-learning means for learning an optimum text creation setting from the input data and output data in the past text-processing stored.

  According to the present invention, each text unit performs learning based on input data and output data of past text processing accumulated by the text learning unit.

  Accordingly, it is possible to learn the setting for creating the most suitable text from a number of summaries created in the past, and it is possible to create the optimum summaries for scenes and situations of content similar to the settings.

  Similarly, the invention described in claim 4 is the summarizing system according to claim 1, wherein the video signal is obtained from a broadcast program broadcast by a television broadcast station.

  Similarly, the invention described in claim 5 is characterized in that, in the summary creation system according to claim 1, the video signal is obtained from a moving image distributed by the Internet.

Similarly, the invention according to claim 6, utterance text describing the utterance content of a person in the content extracted from the video signal about the content, telop text describing the display content of the telop in the content extracted from the video signal, Background image text describing at least one description of scenes, situations, articles and events appearing in the video signal, logo mark text describing a logo, a mark, a mark, a mark including a symbol appearing in the video signal, A step of integrating at least two of the texts to output an integrated text; and a step of creating a summary of the integrated text and outputting a summary text, wherein the step of integrating the texts includes the accumulated past Input data and output data for text integration processing Pre-learning optimal text integration settings, and the step of creating the summary includes pre-learning optimal summary settings from input data and output data of accumulated past summary creation results, This is a summary creation method characterized by this.
According to the present invention, at least two of the utterance text, the telop text, the background image text, and the logo mark text are integrated into an integrated text, and the integrated text is summarized into a summary text. At this time, in the text integration, the optimum text integration setting is learned using the accumulated input data and output data of the past text integration processing as teaching materials. Also, when creating summaries, the optimum summary creation settings are learned using the accumulated input data and output data of past summary creation results as teaching materials. For this reason, it is set as the text integration setting and summary creation state that are optimal for the scene and situation of the content.

  Therefore, it is possible to create a summary text that is optimal for a scene or situation of a content, in which speech text, telop text, background image text, and logo mark text are integrated.

  Similarly, the invention according to claim 7 is the summary creation method according to claim 6, wherein utterance information is extracted from an audio signal of the video signal for the content, the utterance content is recognized from the utterance information, and the utterance Converting the content into text and outputting the utterance text; extracting telop information for displaying the telop from the video signal of the video signal; recognizing the telop content from the telop information; converting the telop content into text; Outputting the background image information from the video signal of the video signal, recognizing the background image content from the background image information, converting the background image content into text, and outputting the background text, the video The logo mark image information is extracted from the video signal of the signal, and the logo mark image information is extracted from the logo mark image information. Recognizing the contents, characterized in that it comprises at least two steps selected the logo content and outputting the logo text and text data, from.

  According to the present invention, utterance information is extracted from an audio signal of a video signal, the utterance content is recognized, and an utterance text is generated. Also, telop information is extracted from the video signal of the video signal, the display content of the telop is recognized, and telop text is generated. Also, background image information is extracted from the video signal of the video signal, the content of the background image is recognized, and background text is generated.

  Therefore, text is generated from speech information, telop information, background image information, and logo mark information, and text is generated based on various information from the video signal of the content. Can be created.

  Similarly, the invention of claim 8 is the summary creating method according to claim 6, wherein the utterance content is converted into text, the telop is converted into text, the background image is converted into text, and a logo mark is added. At least one of the steps of converting to text is characterized by comprising learning an optimal text creation setting from the input data and output data in the past text processing accumulated.

  According to the present invention, learning is performed on the basis of the input data and output data of the past text processing accumulated in each text.

  Accordingly, it is possible to learn the setting for creating the most suitable text from a number of summaries created in the past, and it is possible to create the optimum summaries for scenes and situations of content similar to the settings.

  Similarly, the invention described in claim 9 is the summarizing method according to claim 6, wherein the video signal is acquired from a broadcast program broadcast by a television broadcasting station.

  Similarly, the invention described in claim 10 is characterized in that, in the summary creating method according to claim 6, the video signal is obtained from a moving image distributed by the Internet.

  According to the summary creation system and the summary creation method according to the present invention, an accurate and optimal summary can be generated from video data including moving image data and audio data.

  That is, according to the summary creation system of claim 1, at least two texts among the utterance text, the telop text, the background image text, and the logo mark text are integrated by the text integration unit and output as the integrated text. The integrated text is summarized by the summary creation means, and the summary text is output. At this time, the text integration means includes text integration setting means, and learns the optimal text integration setting using the accumulated input data and output data of the past text integration processing as teaching materials. The summary creation means includes summary setting learning means for learning the optimum summary creation setting using the accumulated past summary creation result input data and output data as learning materials, and learns the optimum summary creation setting. . For this reason, it is set as the text integration setting and summary creation state that are optimal for the scene and situation of the content.

  Therefore, it is possible to create a summary text that is optimal for a scene or situation of a content, in which speech text, telop text, background image text, and logo mark text are integrated.

  According to the summary creation system of claim 2, the utterance content text conversion unit extracts the utterance information from the audio signal of the video signal, recognizes the utterance content, and generates the utterance text. Also, the telop text conversion unit extracts telop information from the video signal of the video signal, recognizes the display content of the telop, and generates telop text. The background image text converting unit extracts background image information from the video signal of the video signal, recognizes the content of the background image, and generates a background text. Further, the logo mark text converting unit recognizes the logo mark text information from the video signal of the video signal and generates the logo mark text.

  Therefore, text is generated from speech information, telop information, background image information, and logo mark information, and text is generated based on various information from the video signal of the content. Can be created.

  According to the summary creation system of the third aspect, each text unit learns based on the input data and output data of the past text processing accumulated by the text learning unit.

  Accordingly, it is possible to learn the setting for creating the most suitable text from a number of summaries created in the past, and it is possible to create the optimum summaries for scenes and situations of content similar to the settings.

  Moreover, according to the summary creation system of Claim 4, a summary can be created from a television program.

  In addition, according to the summary creation system described in claim 5, it is possible to create a summary from an Internet video program.

According to the summary creation method of claim 6, at least two of the utterance text, telop text, background image text, and logo mark text are integrated into an integrated text, and the integrated text is summarized. To be summarized text.
At this time, in the text integration, the optimum text integration setting is learned using the accumulated input data and output data of the past text integration processing as teaching materials. Also, when creating summaries, the optimum summary creation settings are learned using the accumulated input data and output data of past summary creation results as teaching materials. For this reason, it is set as the text integration setting and summary creation state that are optimal for the scene and situation of the content.

  Therefore, it is possible to create a summary text that is optimal for a scene or situation of a content, in which speech text, telop text, background image text, and logo mark text are integrated.

  Furthermore, according to the summary creation method of the seventh aspect, the utterance information is extracted from the audio signal of the video signal, the utterance content is recognized, and the utterance text is generated. Also, telop information is extracted from the video signal of the video signal, the display content of the telop is recognized, and telop text is generated. Also, background image information is extracted from the video signal of the video signal, the content of the background image is recognized, and background text is generated.

  Therefore, text is generated from speech information, telop information, background image information, and logo mark information, and text is generated based on various information from the video signal of the content. Can be created.

  According to the summary creation method of the eighth aspect, learning is performed based on the accumulated input data and output data of the past text conversion process when converting each text.

  Accordingly, it is possible to learn the setting for creating the most suitable text from a number of summaries created in the past, and it is possible to create the optimum summaries for scenes and situations of content similar to the settings.

  Moreover, according to the summary creation system of Claim 9, a summary can be created from a television program.

  Moreover, according to the summary creation system of Claim 10, a summary can be created from an Internet moving image program.

It is a block diagram which shows the whole structure of the summary production system which concerns on embodiment of this invention. The utterance text conversion part of the same summary preparation system is shown, (a) is a block diagram, (b) is a figure which shows the flow of a process. The telop text conversion part of the summary production system is shown, (a) is a block diagram, and (b) is a diagram showing the flow of processing. The background image text-izing part of the same summary preparation system is shown, (a) is a block diagram, (b) is a figure which shows the flow of a process. The logo mark text conversion part of the same summary preparation system is shown, (a) is a block diagram, (b) is a figure which shows the flow of a process. It is a block diagram which shows the text integration part of the summary production system. It is a block diagram which shows the summary preparation part of the summary preparation system. It is a flowchart which shows the process sequence of the summary production system. It is a figure which shows the process example of the summary preparation system.

[First Embodiment]
A summary creation system and a summary creation method according to an embodiment for carrying out the present invention will be described. First, the overall configuration of the summary creation system according to the first embodiment will be described. FIG. 1 is a block diagram showing the overall configuration of a summary creation system according to an embodiment of the present invention.

<Overall Configuration of Summary Production System 10>
As shown in FIG. 1, the summary generation system 10 is a video signal separation unit 20, an utterance text unit 100, a telop text unit 200, a background image text unit 300, a logo mark text unit 400, and a text integration unit. A text integration unit 500 and a summary creation unit 600 as summary creation means are provided. In the present embodiment, the summary generation system 10 acquires a video signal from a broadcast program from the television broadcast station 30 or a moving image distributed on the Internet 40. The video signal can be obtained from a moving image on the Internet. It should be noted that at least two parts selected from the telop text unit 200, the background image text unit 300, the logo mark text unit 400, and the text integration unit 500 may be provided.

  The video signal V including the audio signal and the video signal is separated into the audio signal A and the video signal B by the video signal separation unit 20. The audio signal A is input to the utterance text unit 100, and the video signal B is input to the telop text unit 200, the background image text unit 300, and the logo mark text unit 400.

<Speech text unit 100>
The speech text unit 100 receives the audio signal A and outputs speech text that is text describing the content of a person's speech in the content. The utterance text conversion unit 100 includes an utterance information extraction unit 110, an utterance content recognition unit 120, and an utterance content text conversion unit 130.

  The utterance information extraction unit 110 extracts utterance information from the audio signal A of the video signal V. That is, the noise in the voice signal A is removed, and the information of the human speech voice is extracted. The utterance information can include sound effects and characteristic music.

  The utterance content recognition unit 120 recognizes the utterance content from the utterance information. That is, the utterance information is acoustically and grammatically analyzed to recognize the utterance content as a language. Parameters, conditions, and the like used for this recognition can be generated by machine learning from the past speech text generation data accumulated as described later.

  The utterance content text conversion unit 130 converts the utterance content into text and outputs it. The parameters, conditions, and the like used for this recognition can be generated by machine learning from past speech text input data and generation data accumulated as described later.

<Telop text converter 200>
The telop text converting unit 200 receives the video signal B and outputs telop text that is text describing the telop content in the content. The telop text conversion unit 200 includes a telop information extraction unit 210, a telop content recognition unit 220, and a telop content text conversion unit 230.

  The telop information extraction unit 210 extracts telop information from the video signal B of the video signal V. That is, the background in the video signal B is removed, and only the telop image information is extracted.

  The utterance content recognition unit 120 recognizes the telop content from the telop image information. That is, the telop information is analyzed linguistically and grammatically to recognize the telop display content as a language. Parameters, conditions, and the like used for this recognition can be generated by machine learning from the past telop text input data and generation data accumulated as described later.

  The telop content text conversion unit 230 converts the telop content into text and outputs it. Parameters, conditions, and the like used for this recognition can be generated by machine learning from the past telop text input data and generation data accumulated as described later.

<Background image text unit 300>
The background image text conversion unit 300 receives the video signal B and outputs a background image text that is a text describing the background image content in the content. The background image includes at least one of a scene, a situation, an article, and an event, for example, a person, a person's belongings, a person's facial expression, a landscape, a building situation, an indoor situation, an animal, a vehicle, and other articles. Can be mentioned. The background image text conversion unit 300 includes a background image information extraction unit 310, a background image content recognition unit 320, and a background image content text conversion unit 330.

  The background image information extraction unit 310 extracts background image information from the video signal B of the video signal V. That is, the telop and unclear image in the video signal B are removed, and only the recognizable background image information is extracted.

  The background image content recognition unit 320 recognizes the content of the background image from the background image information. That is, a person, a personal belonging, a facial expression, a landscape, a building situation, an indoor situation, an animal, a vehicle, and other articles represented by analyzing background image information are recognized. Parameters, conditions, and the like used for this recognition can be generated by machine learning from input data and generation data of past background image text accumulated as will be described later.

  The background image content text unit 330 converts the background image content into text and outputs it. Parameters, conditions, and the like used for this recognition can be generated by machine learning from input data and generation data of past background image text accumulated as will be described later.

<Logo Mark Textification Unit 400>
The logo mark text converting unit 400 receives the video signal B and outputs a logo mark text which is a text describing the logo mark contents in the content. Examples of the logo mark include a mark, a mark including a mark, and a mark indicating the origin of the product, and other marks. The logo mark text conversion unit 400 includes a logo mark image information extraction unit 410, a logo mark content recognition unit 420, and a logo mark content text conversion unit 430.

  The logo mark image information extraction unit 410 extracts logo mark image information from the video signal B of the video signal V. That is, the telop and the background image in the video signal B are removed, and only the recognizable logo mark image information is extracted.

  The logo mark content recognition unit 420 recognizes the content of the logo mark from the logo mark image information. That is, it recognizes products, services, stores, facilities, etc. represented by analyzing logo mark image information. Parameters, conditions, and the like used for this recognition can be generated by machine learning from the past logo mark text input data and generation data accumulated as described later.

  The logo mark content text conversion unit 430 converts the logo mark image content into text and outputs it. Parameters, conditions, and the like used for this recognition can be generated by machine learning from the past logo mark text input data and generation data accumulated as described later.

<Text integration unit 500>
The text integration unit 500 integrates the utterance text from the utterance text conversion unit 100, the telop text from the telop text conversion unit 200, the background text from the background image text conversion unit 300, and the background text from the logo mark text conversion unit 400. . That is, inconsistencies and errors in each text are corrected, and an integrated text is generated. Parameters, conditions, and the like used for text integration can be generated by machine learning from past text integration input and output data accumulated as described later.

<Summary creation unit 600>
The summary creation unit 600 summarizes the integrated text from the text integration unit 500. In other words, the content of the summary text is summarized to the number of characters designated. Parameters, conditions, and the like used for this summarization can be generated by machine learning from input data and output data of combination processing such as the past accumulated as will be described later.

Next, machine learning processing of each unit will be described.
<Machine learning process of speech text unit 100>
2A and 2B show an utterance text conversion unit of the summary creation system. FIG. 2A is a block diagram, and FIG. 2B is a diagram showing a flow of processing. As shown in FIG. 6A, the utterance text conversion unit 100 includes a utterance information extraction unit 110, an utterance content recognition unit 120, and an utterance content text conversion unit 130, as well as a machine learning unit 140, which is a text conversion learning unit, and contents A recognition text creation setting unit 150 and a comparative evaluation unit 160 are provided. Further, an existing data storage unit 700 is connected to the speech text unit 100.

  The utterance text conversion unit 100 performs machine learning based on the existing video data stored in the existing data storage unit 700 and the existing utterance text, and optimizes the utterance content recognition unit 120 and the utterance content text conversion unit 130. The existing data storage unit 700 stores an existing video data storage unit 710 that stores a large number of video data that has been used when a person has created an utterance text in the past, and an utterance text created from the utterance content of the video data. An existing utterance text storage unit 720 is provided. These video data and utterance texts are used as machine learning materials.

  Further, the utterance text conversion unit 100 includes switching units 170 and 180 that perform data output switching when machine learning is performed and when an utterance content text is created from new video data.

  The content recognition text creation setting unit 150 sets the speech information extraction processing of the speech information extraction unit 110, the speech content recognition processing setting of the speech content recognition unit 120, and the text conversion processing setting of the speech content text conversion unit 130. And are stored. The utterance information extraction unit 110, the utterance content recognition unit 120, and the utterance content text conversion unit 130 perform utterance information extraction, utterance content recognition, and text conversion according to the conditions and parameters set by the content recognition text creation setting unit 150.

  The comparative evaluation unit 160 includes a comparison unit 161 and an evaluation unit 162. The comparison unit 161 receives the existing video data from the existing video data storage unit 710 and compares the utterance text created by the utterance content text conversion unit 130 with the existing utterance text from the existing utterance text storage unit 720. The evaluation unit 162 performs an evaluation based on the comparison result of the comparison unit 161, and gives a high score when the values match well.

  The machine learning unit 140 receives the evaluation from the evaluation unit 162 and changes the setting state of the content recognition text creation setting unit 150. This process is repeated for the same video data to make the evaluation value of the evaluation unit 162 as high as possible. This process can be repeated for a plurality of video data.

  By performing such machine learning, the abilities of the utterance content recognition unit 120 and the utterance content text conversion unit 130 are improved. After the predetermined machine learning is completed, the utterance text converting unit 100 processes the new video data and is in a state where the optimum utterance text can be output.

  Processing of the utterance text unit 100 will be described. As shown in FIG. 2B, first, a feature amount for speech recognition and text conversion is set in the content recognition text creation setting unit 150 (step SA1). This setting is performed based on the learning result of the machine learning unit 140.

  Next, the utterance information extraction unit 110 extracts speech from a large amount of speech signals based on the set features (step SA2).

  Further, the utterance content recognition unit 120 analyzes the extracted voice based on the set feature (step SA3).

  Then, the utterance content text conversion unit 130 converts the voice into text based on the set feature and outputs the utterance text (step SA4).

<Machine learning of telop text unit 200>
FIG. 3 shows a telop text conversion unit of the summary creation system, where (a) is a block diagram and (b) is a diagram showing a flow of processing. As shown in FIG. 6A, the telop text conversion unit 200 includes a telop information extraction unit 210, a telop content recognition unit 220, and a telop content text conversion unit 230, as well as a machine learning unit 240 and a content recognition text creation setting unit 250. The comparison evaluation unit 260 is provided. An existing data storage unit 700 is connected to the telop text conversion unit 200.

  The telop text conversion unit 200 performs machine learning based on the existing video data stored in the existing data storage unit 700 and the existing telop text, and optimizes the telop content recognition unit 220 and the telop content text conversion unit 230. The existing data storage unit 700 stores an existing video data storage unit 710 that stores a large number of video data used when a telop text was created by a person in the past, and a telop text created from the utterance content of the video data. An existing telop text storage unit 730 is provided. These video data and utterance texts are used as machine learning materials.

  The telop text conversion unit 200 includes switching units 270 and 280 that perform data output switching when machine learning is performed and when an utterance content text is created from new video data.

  The content-recognized text creation setting unit 250 includes a setting of telop information extraction by the telop information extraction unit 210, a setting of text content recognition processing by the telop content recognition unit 220, and a setting of text conversion processing by the telop content text conversion unit 230. Stored. The telop information extraction unit 210, the telop content recognition unit 220, and the telop content text conversion unit 230 perform telop extraction, content recognition, and text conversion according to the conditions and parameters set by the content recognition text creation setting unit 250.

  The comparative evaluation unit 260 includes a comparison unit 261 and an evaluation unit 262. The comparison unit 261 receives the existing video data from the existing video data storage unit 710 and compares the telop text created by the telop content text unit 230 with the existing telop text from the existing telop text storage unit 730. The evaluation unit 262 performs an evaluation based on the comparison result of the comparison unit 261, and gives a high score when the values match well.

  The machine learning unit 240 receives the evaluation from the evaluation unit 262 and changes the setting state of the content recognition text creation setting unit 250. This process is repeated for the same video data to make the evaluation value of the evaluation unit 262 as high as possible. This process can be repeated for a plurality of video data.

  By performing such machine learning, the capabilities of the telop content recognition unit 220 and the telop content text conversion unit 230 are improved. After completing the predetermined machine learning, the telop text conversion unit 200 processes the new video data and is in a state where it can output the optimum telop text.

  The processing of the telop text conversion unit 200 will be described. As shown in FIG. 3B, first, telop information extraction, content extraction, and text conversion feature amounts are set in the content recognition text creation setting unit 250 (step SB1). This setting is performed based on the learning result of the machine learning unit 240 (step SB2).

  Next, the telop information extraction unit 210 extracts a telop from a large amount of video signals based on the set feature (step SB2).

  Further, the telop content recognition unit 220 analyzes the telop extracted based on the set feature (step SB3).

  Then, the telop content text conversion unit 230 converts the telop content into text based on the set feature and outputs it as telop text (step SB4).

<Machine learning of background image text unit 300>
4A and 4B show a background image text conversion unit of the summary creation system. FIG. 4A is a block diagram and FIG. 4B is a diagram showing a flow of processing. As shown in FIG. 6A, the background image text conversion unit 300 is a machine learning which is a text integration setting unit in addition to the background image information extraction unit 310, the background image content recognition unit 320, and the background image content text conversion unit 330. A unit 340, a content recognition text creation / setting unit 350, and a comparative evaluation unit 360. An existing data storage unit 700 is connected to the background image text unit 300.

  The background image text conversion unit 300 performs machine learning based on the existing video data stored in the existing data storage unit 700 and the existing background image text, and optimizes the background image content recognition unit 320 and the background image content text conversion unit 330. To do. The existing data storage unit 700 stores an existing video data storage unit 710 that stores a large number of video data used when a person created telop text in the past, and a background image text created from the utterance content of the video data. The existing background image text storage unit 740 is provided. These video data and background image text serve as machine learning materials.

  The background image text converting unit 300 includes switching units 370 and 380 that perform data output switching when machine learning is performed and when an utterance content text is created from new video data.

  The content recognition text creation setting unit 350 sets the background image extraction processing of the background image information extraction unit 310, the background image content recognition processing of the background image content recognition unit 320, and the text conversion of the background image content text conversion unit 330. Stores processing settings. The background image information extraction unit 310, the background image content recognition unit 320, and the background image content text conversion unit 330 perform background image extraction, background image content recognition, and text conversion according to the conditions and parameters set by the content recognition text creation setting unit 350. Do.

  The comparative evaluation unit 360 includes a comparison unit 361 and an evaluation unit 362. The comparison unit 361 compares the background image text generated by the background image content text unit 330 upon receiving the existing video data from the existing video data storage unit 710 and the existing background image text from the existing background image text storage unit 740. To do. The evaluation unit 362 performs an evaluation based on the comparison result of the comparison unit 361, and gives a high score when the values match well.

  The machine learning unit 340 receives the evaluation from the evaluation unit 362 and changes the setting state of the content recognition text creation setting unit 350. This process is repeated for the same video data to make the evaluation value of the evaluation unit 362 as high as possible. This process can be repeated for a plurality of video data.

  By performing such machine learning, the capabilities of the background image content recognition unit 320 and the background image content text conversion unit 330 are improved. After the predetermined machine learning is completed, the background image text converting unit 300 processes the new video data and can output an optimum background image text.

  The process of the background image text unit 300 will be described. As shown in FIG. 4B, first, background image information extraction, background image recognition, and text conversion feature quantities are set in the content recognition text creation setting unit 350 (step SC1). This setting is performed based on the learning result of the machine learning unit 340.

  Next, the background image information extraction unit 310 extracts a background image from a large amount of video signals based on the set feature (step SC2).

  Further, the background image content recognition unit 320 analyzes the background image extracted based on the set feature (step SC3).

  Then, the background image content text conversion unit 330 converts the content of the background image into text based on the set feature and outputs it as background image text (step SC4).

<Machine learning of logo mark text unit 400>
5A and 5B show a logo mark text conversion unit of the summary creation system, where FIG. 5A is a block diagram and FIG. 5B is a diagram showing a flow of processing. In addition to the logo mark image information extraction unit 410, the logo mark content recognition unit 420, and the logo mark content text conversion unit 430, the logo mark text conversion unit 400 includes a machine learning unit 440, a content recognition text creation setting unit 450, and a comparative evaluation unit 460. Is provided. Further, an existing data storage unit 700 is connected to the logo mark text unit 400.

  The logo mark text conversion unit 400 performs machine learning based on the existing video data stored in the existing data storage unit 700 and the existing logo mark text, and performs the logo mark image information extraction unit 410, the logo mark content recognition unit 420, and the logo mark. The content text unit 430 is optimized. In the existing data storage unit 700, an existing video data storage unit 710 storing a large number of video data used when a person has created logo mark text in the past, and a logo mark text created from the utterance content of the video data are stored. The stored existing logo mark text storage unit 750 is provided. These video data and logo mark text are used as machine learning materials.

  In addition, the logo mark text converting unit 400 includes switching units 470 and 480 that switch data output when machine learning is performed and when an utterance content text is created from new video data.

  The content recognition text creation setting unit 450 stores the settings of the logo mark image content recognition processing of the logo mark content recognition unit 420 and the settings of the text conversion processing of the logo mark content text conversion unit 430. The logo mark image information extraction unit 410, the logo mark content recognition unit 420, and the logo mark content text conversion unit 430 perform logo mark extraction, content recognition, and text conversion according to the conditions and parameters set by the content recognition text creation setting unit 450.

  The comparative evaluation unit 460 includes a comparison unit 461 and an evaluation unit 462. The comparison unit 461 receives the existing video data from the existing video data storage unit 710 and compares the text created by the logo mark content text conversion unit 430 with the existing background image text from the existing logo mark text storage unit 750. The evaluation unit 462 performs an evaluation based on the comparison result of the comparison unit 461, and gives a high score when the values match well.

  The machine learning unit 440 receives the evaluation from the evaluation unit 462 and changes the setting state of the content recognition text creation setting unit 450. This process is repeated for the same video data to make the evaluation value of the evaluation unit 462 as high as possible. This process can be repeated for a plurality of video data.

  By performing such machine learning, the ability of the logo mark content recognition unit 420 and the logo mark content text conversion unit 430 is improved. After the predetermined machine learning is completed, the logo mark text conversion unit 400 processes the new video data and is in a state where it can output the optimum background image text.

  Processing of the logo mark text conversion unit 400 will be described. As shown in FIG. 5B, first, a feature amount of a logo mark is set in the content recognition text creation setting unit 450 (step SC1). This setting is performed based on the learning result of the machine learning unit 340.

  Next, the logo mark image information extraction unit 410 extracts a logo mark from a large amount of video signals based on the set feature (step SD2).

  Further, the logo mark content recognition unit 420 analyzes the background image extracted based on the set feature, and automatically confirms and registers it (step SD3).

  Further, the logo mark content recognition unit 420 recognizes a registered logo mark or a thing that matches a specific logo mark as a logo mark (step SD4).

  Then, the logo mark content text converting unit 430 converts the logo mark content into text based on the set feature and outputs it as logo mark text (step SD5).

<Machine learning of text integration unit 500>
FIG. 6 is a block diagram showing a text integration unit of the summary creation system. The text integration unit 500 includes an integrated text creation unit 510, an integrated text creation setting unit 520, a machine learning unit 530, and a comparative evaluation unit 540. An existing data storage unit 700 is connected to the text integration unit 500.

  The text integration unit 500 performs machine learning on the basis of various existing types stored in the existing data storage unit 700, that is, speech text, telop text, background text, logo mark text, and existing integrated text. Optimize operation. The existing data storage unit 700 stores an existing various text storage unit 760 that stores various text data used when an integrated text was created in the past, and an existing integrated text storage unit 770 that stores an integrated text created from the various texts. With. These various texts and integrated texts serve as machine learning materials.

  In addition, the text integration unit 500 includes switching units 570 and 580 that perform data output switching when performing machine learning and when creating a new integrated text from various new texts.

  The integrated text creation setting unit 520 stores text integration processing settings of the integrated text creation unit 510. The integrated text creation unit 510 performs text integration processing according to the conditions and parameters set by the integrated text creation setting unit 520.

  The comparative evaluation unit 540 includes a comparison unit 541 and an evaluation unit 542. The comparison unit 541 receives the existing various texts from the existing various text storage units 760 and compares the integrated text created by the integrated text creation unit 510 with the existing integrated texts from the existing integrated text storage unit 770. The evaluation unit 542 performs an evaluation based on the comparison result of the comparison unit 541, and gives a high score when the values match well.

  The machine learning unit 530 receives the evaluation from the evaluation unit 542 and changes the setting state of the integrated text creation setting unit 520. This process is repeated for the same various text data to make the evaluation value of the evaluation unit 542 as high as possible. This process can be repeated for a plurality of various text data.

  By performing such machine learning, the ability of the integrated text creation unit 510 is improved. After completing the predetermined machine learning, the text integration unit 500 processes the new video data and is in a state where it can output the optimum integrated text.

<Machine learning of summary creation unit 600>
FIG. 7 is a block diagram showing a summary creation unit of the summary creation system. The summary creation unit 600 includes a summary text creation unit 610, a summary creation setting unit 620, a machine learning unit 630 that is summary setting learning means, and a comparative evaluation unit 640. An existing data storage unit 700 is connected to the summary creation unit 600.

  The summary creation unit 600 performs machine learning based on the integrated text and summary text stored in the existing data storage unit 700, and optimizes the operation of the summary text creation unit 610. The existing data storage unit 700 includes an existing integrated text storage unit 770 that stores the integrated text data used when the summary text was created in the past, and past summary creation results that store the summary text created from the integrated text. And an existing summary text storage unit 780. These integrated texts and summary texts serve as machine learning materials.

  The summary creation unit 600 includes switching units 670 and 680 that perform data output switching when machine learning is performed and when a new summary text is created from a new integrated text.

  The summary creation setting unit 620 stores the summary processing settings of the summary text creation unit 610. The summary text creation unit 610 performs text summary processing according to the conditions and parameters set by the summary creation setting unit 620.

  The comparative evaluation unit 640 includes a comparison unit 641 and an evaluation unit 642. The comparison unit 641 compares the summary text created by the summary text creation unit 610 in response to the existing integration text from the existing integration text storage unit 770 and the summary text from the existing summary text storage unit 780. The evaluation unit 642 performs an evaluation based on the comparison result of the comparison unit 641 and gives a high score when the values match well.

  The machine learning unit 630 receives the evaluation from the evaluation unit 642 and changes the setting state of the summary creation setting unit 620. This process is repeated for the same various text data to make the evaluation value of the evaluation unit 642 as high as possible. This process can be repeated for a plurality of integrated text data.

  By performing such machine learning, the capability of the summary text creation unit 610 is improved. After completing the predetermined machine learning, the summary creation unit 600 can process the new video data and output an optimum summary text.

Next, processing of the summary production system 10 will be described. FIG. 8 is a flowchart showing the operation of the summary creation system.
First, the existing video data storage unit 710, the existing utterance text storage unit 720, the existing telop text storage unit 730, the existing background image text storage unit 740, the existing logo mark text storage unit 750, and the existing various text storage units of the existing data storage unit 700 760, an existing video signal and various text data are read into the existing integrated text storage unit 770 and the existing summary text storage unit 780 (step ST1).

  Next, machine learning processing is performed in the speech text unit 100, the telop text unit 200, the background image text unit 300, and the logo mark text unit 400 (steps ST2a, ST2b, ST2c, and ST2d). This learning process can also be performed sequentially.

  Next, the existing input data and output data are read into the existing data storage unit 550 of the text integration unit 500 and the existing data storage unit 650 of the summary creation unit 600 (step ST3). Next, machine learning processing is performed in the text integration unit 500 and the summary creation unit 600 (steps ST3a and 3b). This learning process can also be performed sequentially. Note that the machine learning processing of the utterance text conversion unit 100, the telop text conversion unit 200, the background image text conversion unit 300, and the logo mark text conversion unit 400 and the machine learning processing of the text integration unit 500 and the summary creation unit 600 are as follows. Regardless of the order of processing, it can be performed in the reverse order.

  When the learning process is completed (yes in step ST4), the video signal to be processed is input to the video signal separation unit 20 (step ST5). Thereby, the utterance text conversion unit 100, the telop text conversion unit 200, the background image text conversion unit 300, and the logo mark text conversion unit 400 execute text conversion processing (steps ST6a, ST6b, ST6c, ST6d).

  Each text is integrated by the text integration unit 500 (step ST7), the integrated text is further summarized by the summary creation unit 600 (step ST8), and summary text is output. finish.

  From the next summarization process, the optimum summarization can be performed only by inputting the target video signal for summarization (step ST5) without performing the machine learning process (steps ST1 to ST4). The machine learning process can be performed as necessary.

  Hereinafter, an example of processing of the summary production system 10 will be described. FIG. 9 is a diagram showing a processing example of the summary creation system. FIG. 9 shows video data. In the video data, video data including a video 810 and audio data including a narration 850 are stored. In this example, a part of a news program that informs the result of a tennis tournament is taken as an example.

  First, in the utterance text conversion unit 100, the utterance information extraction unit 110 extracts the narration 850 from the voice data, and the utterance content recognition unit 120 recognizes the content. The recognized content is converted into text by the utterance content text converting unit 130.

  In this example, the utterance content text generation unit 130, as an utterance text, “XX Open Championship held in ▲▲, XX month ○ day, Japan XX player won the semi-final and advanced to the final. The final will start from ** day *. △△ On TV, the game will be broadcast live from ** ".

  In the telop text converting unit 200, the telop information extracting unit 210 extracts a telop image from the video data. In this example, the telop displayed on the telop display unit 811, the news name 812 and the provider company name 813 displayed on the upper part of the video 810 are extracted. Further, the telop content recognition unit 220 recognizes the content of the telop, and the telop content recognition unit 220 recognizes the content of the telop. The recognized content is converted into text by the background image content text unit 330.

  In this example, the telop content text conversion unit 230 displays “breaking news XX player XX open final advance!” In the telop display unit 811, “△ News” as the news name 812, and “Provision ◆◆ Stock” as the provider name 813 "Company" is output as telop text.

  Further, in the background image text converting unit 300, the background image information extracting unit 310 extracts a person, an article, a situation, and the like from the background image 820. In this example, images of a tennis player 830, a face 831, a cap 832, tennis wear 833, tennis pants 834, a racket 835, a ball 836, a tennis court 837, and the like are extracted. Based on the extracted image, the background image content recognition unit 320 recognizes the content of each background image. For example, gender, race, age, and facial expression are recognized from the image and face 831 of the tennis player 830. In addition, the style, color, and the like of the cap 832, tennis wear 833, and tennis pants 834 are recognized. Further, the color, model number, and the like of the racket 835 and the ball 836 are recognized. The recognized content is converted into text by the background image content text unit 330.

  In this example, “male”, “XX”, “20s”, etc. for the tennis player 830, “sweat”, “suffer” from the face 831, “white” for the cap 832, “white” for the tennis wear 833, “White”, “shorts” for tennis shorts 834, “maker”, “model” for racket 835, “yellow” for ball 836, “clay” for tennis court 837, etc. are output as background image text. The

  Then, in the logo mark text converting unit 400, the logo mark image information extracting unit 410 extracts the logo mark from the video 810. In this example, a logo mark 841 attached to the upper part of the video 810 together with the news name 812, a logo mark 842 attached to the tennis wear 833, and a logo mark 843 attached to the tennis pants 834 are extracted. At this time, since the logo mark 842 and the logo mark 843 are written on the clothes, they are deformed by the movement of the tennis player 830. In this example, extraction is performed in consideration of this deformation. Based on the extracted logo mark, the background image content recognition unit 320 recognizes the content of each logo mark. Then, the logo mark content text conversion unit 430 converts the recognized logo mark content into text.

  In this example, the background image content text conversion unit 330 has “ΔΔN” for the logo mark 841 and “AAA” for the logo mark 842 and “BBB” for the logo mark 842. “AAA brand” and logo mark 843 are “BB” and “CC brand” “BB brand” is output as logo mark text.

  The text integration unit 500 receives the utterance text from the utterance text conversion unit 100, the telop text from the telop text conversion unit 200, the background image text from the background image text conversion unit 300, and the logo mark text from the logo mark text conversion unit 400. Integrate.

In this example, “XX Open Championship held in ▲▲, XX month, XX player in Japan won the semi-final and advanced to the final. The final will be held from XX day □ △△ On TV, this game will be broadcast live from time to time. ”, Breaking News ○○ Player XX Advance to the Open Finals! ”,“ △△ News ”,“ Provision ◆◆ Co., Ltd. ”,“ Male ”,“ XX ”,“ 20's ”,“ Sweat ”,“ Bad ”,“ Cap White ”,“ Tennis Wear White ” "Tennis Wear Short Sleeve", "Tennis Pants White", "Shorts", "Racquet Maker", "Model Number", "Ball Yellow", "Tennis Court Clay", "△△ N", "△△ Television Inc." , Which is a description of “Logo Mark AAA”, and texts such as “BBB Co., Ltd.”, “AAA Brand”, “Logo Mark BB”, “CC Co., Ltd.”, “BB Brand” are comprehensively integrated and output.
This integration text is enormous data, but is suitable for machine search.

  The summary creation unit 600 receives the integrated text from the text integration unit 500 and summarizes and outputs it within a predetermined number of characters, for example, 200 characters. In this example, the summary generation unit 600 says, “Japan XX player won the semi-final of the XX Open Championship on XX month, and advanced to the final. The XX Open Championship was held in ▲▲. The XX player won a victory by wearing a white cap, AAA brand white tennis wear, and BB brand white tennis pants. Will be broadcast live on TV. The summary is easy to understand and can be understood at a glance.

  The above system includes a CPU (Central Processing Unit) as a processing device, a RAM (Random Access Memory), a ROM (Read Only Memory), an HDD (Hard Disc Drive), an SSD (Solid State Drive), and the like as a storage device. It can be realized by executing application software on a computer system. Moreover, each part does not need to be arrange | positioned in the same place, and it can implement | achieve by arrange | positioning a part on cloud and connecting with a network. In addition, since these processes target a large amount of data, it is preferable to perform processing using a GPU (Graphics Processing Unit).

  That is, the integrated text is simply converted into characters such as voice, characters, background video, etc., and is data on a huge amount of text. For this reason, GPU can be processed at high speed by specializing in text processing.

  Furthermore, in addition to speech text, telop text, background image text, and logo mark text, the text integration unit 500 can input text acquired from electronic program guide data when targeting a TV program. Furthermore, in the case of a moving image distributed on the Internet, it is possible to acquire and input a third party's evaluation and reputation as text. This can improve the quality and quantity of integrated text.

[Other Embodiments]
In the present invention, data processing is summarized at high speed and appropriately by AI (artificial intelligence). The AI process can be realized by the machine learning (ML) described above. Furthermore, supervised learning with existing data as correct answers can be adopted as machine learning. Further, it is effective to perform deep learning (DL: Deep Learning) as machine learning.

  In deep learning, a large number of existing video data, various text data corresponding to each video data, integrated text, and summary text are learned as big data. Each machine learning unit includes an input layer, a plurality of intermediate layers, and an output layer, and performs processing by a neural network including a large number of neurons. That is, the new video data input to the summary creation system according to the present invention, various texts based on the video data, the integrated text, and the output of the summary as input are the intermediate layers so as to approach the existing various texts, the integrated text, and the summary. The weights and parameters in the neurons are optimized by a method such as the least square method.

The present invention can be widely applied to systems and apparatuses that create a summary and add metadata to a video signal, and has industrial applicability.

10: summary production system 20: video signal separation unit 30: TV broadcasting station 40: Internet 100: utterance text conversion unit 110: utterance information extraction unit 120: utterance content recognition unit 130: utterance content text conversion unit 140: machine learning unit 150 : Content recognition text creation setting unit 160: comparative evaluation unit 161: comparison unit 162: evaluation unit 170: switching unit 180: switching unit 200: telop text converting unit 210: telop information extracting unit 220: telop content recognizing unit 230: telop content Text conversion unit 240: Machine learning unit 250: Content recognition text creation setting unit 260: Comparison evaluation unit 261: Comparison unit 262: Evaluation unit 270: Switching unit 280: Switching unit 300: Background image text conversion unit 310: Background image information extraction Unit 320: Background image content recognition unit 330: Background image content text conversion unit 340: Machine Learning part 350: Content recognition text creation setting part 360: Comparative evaluation part 361: Comparison part 362: Evaluation part 370: Switching part 380: Switching part 400: Logo mark text converting part 410: Logo mark image information extracting part 420: Logo mark Content recognition unit 430: Logo mark content text conversion unit 440: Machine learning unit 450: Content recognition text creation setting unit 460: Comparison evaluation unit 461: Comparison unit 462: Evaluation unit 470: Switching unit 480: Switching unit 500: Text integration unit 510: Integrated text creation unit 520: Integrated text creation setting unit 530: Machine learning unit 540: Comparison evaluation unit 541: Comparison unit 542: Evaluation unit 550: Existing data storage unit 570: Switching unit 580: Switching unit 600: Summary creation unit 610: Summary text creation unit 620: Summary creation setting unit 630: Machine learning unit 640: Comparative evaluation unit 641 Comparison unit 642: Evaluation unit 650: Existing data storage unit 670: Switching unit 680: Switching unit 700: Existing data storage unit 710: Existing video data storage unit 720: Existing utterance text storage unit 730: Existing telop text storage unit 740: Existing Background image text storage unit 750: Existing logo mark text storage unit 760: Existing various text storage unit 770: Existing integrated text storage unit 780: Existing summary text storage unit

Claims (10)

Utterance text describing the utterance content of the person in the content extracted from the video signal about the content,
Telop text describing the display content of the telop in the content extracted from the video signal,
Background image text describing at least one description of scenes, situations, articles and events appearing in the video signal;
Logo mark text describing a logo, trademark, symbol, mark including a mark appearing in the video signal,
A text integration means for integrating at least two of the texts and outputting an integrated text;
Summary generating means for outputting a summary text summarizing the integrated text;
With
The text integration means comprises text integration setting means for learning the optimal text integration setting using the accumulated input data and output data of past text integration processing as teaching materials,
The summary creation means comprises summary setting learning means for learning optimal summary creation settings using the input data and output data of the past summary creation results accumulated as teaching materials. Speech text conversion comprising means for extracting speech information from the audio signal of the video signal for the content, means for recognizing speech content from the speech information, and means for converting the speech content into text and outputting speech text And
A telop comprising means for extracting telop information for displaying a telop from the video signal of the video signal, means for recognizing telop contents from the telop information, and means for converting the telop contents into text and outputting telop text A texting part;
A background image comprising means for extracting background image information from the video signal of the video signal, means for recognizing the background image content from the background image information, and means for converting the background image content into text and outputting the background text A texting part;
Logo comprising: means for extracting logo mark image information from the video signal of the video signal; means for recognizing logo mark content from the logo mark image information; and means for converting the logo mark content into text and outputting logo mark text Mark text conversion part,
The summary creation system according to claim 1, further comprising at least two text conversion units selected from the above.   At least one of the utterance content text unit, the telop text unit, the background image text unit, and the logo mark text unit is an optimal text creation setting from the input data and output data in the past text processing accumulated The summary creation system according to claim 2, further comprising text learning means for learning   2. The summary creation system according to claim 1, wherein the video signal is obtained from a broadcast program broadcast by a television broadcasting station.   The summary creation system according to claim 1, wherein the video signal is obtained from a moving image distributed via the Internet. Utterance text describing the utterance content of the person in the content extracted from the video signal about the content,
Telop text describing the display content of the telop in the content extracted from the video signal,
Background image text describing at least one description of scenes, situations, articles and events appearing in the video signal;
Logo mark text describing a logo, trademark, symbol, mark including a mark appearing in the video signal,
Integrating at least two of the texts and outputting the integrated text;
Creating a summary of the integrated text and outputting the summary text;
The step of integrating the text includes a step of previously learning an optimal text integration setting from the input data and output data of the past text integration processing accumulated,
The step of creating a summary includes a step of previously learning an optimum summary setting from input data and output data of past summary creation results accumulated. Extracting utterance information from the audio signal of the video signal for the content, recognizing the utterance content from the utterance information, textifying the utterance content, and outputting the utterance text;
Extracting telop information for displaying a telop from the video signal of the video signal, recognizing the telop content from the telop information, converting the telop content into text and outputting the telop text;
Extracting background image information from the video signal of the video signal, recognizing the background image content from the background image information, converting the background image content into text and outputting the background text;
Extracting logo mark image information from the video signal of the video signal, recognizing the logo mark content from the logo mark image information, converting the logo mark content into text and outputting the logo mark text;
The summary creation method according to claim 6, comprising at least two steps selected from:   At least one of the steps of converting the utterance contents into text, converting the telop into text, converting the background image into text, and converting the logo mark into text is a process in the accumulated past text processing. The summary creation method according to claim 6, further comprising the step of learning an optimum text creation setting from the input data and the output data.   2. The summary creation method according to claim 1, wherein the video signal is acquired from a broadcast program broadcast by a television broadcasting station.   The summary creation method according to claim 6, wherein the video signal is obtained from a moving image distributed via the Internet.