JP2018169697A - Video data processing apparatus, video data processing method, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a video data processing apparatus which adds appropriate meta data to video data.SOLUTION: A video data processing apparatus includes: a dictionary storage unit 50 comprising one or both of character recognition dictionaries and voice recognition dictionaries, for each of categories set in advance on the content of video data; an internal text detection unit 30 which detects an internal text included in the video data, by use of a character recognition dictionary selected in advance in the dictionary storage unit 50; a voice recognition unit 40 which recognizes voice added to the video data to generate a voice recognition text; a natural language processing unit 70 which extracts a tag representing an important word from the internal text to generate the tag and importance of the tag, and extracts a tag representing an important word from the voice recognition text to generate the tag and importance of the tag; and a dictionary selection unit 60 which selects a voice recognition dictionary of a category corresponding to the tag of the internal text, and selects a character recognition dictionary of a category corresponding to the tag of the voice recognition text.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a video data processing apparatus, a video data processing method, and a computer program for searching for target video data from a plurality of video data.

  With the diversification of video distribution services, the number of video data that can be viewed is increasing. In addition, since it has become easier to shoot moving images using a digital camera, a portable terminal, etc., the number of video data that can be obtained by other than video distribution services has increased rapidly. As a result, the environment is flooded with video data.

  In such an environment, in order for the user to view the target video data, it is necessary to add appropriate metadata to the video data. A conventional method for assigning metadata of video data is disclosed in, for example, Patent Document 1.

JP 2010-68434 A

  However, the conventional method of assigning metadata is a method in which a keyword selected from keywords registered in advance is used as metadata, and appropriate metadata cannot be assigned to video data. That is, there is a problem that video data that does not correspond to the registered keyword cannot be given appropriate metadata.

  The present invention has been made in view of this problem, and an object of the present invention is to provide a video data processing device, a video data processing method, and a program that can give appropriate metadata to video data.

  The video data processing apparatus according to an aspect of the present embodiment includes, for each video data, a video data identifier indicating the video data, a video data tag indicating the content of the video data, and a tag indicating the importance of the video data tag. A tag database that stores the degree of importance in association with each other; a dictionary storage unit that includes one or both of a character recognition dictionary and a voice recognition dictionary for each of a plurality of categories set in advance for the content of the video data; Included in the video data is an object recognition unit that recognizes an object included in the data and generates a tag representing the object and a likelihood of the tag, an importance generation unit that generates data indicating the importance of the object, and If the character recognition dictionary is stored in the dictionary storage unit, the in-video text is detected using the character recognition dictionary, and the text data is added to the video data. A speech recognition unit that recognizes the recognized speech and generates a speech recognition unit using the speech recognition dictionary if the speech recognition dictionary is stored in the dictionary storage unit; A natural language processing unit for extracting a tag, generating a tag and an importance of the tag, extracting a tag representing an important word from the speech recognition text, and generating the tag and the importance of the tag; and the video The category according to the tag of the text in the video detected by the text detection unit in the video data of the dictionary for character recognition used in the text detection unit in the data and the dictionary for voice recognition used in the speech recognition unit is changed. If the category corresponding to the tag of the speech recognition text detected by the speech recognition unit is changed, the category is changed. The dictionary selection unit that switches to the dictionary for character recognition and selects two or three of the generated three types of tags, or two or three of the three types of tags are similar If the tag importance is calculated based on the importance corresponding to the tag, the video data identifier corresponding to the tag importance, the video data tag, and the tag importance are calculated using the tag as a video data tag. The gist of the present invention is to include an importance level integration unit that associates and stores in the tag database.

  A video data processing method according to an aspect of the present embodiment is a video data processing method performed by the video data processing device, and the video data processing device is configured to display a video indicating the video data for each video data. A tag database that stores a data identifier, a video data tag that represents the content of the video data, and a tag importance that represents the importance of the video data tag, and a plurality of categories that are set in advance for the content of the video data A dictionary storage unit that includes one or both of a character recognition dictionary and a voice recognition dictionary for each of the above, and in the video data processing method, a tag that recognizes an object included in the video data and represents the object; The likelihood of the tag is generated, data indicating the importance of the object is generated, and the text in the video included in the video data is stored in the dictionary storage unit in the character recognition dictionary. Is stored in the dictionary storage unit, the voice recognition text is detected from the video data using the character recognition dictionary, and the voice recognition text recognized from the voice attached to the video data is stored in the dictionary storage unit. If it is generated using the speech recognition dictionary, a tag representing an important word is extracted from the text in the video, the tag and the importance of the tag are generated, and a tag representing the important word is generated from the speech recognition text. Extract the tag and the importance of the tag, and once select the character recognition dictionary and the speech recognition dictionary, the category of the category changed when the category according to the tag of the detected text in the video changes When the category corresponding to the tag of the detected voice recognition text is changed and switched to the dictionary for speech recognition, the category is changed to the dictionary for character recognition when the category is changed. If two or three of the three types of tags matched, or if two or three of the three types of tags are similar, tag importance based on the importance corresponding to the tag The tag is a video data tag, the video data identifier corresponding to the tag importance, the video data tag, and the tag importance are associated with each other and stored in the tag database.

  A computer program according to the present invention causes a computer to function as the video data processing apparatus.

  According to the present invention, a database in which video data is appropriately classified can be generated.

It is a figure which shows the function structural example of the video data processing apparatus which concerns on 1st Embodiment of this invention. It is a figure which shows the operation | movement flow of the tag database production | generation which the video data processing apparatus shown in FIG. 1 performs. It is a figure which shows the example which divided | segmented video data into the scene by the scene division part shown in FIG. It is a figure which shows the example of the object recognized by the object recognition part shown in FIG. It is a figure which shows the example of the text in video data detected by the text detection part in video data shown in FIG. It is a figure which shows the example of the character recognition tag which the natural language process part shown in FIG. 1 produces | generates. It is a figure which shows the example of the speech recognition tag which the natural language process part shown in FIG. 1 produces | generates. It is a figure which shows the example of the importance which the importance generation part shown in FIG. 1 produces | generates. It is a figure which shows the example of the operation | movement flow of the importance integration part shown in FIG. It is a figure which shows the example of a video data tag and tag importance. It is a figure which shows the function structural example of the video data processing apparatus which concerns on 2nd Embodiment of this invention. It is a figure which shows the function structural example of the video data processing apparatus which concerns on 3rd Embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The same reference numerals are given to the same components in a plurality of drawings, and the description will not be repeated.

[First Embodiment]
FIG. 1 shows a functional configuration example of a video data processing apparatus 1 according to the first embodiment. The video data processing apparatus 1 includes a scene dividing unit 10, an object recognizing unit 20, an in-video data text detecting unit 30, a voice recognizing unit 40, a dictionary storing unit 50, a dictionary selecting unit 60, a natural language processing unit 70, and an importance generation unit. 80, an importance integration unit 90, a tag database 100, a video management unit 110, and a video database 120. The video data processing device 1 generates a database in which appropriate metadata is added to video data based on information included in the video data. Hereinafter, the database is referred to as DB.

  The tag DB 100 stores, for each video data, a video data identifier indicating the video data, a video data tag indicating the content of the video data, and a tag importance level indicating the importance level of the video data tag in association with each other. The video data tag is text data representing a keyword or key phrase extracted from the video data.

  The dictionary storage unit 50 includes a character recognition dictionary and a speech recognition dictionary for each of a plurality of categories set in advance for the contents of the video data. Note that both the character recognition dictionary and the speech recognition dictionary need not be provided for each of a plurality of categories. One of the dictionaries may be provided.

  The operation will be described with reference to the operation flow of the video data processing apparatus 1 shown in FIG. Note that FIG. 2 shows a flow of processing object recognition, text detection in video data, and speech recognition in time series, but each processing may be performed simultaneously (in parallel).

  The scene dividing unit 10 divides the video data into a plurality of scenes (step S1). A scene is, for example, a video range obtained by dividing audio added to video data for each audio section. Or you may divide | segment for every area | region of an image | video. Each video section is, for example, a unit in a range where the brightness of pixels constituting video data is in a certain range or a unit in which a change in color tone of a pixel is in a certain range. The division may be divided in units of a certain time.

  The scene dividing unit 10 may be realized by using a general video distribution system API (Application Programming Interface). As the moving image distribution system API, for example, viaPlatz (registered trademark) manufactured by NTTI can be used. The function of the scene dividing unit 10 may be included in each functional component of the object recognizing unit 20, the in-video data text detecting unit 30, and the voice recognizing unit 40 described later.

  FIG. 3 shows an example in which video data is divided by the scene dividing unit 10. The first column in FIG. 3 is a video data identifier for identifying video data. The second column shows the number of the divided scene, and the third column shows the scene time. It can be seen that the scene 1 of the video having the video data identifier A001 is 10 seconds from the start of the video, and the scene 2 is 5 seconds after the scene 1. The time resolution may be, for example, ms of 1 second or less. The resolution may be in units of frames.

  The object recognition unit 20 recognizes an object included in the video data, and generates a tag representing the object and the likelihood of the tag (step S2). The object recognizing unit 20 extracts, for example, a feature amount of a BoF (Bag of Features) method from an image for each scene obtained by dividing video data, and compares the feature amount with a feature amount associated with an object in advance. Recognize objects.

  The object recognizing unit 20 does not need to recognize an object corresponding to a scene. An object may be recognized from a plurality of scenes. The feature amount extracted from the image may be anything. The feature amount may be, for example, a color histogram. Further, the object recognition unit 20 may be configured using a general image recognition API.

  If the frame rate of the video data is 24 f / sec to 60 f / sec, the object recognition unit 20 extracts, for example, one image from 1 second (24 to 60 frames), and recognizes the object from the image. . Note that the image extraction interval may be arbitrary, for example, may be extracted every 2 seconds, or may be extracted every 0.5 seconds.

  FIG. 4 shows an example of an object recognized by the object recognition unit 20. The first column in FIG. 4 is a video data identifier for identifying video data. The second column is an object tag representing the recognized object, the third column is the likelihood of the tag, and the fourth column is the time from the start of the video data that recognized the tag.

  The first line of FIG. 4 indicates that “Kinkakuji” is recognized from the image one second after the start of the video data with the video data identifier “A001”, and its likelihood is “0.7”. As the likelihood is closer to 1.0, it represents that thing.

  In this example, “Kitsu” is recognized from the image 2 seconds after the start of the video data, and “Pond” is recognized from the image 3 seconds later, but multiple tags can be extracted from one image. good. The object tag may be associated with the divided scene number.

The in-video data text detection unit 30 detects the in-video text included in the video data using the character recognition dictionary previously selected by the dictionary selection unit 60 (step S3).
When no dictionary for character recognition is stored in the dictionary storage unit 50, the text detection unit 30 in the video data uses the (common) dictionary provided in the text detection unit in the video data to convert the text in the video from the video data. Is detected.

  The in-video text included in the video data is a character such as a signboard shown in an image constituting the video data. Alternatively, it is a subtitle attached to an image by superimpose or a character projected by telop or the like. The character recognition dictionary that the dictionary selection unit 60 first selects from the dictionary storage unit 50 is set in advance.

  FIG. 5 shows an example of the in-video text detected by the in-video data text detection unit 30. The first column in FIG. 5 is a video data identifier for identifying video data. The second column is the text in the detected video, and the third column is the time from the start of the video data in which the text in the video is detected.

  FIG. 5 shows that the in-video text of “Autumn Kinkakuji” is detected from the image one second after the start of the video data with the video data identifier “A001”. This in-video text is detected from, for example, the telop of “Autumn Kinkakuji” shown in the image.

  The natural language processing unit 70 extracts a character recognition tag representing an important word from the text in the video data, and generates the tag and the importance level of the tag (step S4). The natural language processing unit 70 performs morphological analysis, context analysis, semantic analysis, and the like of the text in the video data, and extracts keywords (important words) from the text in the video data. And the importance of a keyword is produced | generated using TF-IDF method, for example. The importance is, for example, numerical information in the range of 0-100.

  The dictionary selection unit 60 selects a speech recognition dictionary of a category corresponding to the character recognition tag generated from the in-video text detected by the in-video data text detection unit 30 (step S5). For example, the category when the character recognition tag is “Kinkakuji” is “Kyoto Tourism”.

  The voice recognition unit 40 recognizes the voice recognition text from the voice given to the video data by using the voice recognition dictionary previously selected by the dictionary selection unit 60 (step S6). In this example, a speech recognition dictionary corresponding to the category “Kyoto tourism” is selected. In addition, when the dictionary for speech recognition is not stored in the dictionary storage unit 50, the speech recognition unit 40 recognizes the speech recognition text from the speech added to the video data using the standard (common) dictionary provided by itself. .

  The voice recognition unit 40 can be realized by a general voice recognition device. As a speech recognition device, for example, Speech Rec (trademark registration) manufactured by NTT IT can be used.

  For example, if the voice given to the video data is “Kinkakuji Temple with autumn leaves in the background. 40 converts the speech into speech recognition text and outputs it.

  The natural language processing unit 70 extracts a speech recognition tag representing an important word from the speech recognition text, and generates the tag and the importance of the tag (step S7).

  The dictionary selection unit 60 selects a character recognition dictionary of a category corresponding to the speech recognition tag generated from the speech recognition text recognized by the speech recognition unit 40 (step S8). As described above, the dictionary selection unit 60 selects the speech recognition dictionary from the character recognition tags, and selects the character recognition dictionary from the speech recognition tags. That is, the dictionary selection unit 60 operates so that the dictionaries used in the video data text detection unit 30 and the voice recognition unit 40 are determined in correspondence with the categories. As a result, the detection accuracy of the character recognition tag and the recognition accuracy of the voice recognition tag can be improved.

  That is, the dictionary selection unit 60 uses the text recognition dictionary selected by the text data detection unit 30 and the text recognition dictionary selected by the speech recognition unit 40 to detect the text within the video. When the category corresponding to the tag changes, the voice recognition dictionary of the changed category is switched to and selected, and when the category corresponding to the tag of the voice recognition text detected by the voice recognition unit 40 changes, Switch to the dictionary for character recognition and select it.

  FIG. 6 shows an example of the character recognition tag and importance generated by the natural language processing unit 70. The first column in FIG. 6 is the video data identifier, the second column is the character recognition tag, the third column is the importance, and the fourth column is the time.

  FIG. 6 shows an example of the voice recognition tag and importance generated by the natural language processing unit 70. The first column in FIG. 6 is the video data identifier, the second column is the voice recognition tag, the third column is the importance, and the fourth column is the time. In this example, it is shown that video text tags of “autumn” and “Kinkakuji” are generated 1 second from the start of the video data identified by the video data identifier A001.

  FIG. 7 illustrates the speech recognition text generated by the natural language processing unit 70 and its importance. The column direction in FIG. 7 is the same as FIG. 6 except that the tag is a voice recognition tag.

  The example shown in FIG. 7 is a case where the speech recognition text is “Kinkakuji Temple with autumn leaves in the background. It is a stunning scenery reflecting its appearance on the surface of a pond against the background of autumn leaves of trees.” . From the speech recognition text, speech recognition tags of “Autumn”, “Autumn leaves”, “Kinkakuji”, “Kidachi”, “Pond”, and “Water surface” are generated.

  The importance of each speech recognition text is 90 for “Autumn”, 80 for “Autumn leaves”, 90 for “Kinkakuji”, 50 for “Kidachi”, and 60 for “Water”. The time when each voice recognition text was generated is 5 seconds for “Autumn”, 6 seconds for “Autumn leaves”, 8 seconds for “Kinkakuji”, 10 seconds for “Kidachi”, and 11 seconds for “Water”. . The speech recognition text after “water surface” and the notation of importance are omitted.

  The importance generation unit 80 generates data indicating the importance of the object tag recognized by the object recognition unit 20 (step S6). The importance may be the ratio of the object of the object tag to the image. For example, if the ratio of the image of “Kinkakuji” in the image is 60% or more, the importance is 90, for example, if the ratio of the image is 50% or less, the importance is 50. Importance is generated according to the ratio. Alternatively, the likelihood output by the object recognition unit 20 may be converted into importance. For example, a value obtained by multiplying the likelihood by 100 may be used as the importance.

  FIG. 8 shows an example of the importance level generated by the importance level generation unit 80. The columns and rows in FIG. 8 are the same as those in FIG. 4 except that the likelihood is replaced with the importance. The example of importance shown in FIG. 8 is a value obtained by multiplying the likelihood shown in FIG. 4 by 100.

  The importance integration unit 90, when two or three of the generated three types of tags match, or when two or three of the three types of tags are similar, the importance corresponding to the tag The tag importance is obtained based on the tag, and the tag is used as a video data tag, and the video data identifier corresponding to the tag importance, the video data tag, and the tag importance are associated and stored in the tag DB 100 (step S9). The generated three types of tags are the above-mentioned “object tag”, “character recognition tag”, and “voice recognition tag”.

  The similarity of tags is determined with reference to a similar word dictionary and an expression fluctuation dictionary provided in the importance level integration unit 70. In FIG. 1, the similar word dictionary and the expression fluctuation dictionary are not shown.

  The importance level integration unit 90 calculates the tag importance level based on the importance level corresponding to the matching or similar tag, and sets the matching or similar tag as the video data tag. The matching tag is considered to be a tag that better represents the content of the video data.

  Therefore, for example, “Kinkakuji” in which the three types of tags (“object tag”, “character recognition tag”, “voice recognition tag”) coincide with each other is set as a video data tag, and the tag importance is, for example, the sum of each importance ( 70 + 90 + 90 = 250). Then, the video data identifier for identifying the video data, the video data tag, and the tag importance are stored in the tag DB 100 in association with each other.

  On the other hand, the video data is stored in the video DB 120 by the video management unit 110 in association with the video data identifier indicating the video data. Therefore, by searching the tag DB 100 in descending order of tag importance, it is possible to accurately extract the video data identifier of the video data desired to be viewed.

  Thereafter, the desired video data can be retrieved by searching the video DB 120 using the extracted video data identifier. Note that the video data can also be searched in scene units.

  The video search unit performs a process of searching for the video data identifier of the target video data from the tag DB 100 and retrieving the video data corresponding to the video data identifier from the video DB 120. In FIG. 1, the notation of the video search unit is omitted.

  As described above, the video data processing method performed by the video data processing apparatus 1 according to the present embodiment recognizes an object included in the video data, generates a tag representing the object and the likelihood of the tag, Generating data indicating importance, detecting text in video data included in video data from video data using the character recognition dictionary if the dictionary storage unit 50 stores a character recognition dictionary; A tag that represents the speech recognition text that recognizes the speech given to the video data if the speech recognition dictionary is stored in the dictionary storage unit 50 using the speech recognition dictionary, and represents the important word from the text in the video The tag and the importance of the tag are generated, the tag representing the important word is extracted from the speech recognition text, the tag and the importance of the tag are generated, and the character recognition dictionary selected once and voice recognition When the category corresponding to the detected tag in the video text changes, the dictionary is switched to the voice recognition dictionary of the changed category, and when the category changes according to the detected voice recognition text tag. If the two or three of the generated three types of tags are matched or two or three of the three types of tags are similar, The tag importance is obtained based on the importance corresponding to the tag, the video data tag corresponding to the tag importance, the video data tag, and the tag importance are associated with the tag as the video data tag. And stored in the tag DB 100.

  According to the video data processing method of this embodiment, a database (tag DB 100) in which video data is appropriately classified can be generated. Next, the operation of the importance level integration unit 70 will be described in more detail.

[Importance Integration Department]
The operation will be described with reference to the operation flow of the importance integration unit 90 shown in FIG.

  The importance level integration unit 90 determines whether one tag stored in the Doug DB 100 is identical or similar between the three tag types (object tag, character recognition tag, voice recognition tag). Determination is made (step S90). Hereinafter, an example in which the presence / absence of the same tag corresponding to the same video data identifier in FIGS. 6, 7 and 8 is determined will be described.

  When there is no matching tag, for example, for “water surface” where only the voice recognition tag exists, the voice recognition tag is used as the video data tag as it is (step S91), the importance of the voice recognition tag is set as the tag importance, and the video data identifier The data is stored in the tag DB 100 in association with each other (step S92). Then, the process returns to the determination of whether there is a match for other tags (object tag, character recognition tag, voice recognition tag) (NO in step S90 → NO in step S93). This repetition is repeated until all the video data is finished (YES in step S93).

  If there is a matching tag, it is determined whether or not all tag types match (step S94). When all the tag types match (YES in step S94), the importance level integration unit 90 calculates the tag importance level based on the importance level corresponding to the matching tag (step S95). The tag importance is, for example, a value obtained by doubling the total importance corresponding to matching tags. The tag that matches all the tags is “Kinkakuji” in the above example. The tag importance value may be an average value of each importance.

  Next, the importance level integration unit 90 stores the matching tag as a video data tag in the tag DB 100 in association with the video data identifier corresponding to the tag importance level, the video data tag, and the tag importance level. (Step S96).

  If all the tags do not match (NO in step S94), the importance integration unit 90 next determines whether or not the two tag types match (step S97). If the two tag types do not match, the process returns to step S90. That is, when there is no matching tag between different tag types, the importance value is not changed and is stored in the tag DB 100 as the tag importance as it is.

  If the two tag types match (YES in step S97), the importance level integration unit 90 calculates the tag importance level based on the importance level corresponding to the matching tag (step S98). The tag importance when the tags match between the two tag types is, for example, a value obtained by multiplying the total importance corresponding to the matching tags by 1.5. The tag that matches the two tag types is “autumn” in the above example. Thus, the tag importance is a value obtained by multiplying the total value or average value of each importance by a greater weight as the number of matching tags increases.

  Note that the tag importance may be a value that considers the extraction accuracy for extracting matching tags. That is, the tag importance is a value that considers the extraction accuracy for extracting matching tags, and may be obtained by multiplying the importance corresponding to a tag with high extraction accuracy by a large weight. The extraction accuracy is the recognition accuracy of each of image recognition, character recognition, and speech recognition.

  Specifically, for example, when the recognition accuracy of the speech recognition text is higher than the recognition accuracy of the object recognition, the importance corresponding to the speech recognition tag extracted from the speech recognition text is multiplied by a large weight. When the recognition accuracy of object recognition is higher than the recognition accuracy of speech recognition text, a large weight is applied to the object tag recognized by object recognition. In short, tag importance = importance of object tag × weight α + importance of character recognition tag × weight β + importance of voice recognition tag × weight γ, and change each weight α, β, γ according to recognition accuracy May be.

  Next, the importance level integration unit 90 associates the tag having the same two tag types as a video data tag with the video data identifier corresponding to the tag importance level, the video data tag, and the tag importance level associated with each other. Store in the tag DB 100 (step S99).

  FIG. 10 shows an example of video data tags and tag importance levels stored in the tag DB 100. The first column in FIG. 10 is the video data identifier, the second column is the video data tag, and the third column is the tag importance. The tag importance of “Kinkakuji”, which is a video data tag that matches between all tag types, is the highest value at 360. The tag importance of “autumn”, which is a video data tag in which two character recognition tags and voice recognition tags are generated, is 270. The tag importance of “water surface”, which is a video data tag in which only a voice recognition tag is generated, is 60.

  As the number of matching tag types increases by the processing in steps S90 to S99 described above, the tag importance value corresponding to the tag increases. The importance of tags that do not match between tag types does not change from the importance values generated by the importance generation unit 80 and the natural language processing unit 70.

  As the number of matching tag types increases by the processing in steps S90 to S99 described above, the tag importance value corresponding to the tag increases. The tag importance of tags that do not match between tag types does not change from the importance values generated by the importance generation unit 80 and the natural language processing unit 70.

  As described above, the video data processing device 1 increases the tag importance value of the video data tag that better represents the content of the video data and stores the tag importance value in the tag DB 100. According to the video data processing apparatus 1 of the present embodiment, a database in which appropriate metadata is added to video data can be generated.

  In the example shown in FIG. 9, tag matching is determined, but tag similarity may be determined as described above. Also, both tag matching and similarity may be determined.

[Second Embodiment]
FIG. 11 shows a functional configuration example of the video data processing apparatus 2 according to the second embodiment. The video data processing device 2 is different in that a tag management unit 130 is provided instead of the importance level integration unit 90 of the video data processing device 1. The video data processing device 2 does not integrate the importance corresponding to the tag that matches between the tag types.

  The tag management unit 130 sets each of the generated three tags as a video data tag and sets each importance level as a tag importance level. The video data identifier corresponding to the tag importance level, the video data tag, The tag importance is associated and stored in the tag DB 100.

  The tag management unit 130 associates the “object tag” and its importance, the “character recognition tag” and the importance, and the “voice recognition tag” and the importance with the video data identifier from which each tag is extracted, The tag is a video data tag, and the importance corresponding to each tag is stored in the tag DB 100 as the tag importance. That is, each tag and importance are stored in the tag DB 100 as the video data identifier and tag importance as they are.

  The “character recognition tag” and the “speech recognition tag” operate so that the dictionary selection unit 60 determines the dictionary used by the text detection unit 30 and the speech recognition unit 40 in the video data in accordance with the tag category. Is generated. Therefore, the detection accuracy and recognition accuracy of the “character recognition tag” and the “voice recognition tag” are relatively high. That is, the video data processing device 2 can generate a database (tag DB 100) in which video data is appropriately classified.

[Third Embodiment]
FIG. 12 shows a functional configuration example of the video data processing apparatus 3 according to the third embodiment. The video data processing device 3 is different from the video data processing device 2 in that an object recognition unit 23 is provided.

  The object recognizing unit 23 stores feature amounts representing the features of objects corresponding to a plurality of categories, and corresponds to the category of the character recognition dictionary or the category of the speech recognition dictionary selected by the dictionary selecting unit 60, respectively. Use category features.

  For example, when the dictionary selection unit 60 selects the “Kyoto sightseeing” category, the object recognition unit 20 recognizes the image using the feature amount corresponding to the “Kyoto sightseeing” category. The feature amount corresponding to the category of “Kyoto tourism” is, for example, a feature amount related to a shrine and temple located in Kyoto.

  As described above, by making the feature amount of the object recognition unit 23 correspond to the category, the recognition accuracy of the object recognition can be improved.

  In the above embodiment, the description has been given with the example in which the video data identifier indicating the video data is associated with each tag, but the present embodiment is not limited to this example. The scene divided by the scene dividing unit 10 may be associated with each tag. In this case, the target video data can be searched in units of video scenes.

  Further, although an example of whether or not all of the object tag, the character recognition tag, and the voice recognition tag match has been described, it is not necessary to determine matching between all tag types. For example, it is only necessary to determine a match between a character recognition tag and a voice recognition tag. In the video data processing device 3 of the third embodiment, the object recognition 20 of the video data processing device 2 is changed to the object recognition unit 23. However, the object recognition 20 of the video data device is recognized as an object. You may change to the part 23.

  As described above, the present invention is not limited to the above-described embodiment, and various modifications can be made within the scope of the gist thereof.

  When the processing unit in the above apparatus is realized by a computer, the processing content of the function that each processing unit should have is described by a program. Then, by executing this program on a computer, the processing unit in the above apparatus is realized on the computer.

  The program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Further, the program may be distributed by storing the program in a recording device of the server computer and transferring the program from the server computer to another computer via a network.

1: Video data processing device 10: Scene division unit 20, 23: Object recognition unit 30: Text data in text detection unit 40: Voice recognition unit 50: Dictionary storage unit 60: Dictionary selection unit 70: Natural language processing unit 80: Important Degree generation unit 90: Importance integration unit 100: Tag database 110: Video management unit 120: Video database

The video data processing apparatus according to an aspect of the present embodiment includes, for each video data, a video data identifier indicating the video data, a video data tag indicating the content of the video data, and a tag indicating the importance of the video data tag. A tag database that stores the degree of importance in association with each other; a dictionary storage unit that includes one or both of a character recognition dictionary and a voice recognition dictionary for each of a plurality of categories set in advance for the content of the video data; Included in the video data is an object recognition unit that recognizes an object included in the data and generates a tag representing the object and a likelihood of the tag, an importance generation unit that generates data indicating the importance of the object, and If the character recognition dictionary is stored in the dictionary storage unit, the in-video text is detected using the character recognition dictionary, and the text data is added to the video data. A speech recognition unit that recognizes the recognized speech and generates a speech recognition unit using the speech recognition dictionary if the speech recognition dictionary is stored in the dictionary storage unit; Extracting a tag , generating an importance level of the tag in consideration of an occurrence frequency of the tag in the text in the video and an occurrence frequency of other text , and extracting a tag representing an important word from the speech recognition text; The natural language processing unit that generates the importance of the tag in consideration of the frequency of occurrence of the tag in the speech recognition text and the frequency of occurrence in other text, and the character recognition used in the video data text detection unit When the category corresponding to the tag of the text in the video detected by the text detection unit in the video data of the dictionary and the voice recognition dictionary used in the voice recognition unit is changed When the category corresponding to the tag of the speech recognition text detected by the speech recognition unit is changed, it is switched to the character recognition dictionary of the changed category. When two or three of the generated three types of tags match the dictionary selection unit, or two or three of the three types of tags are similar, the importance corresponding to the tag The tag importance is obtained based on the video data tag, and the video data tag corresponding to the tag importance, the video data tag, and the tag importance are stored in the tag database in association with each other. The tag importance is a value considering the extraction accuracy for extracting the matching or similar tags, and the importance corresponding to the tag with high extraction accuracy And gist Rukoto calculated by multiplying the large weight on.

A video data processing method according to an aspect of the present embodiment is a video data processing method performed by the video data processing device, and the video data processing device is configured to display a video indicating the video data for each video data. A tag database that stores a data identifier, a video data tag that represents the content of the video data, and a tag importance that represents the importance of the video data tag, and a plurality of categories that are set in advance for the content of the video data A dictionary storage unit that includes one or both of a character recognition dictionary and a voice recognition dictionary for each of the above, and in the video data processing method, a tag that recognizes an object included in the video data and represents the object; The likelihood of the tag is generated, data indicating the importance of the object is generated, and the text in the video included in the video data is stored in the dictionary storage unit in the character recognition dictionary. Is stored in the dictionary storage unit, the voice recognition text is detected from the video data using the character recognition dictionary, and the voice recognition text recognized from the voice attached to the video data is stored in the dictionary storage unit. If it is generated using the voice recognition dictionary, a tag representing an important word is extracted from the text in the video , and the frequency of occurrence of the tag in the text in the video and the frequency of occurrence in other text are taken into consideration. Tag importance is generated, a tag representing an important word is extracted from the speech recognition text, and the importance of the tag is determined in consideration of the frequency of occurrence of the tag in the speech recognition text and the frequency of occurrence in other text. When the category according to the tag of the detected text in the video is changed, the character recognition dictionary and the speech recognition dictionary selected once are generated. When the category corresponding to the tag of the detected voice recognition text is changed and switched to the dictionary for voice recognition, the category is changed to the dictionary for character recognition when the category is changed. If two or three of the tags match, or if two or three of the three types of tags are similar, the tag importance is calculated based on the importance corresponding to the tag, and the tag is converted into video data. As a tag, an identifier of video data corresponding to the tag importance, the video data tag, and the tag importance are associated and stored in the tag database, and the tag importance matches or is similar a value considering the extraction precision of extracting the tag, and gist Rukoto calculated by multiplying the greater weight to the importance corresponding to the extraction accurate tag.

Claims (5)

A tag database that stores a video data identifier indicating the video data, a video data tag indicating the content of the video data, and a tag importance level indicating the importance level of the video data tag in association with each other;
A dictionary storage unit including one or both of a dictionary for character recognition and a dictionary for speech recognition for each of a plurality of categories set in advance for the contents of video data;
An object recognition unit that recognizes an object included in the video data and generates a tag representing the object and a likelihood of the tag;
An importance generation unit for generating data indicating the importance of the object;
In-video text detection unit for detecting text in video data included in video data using the dictionary for character recognition if the dictionary for character recognition is stored in the dictionary storage unit;
A speech recognition unit that recognizes speech added to video data, and generates a speech recognition unit using the speech recognition dictionary if the dictionary for speech recognition is stored in the dictionary storage unit;
A tag representing an important word is extracted from the text in the video, the tag and the importance of the tag are generated, a tag representing the important word is extracted from the speech recognition text, and the tag and the importance of the tag are generated. A natural language processing unit;
The character recognition dictionary used in the video data text detection unit, and the voice recognition dictionary used in the voice recognition unit are categories corresponding to the text tag detected in the video data text detection unit. If the category is changed according to the tag of the voice recognition text detected by the voice recognition unit, the character recognition dictionary is changed. The dictionary selection part to switch to and select,
If two or three of the three generated tags match, or if two or three of the three types of tags are similar, tag importance based on the importance corresponding to the tag A video data identifier corresponding to the tag importance, the video data tag, and an importance level integration unit that stores the tag importance level in association with each other in the tag database. A video data processing apparatus comprising: A tag management unit is provided instead of the importance integration unit,
The tag management unit sets each of the generated three types of tags as a video data tag and sets each importance level as a tag importance level, an identifier of video data corresponding to the tag importance level, the video data tag, The video data processing apparatus according to claim 1, wherein the tag importance is associated and stored in the tag database.   The object recognition unit stores feature amounts representing features of objects corresponding to a plurality of categories, and corresponds to the character recognition dictionary category or the speech recognition dictionary category selected by the dictionary selection unit, respectively. The video data processing apparatus according to claim 1, wherein an object included in the video data is recognized using the feature amount of the category. A video data processing method performed by a video data processing apparatus,
The video data processing device comprises:
A tag database that stores a video data identifier indicating the video data, a video data tag indicating the content of the video data, and a tag importance level indicating the importance level of the video data tag in association with each other;
A dictionary storage unit including a dictionary for character recognition and a dictionary for speech recognition for each of a plurality of categories set in advance for the contents of video data;
In the video data processing method,
Recognizing an object included in video data, generating a tag representing the object and the likelihood of the tag,
Generating data indicating the importance of the object;
If the text recognition dictionary is stored in the dictionary storage unit, the text in the video included in the video data is detected from the video data using the character recognition dictionary,
Generating a voice recognition text that recognizes the voice given to the video data by using the voice recognition dictionary if the voice recognition dictionary is stored in the dictionary storage unit;
A tag representing an important word is extracted from the text in the video, the tag and the importance of the tag are generated, a tag representing the important word is extracted from the speech recognition text, and the tag and the importance of the tag are generated. ,
Once the selected character recognition dictionary and speech recognition dictionary are selected, the category is changed to the speech recognition dictionary of the category changed when the category corresponding to the detected tag in the video is changed, and the detected speech recognition is selected. When the category corresponding to the tag of the text changes, switch to the character recognition dictionary of the changed category and select,
If two or three of the three generated tags match, or if two or three of the three types of tags are similar, tag importance based on the importance corresponding to the tag The tag is used as a video data tag, the video data identifier corresponding to the tag importance, the video data tag, and the tag importance are associated and stored in the tag database. Video data processing method.   A computer program for causing a computer to function as the video data processing apparatus according to claim 1.

Images