JP2018170001A - Video data processing apparatus, video data processing method, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a video data processing apparatus which adds appropriate meta data to video data.SOLUTION: A video data processing apparatus includes: an object recognition unit 20 which recognizes an object included in video data, and generates a tag representing the object and likelihood of the tag; an internal text detection unit 30 which detects an internal text included in the video data; a voice recognition unit 40 which recognizes voice added to the video data to generate a voice recognition text; a natural language processing unit 50 which extracts a tag representing an important word from the internal text to generate the tag and importance of the tag, and extracts a tag representing an important word from the voice recognition text to generate the tag and importance of the tag; and an importance integration unit 70 which determines tag importance, when two or three of generated three kinds of tags are identical or similar, on the basis of the importance of the tags, to store video data tag in a tag database 80 in association with the tag importance.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a video data processing apparatus, a video data processing method, and a computer program for searching for target video data from a plurality of video data.

  With the diversification of video distribution services, the number of video data that can be viewed is increasing. In addition, since it has become easier to shoot moving images using a digital camera, a portable terminal, etc., the number of video data that can be obtained by other than video distribution services has increased rapidly. As a result, the environment is flooded with video data.

  In such an environment, in order for the user to view the target video data, it is necessary to add appropriate metadata to the video data. A conventional method for assigning metadata of video data is disclosed in, for example, Patent Document 1.

JP 2010-68434 A

  However, the conventional method of assigning metadata is a method in which a keyword selected from keywords registered in advance is used as metadata, and appropriate metadata cannot be assigned to video data. That is, there is a problem that video data that does not correspond to the registered keyword cannot be given appropriate metadata.

  The present invention has been made in view of this problem, and an object of the present invention is to provide a video data processing device, a video data processing method, and a program that can give appropriate metadata to video data.

  The video data processing apparatus according to an aspect of the present embodiment includes, for each video data, a video data identifier indicating the video data, a video data tag indicating the content of the video data, and a tag indicating the importance of the video data tag. A tag database that stores the degree of importance in association with each other, an object that recognizes an object included in the video data, generates a tag that represents the object, and the likelihood of the tag, and data that indicates the degree of importance of the object An importance generation unit that generates a text, an in-video data text detection unit that detects text in the video included in the video data, a voice recognition unit that recognizes a voice given to the video data and generates a voice recognition text, A tag representing an important word is extracted from the text in the video, a tag and an importance of the tag are generated, a tag representing an important word is extracted from the speech recognition text, and the tag and the importance of the tag When two or three of the three types of generated tags match, or when two or three of the three types of tags are similar, it corresponds to the tag The tag importance is obtained based on the importance, and the tag is used as a video data tag, the video data identifier corresponding to the tag importance, the video data tag, and the tag importance are associated with each other in the tag database. The gist is to include an importance level integration unit for storing.

  A video data processing method according to an aspect of the present embodiment is a video data processing method performed by the video data processing device, and the video data processing device is configured to display a video indicating the video data for each video data. A tag database for storing a data identifier, a video data tag representing the content of the video data, and a tag importance level representing the importance level of the video data tag in association with each other; Recognize the included object, generate a tag representing the object and the likelihood of the tag, generate data indicating the importance of the object, detect the text in the video included in the video data, and attach it to the video data A speech recognition text is generated to generate a speech recognition text, a tag representing an important word is extracted from the in-video text, the tag and the importance of the tag are generated, and the speech recognition text A tag representing an important word is extracted from the tag, and the importance of the tag and the tag is generated. If two or three of the generated three tags match, or the three of the tags If two or three of the two are similar, the tag importance is calculated based on the importance corresponding to the tag, the video tag corresponding to the tag importance is used as the video data tag, and the video The gist is to store the data tag and the tag importance in association with each other in the tag database.

  A computer program according to the present invention causes a computer to function as the video data processing apparatus.

  According to the present invention, appropriate metadata can be assigned to video data.

It is a figure which shows the function structural example of the video data processing apparatus of 1st Embodiment. It is a figure which shows the operation | movement flow of the tag database production | generation which the video data processing apparatus shown in FIG. 1 performs. It is a figure which shows the example which divided | segmented video data into the scene by the scene division part shown in FIG. It is a figure which shows the example of the object recognized by the object recognition part shown in FIG. It is a figure which shows the example of the text in video data detected by the text detection part in video data shown in FIG. It is a figure which shows the example of the character recognition tag which the natural language process part shown in FIG. 1 produces | generates. It is a figure which shows the example of the speech recognition tag which the natural language process part shown in FIG. 1 produces | generates. It is a figure which shows the example of the object tag which the importance generation part shown in FIG. 1 produces | generates. It is a figure which shows the example of the operation | movement flow of the importance integration part shown in FIG. It is a figure which shows the example of a video data tag and tag importance. It is a figure which shows the specific example of the character recognition tag memorize | stored in tag DB80, a voice recognition tag, and tag importance. It is a figure which shows the function structural example of the video data processing apparatus of 2nd Embodiment. It is a figure which shows the example of the operation | movement flow of the recognition accuracy evaluation part shown in FIG. It is a figure which shows the function structural example of the video data processing apparatus of 3rd Embodiment. It is a figure which shows the result of having investigated the relationship between the area of a character area, and the relevance rate of a recognition result.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[First Embodiment]
FIG. 1 shows a functional configuration example of the video data processing apparatus 1 according to the first embodiment. The video data processing apparatus 1 includes a scene division unit 10, an object recognition unit 20, a text detection unit 30 in video data, a speech recognition unit 40, a natural language processing unit 50, an importance generation unit 60, an importance integration unit 70, a tag database. 80, a video management unit 90, and a video database 91. The video data processing device 1 generates a database in which appropriate metadata is added to video data based on information included in the video data. Hereinafter, the database is referred to as DB.

  The tag DB 80 stores, for each video data, a video data identifier indicating the video data, a video data tag indicating the content of the video data, and a tag importance level indicating the importance level of the video data tag in association with each other. The video data tag is text data representing a keyword or key phrase extracted from the video data.

  The operation will be described with reference to the operation flow of the video data processing apparatus 1 shown in FIG. Note that FIG. 2 shows a flow of processing object recognition, text detection in video data, and speech recognition in time series, but each processing may be performed simultaneously (in parallel).

  The scene dividing unit 10 divides the video data into a plurality of scenes (step S1). A scene is, for example, a video range obtained by dividing audio added to video data for each audio section. Or you may divide | segment for every area | region of an image | video. Each video section is, for example, a unit in a range where the brightness of pixels constituting video data is in a certain range or a unit in which a change in color tone of a pixel is in a certain range. The division may be divided in units of a certain time.

  The scene dividing unit 10 may be realized by using a general video distribution system API (Application Programming Interface). As the moving image distribution system API, for example, viaPlatz (registered trademark) manufactured by NTTI can be used. The function of the scene dividing unit 10 may be included in each functional component of the object recognizing unit 20, the in-video data text detecting unit 30, and the voice recognizing unit 40 described later.

  FIG. 3 shows an example in which video data is divided by the scene dividing unit 10. The first column in FIG. 3 is a video data identifier for identifying video data. The second column shows the number of the divided scene, and the third column shows the scene time. It can be seen that scene 1 with video data identifier A001 is 10 seconds after the start of the video, and scene 2 is 5 seconds after scene 1. The time resolution may be, for example, ms of 1 second or less. The resolution may be in units of frames.

  The object recognition unit 20 recognizes an object included in the video data, and generates a tag representing the object and the likelihood of the tag (step S2). The object recognizing unit 20 extracts, for example, a feature amount of a BoF (Bag of Features) method from an image for each scene obtained by dividing video data, and compares the feature amount with a feature amount associated with an object in advance. Recognize objects.

  The object recognizing unit 20 does not need to recognize an object corresponding to a scene. An object may be recognized from a plurality of scenes. Further, the feature amount extracted from the image may be anything. The feature amount may be, for example, a color histogram. Alternatively, the object recognition unit 20 may be configured using a general image recognition API.

  If the frame rate of the video data is 24 f / sec to 60 f / sec, the object recognition unit 20 extracts, for example, one image from 1 second (24 to 60 frames), and recognizes the object from the image. . Note that the image extraction interval may be arbitrary, for example, may be extracted every 2 seconds, or may be extracted every 0.5 seconds.

  FIG. 4 shows an example of an object recognized by the object recognition unit 20. The first column in FIG. 4 is a video data identifier for identifying video data. The second column is an object tag representing the recognized object, the third column is the likelihood of the tag, and the fourth column is the time from the start of the video data that recognized the tag.

  The first line in FIG. 4 indicates that “Kinkakuji” is recognized from the image one second after the start of the video data with the video data identifier “A001”, and its likelihood is “0.7”. As the likelihood is closer to 1.0, it represents that thing.

  In this example, “Kitsu” is recognized from the image 2 seconds after the start of the video data, and “Pond” is recognized from the image 3 seconds later, but multiple tags can be extracted from one image. good. The object tag may be associated with the divided scene number.

  The video data internal text detection unit 30 detects video internal text included in the video data (step S3). The in-video text included in the video data is a character such as a signboard shown in an image constituting the video data. Alternatively, it is a subtitle attached to an image by superimpose or a character projected by telop or the like.

  FIG. 5 shows an example of the in-video text detected by the in-video data text detection unit 30. The first column in FIG. 5 is a video data identifier for identifying video data. The second column is the text in the detected video, and the third column is the time from the start of the video data in which the text in the video is detected.

  FIG. 5 shows that the in-video text of “Autumn Kinkakuji” is detected from the image one second after the start of the video data with the video data identifier “A001”. This in-video text is detected from, for example, a telop written as “Autumn Kinkakuji” in the image.

  The voice recognition unit 40 recognizes the voice given to the video data and generates a voice recognition text (step S4). The voice recognition unit 40 can be realized by a general voice recognition device. For example, Speech Rec (registered trademark) manufactured by NTT IT can be used as the speech recognition apparatus.

  For example, if the voice given to the video data is “Kinkakuji Temple with autumn leaves in the background. 40 converts the speech into speech recognition text and outputs it.

  The natural language processing unit 50 extracts a character recognition tag representing an important word from the text in the video data, generates the tag and the importance of the tag, extracts a speech recognition tag representing the important word from the speech recognition text, The tag and the importance level of the tag are generated (step S5). The natural language processing unit 50 performs morphological analysis, context analysis, semantic analysis, and the like of the speech recognition text, and extracts keywords (important words) from the speech recognition text. And the importance of a keyword is produced | generated using TF-IDF method, for example. The importance is, for example, numerical information in the range of 0-100.

  The natural language processing unit 50 performs the same process as the voice recognition text on the text in the video data, and generates the importance of the character recognition tag. FIG. 6 shows an example of a character recognition tag and importance generated by the natural language processing unit 50. The first column in FIG. 6 is the video data identifier, the second column is the character recognition tag, the third column is the importance, and the fourth column is the time.

  The first column in FIG. 6 is the video data identifier, the second column is the character recognition tag, the third column is the importance, and the fourth column is the time. In this example, it is shown that the character recognition tags “autumn” and “Kinkakuji” are generated in one second from the start of the video data identified by the video data identifier A001.

  FIG. 7 illustrates a speech recognition tag generated by the natural language processing unit 50 and its importance. The column direction in FIG. 7 is the same as FIG. 6 except that the tag is a voice recognition tag.

  The example shown in FIG. 7 is a case where the speech recognition text is “Kinkakuji Temple with autumn leaves in the background. It is a stunning scenery reflecting its appearance on the surface of a pond against the background of autumn leaves of trees.” . From the speech recognition text, speech recognition tags of “Autumn”, “Autumn leaves”, “Kinkakuji”, “Kidachi”, “Pond”, and “Water surface” are generated.

  The importance of each speech recognition text is, for example, 90 for “Autumn”, 80 for “Autumn leaves”, 90 for “Kinkakuji”, 50 for “Kidachi”, and 60 for “Water”. The time when each voice recognition text was generated is 5 seconds for “Autumn”, 6 seconds for “Autumn leaves”, 8 seconds for “Kinkakuji”, 10 seconds for “Kidachi”, and 11 seconds for “Water”. . The speech recognition tag and importance notation after “water surface” are omitted.

  The importance generation unit 60 generates data indicating the importance of the object (object tag) recognized by the object recognition unit 20 (step S6). The importance may be obtained from the ratio of the object of the object tag to the image, for example. If the ratio of the image of “Kinkakuji” in the image is 60% or more, the importance is 90, for example, if the ratio of the image is 50% or less, the importance is 50. The importance is generated in correspondence. Alternatively, the likelihood output by the object recognition unit 20 may be converted into importance. For example, a value obtained by multiplying the likelihood by 100 may be used as the importance.

  FIG. 8 shows an example of importance generated by the importance generation unit 60. The columns and rows in FIG. 8 are the same as those in FIG. 4 except that the likelihood is replaced with the importance. The importance shown in FIG. 8 is a value obtained by multiplying the likelihood shown in FIG. 4 by 100.

  The importance integration unit 70, when two or three of the generated three types of tags match, or when two or three of the three types of tags are similar, the importance corresponding to the tag The tag importance is obtained based on the tag, and the tag is used as a video data tag, and the video data identifier corresponding to the tag importance, the video data tag, and the tag importance are associated and stored in the tag DB 80 (step S7). The generated three types of tags are the above-mentioned “object tag”, “character recognition tag”, and “voice recognition tag”.

  The tag match determination may be performed by partial match or after unification processing of half-width / full-width or uppercase / lowercase letters. The tag similarity determination is performed with reference to, for example, a similar word dictionary or an expression fluctuation dictionary provided in the importance level integration unit 70. In FIG. 1, the similar word dictionary and the expression fluctuation dictionary are not shown.

  The importance level integration unit 70 obtains the tag importance level based on the importance level corresponding to the matching or similar tag, and sets the matching or similar tag as the video data tag. A matching or similar tag is considered to be a tag that better represents the content of the video data.

  Therefore, for example, “Kinkakuji” in which the three types of tags (“object tag”, “character recognition tag”, “voice recognition tag”) coincide with each other is set as a video data tag, and the tag importance is, for example, the sum of each importance ( 70 + 90 + 90 = 250). Then, the video data identifier for identifying the video data, the video data tag, and the tag importance are stored in the tag DB 80 in association with each other.

  The processes in steps S1 to S7 described above are repeated until the process is completed for all video data (NO in step S8).

  On the other hand, the video data is stored in the video DB 91 in association with the video data identifier indicating the video data by the video management unit 90. Therefore, by searching the tag DB 80 in descending order of tag importance, the video data identifier of the video data desired to be viewed can be accurately extracted.

  Thereafter, the target video data can be extracted by searching the video DB 91 with the extracted video data identifier. Note that the video data can also be searched in scene units.

  The video search unit performs processing for searching the video data identifier of the target video data from the tag DB 80 and retrieving the video data corresponding to the video data identifier from the video DB 91. In FIG. 1, the notation of the video search unit is omitted.

  The video data processing method performed by the video data processing apparatus 1 according to the present embodiment described above recognizes an object included in the video data, generates a tag representing the object and the likelihood of the tag, and determines the importance of the object. Generating video data, detecting in-video text included in the video data, recognizing audio given to the video data to generate voice recognition text, extracting tags representing important words from the video text, A tag and an importance level of the tag are generated, a tag representing an important word is extracted from the speech recognition text, an importance level of the tag and the tag is generated, and two or three of the generated three kinds of tags If two match or two or three of the three types of tags are similar, the tag importance is calculated based on the importance corresponding to the tag, and the matching tag is used as the video data tag. Video data corresponding to importance An identifier, and the video data tag stored in the tag DB80 in association with the tag importance.

  According to the video data processing method of the present embodiment, it is possible to generate a database (tag DB 80) in which appropriate metadata is added to video data. Next, the operation of the importance level integration unit 70 will be described in more detail.

[Importance Integration Department]
The operation will be described with reference to the operation flow of the importance integration unit 70 shown in FIG.

  The importance level integration unit 70 determines whether one tag stored in the Doug DB 80 matches or is similar between the three tag types (object tag, character recognition tag, voice recognition tag). Determine (step S70). Hereinafter, an example in which the presence / absence of the same tag corresponding to the same video data identifier in FIGS. 6, 7 and 8 is determined will be described.

  If there is no matching tag, for example, for “water surface” where only the voice recognition tag exists, the voice recognition tag is used as it is as the video data tag (step S71), the importance of the voice recognition tag is set as the tag importance, and the video data identifier The data are stored in the tag DB 80 in association with each other (step S72). Then, the process returns to the determination of whether there is a match for other tags (object tag, character recognition tag, voice recognition tag) (NO in step S70 → NO in step S73). This repetition is repeated until all the video data is finished (YES in step S73).

  If there is a matching tag, it is determined whether or not all tag types match (step S74). When all the tag types match (YES in step S74), the importance level integration unit 70 obtains the tag importance level based on the importance level corresponding to the matching tag (step S75). The tag importance is, for example, a value obtained by doubling the total importance corresponding to matching tags. The tag that matches all the tags is “Kinkakuji” in the above example. The tag importance value may be an average value of each importance.

  Next, the importance level integration unit 70 uses the matching tag as a video data tag, stores the video data identifier corresponding to the tag importance level, the video data tag, and the tag importance level in the tag DB 80 in association with each other. (Step S76).

  If all the tags do not match (NO in step S74), the importance integrating unit 70 next determines whether or not the two tag types match (step S77). If the two tag types do not match, the process returns to step S70. That is, when there is no matching tag between different tag types, the importance value is not changed and is directly stored in the tag DB 80 as the tag importance.

  When the two tag types match (YES in step S77), the importance level integration unit 70 obtains the tag importance level based on the importance level corresponding to the matching tag (step S78). The tag importance when the tags match between the two tag types is, for example, a value obtained by multiplying the total importance corresponding to the matching tags by 1.5. The tag that matches the two tag types is “autumn” in the above example. Thus, the tag importance is a value obtained by multiplying the total value or average value of each importance by a greater weight as the number of matching tags increases.

  Note that the tag importance may be a value that considers the extraction accuracy for extracting matching tags. That is, the tag importance is a value that considers the extraction accuracy for extracting matching tags, and may be obtained by multiplying the importance corresponding to a tag with high extraction accuracy by a large weight. The extraction accuracy is the recognition accuracy of each of image recognition, character recognition, and speech recognition.

  Specifically, for example, when the recognition accuracy of the speech recognition text is higher than the recognition accuracy of the object recognition, the importance corresponding to the speech recognition tag extracted from the speech recognition text is multiplied by a large weight. When the recognition accuracy of object recognition is higher than the recognition accuracy of speech recognition text, a large weight is applied to the object tag recognized by object recognition. In short, tag importance = importance of object tag × weight α + importance of character recognition tag × weight β + importance of voice recognition tag × weight γ, and change each weight α, β, γ according to recognition accuracy May be.

  Next, the importance level integration unit 70 associates the tag of the two tag types as a video data tag, and associates the video data identifier corresponding to the tag importance level, the video data tag, and the tag importance level. Store in the tag DB 80 (step S79).

  FIG. 10 shows an example of the video data tag and tag importance stored in the tag DB 80. The first column in FIG. 10 is the video data identifier, the second column is the video data tag, and the third column is the tag importance. The tag importance of “Kinkakuji”, which is a video data tag that matches between all tag types, is the highest value at 360. The tag importance of “autumn”, which is a video data tag in which two character recognition tags and voice recognition tags are generated, is 270. The tag importance of “water surface”, which is a video data tag in which only a voice recognition tag is generated, is 60.

  In FIG. 11, the specific example of the character recognition tag memorize | stored in tag DB80, a voice recognition tag, and tag importance is shown. The object recognition tag is omitted for simplicity of explanation. The tag importance of the integrated tags is calculated with weight β = 0.7 and weight γ = 0.3, for example.

  From the left side of FIG. 11, columns of character recognition tags, voice recognition tags, and tag importance (integrated tags) are arranged, and the row direction is each tag and score. The remarks on tag importance show the formula for calculating the tag importance score.

  The tag “NTT West Japan (D1)” is integrated after unifying uppercase letters (B1) and lowercase letters (C3). “Recommended (B2)” and “One push (C2)” are integrated by the expression fluctuation dictionary. “Kansai (D3)” is integrated by partial matching of character strings (B4 and C5). “Nationwide (D5)” and “Weather (D6)” are integrated by perfect agreement.

  Tag importance of tags extracted by both character recognition and voice recognition can be integrated with a higher score than tags extracted by only one of them. For example, the tag importance of “Nationwide” and “Weather” detected by both character recognition and voice recognition has a higher score than “new service” extracted only by character recognition.

  As a result of the processing in steps S70 to S79 described above, the tag importance value corresponding to the tag increases as the number of matching tag types increases. The tag importance of tags that do not match between tag types does not change from the importance values generated by the importance generation unit 60 and the natural language processing unit 50.

  In this way, the video data processing apparatus 1 increases the tag importance value of the video data tag that better represents the content of the video data and stores it in the tag DB 80. According to the video data processing apparatus 1 of the present embodiment, a database in which appropriate metadata is added to video data can be generated.

  Next, a description will be given of a second embodiment in which the weight for multiplying the tag importance is changed based on the recognition accuracy for recognizing the tag.

[Second Embodiment]
FIG. 12 is a diagram illustrating a functional configuration example of the video data processing device 2 according to the second embodiment. The video data processing device 2 is different from the video data processing device 1 (FIG. 1) in that a recognition accuracy evaluation unit 92 is provided. The recognition accuracy evaluation unit 92 causes the tag importance level calculated by the importance level integration unit 70 when the likelihood output from the object recognition unit 20 is equal to or greater than the threshold value to be calculated by increasing the weight by which the score of the tag representing the object is multiplied.

  FIG. 13 is a diagram illustrating an example of an operation flow of the recognition accuracy evaluation unit 92. The recognition accuracy evaluation unit 92 acquires the likelihood of the tag representing the object from the object recognition unit 20 (step S92). The likelihood is information indicating that the tag representing the recognized object is likely to be closer to 1.0. That is, when the likelihood is large, it is considered that the recognition accuracy of the recognized object tag is high.

  Next, the recognition accuracy evaluation unit 92 determines whether or not the likelihood acquired from the object recognition unit 20 is greater than or equal to a threshold value (step S93). For example, the threshold value is 0.9. If the likelihood is 0.9 or more, the weight α multiplied by the score of the tag representing the object is doubled, for example (step S94). If the likelihood is less than 0.9, the weight α is not changed (NO in step S93).

  By doing in this way, when the tag representing the recognized object seems to be the object, the importance becomes large. As a result, it is possible to increase the tag importance of the video data tag obtained by the importance integration unit 70. That is, by providing the recognition accuracy evaluation unit 92, when the recognition accuracy of a tag representing an object is high, the tag importance can be increased and the tag importance can be sharpened.

  As described above, the video data processing device 2 generates a tag importance level corresponding to the recognition accuracy of the tag representing the object, and increases the tag importance level when the recognition accuracy is high and stores the tag importance level in the DB 80. According to the video data processing apparatus 2 of the present embodiment, a database in which appropriate metadata is added to video data can be generated.

[Third Embodiment]
FIG. 14 is a diagram illustrating a functional configuration example of the video data processing device 3 according to the third embodiment. The video data processing device 3 is different from the video data processing device 1 (FIG. 1) in that a recognition accuracy evaluation unit 93 is provided. The recognition accuracy evaluation unit 93 determines the tag importance obtained by the importance integration unit 70 when the area of the character region of the text in the video output by the text detection unit 30 in the video data is equal to or greater than a predetermined value, Increase the weight multiplied by.

  Here, the area of the character region is a rectangle (parallelogram) circumscribing the character. For example, the coordinates of each point are (0, 0), (1, 0), (1, 16), (0, 16). ), The character area is (1-0) × (16-0) = 16. The area of the text in which a plurality of characters are connected is obtained in the same manner, and divided by the number of characters.

  The area may be represented by the number of pixels. In the case of the number of pixels, the number of pixels in the above example is (1-0 + 1) × (16-0 + 1) = 34.

  FIG. 15 is a diagram illustrating a result of examining the relationship between the area of the character region and the matching rate of the recognition result. The precision is the sum of the number of characters correctly recognized divided by the number of characters. The horizontal axis is the area of the character area, and the vertical axis is the precision. In FIG. 15, 426 × 240 and others represent resolution.

  As shown in FIG. 15, it can be seen that if the area of the character region is 17 or more, the relevance rate is approximately 80% or more. As described above, when the area of the character region is equal to or larger than the predetermined value, it is effective to increase the tag importance and to add a sharpness to the tag importance.

  Furthermore, the recognition accuracy evaluation unit 93 may change the weight (weight multiplied by the score of the character recognition tag) in consideration of the complexity of the image (background) around the character. The complexity of the background can be expressed by, for example, Spatial perception information SI (reference document: ITU-T P.910 5.3.1).

  Also, the weight β multiplied by the score of the character recognition tag may be changed in consideration of the difference between the character color and the background color. The difference between the character color and the background color can be expressed by, for example, the Euclidean distance in the RGB space.

  Further, the size of the weight γ to be applied to the score of the speech recognition tag extracted from the speech recognition text may be changed according to the speech recognition accuracy. The speech recognition accuracy may be evaluated using, for example, the ratio of the sound information frequency band to the other frequency band sound. In FIG. 14, the connection lines connecting the speech recognition unit 40 and the recognition accuracy evaluation unit 93 are not shown.

  In the above embodiment, the example has been described in which the video data identifier indicating the video data is associated with each tag, but the present invention is not limited to this example. The scene divided by the scene dividing unit 10 may be associated with each tag. In this case, it is possible to search in units of scenes obtained by dividing the video data.

  Further, although an example of whether or not all of the object tag, the character recognition tag, and the voice recognition tag match has been described, it is not necessary to determine matching between all tag types. For example, it is only necessary to determine a match between a character recognition tag and a voice recognition tag. In that case, what is necessary is just to memorize | store an object tag and its likelihood in tag DB80 as it is.

  In the example shown in FIG. 9, tag matching is determined, but tag similarity may be determined as described above. Also, both tag matching and similarity may be determined.

  As described above, the present invention is not limited to the above-described embodiment, and various modifications can be made within the scope of the gist thereof.

  When the processing unit in the above apparatus is realized by a computer, the processing content of the function that each processing unit should have is described by a program. Then, by executing this program on a computer, the processing unit in the above apparatus is realized on the computer.

  The computer program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Further, the program may be distributed by storing the program in a recording device of the server computer and transferring the program from the server computer to another computer via a network.

1: Video data processing device 10: Scene division unit 20: Object recognition unit 30: Text detection unit 40: Audio recognition unit 50: Natural language processing unit 60: Importance generation unit 70: Importance integration unit 80: Tag Database 90: Video management unit 91: Video database 92, 93: Recognition accuracy evaluation unit

Claims (7)

A tag database that stores a video data identifier indicating the video data, a video data tag indicating the content of the video data, and a tag importance level indicating the importance level of the video data tag in association with each other;
An object recognition unit that recognizes an object included in the video data and generates a tag representing the object and a likelihood of the tag;
An importance generation unit for generating data indicating the importance of the object;
A text detector in the video data for detecting text in the video included in the video data;
A voice recognition unit that recognizes voice added to video data and generates voice recognition text;
A tag representing an important word is extracted from the text in the video, the tag and the importance of the tag are generated, a tag representing the important word is extracted from the speech recognition text, and the tag and the importance of the tag are generated. A natural language processing unit to
If two or three of the three generated tags match, or if two or three of the three types of tags are similar, tag importance based on the importance corresponding to the tag A video data identifier corresponding to the tag importance, the video data tag, and an importance level integration unit that stores the tag importance level in association with each other in the tag database. A video data processing apparatus comprising:   The tag importance is a value considering extraction accuracy for extracting the matching or similar tags, and is calculated by multiplying importance corresponding to a tag with high extraction accuracy by a large weight. The video data processing device according to 1. A recognition accuracy evaluation unit that receives the likelihood output by the object recognition unit;
The recognition accuracy evaluation unit, when the likelihood is equal to or greater than a threshold value, causes the tag importance obtained by the importance integration unit to be obtained by increasing a weight applied to a tag representing the object. The video data processing apparatus according to 1 or 2. A recognition accuracy evaluation unit that receives the in-video text,
The recognition accuracy evaluation unit evaluates character recognition accuracy from information of a character region of the text in the video, and the character recognition tag used when obtaining the tag importance obtained by the importance integration unit according to the evaluation result The video data processing apparatus according to any one of claims 1 to 3, wherein a weight to be multiplied by an importance level of the video data is changed. A recognition accuracy evaluation unit that receives the speech recognition text;
The recognition accuracy evaluation unit evaluates the recognition accuracy of the speech recognition text from acoustic information, and the importance of the speech recognition tag when obtaining the tag importance obtained by the importance integration unit according to the evaluation result The video data processing apparatus according to claim 1, wherein a weight to be multiplied by is changed. A video data processing method performed by a video data processing apparatus,
The video data processing device comprises:
For each video data, provided with a tag database that stores a video data identifier indicating the video data, a video data tag indicating the content of the video data, and a tag importance level indicating the importance level of the video data tag in association with each other,
In the video data processing method,
Recognizing an object included in video data, generating a tag representing the object and the likelihood of the tag,
Generating data indicating the importance of the object;
Detect text in video included in video data,
Recognize the voice attached to the video data to generate voice recognition text,
A tag representing an important word is extracted from the text in the video, the tag and the importance of the tag are generated, a tag representing the important word is extracted from the speech recognition text, and the tag and the importance of the tag are generated. And
If two or three of the three generated tags match, or if two or three of the three types of tags are similar, tag importance based on the importance corresponding to the tag The matching tag is a video data tag, and the video data identifier corresponding to the tag importance, the video data tag, and the tag importance are associated and stored in the tag database. Video data processing method.   A computer program for causing a computer to function as the video data processing apparatus according to claim 1.

Images