JP4473813B2 - Metadata automatic generation device, metadata automatic generation method, metadata automatic generation program, and recording medium recording the program - Google Patents

Description

  The present invention relates to an automatic metadata generation method for automatically generating metadata relating to the contents of video and images. In particular, metadata is automatically generated based on a technology for automatically recognizing character information in video and images. It relates to the technology to be generated.

  There is a high need for a technique for efficiently generating text information (hereinafter referred to as metadata) that describes video scenes and image contents necessary for realizing a video scene and image search service. In response to this need, many studies have been made on techniques for automatically recognizing character information displayed in video and images and defining the result as metadata. For example, it is described in Non-Patent Document 1 below. Exists.

  Examples of character information displayed in video and images include telop characters that explain the contents of news in news program videos and characters in outdoor signboards that explain store hours information, etc. It plays the role of explaining the explanation object accurately. For this reason, text information in video and images is very useful information as key information when searching for videos and images.

  The system technology described in Non-Patent Document 1 described above recognizes character information in video and images by character recognition technology, defines text information of the recognition result as video and image metadata, This is used as keyword information for image search.

The following Patent Document 1 describes “a character string information extracting device and method and a medium on which the method is recorded”.
Kan Yamamoto / Masanobu Enomoto / Kiyoshi Kurokawa / Hiroaki Udaka / Kazunari Yamayoshi / Taiyoshi Sasaki NTT Technology Journal VOL. 17, PP40-43, electrical engineering edition 2005.1. Japanese Patent Laid-Open No. 2001-76094 “Character String Information Extracting Apparatus and Method and Medium Recording the Method”

  The conventional technology on the system described in Non-Patent Document 1 is that the text information of one type of character recognition result as the whole image even when there are a plurality of character displays having various meaning contents in the video and the image. Is defined as metadata as it is.

Specifically, as shown in FIG. 1, when a plurality of types of characters such as names of persons, places, and event names are displayed in the image (FIG. 1 is an example of a video / image including character display, and a news video It shows the scene of the typhoon report in the middle, with the date and time information “20th (Tue)”, the event name “Typhoon No. 15, approaching” and the character string “Sato reporter”. 2) As shown in FIG. 2, as a character recognition result, a single character string containing the date and time, event name, and personal name of “20th (Tue) Typhoon No. 15, approaching Sato reporter” is stored as meta. Set as data. FIG. 2 shows an example in which one character string “20th (Tue) Typhoon No. 15, approaching Sato reporter” is set in one item (telop recognition result tag) of metadata expressed in XML format. Is. ... (A)
On the other hand, when searching for information using keywords, in general, specify the item name of the information you want to search for, such as the name of a person or place, in addition to the keyword character string, so that unnecessary information does not appear as search results. This is often the case (see FIG. 3). For the user, unnecessary information does not appear as a search result, so that there is an advantage that it is possible to reach the information that the user wants to know in a shorter time than when no item is specified. ... (B)
In order to realize (B), it is necessary to store not only the text to be searched but also the item name and the set that the text means in the text information database for search. When it is desired to perform a search as shown in FIG. 3, the text “typhoon” needs to be stored in the database as information indicating that the item is “news event”.

  However, when the text data for search is created by automatically recognizing the characters displayed in the video and image, the conventional technique has the level of realization as described in (A) above, In other words, it is stored in the database only as a single character string in which character strings of various items such as place names are mixed, and it is not possible to provide the user with the benefits of using the search as described in (B) above. There is.

  The present invention has been made in consideration of the above points, and even when a character string composed of a plurality of items is mixed in a video and an image, as a result of automatic character recognition, a mixed character string as shown in FIG. Instead, the object name and the character string corresponding to the item name are stored in a database as a set so that the user can extract desired information more efficiently than the prior art.

In order to solve the above-described problem, an automatic metadata generation apparatus according to claim 1 of the present invention includes a video / image data reading unit that reads video / image data in which one or more character strings are displayed on a screen, and The video / image data read by the video / image data reading unit is binarized to extract a character area, and a luminance edge is extracted from the read video / image data. All display character strings in video / image data are determined by counting the area and luminance edge for each horizontal line in the image, and determining that the horizontal line whose counted value is equal to or greater than the threshold is within the character string rectangle. vertical coordinate values of the vertices of a circumscribed rectangle of a character string rectangle extracting unit for obtaining character string rectangle information is the size information, the character pattern in the rectangular of the character string rectangle extracting unit in the image extracted in A character recognition unit that performs character recognition processing on the character string and outputs text data as a character recognition result, character string rectangle information extracted by the character string rectangle extraction unit, a metadata item name, and a corresponding character in the image When the information on the upper and lower limits of the vertical coordinate value of the vertex of the column rectangle and the condition information related to the size are compared with the string rectangle condition in the metadata item name database defined by one or more, and the condition is met A metadata item name search unit that outputs a metadata item name corresponding to the condition, a metadata item name output from the metadata item name search unit, and a character recognition result obtained by the character recognition unit and a metadata output unit for outputting as the metadata item name search unit, a vertical coordinate value of the vertex of the circumscribed rectangle, which is defined by the character string rectangle conditionals Compared to upper and lower vertical coordinate values of the vertices of the string rectangles and the result when within the range of upper and lower limits, and wherein the determining and matching with the condition.

Specifically, a video / image data reading unit that reads video / image data in which one or a plurality of character strings are displayed on the screen, and the video / image data read by the video / image data reading unit The predetermined character area extraction process and the luminance edge information extraction process are performed, the extracted character area and the luminance edge are counted for each horizontal line in the image, and the horizontal line where the counted value is equal to or greater than the threshold is A character string rectangle extraction unit that obtains character string rectangle information as vertical coordinate values and size information of circumscribed rectangles of all display character strings in video / image data by determining that they are within the character string rectangle And character recognition processing using a predetermined method for character patterns in the rectangle in the image extracted by the character string rectangle extraction unit, and text data is obtained as a character recognition result. A character recognition unit which outputs the metadata item name and upper and lower vertical coordinate values of the vertexes of the character string rectangle in the image corresponding thereto, the condition of the information about the size of one or a plurality defined metadata items, Compare the name database, the character string rectangle information extracted by the character string rectangle extraction unit and the character string rectangle condition in the metadata item name database, and if the condition matches, the metadata item corresponding to the condition A metadata item name search unit that outputs a name; a metadata output unit that outputs a metadata item name output from the metadata item name search unit and a character recognition result obtained by the character recognition unit as a set ; The metadata item name search unit includes a vertical coordinate value of the vertex of the circumscribed rectangle, an upper limit of a vertical coordinate value of the vertex of the character string rectangle defined by the character string rectangle condition And compared with the lower limit, the result may in the range of upper and lower limits, and wherein the determining and matching with the condition.

In order to solve the above problem, the automatic metadata generation method according to claim 2 of the present invention reads video / image data for reading video / image data in which a single character string or a plurality of character strings are displayed on the screen. And binarizing the video / image data read in the video / image data reading step to extract a character area; and extracting a luminance edge from the read video / image data; The displayed character area and brightness edge are counted for each horizontal line in the image, and all the display in the video / image data is determined by determining that the horizontal line whose counted value is equal to or greater than the threshold is within the character string rectangle. vertical coordinate values of the vertices of a circumscribed rectangle of a character string, a character string rectangle extracting step of acquiring a character string rectangle information is the size information, extracted by the character string rectangle extraction step A character recognition step of performing character recognition processing on a character pattern in a rectangle in the image and outputting text data as a character recognition result, character string rectangle information extracted in the character string rectangle extraction step, and metadata items Information on the upper and lower limits of the vertical coordinate value of the name and the corresponding vertex of the string rectangle in the image, and information on the condition related to the size, or a string rectangle condition in the metadata item name database defined in multiple If the condition matches, the metadata item name search step for outputting the metadata item name corresponding to the condition, the metadata item name output from the metadata item name search step and the character recognition the character recognition result obtained in step a metadata output step of outputting as a set, the metadata item name search stearate Compares the vertical coordinate values of the vertices of the circumscribed rectangle with the upper and lower limits of the vertical coordinate values of the vertices of the character string rectangle defined by the character string rectangle condition, and the result is within the upper and lower limit ranges. In some cases, it is determined that the condition is met .

Specifically, a video / image data reading step for reading video / image data in which one or a plurality of character strings are displayed on the screen, and the video / image data read in the video / image data reading step The predetermined character area extraction process and the luminance edge information extraction process are performed, the extracted character area and the luminance edge are counted for each horizontal line in the image, and the horizontal line where the counted value is equal to or greater than the threshold is A character string rectangle extraction step for acquiring character string rectangle information as vertical coordinate values and size information of circumscribed rectangles of all display character strings in video / image data by determining that the character string rectangle is present And character recognition processing using a predetermined method for the character pattern in the rectangle in the image extracted in the character string rectangle extraction step. A character recognition step of outputting the text data as a result, a character string rectangle information extracted by the character string rectangle extraction step, the upper limit of the metadata item name and vertical coordinate values of the vertexes of the character string rectangle in the image corresponding thereto If the condition information regarding the lower limit and size is one or more than one defined in the metadata item name database, and the condition is matched, if the condition matches, the metadata item corresponding to the condition and metadata item name search step of outputting the name, and a meta data output step of outputting as the metadata item name search set character recognition result obtained by the metadata item name and the character recognition step output from step The metadata item name search step includes a vertical coordinate value of a vertex of the circumscribed rectangle defined by the character string rectangle condition. Compared to upper and lower vertical coordinate value of the character string rectangle vertices and the results when within range of the upper and lower limits, and wherein the determining and matching with the condition.

  In order to solve the above problem, the metadata automatic generation program according to claim 3 of the present invention is configured as a program for causing a computer to execute each step in the metadata automatic generation method according to claim 2. It is characterized by that.

  In order to solve the above problem, a recording medium on which the metadata automatic generation program according to claim 4 of the present invention is recorded is a recording medium on which the computer can read the metadata automatic generation program according to claim 3. It is characterized by being recorded in

  Some images and images in which characters are displayed generally have a fixed display layout pattern according to the content of the characters to be displayed. An example is shown in FIG. FIG. 4A shows telop character information of a news headline displayed at the beginning of each news topic in a news program. In Japan, within the same news program, telop characters having the same position, size and design in the image are displayed at the beginning of each news topic.

  As another example, FIG. 4B is an example of a credit display displayed at the beginning of a video produced by a video production company or the like. The credit display includes characters describing the video content such as the program production date, the program title, the producer, and the broadcast date. As for this credit display, as long as it is produced by the same video production company, the characters in the credit display of any video are displayed in the same position, size and design in the image. In the present invention, using this property, the item name indicated by the character content is also recognized from the layout information such as the position, size, and design of the character in the image, and both the item name and the text of the character recognition result are recognized. It is stored in a database.

  That is, according to the present invention, as described in claim 1, not only the character pattern displayed in the input video / image is recognized and the recognition result text is acquired, but also the position and size of the circumscribed rectangle of the character string By extracting such layout information, it is possible to obtain the metadata item name in the metadata item name database. Thus, according to the present invention, instead of outputting the recognition result of the character display in the image as one text data as in the prior art, the text data of the metadata item name and the character recognition result in character string units. Therefore, the problem to be solved by the conventional technique can be solved.

  According to the present invention, even when a character string consisting of a plurality of items is mixed in a video and an image, as a result of automatically recognizing the character, the metadata item name and the corresponding character string are stored in a database as a set, It is possible to create a state in which the user can extract desired information more efficiently than in the prior art, and the effect that the efficiency of the video / image search processing can be realized is brought about.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiments. FIG. 5 shows an example of a specific apparatus configuration of the metadata automatic generation apparatus according to claim 1 as the first embodiment of the present invention.

  The video / image data reading unit 1 in FIG. 5 reads video / image data in which one or more character strings are displayed on the screen. Specifically, the video / image data reading unit 1 in a VTR (video tape recorder) device is used. The head office can be realized by capturing the output video signal with a personal computer equipped with a video capture board, digitizing it, and writing it in a memory area or hard disk.

  The character string rectangle extraction unit 2 in FIG. 5 performs predetermined character region extraction processing and luminance edge information extraction processing on the video / image data read by the video / image data reading unit 1, Based on the obtained character region and the distribution density of the luminance edge in the image, the position and size information of the circumscribed rectangle of all display character strings in the video / image data is acquired.

  Specifically, it can be realized by implementing a method for realizing the headquarters as a software program on a personal computer. As a method for realizing the headquarters, for example, the method proposed in Patent Document 1 “Character string information extraction device and method and recording medium recording the method” can be used.

  FIG. 6 shows a supplementary diagram for explaining a method of realizing the character string rectangle extraction unit 2. FIG. 6A shows an input image including a character display, and FIG. 6B shows a result of extracting a character region by binarization processing of the image with respect to FIG. 6A. A white portion in the image is a character area. FIG. 6C shows the result of extracting the luminance edge with respect to FIG. FIG. 6D is a histogram obtained by counting the character area and luminance edge information in FIGS. 6B and 6C for each horizontal line in the image. By extracting the peaks of the histogram by threshold processing, the position and size of the character string rectangle can be calculated.

  The character recognition unit 3 in FIG. 5 performs character recognition processing using a predetermined method for the character pattern in the rectangle in the image extracted by the character string rectangle extraction unit 2, and text data is obtained as a character recognition result. Can be realized by software on a computer.

  The metadata item name database 4 in FIG. 5 is one in which the metadata item name and corresponding information on the position and size of the character string rectangle in the image are defined by one or more. Can be realized as information stored in a hard disk or memory on a computer. FIG. 7 shows an example of the contents of the metadata item name database.

In the table of FIG. 7, a program name, broadcast date / time, genre, and producer are set as examples of metadata item names in the left column. In the right column, a character string rectangle condition in the image corresponding to the metadata item name in the left column is set. In the case of the metadata item “program name”, as the condition of the character string rectangle in the corresponding input image,
・ Horizontal coordinates of the upper left vertex of the rectangle: 50 to 60 ・ Vertical coordinates of the upper left vertex of the rectangle: 250 to 260 ・ Rectangle width: 300 to 350 ・ Rectangle height: 40 to 50 An example is shown.

  When the metadata item name search unit 5 in FIG. 5 compares the character string rectangle information extracted by the character string rectangle extraction unit 2 with the character string rectangle condition in the metadata item name database 4 and matches the condition. The metadata item name corresponding to the condition is output, and can be realized by implementing this method as software on a computer. FIG. 8 shows an example of a specific processing flow of this processing unit.

  In step S21 in FIG. 8, the coordinates, width, and height values of the upper left vertex of the character string rectangle extracted by the character string rectangle extracting unit 2 in FIG. 5 are input, and the process proceeds to step S22.

  In step S22 in FIG. 8, 1 is set to the counter variable N when referring to the metadata item name stored in the metadata item name database 4 in FIG. 5, and the process proceeds to step S23. At this time, let M be the number of all metadata item names stored in the metadata item name database.

  Step S23 in FIG. 8 determines whether the coordinates, width, and height of the upper left vertex of the character string rectangle input in step S21 satisfy the character string rectangle condition of the Nth metadata item in the metadata item name database. If it is satisfied, the process proceeds to step S26. Otherwise, the process proceeds to step S24.

As a specific processing result of this step, for example, the horizontal coordinate of the upper left vertex of the character string rectangle input in step S21 is 55, the vertical coordinate is 250, the width is 320, and the height is 45. When the setting content in the item name database is FIG. 7, the input value is a condition of a character string rectangle corresponding to the metadata item name “program name” with N = 1,
・ Horizontal coordinates of the upper left vertex of the rectangle: 50 to 60 ・ Vertical coordinates of the upper left vertex of the rectangle: 250 to 260 ・ Rectangle width: 300 to 350 ・ Rectangle height: 40 to 50 . In such a case, the process proceeds to step S26. If the condition is not satisfied, the process proceeds to step S24.

  Step S24 in FIG. 8 determines whether the counter variable N when referring to the metadata item names stored in the metadata item name database 4 in FIG. 5 is equal to or less than the number M of all metadata item names. If N <= M, the process proceeds to step S25. Otherwise, the processing result is terminated without outputting the search result of the metadata item name.

  Step S25 in FIG. 8 is a step of incrementing the counter variable N by 1 when referring to the metadata item name stored in the metadata item name database 4 in FIG. 5, and the process proceeds to step S23.

Step S26 in FIG. 8 is a step of outputting the Nth metadata item in the metadata item database. For example, in the case of the processing example of step S23, the metadata item name “program name” is output. Is. With the end of this step, the processing of the entire processing unit is also ended.

The metadata output unit 6 in FIG. 5 outputs the metadata item name output from the metadata item name search unit 5 and the character recognition result obtained by the character recognition unit 2 as a set. For example, when the metadata item name output from the metadata item name search unit 5 is “program name” and the character recognition result obtained by the character recognition unit 2 is “fresh news”, “program name” ”And“ fresh news ”are the output results of the metadata output unit.

  Another example is shown in FIG. FIG. 9 shows an example in which the conventional character recognition result shown in FIG. 2 is converted into metadata for each metadata item name according to the present invention. According to the prior art, as shown in FIG. 2, the metadata as one character string “20th (Tue) Typhoon No. 15, approaching Sato reporter”, which is a mixture of all the character strings in the image of FIG. In contrast, according to the present invention, the character string “20th (Tue) Typhoon No.15, approaching Sato reporter” is replaced by “20th (Tue)”, “Typhoon No.15, approaching”, “Sato reporter” And can be output together with metadata item names such as “date”, “news title”, and “reporter name” as shown in FIG.

  Thus, according to the present invention, even when a character string consisting of a plurality of items is mixed in the video and image, as a result of automatically recognizing the character, the metadata item name and the corresponding character string are stored in the database as a set. It is possible to store and create a state that allows the user to retrieve the desired information more efficiently than in the prior art.

  Note that the metadata item name database 4 may be provided outside so that it can be accessed from the metadata automatic generation device.

  Also, a recording medium in which the program code of software for realizing the functions and processing of the metadata automatic generation apparatus in FIGS. 5 and 8 is supplied to the system or apparatus, and the CPU (MPU of the system or apparatus) is supplied. It is also possible to read and execute the program code stored in the recording medium. In this case, examples of the recording medium on which the program code read from the recording medium is recorded include a CD-ROM, DVD-ROM, CD-R, CD-RW, MO, and HDD.

It is explanatory drawing which showed the example of the image | video and image containing a character display, and the scene of the typhoon report relay in a news image | video. It is explanatory drawing which showed the usage example as metadata of the character recognition result by a prior art. It is explanatory drawing which showed the example of the keyword search screen which designated the item name. An example of a standard layout pattern of character display in an image is shown, (a) is an explanatory diagram of a screen containing telop characters, and (b) is an explanatory diagram of a credit screen. It is a block diagram which shows the structure of the metadata automatic generation apparatus in one example of embodiment of this invention. It is explanatory drawing which shows the process image of the character string rectangle extraction part in one example of embodiment of this invention. It is explanatory drawing which shows an example of the content of the metadata item name database in the example of 1 embodiment of this invention. It is a flowchart which shows the flow of a specific process of the metadata item name search part in the example of 1 embodiment of this invention. It is explanatory drawing which shows the example of the metadata output result by one Example of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Video | video data reading part, 2 ... Character string rectangle extraction part, 3 ... Character recognition part, 4 ... Metadata item name database, 5 ... Metadata item name search part, 6 ... Metadata output part

Claims (4)

A video / image data reading unit that reads video / image data in which one or more character strings are displayed on the screen,
The video / image data read by the video / image data reading unit is binarized to extract a character area, and a luminance edge is extracted from the read video / image data, and the extracted character area And the luminance edge for each horizontal line in the image, and by determining that the horizontal line whose counted value is equal to or greater than the threshold is within the character string rectangle, all the display character strings in the video / image data are A character string rectangle extraction unit that acquires character string rectangle information that is the vertical coordinate value and size information of the vertex of the circumscribed rectangle;
A character recognition unit that performs character recognition processing on a character pattern in a rectangle in the image extracted by the character string rectangle extraction unit, and outputs text data as a character recognition result;
The character string rectangle information extracted by the character string rectangle extraction unit, the metadata item name and the upper and lower limits of the vertical coordinate value of the vertex of the character string rectangle in the image corresponding thereto, information on the condition regarding the size, Alternatively, a plurality of metadata item name databases are compared with a character string rectangle condition, and when the condition is met, a metadata item name search unit that outputs a metadata item name corresponding to the condition;
A metadata output unit that outputs a metadata item name output from the metadata item name search unit and a character recognition result obtained by the character recognition unit as a set ;
The metadata item name search unit compares the vertical coordinate value of the vertex of the circumscribed rectangle with the upper limit and lower limit of the vertical coordinate value of the vertex of the character string rectangle defined by the character string rectangle condition, and the result is the upper limit. And an automatic metadata generation apparatus that determines that the condition is met when the value is within a lower limit range . A video / image data reading step for reading video / image data in which one or more character strings are displayed on the screen,
The video / image data read in the video / image data reading step is binarized to extract a character area, and a luminance edge is extracted from the read video / image data, and the extracted character area And the luminance edge for each horizontal line in the image, and by determining that the horizontal line whose counted value is equal to or greater than the threshold is within the character string rectangle, all the display character strings in the video / image data are A character string rectangle extraction step for acquiring character string rectangle information which is vertical coordinate values and size information of the circumscribed rectangle vertices ;
A character recognition step of performing character recognition processing on a character pattern in a rectangle in the image extracted in the character string rectangle extraction step, and outputting text data as a character recognition result;
The character string rectangle information extracted in the character string rectangle extraction step, the upper and lower limits of the vertical coordinate value of the vertex of the character string rectangle in the image corresponding to the metadata item name, and information on the condition regarding the size are singular, Or, comparing a plurality of defined metadata item name databases with a character string rectangle condition, and if the condition matches, a metadata item name search step for outputting a metadata item name corresponding to the condition;
A metadata output step for outputting the metadata item name output from the metadata item name search step and the character recognition result obtained in the character recognition step as a set ;
The metadata item name search step compares the vertical coordinate value of the vertex of the circumscribed rectangle with the upper and lower limits of the vertical coordinate value of the vertex of the character string rectangle defined by the character string rectangle condition, and the result is the upper limit. And an automatic metadata generation method characterized in that if it is within the lower limit range, it is determined that the condition is met . An automatic metadata generation program configured as a program for causing a computer to execute each step in the automatic metadata generation method according to claim 2. A computer-readable recording medium on which the metadata automatic generation program according to claim 3 is recorded.