JP5503507B2 - Character area detection apparatus and program thereof - Google Patents

Description

  The present invention relates to a character area detection device that detects a character area, which is a character area included in an input image, from the input image and a program thereof.

  Subtitles added to video such as television broadcasts are very useful information because they can be used for adding metadata to video and analyzing the content of the video. Therefore, various techniques for detecting a caption area have been proposed.

  For example, the technique described in Non-Patent Document 1 detects edges from each frame image of an input video, and extracts only edges that continue to appear at the same place for a certain period of time. Thereafter, the technique described in Non-Patent Document 1 detects an area where the extracted edges are dense as a caption area.

The technique described in Non-Patent Document 2 detects a caption area based on motion compensation information from a video decoded by an MPEG (Moving Picture Experts Group) method.
In addition, the technique described in Non-Patent Document 3 moves a caption area from a set of edge pairs (edge pairs whose gradient directions are reversed) between frame images so that captions moving in a video can also be handled. The parameter is estimated.
The techniques described in Non-Patent Documents 2 and 3 use a technique for detecting a caption area based on the denseness of temporally continuous edges, as in Non-Patent Document 1.

Hagio, Ariki, Ogata, "Cross-media passage search-Search method for utterance documents using telop and CG flip character strings as search questions-", IEICE Transactions, Vol. J84-D-II, no. 8, pp. 1809-1816, 2001 Sato, Niikura, Taniguchi, Akutsu, Tonomura, Hamada, "High-speed telop area detection method from MPEG encoded video", IEICE Transactions, Vol.J81-D-II, No.8, pp.1847- 1855, 1998 Mori, Kurakake, Sugimura, Shio, Suzuki, "Telop character recognition in video using shape characteristics of background and characters and dynamic correction discriminant function", IEICE Transactions, Vol.J83-D-II, No. 7, pp.1658-1666, 2000

However, in the technologies described in Non-Patent Documents 1 to 3, since the subtitles are based on empirical knowledge (experience knowledge of crowding of edges) that the subtitles continue to be displayed at the same place for a certain period of time, A subtitle whose subtitle size changes over time cannot be detected. In addition, in the techniques described in Non-Patent Documents 1 to 3, if a background or an object other than captions appears in the same place for a certain period of time, there is a risk of erroneous detection.
In addition, in the techniques described in Non-Patent Documents 1 to 3, the subtitle area and other areas are classified based on empirical knowledge of edge crowding, so that a complicated background with many edges appears. The subtitles that appear cannot be detected.

In the techniques described in Non-Patent Documents 1 to 3, since the degree of edge density changes depending on the size of the subtitles (font size), there is a risk of omission of detection when subtitles of various sizes appear. .
Furthermore, in the techniques described in Non-Patent Documents 1 to 3, a plurality of continuous frame images are necessary for detecting the subtitle area, so that the subtitles cannot be detected from one frame image.

  SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to solve the above-described problems and provide a character area detection device and a program therefor that can accurately detect a character area from one image.

  In view of the above-described problems, a character area detection device according to the first invention of the present application is a character area detection device that detects a character area, which is a character area included in an input image, from the input image. A scanning means, an image feature vector calculation means, a character candidate area determination means, a character detection area determination means, an enlargement means, and a character area output means.

  According to such a configuration, the character area detection device receives the input image and converts the input image into one or more low-resolution images having a resolution lower than that of the input image by the resolution conversion unit. In addition, the character area detection apparatus scans the input image and the low resolution image for each of the input image and the low resolution image by scanning the low resolution image and the input image converted by the resolution conversion unit with the scanning window having the same size. A scanning window area image corresponding to the window area is generated.

For example, when a character is included in the input image, the character of the low resolution image becomes smaller as the resolution of the low resolution image becomes lower. On the other hand, the size of the scanning window is constant regardless of the resolution of the input image and the low resolution image. That is, the character area detection device generates scanning window area images in which the same characters as the input image appear in various sizes by the scanning unit.
Further, since the character area detection device scans the entire input image and the low-resolution image by the scanning unit, the character area can be detected regardless of the character appearance position.

  Further, the character area detecting device calculates a feature vector of the scanning window area image generated by the scanning unit for each of the input image and the low resolution image by the image feature vector calculating unit. Then, the character region detection device detects whether the scanning window region image is a character based on the feature vector of the scanning window region image calculated by the image feature vector calculation unit for each input image and low resolution image by the character candidate region determination unit. It is determined by machine learning whether it is a candidate area. In other words, the character region detection device can detect the character region regardless of the character size because the character candidate region determination unit determines the character candidate region using the scanning window region image in which the characters of the input image appear in various sizes. .

  The character area detection device calculates the number of times that the character candidate areas determined by the character candidate area determination unit overlap each other for each of the input image and the low-resolution image by the character detection area determination unit. Character candidate areas that are equal to or greater than the set first threshold are determined as character detection areas.

In addition, the character area detection device uses the enlargement unit to convert the character detection area for each low resolution image determined by the character detection area determination unit into an enlargement ratio at which the low resolution image corresponding to the character detection area has the same resolution as the input image. Zoom in. Then, the character area detection device determines whether any one or more of the character detection area for each low resolution image enlarged by the enlargement means and the character detection area of the input image overlap by the character area output means. Of the character detection areas determined to overlap with each other, the ratio of the overlap of other character detection areas to the reference character detection area that is the character detection area corresponding to the input image or the low resolution image having the maximum resolution is calculated. At the same time, when the calculated overlapping ratio is equal to or greater than a preset second threshold, only the reference character detection area is output as the character area. That is, the character area detection device integrates a plurality of character detection areas corresponding to the same character into one character area by the character area output means.
As a result, the character area detecting device can detect a character area appearing on a complicated background without depending on empirical knowledge of edge crowding.

The character area is information indicating the position (coordinates), shape, and size of an area where a character appears in the input image.
The caption candidate area is information indicating the position (coordinates), shape, and size of a candidate area for the character detection area for each of the input image and the low-resolution image.
The caption detection area is information indicating the position (coordinates), shape, and size of an area where characters appear for each of the input image and the low-resolution image.

Further, the character area detection device according to the second invention of the present application calculates, as an evaluation value, the size of the character detection area or the density of the edge strength determined by the character detection area determination unit for each of the input image and the low resolution image. It is determined whether or not the calculated evaluation value is equal to or less than a preset third threshold value, and a character detection area determined to be equal to or less than the third threshold value is determined from the character detection area determined by the character detection area determination unit. It is preferable to further include character detection area deletion means for deletion.
According to this configuration, when the evaluation value is equal to or less than the third threshold value, the character area detection device determines that the character detection area has been erroneously detected, and deletes the character detection area.

  In the character area detection device according to the third aspect of the present invention, the character area output means determines that the character detection area is determined not to overlap with the character detection area, and the calculated overlapping ratio is less than the second threshold value. It is preferable to further output the character detection area as a character area.

  The character area detection device according to the fourth aspect of the present invention further includes a video decoding unit that decodes the input video and outputs a frame image constituting the video to the resolution conversion unit as an input image when the video is input. The character area output means outputs a character area for each frame image, and the character area for each frame image output by the character area output means exists in another frame image within a preset range from the frame image. And a character area deleting unit that deletes a character area determined not to exist in another frame image from the character area for each frame image output by the character area output unit. Is preferred.

  In ordinary video, characters such as subtitles are rarely displayed for only one frame. Therefore, the character area detecting device determines that the character area is erroneously detected and deletes the character area when the character area detected from a certain frame image does not exist in the frame image in a certain range by the character area deleting unit.

  The character area detection apparatus according to the first invention of the present application is a general computer that includes a resolution conversion means, a scanning means, an image feature vector calculation means, a character candidate area determination means, a character detection area determination means, an enlargement means, and a character area. It can also be realized by a character area detection program that functions as output means.

According to the present invention, the following excellent effects can be obtained.
According to the first invention of the present application, since the entire input image and the low-resolution image are to be scanned, the character area can be detected regardless of the character appearance position, and the scanning window area in which the characters of the input image appear in various sizes Since an image is used, a character area can be detected regardless of the character size. Thus, according to the first invention of the present application, even when the character region appears on a complicated background, the character region is detected from one image with high accuracy without depending on empirical knowledge of edge crowding. be able to.

According to the second aspect of the present invention, since the character detection area having a high possibility of being erroneously detected is deleted, the detection accuracy of the character area can be further increased.
According to the third aspect of the present invention, the character detection areas that do not overlap with each other and the character detection areas with the overlapping ratio less than the second threshold are output as the character areas, and the detection accuracy of the character areas can be further increased.
According to the fourth aspect of the present invention, since the character area that has a high possibility of being erroneously detected is deleted, the detection accuracy of the character area can be further increased.

It is a block diagram which shows the structure of the video caption detection apparatus which concerns on embodiment of this invention. It is a figure explaining the scanning by the scanning means of FIG. It is a figure explaining the determination of a caption candidate area by the caption candidate area determination means of FIG. It is an image explaining determination of a caption detection area by the caption detection area determination means of FIG. It is an image explaining the labeling process by the labeling means of FIG. It is a figure which shows an example of the subtitle detection area which the subtitle detection means of FIG. 1 detected. It is a figure explaining expansion of a caption detection area by the expansion means of FIG. (A)-(c) is a figure explaining the integration of a caption detection area by the caption detection area integration means of FIG. It is a flowchart which shows the whole operation | movement of the video caption detection apparatus of FIG. It is a flowchart which shows the subtitle detection area detection processing by the subtitle detection means of FIG. In the Example of this invention, it is an image which shows the 1st example of the image | video for experiment. In the Example of this invention, it is an image which shows the 2nd example of the image | video for experiment.

[Configuration of video caption detection device]
In the present embodiment, a video caption detection device that detects a caption region included in a video will be described as an example of a character region detection device according to the present invention.
Hereinafter, the configuration of the video subtitle detection apparatus 1 will be described with reference to FIG. As shown in FIG. 1, a video subtitle detection apparatus 1 detects a subtitle area included in a video from the video, and includes video decoding means 10, resolution conversion means 20, storage means 30, and subtitle detection means. 40 and detection result integration means 50.

The video input to the video caption detection device 1 is composed of continuous frame images (input images), and each frame image can be specified by a time code or a frame number.
Also, it is assumed that subtitles are included in this video. For example, the subtitles can include character images, telops, open captions, or scroll supers that are redundant in the video.
The caption area is information indicating the position (coordinates), shape, and size of the area where the caption appears in each frame image of the video, and the time (frame image) when the caption appears in the video.

  The video decoding unit 10 receives a video, decodes the input video, and outputs a frame image constituting the video to the resolution conversion unit 20 as an input image. For example, when an MPEG2 format video is input, the video decoding means 10 is a general MPEG2 format decoder.

The resolution conversion unit 20 receives the frame image from the video decoding unit 10 and converts the input frame image into a low resolution image having a lower resolution than the input image. Specifically, the resolution conversion unit 20 reduces the frame images by the reduction rate α- ^i, and converts the frame images into low resolution images having N types of resolutions. Then, the resolution conversion unit 20 causes the storage unit 30 to store the frame image input from the video decoding unit 10 and the converted low resolution image.

  Here, α is a constant (for example, √2), N is the type of resolution (for example, 6), and i is a counter that satisfies 0 ≦ i <N. Α and N are set to arbitrary values by the operator of the video caption detection device 1.

  The storage means 30 is a storage device such as a memory or a hard disk that stores a frame image, a low-resolution image, a caption candidate area (character candidate area) to be described later, and a caption detection area (character detection area) to be described later.

The caption candidate area is information indicating the position (coordinates), shape, and size of a candidate area for the caption detection area for each of the frame image and the low-resolution image.
The caption detection area is information indicating the position (coordinates), the shape, and the size of the area where the caption appears for each of the frame image and the low resolution image.
In this embodiment, in order to specify the time in the video, the time code of the original frame image is added to the low-resolution image, the caption candidate area, and the caption detection area.

  The caption detection means 40 detects a caption detection area by performing a caption detection area detection process on each of the frame image and the low resolution image. For this reason, the caption detection means 40 includes a scanning means 41, an image feature vector calculation means 42, a caption candidate area determination means (character candidate area determination means) 43, and a caption detection area determination means (character detection area determination means) 44. And labeling means 45, circumscribed area calculating means 46, and filtering means (character detection area deleting means) 47.

  The scanning unit 41 scans the frame image and the low resolution image stored in the storage unit 30 with the scanning window having the same size, thereby scanning the frame image and the low resolution image corresponding to the region of the scanning window. A window area image is generated.

  As shown in FIG. 2, the scanning window Wi has an initial position at the upper left of the image. First, the scanning unit 41 moves the scanning window Wi little by little in the right direction (horizontal direction) from the initial position until the scanning window Wi reaches the right end, and scan image area images corresponding to the area of the scanning window Wi are displayed. Generate. Next, when the scanning window Wi reaches the right end, the scanning unit 41 moves the scanning window Wi slightly downward (vertical direction) from the initial position, and then repeats the horizontal operation as described above. In this way, the scanning unit 41 scans the entire frame image and the low resolution image to generate a scanning window region image.

Here, the scanning window Wi has a constant size (for example, 16 pixels in the vertical direction × 16 pixels in the horizontal direction) in all the frame images and the low-resolution images regardless of the resolutions of the frame images and the low-resolution images.
Note that the number of moving pixels in the horizontal and vertical directions of the scanning window Wi is set to an arbitrary value (for example, 4 pixels) by the operator.

  The image feature vector calculating unit 42 calculates a feature vector of the scanning window region image generated by the scanning unit 41 for each frame image and low resolution image. Here, the image feature vector calculation means 42 calculates a feature vector based on the shape or a feature vector based on the color / texture from the scanning window region image.

<First Example: Feature Vector Based on Shape>
Hereinafter, a first example and a second example of the feature vector calculation method will be described in order.
When a feature vector based on the shape is used, the image feature vector calculation unit 42 performs edge detection processing (for example, a first-order differential filter such as a Sobel filter, a Prewitt filter, and a Roberts filter) on the scanning window region image to generate an edge image. To do. Then, the image feature vector calculation means 42 calculates a feature vector based on the shape from the generated edge image.

Here, examples of the feature vector based on the shape include an edge direction histogram and an edge intensity histogram.
Edge direction histogram: The edge direction is quantized into several types of patterns, and an appearance frequency histogram of each pattern in the scanning window region image is calculated and used as a feature vector.
Edge intensity histogram: The edge intensity is quantized into several types of patterns, and an appearance frequency histogram of each pattern in the scanning window region image is calculated and used as a feature vector.

<Second Example: Feature Vector Based on Color / Texture>
Here, as the feature vector based on the color / texture, for example, a luminance moment, a feature vector by Haar wavelet transform, and a local binary pattern can be cited.
Luminance moment: The average value and variance of the luminance in the scanning window area image are calculated and used as a feature vector.
Haar wavelet: A two-dimensional Haar wavelet transform is applied to the scanning window region image in several stages to calculate the variance in each subband to obtain a feature vector.

Local binary pattern: The scanning window area image is divided into blocks, a local binary pattern is calculated for each pixel in the block, a histogram of the local binary pattern is generated for each block, and the histograms are connected to form a feature vector.
Details of local binary patterns are described in, for example, the documents `` T.Ojala, M. Pietikainen, and T.Maenpaa, Multiresolution gray-scale and rotation invariant texture classification with local binary patterns, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. .24, no. 7, pp. 971-987, 2002.

  The image feature vector calculation means 42 may calculate the feature vector by any of the methods described above. Further, the feature vector may be calculated by combining two or more of the above-described methods.

Hereinafter, the description of the configuration of the video subtitle detection apparatus 1 will be continued.
The subtitle candidate area determination unit 43 determines whether the scanning window area image is a subtitle candidate area based on the feature vector of the scanning window area image calculated by the image feature vector calculation unit 42 for each frame image and low resolution image. Whether or not is determined by machine learning.

  Specifically, the caption candidate area determination unit 43 inputs the feature vector of the scanning window area image calculated by the image feature vector calculation unit 42 to the discriminator, and determines whether or not the scanning window area image is a caption candidate area. Determine whether. After that, the caption candidate area determination unit 43 stores in the storage unit 30 what is determined as a caption candidate area.

  Here, the discriminator uses a feature vector obtained from an image in which a caption area appears (positive example) and a feature vector obtained from an image in which no caption area appears (negative example) as learning data, such as a support vector machine. Prepare in advance by machine learning. For example, in the support vector machine, an identification boundary is set at a position where the Euclidean distance is maximum with reference to a learning data that is closest to another class. That is, in the support vector machine, the identification boundary is set so as to maximize the margin from the class in which the caption area appears to the class in which the caption area does not appear. In the support vector machine, the feature vector of the input scanning window area image is classified into either a class in which a subtitle area appears or a class in which a subtitle area does not appear on the basis of the identification boundary.

As shown in FIG. 3, the caption candidate area determination means 43 performs the caption portion “upper left of the ceasefire XX district damage details” in the frame image of FIG. In the subtitle portion, a large number of rectangular subtitle candidate areas KA are determined.
Note that, in FIG. 3, only a part of the caption candidate areas KA is illustrated for easy understanding of the drawing.

The subtitle detection area determination unit 44 calculates the number of times that the subtitle candidate areas stored in the storage unit 30 overlap each other for each frame image and low-resolution image, and selects a subtitle candidate area where the calculated number of overlaps is equal to or greater than the first threshold. It is determined as a caption detection area.
The first threshold is set to an arbitrary value (for example, 2) by the operator.

  Specifically, the caption detection area determination unit 44 determines whether or not the caption candidate area overlaps with another caption candidate area when the caption candidate area is arranged at the coordinates of the caption candidate area stored in the storage unit 30. Then, for each caption candidate area, the caption detection area determination unit 44 counts the number of times that the caption candidate area is determined to overlap another caption candidate area, and calculates the number of overlaps. Thereafter, the caption detection area determination unit 44 determines that the number of overlapping times for each caption candidate area is equal to or greater than the first threshold as the caption detection area. Here, the subtitle detection area determination unit 44 easily determines whether or not the subtitle candidate areas overlap because all the subtitle candidate areas are the same size as the scanning window area image (that is, the scanning window Wi in FIG. 2). it can.

As shown in FIG. 4, the subtitle detection area determination means 44, for example, the upper left portion of the frame image of FIG. Since a large number of subtitle candidate areas KA overlap in the subtitle portion “ha”, these subtitle portions are determined as subtitle detection regions.
In FIG. 4, the number of times the caption candidate area KA in FIG. 3 overlaps is represented in shades (black and white). The number of overlaps increases as the number of overlaps increases (white), and the number of overlaps decreases as the number of overlaps decreases.

  The labeling unit 45 performs a labeling process on the caption detection area determined by the caption detection area determination unit 44. That is, the labeling means 45 performs a labeling process to check which pixel belongs to which caption detection area. In FIG. 5, pixels belonging to different caption detection areas are represented by different colors such as red, blue, and green by the labeling process.

  The circumscribing area calculation unit 46 calculates a circumscribing rectangle of the caption detection area subjected to the labeling process by the labeling unit 45, and shapes the caption detection area into a rectangular shape. Specifically, the circumscribing area calculation means 46 shapes the caption detection area into a rectangular shape with the upper left point, the upper right point, the lower right point, and the lower left point of the labeled caption detection area subjected to the labeling process as vertices.

  The filtering means 47 calculates the evaluation value of the caption detection area shaped by the circumscribed area calculation means 46 for each frame image and low resolution image, and determines whether or not the calculated evaluation value is equal to or less than the third threshold value. At the same time, the caption detection area determined to be equal to or smaller than the third threshold is deleted from the caption detection area shaped by the circumscribed area calculation means 46.

  Specifically, the filtering unit 47 calculates the size (area) of the caption detection area or the edge strength density of the caption detection area as an evaluation value. Then, the filtering unit 47 deletes the caption detection area in which the evaluation value is determined to be equal to or less than the third threshold as an erroneous detection, and causes the storage unit 30 to store the caption detection area that has not been deleted.

  Here, when the density of the edge strength is used as the evaluation value, the filtering unit 47 performs edge detection processing on the caption detection area to generate an edge image of the caption detection area. Then, the filtering unit 47 calculates the density of the edge strength by dividing the sum of the strengths of the edge components in the edge image of the caption detection region by the area of the caption detection region.

Note that the third threshold is set to an arbitrary value by the operator. For example, the third threshold is 700 square pixels when the size (area) of the caption detection region is used as an evaluation value, and 4 when the density of edge strength is used as the evaluation value.
Further, in the present embodiment, the filtering unit 47 adds the reduction ratio of the original low-resolution image to the caption detection area because the enlargement unit 51 described later expands the caption detection area.

  Therefore, as shown in FIG. 6, the subtitle portion of the upper left portion of the frame image of FIG. It will be determined as the detection area SA.

  The detection result integration unit 50 integrates subtitle detection areas corresponding to N + 1 types of resolutions and outputs the subtitle areas as detection results. Therefore, the detection result integration unit 50 includes an enlargement unit 51, a caption detection area integration unit (character area output unit) 52, and an erroneous detection deletion unit (character area deletion unit) 53.

The enlarging means 51 enlarges the caption detection area stored in the storage means 30 at an enlargement ratio at which the low resolution image corresponding to the caption detection area becomes the same resolution as the frame image. That is, the enlarging means 51 enlarges the caption detection area corresponding to the low resolution image among the caption detection areas stored in the storage means 30. Naturally, the enlarging means 51 does not need to enlarge the caption detection area determined from the frame image.
The enlargement rate is the reciprocal of the reduction rate added to each caption detection area, and can be expressed as α ⁱ .

  Hereinafter, with reference to FIG. 7, the enlargement of the subtitle detection area will be described by focusing on the subtitle portion of the lower left portion of the frame image of FIG. In FIG. 7, illustrations other than the subtitle portion are omitted for simplicity of explanation.

As shown in FIG. 7A, the frame image _Pl includes a caption of the original size. On the other hand, since the low resolution image Ps is reduced by the resolution conversion means 20, the size of the caption is reduced. In this case, as shown in FIG. 7 (b), the caption detection unit 40, the frame image P _l determines the caption detection area SA _l large size from the caption detection area of small size from the low-resolution image P _s SA _s Will be judged. However, in this state, since the subtitle detection areas SA ₁ and SA _s have different reduction rates, it cannot be accurately determined whether or not they overlap.

Therefore, as shown in FIG. 7C, the enlarging means 51 multiplies the coordinates PT _s (x, y) of the caption detection area SA _{s in} FIG. The coordinates PT _s ′ (x ′, y ′) of SA _s ′ are calculated. Furthermore, the enlarging means 51 multiplies the lengths of the vertical and horizontal sides of the caption detection area SA _s by the enlargement ratio to calculate the lengths of the vertical and horizontal sides of the enlarged caption detection area SA _s ′. In this way, the enlarging means 51 enlarges the caption detection area SA _s so that the low-resolution image P _s has the same size as the frame image P _l .

Note that the coordinates PT _l , PT _s , and PT _s ′ of the caption detection areas SA _l , SA _s , and SA _s ′ are the reference points set in the caption detection area with the upper left corner of the image as the origin PT ₀ (0, 0). (For example, the upper left point of the caption detection area). X and x ′ are horizontal coordinates, and y and y ′ are vertical coordinates.

Hereinafter, returning to FIG. 1, the description of the configuration of the video subtitle detection apparatus 1 will be continued.
The caption detection area integration unit 52 determines whether one or more of the caption detection area for each low resolution image enlarged by the enlargement unit 51 and the caption detection area of the frame image overlap, and the caption detection area is The subtitle detection areas are integrated according to the overlapping ratio.

In order to simplify the explanation, as shown in FIG. 8, the caption detection area (that is, the caption detection area corresponding to the low-resolution image) SA _s ′ enlarged by the enlargement unit 51 and the caption detection area corresponding to the frame image An example in which _SAL is integrated will be described.

Caption detection region integration means 52, the caption detection area _SA l, 'coordinates _PT l of, PT _s' SA _s in the caption detection area _SA l, 'when disposed, caption detection area _SA l, _{SA s' SA s} is It is determined whether or not they overlap. Then, when the caption detection areas SA _l and SA _s ′ overlap, the caption detection area integration unit 52 of the total area of the caption detection area SA _s ′ is the area of the portion overlapping the reference caption detection area SA _l (FIG. 8). (Hatched part) is calculated as the overlapping ratio. Further, the caption detection area integration unit 52 determines whether or not the overlapping ratio is equal to or greater than the second threshold value.

Of the caption detection areas that overlap each other, the caption detection area corresponding to the frame image or the low-resolution image with the maximum resolution is referred to as a reference caption detection area (reference character detection area). In the example of FIG. 8, since the caption detection area SA _l corresponds to the frame image, a reference character detection region. In FIG. 8, among the reduced low-resolution images, the subtitle detection area SA _l corresponds to the low-resolution image having the maximum resolution, and the subtitle detection area SA _s ′ corresponds to the low-resolution image having a lower resolution. The caption detection area _SAL becomes the reference character detection area.
The second threshold value is set to an arbitrary value (for example, 80%) by the operator.

Specifically, when the calculated percentage is equal to or greater than the second threshold value (see FIG. 8 (a)), the caption detection region integration means 52 outputs only the reference character detection area SA _l to detect deletion unit 53 erroneously as subtitle region To do. In other words, the caption detection area integration means 52 integrates the caption detection areas SA _l and SA _s ′ so as to delete the caption detection area SA _s ′ from the detection result.

When the calculated ratio is less than the second threshold (see FIG. 8B), the caption detection area integration unit 52 outputs the caption detection areas SA ₁ and SA _s ′ as caption areas to the erroneous detection deletion unit 53. . Other words, the subtitle detection region integration means 52 'so that the maximum area surrounded by both is the one subtitle area, subtitle detection area _SA l, SA _s' caption detection area _SA l, SA _s integrate.

When the caption detection areas SA ₁ and SA _s ′ do not overlap (see FIG. 8C), the caption detection area integration unit 52 erroneously deletes the caption detection areas SA _l and SA _s ′ as separate caption areas. Output to means 53.
In FIG. 8, two subtitle detection areas are described as an example, but the subtitle detection area integration unit 52 determines whether or not all the subtitle detection areas overlap each other.

Hereinafter, returning to FIG. 1, the description of the configuration of the video subtitle detection apparatus 1 will be continued.
The erroneous detection deletion means 53 determines whether or not the caption area for each frame image output by the caption detection area integration means 52 does not exist in other frame images within the range of the preceding and following M frames from the frame image. The subtitle area determined not to exist in the other frame images is deleted from the subtitle area for each frame image output by the subtitle detection area integration means 52.
The frame range M is set to an arbitrary value (for example, 30 frames) by the operator.

  In normal video, it seems that subtitles are not displayed for only one frame. For this reason, the erroneous detection deletion unit 53 deletes a caption area that is considered to be highly likely to be erroneously detected in consideration of temporal continuity between frame images.

Specifically, the erroneous detection deletion unit 53 determines whether or not the caption area of a certain frame image overlaps with any of the caption areas of other frame images located within the range of the preceding and following M frames of the frame image. . Then, when the subtitle area of the frame image does not overlap any of the subtitle areas of other frame images, the erroneous detection deletion unit 53 deletes the subtitle area of the frame image. On the other hand, when the subtitle area of the frame image overlaps with any of the subtitle areas of other frame images, the erroneous detection deletion unit 53 does not delete the subtitle area of the frame image. Further, the erroneous detection deletion means 53 outputs all the subtitle areas in the video as the detection result after processing all the subtitle areas.
In the present embodiment, the erroneous detection deletion means 53 adds the time code of the original frame image to the caption area in order to specify the time when the caption appears in the video.

[Overall operation of video caption detection device]
Hereinafter, the overall operation of the video caption detection device 1 will be described with reference to FIG. 9 (see FIG. 1 as appropriate).

  The video subtitle detection apparatus 1 uses the video decoding means 10 to decode the input video and determines whether or not the output of all frame images of the decoded video has been completed (step S1). Here, when the output of all the frame images is not completed (No in step S1), the video subtitle detection apparatus 1 outputs the next frame image to the resolution conversion unit 20 by the video decoding unit 10 (step S2). ).

  The video subtitle detection apparatus 1 sets the counter i to 0 by the resolution conversion means 20 (step S3). Then, the video subtitle detection apparatus 1 determines whether or not the counter i is less than the number N of low-resolution images by the resolution conversion unit 20 (step S4).

When the counter i is less than the number N of low-resolution images (Yes in step S4), the video subtitle detection apparatus 1 reduces the frame image by the reduction rate α− ⁱ by the resolution conversion unit 20 (step S5).

The video subtitle detection apparatus 1 detects a subtitle detection area by performing subtitle detection area detection processing on the frame image and the low-resolution image by the subtitle detection means 40 (step S6). Details of the caption detection area detection process will be described later.
The video caption detection device 1 increments the counter i by the resolution conversion means 20 (step S7), and returns to the process of step S4.

  If the counter i is not less than the number N of low resolution images (No in step S4), the video subtitle detection apparatus 1 uses the enlargement unit 51 to set a subtitle detection area corresponding to the low resolution image, and the low resolution image is a frame image. The image is enlarged at an enlargement rate that results in the same resolution (step S8).

  The video subtitle detection apparatus 1 determines whether or not one or more of the subtitle detection area for each low resolution image and the subtitle detection area of the frame image overlap each other by the subtitle detection area integration unit 52. The subtitle detection areas are integrated according to the overlapping ratio. Here, when the subtitle detection areas do not overlap each other, or when the overlapping ratio of the subtitle detection areas is less than the second threshold, the video subtitle detection apparatus 1 uses the subtitle detection area integration unit 52 to subtitle each subtitle detection area. Output as a region. On the other hand, when the overlapping ratio of the caption detection areas is equal to or greater than the second threshold value, the video caption detection apparatus 1 outputs only the reference character detection area as the caption area by the caption detection area integration unit 52 (step S9).

In the video caption detection device 1, the erroneous detection deletion unit 53 deletes a caption area that is highly likely to be erroneously detected based on temporal continuity. That is, the video subtitle detection apparatus 1 determines whether or not the subtitle area of a certain frame image exists in other frame images within the range of the preceding and following M frames from the frame image by the erroneous detection deletion unit 53. If the subtitle area of another frame image does not overlap at all, the video subtitle detection apparatus 1 deletes the subtitle area of the frame image by the erroneous detection deletion means 53 (step S10). Thereafter, the video subtitle detection apparatus 1 returns to the process of step S1.
When the output of all the frame images is completed (Yes in step S1), the video subtitle detection device 1 ends the entire operation.

[Subtitle detection area detection processing]
Hereinafter, the caption detection area detection process (step S6 in FIG. 9) will be described with reference to FIG. 10 (see FIG. 1 as appropriate).

  The caption detection unit 40 determines whether or not the scanning unit 41 has finished scanning the entire frame image or low-resolution image (step S61). Here, when the scanning of the entire frame image or low-resolution image is not completed (No in step S61), the caption detection unit 40 moves the scanning window slightly by the scanning unit 41 (step S62).

  The caption detection unit 40 calculates the feature vector based on the shape or the feature vector based on the color / texture from the scanning window region image by the image feature vector calculation unit 42 (step S63).

  The caption detection means 40 determines the caption candidate area by the classifier by the caption candidate area determination means 43. That is, the caption detection means 40 inputs the feature vector of the scanning window area image to the discriminator by the caption candidate area determination means 43, and determines whether or not this scanning window area image is a caption candidate area (step S64). ). Thereafter, the caption detection unit 40 returns to the process of step S61.

  When scanning of the entire frame image or low-resolution image is completed (Yes in step S61), the caption detection unit 40 determines the caption detection region by the caption detection region determination unit 44. That is, the subtitle detection means 40 calculates the number of times that the subtitle candidate areas overlap each other by the subtitle detection area determination means 44, and determines the subtitle candidate area where the number of overlaps is equal to or greater than the first threshold as the subtitle detection area (step S65). .

  The caption detection means 40 labels the caption detection area by the labeling means 45 (step S66). Then, the caption detection means 40 calculates the circumscribed rectangle of the caption detection area by the circumscribed area calculation means 46 (step S67).

  The caption detection means 40 calculates the evaluation value of the caption detection area by the filtering means 47 and performs filtering of the caption detection area based on the evaluation value. That is, the subtitle detection means 40 calculates the size of the subtitle detection area or the density of the edge strength of the subtitle detection area as the evaluation value by the filtering means 47, and the subtitle for which the evaluation value is determined to be equal to or less than the third threshold value. The detection area is deleted as a false detection (step S68).

  As described above, the video subtitle detection apparatus 1 according to the embodiment of the present invention can detect a subtitle area regardless of the subtitle appearance position because the entire frame image and the low resolution image are to be scanned. In addition, since the video subtitle detection apparatus 1 scans the frame image and each low-resolution image with the scanning window having the same size, the subtitles of the frame image appear in the scanning window area image in various sizes. Regardless of the subtitle area, the subtitle area can be detected. Thus, since it does not depend on empirical knowledge of edge crowding, the video subtitle detection apparatus 1 can accurately detect a subtitle area even when the subtitle area appears on a complicated background. For example, the video subtitle detection apparatus 1 can accurately detect a scroll super or a subtitle that moves irregularly compared to a conventional subtitle detection technique.

  Moreover, since the video subtitle detection apparatus 1 deletes the subtitle detection area that is highly likely to be erroneously detected by the filtering unit 47, the detection accuracy of the subtitle area can be further increased. Furthermore, since the video caption detection device 1 deletes a caption area that is highly likely to be erroneously detected by the erroneous detection deletion unit 53, the detection accuracy of the caption area can be further increased.

  Therefore, the video subtitle detection apparatus 1 applies keyword recognition to the detected subtitle area, thereby adding a keyword detection technique for searching a scene with a recording device such as a hard disk recorder, or an analysis for analyzing video content. It can be used as information generation technology. Specifically, the video caption detection device 1 can be expected to be used for digest generation of broadcast programs, extraction of representative frames, similar program search, and related information search.

Here, in the conventional caption detection technique, if an opportunity learning is performed as learning data on images in which captions of various sizes appear, it is considered that the detection accuracy of the caption area is improved regardless of the caption size. However, in this case, the effort for collecting the learning data is enormous, which is not realistic.
On the other hand, the video subtitle detection apparatus 1 determines by the discriminator after converting the subtitles of the frame image into various sizes, so that it is not necessary to machine-learn images in which subtitles of various sizes appear, and the number of machine learning times is reduced. Can be reduced.

In the present embodiment, the circumscribed area calculating unit 46 has been described. However, the present invention is not limited to this. For example, when the subtitle area does not have to be rectangular, the video subtitle detection apparatus 1 does not need to include the circumscribed area calculation unit 46.
In addition, the video caption detection device 1 may have a caption area other than a rectangle.

  In the present embodiment, the video subtitle detection apparatus 1 has been described as detecting video subtitles, but the present invention is not limited to this. For example, the character detection device according to the present invention can detect a character of a background portion (for example, a signboard character) reflected in the still image from a still image captured by a camera.

  In the present embodiment, the caption candidate area determination unit 43 is described as using a support vector machine, but the present invention is not limited to this. For example, the character detection apparatus according to the present invention can use a neural network as machine learning.

  In the present embodiment, the video caption detection device according to the present invention has been described as an independent device, but the present invention uses hardware resources such as a CPU, a memory, and a hard disk of a general computer as the above-described units. It can also be realized by a video subtitle detection program for cooperative operation. This video subtitle detection program may be distributed via a communication line, or may be distributed by writing in a recording medium such as a CD-ROM or a flash memory.

  The character detection device according to the present invention was mounted on a general computer, and a subtitle area detection experiment was performed on an experimental video, and the experimental result was evaluated.

In this experiment, an apparatus (hereinafter referred to as an image caption detection device 1B) in which the image caption detection device 1 of FIG. 1 does not include the erroneous detection deletion unit 53 and does not perform time direction filtering is prepared. In the video subtitle detection apparatus 1B, the number N of low resolution images is set to 1, and only the frame images are processed without generating the low resolution images. In the video caption detection device 1B, the feature vector is calculated by combining the edge intensity histogram, the luminance moment, the Haar wavelet, and the local binary pattern. At this time, a Sobel filter was used for the edge detection process. Furthermore, the video subtitle detection apparatus 1B collects 2500 frame images of positive examples and 2500 frames of negative examples as learning data from another broadcast program, and machine-learns the learning data using a support vector machine.
In addition, what abbreviate | omitted description, such as a 1st threshold value-a 3rd threshold value, is a value similar to above-described embodiment.

  Then, a 30-minute news program was input as an experimental video to the video subtitle detection apparatus 1B. In this experimental video, the resolution is 432 pixels wide × 240 pixels vertical, and 236 types of subtitles have appeared.

As an experimental result, the recall and the precision represented by the following detection result evaluation formula were calculated. Here, Nb is the total number of subtitle areas that can be correctly detected from the experimental video, Nc is the total number of subtitle areas that appear in the experimental video, and No is the total number of subtitle areas detected from the experimental video. . In other words, the recall rate is a value that represents a small number of detection omissions, and the precision rate is a value that represents a small number of false detections.
Reproducibility = Nb / Nc
Compliance rate = Nb / No ... Detection result evaluation formula

In addition, as a comparison target, the subtitle area was detected from the same experimental video using the technique described in Non-Patent Document 1, and the recall rate and the matching rate were calculated using the same detection result evaluation formula.
In the following table, the experimental results of the video caption detection device 1B are shown as examples, and the experimental results of the technique described in Non-Patent Document 1 are shown as comparative examples.

  From the above table, the recall rate was about 80% in the examples and comparative examples. Here, special subtitles such as subtitles that move to the experimental video were not included, and neither subtitle area having a small brightness difference from the background could be detected.

  Regarding the precision, the example improved about 15% over the comparative example. In the comparative example, erroneous detection occurred in a region where background edges are dense, as in the images of FIGS. On the other hand, in the example, the caption area could be accurately detected from the images of FIGS. As a result, there was a big difference in the recall of both methods.

  From the above, it has been found that the character detection device according to the present invention greatly improves the detection accuracy of the caption area only by performing the caption detection area detection process on the frame image without using the low resolution image. If the subtitle detection area detection processing is also performed on the low-resolution image, the character detection device according to the present invention is expected to dramatically improve the subtitle area detection accuracy as compared with the conventional character detection technology. The

1,1B Video caption detection device 10 Video decoding means 20 Resolution conversion means 30 Storage means 40 Subtitle detection means 41 Scanning means 42 Image feature vector calculation means 43 Subtitle candidate area determination means (character candidate area determination means)
44 Subtitle detection area determination means (character detection area determination means)
45 Labeling means 46 circumscribed area calculating means 47 filtering means (character detection area deleting means)
50 Detection result integration means 51 Enlargement means 52 Subtitle detection area integration means (character area output means)
53 False detection deletion means (character area deletion means)

Claims (5)

A character region detection device that detects a character region that is a character region included in an input image from the input image,
A resolution conversion means for inputting the input image and converting the input image into one or more low-resolution images having a resolution lower than that of the input image;
The low resolution image converted by the resolution conversion means and the input image are scanned with the scanning window of the same size, respectively, so that the scanning corresponding to the area of the scanning window is performed for each of the input image and the low resolution image. Scanning means for generating a window area image;
Image feature vector calculating means for calculating a feature vector of a scanning window region image generated by the scanning means for each of the input image and the low resolution image;
Based on the feature vector of the scanning window area image calculated by the image feature vector calculation means for each of the input image and the low-resolution image, whether or not the scanning window area image is a character candidate area is determined by machine learning. Character candidate area determination means for determining;
For each of the input image and the low-resolution image, the number of times that the character candidate areas determined by the character candidate area determination unit overlap each other is calculated, and the calculated number of times of overlap is equal to or greater than a preset first threshold value Character detection area determination means for determining an area as a character detection area;
An enlargement means for enlarging the character detection area for each low resolution image determined by the character detection area determination means at an enlargement ratio at which the low resolution image corresponding to the character detection area is the same resolution as the input image;
It is determined whether at least one of the character detection area for each low-resolution image enlarged by the enlargement means and the character detection area of the input image overlaps, and among the character detection areas determined to overlap each other The ratio of overlapping of other character detection areas to the reference character detection area, which is the character detection area corresponding to the input image or the low resolution image having the maximum resolution, is calculated in advance. A character area output means for outputting only the reference character detection area as the character area when the second threshold is exceeded,
A character area detecting device comprising: For each of the input image and the low resolution image, the size of the character detection area or the density of the edge strength determined by the character detection area determination unit is calculated as an evaluation value, and the calculated evaluation value is set in advance. A character detection area deleting unit that determines whether or not the threshold value is 3 or less, and deletes the character detection area determined to be equal to or less than the third threshold value from the character detection area determined by the character detection area determination unit;
The character region detection device according to claim 1, further comprising:   The character area output means further includes, as the character area, a character detection area determined not to overlap with each other and a character detection area determined to overlap with each other and the calculated overlapping ratio being less than the second threshold value. 3. The character area detecting apparatus according to claim 1, wherein the character area detecting apparatus outputs the character area. A video decoding means for decoding the input video and outputting a frame image constituting the video to the resolution conversion means as the input image;
The character area output means outputs the character area for each frame image,
It is determined whether the character area for each frame image output by the character area output means does not exist in other frame images within a preset range from the frame image, and is output by the character area output means. A character area deleting means for deleting a character area determined not to exist in the other frame image from the character area for each frame image that has been
The character region detection device according to claim 1, further comprising: In order to detect a character region that is a character region included in the input image from the input image,
A resolution converting means for inputting the input image and converting the input image into one or more low-resolution images having a resolution lower than that of the input image;
The low resolution image converted by the resolution conversion means and the input image are scanned with the scanning window of the same size, respectively, so that the scanning corresponding to the area of the scanning window is performed for each of the input image and the low resolution image. Scanning means for generating a window area image;
Image feature vector calculating means for calculating a feature vector of a scanning window area image generated by the scanning means for each of the input image and the low resolution image;
Based on the feature vector of the scanning window area image calculated by the image feature vector calculation means for each of the input image and the low-resolution image, whether or not the scanning window area image is a character candidate area is determined by machine learning. Character candidate area determination means for determining,
For each of the input image and the low-resolution image, the number of times that the character candidate areas determined by the character candidate area determination unit overlap each other is calculated, and the calculated number of times of overlap is equal to or greater than a preset first threshold value Character detection area determination means for determining an area as a character detection area;
Enlarging means for enlarging the character detection area for each low-resolution image determined by the character detection area determining means at an enlargement ratio at which the low-resolution image corresponding to the character detection area becomes the same resolution as the input image;
It is determined whether at least one of the character detection area for each low-resolution image enlarged by the enlargement means and the character detection area of the input image overlaps, and among the character detection areas determined to overlap each other The ratio of overlapping of other character detection areas to the reference character detection area, which is the character detection area corresponding to the input image or the low resolution image having the maximum resolution, is calculated in advance. A character area output means for outputting only the reference character detection area as the character area when the second threshold is exceeded,
Character area detection program to function as