JP3503797B2 - Video telop detection method and apparatus - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for detecting telops in video.

[0002]

2. Description of the Related Art A conventional device for detecting a telop from an image detects a telop using a local feature of one to several frame images.

For example, by utilizing the fact that there is a large brightness difference around the telop, first, the change in the distribution of brightness and hue between frames is examined to find the frame in which the telop appears, and then the frames before and after the telop appears. Take the difference between
There is a method of extracting a telop (for example, Nemoto et al., “Searching for Material Video by Recognition of Telop”, 1994 IEICE Spring Conference D-42).
7, 1994).

Further, targeting one frame image, the telop has a higher brightness than the background and the edge is easily extracted, and the edge extraction is performed on the image by the primary differential to extract the edge image. There is also a method of detecting telops by projecting in the vertical and horizontal directions (for example, Mogi et al., "Indexing Articles Based on Character Recognition in News Video", IEICE Technical Research Report IE95-153, 199).
6 etc.).

Further, by using the property that the telop is stationary and having high luminance, a portion having no movement between two frames is obtained, and further, an area having a luminance equal to or higher than a predetermined value is detected as a subtitle portion. There is also a device (for example, Japanese Unexamined Patent Application Publication No. Hei 8
-331456 "Subtitle moving device").

Further, in a video image coded using correlation between frames such as MPEG, correlation between frames is used,
Moreover, there is a property that pixels coded without using motion compensation are temporally and spatially concentrated in the telop portion.
Using this property, there is also a device that detects a telop by counting the frequency of appearance of coded pixels using correlation between frames and without using motion compensation (Sato et al., " Method of extracting telop area from MPEG image "1996 Information and Systems Society Conference of the Institute of Electronics, Information and Communication Engineers D-273).

[0007]

In the above-mentioned prior art, since the telop is detected using the temporally local information of one or two frame images, the telop is stationary and the brightness is high. However, if there is a subject other than the telop having similar characteristics to the telop, such as a large high-frequency component, there is a problem in that it is erroneously detected as the telop.

On the other hand, if the telop appearing on the screen for a long time temporarily moves or the contour is blurred due to the influence of image quality deterioration, noise, etc., that portion will be missed. For this reason, originally one continuous telop is redundantly detected as separate telops over a plurality of time intervals.

That is, in the prior art, since it is determined whether a telop exists for a certain short section, it is difficult to avoid excessive detection of a subject other than the telop and omission of detection of the telop due to noise. Therefore, when the conventional technique is used for obtaining a list of telops from an image, a subject other than the telops may be erroneously displayed or one telop may be displayed in duplicate.

An object of the present invention is to provide a video telop detection method and apparatus for detecting telops from video without excess or deficiency.

[0011]

In order to achieve the above object, a video telop detection method extracts a telop candidate pixel in the unit of a pixel or a set of pixels, and has a three-dimensional structure including vertical and horizontal spatial axes and a temporal axis. Extraction stage to store in the buffer of
There is a merging step of merging telop candidate pixels on the buffer.

Further, the video telop detection device of the present invention is
A three-dimensional buffer consisting of vertical and horizontal space axes and time axis,
It has extraction means for extracting telop candidate pixels in units of pixels or a set of pixels and storing them in a buffer, and merging means for merging telop candidate pixels on the buffer.

By extracting telop candidate pixels from a video in units of pixels or a set of pixels and storing the telop candidate pixels in a three-dimensional buffer consisting of vertical and horizontal spatial axes and a temporal axis, a video for a longer time than the prior art can be displayed. It becomes possible to process.
Further, by merging the telop candidate pixels on the buffer, it is possible to ignore the minute change for a short time and remove the influence of noise.

According to an embodiment of the present invention, the extraction means is
It has an edge generating means for obtaining an edge of the image, a projecting means for projecting the edge value in the vertical and horizontal directions, and a comparing means for judging the telop candidate pixel based on the comparison result of the projected value and the threshold value.

In the extraction means, the edge generation means obtains the edges of the image, so that an image in which the edges are concentrated around the telop can be obtained based on the feature that the high frequency component of the telop is large. Next, the degree of concentration of edges can be evaluated one-dimensionally by the projection means that projects the edge values in the vertical and horizontal directions. The comparison means can detect a portion where the edges are concentrated by comparing the projected value with the threshold value. Thereby, the telop candidate pixel can be obtained.

According to an embodiment of the present invention, the extraction means is
From the video data coded using the correlation between frames, the number of pixels coded using the correlation between frames and without motion compensation is calculated for each pixel position. It has a counting means for counting inside.

The counting means determines the number of pixels encoded from the video data encoded by utilizing the correlation between the frames, using the correlation between the frames and without using motion compensation. Each position of is counted in a certain counting section. For the telop, use correlation between frames, and
Since there is a tendency that the coded pixels are concentrated without using motion compensation, a large count value is obtained only for the pixels of the telop. As a result, a telop candidate pixel having a larger value can be obtained as the likelihood of the telop is higher.

According to an embodiment of the invention, the merging means comprises
It has smoothing means for smoothing the telop candidate pixels by a three-dimensional smoothing filter.

In the merging means, the smoothing means smoothes the telop candidate pixels by the three-dimensional smoothing filter, so that the adjacent telop candidate pixels are merged and the small isolated telop candidate pixels disappear.

According to an embodiment of the invention, the merging means comprises
It has expansion means for replacing the telop candidate pixel with the maximum value of the neighboring pixels and contraction means for replacing the telop candidate pixel with the minimum value of the neighboring pixels.

In the merging means, the expanding means replaces the telop candidate pixel with the maximum value of the neighboring pixels,
The adjacent telop candidate pixels are merged. The contracting means replaces the telop candidate with the minimum value of the neighboring pixels, so that the small isolated pixel disappears. By the combination of these two means, adjacent telop candidate pixels are merged and small isolated telop candidate pixels disappear.

According to the embodiment of the present invention, the video telop detection device further includes a determination unit that sets a frame before or after a time zone in which no telop candidate pixel exists as a representative frame including a telop.

The determining means can obtain the representative frame representing the telop by setting the frame before or after the time section in which the telop candidate pixel does not exist as the representative frame representing the telop.

According to an embodiment of the present invention, the video telop detection apparatus labels the connected component of the merged telop candidate pixels with a labeling means, and displays a frame containing the labeled telop candidate pixels with a telop. It further has a determination means for determining the frame to be included.

Since the labeling means gives a label to the connected component of the merged telop candidate pixels, it is possible to identify each telop. Since the determination unit sets the frame including the labeled telop candidate pixels as the representative frame representing the telop, it is possible to obtain the representative frame for each telop without excess or deficiency.

[0026]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Next, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing a video telop detection apparatus according to the first embodiment of the present invention.

The video telop detection apparatus of this embodiment detects a pixel or a set of pixels that are candidates for a telop area from a buffer 103 and a video input from the input terminal 101, and extracts telop candidate pixels to be accumulated in the buffer 103. The unit 102 and a merging unit 104 that merges the telop candidates accumulated in the buffer 103 and outputs the merged telop candidates to the output terminal 105.

8 × 8 to 16 × 16 as a set of pixels
Blocks can be used. Buffer 103 is 3
It is a dimension and is represented by an axis t perpendicular to two axes x and y parallel to the screen as shown in FIG. For example, when a 16 × 16 block is used on a screen of 720 × 480 pixels, the width W of the buffer 103 is 45 and the height H is 30. The depth T of the buffer 103 is a value obtained by dividing the time of the target video by the time resolution. For example, when a 10-minute video is processed at 0.5 second intervals, T becomes 1200.

FIG. 3 is a block diagram showing the telop candidate pixel extraction unit 102. The edge generator 202 obtains an edge image from the video input to the input terminal 201 and stores it in the buffer 303. As a method to find the edge,
An image processing operator such as Laplacian or Robert can be used. Next, the vertical projection unit 204 projects the image in the vertical direction to determine the frequency. Then, as shown by v in FIG. 4, the frequency of the portion where the edges are concentrated increases, so this is compared with the threshold value input to the input terminal 206 by the comparison unit 205, and the range (x0 to x1) equal to or more than the threshold value ). Limiting the range to (x0 to x1), the horizontal projection unit 207 further calculates the projection frequency of the edge image in the horizontal direction. The comparison unit 208 obtains the frequency of lateral projection of the edge image. The comparison unit 208 compares this with a threshold value given to the input terminal 209 to obtain a range (y0 to y1) equal to or more than the threshold value. The synthesizing unit 210 outputs to the output terminal 211 as a telop candidate pixel in which the pixel value of the portion of the range (x1 to x1, y0 to y1) obtained as described above is 1, and the other values are 0.

When there are a plurality of ranges equal to or greater than the threshold value in the vertical projection, the horizontal projection is performed for each range. Further, the order of the vertical projection unit 204 and the horizontal projection unit 207 may be exchanged.

FIG. 5 is a block diagram showing another example of the telop candidate pixel extraction unit 102. An example of this is MPEG, etc.
The target is video data encoded by utilizing the correlation between frames. The position of the pixel of the encoded video data input to the input terminal 301 is decoded by the position decoding unit 302, and the decoded video data is output to the address (A) of the counter 304. Similarly, the type decoding unit 303 decodes the type of pixel coding. The type decoding unit 303 outputs "1" only when the type of pixel is coded using inter-frame correlation and without using motion compensation, and otherwise outputs "0". Is output. This signal controls the increase / decrease of the counter 304. The value of the counter 304 is increased / decreased within the counting time and directly output to the output terminal 305 as a telop candidate pixel. After output, counter 30
All four values are reset to zero.

Next, the merging unit 104 will be described.

First, the merging unit 104 using the three-dimensional smoothing filter will be described. As a three-dimensional smoothing filter, consider the following three-dimensional Gaussian filter.

[0035]

[Equation 1] The buffer 103 (B (x, y, t)) is subjected to convolutional integration.

[0036]

[Equation 2] Alternatively, a one-dimensional Gaussian filter

[0037]

[Equation 3] May be convolved in order with respect to the three axial directions of x, y, and t. That is,

[0038]

[Equation 4] And

Next, the merging unit 104 using the expansion processing and the contraction processing will be described. Expansion processing is performed on a certain pixel B
The maximum value of the points included in the neighborhood R (x, y, t) of (x, y, t) is set as the value of the pixel. That is, the pixel B (x,
y, z) is the value of the pixel Be (x, y, z) according to the following equation.

[0040]

[Equation 5] The expansion process serves to fill holes or gaps whose width, height, or depth is smaller than R. For example, when expansion processing is performed on four neighborhoods (four pixels in contact with the target pixel in the vertical and horizontal directions) in FIG. 6A, the result is as shown in FIG. 6B. The gap between the two black areas disappears and the white holes in the black areas are also filled. The contraction processing is performed in the vicinity R of a certain pixel B (x, y, t)
The minimum value of the points included in (x, y, t) is the value of the pixel. That is, the pixel B (x, y, z) becomes the value of the pixel Be (x, y, z) according to the following formula.

[0041]

[Equation 6] The contraction process works to erase an area whose width, height, or depth is smaller than R. For example, when the contraction process of 4 neighborhoods is performed with respect to FIG. 6A, the result is as shown in FIG. 6C. It can be seen that the black area of height 2 disappears. When contraction processing is performed on FIG. 6 (2), which has been subjected to expansion processing, the result is as shown in FIG. 6 (4). 6 (1) and FIG.
Comparing (4), it can be seen that holes and gaps are eliminated while maintaining the size. That is, the expansion / contraction process has a function of compensating for the lack of pixels such as holes and gaps. When the expansion process is performed on the result of the contraction process shown in FIG. 6C, the result is as shown in FIG. 6 (1) and FIG.
Comparing (5), it can be seen that the small region disappears while maintaining the size. That is, the contraction / expansion process has a function of removing noise.

In another example of the merging unit 104, a configuration as shown in FIG. 7 is adopted so that the order of expansion / contraction processing and expansion / expansion can be changed. That is, the telop candidate pixel input to the input terminal 401 is processed by the expansion units 402 and 405 and the contraction units 403 and 404 via the four interlocking switches 406, and is output to the output terminal 407. Switch 4
When the contact piece of 06 is on the upper side, the expansion / contraction process is performed first, and then the contraction / expansion process is performed. When the contact piece of the switch 406 is on the lower side, the order is reversed.

If expansion / contraction processing is performed first, priority is given to supplementing the defect, and if contraction / expansion processing is performed first, noise removal is prioritized.

FIG. 8 is a block diagram of an embodiment in which a judging unit is added to the first embodiment of FIG. From the video input to the input terminal 101, the telop candidate pixel extraction unit 102 detects a pixel or a set of pixels that are candidates for the telop area, accumulates it in the buffer 103, and the merging unit 104 merges the telop candidate pixels. With respect to the merged telop candidate pixels, a representative frame representing the telop is determined by the determination unit 106 and output to the output terminal 105.

Next, two examples of the judging section 106 will be described. First, an example will be described in which a frame before or after a time section in which no telop candidate pixel exists is determined as a representative frame representing a telop.

For example, it is assumed that telops A to G are temporally arranged as shown in FIG. In this figure, the horizontal axis is the time axis (t axis) and the vertical axis is the x or y axis. The frames after the time section in which the telop candidate pixel does not exist are b _{1 to} b ₄ . Further, f _{1 to} f ₄ indicate the previous frame of the time section in which the telop candidate pixel does not exist.

When b _{1 to} b ₄ are representative frames representing telops, telops A, B, D, F and G are reflected,
The telops appearing while other telops are appearing, such as telops C and E, are not reflected. On the other hand, f ₁ ~
When f ₄ is used, telops A, B, C, D, and F are reflected, but telops such as telops E and G that disappear while other telops are appearing are not reflected. This method has an advantage of simplicity because the detection of the section in which the telop candidate pixel does not exist can be realized relatively easily.

Next, an example of the determination unit 106 using labeling will be described.

FIG. 10 shows a determination unit 10 using labeling.
6 is a block diagram of FIG. With respect to the telop candidate pixel input to the input terminal 501, the labeling unit 502 obtains a connected component with a neighboring pixel, and the connected component is stored in the buffer 503 as label information. Label information is managed in a table format as shown in FIG. Here, FIG.
As shown in (2), the position of the label is represented by the coordinate value of the circumscribing rectangular parallelepiped. The determination unit 504 determines that t ₀ ≦ t ≦
Select t in the range of t ₁ and output terminal 5 as the representative frame.
Output to 05.

As an example, FIG. 12 shows a telop time arrangement similar to that in FIG. According to this embodiment, each telop A
~ G can be identified and its time range can be determined. Here, the frame (t ₀ ) in which the telop appears is used as the representative frame, and the times t ₁ to t ₆ are output.

As the representative frame, a frame (t ₁ ) immediately before the telop disappears may be used, or a frame between t ₀ and t ₁ may be used.

The present invention is within the scope of the invention.
Various modifications can be implemented. For example, a representative frame can be displayed using the telop detection result to create a telop list of images.

[0053]

As described above, according to the present invention,
Erroneous detection due to the appearance of a subject having a similar telop for a short time is removed, and temporary telop detection omission due to the influence of image quality deterioration, noise, etc. is compensated, so that it is possible to detect telops without excess or deficiency.

[Brief description of drawings]

FIG. 1 is a block diagram of a video telop extraction device according to an embodiment of the present invention.

FIG. 2 is an explanatory diagram showing a three-dimensional buffer 103.

FIG. 3 is a block diagram of an example of a telop candidate pixel extraction unit 102.

FIG. 4 is an explanatory diagram showing the principle of telop detection.

FIG. 5 is a block diagram of another example of the telop candidate pixel extraction unit 102.

FIG. 6 is an exemplary diagram showing expansion processing and contraction processing of telop candidate pixels.

FIG. 7 is a block diagram of an example of a telop candidate pixel merging unit 104.

FIG. 8 is a block diagram of a video telop extraction device according to another embodiment of the present invention.

FIG. 9 is an exemplary unit showing a determination result according to an embodiment of determining before and after a time period in which a telop candidate pixel does not exist.

FIG. 10 is a block diagram showing an example of a determination unit 106 using labeling.

FIG. 11 is an exemplary diagram showing label information.

FIG. 12 is an exemplary diagram showing a determination result by an example of a determination unit 106 using labeling.

[Explanation of symbols]

101 input terminal 102 telop pixel candidate extraction unit 103 buffer 104 Merger 105 output terminals 106 determination unit 201 input terminal 202 Edge generator 203 buffer 204 Vertical projection unit 205 comparison section 206 input terminal 207 Horizontal projection unit 208 Comparison section 209 input terminal 210 Synthesis Department 211 Output terminal 301 input terminal 302 position decoding unit 303 types decoding unit 304 counter 305 output terminal 401 input terminal 402,405 Inflating section 403,404 contraction section 406 switch 407 output terminal 501 input terminal 502 Labeling section 503 buffer 504 Judgment unit 505 output terminal

─────────────────────────────────────────────────── ─── Continuation of the front page (56) Reference JP-A-9-18798 (JP, A) JP-A-8-317301 (JP, A) JP-A-8-212231 (JP, A) JP-A-7- 192003 (JP, A) (58) Fields investigated (Int.Cl. ⁷ , DB name) H04N 5/262-5/28 G06T 7/00-7/60 G06K 9/00 G06K 9/46-9/52 G06K 9/62-9/82

Claims (8)

(57) [Claims] 1. A method of detecting a telop from an image, comprising extracting telop candidate pixels in units of pixels or a set of pixels and storing the telop candidate pixels in a three-dimensional buffer having vertical and horizontal spatial axes and a temporal axis. And a merging step of merging telop candidate pixels on the buffer. 2. A device for detecting a telop from an image, comprising: a three-dimensional buffer having vertical and horizontal spatial axes and a temporal axis; extracting telop candidate pixels in the unit of a pixel or a set of pixels; An image telop detection device comprising: an extracting unit that stores the telop candidate pixels on the buffer; 3. The telop based on the comparison result between the projected value and the threshold, the extraction means obtaining the edge of the image, the projection means for projecting the edge value in the vertical and horizontal directions. The video telop detection device according to claim 2, further comprising a comparison unit that determines a candidate pixel. 4. The number of pixels encoded from the video data encoded by utilizing the correlation between frames using the correlation between frames and without using motion compensation, 3. The video telop detection device according to claim 2, further comprising a counting unit that counts each position of each pixel within a counting section. 5. The video telop detection device according to claim 2, wherein the merging unit has a smoothing unit that smoothes the telop candidate pixels with a three-dimensional smoothing filter. 6. The merging means has expansion means for replacing the telop candidate pixel with the maximum value of the neighboring pixels and contraction means for replacing the telop candidate pixel with the minimum value of the neighboring pixels. The video telop detection device according to. 7. The video telop detection device according to claim 2, further comprising a determination unit that determines a frame before or after a time section in which no telop candidate pixel is present as a representative frame representing a telop. . 8. The method according to claim 2, further comprising labeling means for giving a label to the connected components of the merged telop candidate pixels, and determining means for making a frame including the labeled telop candidate pixels a representative frame representing the telop. 7. The video telop detection device according to any one of 1 to 6.