KR19990047501A - How to extract and recognize news video subtitles - Google Patents

Abstract

The present invention relates to a method of automatically extracting and recognizing subtitles from a video image, and conventionally, a simple index retrieval method does not express various meanings of an image and thus does not satisfy a search requirement of various users.

Therefore, when the video input image (contrast value image) of one frame comes in, the present invention extracts a place where there is a sign of a news subtitle from the whole image, and examines whether the extracted candidate region continues in successive frames. Extract the subtitles, perform large-scale segmentation and verification to verify that the candidate subtitles of the extracted subtitles are true subtitles, and perform the extracted subtitle recognition step using the recognition means for the individual character regions obtained in the segmentation process. As a result, news video subtitle extraction and recognition is possible.

Description

How to extract and recognize news video subtitles

The present invention relates to a method for automatically extracting and recognizing a subtitle from a video image. More particularly, the present invention relates to a method of finding a subtitle area position in real time as a video (or image) database technology, which is a field of a multimedia database.

In the conventional video (or image) database technology, an image is stored in a compressed file by attaching a simple index to the image, and information is searched through an index search according to a user's request. This simple index retrieval method does not express the meaning of the image in various ways, and thus does not satisfy the search needs of various users. Therefore, this problem can be solved by extracting the semantic information of the image, storing it as metadata, and retrieving it. The semantic information of the important video in the news video may include caption information, color information, person information, and the like.

Conventional subtitle extraction method is to extract the subtitle character using the region splite and merge method for the whole image, another method is to divide the entire image into a predetermined region and then The portion where the change in contrast value changes rapidly is determined and extracted as the subtitle area. The problem with the conventional method is that processing the entire image takes a lot of processing time and has a disadvantage in reliability.

Extracting semantic information (caption, color, motion, shape, etc.) of images is an important process when constructing a content-based video (news video) database system. In particular, in the news video, the role of the subtitles is important because the subtitles are used to emphasize the contents explained by the anchor or the reporter.

Accordingly, an object of the present invention is to provide a method for extracting subtitles in a large amount of news video in real time by extracting a subtitle region without processing the entire image in order to solve the conventional problems.

1 is a system configuration of the news video subtitle extraction according to the present invention.

2 is a flowchart of caption candidate region extraction according to the present invention;

3 is a flowchart for caption region verification and regionation according to the present invention;

Figure 4 is a large area segmentation extraction flow chart according to the present invention.

5 is a flow chart of large character area verification and individual character area segmentation according to the present invention;

* Description of the symbols for the main parts of the drawings *

100: news video input image 200: subtitle candidate region extraction step

300: subtitle area verification and area step 400: extracted subtitle recognition step

In order to achieve the above object, the present invention determines a final candidate area in consideration of continuity between candidate areas of subtitles extracted from each frame. Verification is performed to determine whether the extracted final candidate region is a true subtitle region. The verification method uses a large-scale segmentation method that clusters the caption characters to segment and verify the word characters of the caption. The large character segmentation method sorts each extracted character element in ascending order based on min-x coordinates. The first component makes the first large area. The second component checks the condition whether it belongs to the formed large area, and if it meets the condition, it belongs to the large area. If there is not many conditions, it creates a new large area. This process is repeated for all components to create a large area of subtitle words. A non-subtitle area may exist among the large areas that satisfy the feature condition of the subtitle. In order to remove such non-large-area regions, the feature conditions of the large-area regions are compared to remove unsuitable large-area regions as subtitle sentences. Since there may be a non-subtitle area in a large area that satisfies a condition, a true subtitle area is extracted through individual area characterization and condition checking.

1 is a flowchart illustrating a method of extracting news video subtitles according to the present invention. As shown in FIG. A candidate region extraction step (200) of extracting a place where there is an indication of existence and determining a final candidate region by checking whether the extracted candidate region continues in a continuous frame; A caption region verification and regionation step (300) of performing large area segmentation and verification to confirm whether the candidate region of the extracted caption is a true caption region; The extracted subtitle recognition step 400 is performed by using the recognition means for the individual character areas obtained in the regionation process.

The news video subtitle extraction and recognition method according to the present invention made as described above proceeds as follows. When a video input image (contrast value image) of one frame comes in (100), a place where there is a sign of a news subtitle is extracted from the entire image, and the extracted candidate region is examined to determine whether the continuity continues in successive frames. To extract (200). In order to check whether the candidate region of the extracted caption is a true caption region, large-area segmentation and verification are performed (300). The caption region verification and regionization are performed through the RGB color information from the news video input image. In operation 400, the extracted subtitles are recognized using the recognition means of the individual character area obtained in the regioning process.

2 illustrates a candidate region extraction method according to the present invention, which uses an existing candidate region extraction method. The extracted candidate region determines whether or not continuity continues in successive frames to determine a final candidate region. As shown, first, candidate regions in consecutive frames are selected by time (a candidate region extracted from a frame at time _t : F _t (x, y) _{1 to n} , n: number of candidate regions extracted at time t) Extract (210). That is, first, candidate regions of time t-1 and time t are extracted, and a final candidate region of time t is determined at time t + 1. For all candidate regions F _t (x, y) _{1 to n} at time t, candidate regions F _t-1 (x, y) _{1 to s} at all times _t-1 or candidate regions at time t + 1 If (F _{t + 1} (x, y) _{1 ~ k} ) coincides with each other (230) and if they match, the corresponding coordinate is determined as the final candidate region (240). If it does not match, the above operation is performed by increasing the candidate areas one by one (250, 220). After all investigations at time t are completed, the candidate area at time t + 1 is changed to the candidate area at time t (260).

In FIG. 3, a step of terminating and verifying whether the extracted candidate region is a true subtitle region is described. Since the caption character feature in the news video input image has various color values (RGB color), color is used for caption character extraction (310). The RGB image is converted into an HSI image for easy color value processing, and the subtitle character is binarized using the unique values of H, S, and I (320). Labeling is performed to extract component coordinates in units of characters from the binarized image (330). In operation 340, a large area segmentation for clustering the extracted elements of the character unit is performed. The method determines whether the extracted large-area area is a true subtitle area, removes the non-subtitle area, and extracts a subtitle area by character unit 350 to extract a subtitle area.

4 shows a large region extraction method. First, ascending order based on the min-x coordinate of each extracted character unit component is obtained, and R1 _{to k are} obtained (341, 342). R ₁ unconditionally incorporates large region LR ₁ (343), examines the condition that region R ₂ can belong to large region LR ₁ (345), and if so belongs to LR ₁ (346), If not, create a new LR ₂ to incorporate R ₂ (347). The condition that any R _i can be incorporated into LR _{1 ~ m} is that the y coordinate of R _i (min-y <-> max-y) is the y coordinate (min-y) of any LR _s (1≤s≤m). <-> max-y) and more than 85% overlap R _i is considered to be a condition to be incorporated into LR _s . This process is repeated to R _{1 ~ k} to generate LR _m (348,344). The generated large-area areas check the subtitles by performing large-character area verification and character-based individual area, and perform individual area (350).

In FIG. 5, the large character area verification and the character unit individual area are described in detail. Various area coefficients are obtained for the large area LR _i (351, 352, 353).

Size of the region = (max-x-min-x + 1) * (max-y-min-y + 1)

Aspect ratio of the area = (max-y-min-y + 1) / (max-x-min-x + 1)

Character area to background area ratio = number of character pixels / area size

If the area coefficient does not satisfy the condition of the caption, the next large area is examined (354,358). If the area coefficient is satisfied, the individual character area is extracted in the large character area (354,355). In the method of extracting individual text areas, the text areas are extracted by combining the phoneme characters (first, neutral, and final) into one letter unit. When viewing the news subtitles, the news icon subtitles with anchors are composed of at least 4 to 9 characters, so if the number of extracted individual character areas is 4 or more, the subtitle area is extracted (356). In operation 357, individual region coordinates of the extracted LR _i are stored.

Subtitle recognition is performed through the character recognizer on the extracted individual character regions (400).

According to the present invention, it is possible to extract a subtitle area in real time without processing the entire image when extracting the subtitle area, thereby processing a large amount of video video subtitles in real time, thereby reducing the processing time of a large video subtitle. It will bring you advanced.

Claims (7)

When a video input image (contrast value image) of one frame is received, the candidate is determined by extracting a place where there is a sign of a news subtitle from the whole image and checking whether the extracted candidate region is continuous in successive frames. Region extraction step; Caption region verification and regionation step of extracting a large unit area, extracting a large character area, and extracting a character unit individual area to confirm whether the candidate region of the extracted caption is a true caption area; And a caption recognition step of extracting the individual character areas obtained in the caption area verification and regionation step using a recognition means. The method of claim 1, The candidate region extraction step, In the consecutive frames, candidate regions are extracted by time (the candidate regions extracted from the frame at time _t : F _t (x, y) _{1 to n} , n: the number of candidate regions extracted at the time t) and t-1, t extracting a candidate region at time t + 1 and determining a final candidate region at time t + 1; For all candidate regions F _t (x, y) _{1 to n} at time t, candidate regions F _t-1 (x, y) _{1 to s} at all times _t-1 or candidate regions at time t + 1 Checking whether (F _{t + 1} (x, y) _1-k ) matches; Increasing the candidate regions one by one if they do not match, and determining corresponding coordinates as the final candidate regions if they match; and after all investigations at time t are completed, changing the candidate area at time t + 1 into a candidate area at time t. The method of claim 1, The caption region verification and regioning step, A binarization step of converting an RGB image into an HSI image for processing various color values of the caption character of the news video input image, and binarizing the caption character using the unique values of H, S, and I; A labeling step of labeling to extract component coordinates in units of characters from the binarized image; A large unit area extracting step of extracting a large unit area for clustering the extracted character units; News video characterized in that the large character region verification and character unit individual segmentation step of extracting the subtitle region are performed by verifying whether the extracted large unit region is a true subtitle region, removing the non-subtitle region, and region-by-character region. How to extract and recognize subtitles. The method of claim 3, wherein The large unit area extraction step, Obtaining an ordered result R _{1 to k} by ascending order based on the min-x coordinate of each character unit component; Unconditionally incorporating R ₁ into the large region LR ₁ ; If region R ₂ can belong to investigate the conditions that can belong to a large-scale area LR ₁ comprising belong and, incorporated into the R ₂ to produce a new LR ₂ unless the conditions ₁ and LR; Repeating the above process from R _{1 to k} to conditionally check whether any R _i can be incorporated into LR _{1 to m} to generate LR _m . The method of claim 4, wherein Generating the LR _m , Conditions under which R _i can be incorporated into LR _{1 to m} , R _i in the y-coordinate (min-y <-> max -y) which LR _s (1≤s≤m) of the y-coordinate (min-y <-> max -y) and when at least 85% overlap R _i is A method for extracting and recognizing news video subtitles, characterized by being incorporated into LR _s . The method of claim 3, wherein The large character area verification and character unit individual area step, Obtaining area coefficients for each area of the large unit area LR _{i including} the size of the area, the aspect ratio of the area, and the ratio of the text area to the background area; If the area coefficient of the large area LR _i does not satisfy the condition of the caption, inspecting the next large area, and if it satisfies the area, extracts the area by individual character units in the large character area; Extracting and recognizing news video subtitles if the number of the extracted individual character areas is 4 or more and storing the coordinates of the individual areas of LR _i . The method of claim 6, The individual text area extraction step, A method of extracting and recognizing a news video subtitle, comprising extracting a character region by combining characters of a character (first, neutral, and final) into a single character unit.