JPH08317301A - Video output device - Google Patents

Abstract

PURPOSE: To transmit switching timing information to a user without disturbing the viewing of main video by deciding the presence/absence of subtitles from the feature values of subtitles by the divided areas of image data, performing the reduction/move processing of extracted subtitles and compositing them with the main video. CONSTITUTION: A main video input part 100 converts an analog video signal inputted from a video reproducing device 7 to a digital image signal 5-6. A subordinate video input part 200 converts an analog video signal inputted from a television tuner 8 to a digital image signal 5-7. A subtitle area extracting part 300 prepares a subtitle area signal 5-20 by detecting a subtitle area while using a luminance counting means by areas and an edge counting means by areas. A video control part 400 extracts the picture elements of subtitles from the subordinate video signal 5-7 and the subtitle area signal 5-20, the picture elements are thinned and reduced according to the designation of reduction, and the position of subtitles is moved according to the designation of place. This processed subtitle signal is composited with the main video signal 5-6 and outputted. Further, the format of subtitles is detected from the subtitle area signal 5-20 and when it is equal with the designation of user, the main video and the subordinate video are switched.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a video output apparatus for automatically monitoring a sub-picture of a back program so as to support overlooking of important scenes of the sub-picture even when the viewer is watching the main picture. Regarding

[0002]

2. Description of the Related Art In recent years, with the increase of video channels, the number of video output devices for outputting a plurality of videos on the same screen on the video reception system side is increasing.

Conventional video simultaneous output systems in these video output devices can be roughly classified into the following three types. The first method is a so-called picture-in-picture method in which a sub-picture is reduced and displayed in the main picture. The second method is a method of dividing the display area (screen) of the image into two and displaying the main image and the sub-image side by side. The third method is a method of superimposing and displaying a text broadcast instead of a sub-picture.

The viewing mode of the user of such a device is such that the video channel to be focused on is the main video and the video channel to be watched only when the user is interested is the sub-video. For example, there is a viewing mode in which a drama is the main video and a sports program is the sub video. In this case, the user's attention is paid to the sports program, which is the sub-picture, for checking the game situation such as points, scoring scenes, and scenes in which favorite players appear.

However, the conventional video output system has various problems in consideration of the viewing mode of the user.

First, in the first method, since the display area of the sub-picture is small, it is difficult to confirm the contents.
Therefore, even if the sub-pictures are output at the same time, it is easy to overlook the important scene and the desired effect cannot be fully exerted.

In the second method, since the display area is divided into two, there is a problem that both the main image and the sub image are distorted. In addition, when watching the main video, there is a drawback in that the scenes of the sub-video are switched or moved so that the sub-picture becomes difficult to see.

In the third method, the character information is used as a substitute for the sub-picture, and the sub-picture is not viewed directly. In addition, this system has a problem that it is limited to programs in which teletext is being broadcast.

[0009]

SUMMARY OF THE INVENTION The problem to be solved by the present invention is that the main video can be viewed without making the main video smaller, distorted, or distracted, and the importance of the back program. An object of the present invention is to provide a video output device capable of preventing a scene from being overlooked.

[0010]

First, a method for extracting subtitles includes means for extracting a feature amount of a subtitle for each segmented region from digital image data, and presence or absence of subtitles for each region based on the feature amount for each segmented region. Provide a means to judge.

Next, as a method of combining the extracted subtitles with the main video, there are two methods, that is, a subtitle pixel extracting means for extracting only the subtitles in the subtitle area, a subtitle reducing means for reducing the subtitles, and a subtitle moving means for moving the subtitles. Provided is a combined caption processing means and an image combining means for combining the caption with the main video being viewed.

Further, as a method of switching between the main video and the sub video, a format detecting means for detecting the format of the subtitle from the subtitle area and a video switching means for switching the output video are provided.

[0013]

In the sub-video input by the video input means, the feature extraction section extracts the feature amount for each divided area, and the subtitle area determination means determines the presence or absence of subtitles for each area. As a result, even if the number of subtitle characters is small on the entire screen, partial extraction is possible.

Next, the subtitle pixel extracting means removes the background from the subtitle presence area to extract only the subtitle. The extracted subtitles are processed by the subtitle reducing unit and the subtitle moving unit, and displayed in any size on the screen at any size. Finally, the video synthesizing means synthesizes the main video and the subtitles and displays them on the display. Therefore, it is possible to obtain the information of other video without impairing the viewing of the main video, and to inform the user of whether or not to switch from the main video to the sub video.

When video switching is performed instead of composite display, the format is first detected by the format detecting means from the extracted subtitle area. Next, the video switching means determines whether the detected format matches the format specified by the user, and if they match, the main video and the sub video are switched. As a result, the video is switched without paying attention to the displayed subtitles, so that important scenes are not missed.

[0016]

1 is a block diagram of the overall processing of the present invention. The main video input unit 100 receives the video signal sent from the video playback device 7. The video signal is analog data, which is digitized to output main image data 5-6. Similarly, the sub-picture input unit 200 is the TV tuner 8
Input the video signal sent from. The video signal is analog data, which is digitized to generate sub-image data 5
Output -7.

The subtitle area extraction unit 300 uses the sub-image data 5-
Input 7 Then, the sub image data 5-7 is processed and the subtitle area data 5-20 is output. The subtitle area data 5-20 is data in which the presence or absence of subtitles is extracted for each divided area. Details of the subtitle area extraction unit 300 will be described with reference to FIG.

The video control section 400 receives as input three data: main image data 5-6, sub-image data 5-7, and subtitle area data 20. Then, processing for combining or switching images from these data is performed. The display image data 5-29 output by the video control unit 400 is output to the display 1.

FIG. 2 is a processing block diagram showing details of the video control unit 400. The processing of the video control unit 400 is largely the video synthesizing processing 400-a and the video switching processing 400-b.
It can be divided into two. The control unit 410 switches which of the video compositing process 400-a and the video switching process 400-b is to be controlled according to the user's designation.

First, the image synthesizing process 400-a will be described. The subtitle pixel extraction unit 420 uses the subtitle area data 5-2.
0 and sub-image data 5-7 are input. Subtitle area data 5
Only subtitle pixels are extracted from the area determined to have subtitles at -20. As a result, the subtitle pixel data 5-21 with the background removed is output. The subtitle pixel extraction unit 420
The details of the process will be described with reference to FIG. The subtitle reduction unit 430 and the subtitle moving unit 440 prevent the main image combined by the subtitle pixel data 5-21 from being disturbed. The subtitle reduction unit 430 inputs the subtitle pixel data 5-21. At that time, caption pixels are thinned out only when the caption of the caption pixel data 5-21 is large and there is a reduction designation. The data obtained by reducing the caption in this way is re-inserted into the caption pixel data 5-21. The subtitle moving unit 440 also inputs the subtitle pixel data 5-21. The subtitle of the subtitle pixel data 5-21 hinders viewing of the main video, and the subtitle pixel is moved to an arbitrary position only when movement is designated. This allows
The data is put in the subtitle pixel data 5-21 again.

The video synthesizing section 450 inputs the main image data 5-6 and the subtitle pixel data 5-21. Then, the main image data 5-6 and the subtitle pixel data 5-21 are combined and output as display image data 5-29. Thus, no matter what subtitles are displayed on the sub-video, subtitles can be added without disturbing the viewing of the main video, and the user can be informed of whether or not the main video is switched to the sub-video.

Next, the image switching process 400-b will be described. First, the format detection unit 460 inputs the subtitle area data 5-20. Format data 5-22 is detected from the subtitle area data 5-20 and is output. Details of the processing of the format detection unit 460 will be described with reference to FIGS. 16 and 17. The video switching unit 450 inputs the main image data 5-6, the format data 5-22, and the sub image data 5-7. It is determined whether the format data 5-22 is the same as the format designated by the user. When it is determined that the formats are the same, the sub-image data 5-
21 is output by substituting the display image data 5-29. As a result, the user does not miss an important scene without paying attention to the subtitles.

FIG. 3 shows the video composition process 400 described with reference to FIG.
It is a flowchart of the image data in -a. The subtitle area extraction unit 300 inputs the sub-image data 5-7. The subtitle area extraction unit 300 outputs subtitle area data 5-20 in which an area having subtitles is extracted. In the subtitle area data 5-20, 1 is assigned to the area including the subtitle, and 0 is assigned to other areas.
The subtitle pixel extraction unit 420 inputs the subtitle area data 5-20 and the sub-image data 5-7. The subtitle pixel extraction unit 420 performs processing for extracting subtitle pixels from the subtitle area of the subtitle area data 5-20. Output subtitle pixel data 5-2
The subtitle pixel of 1 contains the pixel value of the sub-image data 5-7. The other pixels have a black color value. The caption reduction unit 430 thins out the caption pixels only when the caption of the caption pixel data 5-21 is large and there is a contraction designation. The data obtained by reducing the caption in this way is re-inserted into the caption pixel data 5-21. The subtitle moving unit 440 uses the subtitle pixel data 5-21.
Enter The subtitle of the subtitle pixel data 5-21 hinders viewing of the main video, moves the subtitle pixel to an arbitrary position only when movement is designated, and inserts the subtitle pixel data 5-21 again.

The video synthesizing section 450 inputs the main image data 5-6 and the subtitle pixel data 5-21. Main image data 5-
6 and subtitle pixel data 5-21 are combined to display image data 5
Output as -29. Output display image data 5-
29 is displayed on the display 1. This allows subtitles to be added without hindering viewing of the main video regardless of what subtitles are displayed on the sub video, and it is possible to inform the user of a trigger for switching from the main video to the sub video.

FIG. 4 shows the video composition processing 400 described with reference to FIG.
It is a figure which shows the example of a display which synthesize | combined the subtitles in -a.

An example of composition of 5-29-a is subtitle composition processing 40.
The subtitle processing is not performed on subtitle pixel data 5-21 at 0-a. As a result, composite display is performed at the position of the subtitle displayed in the sub image data 5-7 at the time of input. 5-29-
b is the subtitle pixel data 5-21 in the subtitle reduction unit 420 first.
Perform subtitle reduction processing. Further, in this example, subtitle moving processing is further performed by the subtitle moving unit 430 and combined display is performed. 5-29
-C is an example in which the subtitle pixel data 5-21 of the subtitled area is displayed on the liquid crystal panel 1-1.

FIG. 5 shows the video switching process 4 described with reference to FIG.
It is a flowchart of the image data in 00-b. The subtitle area extraction unit 300 inputs the sub-image data 5-7. The subtitle area extraction unit 300 outputs subtitle area data 5-20 in which an area having subtitles is extracted. Subtitle area data 5-2
For 0, 1 is assigned to the area including subtitles, and 0 is assigned to other areas. The format detection unit 460 inputs the subtitle area data 5-20. The format detection unit 460 detects and outputs the format data 5-22 from the arrangement state of the subtitle area. Video switching unit 4
70 inputs the main image data 5-6, the format data 5-22, and the sub-image data 5-7. It is determined whether the format data 5-22 and the format designated by the user are the same. When it is determined that the formats are the same, the sub-image data 5-21 is substituted into the display image data 5-29 and output. The output display image data 5-29 is displayed on the display 1.

FIG. 6 is a processing block diagram of the subtitle area extracting unit 300. In the figure, 5-7 is a sub-picture input unit 200.
It is digital image data converted by. The feature extraction unit 350 extracts the feature of the subtitle from the input image and outputs it as a count value. There are two types of count values, that is, the values counted by the area-specific brightness counting section 600 and the area-specific edge counting section 700. Further, in the processing described in this figure, the screen is divided into divided areas and the processing is performed for each area.

The region-brightness counting section 600 extracts high-brightness pixels in each region and outputs the number thereof. The area-based edge counting unit 700 extracts pixels (hereinafter, strong edges) having a large color difference from the adjacent pixels in each area, and outputs the number. The subtitle area determination unit 800 receives as input the count values obtained by the area-based luminance counting section 600 and the area-based edge counting section 700. When both count values satisfy a certain condition, it is determined that there is a caption in the area. The subtitle area data 5-20 output by the subtitle area determination unit 800 is 1-bit data indicating the presence or absence of subtitles for each area. 1 for data in the area containing subtitles
Is assigned, and 0 is assigned to all other values.

In the caption determination of this embodiment, both high-luminance pixels and strong pixels are checked as a feature of the caption. Therefore, there is no edge like light illumination and high brightness area. A region such as a shogi board that has edges but low luminance is not extracted, which has the effect of reducing erroneous recognition. Furthermore, since the characteristics of subtitles are determined for each divided area, even if the number of subtitle characters is small on the entire screen, partial extraction can be performed.

FIG. 7 is an explanatory diagram of programs and data. In the figure, a program 5-1 is a computer program for the subtitle extraction / synthesis processing of FIG. Also, 5
-2 to 5-29 are the data that the program 5-1 refers to, and among them, 5-2 to 5-5 are the parameters to be preset for extraction, 5-6 to 5-29.
Is data for work used for one-time processing.

The data will be described below. First, the threshold value 5-
2, 5-3 are thresholds used when extracting the feature amount of the subtitles from the image, and thresholds 5-4, 5-5 are thresholds for determining the subtitles from the feature amount, which are set in advance. Keep it.

The main image data 5-6 is the main image conversion device 10.
The sub-image data 5-7 is digital image data of the latest frame converted by the sub-image conversion device 1.
It is the digital image data of the latest frame converted by 0. This image data is red image data 5 for processing.
-7-1, green image data 5-7-2, blue image data 5-
7-3 is divided into three types of color component data.

Next, various data created by the feature extraction unit 350 will be described. The brightness data 5-8 is array data in which high brightness pixels are extracted from the image data. The horizontal edge data 5-9 is array data obtained by extracting pixels having a large color difference in the horizontal direction of the image, and the vertical edge data 5-10 is array data obtained by extracting pixels having a large color difference in the vertical direction of the image. The previous frame luminance data 5-11, the previous frame horizontal edge data 5-12, and the previous frame vertical edge data 5-13 are the luminance data 5-9 and horizontal edge data 5 one frame before the frame currently being processed temporarily. -10, vertical edge data 5-
11 is the array data in which 11 is stored.

The brightness collation data 5-14 is the brightness data 5
Both -8 and the previous frame brightness data 5-11 are array data in which high brightness pixels are stored. Further, the horizontal edge matching data 5-15 is array data in which the horizontal edge data 5-9 and the previous frame horizontal edge data 5-12 both store pixels of strong edges. Similarly, the vertical edge matching data 5-16
Is the array data in which the vertical edge data 5-10 and the previous frame vertical edge data 5-13 both store pixels of strong edges.

The brightness area data 5-17 is array data which stores the result of counting the number of high brightness pixels of the brightness matching data 5-14 for each divided area. Horizontal edge area data 5
-18 is array data in which the result of counting the number of pixels of the strong edge of the horizontal edge matching data 5-15 is stored for each divided area. Similarly, the vertical edge area data 5-19 is array data in which the result of counting the number of pixels of the strong edges of the vertical edge matching data 5-16 is stored for each divided area. In addition, it is desirable that each of the divided areas has a size such that one subtitle character is included therein. In this embodiment, the screen is divided into 16 horizontally and 10 vertically.

Next, the data created by the subtitle area determining section 400 will be described. The subtitle area data 5-20 is array data that stores the result of determining the presence or absence of subtitles for each divided area.
Further, the subtitle pixel data 5-21 created by the subtitle pixel extraction unit 900 is image data in which only subtitles are extracted.

The format data 5-22 is the position and direction data of the extracted subtitle of the representative image. The row count data 5-23 is array data that stores the number of areas determined to have captions on a row-by-row basis. Maximum row count data 5-
Reference numeral 24 is data in which the number of row areas having the largest number of row count data 5-23 is stored. The maximum row position data 5-25 is data that stores the row number of the maximum row. Similarly, the column count data 5-26 is array data that stores the number of regions determined to have captions on a column-by-column basis. The maximum column count data 5-27 is data that stores the maximum value of the column count data 5-26. The maximum column position data 5-28 is data that stores the column number of the column having the maximum value.

The display image data 5-29 is the sub-image data 5
-7 is data to be displayed on the display after being subjected to image composition or image switching processing by the image control unit 400,
If the sub-image data does not have a subtitle, the main image data is transferred to the display image data as it is.

Hereinafter, with reference to FIG. 7, the processing procedure of the area-specific luminance counting section 600, the area-specific edge counting section 700, the subtitle area determining section 800, and the subtitle pixel extracting section 900 of FIG. 6 will be described in detail using a flowchart. explain.

FIG. 8, FIG. 9, and FIG.
00, the processing procedure of the edge-by-area counting section 700 is shown in FIGS. 11, 12, and 13, and the processing procedure of the subtitle area determination section 800 is shown in FIG.

First, the processing procedure of the area-specific luminance counting section 600 will be described with reference to FIGS. 8, 9 and 10. Process 601
The process from the process 609 to the process 609 (FIG. 8) is a process of extracting a high-luminance pixel in the input image. In step 601, variables X and Y used in steps 602 to 609 are initialized to 0. In the process 602, each color component of the input red image data 5-7-1, green image data 5-7-2, and blue image data 5-7-3 of the sub-image data 5-7 has a luminance of a threshold value of 1 or more. If all the color components have luminances equal to or greater than the threshold value 1, the process proceeds to step 603, and if not, the process proceeds to step 604. Processing 60
In 3, the 1 is written in the array of the brightness data 5-8 at the position equal to the position on the image whose brightness is examined in the process 602, and the process 6
In 04, 0 is written in the array of the brightness data 5-8 at the position equal to the position on the image whose brightness is checked in the process 602. Processes 605 to 609 are address updating processes for performing the above-described process on all pixels. When all the processes are completed, the process moves to the process 610.

Processes 610 to 618 (FIG. 9) are processes for extracting pixels in which high brightness continues over a plurality of frames. In process 610, variables X and Y used in processes 611 to 618 are initialized to 0.
A process 611 checks whether the values of the brightness data 5-8 and the previous frame brightness data 5-11 are both 1, and if both are 1, the process proceeds to a process 612, and if not, the process proceeds to a process 613. In process 612, 1 is written in the array of the brightness collation data 5-14 at the position equal to the position on the brightness data checked in process 611, and in process 613, the brightness collation data at the position equal to the position on the brightness data checked in process 611. 5-14
Write 0 to the array. Processes 614 to 618 are address update processes for performing the above process on all pixels. When all the processes are completed, the process moves to process 619.

Processes 619 to 625 (FIG. 9) are processes for updating the previous frame luminance data 5-11.
In process 619, variables X and Y used in processes 620 to 625 are initialized to 0. In process 620, the brightness data is substituted for the previous frame brightness data. Processes 621 to 625 are address update processes for performing the above process on all pixels. When all the processes are completed, the process moves to the process 626.

Processes 626 to 639 (FIG. 10) are processes for counting the number of high-luminance pixels for each divided area. In process 626, variables i, j, Xb, and Yb used in processes 627 to 239 are initialized to 0. Process 62
7 checks whether or not the brightness collation data 5-14 is 1, and if 1 then moves to processing 628, and otherwise moves to processing 629. In process 628, 1 is added to the array of the brightness region data 5-17 corresponding to the segmented region to which the position of the brightness matching data checked in process 627 belongs. Process 629 to Process 63
Reference numeral 9 is an address update process for performing the above process on all pixels. When all the processes are completed, the brightness count result for each area is stored in the brightness area data 5-17.

Next, the processing procedure of the edge-by-area counter 700 will be described with reference to FIGS. 11, 12, and 13. Processes 701 to 712 (FIG. 11) are processes for extracting pixels of strong vertical and horizontal edges. In process 701, variables X and Y used in processes 702 to 712 are initialized to 0. In the process 702, whether the color components of the red image data 5-7-1, green image data 5-7-2, and blue image data 5-7-3 of the input image data have a color difference of 2 or more in the horizontal direction. The color difference of all color data is 2
If it is above, the process proceeds to step 703, and if not, process 7
Move to 04. In process 703, 1 is added to the array of the horizontal edge data at the same position as the position in the image for which the color difference is checked in process 702.
In step 704, 0 is written in the array of horizontal edge data at the position equal to the position in the image for which the color difference is checked in step 702.
Write. In steps 705 to 707, similarly,
Extract vertical edge data. Steps 708 to 712
Is an address update process for performing the above process on all pixels. Processing 713 when all processing is completed
Move on to.

Processings 713 to 324 (FIG. 12) are processings for extracting pixels in which a strong edge continues over a plurality of frames. In the processing 713, the processing 714
The variables X and Y used in the process 724 are initialized to 0. A process 714 checks whether the values of the horizontal edge data 5-9 and the previous frame horizontal edge data 5-12 are both 1, and if both are 1, the process proceeds to a process 715, and if not, the process proceeds to a process 716. In process 715, 1 is written in the array of the horizontal edge collation data 5-15 at the position equal to the position on the horizontal edge data checked in process 714, and in process 716, the position equal to the position on the horizontal edge data checked in process 714 is written. Write 0 to the array of the lateral edge matching data 5-15.
Similarly, vertical edge matching data is calculated in processing 717 to processing 719. Processes 720 to 724 are address update processes for performing the above-described process on all pixels. When all the processes are completed, the process moves to process 725.

Processes 725 to 731 (FIG. 12) are processes for updating the previous frame horizontal edge data 5-12 and the previous frame vertical edge data 5-13. Process 72
5, the variable X used in processing 726 to processing 731,
Initialize Y to 0. In the process 726, the horizontal edge data 5-9 is substituted into the previous frame horizontal edge data 5-15, and the vertical edge data 5-1 is added to the previous frame vertical edge data 5-13.
Substitute 0. Processes 727 to 731 are address update processes for performing the above-mentioned process on all pixels. When all the processes are completed, the process moves to the process 732.

Processes 732 to 748 (FIG. 13) are processes for counting the number of pixels of vertical and horizontal strong edges for each divided area. In process 732, process 733 to process 74
The variables i, j, Xb, and Yb used in 8 are initialized to 0. A process 733 checks whether or not the horizontal edge collation data 5-15 is 1, and if it is 1, the process proceeds to a process 734, and if not, the process proceeds to a process 735. In process 734, 1 is added to the array of the horizontal edge region data 5-18 corresponding to the segmented region to which the position of the horizontal edge matching data checked in process 733 belongs. Similarly, in processing 735 and processing 736, vertical edge area data is calculated. Processes 737 to 748 are address update processes for performing the above-described process on all pixels. When all the processing is completed, the edge count result for each area is stored in the horizontal edge area data 5-18 and the vertical edge area data 5-19.

By the above processing, the feature extraction of subtitles is performed for each area.

Next, the processing procedure of the subtitle area determination unit 800 will be described with reference to FIG. Process 801 to process 809
The process up to (FIG. 15) is the process of determining the presence or absence of subtitles for each divided area. In process 801, variables Xb and Yb used in processes 802 to 809 are initialized to 0. Process 802
Examines whether the luminance area data has a threshold value of 3 or more, the horizontal edge area data has a threshold value of 3 or more, and the vertical edge area data has a threshold value of 3 or more. If three conditions are satisfied, the process proceeds to step 803, and so on. If not, the process proceeds to process 804. In process 803, 1 is written in the array of the subtitle region data 5-20 in the same region as the region checked in process 802, and in process 804, process 8 is performed.
Subtitle area data 5-2 of an area equal to the area checked in 02.
Write 0s to the 0 array. Processes 805 to 809 are address update processes for performing the above-described process on all areas. When all the processes are completed, the process moves to process 810. By the above, the presence or absence of subtitles for each area is obtained.

The processing procedure of the subtitle pixel extraction unit 420 and the format detection unit 460 of FIG. 2 will be described in detail below with reference to FIG. 7 using a flowchart. FIG. 15 shows the processing procedure of the subtitle pixel extraction unit 420, and FIGS. 16 and 17 show the processing procedure of the format detection unit 460.

First, the subtitle pixel extraction section 420 will be described.
Processes 420-1 to 420-16 are processes for checking each pixel for a subtitle and extracting pixels corresponding to the subtitle. In process 420-1, variables i, j, Xb, and Yb used in processes 420-2 to 420-16 are set to 0.
Initialize with. Process 420-2 is subtitle area data 5-2.
It is checked whether 0 is 1 or not, and if it is 1, the process proceeds to process 420-3, and if not, the process proceeds to process 420-05. Process 42
In 0-03, it is checked whether the red image data, the blue image data, and the green image data of the corresponding pixel have a brightness of threshold value 1 or more,
If all three are greater than or equal to the threshold value 1, the process proceeds to processing 420-04,
If not, the process proceeds to process 420-05. Process 420-
In 04, the value of the sub-image data is substituted for the subtitle pixel data at the pixel position checked in the process 420-03. Process 420-0
In 5, 0 is substituted for the subtitle pixel data at the pixel position checked in the process 420-03.

Processes 420-06 to 420-16 are address update processes for performing the above-mentioned process on all pixels. When all the processing is completed, only the subtitles in the sub-image are extracted.

Next, the format detector 460 will be described. Processes 460-1 to 460-7 (FIG. 16) are processes for projecting subtitle region data in the row direction. Similarly, processes 460-8 to 460-14 (FIG. 16)
This is a process of projecting subtitle region data in the column direction. The processing in the row direction will be described in detail. In the process 460-1,
Variable X used in processing 460-2 to processing 460-7
b and Yb are initialized to 0. In process 460-2, the number of captioned areas is added for each row. Processes 460-3 to 460-7 are address update processes for performing this process on all areas. When all the processes are completed, the process proceeds to process 460-8, and similarly, the process of projecting in the column direction is performed. When the process in the column direction is completed, the process proceeds to process 460-15. Through the above processing, the number of subtitled areas in the row and column directions is stored in the row count data 5-23 and the column count data 5-26.

Processes 460-15 to 460-33
In FIG. 17, the results of the individual areas are integrated, and the process of finally determining the presence or absence of subtitles and determining the position and direction of subtitles is performed. In process 460-15, variables Xb and Yb used in processes 460-16 to 460-25 are initialized to 0. Since the process 460-16 checks the line having the largest value in the line count data, it is checked whether the line count data is larger than the maximum line count data. If the line count data is larger, the process proceeds to process 460-17. If the line count data is smaller, the process 460- Move to 19. In process 460-17, the maximum line count data is replaced with the value of the line count data, and in process 460-18, the line number at that time is stored in the maximum line position data. Process 4
The processing from 60-19 to processing 460-20 is an address updating processing for performing the above processing on all the rows. Further, the processes 46-21 to 460-25 similarly calculate the maximum column count data and the maximum column position data.

In process 460-26, it is checked whether the maximum row count data is greater than or equal to the threshold value 4 or the maximum column count data is greater than or equal to the threshold value 4. If the conditions are satisfied, it is determined that captions are present, and the process 460- If 27 is not satisfied, it is determined that there is no subtitle.

When it is determined that there is a subtitle, it is classified whether the subtitle has a format of vertical left writing, vertical right writing, horizontal overwriting, or horizontal draft on the image.

First, in processing 460-27, the maximum row count data and the maximum column count data are compared, and if the maximum row count data is large, it is determined to be "horizontal writing" and processing 460.
If the maximum column count data is large, it is determined to be "vertical writing" and the flow proceeds to processing 460-31. Process 460-
In 28, the maximum line position data is in the center line (5 in this embodiment).
(Line) It is checked whether it is more than or equal to it, and if it is greater than or equal to it, it is determined to be “overwrite” and the process proceeds to processing 460-29. And process 46
In 0-29, "upper horizontal writing" is written in the format data, and in processing 460-30, "lower horizontal writing" is written in the format data. On the other hand, in process 460-31, it is checked whether or not the maximum column position data is in the center column (8th column in this embodiment) or more. If it is, the right writing is determined and the process proceeds to process 460-32. If so, it is determined as “left writing” and processing 460-33.
Move to. In process 460-32, "right vertical writing" is written in the format data, and in process 460-33, "left vertical writing" is written in the format data.

With the above processing, the subtitle format can be determined. In this embodiment, the format is determined by dividing it into two vertically and horizontally, but the format can be distinguished in more detail by increasing the number of divisions, and the number of subtitle areas is also counted.
It goes without saying that it is possible to distinguish the formats in which the proportion of subtitles in the screen is large when the number of areas is large, and the proportion of subtitles in the screen is small when the number of areas is small.

The subtitle feature extraction section and subtitle determination section have been disclosed in detail above with reference to the flowcharts.

According to the present subtitle extraction example, since the feature extraction unit determines whether or not subtitles appear for each divided area from the digital image data, even if the number of characters in the subtitles is small on the entire screen, Partial extraction is possible. Further, since the subtitle area determination unit projects the subtitle presence / absence information in the row and column directions to determine, it is possible to distinguish whether the appearing subtitle is vertical writing or horizontal writing. Further, the feature extraction unit counts the number of high-brightness pixels equal to or higher than the first threshold value included in each region and the number of edges equal to or higher than the second threshold value included in each region as a feature. Therefore, it is not judged as a subtitle only by the appearance of a screen with high brightness. Further, the edge is obtained for each of a plurality of directions, and the complexity of the character portion can be faithfully extracted.

[0063]

According to the present invention, since the subtitles extracted from the sub video are combined with the main video that is being viewed, it is possible to obtain text information that is a substitute for the sub video even if text broadcasting is not being performed.
In addition, the size and display position of the subtitles to be combined can be arbitrarily changed by the user, and viewing of the main video is not hindered.

Further, since the sub-picture is switched and displayed depending on the subtitle format so that the subtitle can be viewed, the user does not miss an important scene without paying attention to the subtitle.

[Brief description of drawings]

FIG. 1 is a block diagram of the overall processing of the present invention.

FIG. 2 is a processing block diagram showing details of a video control unit 400.

FIG. 3 is a flowchart of image data in the video compositing process 400-a described in FIG.

4 is an explanatory diagram of a display example in which captions are combined in the image combining process 400-a described in FIG.

5 is a flowchart of image data in the video switching process 400-b described with reference to FIG.

FIG. 6 is a processing block diagram of a subtitle area extraction unit 300.

FIG. 7 is an explanatory diagram of programs and data.

FIG. 8 is a flowchart showing a processing procedure of an area-brightness counting section 600.

FIG. 9 is a flowchart showing a processing procedure of an area-brightness counting section 600.

FIG. 10 is a flowchart showing a processing procedure of an area-brightness counting section 600.

FIG. 11 is a flowchart showing a processing procedure of the edge-by-region counting unit 700.

FIG. 12 is a flowchart showing a processing procedure of a region-based edge counting section 700.

FIG. 13 is a flowchart showing a processing procedure of a region-based edge counting section 700.

FIG. 14 is a flowchart showing a processing procedure of a subtitle area determination unit 800.

FIG. 15 is a flowchart showing a processing procedure of a subtitle pixel extraction unit 420.

FIG. 16 is a flowchart showing a processing procedure of a format detection unit 460.

FIG. 17 is a flowchart showing the processing procedure of the format detection unit 460.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Display, 1-1 ... Liquid crystal panel, 7 ... Main image reproducing device, 8 ... TV tuner, 100 ... Main image input unit,
200 ... Sub-picture input section, 300 ... Subtitle area extraction section, 40
0 ... Video control unit.

Claims (6)

[Claims] 1. A video output device for inputting a plurality of videos from a VTR or a TV tuner and synthesizing and outputting to a display, a first means for inputting a main video, a second means for inputting a sub-video, and a video output. A third means for extracting subtitle information from the sub-picture specified by the user, and a fourth means for processing and outputting the sub-picture and the main video from the sub-title information obtained by the third means. A video output device characterized by the above. 2. The video output device according to claim 1, wherein the third means comprises a fifth means for judging the presence or absence of the subtitle for each area. 3. The fourth means according to claim 1, further comprising: sixth means for detecting subtitle pixels in an area determined to have the subtitle; and seventh means for reducing the detected subtitle. A video output device comprising: an eighth means for moving and a ninth means for synthesizing the main video at an arbitrary position. 4. The fourth means according to claim 1, wherein the tenth means for detecting a subtitle format from the area determined to have the subtitle and the detected format coincides with a format designated by a user. A video output device comprising eleventh means for switching the sub video to the main video for display. 5. The video output device according to claim 4, wherein the tenth means detects the format based on vertical and horizontal writing of the subtitles, and the display positions of up, down, left and right. 6. The fifth means according to claim 2,
A video output device for determining the presence or absence of subtitles on the condition that the number of high-brightness pixels and the number of edges in a video and those of them remain stationary for a certain period of time.