KR20100000256A - Method for displaying caption of moving picture - Google Patents

Abstract

PURPOSE: A method for displaying a caption of a moving picture is provided to display a caption of a moving picture on the surroundings of a speaker according to the location of the determined speaker, thereby increasing visual immersion of a viewer about the moving picture. CONSTITUTION: Time information on which a caption is to be displayed is obtained(S710). A face is detected from image frames corresponding to the time information(S720). A speaker corresponding to the image frames is determined(S730). A face area range of the speaker is determined using the face(S740). An area on which the caption is to be displayed is adaptively determined. The caption is displayed on the determined area.

Description

How to display captions in a video {Method for Displaying Caption of Moving Picture}

The present invention relates to video display, and more particularly, to a method of displaying a caption of a video.

Recently, due to the development of science and technology and the improvement of economic level, the spread of communication network such as high-speed Internet and the rapid increase of users of high-speed communication network have been made, and the rapid increase of users of high-speed communication network has made it possible to disseminate various video contents through communication network.

Types of the above-described image contents include videos of a professional field such as education, medicine, or science, and videos of entertainment / culture such as broadcasting or music video.

In the case of a video produced in a foreign language among such videos, subtitles are displayed in the video to help viewers understand, and the viewers can understand the conversations of the speakers in the video.

On the other hand, recently, even in the case of a video produced in a foreign language as well as a video produced in a foreign language, subtitles are displayed in the video to add fun to the video.

However, conventionally, in displaying subtitles of such videos, the subtitles are fixedly displayed on one side of the screen, for example, at the bottom of the screen, regardless of the position of the speaker appearing in the video. In order to understand the contents, the subtitles displayed at the bottom of the video and the video cannot be watched alternately, so that the visual immersion of the video is lowered and the understanding of the video is inevitably reduced.

SUMMARY OF THE INVENTION The present invention has been made in view of the above-described problem, and provides a caption display method of a video that can adaptively determine a position at which a caption of a video is output according to the location of a speaker determined using audio information and video information constituting a video. It is technical problem to provide.

According to an aspect of the present invention, there is provided a method for displaying a caption of a video, the method including: obtaining time information for displaying the caption from a caption displayed on the video; Detecting a face from image frames corresponding to the time information among the image frames constituting the video; Determining a speaker corresponding to the image frames corresponding to the time information by changing whether a feature point of a face detected from each of the image frames corresponding to the time information is changed; Determining a range of the speaker's face region in each of the image frames corresponding to the time information by using the determined speaker's face; And adaptively determining a region in which the caption is to be displayed according to a range of the speaker's face region and displaying the caption in the determined region.

The displaying of the subtitles may include: setting a candidate area in which the subtitles are to be displayed among areas within a predetermined distance from a range of the speaker's face area; And determining a candidate region having the least color deviation and the highest saturation among the candidate regions as a region where the subtitle is to be displayed.

At this time, in the subtitle display step, the subtitles are displayed in a color complementary to the color of the area in which the subtitles are to be displayed.

On the other hand, the above-described subtitle display method of the video further comprises the step of analyzing the voice information constituting the video before the subtitle display step to determine the spatial location where the voice is generated, in the subtitle display step, The location of the caption is determined by using the spatial position of.

At this time, the spatial position of the voice is characterized by using at least one of the difference between the voice signal output for each channel and the voice frequency band.

According to the present invention, there is an effect that the viewer's visual immersion of the video can be enhanced by displaying subtitles of the video around the speaker according to the location of the speaker determined using the video information and the audio information constituting the video.

In addition, according to the present invention, the viewer with a hearing impairment can easily understand the conversation contents of the speaker of the video when two or more speakers talk in the video.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram showing a video player according to an embodiment of the present invention. The video reproducing apparatus 100 displays subtitles of a video and a video through a display (not shown), and as illustrated, the receiver 110, the time information detector 120, the image analyzer 130, and the speaker are determined. The unit 140 includes a caption position determiner 150 and a video output unit 160.

The receiver 110 receives subtitle data of a video and a video from a storage medium (not shown), separates the received video into video information and audio information, and provides the video information to the image analyzer 130 to be described later. The caption data of the captured video is provided to the time information detector 120 to be described later.

The above-described storage medium may be configured separately from the video playback device 100 but may be included in the video playback device 100. In addition, in the above-described embodiment, the reception unit 110 is described as receiving subtitle data of a video and a video from a storage medium. In the modified embodiment, the subtitle data of the video and a video may be downloaded in a streaming format through the Internet. Could be

The time information detector 120 detects time information or image frame information for displaying each subtitle from the subtitle data provided from the receiver 110 and provides the image information to the image analyzer 130. Here, the subtitle may be a file format such as smi, idx, sub, srt, psb, ssa, ass, and usf, and an example of a subtitle produced in the smi file format is described below.

<SYNC start = 46085> <P Class = KRCC>

Mulder! Look at this.

<SYNC start = 48341> <P Class = KRCC>

What's wrong? Scully.

In the example above, it can be seen that the subtitle "Mulder! Look at this" is displayed from 46.085 seconds, and the subtitle "Why? Scully." Is displayed from 48.341 seconds. From the caption data, the time information detector 120 displays 46.085 seconds to 48 hours of time information for displaying the caption "Mulder! Look at this." You will get 341 seconds of information.

On the other hand, in the above-described embodiment, it is described as extracting time information from the caption data, but if the frame number for displaying the caption is included in the caption data, the frame number is directly extracted and provided to the image analyzer 130. Could be

The image analyzer 130 detects a face and a face position from each image frame corresponding to the time information provided from the time information detector 120 and provides the same to the speaker determiner 140. The image analyzer 130 may detect a face using various methods. In one embodiment, the face detector may detect a face using a skin color extraction method.

In the case of using the skin color extraction method, the image analyzer 130 displays an area having a color deviation value within a threshold value within the image frame in white, displays the remaining area in black, and then displays the area in white. I find my eyes. At this time, the eyes are geometrically recessed in the face, and since the eyes are distinctly different from the skin color, they can be easily found in the white area. According to this method, the white area with two black dots is determined as the face.

On the other hand, in order to compensate for the disadvantages that may occur when using the skin color extraction method (for example, it may not be able to detect a face or a face wearing glasses), the contour of the face is used by using the edge difference image based on the movement. It may be used by a method of finding or using a method of detecting a face by blinking eyes.

In addition to these methods, a face may be detected from each image frame using various methods such as a method based on local features of a face or a method based on a template-based face.

The speaker determiner 140 determines a speaker in each image frame corresponding to the caption by using a face in each image frame detected by the image analyzer 130. The video is usually composed of 24 to 30 video frames per second, and since each character may have other characters besides the speaker, the faces detected by the image analyzer 130 are all the speaker's faces. You can't. Accordingly, the speaker determiner 140 determines the speaker who is speaking in the current frame by detecting the speaker's face among the faces detected by the image analyzer 130.

A method of determining the speaker by using the face in each image frame detected by the image analyzer 130 will be described in more detail with reference to FIG. 2.

The speaker determiner 140 determines the speaker for the corresponding image frame by determining a face whose feature point is changed among faces of each image frame detected by the image analyzer 130 as the speaker's face. In one embodiment, the feature point may be an area under the eye of the face, such as the mouth 210 or the area 212 around the chin. This is generally because the area 212 around the mouth or chin moves when a person speaks.

For example, as illustrated in FIG. 2, when the image frame corresponding to the time information on which the subtitle is to be displayed is n frames and n + 1 frames, the feature points 210 and 212 of the face 200 are changed at n + 1 frames. Thus, the face 200 of FIG. 2 is determined as the speaker's face.

For example, if the width and height of a face detected in n frames are 200 X 200, the detected faces are equally divided into 30 widths and 30 heights, and then the faces detected in n + 1 frames are equally divided. When the two frames are overlapped with each other, the color of the area under the eye in the n + 1 frame determines the face different from the color of the same area among the n frames.

On the other hand, when the speaker is determined, the speaker determiner 140 provides the position of the face determined by the speaker to the subtitle position determiner 150 to be described later.

The caption position determiner 150 determines the position at which the corresponding caption is to be displayed by using the face position of the speaker determined by the speaker determiner 140. Specifically, the caption position determiner 150 first determines a candidate region in which captions are to be displayed using the speaker's face position. In one embodiment, the candidate region in which the caption is to be displayed is an area within a predetermined distance from the speaker face region range 310 detected from each of the image frames in which the caption is to be displayed, for example, the speaker's face. A region where the height does not deviate by more than one line from the region range 310 or the horizontal axis does not deviate by two letters from the speaker's face region range 310 may be determined as a candidate region for displaying the subtitle. In this case, the speaker's face region range 310 refers to a region in which the speaker's face region extracted from the respective image frames for which the caption is to be displayed is changed according to each image frame. In FIG. 3, when there are three image frames in which subtitles are to be displayed, an area in which the speaker's face included in each of the three image frames is changed is determined as the speaker's face region range 310.

In this case, when there is no candidate region 320 that satisfies such a condition, as illustrated in FIG. 4, the region of the speaker's face region 310, which is to the left or the right of the speaker's face region range 310, is not affected. You can set the area that can be displayed by wrapping the contents of the subtitle at least once without being selected as a candidate area.

On the other hand, when the subtitles are displayed among the candidate areas, the area where the subtitles leave the screen or the speaker may be excluded from the candidate areas.

Subsequently, the caption position determiner 150 determines the caption display region among the candidate regions. In one embodiment, the caption position determiner 150 has the smallest color deviation and the highest saturation region among the candidate regions. Can be determined as the subtitle display area. If the saturation value of the determined subtitle display area is less than or equal to the threshold value, the color of the subtitle may be determined as an average color and a complementary color of the determined subtitle display area.

In the above-described embodiment, the caption position determining unit 150 determines the caption position using the range of the speaker's face area determined by the speaker determination unit 140, but in the modified embodiment, the caption position is determined. In this case, the caption position may be determined by using the audio information constituting the video together or by using only the audio information. To this end, the video reproducing apparatus 100 may further include a voice analyzer 155.

The voice analyzer 155 receives voice information constituting a video from the receiver 110 described above, and receives time information for displaying a corresponding subtitle from the time information detector 120 to determine a video frame corresponding to the time information. The spatial location at which the voice is generated is determined from the voice information, and the determined spatial location is provided to the speaker determiner 140 described above.

Specifically, the voice analyzer 155 first analyzes the standard of voice information provided from the receiver 110. In one embodiment, the specification of the voice information may be stereo, Dolby Digital, Dolby Surround, Dolby Pro Logic, Digital Surround, DTS, etc., in which a voice signal for each channel is output differently. 155) is to analyze whether the audio standard of the video corresponds to.

Thereafter, the voice analyzer 155 determines the spatial location where the voice is generated using the difference of the voice signal of each channel or the voice frequency band. In general, the audio information of a video is stored in two left and right channels or more in order to match the position of a sound source generated in the video frame constituting the video. It can be inferred. More specifically, when using the difference in the voice signal, assuming that the analyzed voice standard is stereo or Dolby Digital, since the voice standard is to record the voice in different directions using a separate microphone, the size of the voice for each channel is different In addition, since the time difference (phase difference) for the voice to reach the microphone for each channel is inevitably generated, the voice analyzer 155 determines the left and right sides of the voice generation position using this difference. In one embodiment, the voice analyzer 155 determines the left and right positions of the voice by using the time difference of the voice signal in the low band, and determines the left and right positions by the size difference of the voice in the high band.

On the other hand, in the case of using the voice frequency band, the voice analyzer 155 determines the top and bottom of the voice generation position using the spectral characteristics of the voice.

When the spatial position of the voice determined by the above method is provided to the caption position determiner 150, the caption position determiner 150 determines the caption display area in consideration of the spatial position of the determined sound.

On the other hand, if the subtitle position determiner 150 cannot determine the area where the subtitle is to be displayed through the above-described methods, for example, the speaker does not appear in an image frame corresponding to time information and thus detects the speaker's face and face position. If it is not possible, if the speaker appears in the video frame but the video frame is dark and cannot recognize the speaker, or if the audio information is not suitable for determining the spatial location where the voice is generated, One side, such as the right side or the bottom of the screen, may be determined as the position where the subtitle is to be displayed.

In consideration of such various situations, the caption positions determined by the caption position determiner 150 are summarized in a table as shown in FIG. 5.

Referring back to FIG. 1, the video output unit 160 outputs a video and a caption of the video through a display (not shown). At this time, the caption of the video is determined by the caption position determiner 150 described above. The subtitle display area is displayed. That is, as shown in FIG. 6, the video output unit 160 outputs subtitles around the speaker in the video frame.

Hereinafter, a caption display method of a video will be described with reference to FIG. 7.

7 is a flowchart illustrating a caption display method of a video according to an embodiment of the present invention. First, when the caption data of the video and the video are received (S700), time information for displaying each caption is obtained from the caption data of the video (S710). In one embodiment, the time information may be information about a time at which the corresponding subtitle is to be displayed or may be a frame number at which the corresponding subtitle is to be displayed.

Next, in operation 7720, a face and a face position are detected from image frames corresponding to time information in which a corresponding caption is to be displayed among image frames constituting a moving image. In one embodiment, the face detection may be performed using a skin color method, and the face is detected using a skin color method or a method of finding a contour of a face using a motion-based edge difference image or blinking eyes. Can be used.

Subsequently, it is determined whether the detected feature point of the face is changed to determine a speaker corresponding to each image frame corresponding to the time information (S730). The reason for determining the speaker here is that since other characters other than the speaker may appear in each image frame, not all faces detected in S620 are the speaker's face. This is to determine the speaker who is speaking in the current frame.

In determining the speaker, the face whose feature point is changed in the detected face may be determined as the speaker's face. In this case, the feature point may be an area under the eye of the face, for example, an area around the mouth or the chin. More specifically, when the video frame corresponding to the time information to display the subtitle is n frames and n + 1 frame, the difference between the faces detected in the n frame and n + 1 frame is calculated, and the calculated difference Through this, the face determined to be the part corresponding to the feature point of the face is determined as the speaker's face.

Next, the range of the speaker's face region is determined in each image frame corresponding to the time information on which the subtitle is to be displayed using the speaker's face determined in S730 (S740). In one embodiment, the entire face area generated when the faces detected in each of the image frames overlap each other is determined as the speaker's face area range.

Subsequently, the subtitle may be displayed within a predetermined distance from the range of the speaker's face area determined in S740, for example, a region where the height does not deviate by more than one line from the range of the speaker's face area or the horizontal axis does not deviate by two letters from the range of the speaker's face area. The candidate area is set (S750). In this case, when there is no area satisfying such a condition, one side of the screen (for example, the right or the bottom of the screen) may be set as the candidate area.

Next, a subtitle display area is selected from the candidate areas where subtitles are to be displayed (S760). In an exemplary embodiment, the caption display area may be determined to have the smallest color deviation and the highest saturation among the candidate areas.

Meanwhile, in the above-described embodiment, the position of the subtitle is determined by using the speaker's face detected in each video frame. However, in the modified embodiment, the voice information constituting the video frame corresponding to the corresponding time information is described. By determining the spatial position of the voice (S770), it is also possible to determine the subtitle display area using the spatial position of the determined voice. In this case, the spatial position of the voice may be determined by analyzing the standard of the voice signal and then using the difference of the voice signal of each channel and the frequency spectrum characteristic of the voice signal according to the analyzed voice standard.

Thereafter, finally, the subtitles of the video and the video are output, and the subtitles are output to the subtitle display area determined in step S770 (S780). In this case, the caption may be output in a color having a complementary color relationship with the color of the caption display area.

The above-described subtitle display method of the video may be implemented in a program form that can be performed using various computer means. In this case, the program for performing the subtitle display method of the video may include a hard disk, a CD-ROM, a DVD, a ROM ( ROM), RAM, or flash memory, such as a computer readable recording medium.

Those skilled in the art to which the present invention pertains will understand that the present invention can be implemented in other specific forms without changing the technical spirit or essential features.

Therefore, it is to be understood that the embodiments described above are exemplary in all respects and not restrictive. The scope of the present invention is shown by the following claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

1 is a schematic block diagram of a video reproducing apparatus according to an embodiment of the present invention.

2 is a view showing a method of determining a speaker using whether or not to change a feature point of a detected face.

3 and 4 show caption display candidate regions;

5 is a view illustrating a caption position according to audio information and video information.

6 is a view showing a video display captions according to an embodiment of the present invention.

7 is a flowchart illustrating a caption display method of a video according to an embodiment of the present invention.

<Description of the symbols for the main parts of the drawings>

100: video playback device 110: receiver

120: time information detector 130: image analyzer

140: speaker determination unit 150: subtitle position determination unit

155: voice analysis unit 160: video output unit

Claims (5)

Obtaining time information for displaying the caption from the caption to be displayed in the video; Detecting a face from image frames corresponding to the time information among the image frames constituting the video; Determining a speaker corresponding to the image frames corresponding to the time information by changing whether a feature point of a face detected from each of the image frames corresponding to the time information is changed; Determining a range of the speaker's face region in each of the image frames corresponding to the time information by using the determined speaker's face; And And adaptively determining a region in which the caption is to be displayed according to a range of the speaker's face region and displaying the caption in the determined region. The method of claim 1, wherein the displaying subtitles, Setting a candidate area in which the subtitle is to be displayed among areas within a predetermined distance from the speaker's face area range; And And determining a candidate region having the least color deviation and the highest saturation among the candidate regions as a region where the subtitle is to be displayed. The method of claim 1, wherein in the caption display step, The caption display method of the caption of the video, characterized in that for displaying the subtitle in a color complementary to the color of the area to be displayed. The method of claim 1, Before the subtitle display step, further comprising the step of analyzing the voice information constituting the video to determine the spatial location where the voice is generated, The subtitle display method of the video, characterized in that for determining the area to display the subtitle using the spatial position of the voice together. The method of claim 4, wherein The spatial position of the voice is determined using at least one of the difference between the voice signal output for each channel and the voice frequency band.

Images