KR20120029247A - Apparatus and method for detecting output error of audiovisual information of video contents - Google Patents

Abstract

PURPOSE: An apparatus for detecting output error of audiovisual information of a video content and a method thereof are provided to reduce the generation of sync error of caption and audio data. CONSTITUTION: A voice extraction unit(220) checks the start time and the finish time of the voice signal from the audio signal of a video content. A subtitle detection unit(230) detects a character string from a designated subtitle area of the video content. A subtitle synchronization decision unit(240) compares the start time and a start flag. The caption synchronization determining unit compares the finish time and an ending flag.

Description

Apparatus and method for detecting audio visual information output error of video content {APPARATUS AND METHOD FOR DETECTING OUTPUT ERROR OF AUDIOVISUAL INFORMATION OF VIDEO CONTENTS}

The present invention relates to an apparatus and method for detecting an error in which audiovisual information of video content is abnormally output. More particularly, the present invention relates to a mismatch between a timing at which subtitles and audio are output from video contents and a timing at which video and audio are output. An apparatus and method for detecting the same.

Information transmission media are becoming more diversified and more abundant in a rapidly changing modern society. In the past, information delivery media used a text-oriented expression method, but nowadays, a video media that simultaneously stimulates a user's vision and hearing is used as the most popular means of information transmission.

In particular, as the technology of TV, computer, Internet, etc. has become popular, it is very easy to access information provided in the form of moving pictures. Recently, with the development of video content encoding technology, high-definition video of tens to hundreds of megabytes can be easily accessed.

In addition, video content photographing equipment that can produce such a video has also been developed, such as mobile phone cameras, digital cameras, camcorders, etc. are often used by experts. Accordingly, various video contents such as UCC and movies are continuously produced, and this trend is expected to accelerate further. In addition, data storage media such as hard disk drives, memory cards, etc. are also becoming inexpensive as they become larger in capacity.

In addition, the media playback terminals that can watch video content has also been diversified, including mobile phones, smart phones, PMPs, mobile DMB terminals, laptops, and vehicle navigation systems. In the past, information transmission of video contents was only terrestrial TV broadcasting, but there are various methods such as IPTV, satellite broadcasting, and digital cable broadcasting.

Meanwhile, before serving the produced video content through a medium such as an IPTV, the defect of the video content itself should be investigated. This is because, when there is a subtitle or lip sync error in the video content, the viewer's complaint is directed to the content service subject, not the content producer.

1 is a diagram illustrating a method of detecting an audiovisual output information error of a conventional video content.

As shown in FIG. 1, in general, video content is mostly examined by the human eye in order to detect errors in subtitle information and audio information, which are audiovisual output information. However, it is practically impossible for a company that provides video content to detect all errors in video content that are distributed hundreds of times a day.

In addition, even if the video content is played back quickly to investigate an error, the conventional technology requires a lot of time, manpower, and cost. In addition, since a person detects an error by hand, there is a problem in that the deviation is large according to a person's ability to detect an error and the error is not completely eliminated even after the visual inspection is completed.

In order to solve the above-mentioned problems of the prior art, some embodiments of the present invention provide an apparatus and method for detecting a subtitle display error by detecting the presence or absence of a subtitle in a region where a subtitle of video content is located and extracting a voice.

In addition, some embodiments of the present invention provide an apparatus and method for detecting a voice output error by analyzing a mouth shape of a face of video content and extracting a voice by using face recognition technology.

The objects of the present invention are not limited to the above-mentioned objects, and other objects not mentioned can be clearly understood from the following description.

As a technical means for achieving the above-described technical problem, a first aspect of the present invention is a voice extractor for checking the start time and end time of a voice signal using frequency information from the audio signal of the video content, the video content Detects whether there is a string in a predetermined subtitle area, sets a start flag to the Nth video frame in which the string is first detected, and changes the string of the Nth video frame and the subtitle area from an N + 1th video frame A subtitle detector configured to set an end flag to a video frame in which a character string is not detected for the first time in the subtitle area, and compare the start time with the start flag, and compare the end time with the end flag, respectively, and specify a first predetermined first flag. Determine who has exceeded the caption error setting time A caption display error detection apparatus including a film synchronization determination unit may be provided.

In addition, the second aspect of the present invention is a voice extractor for checking the start time and the end time of the voice signal using the frequency information in the audio signal of the video content, facial recognition of the shape of the mouth of the talker appearing in the video content The image recognition unit which grasps by a technique and checks the start time of the first video frame where the mouth shape is opened, and compares the start time of the audio signal extracted by the voice extractor and the start time of the first video frame checked by the image recognition unit. It is possible to provide a voice sync error detection device including a voice synchronization determination unit for determining whether to exceed a predetermined first voice error setting time.

In order to achieve the above object, a third aspect of the present invention is (a) extracting a speech signal using frequency information from the audio signal of the video content, (b) the start time of the speech signal extracted from the video content And extracting an end time; (c) checking whether a character string exists in a predetermined subtitle region of the N th video frame, and (d) a start flag in the N th video frame when the character string exists in the subtitle region. And setting an end flag in the N + M th video frame in which the character string disappears in the subtitle area, and (e) displaying the subtitle by comparing the start time and end time of the voice signal with the start flag and end flag, respectively. It is possible to provide a caption display error detection method comprising the step of identifying an error.

In addition, a fourth aspect of the present invention is to (a) extracting a speech signal using the frequency information from the audio signal of the video content, (b) to the facial recognition technology in the shape of the speaker's mouth appearing in the N-th video frame Determining the start time of the first video frame in which the mouth shape is opened, and (c) comparing the start time of the voice signal with the start time of the checked first video frame to identify a voice sync error. It may provide a voice sync error detection method comprising a.

BRIEF DESCRIPTION OF THE DRAWINGS The above and other objects, features and advantages of the present invention will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which: FIG.

The present invention may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. It is provided to fully inform the owner of the scope of the invention.

According to one of the problem solving means of the above-described audiovisual output information error detecting apparatus of the present invention, a caption display error and a voice sync error of video content can be detected.

In addition, according to the above-described problem solving means of the present invention, the subtitle display error and the audio sync error of the video content does not require a separate error inspection manpower, or it can reduce the cost by reducing the time required.

In addition, according to the above-described problem solving means of the present invention, it is possible to more accurately detect the subtitle display error and the audio sync error of the video content can be used for the integrity verification and quality verification of the content.

In addition, according to the above-described problem solving means of the present invention, since it can be widely used in video content encoding process, transcoding process, format conversion process currently performed by a video content service company, more video content can be produced and distributed. have.

1 is a diagram illustrating a method of detecting an audiovisual output information error of a conventional video content.
2 (a) is a block diagram illustrating a caption display error detection apparatus according to an embodiment of the present invention, Figure 2 (b) is a view for explaining a voice sync error detection apparatus according to another embodiment of the present invention It is a block diagram.
3 is a block diagram illustrating an apparatus for detecting audiovisual output information error according to another embodiment of the present invention.
4 (a) and 4 (b) are diagrams for describing an area where a subtitle of video content is located according to an embodiment of the present invention.
5 is a diagram for explaining an example in which a caption is included in a video frame itself according to an embodiment of the present invention.
FIG. 6 is a diagram for explaining an example of adding caption to a video frame according to an embodiment of the present invention. FIG.
7 is a diagram for describing a method of detecting an error of a caption according to an embodiment of the present invention.
8 is a diagram for describing a mouth shape recognition method using face recognition technology according to another exemplary embodiment of the present invention.
9 is a diagram for describing a method of detecting an error of an image and a sound of video content according to another exemplary embodiment of the present invention.
10 and 11 are flowcharts illustrating a caption display error detection method according to an exemplary embodiment of the present invention.
12 is a flowchart illustrating a voice sync error detection method according to another embodiment of the present invention.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and like reference numerals designate like parts throughout the specification.

Throughout the specification, when a part is "connected" to another part, this includes not only "directly connected" but also "electrically connected" with another element in between. . In addition, when a part is said to "include" a certain component, which means that it may further include other components, except to exclude other components unless otherwise stated.

In addition, when a part is said to "include" a certain component, it means that it may further include other components, except to exclude other components unless specifically stated otherwise.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings.

2 (a) is a block diagram illustrating a caption display error detection apparatus according to an embodiment of the present invention, Figure 2 (b) is a view for explaining a voice sync error detection apparatus according to another embodiment of the present invention It is a block diagram.

As shown in FIG. 2 (a), the caption display error detecting apparatus 200 according to an exemplary embodiment of the present invention may determine the video content input unit 210, the voice extractor 220, the caption detector 230, and the caption synchronization. It may include a portion 240.

In the present invention, "subtitle display error" means that a subtitle corresponding to a specific voice is not displayed on the screen at an appropriate moment. That is, this may be the case when the subtitles of the internal language or the foreign language of some dialogue spoken by the talker are accidentally omitted while the image of the video content is output.

For example, if the subtitles are not displayed when the mouth of the speaker in the video moves, when the subtitles are not displayed when the voice of the speaker in the video is output, the user of the video content may If everything disappears too quickly to read, the general reading level viewers may have read the subtitles, but the subtitles do not disappear. In addition, subtitles are not fastened or slowed in the video content, but the subtitles themselves are not input, that is, the subtitles are not loaded. In this case, they are separately managed as "no subtitles" without corresponding to the subtitle display error. You may.

Now, when describing each component, the content input unit 210 may receive video content data from a device connected to the caption display error detecting apparatus 200 by wire or wirelessly. The input video content may be stored in a database not shown to implement the present invention.

Here, the video content refers to general video data composed of various types of scenes. In the video content, a dialogue person who speaks a language such as a foreign language user or a Korean language user may appear. In addition, subtitles may be provided in the video content for a specific purpose. For example, Korean subtitles may be provided in each scene so that the hearing impaired person can use the content of the content, or difficult to understand based on a person using a standard language. Korean subtitles may be provided to make it easier to understand the dialect, or appropriately summarized Korean subtitles may be provided when the talker speaks professional or difficult contents.

The voice extractor 220 may extract a voice signal using the frequency information of the video content input by the content input unit 210, and check a start time and an end time of each frame of the video content. The extracted voice data is necessary to confirm whether or not the caption appears at an appropriate timing or occurs with a time difference in the portion where the voice data is output.

If the audio is output but there is no subtitle, it can be classified as no subtitle and managed separately. This separately managed portion may be subjected to the subtitle display error detection process again after the subtitle insertion operation.

Here, the voice is an audio signal extracted from the video content data and may be human voice data. In addition, any video object in which a subject capable of making sounds such as an animal, an alien, and a robot, in addition to a human voice, such as an animal documentary, an animation, and a science fiction movie, can be extracted and used from the object. In the following description, voice data extracted from a person will be described as an example.

In addition, the audio data is extracted by using different points from other data. For example, the voice frequency band from a typical speaker's vocal cords is between about 300 Hz and about 3.5 kHz. Therefore, by extracting a similar signal in this frequency band, it can be used as audio data required in the present invention. The speech extraction method is not limited to a specific method and various methods may be used. In addition, the extracted voice data need not be analyzed what kind of voice it is, but only the time when the voice is present and whether the voice is present.

The caption detector 230 detects whether a character string exists in a predetermined caption region from data of video content, sets a start flag in a first video frame in which a character string is first detected, and a difference from a next video frame to a previous video frame. The caption area may be detected using, and an end flag may be set in a video frame in which no character string exists in the caption area.

Here, the caption area is a portion where the caption is displayed, which can be anywhere, such as the bottom, top, right side, left side of the screen, but can increase the caption recognition rate by setting an area where the caption appears. In this embodiment, various video media such as general video contents, for example, movie film, TV, Blu-ray, DVD, CD, VHS tape, etc. are mainly positioned in the lower quarter of the image, which is a position for displaying subtitles. something to do.

4 (a) and 4 (b) are diagrams for describing an area where a subtitle of video content is located according to an embodiment of the present invention.

4A is an example of a standard-definition screen 410 used in a video graphic display device such as a digital standard television. Since the SD class screen usually has a resolution of 720 x 480, an area about 120 pixels from the bottom of the screen, which is a quarter of the vertical length, may be set as the first subtitle area 415.

4B is an example of a high-definition screen 420 used in a video graphic display device such as a high definition television. Since the HD level screen usually has a resolution of 1920 x 1080, an area of about 270 pixels from the bottom of the screen, which is a quarter of this size vertically, can be set as the second subtitle area 425.

5 is a diagram for explaining an example in which a caption is included in a video frame itself according to an embodiment of the present invention.

5, in the video frame itself, "came breakdown of North Korea arrived in Uljin!" This is the case in which the picture 510 of the letter shape is inserted. As described above, unlike the case where the caption is inserted into the caption region of the video, the video frame itself is an example of another video in the form of caption. Therefore, in such a case, the caption detector 230 may determine that there is no caption when checking whether there is a caption in the caption area, and thus may not detect the caption. In addition, such a letter-shaped picture 510 represents the voice of the talker in the subtitle area, and is different from a subtitle inserted in a large font such as the size of the picture 510. Another example of such a picture may be a phrase that summarizes a person's voice content or situation in an easy and understandable way.

The caption detection unit 230 first determines whether a caption exists in each caption area of FIGS. 4A and 4B, and the caption-shaped picture is a character in the video content itself or a caption separately inserted into the image. Perception can be distinguished. There are a variety of methods for distinguishing subtitles (hereinafter referred to as subtitles) embedded in an image or picture-type pictures (hereinafter referred to as subtitles).

First, it is possible to determine whether an inserted caption or a picture caption is displayed by looking at a position where letters appear on the screen. For example, it may be determined whether the caption exists only at the bottom of the screen, and if the caption is in the middle or top of the screen rather than the bottom, it may be determined as a picture caption. In addition, even if the subtitle is present at the bottom, it is possible to determine more precisely by further checking whether the letters are arranged neatly or randomly arranged.

In addition, it is possible to determine whether the caption is a subtitle or a picture caption by the size of the text displayed on the screen. For example, in case of SD class, less than 25 x 25 pixels per character, and in case of HD class, less than 50 x 50 pixels can be viewed as embedded subtitles. However, if the font on the screen is too small or the size is not constant, it can be regarded as not an embedded subtitle.

In addition, it is possible to determine whether the text displayed on the screen is inserted with subtitles or picture captions depending on how much temporal similarity, that is, correlation. That is, the character displayed in the caption area on the screen must be at the same position without moving for at least 1 second (30 frames) in time to be determined as the inserted caption.

In addition, it is possible to determine whether the text displayed on the screen is inclined in the horizontal direction to determine whether the caption is an inserted caption or a picture caption. In general, since the inserted subtitles appear repeatedly in a predetermined subtitle area, the letters are arranged in a direction perpendicular to the horizontal direction so that the viewer can easily view the subtitles.

The various criteria described above may be arbitrarily determined by the producer according to the characteristics of the video content and may not be applied uniformly to all the video contents.

In addition, in the present invention, a technique for detecting a caption region may use various methods and techniques such as edge map usage, stroke filtering, and a method of using color and luminance values, and is not limited to a specific caption region detection technique.

FIG. 6 is a diagram for explaining an example of adding caption to a video frame according to an embodiment of the present invention. FIG.

As shown in FIG. 6, the general embedded subtitle appears at the position of the subtitle area 425 at the bottom of the screen. For example, the size of the inserted subtitle is small. Usually, the size of the subtitle is 25 x 50 pixels in width x height.

The subtitle synchronization determiner 240 compares the start time and the end time of the voice signal extracted by the voice extractor 220 with the start flag and the end flag of the character string of the subtitle area detected by the subtitle detector 230, respectively. It may be determined whether the first subtitle error setting time is exceeded.

First, in order to detect a subtitle error of video content, character detection is performed in a predetermined portion of the lower part of the image, for example, a quarter of a vertical size, and the subtitle error is compared by mutual comparison with voice extracted from audio data. An algorithm for detecting can be used. Figure 7 specifically illustrates such a method.

7 is a diagram for describing a method of detecting an error of a caption according to an embodiment of the present invention.

7 illustrates a method of comparing the start and end of the subtitle detected by the subtitle detector 230 with the voice signal extracted by the voice extractor 220. The start flag may be set in the first frame where the caption area is detected, and the caption area may be detected from the next frame by using a difference from the previous image. When the caption region ends, an end flag is set to use an algorithm for comparing the start flag and the end flag of the voice signal extracted from the voice extractor 220, respectively.

The caption displayed between the caption start flag 710 and the caption end flag 720 is the same as the time at which the audio is output, and thus shows an example of the section A, that is, the section in which the caption is normally displayed. That is, it is a section indicating that the start and end of the voice are almost identical to the start end of the subtitle normally. To distinguish between normal subtitles and error subtitles, an error limit of the time and end time of the audio file may be set in seconds. For example, it can be set at a difference of about 0.5 seconds (15 frames).

In this case, the time at which the start and end points of the voice signal can be determined can be known using a PTS (Presentation Time Stamp) of the audio file. In addition, the time calculation method of the start and end of the speech signal that can be applied in the present invention is not limited to a specific method and various methods may be used.

Sections B and C each represent an example of a caption error section in which an error caption appears.

Section B represents a case where the audio signal is output but there is no subtitle. Accordingly, the caption synchronization determiner 240 may determine that the caption error is detected if the time when the caption is not displayed despite the voice signal continues beyond the predetermined threshold. In this case, the threshold value, that is, the threshold time of the time without the subtitle display may be set to, for example, about 2 seconds, which may be set to a different time according to the type of video content. That is, when a predetermined time difference occurs between the start of the extracted voice signal and the start flag of the subtitle region (for example, 2 seconds), an algorithm for determining that the subtitle display error is present without the subtitle display but with the subtitle display may be used.

In contrast to section B, section C has no audio signal and subtitles are displayed. The audio signal is not output even when the caption is displayed between the caption start flag 730 and the caption end flag 740. This case can also be treated as a subtitle error. Such subtitle errors occur with a probability of less than about 1% in one video content, and the subtitles correspond to one or two measures of the subtitles in which the error occurs. Therefore, it may be efficient in some cases to use an algorithm that has a higher proportion of caption error in the section B, which is more frequent. This is because more subtitle errors may be detected than actual subtitle errors. Therefore, in the present invention, an algorithm for detecting a caption error in section B may be included, and whether or not the caption error in section C is included according to circumstances.

Next, a voice sync error detection apparatus 250 according to an embodiment of the present invention will be described with reference to FIG. 2 (b). The same reference numerals are given to the components already described in FIG. 2 (a), and redundant descriptions are omitted.

As shown in FIG. 2 (b), the audio sync error detection apparatus 250 according to another embodiment of the present invention may include a video content input unit 210, a voice extractor 220, an image recognition unit 235, and a voice synchronization. The determination unit 245 may be included.

In the present invention, the voice sync error means that the voice is not output as a sound at an appropriate moment, and corresponds to a lip synchronization error. For example, if the voice of the speaker in the video is not output at the same time as the voice of the speaker is moving, if the voice is output before or after the mouth of the speaker in the video, the voice of the speaker in the video is not output. If not, the voice may be output even though the speaker in the video does not move.

When a voice sync error occurs, there is a mismatch between the video and the audio in the video content, so if the sync is lost in one part, the video and audio appear later. That is, it continues from the moment when synchronization is lost until the end of the video content. In general, a voice sync error is not easy to detect, but it is possible to detect an error if the timing at which the voice starts to be output (the instant of conversation start) can be accurately understood.

The content input unit 210 may receive video content data from a device connected to the voice sync error detection device 250 by wire or wirelessly, and the voice extractor 220 may be a frequency of the video content input by the content input unit 210. The audio signal may be extracted using the information, and a start time and an end time of each frame of the video content may be checked.

The image recognizing unit 235 may recognize the shape of the talker's mouth using face recognition technology from the data of the video content, and check the start time and the end time of the video frame in which the mouth shape first opens. Therefore, in order to determine whether the voice is properly output from the image data, a procedure of analyzing the face of the dialogue person of the image data is required. Referring to Figure 8 will be described in this regard.

8 is a view for explaining a mouth shape recognition method using face recognition technology according to another embodiment of the present invention.

As shown in FIG. 8, the image recognition unit 235 may recognize the face of the talker in the entire image by distinguishing it from a background or a body part, and may recognize the shape of the mouth part of the face. The method of distinguishing the face and the mouth from other images may be determined by comparing the graphic in the image with the human face template (for example, the shape of a circle, eyes, nose, and mouth) and the general skin color of the person. Since such a technique is widely used in digital cameras, CCTVs, facial recognition software, etc., various methods can be used in the present invention without being limited to a specific facial recognition technique.

For example, the face 810 of the talker is recognized and the mouth 820 of the talker is recognized through face recognition technology. You can see the moment when the mouth of the talker is closed in the first shape and the mouth opens. The timing at which the voice signal extracted from the audio data corresponding to the conversation start timing from the frame where the mouth is opened can be compared with each other. 9 shows a method of comparing each timing with each other.

9 is a diagram for describing a method of detecting an output error between a video and audio of video content according to another embodiment of the present invention.

The voice synchronization determiner 245 compares the start time and the end time of the voice signal extracted by the voice extractor 220 with the start time and end time of the video frame detected by the image recognition unit 230, respectively. 1 It may be determined whether the voice error setting time is exceeded.

As described above, the face and the mouth are respectively recognized in the image, and the time at which the mouth starts to open after the first letter is detected, and the start time of the extracted voice signal is detected to detect a predetermined reference time (for example, 165 msec or 5). Frame), you can create an algorithm that determines a voice sync error.

9 illustrates an error between the start time of the extracted voice signal and the talk start time of the talker. If this error is ahead or behind a predetermined reference time, that is, the threshold time, it can be regarded as a voice sync error. Here, the threshold time may be, for example, plus minus 165 msec (5 frames).

The interval D 910 is a case where the difference between the start time of the extracted voice and the talk start time exceeds the aforementioned threshold time and is less than a predetermined time, for example, less than 2 seconds (60 frames). In this case, the voice synchronization determiner 245 determines that the voice sync error. On the other hand, in the case of the interval E 920, the difference between the start time of the extracted voice and the start time of the conversation is more than 2 seconds (60 frames), which is too large to be referred to as a voice sync error. Therefore, in this case, the voice synchronization determiner 245 may determine that the extracted voice signal is the voice of another person other than the conversation person rather than the probability of the voice sync error.

In addition, since there is a possibility that the voice synchronization determiner 245 incorrectly recognizes the conversation start time of the conversation, the voice synchronization determination unit 245 may not include the voice sync error when the difference is more than 2 seconds. That is, if the first time the mouth starts to move, i.e., the start time of the conversation and the start time of the extracted speech signal differ by more than a predetermined reference time, e. It can also be implemented.

 In addition, the above-described facial recognition technology has a limited recognition rate because the angle of face recognition is limited, and in particular, the recognition rate is further deteriorated when the talker starts a conversation. However, the voice sync error is not a one-time error but occurs once and affects all voice signals until the end of playback of the video content, so that all voice syncs are not matched. Therefore, even if the recognition rate decreases when the talker starts the conversation, the problem may be solved by recognizing the shape of the face and mouth in the image that is easy to recognize the face at any time from the time of the voice sync error.

3 is a block diagram illustrating an audiovisual output information error detecting apparatus 300 according to another exemplary embodiment of the present invention.

When the caption display error detection apparatus 200 described above and the voice sync error detection apparatus 250 are combined, the audio visual output error detection apparatus 300 as shown in FIG. 3 is obtained. The video content received by the content input unit 210 is divided into voice data, caption data, and image data, and analyzed by the voice extractor 220, the caption detector 230, and the image recognizer 235, respectively. The caption synchronization determiner 240 determines whether the caption display is appropriate for the speech using respective data extracted by the speech extractor 220 and the caption detector 230, and the speech synchronization determiner 245 uses the speech extractor ( The audio sync is determined according to the image using the respective data extracted by the image recognition unit 220 and 220. The subtitle synchronization determiner 240 and the voice synchronization determiner 245 may output corrected video content capable of outputting a subtitle and an image at an appropriate timing based on a commonly used voice signal. Detailed description of each component is the same as described above, and thus duplicated description is omitted.

10 and 11 are flowcharts illustrating a caption display error detection method according to an exemplary embodiment of the present invention.

First, in operation S1010, the caption display error detecting apparatus 200 receives video content and extracts a voice signal from the input video content in operation S1020.

In operation S1020, the caption display error detecting apparatus 200 starts playback from the Nth frame of the video content. At this time, the caption display error detecting apparatus 200 checks whether caption is detected in the caption region of the corresponding frame. If the caption is not detected, it moves to the N + 1 th frame, which is the next frame (S1045). If a caption is detected, a flag indicating that the caption is started is set to 1 (S1050).

In step S1060, the frame in which the subtitle start is set is compared with the frame of which the frame number is incremented by one. If the subtitles are not changed in the subtitle area, the next frame is moved and the frame is compared with the frames which are increased by one. This comparison method can be repeated to detect subtitles. Subtitles are usually conversational content, so they can last for a few seconds, or 30 frames per second for typical video content. Once detected, subtitles usually remain the same for 30 to 150 frames.

The method for detecting the presence or absence of subtitles as compared to the continuous frames described above can be detected using various algorithms after obtaining the difference between the nth subtitle area and the N + 1th subtitle area.

In operation S1080, the caption display error detecting apparatus 200 sets a flag to 1 to indicate that the caption is over after detecting that the caption is over. Through the above-described repetition process, the end flag is terminated at the N + M th video frame in which the character string disappears in the subtitle area. Here, a flag indicating the start of the subtitle and the end of the subtitle is necessary for comparing the time of the start and the end of the audio extracted from the audio.

In operation S1090, the caption display error detecting apparatus 200 determines whether the corresponding frame is the last frame, and if it is the last frame, terminates playback of the video content (S1100), otherwise moves to the next frame again (S1095). It is determined whether a subtitle is detected in the subtitle area of the.

The voice signal extracted in step S1110 and the flagged video content are simultaneously started. Then, it is checked whether a flag indicating that there is a subtitle within a predetermined time after the reproduction of the audio signal starts (S1120). If there is no setting, a synchronization error is checked at the corresponding position (S1135), and playback continues to check whether the caption flag is set within a predetermined time after the playback of the audio signal is started.

In operation S1130, the caption display error detecting apparatus 200 determines whether there is a voice start signal within a predetermined time after detecting the caption flag when the caption flag is set within a predetermined time after the signal is started playing. . If the audio start signal is within the corresponding time, the synchronization error is checked at the corresponding position (S1135). If it cannot be determined, playback continues and the process returns to step S1120. If it is determined, playback is terminated.

12 is a flowchart illustrating a voice sync error detection method according to another embodiment of the present invention.

In operation S1210, the voice sync error detecting apparatus 250 receives video content, and extracts a voice signal from the video content (S1220). Then, the extracted voice signal and the video content start to be played simultaneously (S1230), and the time point at which the talker displayed on the screen starts talking, that is, the start time of the conversation, is checked (S1240).

In step S1250, it is checked whether the difference between the start time of the conversation and the start time of the voice signal exceeds a predetermined first voice error setting time. If the first voice error setting time is exceeded, it is again checked whether the difference between the start time of the conversation and the start time of the voice signal is within the predetermined second voice error setting time (S1260).

In operation S1270, when a difference between the conversation start time and the start time of the voice signal occurs within the second voice error setting time, the voice sink error detecting apparatus 250 checks for a synchronization error at the corresponding position. Then, playback of the video content is stopped (S1280).

In operation S1290, the voice sink error detecting apparatus 250 determines whether to finish playing the video content when the difference between the start time of the conversation and the start time of the voice signal exceeds a predetermined first voice error setting time. If the playback of the video content is not completed, playback continues to be performed again from step S1240, and if it is completed, execution ends.

In addition, although the audiovisual output information error detection method for executing both the caption display error detection method and the voice sync error detection method is not shown in the drawing, the caption display error detection method and the voice sync error detection method are induced and combined respectively. Since the method is obvious to those skilled in the art, a description thereof will be omitted.

Each embodiment of the present invention can also be implemented in the form of a recording medium including instructions executable by a computer, such as a program module executed by the computer. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. In addition, computer readable media may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Communication media typically includes computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, or other transmission mechanism, and includes any information delivery media.

Although the methods and apparatus of the present invention have been described in connection with specific embodiments, some or all of their components or operations may be implemented using a computer system having a general purpose hardware architecture.

The foregoing description of the present invention is intended for illustration, and it will be understood by those skilled in the art that the present invention may be easily modified in other specific forms without changing the technical spirit or essential features of the present invention. will be. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is shown by the following claims rather than the above description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present invention. do.

200: caption display error detection device
210: content input unit
220: voice extraction unit
230: Caption detection unit 235: Image recognition unit
240: subtitle synchronization determiner 245: voice synchronization determiner
250: voice sync error detection device
300: audio visual output error detection device

Claims (18)

An apparatus for detecting a subtitle display error of video content, the apparatus comprising:
A voice extraction unit which checks a start time and an end time of the voice signal using frequency information from the audio signal of the video content;
Detect whether the video content has a character string in a predetermined subtitle area, set a start flag to an N th video frame in which the character string is first detected, and start the N + 1 th video frame from the N th video frame and the subtitle area. A caption detector for detecting whether a string is changed and setting an end flag to a video frame in which the string is not detected for the first time in the caption area;
A caption synchronization determining unit comparing the start time with the start flag and comparing the end time with the end flag to determine whether the first caption error setting time is exceeded;
Subtitle display error detection device comprising a.
The method of claim 1,
And the caption detection unit classifies the caption area according to a position where the character string is displayed.
The method of claim 1,
And the caption detection unit classifies the caption area according to the size of the character string.
The method of claim 1,
And the caption detection unit classifies the caption area based on a temporal similarity in which the character string is displayed.
The method of claim 1,
And the caption detection unit classifies the caption area by an angle at which the character string is inclined with respect to a horizontal direction.
The method of claim 1,
The caption synchronization determining unit may determine that the caption error is present but the character string does not exist when the time difference between the start time of the sound signal and the start flag of the caption area exceeds the first caption error setting time. Subtitle display error detection device.
An apparatus for detecting a voice sync error of video content,
A voice extraction unit which checks a start time and an end time of the voice signal using frequency information in the audio signal of the video content;
An image recognition unit which grasps the shape of the mouth of the talker appearing in the video content by a face recognition technology and checks the start time of the first video frame in which the mouth shape is opened;
The voice synchronization determining unit which compares the start time of the voice signal extracted by the voice extractor and the start time of the first video frame checked by the image recognition unit to determine whether the first predetermined voice error setting time is exceeded.
Voice sync error detection device comprising a.
The method of claim 7, wherein
And the voice synchronization determining unit does not determine a voice error when a time difference between the start time of the voice signal and the start time of the first video frame exceeds the first voice error setting time.
A method of detecting subtitle display error of video content, the method comprising:
(a) extracting a voice signal from the audio signal of the video content by using frequency information;
(b) extracting a start time and an end time of the audio signal extracted from the video content;
(c) checking whether a string exists in a predetermined subtitle region of the Nth video frame,
(d) setting a start flag in the N-th video frame when the character string exists in the caption area, and setting an end flag in the N + M-th video frame in which the character string disappears in the caption area;
(e) checking a caption display error by comparing a start time and an end time of the voice signal with the start flag and the end flag, respectively;
Subtitle display error detection method comprising a.
The method of claim 9,
In step (d),
(d1) checking a synchronization error at a position without the start flag within a first predetermined subtitle error setting time after a start time of the voice signal;
Subtitle display error detection method comprising a.
The method of claim 9,
In step (d),
(d2) checking for a synchronization error at a position without the end flag within a first predetermined subtitle error setting time after a start time of the voice signal;
Subtitle display error detection method comprising a.
The method of claim 9,
And the caption detection unit classifies the caption area according to a position where the character string is displayed.
The method of claim 9,
And the caption detection unit classifies the caption area according to the size of the character string.
The method of claim 9,
The caption detection unit, the caption display error detection method for classifying the caption area by the temporal similarity (correlation) in which the character string is displayed.
The method of claim 9,
And the caption detection unit classifies the caption area by an angle at which the character string is inclined relative to a horizontal direction.
In the method for detecting a voice sync error of video content,
(a) extracting a voice signal from the audio signal of the video content by using frequency information;
(b) identifying the shape of the mouth of the talker appearing in the N-th video frame by facial recognition technology and checking the start time of the first video frame in which the mouth shape occurs;
(c) checking a voice sync error by comparing the start time of the voice signal with the start time of the checked first video frame.
Voice sync error detection method comprising a.
17. The method of claim 16,
(c) step,
(c1) determining whether a difference between the conversation start time and the start time of the voice signal is less than or equal to a second predetermined voice error setting time when the first voice error setting time is exceeded after the start time of the voice signal.
Voice sync error detection method comprising a.
17. The method of claim 16,
(c1) step,
(c-1) checking a synchronization error at a location not exceeding the video content if a difference between the conversation start time and the start time of the voice signal does not exceed the second voice error setting time;
Voice sync error detection method further comprising.

Images