KR102311386B1 - Method and Apparatus for Lip Synchronization Control of Audio/Visual Signals - Google Patents

Abstract

A lip sync control method is provided. The method comprises the steps of decoding a compressed video signal to provide a decompressed video signal, causing the decompressed video signal to be output to a TV set together with a compressed audio signal, by decoding the compressed audio signal providing a decompressed audio signal, sampling the decompressed audio signal to provide reference audio signal samples, providing loopback audio signal samples via a sound output output from the TV set, and the reference and controlling the timing at which the compressed video signal is decoded or the timing at which the decompressed video signal is output to the TV set based on comparing the audio signal samples and the loopback audio signal samples.

Description

Method and Apparatus for Lip Synchronization Control of Audio/Visual Signals

The present invention relates to an AV (Audio/Visual) signal processing technology.

Recently, in line with the development of multimedia technology, the demand for the development of realistic audio technology for reproducing realistic audio is rapidly increasing. In response to these demands, research on a 3D sound system capable of providing realistic audio effects is being actively conducted. One of the 3D sound systems created as a result of this research is Dolby Atmos. Dolby Atmos is a three-dimensional sound system that immerses the audience by maximizing the realism and effect of movie scenes by allowing you to listen to movie scenes with realistic surround sound that gives a sense of space. Dolby Atmos is a 360° stereophonic sound system based on the concept of an audio object that is independently positioned and mastered in three-dimensional space. Worldwide, there are about 1,300 content mastered with Dolby Atmos, which is a considerable number, and in the case of Netflix, about three-quarters of new content is mastered with Atmos.

An object of the present invention is to correct a disparity between audio and video when content is reproduced so that they can be synchronized.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

In one aspect, a lip sync control method is provided. The method comprises the steps of decoding a compressed video signal to provide a decompressed video signal, causing the decompressed video signal to be output to a TV set together with a compressed audio signal, by decoding the compressed audio signal providing a decompressed audio signal, sampling the decompressed audio signal to provide reference audio signal samples, providing loopback audio signal samples via a sound output output from the TV set, and the reference and controlling the timing at which the compressed video signal is decoded or the timing at which the decompressed video signal is output to the TV set based on comparing the audio signal samples and the loopback audio signal samples.

In one embodiment, the step of converting a sound output output from the TV set into an electrical signal to provide loopback audio signal samples comprises: converting a sound output output from the TV set into an electrical signal, and converting the electrical signal to A/ D transform and provide samples of the loopback audio signal.

In an embodiment, the A/D conversion of the electrical signal to provide the loopback audio signal samples includes: A/D converting the electrical signal to provide digital samples and noise filtering on the provided digital samples and providing the loopback audio samples.

In one embodiment, A/D converting the electrical signal to provide loopback audio signal samples and sampling the decompressed audio signal to provide reference audio signal samples are performed in synchronization with each other.

In one embodiment, the reference audio signal samples and the loopback audio signal samples are samples within the same time window.

In one embodiment, controlling the timing at which the compressed video signal is decoded or the timing at which the decompressed video signal is output to the TV set based on comparing the reference audio signal samples and the loopback audio signal samples The step is to calculate a disparity value between the reference audio signal samples and the loopback audio signal samples, and based on the disparity value, the decompressed video signal is output to the TV set or the compressed video signal is and delaying presentation of the decoded and decompressed video signal.

In one embodiment, calculating a disparity value between the reference audio signal samples and the loopback audio signal samples, and based on the disparity value, the decompressed video signal is output to the TV set or the compressed video Delaying a signal from being decoded and presented as a decompressed video signal comprises: determining a delay value based on the disparity value; and outputting the decompressed video signal to the TV set or the compressed and causing the decoded video signal to be decoded and provided as a decompressed video signal delayed by the delay value.

In an embodiment, the delay value is equal to the disparity value.

In an embodiment, the delay value is a value obtained by subtracting a processing delay value from the disparity value.

In one embodiment, the method comprises: receiving a Dolby Atmos signal comprising the compressed video signal and the compressed audio signal; and converting the Dolby Atmos signal to the compressed video signal and the compressed audio signal. It further comprises the step of separating into

In an embodiment, the decoding of the compressed video signal to provide a decompressed video signal may include synchronizing the provided decompressed audio signal with reference to time information included in a header of the compressed audio signal. and providing the decompressed video signal accordingly.

In one embodiment, the TV set comprises a TV.

In one embodiment, the TV set includes a TV and a loudspeaker connected to the TV.

In another aspect, a set-top box is provided. The set-top box is a demultiplexer configured to demultiplex a signal including a compressed video signal and a compressed audio signal to output a compressed video signal to a first terminal and output a compressed audio signal to a second terminal - the second terminal is connected to an output terminal for connection with a TV - an audio decoder configured to decode the compressed audio signal output to the second terminal to output a decompressed audio signal, the compression output to the first terminal A delay generation unit configured to receive and output a video signal as it is or with a delay, a video decoder configured to decode the compressed video signal output from the delay generation unit and output a decompressed video signal to the output terminal, the TV a sound wave converter configured to receive a sound wave output from the , and provide loopback audio signal samples, and sample the decompressed audio signal to provide reference audio signal samples, the reference audio signal samples and the loopback audio signal samples and a delay value determiner configured to determine and output a delay value based on . The delay generator may be further configured to receive and output the compressed video signal as it is, and to receive the compressed video signal in response to receiving the delay value from the delay value determiner and delay the output by the delay value. have.

In one embodiment, the audio decoder is a software-based Atmos audio decoder.

In one embodiment, the sound wave converter comprises a microphone configured to convert a sound wave output from the TV into an electric signal, and the sound wave converter is further configured to A/D convert the electric signal to provide loopback audio signal samples .

In an embodiment, the set-top box further comprises a noise removal filter unit configured to remove noise filter the loopback audio signal samples.

In one embodiment, the reference audio signal samples and the loopback audio signal samples are samples within the same time window.

In an embodiment, the delay value determiner is further configured to calculate a disparity value between the reference audio signal samples and the loopback audio signal samples and determine the delay value based on the disparity value.

In an embodiment, the audio decoder, the sound wave converter, and the delay value determiner are operated such that the delay generator is deactivated in response to receiving the delay value from the delay value determiner.

In another aspect, a set-top box is provided. The set-top box is a demultiplexer configured to demultiplex a signal including a compressed video signal and a compressed audio signal to output a compressed video signal to a first terminal and output a compressed audio signal to a second terminal - the second terminal is connected to an output terminal for connection with a TV - an audio decoder configured to decode the compressed audio signal output to the second terminal to output a decompressed audio signal, the compression output to the first terminal A video decoder configured to decode the compressed video signal to output a decompressed video signal, a delay generator configured to receive the decompressed video signal and output it as it is or with a delay to the output terminal, and receive a sound wave output from the TV a sonic converter configured to provide loopback audio signal samples by: sampling the decompressed audio signal to provide reference audio signal samples, and determining a delay value based on the reference audio signal samples and the loopback audio signal samples It may include a delay value determiner configured to determine and output. The delay generator receives the decompressed video signal and outputs it as it is, and in response to receiving the delay value from the delay value determiner, receives the decompressed video signal and delays the output by the delay value. can be

In one embodiment, the audio decoder is a software-based Atmos audio decoder.

In one embodiment, the sound wave converter comprises a microphone configured to convert a sound wave output from the TV into an electric signal, and the sound wave converter is further configured to A/D convert the electric signal to provide loopback audio signal samples .

In an embodiment, the set-top box further comprises a noise removal filter unit configured to remove noise filter the loopback audio signal samples.

In one embodiment, the reference audio signal samples and the loopback audio signal samples are samples within the same time window.

In an embodiment, the delay value determiner is further configured to calculate a disparity value between the reference audio signal samples and the loopback audio signal samples and determine the delay value based on the disparity value.

In an embodiment, the audio decoder, the sound wave converter, and the delay value determiner are operated such that the delay generator is deactivated in response to receiving the delay value from the delay value determiner.

According to embodiments of the present invention, there is a technical effect of providing a more realistic audio effect to a viewer by synchronizing the disparity between audio and video when playing Dolby Atmos content.

1 is a view showing a first embodiment of a block diagram of a set-top box having a lip sync control function.
FIG. 2 is a diagram illustrating an embodiment of a block diagram of a TV set that can be connected to the set-top box of FIG. 1 .
3 is a diagram illustrating a second embodiment of a block diagram of a set-top box having a lip sync control function.
FIG. 4 is a diagram for conceptually explaining a process of calculating a disparity value between reference audio signal samples and loopback audio signal samples in the delay value determiner of FIG. 1 .
5 is a diagram illustrating a first embodiment of a flowchart for explaining a lip sync control method performed in a set-top box.
6 is a diagram illustrating a second embodiment of a flowchart for explaining a lip sync control method performed in a set-top box.

Advantages and features of the present invention and a method of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in a variety of different forms, and these embodiments merely allow the disclosure of the present invention to be complete and those of ordinary skill in the art to which the present invention pertains. It is provided to fully inform the person of the scope of the invention, and the present invention is only defined by the scope of the claims.

The terms used herein are used only to describe specific embodiments and are not intended to limit the present invention. For example, a component expressed in a singular should be understood as a concept including a plurality of components unless the context clearly means only the singular. In addition, in the specification of the present invention, terms such as 'comprise' or 'have' are only intended to designate that the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification exist, and such The use of the term does not exclude the possibility of the presence or addition of one or more other features or numbers, steps, operations, components, parts, or combinations thereof. Also, in the embodiments described in this specification, a 'module' or a 'unit' may mean a functional part that performs at least one function or operation.

In addition, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the specification of the present invention, it should be interpreted in an ideal or excessively formal meaning. doesn't happen

Hereinafter, embodiments of the present invention will be described in more detail with reference to the accompanying drawings. However, in the following description, if there is a risk of unnecessarily obscuring the gist of the present invention, detailed descriptions of well-known functions or configurations will be omitted.

1 is a view showing a first embodiment of a block diagram of a set-top box having a lip sync control function. It should be recognized that the components shown in FIG. 1 do not reflect all functions of the set-top box and are not essential, so the set-top box may include more components or fewer components than the illustrated components. .

As shown in FIG. 1 , the set-top box 100 decodes an AV signal in which compressed video signals and compressed audio signals are multiplexed, for example, Dolby Atmos content. and a demultiplexer 110 configured to separate the compressed video signal and the compressed audio signal from each other by multiplexing. The demultiplexer 110 may output a compressed video signal to the first terminal 112 and output a compressed audio signal to the second terminal 113 . The second terminal 113 may be connected to an output terminal 123 of the set-top box 100 for connection to a TV set. Accordingly, the compressed audio signal is output through the output terminal 123 of the set-top box 100 to be input to the TV set. In one embodiment, the output terminal 123 is a High Definition Multimedia Interface (HDMI) terminal. The set-top box 100 may further include an audio decoder 115 configured to decode the compressed audio signal output to the second terminal 113 and output decompressed audio signals. . In one embodiment, the audio decoder 115 is a software-based Atmos audio decoder. The set-top box 100 may further include a delay generator 117 configured to receive the compressed video signal output to the first terminal 112 and output it as it is or with a delay. The set-top box 100 is a video decoder 120 configured to decode the compressed video signal output from the delay generator 117 and output decompressed video signals to the output terminal 123 . may further include. In one embodiment, video decoder 120 is a video decoder conforming to H.264, H.265, or High Efficiency Video Coding (HEVC) standards. The decompressed video signal may be output through the output terminal 123 and input to the TV set. The video decoder 120 decompresses in synchronization with the decompressed audio signal output from the audio decoder 115 with reference to time information included in a header of the compressed audio signal input to the audio decoder 115 . It can be operated to output a video signal. Although the signal output from the set-top box 100 is a decompressed video signal and a compressed audio signal that are out of synchronization with each other, the audio in the set-top box 100 is performed by the above-described synchronization control operation by the video decoder 120 . As a result, the decoder 115 outputs a decompressed audio signal synchronized with the decompressed video signal.

The set-top box 100 includes a microphone 130 configured to convert a sound output, that is, a sound wave output from a TV set that receives the decompressed video signal and the compressed audio signal output from the set-top box 100, that is, into an electrical signal. may include In one embodiment, the microphone 130 is a voice recognition microphone. The set-top box 100 may further include an A/D converter 135 configured to A/D convert an electrical signal to provide loopback audio signal samples. The microphone 130 and the A/D converter 135 may collectively constitute a sound wave converter. The loopback audio signal samples are signal samples collected by the TV set using the compressed audio signal received from the set-top box 100 and collecting the audio output emitted through a loud speaker in the set-top box 100 again. can be comprehended. In one embodiment, the A/D converter 135 may be operated to digitally sample the input electrical signal for a predetermined time period. The set-top box 100 may further include a noise removal filter unit 140 connected to the A/D converter 135 and configured to remove noise and filter the loopback audio signal samples. In one embodiment, the noise removal filter unit 140 is a digital bandpass filter that passes the audible frequency band.

The set-top box 100 may further include a delay value determiner 145 configured to provide reference audio signal samples by sampling the decompressed audio signal output from the audio decoder 115 . In an embodiment, the delay value determiner 145 may be operated to digitally sample the decompressed audio signal for a predetermined time period. In an embodiment, the delay value determiner 145 samples the decompressed audio signal in synchronization with sampling by the A/D converter 135 . For this embodiment the reference audio signal samples and the loopback audio signal samples are samples within the same time window. The delay value determiner 145 may be further configured to determine and output a delay value based on the reference audio signal samples and the loopback audio signal samples. In an embodiment, the delay value determiner 145 may be further configured to calculate a disparity value between the reference audio signal samples and the loopback audio signal samples and determine the delay value based on the disparity value.

Referring to FIG. 4 , which is a diagram for conceptually explaining a process in which the delay value determiner 145 calculates a disparity value between the reference audio signal samples and the loopback audio signal samples, the delay value determiner 145 may include As described above, a difference (T′ - T) between a sample time (T′) of an arbitrary sample among loopback audio signal samples and a sample time (T) of a corresponding reference audio signal sample may be calculated as a disparity value. In an embodiment, the delay value determining unit 145 is configured to determine the disparity value by cross-correlation between the reference audio signal samples and the loopback audio signal samples. Since the reference audio signal samples are signal samples obtained by sampling the decompressed audio signal synchronized with the decompressed video signal output from the set-top box 100, the TV set transmits the audio output emitted through the loudspeaker to the set-top box ( 100), the disparity calculated between the loopback audio signal samples and the reference audio signal samples, which are signal samples collected again, may be considered to reflect the disparity between the video output and the audio output actually reproduced in the TV set. In an embodiment, the delay value determining unit 145 is configured to determine the disparity value as the delay value. In an embodiment, the delay value determining unit 145 is configured to determine a value obtained by subtracting a processing delay value from a disparity value as a delay value. In one embodiment, the processing delay value is a delay that may be generated while passing through the microphone 130 , the A/D converter 135 , and the noise removal filter unit 140 , and may be generated by the delay value determiner 145 . It can be determined a priori taking into account the delay.

The delay generator 117 receives the compressed video signal output from the demultiplexer 110 and outputs it as it is, and receives the compressed video signal in response to the delay value input from the delay value determiner 145 and receives the compressed video signal as much as the delay value. It may be further configured to output with a delay. In an embodiment, the audio decoder 115 may be operated such that the delay generator 117 is deactivated in response to receiving a delay value from the delay value determiner 145 . In this embodiment, the video decoder 120 stops the synchronization control operation with the audio decoder 115 and decodes the compressed video signal inputted with delay by the delay value from the delay generator 117 to output the decompressed video signal. This means that the decompressed video signal is output with that much delay, and as a result, the time difference between the video output and the audio output output from the TV set can be compensated. In an embodiment, the A/D converter 135 , the noise removal filter unit 140 , and the delay value determiner 145 also delay the delay generator 117 in receiving the delay value from the delay value determiner 145 . It can be actuated to be deactivated in response.

FIG. 2 is a diagram illustrating an embodiment of a block diagram of a TV set that can be connected to the set-top box of FIG. 1 . It should be appreciated that the components shown in FIG. 2 also do not reflect all functions of the TV set and are not essential, so that the TV set may include more or fewer components than the illustrated components. .

As shown in FIG. 2 , the TV set 200 includes an input terminal 210 for connection to the set-top box 100 . In one embodiment, the input terminal 210 is an HDMI terminal. The TV set 200 may further include a TV panel 220 connected to the input terminal 210 . In one embodiment, the TV panel 220 is a liquid crystal display (LCD) panel, organic light emitting diode (OLED), such as a twisted nematic (TN) panel, an in-plane switching (IPS) panel, and a vertical alignment (VA) panel. It may be a panel or an Active Matrix Organic Light Emitting Diode (AMOLED) panel, but it should be recognized that the type of the TV panel 220 is not limited thereto. The TV set 200 may further include an audio decoder 230 connected to the input terminal 210 . The audio decoder 230 may be configured to output a decompressed audio signal by decoding the compressed audio signal input from the input terminal 210 . The TV set 200 may further include a D/A converter 240 connected to the audio decoder 230 . The D/A converter 240 may be configured to convert a decompressed audio signal that is a digital signal into an analog signal. The TV set 200 may further include a loudspeaker 250 connected to the D/A converter 240 . The loudspeaker 250 may be used to output audio mixed with the video displayed on the TV panel 220 as sound waves. Although not shown, in one embodiment, the TV set 200 may include an external loudspeaker.

3 is a diagram illustrating a second embodiment of a block diagram of a set-top box having a lip sync control function. The components shown in FIG. 3 do not reflect all functions of the set-top box and are not essential, so it should be recognized that the set-top box may include more components or fewer components than the illustrated components. .

The demultiplexer 310 , the audio decoder 315 , the output terminal 323 , the microphone 330 , the A/D converter 335 , and the noise removal filter unit 340 which are components of the set-top box 300 shown in FIG. 3 . ) and the delay value determining unit 345 have substantially the same function as the corresponding components in the set-top box 100 shown in FIG. 1 and may be configured in the same manner, so a description thereof will be omitted. The set-top box 300 shown in FIG. 3 is different from the set-top box 100 shown in FIG. 1 in a connection method between the delay generator 317 and the video decoder 320 . The video decoder 320 is configured to receive a compressed video signal from the demultiplexer 310 through the first terminal 312, decode it, and output a decompressed video signal, and the delay generator 317 includes the video decoder ( 320) receives the decompressed video signal and outputs it as it is to the output terminal 323, and in response to receiving the delay value from the delay value determiner 345, receives the decompressed video signal and delays the output by the delay value. It may be configured to output to the terminal 323 . The video signal processing in the set-top box 100 shown in FIG. 1 is performed by a method of pre-delay and post-decoding, whereas the video signal processing in the set-top box 300 shown in FIG. 3 is performed by a method of pre-decoding and post-delay. can be said to be made with

In the above description, each of the components/circuits of the set-top boxes 100 and 300 has been described as separate and separate components, but all components/circuits or some components/circuits of the set-top boxes 100 and 300 are ASICs. Based on the (Application Specific Integrated Circuit), PLD (Programmable Logic Device) or FPGA (Field-Programmable Gate Array) design method, it can be implemented with a single IC chip.

5 is a diagram illustrating a first embodiment of a flowchart for explaining a lip sync control method performed in a set-top box.

The lip sync control method starts with a step ( S505 ) of receiving a signal including a compressed video signal and a compressed audio signal, for example, a Dolby Atmos signal. In step S510 , the demultiplexer 110 separates the Dolby Atmos signal into a compressed video signal and a compressed audio signal, and outputs them to the first terminal 112 and the second terminal 113 , respectively. In step S515 , the compressed audio signal is input to the audio decoder 115 , the decompressed audio signal is output from the audio decoder 115 , and the compressed video signal is inputted to the video decoder 120 , so that the video A decompressed video signal is output from the decoder 120 . In this step, the video decoder 120 synchronizes with the decompressed audio signal output with reference to the time information included in the header of the compressed audio signal input to the audio decoder 115, and the decompressed video signal A synchronous control operation may be performed to output . In step S520 , the compressed audio signal and the decompressed video signal are supplied to the TV set 200 . This step does not require actual operation, but can be implicitly performed by connecting the output of the video decoder 120 to the output terminal 123 and the second terminal 113 of the demultiplexer 110 to the output terminal 123 . It should be noted that there is In step S525 , the sound output from the TV set 200 is converted into an electrical signal using the microphone 130 . In step S530, an electric signal is A/D converted using the A/D converter 135 to provide loopback audio signal samples. In this step, an electric signal is A/D converted using the A/D converter 135 to provide digital samples, and noise filtering is performed on the digital samples provided by the noise removal filter unit 140 to generate loopback audio samples. can provide In step S535 , the decompressed audio signal output from the audio decoder 115 is sampled by the delay value determiner 145 to provide reference audio signal samples. In an embodiment, this step may be performed in synchronization with the aforementioned step S530 of providing loopback audio signal samples by A/D converting an electrical signal. According to this embodiment, the reference audio signal samples and the loopback audio signal samples may be samples within the same time window. In step S540 , a disparity value between the reference audio signal samples and the loopback audio signal samples is calculated by the delay value determiner 145 , and a delay value is determined based on the calculated disparity value, and the delay generating unit 117 . causes the input of the compressed video signal to the video decoder 120 to be delayed by a delay value. In an embodiment, the delay value determiner 145 determines the delay value to be the same as the disparity value. In an embodiment, the delay value determiner 145 determines a value obtained by subtracting a processing delay value from a disparity value as a delay value.

6 is a diagram illustrating a second embodiment of a flowchart for explaining a lip sync control method performed in a set-top box.

The lip sync control method starts with a step ( S605 ) of receiving a signal including a compressed video signal and a compressed audio signal, for example, a Dolby Atmos signal. In step S610 , the demultiplexer 310 separates the Dolby Atmos signal into a compressed video signal and a compressed audio signal, and outputs them to the first terminal 312 and the second terminal 313 , respectively. In step S615, the audio signal compressed by the audio decoder 315 is decoded to output the decompressed audio signal. In step S620, the video signal compressed by the video decoder 320 is decoded to output the decompressed video signal. In this step, the video decoder 320 may perform a synchronization control operation so that the decompressed video signal is output in synchronization with the output decompressed audio signal with reference to time information included in the header of the compressed audio signal. . In step S625 , the compressed audio signal and the decompressed video signal are supplied to the TV set 200 . This step does not require an actual operation, but the output of the video decoder 320 is directly transferred to the output terminal 323 without delay through the delay generator 317 , and the second terminal 313 of the demultiplexer 310 is output. It should be noted that this can be done implicitly by connecting to terminal 323 . In step S630 , the sound output from the TV set 200 is converted into an electrical signal using the microphone 330 . In step S635, an electric signal is A/D converted using the A/D converter 335 to provide loopback audio signal samples. In this step, the A/D converter 335 is used to A/D convert the electrical signal to provide digital samples, and noise filtering is performed on the digital samples provided by the noise removal filter unit 340 to generate loopback audio samples. can provide In step S640 , the decompressed audio signal output from the audio decoder 315 is sampled by the delay value determiner 345 to provide reference audio signal samples. In an embodiment, this step may be performed in synchronization with the aforementioned step ( S635 ) of A/D converting an electrical signal to provide loopback audio signal samples. According to this embodiment, the reference audio signal samples and the loopback audio signal samples may be samples within the same time window. In step S645, the delay value determining unit 345 calculates a disparity value between the reference audio signal samples and the loopback audio signal samples, and determines a delay value based on the calculated disparity value, and the delay generation unit 317 causes the supply of the decompressed video signal to the TV set 200 to be delayed by a delay value. In an embodiment, the delay value determiner 345 determines the delay value to be the same as the disparity value. In an embodiment, the delay value determiner 345 determines a value obtained by subtracting a processing delay value from a disparity value as a delay value.

In the above description, the meaning of the description that a component is connected to or coupled to another component means that the component is directly connected or coupled to the other component, as well as one or more other components interposed therebetween. It should be understood to include the meaning that may be connected or coupled through an element. In addition, terms for describing the relationship between elements (eg, 'between', 'between', etc.) should also be interpreted with similar meanings.

In the embodiments disclosed herein, the arrangement of the illustrated components may vary depending on the environment or requirements in which the invention is implemented. For example, some components may be omitted or some components may be integrated and implemented as one. Also, the arrangement order and connection of some components may be changed.

In the above, various embodiments of the present invention have been shown and described, but the present invention is not limited to the specific embodiments described above, and the above-described embodiments depart from the gist of the present invention as claimed in the appended claims. Without this, various modifications may be made by those of ordinary skill in the art to which the present invention pertains, and these modified embodiments should not be understood separately from the technical spirit or scope of the present invention. Accordingly, the technical scope of the present invention should be defined only by the appended claims.

100, 300: set-top box
110, 310: demultiplexer
112, 312: first terminal
113, 313: second terminal
115, 315: audio decoder
117, 317: delay generating unit
120, 320: video decoder
123, 323: output terminal
130, 330: microphone
135, 335: A/D converter
140, 340: noise removal filter unit
145, 345: delay value determining unit
200: TV set
210: input terminal
220: TV panel
230: audio decoder
240: D/A converter
250: loudspeaker

Claims (27)

A lip sync control method comprising:
decoding the compressed video signal to provide a decompressed video signal;
causing the decompressed video signal to be output to a TV set along with a compressed audio signal;
decoding the compressed audio signal to provide a decompressed audio signal;
sampling the decompressed audio signal to provide reference audio signal samples;
providing loopback audio signal samples via a sound output output from the TV set; and
and controlling a timing at which the compressed video signal is decoded or a timing at which the decompressed video signal is output to the TV set based on comparing the reference audio signal samples and the loopback audio signal samples. control method.
According to claim 1,
The step of converting a sound output output from the TV set into an electrical signal to provide loopback audio signal samples may include converting the sound output output from the TV set into an electrical signal, and A/D converting the electrical signal to provide the loopback audio signal samples. A method of controlling lip sync comprising the step of providing audio signal samples.
3. The method of claim 2,
The A/D conversion of the electrical signal to provide the loopback audio signal samples may include: A/D converting the electrical signal to provide digital samples and performing noise filtering on the provided digital samples to provide the loopback audio signal A method of controlling lip sync comprising providing samples.
3. The method of claim 2,
The A/D-converting the electrical signal to provide loopback audio signal samples and the sampling the decompressed audio signal to provide reference audio signal samples are performed in synchronization with each other.
5. The method of claim 4,
wherein the reference audio signal samples and the loopback audio signal samples are samples within the same time window.
According to claim 1,
Controlling the timing at which the compressed video signal is decoded or the timing at which the decompressed video signal is output to the TV set based on comparing the reference audio signal samples and the loopback audio signal samples comprises: Calculating a disparity value between audio signal samples and the loopback audio signal samples, and based on the disparity value, the decompressed video signal is output to the TV set or the compressed video signal is decoded and decompressed A method of controlling lip sync, comprising the step of delaying presentation of a video signal.
7. The method of claim 6,
Calculating a disparity value between the reference audio signal samples and the loopback audio signal samples, and based on the disparity value, the decompressed video signal is output to the TV set or the compressed video signal is decoded and compressed delaying the presentation of the released video signal comprising:
determining a delay value based on the disparity value; and
and causing the output of the decompressed video signal to be output to the TV set or the decoding of the compressed video signal to be provided as a decompressed video signal delayed by the delay value.
8. The method of claim 7,
and the delay value is the same as the parallax value.
9. The method of claim 8,
The delay value is a value obtained by subtracting a processing delay value from the disparity value.
According to claim 1,
receiving a Dolby Atmos signal comprising the compressed video signal and the compressed audio signal; and
Separating the Dolby Atmos signal into the compressed video signal and the compressed audio signal.
According to claim 1,
The decoding of the compressed video signal to provide a decompressed video signal may include referring to time information included in a header of the compressed audio signal to synchronize with the provided decompressed audio signal and to provide the decompressed video signal. A method of controlling lip sync comprising the step of providing a video signal.
According to claim 1,
The TV set includes a TV, lip sync control method.
According to claim 1,
wherein the TV set includes a TV and a loudspeaker connected to the TV.
As a set-top box,
A demultiplexer configured to demultiplex a signal including a compressed video signal and a compressed audio signal to output a compressed video signal to a first terminal and output a compressed audio signal to a second terminal, wherein the second terminal is connected to a TV connected to the output terminal for -,
an audio decoder configured to decode the compressed audio signal output to the second terminal to output a decompressed audio signal;
a delay generator configured to receive the compressed video signal output to the first terminal and output it as it is or with a delay;
a video decoder configured to decode the compressed video signal output from the delay generator and output a decompressed video signal to the output terminal;
a sound wave converter configured to receive a sound wave output from the TV and provide loopback audio signal samples; and
a delay value determiner configured to sample the decompressed audio signal to provide reference audio signal samples, and to determine and output a delay value based on the reference audio signal samples and the loopback audio signal samples;
The delay generator is further configured to receive and output the compressed video signal as it is, and to receive the compressed video signal in response to receiving the delay value from the delay value determiner and delay the output by the delay value. set top box.
15. The method of claim 14,
The audio decoder is a software-based Atmos audio decoder, a set-top box.
15. The method of claim 14,
The sound wave conversion unit includes a microphone configured to convert the sound wave output from the TV into an electrical signal,
The sound wave converter is further configured to A/D convert the electrical signal to provide loopback audio signal samples.
15. The method of claim 14,
and a denoising filter unit configured to denoise the loopback audio signal samples.
15. The method of claim 14,
wherein the reference audio signal samples and the loopback audio signal samples are samples within the same time window.
15. The method of claim 14,
The delay value determining unit is further configured to calculate a disparity value between the reference audio signal samples and the loopback audio signal samples and determine the delay value based on the disparity value.
15. The method of claim 14,
The set-top box, wherein the audio decoder, the sound wave conversion unit, and the delay value determining unit is operated to be deactivated in response to the delay generating unit receiving the delay value from the delay value determining unit.
As a set-top box,
A demultiplexer configured to demultiplex a signal including a compressed video signal and a compressed audio signal to output a compressed video signal to a first terminal and output a compressed audio signal to a second terminal, wherein the second terminal is connected to a TV connected to the output terminal for -,
an audio decoder configured to decode the compressed audio signal output to the second terminal to output a decompressed audio signal;
a video decoder configured to decode the compressed video signal output to the first terminal to output a decompressed video signal;
a delay generator configured to receive the decompressed video signal and output it as it is or with a delay to the output terminal;
a sound wave converter configured to receive a sound wave output from the TV and provide loopback audio signal samples; and
a delay value determiner configured to sample the decompressed audio signal to provide reference audio signal samples, and to determine and output a delay value based on the reference audio signal samples and the loopback audio signal samples;
The delay generator receives the decompressed video signal and outputs it as it is, and in response to receiving the delay value from the delay value determiner, receives the decompressed video signal and delays the output by the delay value. A set-top box.
22. The method of claim 21,
The audio decoder is a software-based Atmos audio decoder, a set-top box.
22. The method of claim 21,
The sound wave conversion unit includes a microphone configured to convert the sound wave output from the TV into an electrical signal,
The sound wave converter is further configured to A/D convert the electrical signal to provide loopback audio signal samples.
22. The method of claim 21,
and a denoising filter unit configured to denoise the loopback audio signal samples.
22. The method of claim 21,
wherein the reference audio signal samples and the loopback audio signal samples are samples within the same time window.
22. The method of claim 21,
The delay value determining unit is further configured to calculate a disparity value between the reference audio signal samples and the loopback audio signal samples and determine the delay value based on the disparity value.
22. The method of claim 21,
The set-top box, wherein the audio decoder, the sound wave conversion unit, and the delay value determining unit is operated to be deactivated in response to the delay generating unit receiving the delay value from the delay value determining unit.

Images