JPH099224A - Dynamic image and speech codec using lip-synchronous controller - Google Patents

Abstract

PURPOSE: To match the movement of the mouth of an object with speech. CONSTITUTION: A motion vector detection circuit 41 improves performance by making a block size small when the movement of the object image picked up by a camera 1 is large, outputs the lip-synchronous control signals Ls of a delay level corresponding to the arithmetic amount of the block matching arithmetic circuit group to a lip-synchronous control circuit 14a and performs lip synchronization. Also, when the movement of the object is small, the block size is enlarged, the time of the arithmetic processing of the block matching arithmetic circuit group is substantially shortened and the delay time of the lip synchronization is adaptively shortened.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving image and audio codec device in the field of communication equipment such as a video conference system, and more particularly to a lip sync control device for performing compression and expansion of audio data.

[0002]

2. Description of the Related Art In recent years, in a lip sync control device, although the delay due to the compression and expansion of the image data on the image codec side changes depending on the motion of the image, the delay amount of the audio data is mainly controlled by the transmission rate. Has become.
Hereinafter, a conventional lip sync control device will be described with reference to the drawings.

FIG. 5 is a functional block diagram of a moving picture and audio codec device using a conventional motion vector detecting circuit. First, the image system is the input video signal from the camera 1.
(NTSC) is A / D converted by the A / D converter 2,
The line converter 3 performs conversion to an intermediate format commonly used in the world called ITC-TH261 (for example, CIF). And C
The video signal encoded by the IF conversion is motion-compensated by the motion vector detection circuit 4, and is encoded and encoded by the DCT unit 5 into a motion-compensated inter-frame prediction signal. In addition, the difference signal from the reproduced image signal of the previous image frame generated from the motion vector detection circuit 4 is encoded, and the DCT of the DCT unit 5 performs discrete cosine transform to transform coefficients into a zigzag scan array, and then quantized. It is quantized by the unit 6. Then, after the quantized data is further compressed by the variable length coding unit 7, the error detection and correction unit 8 performs error correction coding and follows the data structure defined by the multimedia multiplexing and demultiplexing unit 9. It is multiplexed into one bit string and transmitted to the public network N.

Next, in the audio system, the input audio signal from the microphone 11 is A / D converted by the A / D converter 12, compressed by the audio encoder 13, and delayed by the lip sync control circuit 14 in the image system. The delay of the voice corresponding to
Fixed control via ₁ . Then, it is multiplexed into one bit string according to the data structure defined by the multimedia multiplexing and demultiplexing unit 9 and transmitted to the public line network N.

The CPU 10 controls the direction of the camera 1 via the control detector 15, and controls the audio encoder 13 and the lip sync control circuit 14 in accordance with the direction control.

FIG. 6 is a block diagram of the conventional motion vector detecting circuit 4 shown in FIG. 5, in which 16 is an input image.
Input block data a consisting of × 16 pixels and block data b consisting of 32 × 32 pixels as a stored image,
A window search circuit that gradually shifts the position of the block read from the block data b of the stored image, 17 is an arithmetic element unit having 256 arithmetic circuits for performing a block matching operation and calculating a residual f, and 18 is obtained. A comparison circuit for obtaining the minimum residual fm from the residual value, and 19 is an output circuit for serially outputting the residual fs at the same coordinates as the motion vector (coordinates Vx, Vy, residual fm) as motion vector information. Is.

First, in the block matching calculation in the calculation element section 17, the sum of absolute differences between the block data a and b expressed by the following (Equation 1) is calculated as a residual f.

[0008]

[Equation 1]

The block data b of 32 × 32 pixels is converted into 16
Since the search is performed using the window of 16 pixels (block data a), 16 trials are required in each of the horizontal and vertical directions, for a total of 256 trials.

In this way, the residual output f from the arithmetic circuits 17a, 17b ... 17n of each arithmetic element unit 17 is output to a total of 256 comparing circuits 18, which compare the magnitudes of these. Then, the minimum residual fm is detected, and the coordinates Vx and Vy of the motion vector are calculated from the clock coefficient value at that time.

Since the delay information due to such motion vector calculation of the video system is not transmitted to the audio system, the lip sync control circuit 14 of FIG. 5 uses the fixed delay estimated from the average delay of the video system. It is the set value. The fixed set value is about 0 to 500 msec in one direction and is set by the CPU 10.

[0012]

However, in the above-mentioned conventional motion vector detection circuit, the arithmetic element unit is
Execute block matching operation by full search in 17,
Since all calculations are performed regardless of the motion level of the subject, the delay time of the lip sync has been set to a fixed value without a means for switching according to the motion level. As a result, the movement of the subject's mouth and the voice did not match, and the naturalness was impaired.

The present invention solves the above problems, and an object of the first invention is to reasonably estimate the motion of a pixel block between consecutive frames and control the number of image frames to be processed per second.
(Thinning) to perform calculation according to the motion level of the subject to improve the prediction accuracy between image frames, and predict and compensate the motion vector of the thinned image frames to reduce the number of calculation trials. A motion vector detection circuit capable of executing efficient calculation, and a motion vector calculation delay information of the motion vector are transmitted to a lip sync control circuit to adaptively delay audio data.

A second object of the present invention is to reasonably estimate the transmission rate between consecutive image frames, transmit the transmission rate information between the image frames to a lip sync controller, and adaptively transmit voice data. It is delayed.

[0015]

The first invention of the present invention is as follows:
To achieve the object, a CPU that controls the direction of the camera according to the movement of a subject, an image frame control circuit that controls the number of image frames processed every second when the CPU controls the direction of the camera, and the image frame control A motion vector prediction / detection circuit that predicts a motion vector of an image frame decimated from the output from the circuit, a filter that determines the motion level of the subject of the input image, and a switch circuit group that allows the user to select the block size, and a block A window search circuit and a block matching calculation circuit group that perform matching calculation, a comparison circuit group that compares the outputs of the window search circuit and the block matching calculation circuit group, and an output that outputs the output of the comparison circuit group as a motion vector Motion vector detection circuit including circuit and the comparison A delay level output circuit that delays n-bit delay information from the path group, a lip sync prediction detection circuit that predicts a lip sync of an image frame decimated from the output from the image frame control circuit, and encoded audio A lip sync control circuit comprising a multiplexer for switching a delay table in which a delay amount of the audio data is determined by a lip sync control signal for data is provided.

In order to achieve the object, a second invention of the present invention comprises an image compression circuit for compressing a moving image from a camera, and an audio compression circuit for compressing a voice from a microphone.
The time series consistent number TR data for each image processed by the image compression circuit is read, the transmission time between the previous image frame and the next image frame is discriminated from the TR data difference within a unit time, and from the discriminated time, It is characterized in that it has a lip sync control circuit composed of a CPU for delaying an audio signal from an inter-image frame time and an audio data synchronization deviation amount table obtained in advance to synchronize an image and audio.

[0017]

According to the first aspect of the present invention, when the movement of the subject is large, the block size is reduced to improve the performance and the lip sync operates at a delay level according to the amount of calculation, and when the movement of the subject is small, The block size can be increased to greatly reduce the calculation processing time, and the lip sync delay time can be adaptively shortened. In addition, it makes it possible to improve the followability with sound for the movement of the camera,
It also allows the user to select the block size and adjust the image quality.

According to the second aspect of the invention, the CPU reads the time-series consistent number TR data for each image processed by the image compression circuit, and detects the difference between the previous image frame and the next image frame from the TR data difference within a unit time. It is possible to synchronize the image and the voice by delaying the voice signal from the time between image frames and the voice data synchronization deviation amount table which are obtained in advance from the determined transmission time.

[0019]

FIG. 1 is a functional block diagram of a moving image and audio codec device using a lip sync control device according to a first embodiment of the present invention. In FIG. 1, the same functional blocks as those in FIG. 5 of the conventional example are designated by the same reference numerals, and the description thereof will be omitted. Here, unlike the conventional example, the sound delay corresponding to the delay of the image system is not fixedly controlled by the CPU 10 and is lip-controlled to the lip sync control circuit 14a via the control line L ₂ from the motion vector detection circuit 4a. It is characterized in that the sync control signal Ls is adaptively added and controlled.

FIG. 2 is a block diagram of the motion vector detection circuit 4a and the lip sync control circuit 14a shown in FIG. A reference numeral 21 in the motion vector detection circuit 4a is a switch circuit group consisting of a filter for discriminating the motion level of the object of the input image and switches 21a, 21b. Is. Reference numeral 22 denotes a window search circuit and a block matching calculation circuit group that perform a block matching calculation. This is the block size within the block
22a, 22b …… 22n

[0021]

[Outside 1]

The block sizes are arranged in order from the left of FIG. Then, as will be described later, as the block size decreases, the amount of calculation increases, and the delay shifts from the smaller delay to the larger delay. Reference numeral 23 is a comparison circuit group for comparing the outputs of the window search circuit and the block matching calculation circuit group 22.

These filter and switch circuit group 2
1, the window search circuit and block matching arithmetic circuit group 22, and the comparison circuit group 23 are each composed of n + 1 units corresponding to each other. Further, 24 is an output circuit, and 26 is an image frame control circuit, which controls the number of image frames to be processed per second when the CPU 10 controls the direction of the camera 1. A motion vector prediction / detection circuit 27 predicts the motion vector 20 of the image frame thinned out from the output from the image frame control circuit 26. 28 is a switch for the user to select the block size 22a, 22b ... 22n.

Next, 29 in the lip sync control circuit 14a
Is a delay level output circuit for delaying the n-bit delay information (lip sync control signal Ls in FIG. 1) from the comparison circuit group 23. Reference numeral 30 is a lip sync prediction detection circuit for predicting lip sync of the image frames thinned out from the output from the image frame 26. Reference numeral 31 is a delay table in which the amount of delay of the audio data 33 encoded by the audio encoder 13 is determined by the lip sync control signal Ls.
It is a multiplexer that switches between 32 (0, 5 ... 500 ms).

FIG. 3 is a geometrical relationship diagram between a reference image for block matching and a previous image frame search area in the first embodiment of the present invention, in which C is a reference image and M × N. The current frame consisting of blocks is shown, and D is the search area image in the previous image frame. The operation of the motion vector detection circuit and the lip sync control circuit shown in FIG. 2 configured as described above will be described.

Now, assuming that the image of the (n-1) th frame, which is the frame immediately preceding the nth frame, is stored at both ends of transmission and reception, this information and the difference information of the next nth frame are stored. Based on this, the moving area of the subject is detected. To do that, divide the screen into sub-blocks,
The motion vector (direction and movement amount) is detected depending on how many pixels the sub-block is moved in any direction and pattern matching can be achieved in the nth frame.
This is done for all sub-blocks and the motion vector is transmitted. At the same time, at the transmitting end, the motion vector is used to create the motion-predicted nth frame image data.
The difference from the actual nth frame image data is called a prediction error and is encoded and transmitted. At the receiving end, the nth frame is restored from the already stored (n-1) th frame information, the transmitted motion vector and prediction difference.

Here, in the block matching calculation in the block matching calculation circuit group 22 of FIG. 2, the previous frame is the entire area of (M + 2p) × (N + 2p) in FIG. 3, and the sub-block C (reference image) is Depending on the level of the movement, if there is no movement, (M + 2p) × (N + 2p) is set as the sub-block C, or if the movement level is large, the maximum (N × N) size is used for the block matching calculation. The area to be searched at this time is (D area) C for each area corresponding to the sub-block C in the previous frame D.
Compare with

Then, the sub-block C represented by (Equation 2),
The sum of absolute differences between the block D and the block D is calculated as the residual f.

[0029]

[Equation 2]

In this process, motion estimation is performed in the range of ± P pixels or lines / frames, and all pixels are shifted.
The brute force search considering (horizontal direction) and all line shifts (vertical direction) requires (2P) ² cost function calculations.

At this time, the block size (M
Since the calculation amount of the cost function also changes when (× N) is changed according to the motion level, for example, in the stationary state, the block size (M × N) is set to be large to reduce the calculation amount of the cost function. Can be efficient.

Here, as shown in FIG. 2, a switch 28 that is turned off when the next stage is turned on is provided between each stage of the filter and switch circuit group 21, so that, for example, the block data b of the stored image remains static. When it is an image, the first stage filter and switch circuit 21a is turned on,
Switch 2 between filter and switch circuit group 21b
8 is the block size 22a (M / 2 ⁰ × N / of the first stage of the window search circuit and block matching arithmetic circuit group 22)
2 ⁰⁾ and only the valid, since the disabled next stage following the matching operation by the largest block size is performed.

When the motion level of the block data b of the stored image rises from this, the second stage filter and switch circuit 21b and the second stage window search circuit and 22b of the block matching arithmetic circuit group 22 (block size M / 2 ¹ × N / 2 ¹ ) is only valid. Furthermore, by changing the block size according to the motion vector,
The amount of calculation can be reduced as much as possible to improve the followability of the communication device.

The filter and switch circuit group 21 has a switch structure in which the CPU 10 can switch the block sizes 22a, 22b ... 22n, and manual control for fixing the block size is also possible.

Next, the motion vector detecting circuit 4a shown in FIG.
The operation of FIG. 1 using the lip sync control circuit 14a will be described. When the CPU 10 controls the direction of the camera 1, the CPU 10 controls the block size and the number of image frames based on the content of the direction control. For example, when the control speed of the camera 1 is high, the means for improving the followability to the movement of the camera 1 is as follows.

First, the CPU 10 sets the block size large within an allowable range to reduce the calculation amount of the cost function, and the image quality is deteriorated such as block distortion, but the followability to motion can be prioritized. . And the control line L
_As the lip sync control signal Ls to ₂ , the n-bit information based on the selected block size is transmitted.

Secondly, the CPU 10 sets the block size to be small in order to improve the followability to the motion without degrading the image quality, and the cost function increases and the processing time becomes longer. Control (decimate) the number to reduce the amount of information. Then, the motion vector of the thinned image frame is predicted and detected from the motion vector of the preceding and succeeding image frames to be supplemented. The lip sync control signal Ls of the control line L ₂ is also n-bit information based on the selected block size and n-bit information predicted and detected from the delay levels before and after the thinned image frame is transmitted. Table 1 shows an example of the delay level of the lip sync control signal.

[0038]

[Table 1]

As described above, in this embodiment, since the motion vector calculation process is performed with the block size corresponding to the speed of movement of the subject, the lip sync process is also delayed and set based on the selected block size. . As a result, the movement of the subject's mouth matches the voice regardless of the movement level of the subject.

FIG. 4 is a functional block diagram of a moving picture and audio codec device using a lip sync control device according to the second embodiment of the present invention. The same functional blocks as those in FIG. 1 are designated by the same reference numerals, and the description thereof will be omitted. In FIG. 4, 34 is an image compression circuit for compressing a moving image from the camera 1, 35 is a voice compression circuit for compressing the voice from the microphone 11, 36 is a CPU for the voice compressed by the voice compression circuit 35.
A signal delay circuit that delays for a predetermined time by controlling 10
Reference numeral 37 is a previously obtained time between image frames and audio data synchronization deviation amount table, and 38 is a line interface (I / I).
F).

Next, the operation will be described. First, the image signal input from the camera 1 is the image compression circuit 34 "ITC-TH.26".
Compressed with 1 ”algorithm. Next, it is multiplexed with a voice signal and other data by the multimedia multiplexing / separating unit 9 and transmitted to the public line network N through the line interface 38.

Here, the time series consistent number "TR" data for each image processed on the "ITC-TH.261" algorithm is read into the CPU 10 to calculate the image discharge speed. The speed for each image is calculated by the following procedure.
First, the "TR" data read by the CPU 10 is stored as {n2}. The "TR" data is read after the unit time set next, and this is designated as {n1}. By calculating {n1-n2} from these two data, the number of frames of the image within the unit time can be obtained. The transmission time between the previous image frame and the next image frame is discriminated from the "TR" data difference within this unit time.

A signal delay circuit inserted in the audio signal processing circuit line from the inter-image frame time and audio data synchronization deviation amount table 37 obtained in advance from the determined time.
Controls 36 to delay the audio signal and synchronize the image and audio.

As described above, in the present embodiment, the image processing speed is judged from the image frame speed carried out from the image compression block to control the delay amount of the lip sync processing. As a result, when the image frame speed is fast, the delay amount is small, and conversely, when the image frame speed is slow, the delay amount is increased so that the movement of the subject's mouth and the sound match.

[0045]

As described above, according to the first aspect of the present invention, since the motion vector calculation process is performed with the block size according to the speed of movement of the subject, the lip sync process also uses the selected block size. By being delayed with respect to the reference, the movement of the subject's mouth and the voice match regardless of the movement level of the subject.

The second aspect of the invention is to judge the image processing speed from the image frame speed carried out from the image compression block and control the delay amount of the lip sync processing so that when the image frame speed is high, When the amount of delay is small and conversely slow, the amount of delay is increased to match the movement of the subject's mouth with the sound.

[Brief description of drawings]

FIG. 1 is a functional block diagram of a moving picture and audio codec device using a lip sync control device according to a first embodiment of the present invention.

FIG. 2 is a block diagram of a motion vector detection circuit and a lip sync control circuit of FIG.

FIG. 3 is a geometrical relationship diagram between a reference image for block matching and a previous image frame search area in the first embodiment of the present invention.

FIG. 4 is a functional block diagram of a moving image and audio codec device using a lip sync control device according to a second embodiment of the present invention.

FIG. 5 is a functional block diagram of a moving image and audio codec device using a conventional motion vector detection circuit.

6 is a functional block diagram of the motion vector detection circuit of FIG.

[Explanation of symbols]

1 ... Camera, 2, 12 ... A / D converter, 3 ... Line converter, 4a ... Motion vector detection circuit, 5 ... Coding and DCT section, 6 ... Quantization section, 7 ... Variable length coding section, 8 ... error detection / correction unit, 9 ... multimedia multiplexing, separation unit, 10 ... CPU, 11 ... microphone, 13 ... voice encoder, 14a ... lip sync control circuit, 15 ... control detection unit, 21 ... filter and switch circuit group , twenty two…
Wind search circuit and block matching arithmetic circuit group, 23 ... Comparison circuit group, 24 ... Output circuit, 26 ... Image frame control circuit, 27 ... Motion vector prediction detection circuit, 28
... switch, 29 ... delay level output circuit, 30 ... lip sync prediction detection circuit, 31 ... multiplexer, 32 ... delay table, 34 ... image compression circuit, 35 ... audio compression circuit, 36 ... signal delay circuit, 37 ... between image frames Time and audio data synchronization deviation amount table, 38 ... Line interface (I / F).

Claims (3)

[Claims] 1. A CPU for controlling the direction of a camera according to the movement of a subject, an image frame control circuit for controlling the number of image frames processed every second when the CPU controls the direction of the camera, and the image frame control circuit. Motion vector predictive detection circuit that predicts the motion vector of the image frame thinned out from the output from the, the filter that determines the motion level of the subject of the input image and the switch circuit group that the user can select the block size, and the block matching A window search circuit and a block matching calculation circuit group that perform calculation, a comparison circuit group that compares the outputs of the window search circuit and the block matching calculation circuit group, and an output circuit that outputs the output of the comparison circuit group as a motion vector. From the motion vector detection circuit and the comparison circuit group consisting of Delay level output circuit for delaying the n-bit delay information, a lip sync prediction detection circuit for predicting the lip sync of the image frame thinned out from the output from the image frame control circuit, and the encoded audio data. A moving picture and audio codec device using a lip sync control device, comprising a lip sync control circuit including a multiplexer for switching a delay table in which a delay amount of the audio data is determined by a sync control signal. 2. The CPU, in the operation control of the block matching arithmetic circuit group, controls the size of the block size to reduce the calculation amount of the cost function and the information amount. A video and audio codec device using the lip sync control device according to 1. 3. An image compression circuit for compressing a moving image from a camera, and an audio compression circuit for compressing audio from a microphone,
The time series consistent number TR data for each image processed by the image compression circuit is read, the transmission time between the previous image frame and the next image frame is discriminated from the TR data difference within a unit time, and from the discriminated time, A lip sync control device having a lip sync control circuit including a CPU for delaying an audio signal from an image frame inter-frame time and an audio data synchronization deviation amount table obtained in advance to synchronize an image and audio is used. Video and audio codec device used.