JPH0714208B2 - Video transmission system - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION [Industrial field of use] The present invention relates to a moving image transmission system used for videophones and video conferences. The present invention, while compressing the amount of transmitted information,
The present invention relates to a moving image transmission method capable of transmitting more natural images.

〔Overview〕

The present invention relates to a moving image transmission method in which a model image is transmitted, motion information of the subsequent image is transmitted, and the model image is deformed to synthesize and display the moving image. Realization of a more natural moving image by transmission of a small amount of information by extracting and transmitting changes as isodensity line information and correcting the density of the moving image synthesized from the model image based on this isodensity line information Is.

[Conventional technology]

As a moving image transmission method for transmitting a face image used in a videophone, a video conference, etc., various moving image transmission methods capable of compressing a transmission information amount and compressing a used band of a transmission line have been studied. As such a moving image transmission method, an intelligent encoding method has been studied.

This method is a method in which only one model image is transmitted in advance prior to a call, and only information about the face movement of the caller is transmitted during the call. More specifically, prior to the call, one model image of the face of the caller (still image, with no expression) is transmitted from the sender side to the receiver side. A face shape model of the caller is formed based on the image. During an actual call, the transmitting side recognizes the movement of each part of the face of the caller, analyzes the movement, and transmits the analysis information of this movement to the receiving side. On the receiving side, the shape model based on the above-mentioned model image is transformed based on the received motion analysis information to synthesize a moving image, and this is displayed.

[Problems to be solved by the invention]

As a problem of such a conventional moving image transmission system, there is a problem that information on a change in how the face is illuminated is not transmitted due to a change in the shape of the face. This is information that the density value of the portion changes with the movement because the way the face is illuminated, that is, the irradiation angle changes due to the change in the shape of the face. Even if the transmitting side can accurately extract the facial movement, if this information is not transmitted, the same image as the transmitting side cannot be reproduced only by deforming the model on the receiving side.

A more specific example is the expression of wrinkles. Wrinkles appear as shadows due to changes in the shape of the face, and the model image does not include density information related thereto. For this reason, it is impossible to express wrinkles simply by deforming the model.

The present invention solves the above-mentioned problems, and transmits density change information of an input image as isodensity line information and performs density correction using this information, thereby transmitting a natural image closer to the input image. It is an object of the present invention to provide a moving image transmission system capable of performing the above.

[Means for solving problems]

The present invention includes, on the transmitting side, means for transmitting a model image prior to a call, and means for analyzing movement of an image during a call and transmitting motion information to the receiving side. In a moving image transmission system equipped with means for transforming the model image according to the information on the image motion transmitted from the device and synthesizing an image accompanied by motion, on the transmitting side, a point where the density of the image during the call is equal is drawn. And a means for sequentially transmitting the isoconcentration line information extracted by this means to the receiving side, and using the isoconcentration line information received by the receiving side. It is characterized in that a means for correcting the density of the combined image is provided.

Further, the present invention includes, on the transmission side, means for extracting an isodensity line connecting points having the same density of an input image during a call with a line, and means for encoding the extracted isodensity line, On the side, a means for restoring a grayscale image from the received isoconcentric line information, a means for extracting an isoconcentric line of the image composed by combining the motions from the model image, and an image composed as described above from the extracted isoconcentric lines Means for restoring the grayscale image, a means for calculating the density difference between the grayscale image restored from the received iso-concentration line information and the grayscale image of the image synthesized from the model image, and the density difference And a means for correcting the density of the image.

[Action]

In the present invention, first, a static model image of the caller is transmitted prior to the call. The receiving side extracts characteristic points from this model image and stores them as characteristic point data.

During a call, the transmitting side extracts the characteristic points of the input image and transmits it to the receiving side as image movement information, and the receiving side deforms the model image based on this information to create a composite image.

On the transmitting side, the density of the image is extracted as time-series information of the contour line information together with the information on the movement of the image, and this is sequentially transmitted to the receiving side. The receiving side corrects the density of the composite image transformed from the model image based on the contour line information, and displays it as an output image to the receiving party.

As described above, by using the contour line information, the amount of transmission information can be reduced, so that the transmission efficiency is improved as compared with the case of transmitting the density information of all pixels. Further, the temporal change of the contour lines well represents the movement of the face, so that a natural restored image can be obtained.

〔Example〕

 Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram showing the configuration of a moving image transmission system according to an embodiment of the present invention.

In the moving image transmission system of this embodiment, the transmitting side uses the photographing means 2 for photographing the face of the caller 1, the image analyzing means 3 for analyzing the image photographed by the photographing means 2, and the image analyzing means 3. And a transmitting means 4 for transmitting the analyzed image information onto the transmission path 5. Also, the receiving side displays a moving image combined with the receiving means 6 for receiving the image information on the transmission path 5 and the image combining means 7 for combining the moving image from the image information received by the receiving means 6 to the caller 9. The image display means 8 is provided. The image pickup means 2 is a normal television image pickup device, and the image display means 8 is a television display device.

The image analysis means 3 on the transmission side extracts a model image, which is a still image to be transmitted to the reception side prior to the call, from the image information of the caller 1 captured by the imaging means 2, and then extracts the model image to be transmitted to the reception side. A means 31 and a movement analysis means 32 for analyzing the movement of the face of the calling party 1 during the call and outputting the movement information.
An iso-dense line extracting means 33 for extracting an iso-dense line which is information on the density of the face of the caller 1 during a call is provided.

The image synthesizing means 7 on the receiving side is a model creating means for extracting the feature points from the model image received prior to the call.
71, a motion synthesizing unit 72 that transforms the model image created above based on the received motion information to create a composite image that has been subjected to motion synthesizing, and motion synthesizing based on the received iso-concentration line information. And a density correction unit 73 that corrects the density of the synthesized image that has been executed.

Note that this moving image transmission system is bidirectional, and reverse transmission is also performed with the same configuration.

The feature of the present invention is that, in order to generate a natural moving image, the change amount information of the degree of reflection of the face due to the movement is reduced as an information amount, and is represented as a contour line that well represents the rough density distribution of the face. It is extracted and transmitted to the receiving side, and the receiving side calculates the difference between the density of the image reconstructed based on the contour lines and the density of the image reconstructed from the model image, and this difference is added to the composite image subjected to motion composition. This is to add and correct the density to transmit a natural image during a call.

Next, FIG. 2 shows a block diagram of the flow of image processing in the moving image transmission system of this embodiment. This block diagram shows the details of the image processing performed by the image analysis means 3 and the image synthesis means 7 in FIG.

Image processing in the moving image transmission system of this embodiment is roughly divided into three types: model creation, motion analysis and synthesis, isodensity line extraction, and density correction. Each image processing will be described in detail below.

(1) Model Creation In this example, as a priori knowledge, one model image of the caller 1 is prepared on the receiving side, and a wire frame representing the face shape of the caller 1 is created based on this model image. .
The procedure for creating the wire frame will be described.

First, on the sending side, one of the face images of the caller 1 is model image 11
Extracted from the input from the image pickup means 2 as 1. This face image is a still image and has no expression. This model image is transmitted to the receiving side before the call. On the receiving side, 120 feature points of the face of the caller 1 and the center and radius of the iris are extracted from the received model image 112. Then, the positions of the 385 wire frame composing points are linearly calculated using the feature points of the face. Then, the characteristic points and the constituent points of the wire frame are connected by a triangular frame to form a model wire frame 114. Then, the model image 112, the model feature point information (including information on the center and radius of the iris) 113, and the model wire frame 114 are stored in the knowledge database 115.

(2) Motion analysis and synthesis The motion of the face of the talker 1 during a call is extracted on the basis of feature points on the transmitting side and transmitted to the receiving side, and on the receiving side a model image is based on this motion information. It transforms and synthesizes an image with facial movements. This motion analysis and extraction is performed for each facial organ,
For example, it is performed based on the positions of the eyes, the nose, and the like.

First, prior to the analysis of the motion, in order to simplify the processing, a region indicating a rough position of each organ of the face, that is, a candidate region is set. This candidate area setting process is performed by sequentially searching for an area representing each organ from a binarized image using a local threshold in the initial frame of a call,
In the subsequent frames, the feature points extracted in the previous frame are used.

Differentiation and thinning are performed within this candidate region to extract the contour lines of each facial organ, that is, the feature lines. At this time, the type and size of the differential mask, the thinning algorithm, and the like differ depending on the characteristic line to be extracted.

Next, 120 points on the extracted characteristic line are extracted as characteristic points and transmitted to the receiving side. Further, the center and radius of the iris are extracted by performing Hough transform using the differential image in the eye candidate region and using the differential value as the weight, and are transmitted to the receiving side as the feature point information 122 together with the above-mentioned feature points. To transmit.

The flowchart of the motion analysis operation on the transmitting side is described in the third section.
Shown in the figure.

The creation of a composite image that combines the motions on the receiving side is basically based on the affine transformation of the model image using the wire frame.

First, the feature point 120 of the input image in the received feature point information 123
The input wireframe 124 is created by the same process as the wireframe creation of the model image using the points. Next, the model wire frame 126 is extracted from the knowledge database 115, and the model image 128 is deformed using this model wire frame 126 and the input wire frame 124. This modification is performed by affine-transforming the pixels of the model image included in each triangular frame of the model wire frame 126 into the corresponding triangular frame of the input wire frame 124. The obtained image is the composite image 129.

Next, inside the eye, the iris portion is extracted from the iris portion of the model image, and the white eye portion is extracted from the white eye portion of the model image, using information about the center and radius of the iris of the model image and the input image. Pixels are affine transformed and combined. As a result, the composite image 130 is obtained.

The flowchart of the image synthesizing operation on the receiving side is described in the fourth section.
Shown in the figure.

(3) Density Correction of Composite Image by Contour Lines Here, a line obtained by connecting pixels having the same density value in an image is called a contour line.

On the transmitting side, in order to detect the density value of the image, first, the input image 121 is quantized using a threshold value that divides the dynamic range of the image into eight equal parts. For this quantized image, the image is smoothed by the majority logic using a mask of 9 × 9 pixels centered on the pixel of interest. This is for averaging pixels having noise picked up by smoothing or having a value different from that of surrounding pixels. Then, the portion where the density value of the smoothed input image 121 changes, that is, the boundary of the above eight levels is actually extracted as a contour line 131. The extracted contour lines are encoded by a chain code and transmitted to the receiving side. As this chain encoding method, for example, there is an encoding method in which the coordinates of one point having a contour line are designated as the starting point and the direction of the line is designated as 8 directions or 4 directions.

FIG. 5 shows a flowchart of the operation of extracting the contour lines on the transmitting side.

By extracting the density of this input image as a contour line, it is possible to reduce the amount of information transmitted to the receiving side as compared to transmitting the change in density of all pixels.

On the receiving side, the contour lines 134 obtained by decoding the received encoded contour lines information are obtained. Further, the receiving side extracts the iso-density line 136 from the composite image 130 in which the motion is corrected based on the model image by the same processing as the transmitting side.

Then, the contour lines 134 of the input image and the contour lines 136 of the composite image are converted into grayscale images 135 and 137 by interpolating the density values of the pixels between the contour lines. As this interpolation method,
There is an interpolation method that interpolates the density value by obtaining the distance to the intersection and the intersection of the upper, lower, left, and right lines for one pixel between the isodensity lines. Further, instead of four points, it is also possible to obtain two or more intersections in the vertical and horizontal directions and interpolate the density value.

In this restoration of the grayscale image, the information of the transmitted isodensity lines is information obtained by extracting density information as rough information on the transmission side and compressing it. The value is roughly restored. In addition, the extraction of the contour lines 136 from the composite image 130 and the contour lines 136
In order to obtain the difference between the grayscale image 137 and the grayscale image 135 restored from the input image, the grayscale image 137 is compared with the input image 121 on the basis of the same motion-synthesized synthetic image. This is because.

In this embodiment, since the first transmitted model image contains fine density information, not all density information is sent, but the difference can be detected on the receiving side as a rough change in density distribution. It is configured such that it is sent as isodensity line information, the difference is detected on the receiving side, and this is added to the density value information of the first model image.

In this way, the grayscale image 135 obtained by converting the contour lines of the input image 121 is a grayscale image 137 obtained by converting the rough density distribution of the input image 121 into the contour lines of the composite image 130. Are considered to represent the rough density distribution of the composite image, respectively. Therefore, the amount of density change due to the change of the face can be roughly estimated by taking these differences. Therefore, if the difference between the two grayscale images 135 and the grayscale image 137 reconstructed from the composite image is calculated, a difference image 138 that is the change in density is obtained. By adding the difference image 138 to the composite image 130, the output image 140 corrected for the density change is obtained.

Here, when the output image 140 is output to the image display unit 8, the background is also combined with the face and output.

FIG. 6 shows a flowchart of the density correction operation on the receiving side.

Further, FIG. 7 shows a schematic example of an image corresponding to FIG. 2 which is image-converted by this embodiment.

〔The invention's effect〕

As described above, according to the present invention, a portion that does not exist in the model image such as the inside of the mouth is naturally synthesized and transmitted as isodensity line information that well represents the change in density due to the movement of the face.
The density change accompanying the shape conversion such as wrinkles is naturally synthesized, and a more natural moving image can be transmitted. In addition, since the face density information can be sent not as for all pixels but as compressed and chain-encoding isodensity line information,
The amount of transmitted information can be small, and the transmission efficiency can be improved.

[Brief description of drawings]

FIG. 1 is a block diagram showing an embodiment of the present invention. FIG. 2 is a block diagram for explaining the processing operation of the embodiment. 3 to 6 are flow charts for explaining each processing of the embodiment. FIG. 7 is a diagram showing an example of a processed image of the embodiment. 1, 9 ... caller, 2 ... imaging means, 3 ... image analysis means, 4
... sending means, 5 ... transmission path, 6 ... receiving means, 7 ... image synthesizing means, 8 ... image displaying means, 31 ... model image extracting means, 32
... motion analysis means, 33 ... iso-dense line extraction means, 71 ... model creation means, 72 ... motion synthesis means, 73 ... density correction means.

Claims (2)

[Claims] 1. A transmission side is provided with means for transmitting a model image prior to a call and means for analyzing a movement of an image during a call and transmitting movement information to the reception side. In the moving image transmission system, which is equipped with a means for synthesizing an image with movement by transforming the model image according to the information on the image movement transmitted from the side The iso-concentration line information received by the receiving side is provided with means for extracting the iso-concentration line information connected by lines in time series and means for sequentially transmitting the iso-concentration line information extracted by this means to the receiving side. The moving image transmission system is provided with means for correcting the density of the combined image according to. 2. The transmitting side comprises means for transmitting a model image prior to a call, and means for analyzing a movement of an image during a call and transmitting movement information to the receiving side. In the moving image transmission method, the model image is transformed by the information of the image movement transmitted from the side to synthesize an image accompanied by movement. It is equipped with a means to extract the contour lines connecting the lines and a means to encode the extracted contour lines, and to the receiving side, a means to restore the grayscale image from the received contour lines information, and a model image. Means for extracting the contour lines of the image synthesized by motion synthesis from the above, means for restoring the grayscale image of the synthesized image from the extracted contour lines, and the grayscale image restored from the received contour line information And synthesized from the model image above Moving image transmission method characterized by comprising means for calculating the density difference between the grayscale image of the image, and means for performing density correction of the synthesized image by the density difference.