JPH0760460B2 - Continuous viewing area image reproduction method - Google Patents

Description

Description: TECHNICAL FIELD The present invention relates to a reproducing method for obtaining a continuous-view object image at an arbitrary angle by rotating an object imaged by a camera by image processing.

(Prior Art) Conventionally, as a method of reproducing an image at an arbitrary angle from a plurality of images captured at different angles, a linear key frame method, a method using a density contour line, a method of rotating an object by image processing ( Hereinafter referred to as a rotation method). The linear key frame method will be described below.

FIG. 4 is a diagram showing a flow of processing of a linear key frame method using two original images (key frame images) captured by two cameras on the same plane. Since this basically targets a binary image, when a gradation image is used, the binarization processing (a) is performed after the left and right images are captured (a).

Next, the corresponding points are searched (c) from the left and right images that have undergone the binarization process. Generally, a correlation function is used to search for the corresponding points. Among the corresponding points found in (c), for example,
One of the right images is used as a reference point, and the left image is translated (D) so that the coordinates of the corresponding points of the left image on the two-dimensional plane coincide with the reference points of the right image.

After the parallel movement, the distance (e) between the corresponding points is calculated, and the internally dividing point that internally divides the corresponding points into m: n is calculated (f). This part
m and n are determined by the rotation angle. When the calculated internal division point is displayed (K), the target image after rotation can be obtained.

Then, an image viewed from an arbitrary angle can be obtained by appropriately selecting the internal divisions m and n between the corresponding points. As described above, the linear key frame method is a method of obtaining an image at an arbitrary angle by obtaining the internally dividing points from the corresponding points of the left and right binary images, but it can process only binary images. There are drawbacks.

Next, in the method using the density contour line, the density distribution of the gradation image is described by a binary line image, and the gradation image can be efficiently described. The density contour line extracted from the gradation image is defined as follows.

The gradation image F (X, Y) is set to n density thresholds t _k (k = 1,
2, ..., N), the binarization is performed as follows.

When F (X, Y) ≧ t _k B _k (X, Y) = 1 (black pixel) When F (X, Y) <t _k B _k (X, Y) = 0 (white pixel) k = 1,2, ..., n n binary images B _k (k = 1,2, ..., n obtained by the above equation
The density contour line is obtained by extracting the boundary line between each of the black and white image areas in n).

This is also handled by extracting n density contour lines from two original images (key frame images) captured by two cameras on the same plane and using the same method as the linear key frame method described above. An image at an arbitrary angle can be obtained by internally dividing the contour lines and reproducing a gradation image from the obtained n images of internally dividing points. Although this provides a natural gradation image, it has drawbacks such as requiring artificial means when searching for corresponding points and enormous calculation time.

Next, a rotation method for rotating an object by image processing will be described (for example, Japanese Patent Application No. 62-269027). In this rotation method, the object imaged by the camera is cut in the horizontal direction, its cross section is represented by a simple two-dimensional model, the imaged pixels are imaged on the model, and the model is rotated at an arbitrary rotation angle. It is a method of obtaining a desired image by rotating with. The case where the rotation method is used for the images captured by the two cameras will be described below.

FIG. 5 shows a spatial coordinate system of a human image of a subject with two cameras, and FIG. 6 shows the subject of FIG.
The relationship between the circular model and the two-dimensional coordinate system in the case of cutting in the z plane and using the circular model for the cut surface is shown.

Here, 1L is a left image camera, 1R is a right image camera, 2
Is the object, x ₀ is the center point of the model, point p is the two-dimensional coordinate point (x, j, z) before rotation, and p ′ is the two-dimensional coordinate point (x ′,
j, z '), r (function of y) is the radius of rotation, θ is the angle formed by the two cameras, α is the angle of rotation when the captured image is rotated to the desired position (the counterclockwise direction is positive) Yes, a is the leftmost coordinate point of the subject image, and b is the rightmost coordinate point of the subject image.

FIG. 7 is a diagram showing a flow of processing when, for example, a front face image is created using the circle model of FIG. 6, and details of the processing will be described below with reference to FIG.

(A) Edge and face contour extraction Left and right human face images captured by the two cameras 1L and 1R are digitized and taken in, and after applying the edge detection filter, only the face contour line To extract.

(A) Position correction Of the left and right images contour-extracted in (a), the left face contour line is moved in parallel so that it matches the reference point (for example, the crown) of the two-dimensional coordinate of the right face contour image. to correct.

(C) Cutting out the nose and creating a nose facing the front Cut out the nose from the left and right original images that were position-corrected in (A). The nose cutout processing includes a method of estimating and cutting out the nose from the contour line of the face, a method of recognizing and cutting out the nose along with eyes and mouth, and the like. Next, the nose muscles are detected from the cut out left and right noses ((c) -1). Using the detected right and left nose muscles, the frontal nose muscles are calculated by the key frame method ((C) -2). Finally, the pixels of the left and right original images are moved along the frontal nose muscle to create a frontal nose ((C) -3).

(D) Rotational movement From the left and right face contour lines whose position has been corrected in (b), the contour radius r
(Y) is calculated, and the pixels of each original image are rotationally moved to create a moving processed image.

(E) Creation of composite face contour From the left and right face contour lines that have been position-corrected in (a), a front face-facing composite face contour line is created using the key frame method.

(F) Compositing of moving processed images The left and right moving processed images are combined into one image.

(G) Synthesis of nose The nose created in (C) -3 is synthesized with the nose portion of the composite movement processing image.

(H) Correction by Synthetic Face Contour Line The face contour of the movement-processed image is corrected by using the synthetic face contour line obtained in (e).

Through the above processing, it is possible to obtain a front-facing image from the left and right two face images. As is clear from the flow of the processing, the above processing is performed by relatively simple image processing, so that a multi-valued image can be processed as compared with the conventional linear key frame method, and a method using a density contour line. Execution is possible in a short time compared with.

However, since this rotation method uses a circular model, the eyes, nose, and mouth move to the front, but the nose muscle remains bent. Therefore, as shown in the process of (c), only the nose of the uneven part is cut out from the left and right images and the key frame method is used together to create the front facing nose and the process (ki) to combine it with the final processed surface is required. Has the drawback that

(Object of the Invention) The present invention solves the above-mentioned drawbacks of the conventional rotation method,
It is an object of the present invention to obtain a reproduced image of an arbitrary rotation angle in a short time from a gradation image picked up by one camera or two cameras on the same plane by simple image processing.

(Structure of the Invention) (Differences between Features of the Invention and Prior Art) The present invention cuts a subject imaged by a camera in the horizontal direction, and expresses its cross section in a two-dimensional elliptical model, and the long axis of the elliptical model. And the minor axis is changed in the vertical direction, and the elliptical model is rotated at an arbitrary rotation angle to obtain a desired image in a short time, and the conventional linear key frame method is used or the density contour line is used. The method itself is different from the method. Further, the method of treating uneven portions such as the nose is different from the conventional rotation method.

(Example) As an example of the present invention, a case where a human image is used as a subject and the images are picked up by two cameras arranged on the same plane will be described.

FIG. 1 is a diagram in which the subject of FIG. 5 is cut along the xz plane where y = j and an elliptical model is applied to the cut surface of the subject.
In FIG. 1, the reference coordinate system is the x-z axis shown in the same figure, and the object before rotation is represented by an ellipse model with a short axis c and a long axis d. Angle α in the center
The figure shows an example in which the subject is imaged from the front by rotating the elliptical model corresponding to the case where the subject is imaged from an angle.

Generally, the coordinate system of the elliptical model before rotation is used as a reference,
The point p (x, j, z) is a coordinate point on the ellipse model before rotation, and the point p ′ (x ′, j, z ′) is an angle α about the center point O of the ellipse.
If the coordinate point is rotated by (counterclockwise is positive), the following equation holds.

Here, β is a function β (y) of the vertical axis y and is expressed by a simple function. FIG. 2 shows the functional form of β (y) in this embodiment. Β (y) is a constant value from the top of the nose to the upper part of the nose, and β (y) is increased by a linear function to express the height of the nose from the upper part of the nose to the neck part. It is reduced by a quadratic function to express the part. In this way, in addition to the height of the nose, even the shoulder portion can be represented by the elliptical model. When the captured image is rotated counterclockwise by -α on the xz plane, it can be obtained by substituting -α for α in the equation (2).

FIG. 3 is a diagram showing the flow of processing when, for example, a front face image is created using the ellipse model. The details of the processing of the rotation method using the ellipse model will be described below with reference to FIG. .

(A) Edge / face contour extraction The left and right human face images captured by two cameras are digitized and taken in, and after applying an appropriate edge extraction filter, a contour line is extracted.

(A) Position correction Among the left and right images whose contours have been extracted in (a), the left face contour line is moved in parallel to match the reference point (for example, the crown) of the two-dimensional coordinates of the right face contour image. Similarly, each pixel of the face original image on the left side is translated by the same amount as the face contour line, and the position is corrected not only for the face outline but for the entire face original image. The above position correction processing is the same as that of the conventional rotation method described above.

(C) Rotational movement Left and right person images (two-dimensional images) position-corrected in (A)
Are horizontally cut, and an ellipse model is applied to each pixel on the cutting line in the face contour line. This operation is repeated from the top of the face to the bottom. Specifically, the procedure is as follows. The distance between the leftmost point and the rightmost point, which are the intersections of the face contour line and the cutting line, is obtained and is defined as the minor axis c of the elliptical model.
In addition, the midpoint between the leftmost point and the rightmost point is obtained and used as the origin of the elliptical model. The value of β corresponding to the cutting line is obtained from FIG. 2,
From this β and the short axis c obtained above, the long axis d is obtained by the equation (3). As the angle α, an angle formed between the direction in which the left image or the right image is captured and the direction in which the image is to be reproduced is set. The x-coordinate corresponding to each pixel of the face original image on the cutting line is obtained, and the z-coordinate value is obtained by the above equation (1) from the short axis c and the long axis d of the elliptic model obtained in the above. Substituting the α, x-coordinate, and z-coordinate obtained in step (3) into equation (3), x'is obtained. The above is repeated for all the pixels on the cutting line in the face contour line, and all the pixels are converted into x ′. The above steps (1) to (5) are executed for all scanning lines below the parietal region to convert the entire face original image. As described above, the left and right rotational movement processed images rotated counterclockwise by α are created.

In the above description, the distance between the leftmost point and the rightmost point, which is the intersection of the face contour line and the cutting line, is defined as the short axis c of the elliptical model. Alternatively, the long axis d may be used.

The above-described method of allocating to the ellipse model is most applicable when the subject is photographed from the front, but since it can be realized by relatively simple processing, if α is not too large, it is sufficient even when photographed from an angle. This is an effective method.

(D) Creating a synthetic face contour The left and right face contours that have been position-corrected in (a) are cut horizontally, and the intersection of the left part of the left face contour and the cutting line and the left part of the right face contour are cut. Find the intersection of the cutting lines, and use this as the corresponding point,
The key frame method described above is applied to find the point internally divided to m: n. The ratio of m: n is determined in correspondence with the angle α described in (c) above, and the angle formed by the direction in which the left image is captured and the direction in which the image is reproduced and the direction in which the right image is captured. And the angle formed by the direction of reproduction. In the above (a), when two left and right face images are taken by installing two cameras at the same angle to the left and right of the front of the subject, to reproduce the front face image, m = 1 and n = 1, which is a point to be internally divided into 1: 1. Similarly, an intersection between the right part of the left face contour line and the cutting line and an intersection of the right part of the right face contour line and the cutting line are obtained, and a point internally divided into 1: 1 is obtained. By repeating the above operation from the top end to the bottom end of the face contour line, a front face-combined face contour line is created. The above-described synthetic face contour creation processing is the same as the above-described conventional rotation method.

(E) Compositing of movement processed images One image is composed from the rotation movement processed images of (c).
The composition ratio is changed according to the rotation angle. In the present embodiment, in order to create a front image, the composite face contour line created in (d) above is cut in the horizontal direction, and the intersection of the left part of the composite face contour line and the cutting line and the right part of the composite face contour line are cut. Find the midpoint from the intersection of the cutting line and. The above operation is repeated from the uppermost end to the lowermost end of the synthetic face contour line to obtain the vertical center axis of the face consisting of a set of midpoints. Then, the left movement processed image is assigned to the left side of the central axis, and the right movement processed image is assigned to the right side, and the left and right movement processed images are combined.

(F) Correction by composite face contour line The face contour of the movement-processed image is corrected by using the composite face contour line obtained in (d). In other words, the correction is performed by fitting the composite face contour line of (e) to the movement processed image of (e), ignoring the pixels protruding from the composite face contour line, and replacing it with the background image. Note that when the image is picked up by one camera, this moving process image combination process is unnecessary.

By the above processing, the front-faced images from the two left and right face images are combined. Further, an image at any angle other than the front can be combined by the same process except for the combining method of the moving processed image. The moving processed images are combined by changing the ratio of the combined areas of the left and right processed images according to the rotation angle.

As is clear from the flow of the process, the above process is performed by simple image processing, and can be executed in a short time as compared to the conventional rotation method, because no special process for uneven parts such as the nose is required. .

(Effects of the Invention) As described above, according to the present invention, by capturing an image of a single subject in which two left and right cameras are arranged on the same plane, and by rotating the captured subject at an arbitrary angle, A desired image can be reproduced in a short time. If the present invention is applied to an image database, a rotated image can be created from two subject images captured by two cameras that are practically on the same plane, and all the rotated images are recorded in the database. Effective in effective use of the database compared to.

Further, conventionally, the videophone device has a drawback that the lines of sight of the two do not match, but the present invention is applied to a videophone device having two cameras, and from two images taken by the two cameras, By combining front-facing images using the rotation method of the present invention and transmitting the composite images to the other party, a videophone device in which the lines of sight of the two are matched can be realized.

[Brief description of drawings]

FIG. 1 is a diagram showing a relationship between an ellipse model and a two-dimensional coordinate system when the subject of FIG. 5 is cut in the xz plane where y = j and the cut surface is applied with the elliptical model used in the present invention. Second
FIG. 3 is a diagram showing a function form of β, FIG. 3 is a diagram showing a processing flow of a rotation method using an elliptic model according to the present invention, and FIG. 4 is a diagram showing a processing flow of a conventional linear key frame method, FIG. 5 is a diagram showing a spatial coordinate system of a subject and a camera when the rotation method is used, and FIG. 6 is a cross-sectional view of the subject of FIG. 5 cut along the xz plane where y = j, and a circular model is formed on the cut surface. FIG. 7 is a diagram showing the relationship between the circular model and the two-dimensional coordinate system when using, and FIG. 7 is a diagram showing the flow of processing using the circular model. 1 ... Imaging camera, 1L ... Left image camera, 1R ... Right image camera, 2 ... Subject.

Claims (1)

[Claims] 1. An object is imaged by two cameras on the same plane, and an object is imaged from an arbitrary position between the left image and the right image, which are two-dimensional images of the obtained two objects. A continuous viewing area image reproduction method for automatically reproducing a two-dimensional image of a subject obtained in this case, wherein a left contour line which is a contour line of the left image and a right contour line which is a contour line of the right image are extracted. And a step of correcting the position by translating the left image and the right image in parallel so that corresponding points of the left contour line and the right contour line coincide with each other, and an ellipse of a cut section of the subject in a horizontal plane. To represent with a model, first, for the position-corrected left image,
The left contour line is cut in the horizontal direction, and the intersection line of the cutting line and the left contour line and the midpoint of the intersection line are taken as the minor axis (long axis) and the center point of the ellipse model, respectively, and the ellipse The shape of the elliptical model is determined by previously giving a value obtained by changing the ratio of the major axis to the minor axis of the model in the vertical direction according to the shape of the subject, and the left image on the intersection line in the left contour line is determined. Each of the above pixels is fitted to a point on the ellipse model, the ellipse model is rotated about the center point by an angle corresponding to the position of the two-dimensional image to be reproduced, and the ellipse model is rotated. A point on the ellipse model is projected in the short axis (long axis) direction before the rotational movement, and this operation is repeated from the uppermost end to the lowermost end of the left contour line, and then the above steps are performed on the right image. The left image and the right image by executing A process of creating a rotational movement processing image that is an image in which each of them is rotationally moved, and each of the left contour line and the right contour line is horizontally cut, and the cutting line and the left part of the left contour line are cut. Between the intersections of the left part of the right contour line and between the intersections of the cutting line and the right part of the left contour line and the right part of the right contour line, from the left image when the rotational movement processing image is created. Of the left contour line and the right contour line by repeating the operation from the top edge to the bottom edge of the left contour line and the right contour line. And a step of creating a composite contour line that is a contour line corresponding to the two-dimensional image, in which the composite contour line is cut in the horizontal direction, and an interior division point between intersections of the cut line and the composite contour image is determined. , Obtained at the same ratio as the process of creating the composite contour line, By repeating this operation from the uppermost end to the lowermost end of the composite contour line, a vertical centerline composed of the set of internal division points is obtained, and the left image and the right image are rotationally moved with this centerline as a boundary. And a step of performing contour correction by applying the synthesized contour line to an image obtained in the step of synthesizing by allocating the rotational movement processing image. Continuous viewing zone image reproduction method.