JP2000251074A - Method and device for image generating processing - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily generate the synthetic image of two objects watched from any arbitrary viewpoint. SOLUTION: The image of the head is photographed by a TV camera 2 while letting a person sit on a motor-driven rotary chair 1 and rotating the chair, images for one rotation are stored in a storage part 3 for multi-viewpoint images, and a rotation angle at every point of time of photographing the image is reported from the motor-driven rotary chair 1 to an acquisition part 4 for viewpoints. In a face image generating part 5, the viewpoint closest to a viewpoint arbitrarily designated by a viewpoint setting part 8 is selected, the area of the face in the multi-viewpoint images corresponding to the viewpoint is extracted and outputted and in a position aligning and synthesizing part 6, first of all, a rotary axis in the image is detected on the basis of the image in the face area. Next, the lowest part of the jaw is detected by a jaw position detecting part 6b in respective front and side images after area extraction. In an image synthesizing part 6d, the image of a human body and the image of the head after position alignment are synthesized.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image generation processing method and apparatus, and more particularly, to an image generation processing method and apparatus based on a plurality of still images obtained by photographing an object from a plurality of viewpoints.
The present invention relates to a method and an apparatus for generating an image viewed from an arbitrarily set viewpoint, and a method and an apparatus for synthesizing an image generated in this way by aligning an image obtained when another object is viewed from the same viewpoint. .

[0002]

2. Description of the Related Art For example, in the apparel industry, it is desired to realize a new production / distribution system in which a customer's request is promptly reflected in one-item production by linking a three-dimensional clothing simulation with a three-dimensional CAD. However, at that time, the customer's own face was photographed at the store, and it was
If the image of the clothes created by (computer graphics) is synthesized and displayed three-dimensionally, it is possible to more accurately grasp the appearance of the customer wearing the clothes. As another example, it is desired to realize a system that can quickly find favorite glasses by combining the glasses created by CG and the face of the customer. In addition, the target is not necessarily limited to the face. For example, based on a photographed image of an art work such as a pot or a craft from a plurality of viewpoints, C
There are many demands for a system that realizes an electronic museum by generating an image viewed from an arbitrary viewpoint inside the museum created in G.

In order to realize such a system,
Based on images of certain faces and pots taken from multiple viewpoints,
There is a need for a technique for generating an image of the object as viewed from an arbitrary viewpoint, and combining the generated image with a human body or eyeglasses generated by, for example, CG, or with the interior of a museum.

In order to meet such demands,
In order to combine the first thing and the second thing together,
A method has been proposed in which both images viewed from a fixed viewpoint are aligned by manual operation.

If this method is adopted, for example, when aligning the position of the head with the clothing image, two images, that is, an image of a person's head taken from the front and a clothing image viewed from the front, are taken. By preparing the head image and moving the head image in parallel using a mouse or the like, the position of the head image is aligned with the clothing image, and then the two can be combined.

[0006]

However, when this method is adopted, there is an inconvenience that it requires labor. In particular, in order to generate a composite image from an arbitrary viewpoint, it is necessary to manually perform alignment at each viewpoint, which is disadvantageous in that the labor is enormous.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and a first object of the present invention is to provide an image generation processing method and an apparatus for easily generating an image of an object viewed from an arbitrary viewpoint. 2 from an arbitrary viewpoint
It is a second object of the present invention to provide an image generation processing method and apparatus capable of easily generating a composite image of two things.

[0008]

According to a first aspect of the present invention, there is provided an image generation processing method comprising: a multi-view image in which a first object is viewed from a plurality of viewpoints having different positions and orientations relative to the first object; While storing the positions and orientations of the plurality of viewpoints, and in response to arbitrarily setting the viewpoints, based on the multi-viewpoint image and the positions and orientations of the plurality of viewpoints, This is a method of generating an image of a first thing viewed from the set viewpoint.

The image generation processing method according to a second aspect of the present invention stores a multi-view image in which a first object is viewed from a plurality of viewpoints having different positions and orientations relative to the first object. The positions and orientations of the plurality of viewpoints are acquired, and in response to setting the viewpoints arbitrarily, based on the multi-viewpoint image and the positions and orientations of the plurality of viewpoints, from the set viewpoints. Generating a first image of the viewed first thing; generating a second image of a second thing that is different from the first thing as viewed from the set viewpoint; In this method, the first image and the second image are aligned with each other and combined to generate a combined image.

An image generation processing apparatus according to a first aspect of the present invention is a multi-viewpoint image storing a multi-viewpoint image in which a first thing is viewed from a plurality of viewpoints different from each other in position and orientation relative to the first thing. Storage means, viewpoint acquisition means for acquiring the positions and orientations of the plurality of viewpoints, viewpoint setting means for arbitrarily setting viewpoints, and setting based on the multi-viewpoint image and the positions and orientations of the plurality of viewpoints. And a first image generating means for generating an image of the first thing as viewed from the point of view.

An image generation processing apparatus according to a second aspect of the present invention is a multi-view image storing a multi-view image in which a first object is viewed from a plurality of viewpoints having different positions and orientations relative to the first object. Storage means, viewpoint acquisition means for acquiring the positions and orientations of the plurality of viewpoints, viewpoint setting means for arbitrarily setting viewpoints, and setting based on the multi-viewpoint image and the positions and orientations of the plurality of viewpoints. First image generating means for generating a first image of a first thing viewed from the set viewpoint, and a second image of a second thing different from the first thing viewed from the set viewpoint. A second image generating unit for generating an image; and a composite image generating unit for generating a composite image by performing positioning and combining of the first image and the second image in accordance with the set viewpoint.

[0012]

According to the image generation processing method of the first invention, the first method
In addition to storing a multi-view image in which the object is viewed from a plurality of viewpoints different from each other in relative position and orientation with respect to the first object, the positions and orientations of the plurality of viewpoints are acquired, and In response to the setting of the viewpoint, the image of the first thing viewed from the set viewpoint is generated based on the multi-view image and the positions and orientations of the plurality of viewpoints. , It is possible to easily generate a high-quality image of the first thing viewed from the viewpoint.

According to the image generation processing method of the second invention,
A multi-view image of the first thing viewed from a plurality of viewpoints different from each other in relative position and orientation with respect to the first thing is stored, and the positions and directions of the plurality of viewpoints are acquired. In response to arbitrarily setting the viewpoint, based on the multi-view image and the positions and orientations of the plurality of viewpoints,
Generating a first image of a first thing viewed from the set viewpoint, and generating a second image of a second thing different from the first thing viewed from the set viewpoint; Since the first image and the second image are aligned according to the set viewpoint and combined to generate a combined image, a combined image of two things viewed from an arbitrary viewpoint can be easily created. Can be generated.

According to the image generation processing apparatus of the first invention,
The multi-viewpoint image storage means stores a multi-viewpoint image of the first thing viewed from a plurality of viewpoints different from each other in relative position and orientation with respect to the first thing, and the viewpoint acquisition means stores the plurality of viewpoints. The position and orientation of. If the viewpoint is arbitrarily set by the viewpoint setting means,
The first image generating means can generate an image of the first thing viewed from the set viewpoint based on the multi-view image and the positions and orientations of the plurality of viewpoints.

Therefore, a high-quality image of the first object viewed from an arbitrary viewpoint can be easily generated.

According to the image generation processing apparatus of the second invention,
The multi-viewpoint image storage means stores a multi-viewpoint image of the first thing viewed from a plurality of viewpoints different from each other in relative position and orientation with respect to the first thing, and the viewpoint acquisition means stores the plurality of viewpoints. The position and orientation of. If the viewpoint is arbitrarily set by the viewpoint setting means,
The first image generation means can generate a first image of a first thing viewed from the set viewpoint based on the multi-view image and the positions and orientations of the plurality of viewpoints. The second image generating means generates a second image of the second thing as seen from the set viewpoint, the second thing different from the first thing, and the synthesized image generating means generates the second image of the second thing. The first image and the second image can be aligned and synthesized according to the viewpoint to generate a synthesized image.

Therefore, a composite image of two things viewed from an arbitrary viewpoint can be easily generated.

[0018]

Embodiments of the present invention will be described below in detail with reference to the accompanying drawings.

FIG. 1 shows a first embodiment of an image generation processing apparatus according to the present invention.
FIG. 3 is a block diagram showing an embodiment of the present invention.

This image generation processing apparatus includes a human head and C
The clothing image created in G is aligned at an arbitrary viewpoint to synthesize an image.

In this image generation processing apparatus, first, an image of the head is taken by the TV camera 2 while the person is sitting on the electric swivel chair 1 and rotating. Then, an image for one round, for example, 90 images every 4 ° is stored in the multi-viewpoint image storage unit 3. The electric swivel chair 1 notifies the viewpoint acquisition unit 4 of the rotation angle at each time of capturing an image. In this way, it is recognized from which direction (viewpoint) each of the 90 images was taken with respect to the human head.

Next, the face image generation unit 5 selects the viewpoint closest to the viewpoint arbitrarily designated by the viewpoint setting unit 8 from the viewpoints acquired by the viewpoint acquisition unit 4, and corresponds to the viewpoint. A face area is extracted and output from the multi-viewpoint image to be output.

In the alignment synthesizing unit 6, first, based on the image output from the face area extracting unit 5b, the rotation axis detecting unit 6a
Detects the rotation axis in the image. FIG. 2 shows the rotation axis detection process.

The center of gravity calculating section calculates the position of the center of gravity of the face area portion of the image at each viewpoint output from the face area extracting section 5b. The temporary rotation axis calculation unit calculates the average of the center of gravity at each viewpoint, and calculates the temporary rotation axis on the assumption that the rotation axis passes through the average position of the center of gravity. The feature point extraction unit extracts, as feature points, a portion having a large variation in brightness from the image after region extraction at each viewpoint. More specifically, the Gaussian Laplacian processing {the original image is first blurred by a normal function (Gaussian function), and then the Laplacian is applied} Are extracted as feature points. Feature point 3
The three-dimensional coordinate calculator calculates three-dimensional coordinates of a plurality of feature points based on the provisional rotation axis and the feature points at each viewpoint. Specifically, three-dimensional coordinates are calculated on the assumption that the visual axis of the TV camera 2 passes through the rotation axis. In the rotation axis calculation unit, the distance between the projected point and the feature point extracted by the feature point extraction unit when projecting the three-dimensional coordinates of the calculated feature point onto the image at each viewpoint is minimized. It calculates how much the visual axis of the TV camera 2 is deviated from the rotation axis based on the result, and corrects the position of the rotation axis based on the result.

Next, in the position synthesizing unit 6, the lowermost part of the chin is detected by the chin position detecting unit 6b in each of the front image and the side image after the region extraction. FIG. 3 shows the flow of the processing.

First, as shown in FIG. 3A, in the front image, the hair region is deleted based on the color difference between the hair region and the skin region, and only the skin region is left. Next, in the skin region, the lips are detected based on the fact that the lips are substantially below the skin region and are oriented in the horizontal direction, and the center position thereof is detected. Thereafter, the lowermost part of the chin is searched downward starting from the center of the lip. At this time, the point at which the brightness of the image changes abruptly in the downward direction is determined to be the lowermost part of the chin. Next, (b) in FIG.
As shown in (2), the lowermost part of the chin is detected in both left and right images. For example, in the right side image, the hair region is deleted in the same manner as described above, and then the lower right portion of the skin region is detected as the lower portion of the chin. In the left side image, the lowermost part of the jaw is detected by the same processing. Based on the lowermost position of the jaw detected in this way, the three-dimensional lowermost position of the jaw is calculated based on the principle of triangulation. In the positioning unit 6c, the head image is moved so that the rotation axis of the head coincides with the rotation axis of the human body, and the head image is moved so that the lowermost part of the chin is at a predetermined position with respect to the human body part. To move further.

The image synthesizing unit 6d synthesizes the image of the human body and the image of the head after the alignment.

When the target is other than a face, an electric rotating base is used. Also, instead of the lowermost part of the chin, a part that is easily detected according to the target is detected.

The human body image generation section 7 performs a three-dimensional human body image rotation process based on the set viewpoint to generate a human body image corresponding to the viewpoint.

In the case of adopting this embodiment, the position of the viewpoint of the TV camera 2 is obtained by the notification from the electric swivel chair, so that there is an advantage that the positioning accuracy is high.

FIG. 4 shows a second embodiment of the image generation processing apparatus according to the present invention.
FIG. 3 is a block diagram showing an embodiment of the present invention. This image generation processing apparatus differs from the image generation processing apparatus of FIG. 1 in that an electric swivel chair 1 that rotates at a known speed pattern is adopted, and a TV camera based on the speed pattern is used instead of the viewpoint acquisition unit 4. This is the only point that employs the viewpoint calculation unit 4a that estimates the position of the second viewpoint.

Therefore, when this embodiment is employed, the notification of the viewpoint from the electric swivel chair 1 becomes unnecessary, and the configuration can be simplified.

FIG. 5 shows a third embodiment of the image generation processing apparatus according to the present invention.
FIG. 3 is a block diagram showing an embodiment of the present invention. This image generation processing apparatus is significantly different from the image generation processing apparatus of FIG. 4 in that the rotary chair 1a rotated by inertia is used instead of the electric rotary chair 1.
This is the point adopted. When the swivel chair 1a is rotated by inertia with momentum, the rotation speed gradually decreases due to frictional resistance. In this embodiment, the viewpoint at each time point is obtained on the assumption that the rotation speed decreases at a constant rate.

As shown in FIG. 5, this viewpoint is obtained through the flow of similarity calculation, image detection after one rotation (image detection after one round), speed / acceleration calculation, and viewpoint calculation. In this embodiment, it is assumed that an image of about one half turn or more is input in order to calculate the viewpoint.

The similarity calculation unit 4b evaluates the similarity between the images of the multi-viewpoint image. Specifically, as shown in FIG. 6, first, a face area is extracted from each of the multi-viewpoint images, and then the pixel values of the face area are vertically accumulated by the projection image calculation unit. In this way, a projected image is calculated for each of the multi-viewpoint images. After that, the similarity between the projected images is evaluated by the correlation coefficient. That is, it is determined that the larger the correlation coefficient between the projected images is, the more similar the images are.

The one-round image detection unit 4c uses the fact that the images of the first one-half circle are similar to each other for the first half-circle images, so that the image of the first one-half circle can be detected. To detect. That is, it is determined that the image having the highest similarity is the image just after one round.

Next, the speed / acceleration calculating section 4d calculates the speed and acceleration for each of the first half-circle images in accordance with the number of subsequent images that are exactly one round later. calculate. Since the rotation speed gradually decreases, the time required for one rotation gradually increases, and accordingly, the number of images required for one rotation also gradually increases. In addition, the rate of reduction in speed is obtained from the change in the number of images required for one rotation.

When the rotational speed and the acceleration are obtained in this manner, the viewpoint calculation unit 4e determines the viewpoint of each image of the multi-viewpoint image (at what angle the image was taken at which angle) based on the rotation speed and the acceleration. calculate.

The processing other than that of the viewpoint obtaining unit 4 is the same as that of the first embodiment, and a description thereof will be omitted.

When this embodiment is adopted, the swivel chair 1a is used instead of the electric swivel chair 1, so that
The configuration can be simplified.

FIG. 7 shows a fourth embodiment of the image generation processing apparatus according to the present invention.
FIG. 3 is a block diagram showing an embodiment of the present invention.

This embodiment is largely different from the third embodiment in that the multi-viewpoint image is inputted by rotating the swivel chair 1a by applying a force little by little by human power, instead of rotating the swivel chair 1a by momentum. This is the only point that the content of the processing of the viewpoint obtaining unit is set in accordance with the difference in the method of rotation and the method of rotation.

In this embodiment, a viewpoint is obtained by the flow of similarity calculation, one-round image detection, average speed calculation, and viewpoint calculation. The processing up to the detection of the image after one rotation is the same as that of the third embodiment, but the number of images to be input is not necessarily about one and a half, and may be smaller if it is one or more. However, as the number of images increases, the accuracy of the calculated viewpoint increases. The average speed calculation unit 4f calculates the average of the number of images required to make exactly one round, for each of the first plurality of images, according to how many images after the one round obtained by the preceding processing. Is calculated to determine the average speed. The viewpoint calculation unit 4e calculates a rotation angle (viewpoint) when each of the multi-view images is photographed, on the assumption that the swivel chair 1a is rotating at the average speed.

When this embodiment is adopted, there is no need to rotate the swivel chair 1a with momentum, so that there is an advantage that the impact received by the occupant during rotation can be reduced.

FIG. 8 shows a fifth embodiment of the image generation processing apparatus according to the present invention.
FIG. 3 is a block diagram showing an embodiment of the present invention.

This embodiment is significantly different from the fourth embodiment in that the marker 1b attached to the swivel chair 1a is detected to obtain a viewpoint, and the rotation axis is detected by the positioning unit 6. Is performed using the detected marker.

In this embodiment, the viewpoint is acquired by the flow of marker detection and viewpoint detection. The marker detection unit 4g detects the position of the known marker 1b attached to the swivel chair 1a in each of the multi-viewpoint images. In the viewpoint calculation unit 4e, the marker 1b detected in each viewpoint
, The viewpoint (rotation angle) when each multi-viewpoint image is input is calculated. The markers 1b are, for example, written at equal intervals on the side surface of a thin disk centered on the swivel chair 1a, and the position (angle from the front) can be specified by the pattern of the markers 1b. By determining the pattern when the marker 1b is detected, the rotation angle of the swivel chair 1a at the time when the image is captured is calculated.
That is, after determining the angle of the pattern of the marker 1b located at the center and calculating a rough angle, the angle is corrected based on the position of the marker 1b.

The rotation axis detector 6a detects the positions of the markers 1b on both sides in each multi-viewpoint image, and averages the positions at which the two points are equally divided in all the multi-viewpoint images, thereby obtaining the rotation axis. Is detected.

When this embodiment is employed, even if the rotation speed is irregular, the rotation angle at the time of capturing each multi-viewpoint image can be calculated with higher accuracy, and the rotation axis detection accuracy can be improved. Positioning can be performed with high accuracy.

FIG. 9 shows a sixth embodiment of the image generation processing apparatus according to the present invention.
FIG. 3 is a block diagram showing an embodiment of the present invention.

This embodiment is significantly different from the fourth embodiment in that a viewpoint acquiring unit 4 extracts feature points from each multi-view image, acquires a viewpoint based on the positions thereof, and detects a rotation axis. Is the only point that is performed.

In this embodiment, the viewpoint is obtained in the sequence of feature point extraction, feature point tracking, and viewpoint calculation. The feature point extraction unit 4h performs the same processing as the feature point extraction of FIG. The feature point extraction unit 4i cuts out, for each of the extracted feature points, a density pattern of, for example, a 7 × 7 rectangular area centered at the point from the multi-view image, and selects between the multi-view images having different viewpoints. It is determined that the points where these 7 × 7 density patterns are similar to each other are the same feature point, and how each feature point moves as the angle at the time of shooting changes is tracked. I do. In the viewpoint calculation and the detection of the rotation axis, the viewpoint (rotation angle) at which each multi-view image is captured is calculated based on the manner in which each feature point moves between the multi-view images, and the rotation is calculated. Calculate the position of the axis.

The method of calculating the rotation angle and the rotation axis will be described in detail with reference to FIG.

FIG. 10 is a top view of an object rotating in front of the TV camera 2. At the time of capturing the m-th image, the i-th feature point is located at a distance ri from the rotation axis, and the angle with respect to a reference axis parallel to the visual axis of the TV camera 2 is θi + θm. Where θm
Is the direction of the front of the face, for example. Also, the distance between the rotation axis and the visual axis is e, the distance between the viewpoint and the perpendicular leg lowered from the rotation axis to the visual axis is d, the distance between the viewpoint and the image plane (focal length) is f, and the i-th feature. Let x be the x coordinate of the point where the point is observed. At this time, as can be seen from the figure, f / xim = {d-ri · cos (θi + θm)} /
There is a relationship of {ri · sin (θi + θm) + e}. If the number of images is M and the number of feature points is N, M × N such relational expressions hold. However, in practice, the feature points may not be observed because they are hidden, so the number of relational expressions is smaller than this. On the other hand, since the focal length f is generally known, the number of unknowns is N for ri, N for θi, M for θm, and 1 for each of e and d.
And a total of (2N + M + 2). Therefore,
When N and M are sufficiently large, the number of relational expressions is greater than the number of unknowns, and thus unknowns can be obtained from these relational expressions. Thus, the position d of the rotation axis and the rotation angle θm are obtained.

FIG. 11 is a block diagram showing a seventh embodiment of the image generation processing device of the present invention.

This embodiment differs greatly from the first embodiment only in that the TV camera 2 is rotated so as to surround the chair 1c instead of rotating the chair.

In this embodiment, the TV camera 2 is rotated by an electric motor or the like, and the position of the viewpoint is notified to the viewpoint acquisition unit 4. Other processes are the same as those in the first embodiment.

FIG. 12 is a block diagram showing an eighth embodiment of the image generation processing apparatus according to the present invention.

This embodiment is significantly different from the seventh embodiment in that the TV camera 2 is previously arranged at a plurality of positions so as to surround the chair on a horizontal plane across the head instead of rotating the TV camera 2. The only thing to keep in mind.

In this embodiment, the position of the viewpoint is determined based on which TV camera 2 performs photographing.
Notify. Other processes are the same as those in the first embodiment.

FIG. 13 is a block diagram showing a ninth embodiment of the image generation processing device of the present invention.

This embodiment is significantly different from the fifth embodiment only in that a multi-viewpoint image is photographed while the TV camera 2 is freely moved manually, for example, so as to surround a person in the horizontal direction. It is.

In this embodiment, since the positional relationship between the TV camera 2 and the person is not necessarily a rotation, the configuration of the positioning synthesizing unit is different from that of the embodiment shown in FIG.

In the coordinate system fixed to the chair 1c, i
Xi = (xi, yi,
zi), the i-th marker 1 in the m-th multi-viewpoint image
The position where b is observed is (xim, yim), and the position of the viewpoint when the m-th image is captured is Um = (um, vm,
wm), a 3 × 3 rotation matrix that determines the orientation of the TV camera 2 when the m-th image is shot is Rm, the focal length is f, and Xim = (xim, yim, zim). The equation holds. Xim = {fRm (Xi-Um)} / {(Xi-Um)
Here, assuming that the number of markers is N and the number of images is M, the number of these relational expressions is N × M. Actually, the number of relational expressions is smaller than this, since the marker may be hidden behind and cannot be seen. On the other hand, the number of unknowns is Rm M × 3,
Um is M × 3, Xi is N × 3, and a total of (6M
+ 3N). Therefore, when N and M are sufficiently large, the number of relational expressions is greater than the number of unknowns, and these unknowns can be obtained from these relational expressions.

The alignment synthesizing unit first detects the lowermost part of the chin in the front image and the side images in the same manner as described in the first embodiment (see FIG. 3). Next, based on the principle of triangulation based on the principle of triangulation, based on the position and orientation of the viewpoint of the TV camera 2 when these three images were taken, and the position of the lowermost part of the jaw in each image, Calculate the three-dimensional position. Other processes are the same as in the fifth embodiment.

FIG. 14 is a block diagram showing a tenth embodiment of the image generation processing apparatus according to the present invention.

This embodiment is significantly different from the ninth embodiment in that a marker is not attached to a chair, and instead of detecting the marker, feature points are extracted from each of the multi-viewpoint images and the feature points are multiplied. Only points that are tracked between viewpoint images.

Here, the extraction of the feature points and the tracking of the feature points are the same as in the fifth embodiment, and the subsequent processing is the same as in the ninth embodiment.

FIG. 15 is a block diagram showing an eleventh embodiment of the image generation processing device according to the present invention.

This embodiment is significantly different from the tenth embodiment only in that the face image generation unit 5 includes a coordinate conversion unit 5c. FIG. 16 shows the flow of the process performed by the coordinate conversion unit 5c.

First, feature points are extracted from each of the multi-viewpoint images after the region extraction by the method described in the first embodiment (see FIG. 2). FIG. 17 shows the flow of the three-dimensional coordinate calculation. In the feature point pair selection, two randomly selected multi-viewpoint images are selected, and feature points are randomly selected one by one from feature points extracted from each of the images. In the calculation of a three-dimensional feature point candidate, it is assumed that the two feature points are obtained by observing the same feature point on the object, and the three-dimensional coordinates of the feature points are calculated based on the principle of triangulation. In the verification, the calculated three-dimensional feature point candidates are projected on each multi-viewpoint image, the number of multi-viewpoint images in which feature points are actually observed near the projected point is counted, and the number is counted as a predetermined number. When the value is equal to or larger than the threshold value, the three-dimensional feature point candidate is output. Otherwise, a feature point pair is selected again. Such a hypothesis / verification process is repeated until a predetermined number of three-dimensional feature points are extracted. After calculating a predetermined number of three-dimensional coordinates,
Projecting them on the image at the set viewpoint and the image at the viewpoint selected by the method described in the first embodiment;
Calculate projection feature points. The coordinate conversion coefficient unit calculates a coordinate conversion for converting the projection feature point at the selected viewpoint into the projection feature point at the set viewpoint. The coordinate transformation execution unit transforms and outputs the multi-viewpoint image after the region extraction at the selected viewpoint in accordance with the remaining coordinate transformation.

FIG. 18 is a block diagram showing a twelfth embodiment of the image generation processing apparatus according to the present invention.

This embodiment is significantly different from the eleventh embodiment in that the face image generating unit 5 uses a plurality of (for example, a plurality of (for example, (2) viewpoints are selected, and a multi-viewpoint image from which two regions have been extracted is interpolated and synthesized and output.

The multiple viewpoint selecting section 5d selects, for example, two viewpoints close to the set viewpoint. Interpolation synthesis unit 5e
19 is shown in FIG. The processing up to the calculation of the three-dimensional coordinates is the same as in the eleventh embodiment. After that, the calculated three-dimensional coordinates are projected on an image at the set viewpoint and on a selected image at, for example, two viewpoints. Next, a coordinate conversion coefficient for converting the selected projection feature point projected on the image at the two viewpoints into the projection feature point projected on the image at the set viewpoint is, for example, 2
Calculate pairs. The coordinate transformation execution unit performs coordinate transformation on the calculated multi-viewpoint image, for example, after extracting a region at two viewpoints, selected according to, for example, two sets of coordinate transformation coefficients.
The interpolation unit averages and outputs, for example, two images that have been subjected to the coordinate conversion.

FIG. 20 is a block diagram showing a thirteenth embodiment of the image generation processing apparatus according to the present invention.

This embodiment differs greatly from the twelfth embodiment only in the configuration of the alignment synthesizing unit 6.

The alignment synthesizing unit 6 in this embodiment
Then, the alignment is manually performed according to the procedure shown in FIG.

First, for example, a head and a human body are superimposed and displayed on a screen at a viewpoint corresponding to the front and a viewpoint corresponding to the side. Next, the human head is moved using the mouse or the like by the manual positioning unit 6c 'so that the head of the person is naturally connected to the human body. Based on the movement amount, a movement amount in a three-dimensional space is calculated according to the principle of triangulation. The three-dimensional coordinates calculated by the method described in the twelfth embodiment are moved based on the calculated three-dimensional movement amount. In the calculation of the projection feature points, the moved three-dimensional coordinates are projected on the image at the set viewpoint. In the coordinate conversion coefficient calculation unit, the twelfth
In the method described in the embodiment, a coordinate conversion coefficient for converting the projection feature point at the selected viewpoint calculated by the face image generation unit into the projection feature point calculated here is calculated. The coordinate transformation executing section performs coordinate transformation on the interpolated image generated by the method described in the twelfth embodiment according to the calculated coordinate transformation coefficient, and outputs the result.

FIG. 22 is a block diagram showing a fourteenth embodiment of the image generation processing apparatus according to the present invention.

In this embodiment, after the positioning is performed by the method described in the twelfth embodiment, the result is displayed on the screen, and the image synthesizing / confirming unit 6e allows the operator to judge the quality. Only when it is not good, the thirteenth
The manual alignment is performed by the method described in the embodiment.

FIG. 23 is a block diagram showing a fifteenth embodiment of the image generation processing apparatus according to the present invention.

In this embodiment, a multi-viewpoint image is input by shooting while freely moving the TV camera 2 without necessarily moving in the horizontal direction. In this way, not only one point but also three or more predetermined characteristic points such as the lowermost part of the chin and the corners of both eyes are extracted, and these characteristic points are positioned at predetermined positions with respect to the human body. Is to be aligned.

When the TV camera 2 is freely moved, it is not always possible to perform positioning only by parallel movement, and it is necessary to adjust the direction and size of the head.
Therefore, it is necessary to extract three or more feature points.

Specifically, in the three feature point extracting unit 6f, the lowermost part of the chin in each of the plurality of multi-viewpoint images,
The three points at the outer corners of both eyes are extracted. Next, the three-dimensional position calculation unit 6g
, The three-dimensional coordinates of these three points are calculated based on the principle of triangulation. Thereafter, alignment is performed in the procedure shown in FIG.

First, the amount of translation, the amount of rotation, and the amount of movement of the extracted three feature points at a predetermined position with respect to the human body part.
The three-dimensional movement amount of the amount of expansion and contraction is calculated, and the movement amount of the three-dimensional coordinates of the feature amount calculated by the face image generation unit is calculated based on these movement amounts. Thereafter, a coordinate conversion coefficient for converting the projection feature point calculated by the face image generation unit into the projection feature point calculated here is calculated, coordinate-converted based on the coefficient, and output.

[0086]

The present invention has a unique effect that a high-quality image of the first object viewed from an arbitrary viewpoint can be easily generated.

The other invention has a specific effect that a composite image of two things viewed from an arbitrary viewpoint can be easily generated.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a first embodiment of an image generation processing apparatus according to the present invention.

FIG. 2 is a flowchart illustrating a flow of a rotation axis detection process.

FIG. 3 is a flowchart illustrating a flow of processing for detecting a lowermost part of a jaw.

FIG. 4 is a block diagram showing a second embodiment of the image generation processing device of the present invention.

FIG. 5 is a block diagram showing a third embodiment of the image generation processing device of the present invention.

FIG. 6 is a flowchart illustrating a flow of a similarity evaluation process.

FIG. 7 is a block diagram showing a fourth embodiment of the image generation processing device of the present invention.

FIG. 8 is a block diagram showing a fifth embodiment of the image generation processing device of the present invention.

FIG. 9 is a block diagram showing a sixth embodiment of the image generation processing device of the present invention.

FIG. 10 is a diagram illustrating a method of calculating a rotation angle and a rotation axis.

FIG. 11 is a block diagram showing a seventh embodiment of the image generation processing device of the present invention.

FIG. 12 is a block diagram showing an eighth embodiment of the image generation processing device of the present invention.

FIG. 13 is a block diagram showing a ninth embodiment of the image generation processing device of the present invention.

FIG. 14 is a block diagram showing a tenth embodiment of the image generation processing device of the present invention.

FIG. 15 is a block diagram showing an eleventh embodiment of the image generation processing device of the present invention.

FIG. 16 is a flowchart illustrating a flow of a process performed by a coordinate conversion unit.

FIG. 17 is a flowchart illustrating a flow of three-dimensional coordinate calculation.

FIG. 18 is a block diagram showing a twelfth embodiment of the image generation processing device of the present invention.

FIG. 19 is a flowchart illustrating the flow of a process performed by an interpolation / synthesis unit.

FIG. 20 is a block diagram showing a thirteenth embodiment of the image generation processing device of the present invention.

FIG. 21 is a flowchart illustrating the flow of a process performed by the alignment synthesizing unit.

FIG. 22 is a block diagram showing a fourteenth embodiment of the image generation processing device of the present invention.

FIG. 23 is a block diagram showing a fifteenth embodiment of the image generation processing device of the present invention.

FIG. 24 is a flowchart illustrating the flow of a positioning process.

[Explanation of symbols]

 Reference Signs List 1 electric swivel chair 1a swivel chair 1b marker 1c fixed chair 2 TV camera 3 multi-viewpoint image storage unit 4 viewpoint acquisition unit 4a viewpoint calculation unit 4b similarity calculation unit 4c one-round image detection unit 4d speed / acceleration calculation unit 4e viewpoint Calculation unit 4f Average speed calculation unit 4g Marker detection unit 4h Feature point extraction unit 4i Feature point tracking unit 5 Face image generation unit 5a View point selection unit 5b Face region extraction unit 5c Coordinate conversion unit 5d Multiple viewpoint selection unit 5e Interpolation synthesis unit 6 Position Alignment synthesis unit 6a Rotation axis detection unit 6b Jaw position detection unit 6c Position alignment unit 6c 'Manual alignment unit 6d Image synthesis unit 6e Image synthesis / confirmation unit 6f 3 Feature point extraction unit 6g 3D position calculation unit 7 Human body image generation unit 8 viewpoint setting part

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Takashi Torii 4-1-1, Kamiodanaka, Nakahara-ku, Kawasaki-shi, Kanagawa Prefecture Inside Fujitsu Limited (72) Inventor Toshio Endo 4-1-1, Kamiodanaka, Nakahara-ku, Kawasaki-shi, Kanagawa Prefecture No. 1 Inside Fujitsu Limited (72) Inventor Satoshi Otake 4-1-1, Kamiodanaka, Nakahara-ku, Kawasaki City, Kanagawa Prefecture Inside Fujitsu Limited (72) Inventor Masaaki Yoshikawa 1-3-3 Konyacho, Takamatsu City, Kagawa Prefecture Shares Company Fujitsu Kagawa System Engineering (72) Inventor Noritaka Sekiguchi 1-1-1, Sonoyama-cho, Otsu-shi, Shiga Toray Co., Ltd. Shiga Plant F-term (reference) 5B050 BA06 BA10 BA12 DA07 EA03 EA05 EA19 FA19 5B057 CA08 CA12 CA16 CB08 CB12 CB16 CE08 CE09 CH08 DC02

Claims (37)

[Claims] 1. A multi-view image in which a first object is viewed from a plurality of viewpoints having different positions and orientations relative to the first object, and the positions and orientations of the plurality of viewpoints are stored. In response to setting the viewpoint arbitrarily, based on the multi-view image and the positions and orientations of the plurality of viewpoints, the image of the first thing viewed from the set viewpoint is obtained. An image generation processing method characterized by generating. 2. A multi-view image in which a first object is viewed from a plurality of viewpoints having different positions and orientations relative to the first object is stored, and the positions and orientations of the plurality of viewpoints are stored. In response to setting the viewpoint arbitrarily, based on the multi-view image and the positions and orientations of the plurality of viewpoints, the first thing of the first thing viewed from the set viewpoint is obtained. Generating an image, generating a second image of a second thing as viewed from a second viewpoint different from the first thing from the set viewpoint, and generating a first image and a second image according to the set viewpoint. An image generation processing method, comprising: performing image positioning and synthesizing an image to generate a synthesized image. 3. The multi-viewpoint image is an image obtained by photographing a first thing while rotating a first thing by an electric rotating means, and the direction of the viewpoint is a photographing of the first thing. 3. The method according to claim 1, wherein the calculation is performed based on a rotation angle signal output from the electric rotation means.
2. The image generation processing method according to 1. 4. The multi-viewpoint image is an image obtained by photographing the first thing while rotating the first thing with a speed pattern specified by a parameter unique to the electric rotating means by the electric rotating means. The image generation processing method according to claim 1, wherein the direction of the viewpoint is calculated based on a parameter unique to the electric rotating unit. 5. The multi-viewpoint image is an image obtained by photographing a first thing while rotating the first thing by inertia, and the direction of the viewpoint is determined by a similarity between the multi-viewpoint images. 3. An image capturing time point after one rotation is obtained based on the image data, a rotation speed and a rotation acceleration are detected from the image capturing time point, and the rotation speed and the rotation acceleration are calculated based on the rotation speed and the rotation acceleration. The image generation processing method described in the above. 6. The multi-view image is an image obtained by photographing a first thing while rotating the first thing, and the direction of the viewpoint is based on a similarity between the multi-view images. 3. The image generation processing method according to claim 1, wherein an image shooting time point after one rotation is obtained, an average rotation speed is detected from the image shooting time point, and calculation is performed based on the average rotation speed. . 7. The multi-view image is an image obtained by photographing a first object while rotating the object at the same time as an object to which a predetermined pattern is added, and the direction of the viewpoint is determined from each of the multi-view images. The image generation processing method according to claim 1, wherein the calculation is performed by detecting the predetermined pattern. 8. The multi-viewpoint image is an image obtained by photographing a first thing while rotating the first thing, and the direction of the viewpoint is obtained by extracting a feature point from each of the multi-viewpoint images. 3. The image generation process according to claim 1, wherein a correspondence between feature points is extracted, and the calculation is performed based on a rotation angle of the first thing calculated based on the correspondence. Method. 9. The multi-viewpoint image is an image obtained by shooting while moving around a first thing, and the direction of the viewpoint is calculated based on a moving position. The image generation processing method according to claim 1 or 2. 10. The multi-viewpoint image is an image obtained by photographing at a plurality of photographing positions arranged around a first thing, and the position and orientation of the viewpoint are based on the corresponding photographing position. The image generation processing method according to claim 1, wherein the image generation processing is performed. 11. The multi-view image is an image obtained by simultaneously photographing a first thing and an object to which a predetermined pattern is added, and the direction of the viewpoint is detected from each of the multi-view images. The image generation processing method according to claim 9, wherein the calculation is performed based on a positional relationship between the patterns. 12. The direction of the viewpoint is calculated based on the correspondence between the feature points by extracting five or more feature points from each of the multi-viewpoint images. The image generation processing method according to claim 9. 13. The method according to claim 1, wherein the generation of the image of the first thing is performed by cutting out a desired area from an image selected from an obtained viewpoint.
3. The image generation processing method according to any one of 2. 14. The method according to claim 1, wherein the generation of the image of the first thing is performed by cutting out a desired area from an image selected from an obtained viewpoint and performing coordinate transformation on the area. An image generation processing method according to claim 12. 15. The method according to claim 15, wherein the generation of the image of the first thing is performed by cutting out a desired area from a plurality of images selected from the obtained viewpoint and interpolating and synthesizing these areas. The image generation processing method according to claim 1. 16. The composition of the first image and the second image,
A predetermined feature point is detected from each of the axis of the first thing and each of the multi-viewpoint images, a three-dimensional position of the predetermined feature point is calculated, and the axis of the first thing and the second thing are detected. 9. The positioning is performed so that the axes of the second object coincide with each other and a predetermined feature point is positioned at a predetermined position with respect to the second object in the three-dimensional space. An image generation processing method according to any one of claims 12 to 15. 17. The composition of the first image and the second image,
Detecting the axis of the first thing, moving each of the predetermined number of images among the multi-view images so that the axis is at the predetermined position, and moving the moved image to an appropriate position with respect to the second thing. 16. The method according to claim 2, wherein the first object is moved so as to move, and the two images are aligned based on the amount of movement. Image generation processing method. 18. The composition of the first image and the second image,
Three feature points are detected from each of a plurality of images captured from a predetermined viewpoint in the multi-viewpoint image, three-dimensional positions of the feature points are calculated, and three feature points are detected in a three-dimensional space. 16. The image generation processing method according to claim 2, wherein the positioning is performed such that the position of the image is at a predetermined position with respect to the second thing. 19. Multi-view image storage means for storing a multi-view image in which a first thing is viewed from a plurality of viewpoints different from each other in relative position and orientation with respect to the first thing; Viewpoint acquiring means for acquiring a position and an orientation, viewpoint setting means for arbitrarily setting a viewpoint, and a first viewpoint viewed from the set viewpoint based on a multi-view image and positions and orientations of a plurality of viewpoints. An image generation processing apparatus, comprising: first image generation means for generating an image of an object. 20. A multi-viewpoint image storing means for storing a multi-viewpoint image obtained by viewing a first thing from a plurality of viewpoints having different positions and orientations relative to the first thing; Viewpoint acquiring means for acquiring a position and an orientation, viewpoint setting means for arbitrarily setting a viewpoint, and a first viewpoint viewed from the set viewpoint based on a multi-view image and positions and orientations of a plurality of viewpoints. First image generation means for generating a first image of an object, and second image generation means for generating a second image of a second thing as viewed from a second viewpoint different from the first thing from the set viewpoint An image generation processing apparatus comprising: a first image and a second image that are aligned according to the set viewpoint and synthesized to generate a synthesized image. 21. The multi-viewpoint image is an image obtained by photographing a first thing while rotating a first thing by an electric rotating unit, and the viewpoint acquiring unit is configured to photograph the first thing. 21. The image generation processing apparatus according to claim 19, wherein the direction of the viewpoint is calculated based on a rotation angle signal output from the electric rotation unit at times. 22. The multi-viewpoint image is an image obtained by photographing the first thing while rotating the first thing with a speed pattern specified by a parameter unique to the electric rotating means by the electric rotating means. 21. The image generation processing apparatus according to claim 19, wherein the viewpoint acquiring unit calculates the direction of the viewpoint based on a parameter unique to the electric rotating unit. 23. The multi-viewpoint image is an image obtained by photographing the first thing while rotating the first thing by inertia, and the viewpoint acquiring means calculates the similarity between the multi-viewpoint images. A similarity calculating means for calculating, a one-turn image detecting means for detecting, based on the similarity, an image which has just rotated one time with an image at a certain point in time among the multi-view images, and a predetermined plurality of multi-view images among the multi-view images A speed / acceleration calculating means for calculating a rotation speed and a rotation acceleration from a relationship between a time when each of the images is taken and a time when the image is taken after one rotation of the images; 21. The image generation processing apparatus according to claim 19, wherein the direction of the viewpoint is calculated based on the viewpoint. 24. The multi-viewpoint image is an image obtained by photographing a first thing while rotating the first thing,
The viewpoint acquisition means includes: a similarity calculation means for calculating a similarity between multi-view images; and a rotation after one rotation for detecting, based on the similarity, an image of the multi-view image just after one rotation with an image at a certain time. Image detection means, and one of the multi-viewpoint images
Speed calculation means for calculating an average rotation speed from a relationship between a time point when each of the one or more images is captured and a time point when the image after one rotation of the images is captured, and a viewpoint orientation based on the average rotation speed 21. The image generation processing device according to claim 19, wherein the image generation processing device calculates the following. 25. The multi-viewpoint image is an image obtained by photographing a first thing while rotating at the same time as an object to which a predetermined pattern is added. 21. The image generation processing apparatus according to claim 19, wherein a direction of a viewpoint is calculated by detecting the predetermined pattern. 26. The multi-viewpoint image is an image obtained by photographing a first thing while rotating the first thing,
The viewpoint obtaining means includes: a feature point extracting means for extracting a feature point from each of the multi-view images; a feature point tracking means for extracting a correspondence between the feature points between the multi-view images; And a rotation angle calculating means for calculating a rotation angle of the first thing at the time of capturing the multi-viewpoint image, and calculating the viewpoint direction from the rotation angle at the time of capturing each of the multi-viewpoint images. The image generation processing apparatus according to claim 19, wherein: 27. The multi-view image is an image obtained by shooting while moving around a first thing, and the viewpoint obtaining means calculates a direction of a viewpoint based on a movement position. Claim 19 or Claim 2
0. The image generation processing device according to 0. 28. The multi-viewpoint image is an image obtained by photographing at a plurality of photographing positions arranged around a first thing, and the viewpoint acquiring means determines a viewpoint based on a corresponding photographing position. 21. The image generation processing apparatus according to claim 19, wherein the position and orientation of the image are calculated. 29. The multi-viewpoint image is an image obtained by simultaneously photographing a first thing and an object to which a predetermined pattern has been added, and the viewpoint obtaining means determines the pattern from each of the multi-viewpoint images. 29. The image generation processing apparatus according to claim 27, further comprising: a pattern detection unit configured to detect a direction of a viewpoint based on a positional relationship of a pattern detected from each of the multi-viewpoint images. . 30. A feature point extracting means for extracting five or more feature points from each of the multi-view images, and a feature point tracking for extracting the correspondence of each feature point between the multi-view images. 29. The image generation processing device according to claim 27, wherein the image generation processing device includes means for calculating based on the correspondence. 31. The first image generating means includes an area extracting means for cutting out a desired area from a multi-view image as needed, and one viewpoint obtained by a viewpoint acquiring means according to an arbitrary set viewpoint. 31. A multi-viewpoint image after extracting a region corresponding to the selected viewpoint is output, comprising: a viewpoint selecting means for selecting a selected viewpoint.
The image generation processing device according to any one of the above. 32. The first image generating means includes: an area extracting means for cutting out a desired area from a multi-view image as needed; and a first image generating means for extracting a desired area from a viewpoint obtained according to an arbitrary viewpoint set by a viewpoint setting means. And a viewpoint selecting means for selecting one of the viewpoints, and outputting the multi-viewpoint image after the region extraction corresponding to the selected viewpoint in accordance with the relationship between the set viewpoint and the selected viewpoint after the coordinate conversion. 2. The method according to claim 1, wherein
The image generation processing device according to any one of claims 9 to 30. 33. An image processing apparatus according to claim 27, wherein the first image generation unit includes a region extraction unit that cuts out a desired region from the multi-view image as needed, And a viewpoint selecting means for selecting a viewpoint from among the plurality of viewpoints, wherein a plurality of images after region extraction corresponding to the selected viewpoint among the multi-viewpoint images are interpolated and synthesized and output. The image generation processing device according to claim 30. 34. The composite image generating means, wherein a rotation axis detecting means for detecting a rotation axis of a first thing which rotates relatively to the photographing means, and an image taken from a predetermined viewpoint in a multi-viewpoint image. A feature point detection unit that detects a predetermined feature point from each of the plurality of images; and a calculation unit that calculates a three-dimensional position of the feature point, and a rotation axis of the first thing and the second thing. Are aligned with each other, and a predetermined feature point is positioned at a predetermined position with respect to the second object in the three-dimensional space. The image generation processing device according to any one of claims 29 to 32. 35. A combined image generating unit comprising: a rotating axis detecting unit configured to detect a rotating axis of a first thing that rotates relative to a photographing unit; and a rotating axis detecting unit configured to move the rotating axis to a predetermined position of the image. First moving means for moving each of a predetermined number of images among the multi-viewpoint images, display means for displaying the moved image so as to be superimposed on the image of the second thing, and first object on the displayed screen Second moving means for manually moving the first thing so that the first thing is at an appropriate position with respect to the second thing, and the first thing and the second thing based on the amount of movement when manually moving. The image generation processing apparatus according to any one of claims 19 to 25 and 29 to 32, further comprising a positioning means for performing positioning with an object. 36. The composite image generating means, comprising: a rotation axis detecting means for detecting a rotation axis of a first thing rotating relatively to the photographing means; and a plurality of multi-view images photographed from a predetermined viewpoint. Characteristic point detecting means for detecting a predetermined characteristic point from each of the images, position calculating means for calculating the three-dimensional position of the characteristic point, and the rotation axes of the first and second objects are mutually different. A positioning means for positioning so that they coincide with each other and a predetermined feature point comes to a predetermined position with respect to the second object in the three-dimensional space;
Display means for combining the image with the image of the object and displaying the image on the screen, and manually moving the first object so that the first object is located at an appropriate position with respect to the second object on the displayed screen. Second
In response to the position of the first thing and the second thing not being appropriate in the displayed image, based on the amount of movement when manually moving the first thing and the first thing again. The image generation processing apparatus according to any one of claims 19 to 25 and 29 to 32, further comprising a positioning means for performing positioning with a second thing. 37. A feature point detecting means for detecting three feature points from each of a plurality of images taken from a predetermined viewpoint among multi-viewpoint images, and a feature point detecting means for detecting the three feature points. Position calculating means for calculating a three-dimensional position, wherein the three feature points are aligned with respect to the second object so as to be at a predetermined position in the three-dimensional space. The image generation processing device according to any one of claims 19 to 32.