JPH1145349A - Wire frame model matching device, method therefor and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make it possible to correctly extract feature point so as to obtain natural image even when side face is displayed. SOLUTION: This wire frame model matching device for matching a three- dimensional wire frame model with the image of the correspondent face is composed of a distance image input device 10 such as a range finder, arithmetic processor 30 for performing arithmetic processing based on an inputted distance image, and display device 20 for displaying the output of the arithmetic processor 30. At the arithmetic processor 30, a curvature operating part 36, beautiful part operating part 38 and feature extracting part 40 extract feature points on the face image corresponding to the control points of this wire frame model based on the information of a distance, which is inputted from the distance image input device 10, from a reference to each point on the face. Then, a matching part 42 at the arithmetic processor 30 matches the correspondent control points of this wire frame model with these respective extracted feature points on the face image.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a wire for matching a three-dimensional wire frame model to a corresponding object image, for example, matching a face wire frame model for performing facial expression synthesis to a face image. The present invention relates to a frame model matching device and method, and a machine-readable recording medium storing a matching program for matching such a three-dimensional wire frame model to a corresponding object image, and in particular, to each part of an object using a distance image. Automatically extract features and perform matching.

[0002]

2. Description of the Related Art For example, in the fields of communication in remote places, virtual reality, and human interface, a technique for synthesizing facial expressions by image processing using a facial image is required.

Conventionally, as a method of performing facial expression synthesis in face image processing, a method of matching a wire frame model with a face image has been used. If the face image is represented by a wireframe structure and the vertex on the wireframe model is moved to perform the face image deformation operation, it is advantageous in terms of information compression. This method is described in IEICE, J
73-A, 1270-1280 (1990), using a wireframe model created according to a standard three-dimensional shape of a face, the control points on the wireframe model are set from the front. Eyes of the face image taken,
Match the wireframe model to the target individual's face image by setting it to feature points such as nose, mouth, eyebrows, face contour, hair, etc., and transforming points other than control points by affine transformation etc. Things. This face standard wireframe model is based on the FACS (Facial Action Coding System, Cons
ulting Psychologist Press (1977)), which is a model capable of synthesizing facial expressions by using face deformation rules. With this model, six basic facial expressions (anger, disgust, fear, joy, sadness, surprise)
And various expressions can be synthesized.

Further, IEICE, J56-D, 226-23
2 (1973), IEICE Technical Report, HC 91-43 (199
2) and as shown in JP-A-8-77334,
Example of performing automatic matching of a wireframe model by detecting feature points from a face image and matching control points of the wireframe model to extracted feature points in order to automatically match a wireframe model to a face image There is.
In this technique, a Gaussian filter or a Sobel filter is applied to a photographed face image to extract features such as edges, and the feature is associated with a face feature to automatically match a wireframe model.

Further, IEICE Technical Report, PRU92-79 (1)
992), there has been proposed a method of matching depth information of a wireframe model using a distance image in order to accurately perform deflection matching including the depth direction. In this method, the depth component of each vertex is matched using the front distance image with respect to the matched wire frame model.

[0006]

However, since the conventional matching of the wireframe model to the face image is performed using the face image photographed from the front, accurate matching is performed for the face side. This cannot be performed, and there is a problem that an image becomes unnatural when a profile is displayed.

In the feature point extraction for automatic matching of the wire frame model, which is performed using a color image or a gray-scale image, the feature points to be extracted differ depending on the photographing conditions such as lighting conditions. There is a possibility that
There is a problem that it is difficult to always extract features under the same conditions.

The present invention has been made in view of the above points, and enables a three-dimensional wire frame model that enables accurate feature point extraction and obtains a natural image even when a profile is displayed. Wire frame model matching device and method for matching an image of an object to be matched, and a machine-readable recording medium storing a matching program for matching such a three-dimensional wire frame model to a corresponding object image. With the goal.

[0009]

In order to achieve the above object, a wire frame model matching device according to the present invention is a wire frame model matching device for matching a three-dimensional wire frame model with a corresponding object image. A feature point extracting unit that extracts feature points on an object image corresponding to control points of the wireframe model based on distance information from a reference to each point on the object; and a feature point extracting unit that extracts feature points. And matching means for matching the control points of the wireframe model with each feature point on the object image.

A wire frame model matching method according to the present invention is a wire frame model matching method for matching a three-dimensional wire frame model to an image of a corresponding object, the distance information from a reference to each point on the object. And a feature point extraction step of extracting feature points on the object image corresponding to the control points of the wire frame model based on the distance information. And a matching step of matching corresponding control points of the wireframe model.

Further, a recording medium according to the present invention is a machine-readable recording medium recording a matching program for matching a three-dimensional wireframe model to a corresponding object image, and the matching program is stored in a computer. ,
The feature points on the object image corresponding to the control points of the wire frame model are extracted based on distance information from the reference to each point on the object, and the feature points on the extracted object image are It is characterized in that the corresponding control points of the frame model are matched.

That is, according to the wire frame model matching apparatus, method and recording medium of the present invention, feature points are extracted based on distance information, so that accurate feature point extraction becomes possible.

[0013]

Embodiments of the present invention will be described below with reference to the drawings.

[First Embodiment] FIG. 1A is a block diagram of a wire frame model matching apparatus according to a first embodiment of the present invention. The wire frame model matching device according to the present embodiment is a distance image input device 1
0, a display device 20, and an arithmetic processing device 30. It should be noted that the display device 20 and the arithmetic processing device 30 may be at a close position or at a remote place via a network.

In the present embodiment, a case where the three-dimensional structure of the face wireframe model is displayed on the display device 20 will be described as an example. Here, (B) to (D) of FIG.
The three-dimensional structure of the wireframe model of the face is displayed from the front, the oblique direction, and the lateral direction.

The range image input device 10 is a device used to capture an image of a measurement object (in the present embodiment, face).
This is a full range type rangefinder that can measure the entire circumference of °. The all-around type range finder can measure all-around type color information (texture image) 100 and all-around type distance information 102 of the measurement object as shown in FIGS. 2A and 2B. . The omnidirectional distance image 102 is an image in which the value of each point on the image represents the distance from the rotation center of the range finder. In this example, the brighter the portion is drawn, the greater the distance from the rotation center is. The measured all-around texture image and the distance images 100 and 102 are shown as images in which the horizontal axis indicates the angle indicating the measurement direction and the vertical axis indicates the measured height of the measurement target.

The arithmetic processing unit 30 performs arithmetic processing based on the full-circle type texture image and the distance images 100 and 102 of the measurement object (face) input from the distance image input device 10 and obtains the arithmetic result. Is output to the display device 20. Here, the arithmetic processing device 30 includes a control unit 32, a memory 3
4. Curvature calculation unit 36, differentiation calculation unit 38, feature extraction unit 4
0, a matching unit 42, and a developing unit 44.
Each of these units can be actually composed of one CPU and a memory, and the curvature calculation unit 36,
The differential operation unit 38, the expansion unit 44, and the like may be configured by a dedicated DSP.

The control unit 32 controls the inside of the arithmetic processing unit 30. The control unit 32 receives the all-around type texture image and the distance images 100 and 102 from the distance image input device 10 via a signal line 50, Further, the result of the arithmetic processing is transmitted to the display device 20 via the signal line 60. Memory 34
Is connected to the control unit 32 and receives the omnidirectional texture image and the distance image 10 from the distance image input device 10.
And a work memory for storing other information.

The curvature calculator 36 calculates the curvature at a point on the object (face) from the all-round distance image 102 stored in the memory 34. The differential operation unit 38 has a differential filter such as a Sobel filter, and extracts an edge of a target object (face) from the all-around distance image 102 stored in the memory 34. The feature extraction unit 40 is configured to perform the above-described curvature calculation unit 36
An operation for extracting a feature point corresponding to the control point of the wire frame model is performed based on the feature information such as the curvature information calculated in step (1) and the edge information extracted by the differential operation unit. The matching unit 42 matches the control points of the wireframe model with the feature points extracted by the feature extraction unit 40.
The developing unit 44 generates a developed wire frame to be matched with the all-around distance image 102 stored in the memory 34 in order to match the depth components.

In other words, 3
After performing dimensional shape modeling and setting so that the vertical direction is displayed correctly, the all-around texture image 100
The front texture image displayed as a texture mapping image of the front viewpoint image by using the front texture image 1 shown in FIG.
In order to match the wire frame model to the wireframe model 04, the feature extracting unit 40 uses
Is used to extract feature points of the face image. In order to perform this feature point extraction, it is necessary to perform preprocessing on each image.

FIG. 2D shows a 3D image set to be displayed from the same viewpoint as the front texture image 104.
This is a front distance image 106 showing the distance distribution of the two-dimensional shape modeling image with respect to the display plane.
Reference numeral 06 is displayed so that each point on the image indicates distance information from the display plane. In this example, the bright part has a large distance from the display plane.

The curvature calculator 36 calculates the curvature at the position indicated by the point on the object to be measured for each point on the whole circumference type distance image 102 with respect to the image. FIG. 3A is an all-round curvature image 108 displayed so that each pixel has a curvature value. Here, the positive and negative of the curvature are defined such that a point that is convex in the front direction with respect to the display plane of the full-circular curvature image 108 is positive, and a point that is concave in the front direction is negative. In this example, the negative curvature is increased as the brightness increases, and the positive curvature is increased as the brightness decreases. Considering this curvature distribution as a texture, 3
The front curvature image attached to the three-dimensional shape modeling image and displayed from the front is the image indicated by reference numeral 110 in FIG.

The differential operation section 38 applies a Sobel filter (differential filter) to perform edge extraction on the front distance image 106. As a result, a front edge image 112 as shown in FIG. 3C is obtained.
In this example, the edge information, that is, the larger the differential value is, the brighter it is drawn. The above-mentioned all-around curvature image 10
8. Front curvature image 110 and front edge image 112
Is that any point is the entire circumference type texture image 100, the entire circumference type distance image 102 or the front side texture image 104,
It is associated with the front distance image 106, and both the distance information and the color information can be referred to.

The front texture image 104, the front distance image 106, the all-around curvature image 108, the front curvature image 110, and the front edge image 1 which have been subjected to such pre-processing.
The feature extraction unit 40 performs feature point extraction using a series of twelve images. Then, the matching unit 42 matches the feature points extracted by the feature extracting unit 40 with some of the control points on the wire frame as shown in FIG. I do.

The above-mentioned series of images 10
The feature points to be extracted from 4 to 112 include the nose apex, the lower end of the nose, both ends of the nose, the boundary line between the upper lip and the lower lip, both ends of the mouth, upper and lower ends of the mouth,
Includes facial contours. The extraction order of each feature point is performed according to the flow shown in FIG.

That is, first, it is necessary to extract both ends of the face (step S10). This feature point is a feature point defined as the intersection point of the face outline on the horizontal line of the eye position. The detection of the feature points is manually performed in order to finely adjust the position while considering the position of the eyes.

Next, a nose tip is extracted (step S1).
2). This nose apex extraction is extracted as the position of the foremost point in the front distance image 106. First, in order to remove noise, a noise removal operation is performed by a smoothing filter, and the filtered distance image The point where the distance value is closest to the above is set as the position of the nose. Also,
If a hair part that is difficult to measure with the range finder is included in the feature point search area, the possibility of erroneous extraction increases. For this reason, in the display area, for example, when the nose is detected only in the lower one-third area, and when the front texture image 104 and the front distance image 106 are displayed, the area where the hair part measurement is difficult is performed. The display is set in the upper third area, and is excluded from the nose apex detection area.

Then, from the front curvature image 110, the nose contour (the lower end of the nose, both ends of the nose) and the boundary line between the upper lip and the lower lip are extracted as the position where the negative curvature becomes large, and the position where the positive curvature is large is extracted. Extract the top and bottom of the mouth. Further, the lower edge of the face and the face outline are extracted from the front edge image 112.

That is, the extraction of the characteristic points representing the lower end of the nose, both ends of the nose, the upper and lower ends of the mouth, and the face contour is performed by the method shown in FIG. In the figure, darker portions are indicated by finer hatching, and brighter portions are indicated by no hatching. In other words, in the image shown in the figure, it is assumed that the brighter the area, the larger the feature amount. Then, when there is an area having a large feature amount in the form indicated by reference numeral 200, the rectangular window 202 is scanned on this image. In other words, the feature points to be extracted are regions where the feature amount is maximum or minimum, and in such regions, the region having a value close to the maximum or minimum value of the feature amount is often elongated. . Therefore, a rectangular window w (x, y) of a fixed size indicated by reference numeral 202 centered on the point P (x, y) on the image C (x, y) on which the feature point search is performed is referred to. scanned within a certain range, as indicated at 204, point P (x, y) transverse length w _x about the vertical of the total of the feature amount of the pixels included in the window of length w _y It is defined as f (x, y), and the position at which this function is maximum or minimum is defined as a feature point.
In this figure, when the rectangular window 202 is at the position indicated by the reference numeral 206, the function f (x, y) is maximized, and the center of the rectangle is the characteristic position. here,
The size of the window w (x, y) is determined according to the size of the wireframe model that has been matched by the operation before the operation for detecting the feature point.

Since it is necessary to use the size information of the wire frame model that has been matched up to that point in the extraction of each feature point starting from step S16, first, the feature at the lower end of the face is used. Point extraction is performed, the points at the tip of the jaw of the wire frame model are matched, and the size of the wire frame model is adjusted to the face image to some extent (step S14).

The feature extraction of the lower end of the face in this step S14 is performed by using the front edge image 112 as follows: w _x × w _y = (interval of the two ends of the face of the initially matched face on the wire frame model / 4) ) × 5 pixels The window 202 is scanned from the position corresponding to the nose vertex extracted in step S12 to the position corresponding to the lower end of the front texture image 104, and the edge intensity f (x, y The position where () is the maximum is defined as the lower end of the face.

Next, the lower end of the nose is extracted (step S1).
6). In this nose lower end extraction, a window 202 having a size of w _x × w _y = (width of a nose of a wire frame model) × 5 pixels is added to the nose apex extracted in step S12 using the front curvature image 110. From the corresponding position, scanning is performed to the height of the upper end of the mouth of the wireframe model, and the position where the curvature f (x, y) becomes minimum is defined as the lower end of the nose.

Next, both ends of the nose are extracted (step S1).
8). This nose both ends extraction uses the front curvature image 110 to create a window 202 having a size of w _x × w _y = 5 × (the length from the nose vertex to the lower end of the nose of the wire frame model) pixels. Along the horizontal line of the height of the nose vertex and the center of the lower end of the nose from the center line of the face including the position corresponding to the nose vertex extracted in step S12 (the face based on both ends of the face extracted in step S10). Scan by the width of 4/4), and the curvature f
The position where (x, y) is minimum is defined as one end of the nose.

Next, feature points of the mouth are extracted. The mouth feature points include the mouth both ends, the mouth center, and the upper and lower ends of the mouth. To extract these, first, a temporary mouth center is extracted (step S20), and the extracted temporary mouth center is used. , And extract accurate mouth feature points. That is, the mouth center tentative, by using the front curvature image 110, the w _x × w _y = (width of the mouth of the wire frame model) × 5 pixels in size of the window 202, extracted by the step S16 nose Step S14 from the position corresponding to the lower end
Is scanned to the position corresponding to the lower end of the face extracted in step
The position where (x, y) is minimum is set as the temporary mouth center.

Then, in order to accurately extract each mouth feature point, mouth both ends are first extracted (step S22). This mouth end extraction uses the front curvature image 110 and the step S
(Interval between points at both ends of face initially matched on the wireframe model / 2) × (interval between lower end of nose of wireframe model and temporary mouth center, centered on temporary mouth center position extracted in step 20) When a rectangular area having a size of (length × 2) pixels is set, and the average value of the feature values inside this rectangular area is C _mean and the standard deviation is C _std , the feature value is larger than C _mean + C _std. The leftmost point among the possessed points is the left end of the mouth, and the rightmost point is the right end of the mouth.

When the both ends of the mouth are determined in this way, a position having the minimum curvature is searched for on the vertical line of 40 pixels in length centering on the middle point, and that position is set as the accurate mouth center (step S24). .

Then, the upper and lower ends of the mouth are extracted (step S26). This mouth upper and lower edge extraction uses the front curvature image 110, and w _x × w _y = (width of mouth of wire frame model / 2) ×
The window 202 having a size of 5 pixels is moved from the position corresponding to the center of the mouth extracted in step S24 to the position corresponding to the lower end of the nose extracted in step S16 upward, and downward in step S14. Scanning is performed up to the positions corresponding to the extracted lower end of the face, and the positions where the curvature f (x, y) becomes the maximum are set as the upper end and the lower end of the mouth.

Finally, the face outline is extracted (step S).
28). This face contour extraction is performed by extracting the front edge image 112
Is used to extend a straight line from the position corresponding to the nose apex extracted in step S12 to each contour control point, and a window having a size of 15 × 5 pixels, which is long in a direction perpendicular to the straight line, is set horizontally for each control point. On the line, scanning is performed from the inside of 50 pixels from each control point to the outside of the position corresponding to both ends of the face extracted in step S10, and 10 pixels outside, to obtain the edge intensity curvature f.
The position where (x, y) is the maximum is defined as the contour point of the face.

Next, the matching section 42 performs automatic matching of the wire frame by matching each control point of the wire frame with each of the extracted feature points. Then, the depth component is matched with the wire frame model using the front distance image 106. That is, for all vertices p (x, y, z) on the wireframe model,
In two-dimensional matching, the values of x and y are accurate,
Since the value of z is incorrect, the value of z is matched using the front distance image 106.

Further, with respect to the wire frame model matched using the front distance image 106, the entire circumference type distance image 102 Is performed.
Therefore, the developing unit 44 converts the wireframe model matched in the work up to now into the same coordinate system as the all-around distance image 102 by performing cylindrical coordinate conversion. That is, the vertex p on the wire frame in the Cartesian coordinate system
(X, y, z) is converted into cylindrical coordinates, and p (r, θ,
y). FIG. 6 shows the wireframe model thus converted.

The matching unit 42 superimposes the wireframe model converted by the developing unit 44 on the above-described perimeter-type texture image 100 and the front distance image of a feature point shifted from the wireframe model. Correction matching is performed for points on the side of the face indicated by black circles in FIG. 6 where accurate matching cannot be performed only by performing matching using 106.

The wire frame model thus corrected and matched is output to the display device 20 as the final result of the arithmetic processing by the arithmetic processing device 30.

[Second Embodiment] Next, a second embodiment of the present invention will be described.
An embodiment will be described.

The standard wire frame model is created using an average face as a model. Normally, in order to reduce the amount of information, each triangular patch forming the wire frame model is created roughly. However, in the process of matching to a personal face, the size of the triangular patch becomes too large in some cases, and depending on the location, there may be a part where the smoothness is lost when displaying the face image. is there. Therefore, it is necessary to minimize the increase in the amount of information and to keep the surface shape smooth.

Therefore, in the second embodiment, as shown in FIG. 1A, the arithmetic processing unit 30 of the wire frame model matching apparatus in the first embodiment further includes a subdivision unit. 46 is provided.

The subdivision unit 46 performs threshold processing on the points inside each triangular patch of the wireframe model using the above-described full-circular curvature image 108. Is finely divided to smooth the surface shape.

In this case, the wire frame subjected to the cylindrical coordinate conversion shown in FIG. 6 is superimposed on the above-described full-circular curvature image 108, and the determination as to whether or not each triangular patch is subdivided is made based on the absolute value of the curvature. . That is, the absolute value of the internal curvature C in each triangle | when the _max, | | C | of the maximum value | C C | curvature for _max radius r _max = 1 / | C | _max the square root of the area of the triangle

[0048]

(Equation 1) Amount normalized by

[0049]

(Equation 2) Is compared with a preset threshold value c _th , and if the following inequality (1) is not satisfied, subdivision is performed.

[0050]

(Equation 3) In this division, as shown in FIG. 7A, each triangle patch is divided into three triangles formed by drawing a dotted line from each vertex to a center point (centroid, inner center, outer center, etc.). By doing.

Alternatively, as shown in FIG.
The division may be made by a dotted line connecting the vertices of the adjacent sides of two adjacent triangles.

After such division, the matching unit 42 matches the distance value r of the coordinates of the vertex to the newly generated vertex using the distance image.

Although the present invention has been described based on the embodiments, the present invention is not limited to the above-described embodiments, and various modifications and applications are possible within the scope of the present invention. .

Although the above description has been made with reference to a face image, this method is not particularly limited to a face image, and can be used for a more general range image. For example, the present invention can be applied to an arbitrary object such as a person, an animal, a plant, and a landscape. By using the whole circumference type distance image of the whole body of a person or an animal, it is possible to match a wire frame model of the whole body and to generate a motion model of a human body or an animal. Further, by measuring the shape of a part of the body such as a hand, an arm, or a foot, it is possible to perform matching with these models, and similarly, it is possible to generate a motion model of these parts. In order to perform such model matching, a bendable point such as a joint can be used as a feature point. Furthermore, by performing distance image measurement during the operation and performing matching with an internal organ model such as the heart or stomach, deformation simulation of these organs can be performed. In addition, by measuring the shapes of plants such as trees and roses, it is possible to match these models to simulate growth and environmental changes. A deformation simulation can be similarly performed on a desk, a chair, a robot, an electronic device, a building, a mountain, a landscape, and the like.

Here, the gist of the present invention is summarized as follows.

(1) In a wire frame model matching apparatus for matching a three-dimensional wire frame model with an image of a corresponding object, control points of the wire frame model are determined based on distance information from a reference to each point on the object. Feature point extracting means for extracting a feature point on the object image corresponding to the above, and matching for matching the corresponding control point of the wire frame model with each feature point on the object image extracted by the feature point extracting means. Means for matching a wire frame model.

That is, since the feature points are extracted based on the distance information, accurate feature point extraction becomes possible. In the conventional example, information in the depth direction is supplemented by a distance image after wire frame matching, and the distance image does not survive feature point extraction.

(2) Based on the distribution of the distance information,
The wire frame model matching device according to the above (1), further comprising a subdividing means for subdividing the wire frame model.

That is, an object can be faithfully reproduced in a portion where the characteristic amount changes drastically.

(3) The characteristic point extracting means has a curvature calculating means for calculating a curvature at each point on the object based on the distance information, and the characteristic point extracting means determines the curvature based on the curvature information obtained by the curvature calculating means. The wire frame model matching device according to (1), wherein a feature point on the object image corresponding to a control point of the wire frame model is extracted.

That is, attention is paid to the curvature as the feature amount of the object shape.

(4) Based on the distribution of the curvature information,
The wire frame model matching device according to the above (3), further comprising a subdividing means for subdividing the wire frame model.

That is, the object can be faithfully reproduced in a portion where the characteristic amount changes drastically.

(5) The feature point extracting means has differentiating means for obtaining a differential value of the distance information at each point on the object, and the wire frame model is obtained based on the differential value obtained by the differentiating means. The feature point on the object image corresponding to the control point is extracted.

That is, attention is paid to the edge component as the feature amount of the object shape.

(6) Based on the distribution of the differential information,
The wire frame model matching device according to the above (5), further comprising a subdividing means for subdividing the wire frame model.

That is, an object can be faithfully reproduced in a portion where the characteristic amount changes greatly.

(7) The feature point extracting means includes a feature on an object image corresponding to a control point of the wire frame model, based on an all-around type distance image having distance information at each point in all directions of the object. Wherein the matching means includes a developing means for developing the wireframe model on a plane, and matches the developed wireframe model to the all-around distance image. The wire frame model matching device according to 1).

That is, matching in all directions is possible.

(8) The distance information includes an all-around distance image, and the expanding means includes cylindrical coordinate conversion means for performing cylindrical coordinate conversion on the wire frame model. Wire frame model matching device.

That is, the full-circle distance image input by the full-circle range finder is matched with the wire frame model developed on the plane by the cylindrical coordinate conversion, and accurate matching can be obtained.

(9) In a wire frame model matching method for matching a three-dimensional wire frame model to an image of a corresponding object, a step of inputting distance information from a reference to each point on the object, based on the distance information A feature point extraction step of extracting feature points on the object image corresponding to the control points of the wireframe model, and, for each feature point on the extracted object image, a control point corresponding to the wireframe model. And a matching step of matching.

That is, since feature points are extracted based on distance information, accurate feature point extraction is possible. In the conventional example, information in the depth direction is supplemented by a distance image after wire frame matching, and the distance image does not survive feature point extraction.

(10) The wire frame model matching method according to (9), further comprising a subdividing step of subdividing the wire frame model based on the distribution of the distance information.

That is, an object can be faithfully reproduced in a portion where the characteristic amount changes drastically.

(11) The feature point extracting step includes a curvature calculating step of calculating a curvature at each point on the object based on the distance information, and the characteristic point extracting step is performed based on the curvature information obtained in the curvature calculating step. The wireframe model matching method according to (9), wherein a feature point on the object image corresponding to a control point of the wireframe model is extracted.

That is, attention is paid to the curvature as the feature amount of the object shape.

(12) The wire frame model matching method according to (11), further comprising a subdividing step of subdividing the wire frame model based on the distribution of the curvature information.

That is, an object can be faithfully reproduced in a portion where the characteristic amount changes drastically.

(13) The feature point extracting step includes a differentiating step of obtaining a differential value of the distance information at each point on the object, and the wire frame model is obtained based on the differential value obtained by the differentiating step. The method according to (9), wherein feature points on the object image corresponding to the control points are extracted.

That is, attention is paid to the edge component as the feature amount of the object shape.

(14) The wire frame model matching method according to (13), further comprising a subdividing step of subdividing the wire frame model based on the distribution of the differential information.

That is, an object can be faithfully reproduced in a portion where the characteristic amount changes drastically.

(15) In the feature point extracting step, the feature on the object image corresponding to the control point of the wire frame model is based on an all-around type distance image having distance information at each point in all directions of the object. This is the process of extracting points,
The wire frame model according to (9), wherein the matching step includes a developing step of developing the wire frame model on a plane, and matching the developed wire frame model with the all-around distance image. Alignment method.

That is, matching in all directions is possible.

(16) The distance information includes an all-around distance image, and the expanding step includes a cylindrical coordinate conversion step of performing cylindrical coordinate conversion on the wire frame model. Wireframe model matching method.

That is, the entire circumference type range image input by the full circumference range finder is matched with the wire frame model developed on the plane by the cylindrical coordinate conversion, and accurate matching can be obtained.

(17) A machine-readable recording medium storing a matching program for matching a three-dimensional wireframe model to a corresponding object image, wherein the matching program is transmitted from a computer to a computer from a reference. The feature points on the object image corresponding to the control points of the wire frame model are extracted based on the distance information to each point, and the feature points on the extracted object image correspond to the feature points on the wire frame model. A recording medium storing a matching program for matching a three-dimensional wireframe model with a corresponding object image, wherein the matching program matches control points.

That is, since feature points are extracted based on distance information, accurate feature point extraction becomes possible. In the conventional example, information in the depth direction is supplemented by a distance image after wire frame matching, and the distance image does not survive feature point extraction.

(18) The matching program causes the computer to subdivide the wireframe model based on the distribution of the distance information, and supports the three-dimensional wireframe model according to (17). A recording medium storing a matching program for matching an image of an object to be adjusted.

That is, an object can be faithfully reproduced in a portion where the characteristic amount changes drastically.

(19) The matching program causes a computer to calculate a curvature at each point on the object based on the distance information, and based on the obtained curvature information, a computer corresponding to a control point of the wire frame model. A recording medium storing a matching program for matching a three-dimensional wire frame model according to the above (17) with a corresponding object image, wherein feature points on the object image are extracted.

That is, attention is paid to the curvature as the feature amount of the object shape.

(20) The three-dimensional wireframe model according to (19), wherein the matching program causes the computer to subdivide the wireframe model based on the distribution of the curvature information. A recording medium storing a matching program for matching an image of a corresponding object.

That is, an object can be faithfully reproduced in a portion where the characteristic amount changes drastically.

(21) The matching program causes a computer to determine a differential value of the distance information at each point on the object, and based on the obtained differential value, the object corresponding to the control point of the wire frame model. A recording medium storing a matching program for matching a three-dimensional wireframe model according to the above (17) with an image of a corresponding object, wherein a feature point on the image is extracted.

That is, attention is paid to the edge component as the feature amount of the object shape.

(22) The matching program according to (21), wherein the matching program causes a computer to subdivide the wire frame model based on the distribution of the differential information. A recording medium storing a matching program for matching an image of a corresponding object.

That is, an object can be faithfully reproduced in a portion where the characteristic amount changes greatly.

(23) The above-mentioned matching program allows the computer to execute a feature on an object image corresponding to a control point of the above-mentioned wire frame model on the basis of an all-around type distance image having distance information at each point in all directions of the object. A point is extracted, the wireframe model is developed on a plane, and the developed wireframe model is matched with the all-around distance image. The three-dimensional wireframe model described in (17) above is supported. A recording medium on which a matching program for matching an image of an object is recorded.

That is, matching in all directions is possible.

(24) The three-dimensional image according to (23), wherein the distance information includes an all-around distance image, and the matching program causes the computer to perform cylindrical coordinate conversion on the wire frame model. A recording medium on which a matching program for matching a wireframe model to a corresponding object image is recorded.

That is, the full-circle range image input by the full-circle range finder is matched with the wire frame model developed on the plane by the cylindrical coordinate conversion, and accurate matching can be obtained.

[0104]

As described in detail above, according to the present invention,
A wire frame model matching apparatus and method for matching a three-dimensional wire frame model with a corresponding object image, which enables accurate feature point extraction, so that a natural image can be obtained even when a profile is displayed, and the like. Three
A machine-readable recording medium on which a matching program for matching a two-dimensional wireframe model with a corresponding object image can be provided.

[Brief description of the drawings]

FIG. 1A is a block configuration diagram of a wire frame model matching device according to a first embodiment of the present invention;
(B) to (D) are diagrams showing the three-dimensional structure of the wireframe model of the face from the front, the oblique direction, and the lateral direction, respectively.

FIGS. 2A to 2D are photographs of an intermediate image displayed on a display, showing all-around color information (texture image), all-around distance information, a front texture image, and a front distance image, respectively. .

FIGS. 3A to 3C are full-circle curvature images, respectively.
It is a photograph of an intermediate image displayed on a display showing a front curvature image and a front edge image.

FIG. 4A is a diagram showing control points on a wire frame, and FIG. 4B is a diagram for explaining a feature point extraction window.

FIG. 5 is a diagram showing the order of extracting feature points.

FIG. 6 is a diagram showing a wireframe model subjected to cylindrical coordinate conversion.

FIGS. 7A and 7B are diagrams illustrating examples of how to divide a triangular patch according to the second embodiment of the present invention.

[Explanation of symbols]

 REFERENCE SIGNS LIST 10 distance image input device 20 display device 30 arithmetic processing device 32 control unit 34 memory 36 curvature operation unit 38 differentiation operation unit 40 feature extraction unit 42 matching unit 44 development unit 46 subdivision unit 100 whole-circumference type color information (texture image) 102 Full circumference type distance information 104 Front texture image 106 Front distance image 108 Full circumference curvature image 110 Front curvature image 112 Front edge image 202 Window

Claims (3)

[Claims] 1. A wireframe model matching apparatus for matching a three-dimensional wireframe model to an image of a corresponding object, comprising: a distance information from a reference to each point on the object;
Feature point extraction means for extracting feature points on the object image corresponding to the control points of the wire frame model; and correspondence of the wire frame model to each feature point on the object image extracted by the feature point extraction means. And a matching means for matching the control points to be performed. 2. A wire frame model matching method for matching a three-dimensional wire frame model to an image of a corresponding object, comprising: inputting distance information from a reference to each point on the object; Extracting a feature point on the object image corresponding to the control point of the wireframe model, and matching the corresponding control point of the wireframe model to each feature point on the extracted object image. A wire frame model matching method, comprising: 3. A machine-readable recording medium on which a matching program for matching a three-dimensional wireframe model to an image of a corresponding object is recorded. A feature point on the object image corresponding to the control point of the wire frame model is extracted based on distance information to each point, and a control corresponding to the wire frame model is provided for each feature point on the extracted object image. A recording medium storing a matching program for matching a three-dimensional wire frame model to a corresponding object image, wherein the matching is performed on points.