JPH11211440A - Three-dimensional shape restoring device and method, and computer readable storage medium having three-dimensional restoring program recorded thereon - Google Patents

Abstract

PROBLEM TO BE SOLVED: To precisely restore the three-dimensional shape in a short time on the basis of luminance information from an image data obtained by photographing a three-dimensional target. SOLUTION: A color image data obtained by photographing a target having a three-dimensional shape is fetched by an image fetching part 24, a normal vector detecting part 26 divides the target into, for example, minute planes of picture element unit, and detects the normal vector of each minute plane on the basis of luminance information. A three-dimensional shape restoring part 50 calculates the three-dimensional data of the target on the basis of the normal vector group, and restores and displays it. The normal vector detecting part 26 divides the normal vector of the minute lane in the target to be photographed into a light source directional vector vertical component and a vector horizontal component orthogonal to the light source direction. The vertical component is proportional to the luminance of the minute plane, and the horizontal component is determined from the tangential direction of the equivalence line of the luminance.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a three-dimensional shape restoring apparatus and method for creating three-dimensional shape data of an object from a color image of the object having a three-dimensional shape and restoring the data by computer graphics. In addition, the present invention relates to a computer-readable recording medium on which a three-dimensional shape restoration program is recorded, and in particular, to quickly and accurately restore a three-dimensional shape of an object based on a luminance image converted from a color image of the three-dimensional object. And a computer-readable recording medium on which a three-dimensional shape restoring program is recorded.

[0002]

2. Description of the Related Art In recent years, with the spread of digital cameras, digital videos, and the like, photographed image data is taken into a personal computer and used for presentations and the like. In this case, if the three-dimensional shape data can be generated on the personal computer from the captured image data, the object captured by the computer graphics can be restored, and the use range of the captured image data is greatly expanded.

Conventionally, a method of restoring the three-dimensional shape of an object based on luminance information of image data obtained by photographing the object having the three-dimensional shape is generally called “Shape From Shading”.
Known as the problem, various approaches have been attempted. Typical examples of the restoration method include a characteristic curve expansion method and a relaxation method. The characteristic curve development method is a method formulated by Horn to obtain three-dimensional shape information by solving an illumination equation of an image, that is, a non-linear first-order differential equation relating to a surface inclination along a characteristic curve. Specifically, a characteristic curve is obtained by sequentially connecting points where the difference in brightness is the largest around an arbitrary point on the screen. The inclination of the surface on the characteristic curve is obtained from the characteristic curve and the reflectance distribution map, and the three-dimensional shape is reconstructed.

The relaxation method is a method based on outer edge line constraint and smooth constraint. On the outer edge of the curved body, that is, on the apparent contour, the normal vector of the surface is squared with the square error near the pixel and the reflection function and the luminance of the observed image using the initial condition that the normal vector is orthogonal to the line of sight. To minimize the sum of the squared errors,
This is a method of solving the surface inclination of a simultaneous equation proportional to the number of pixels by the Gauss-Seidel iterative method.

In other words, the relaxation method utilizes the fact that the inclination of the surface can be determined by the inclination of the tangent of the outline on the contour line of the object, and using this as the boundary condition, the inclination of the surface of the entire shape is determined to obtain the three-dimensional shape. Reconfigure.

[0006]

However, the conventional method of restoring a three-dimensional shape based on luminance information of a photographed image has the following problems. First, in the characteristic curve expansion method, to find a solution along the characteristic curve,
A solution cannot be obtained in a portion where the characteristic curve cannot be obtained. At the starting point of the characteristic curve, it is necessary to approximately give the surface inclination artificially, but an error at this time tends to affect the entire restored shape.

In the relaxation method, it is difficult to accurately determine an apparent contour in a natural image or the like, and an error is likely to be included in an initial condition. Further, since simultaneous equations proportional to the number of pixels are solved by an iterative method, a long time is required for processing. The present invention has been made in view of such a conventional problem, and based on image data obtained by photographing a three-dimensional object,
It is an object of the present invention to provide a three-dimensional shape restoring device and method capable of restoring a three-dimensional shape accurately and in a short time based on luminance information, and a computer-readable recording medium recording a three-dimensional shape restoring program. .

[0008]

FIG. 1 is a diagram illustrating the principle of the present invention. FIG. 1A shows a three-dimensional shape restoration apparatus according to the present invention.
And an image capturing unit 24 that captures color image data obtained by capturing an object having a three-dimensional shape, and divides the object into minute surfaces in, for example, pixel units, and normal vectors of each minute surface based on luminance information. Normal vector extraction unit 26 for extracting
And a three-dimensional shape restoring unit 50 for calculating and restoring and displaying three-dimensional shape data of the object based on the normal vector group.

In the present invention, as shown in FIG.
The normal vector 72 of the minute surface 70 of the target object 66 to be photographed is decomposed into a vector vertical component 76 in the line-of-sight direction 68 and a vector horizontal component 74 of a surface 78 orthogonal to the line-of-sight direction. Proportional and horizontal component 7
No. 4 is determined from the tangent direction of the luminance contour.
The normal vector detection unit 26 includes a luminance image conversion unit 28 that converts a color capture image into a luminance image, a horizontal component extraction unit 30 that extracts a horizontal component of a normal vector of each minute surface based on the luminance image, A vertical component calculation unit 32 that extracts a vertical component of a normal vector of each minute surface based on the normal vector normalization that calculates a normal vector normalized by a horizontal component and a vertical component of the normal vector of each minute surface A part 48 is provided.

The horizontal component 74 of the normal vector extracted by the horizontal component extraction unit 30 is a unit vector length (a size of 1.0) projected on a plane orthogonal to the viewing direction 68 of the object 66 in FIG. The vertical component 76 of the normal vector extracted by the vertical component extraction unit 32 is a component in the line-of-sight direction proportional to the luminance of the target object. The horizontal component extraction unit 30 includes a luminance image smoothing unit 34 for smoothing the luminance image, and a luminance contour conversion for rounding the pixel luminance value of the smoothed luminance image by an arbitrary step width to convert it into a luminance contour image (luminance contour image). Unit 36, a luminance contour line generating unit 38 that generates a luminance contour line at an arbitrary step width based on the luminance contour image, and calculates a horizontal component of a normal vector for each surface of a small area on the luminance contour line A horizontal component calculator 40 is provided.

The luminance contour generating unit 38 searches for a pixel whose luminance value changes on the contour image, for example, as a tracking start point, and examines eight surrounding pixels around the tracking start point in a predetermined direction. If the luminance value of the pixel is the same as the tracking start point, and the luminance value of the pixel of interest next is greater than the tracking start point,
The current pixel of interest is set as the next tracking start point, and a plurality of tracking start points are connected to generate a luminance contour line.

The horizontal component calculation unit 40 generates a tangent vector of a luminance isoline passing through a minute surface, and obtains a horizontal component of a normal vector by rotating the tangent vector by 90 degrees. in this case,
When the surface shape on which the minute surface exists is a convex surface, the horizontal component of the normal vector is obtained by rotating the tangent vector of the luminance isoline by 90 degrees in a direction in which the luminance becomes dark. On the other hand, when the surface shape on which the minute surface exists is concave, the tangent vector of the luminance isovalue line is rotated 90 degrees in the direction in which the luminance becomes brighter, and the horizontal component of the normal vector is obtained.

The vertical component extraction unit 32 calculates the vertical component of the normal vector as a value proportional to the luminance value of each minute surface of the smoothed luminance image. Specifically, a photographing condition input unit 42 for inputting photographing conditions such as a line-of-sight direction and a photographing direction when a color image is photographed, a reflection table defining a relationship between a surface angle of a processing target object with respect to the line-of-sight direction and a luminance value. And a vertical component calculation unit 46 that calculates the vertical component of the normal vector of each surface by referring to the reflection table based on the luminance value of each minute surface of the luminance image.

The reflection table generation unit 44 generates a linear reflection table storing a luminance value B proportional to the surface angle θ,
The vertical component calculator 46 calculates the brightness Bs of each minute surface of the brightness image.
Is multiplied by a predetermined scale coefficient K to convert to a luminance value B of a linear reflection table, and a corresponding surface angle θ is obtained by referring to the linear reflection table based on the luminance value B, and a vertical component Nz of a normal vector of length 1 is obtained. Is calculated by Nz = sin θ.

The reflection table generator 44 calculates the surface angle θ.
In addition to the linear reflection table storing the luminance value B proportional to the linear reflection table, a non-linear reflection table in which the relationship of the luminance value B to the surface angle θ of the linear reflection table is modified based on the material of the object, the type of the light source, and the like is generated. . In this case, the vertical component calculation unit 46 obtains the brightness value B of the reflection table by multiplying the brightness Bs of each minute surface of the smoothed brightness image by a predetermined scale coefficient K, and then obtains the nonlinear reflection by the table brightness value B. The plane angle θ is obtained by referring to the table, and the vertical component Nz of the normal vector is calculated.

As the scale factor K used in the vertical component calculator 46, a ratio K = (B / Bs) between the luminance value B of the reflection table and the luminance value Bs of the photographed image at the same surface angle θ is calculated. For example, a ratio K = (Bmax / Bsmax) between the maximum luminance value Bmax of the reflection table and the luminance value Bsmax of the captured image at a surface angle of 90 degrees perpendicular to the light source direction is calculated.

The normal vector normalizing section 48 converts a normal vector obtained by combining the horizontal component (Nx, Ny) and the vertical component (Nz) of the unit vector length (size 1) into the unit vector length (size 1.0). That is, the normal vector normalizing unit 48 calculates a normalized horizontal component HL for synthesizing a normal vector having a unit vector length based on the vertical component Nz as HL = √ (1−Nz ² ).

Next, the ratio (HL / 1.0) of the normalized horizontal component HL to the unit vector length of 1.0 is multiplied by each of the horizontal components (Nx, Ny) of the normal vector to obtain a normalized horizontal component (Nnx). , Nny) as Nxn = Nx · (HL / 1.0) Nyn = Ny · (HL / 1.0) to normalize the normal vector.

A three-dimensional shape restoring unit 50 shown in FIG. 1A integrates a normalized normal vector to generate a three-dimensional shape of a target range, and a result of restoring the three-dimensional shape. The display unit 52 includes a restoration display unit 52 for displaying as a message. The integration processing unit 51 sets a reference plane orthogonal to the line-of-sight direction, sets a height in the line-of-sight direction to a projected micro-plane on the reference plane with an arbitrary micro-plane of the target object as an initial position, and sets an initial value. Calculate the height from the reference plane sequentially by calculating the height variation of the other minute planes adjacent to the minute plane from the inclination of the plane based on the normal vector and adding it to the initial height value. To generate three-dimensional shape data.

Further, an image range selection unit 25 for selecting an image range for restoring a three-dimensional shape from the captured color image.
Is provided. The image range selection unit 25 labels the selection range using the number of selections of each pixel data as a label number. The image range selection unit 25 specifies, for example, an arbitrary color pixel included in the object, and selects a pixel that has a color pixel value within a predetermined allowable range with respect to the color pixel value of the specified pixel and is continuous with the specified pixel.

The present invention also provides a three-dimensional shape restoring method, in which an image capturing step of capturing color image data of an object having a three-dimensional shape is performed; A normal vector extraction step of extracting a normal vector of each minute surface based on the image; a three-dimensional shape restoring step of calculating and restoring and displaying three-dimensional shape data of the object based on the normal vector group It is characterized by.

The normal vector detection process in the three-dimensional shape restoration process includes a brightness image conversion process of converting a color captured image into a brightness image; a horizontal component for extracting a horizontal component of a normal vector of each minute surface based on the brightness image. Extraction process; Vertical component extraction process for extracting the vertical component of the normal vector of each minute surface based on the luminance image; Method of calculating the normal vector normalized by the horizontal and vertical components of the normal vector of each minute surface A line vector normalizing step.

The three-dimensional shape restoration process includes an integration process of generating a three-dimensional shape of a target range by integrating a normalized normal vector; and a restoration display process of displaying the three-dimensional shape as a restoration result. Prepare. Other points are the same as those of the three-dimensional shape restoration device.

Further, the present invention provides a computer-readable recording medium storing a three-dimensional shape restoring program. The image capturing part captures color image data of an object having a three-dimensional shape. A normal vector extraction unit that divides an object into minute surfaces and extracts the normal vector of each minute surface based on luminance information, and calculates and restores the three-dimensional shape data of the object based on the normal vector group A three-dimensional shape restoring unit that performs

Here, the normal vector detecting section includes a luminance image converting section for converting the color captured image into a luminance image, a luminance image smoothing section for smoothing the luminance image, and a normal vector detection section for each minute surface based on the luminance image. A horizontal component extraction unit that extracts a horizontal component,
A vertical component extracting unit that extracts a vertical component of a normal vector of each minute surface based on a luminance image, and a normal that calculates a normal vector normalized by a horizontal component and a vertical component of the normal vector of each surface of the minute region. A vector normalization unit is provided.

The three-dimensional shape restoration unit includes an integration processing unit that generates a three-dimensional shape of the target range by integrating the normalized normal vector, and a restoration display unit that displays the three-dimensional shape as a restoration result. Prepare. Other points are the same as those of the three-dimensional shape restoration device.

[0027]

DESCRIPTION OF THE PREFERRED EMBODIMENTS <Table of Contents> 1. Device configuration and function 2. Image capture and range selection 3. Extraction of horizontal component of normal vector 4. Extraction of normal vector vertical component Normalization of normal vector Reconstruction of three-dimensional shape FIG. 2 is a block diagram of the hardware configuration of the three-dimensional shape restoration device according to the present invention. In FIG. 2, the three-dimensional shape restoration apparatus of the present invention has a processing device 10 using a computer device such as a PC / AT compatible device.
For example, a digital camera device 12, a video camera device 14, and an image scanner device 16 are provided as image pickup devices for inputting a color image to be processed.

The interface of the digital camera device 12, the video camera device 14, and the image scanner device 16 with the processing device 10 is, for example, SCSI or RS23.
2C or the like is used. The processing device 10 using the PC / AT compatible has a main memory 18 of, for example, 32 megabytes or more.
It has. The processing device 10 is a digital camera device 1
2. Capture color image data from the video camera device 14 or the image scanner device 16 and, for any target (or target range) included in the color image data,
After converting to a luminance image, the normal vector of each surface, for example, a minute surface that is a pixel unit, which constitutes the object, is obtained, and the shape that creates the obtained normal vector group is obtained in reverse, thereby obtaining the three-dimensional shape of the object. Generate data and restore the object.

The processing device 10 is followed by a 3D accelerator 20. The processing device 10 receives three-dimensional shape data generated from a group of normal vectors of a minute surface and inputs the three-dimensional shape data to an arbitrary device by a computer graphics technique. The image of the object viewed from the line of sight is restored and displayed on the display 22. FIG. 3 is a functional block diagram of the three-dimensional shape restoration processing of the present invention realized by the three-dimensional shape restoration software installed in the processing device 10 of FIG.

Referring to FIG. 3, the three-dimensional shape restoring device of the present invention comprises an image capturing unit 24, an image range selecting unit 25, a normal vector extracting unit 26, and a shape restoring unit 50.
The image capturing unit 24 captures, for example, RGB color image data from the digital camera device 12, the video camera device 14, or the image scanner device 16. Image range selection unit 2
Reference numeral 5 performs labeling for selecting an image range in which a three-dimensional shape in a color image captured by the image capturing unit 24 is to be restored. The normal vector extraction unit 26 obtains the normal vector of each of the minute surfaces constituting the object,
Focusing on the luminance information of the color image, a normal vector of a minute surface is extracted by dividing it into a horizontal component and a vertical component.

Here, the horizontal component and the vertical component obtained by decomposing the normal vector of the minute surface of the object are determined as shown in FIG. 4 irradiates the object 66 with light from a light source direction 69 and captures an image of the object from a line-of-sight direction 68. Focusing on a certain minute surface 70 on the object 66, a normal vector 72 is set outside the minute surface 70 orthogonal to the surface direction.

In the present invention, the normal vector 72 is decomposed into a horizontal component 74 and a vertical component 76 and extracted. The horizontal component 74 is a vector component parallel to a plane 78 orthogonal to the viewing direction 68 with respect to the object 66. The vertical component 76 is a vector component parallel to the viewing direction 68 with respect to the target 66. Here, a plane 78 orthogonal to the line-of-sight direction 68 is defined as a three-dimensional coordinate (X, Y, Z) by a two-dimensional coordinate (X, Y, Z).
Y) The plane is set, and the line-of-sight direction 68 is set as the Z-axis direction. Therefore, assuming that the magnitude of the normal vector 72 is N, the horizontal component 74 is represented by (Nx, Ny), and the vertical component 76 is represented by (Nz).

Referring again to FIG. 3, in the normal vector extraction unit 26, the vertical component 76 of the normal vector 72 shown in FIG. The horizontal component 74 is determined by the tangential direction of the luminance isolines passing through the minute surface 70, considering the luminance isolines on the object 66. Since the normal vector is decomposed into a horizontal component and a vertical component and extracted as described above, the normal vector extracting unit 2
6 is provided with a luminance image conversion unit 28, followed by a horizontal component extraction unit 30 and a vertical component extraction unit 32.

The horizontal component extracting section 30 is provided with a luminance image smoothing section 3.
4, a luminance contour converter 36, a luminance iso-line generator 38, and a horizontal component calculator 40. On the other hand, the vertical component extraction unit 32 includes a shooting condition input unit 42 for setting a light source direction and a shooting direction, a reflection table creation unit 44, and a vertical component calculation unit 46. Further, the normal vector extraction unit 26 is provided with a normal vector normalization unit 48. As will be described later, the normal vector normalization unit 48 obtains the horizontal component of the normal vector in the horizontal component extraction unit 30 as a unit vector component having a size of 1, and the vertical component Is calculated as the correct vector magnitude proportional to the actual luminance value, so that the horizontal component based on the vertical component is changed so that it becomes the horizontal component of the normal vector having a unit vector length of magnitude 1. Normalize based on components.

The shape restoration unit 50 provided following the normal vector extraction unit 26 includes an integration processing unit 51 and a restoration display unit 52.
Is provided. The integration processing unit 51 performs integration processing to reversely obtain a three-dimensional shape that generates a normal vector group of each minute surface of the object extracted by the normal vector extraction unit 26, and generates three-dimensional shape data of the object. I do. The restoration display unit 52 creates two-dimensional display data by setting an arbitrary line-of-sight direction to the three-dimensional shape data of the object obtained by the integration processing unit 51, and uses the RGB data obtained from the read color image. The object is restored using computer graphics and displayed on the display.

FIGS. 5 and 6 show processed images according to the processing procedure of the three-dimensional shape restoration apparatus of the present invention shown in FIG. First, FIG. 5A is a color captured image 54 captured from an appropriate imaging device by the image capturing unit 24 in FIG.
The captured image 54 of the fist holding the right hand is captured. This color capture image 54 is the same as the normal vector extraction unit 26 shown in FIG.
Is converted into a luminance conversion image 56 as shown in FIG. 5B.

The luminance conversion image 56 is further processed by the luminance contour converter 36 and the luminance contour generator 38 shown in FIG. 3 to generate a luminance contour image 58 as shown in FIG. 5C. The luminance contour image 58 is a set of lines connecting pixels of the same luminance in the luminance conversion image 56. FIG. 6 is a set of horizontal components extracted by the horizontal component calculation unit 40 provided in the normal vector extraction unit 26 of FIG.
A normal vector horizontal component extraction image 60 as shown in FIG. 5D is obtained based on the luminance iso-line image 58 in FIG.

On the other hand, the vertical component extraction unit 32 shown in FIG. 3 extracts a normal vector vertical component for each minute surface as shown in FIG. 6E based on the luminance information of the luminance conversion image 56 shown in FIG. An image 62 is determined. FIG. 6 (D)
The extracted normal vector horizontal component image 60 is a vector horizontal component that has been normalized based on the vertical component by the normal vector normalization unit 48 in FIG.

When the normal vector horizontal component extraction image 60 of FIG. 6 (D) and the normal vector vertical component extraction image 62 of FIG. 6 (E) are obtained in this manner, the normal vector group of FIG. An integration processing unit 51 provided in the shape restoration unit 50 performs integration processing for obtaining a three-dimensional shape for creating a normal vector group in reverse to generate three-dimensional shape data of the object, and based on the three-dimensional shape data. The restoration display unit 52 generates and displays a three-dimensional shape restoration image 64 as shown in FIG. In the three-dimensional shape restored image 64, the object of the color captured image 54 viewed from the front in FIG.

FIG. 7 is a flowchart of the processing procedure of the three-dimensional shape restoring apparatus of the present invention shown in FIG. First, in step S1, a color image captured by the digital camera device 12, the video camera device 14, or the image scanner device 16 is captured by the image capturing unit 24. Next, in step S2, an arbitrary target range is selected by the image range selection unit 25 for the captured color image, and labeling is performed using a label number using one or more values.

Here, the user can freely select the target range for the color captured image as needed.
If not particularly required, the entire range may be labeled and selected. Also, when selecting the target range, the user can arbitrarily set a label number 0 for a part that is not to be restored, so that the part is excluded from the three-dimensional shape restoration processing according to the present invention, and the three-dimensional shape restoration according to the present invention is performed. The three-dimensional shape may be supplemented by post-processing after the processing is completed.

Next, in step S3, the captured color image is converted into a luminance image by the luminance image conversion unit 28, and further, in step S4, the luminance image is smoothed. In the smoothing of this luminance image, even if it is recognized as the same color by human eyes in a natural image or the like, there is a part where the value is extremely different when viewed as a numerical value as a luminance value. However, when the three-dimensional shape is restored, it has an adverse effect as noise, so that smoothing is performed to eliminate an extreme shift in the luminance value.

Next, in step S4, a horizontal component of a normal vector of a pixel forming a micro surface is extracted based on the luminance image. In the extraction of the horizontal component, a luminance contour line is generated after conversion into a luminance contour image, and finally the horizontal component is calculated based on the tangent vector. Subsequently, in step S6, the vertical component of the normal vector of each pixel corresponding to the minute surface of the luminance image is extracted. The extraction of the vertical component of the normal vector is performed by inputting shooting conditions such as a light source direction and a shooting direction, creating a reflection table for defining a correspondence relationship between luminance and a surface angle of a minute surface for an object, and acquiring a captured image. Also, the vertical component of the normal vector is calculated by referring to the reflection table based on the pixel luminance value.

Next, in step S7, a normal vector of magnitude 1 normalized by the horizontal and vertical components of the normal vector of each pixel constituting the object is calculated. Specifically, a horizontal component corresponding to a normal vector having a magnitude of 1 is obtained based on the vertical component. If the normal vector group of the pixel corresponding to each minute surface of the object can be extracted in this way, in step S8, an integration process for inversely finding the shape of the minute surface that produces the normalized normal vector group In step S9, two-dimensional shape data is generated from the three-dimensional shape data created by setting an arbitrary line-of-sight direction, and the corresponding color pixel value of the color image captured in step S1 To restore the object.

Next, the details of the three-dimensional shape restoration processing of the present invention shown in FIGS. 3 and 7 will be described. 2. Image Capture and Range Selection In the digital camera device 12 shown in FIG. 3, a captured image is stored in a storage device such as a built-in memory or a mini disk. The video camera device 14 stores the captured image on a tape. Further, the image recorded on paper or film is read by the image scanner 16 so that
It can be digitized color image data.

The color image data from the digital camera device 12, the video camera device 14, or the image scanner device 16 is supplied to the image capturing unit 24 as shown in FIG.
The image is captured as a color captured image as shown in FIG. 8A, and the captured result is stored in, for example, an RGB table 82 as shown in FIG. The RGB table 82 exemplifies (4 × 4) dot pixels P11 to P44 as an example.
14, the R component 84, the G component 85, and the B component 86
Stores GB pixel values.

Each of the R component 84, the G component 85, and the B component 86 is 8-bit data, and takes a value of 0 to 255. Further, a valid flag 88 is provided, and by setting bit 1, the RGB pixel value is valid, and by setting bit 0, the RGB pixel value is invalid. This valid flag 88
The value stored in the table indicates whether or not the processing is significant. The processing is performed only on the pixels for which the validity flag 88 is valid, thereby improving the overall processing speed.

In the case of the RGB table 82 shown in FIG. 8, the valid flags 88 are all valid at the time of image reading, and when a target range described later is selected, the validity within the target range is invalid and the validity outside the target range is invalid. Become. The color image data fetched as in the RGB table 82 in FIG. 8 is provided to the image range selection unit 25 in FIG. 3 to select a target range for a three-dimensional shape restoration target.

The flowchart of FIG. 9 is a flowchart of the target range selection processing by the image range selection unit 25 of FIG. 3. The target range can be automatically selected by specifying an arbitrary pixel on the color image. . That is, when an arbitrary pixel on the color image is specified in step S1, in step S2, a pixel value of the specified pixel, that is, a pixel having an RGB value included in an arbitrary allowable width α based on the RGB value, A pixel following the designated pixel is selected and registered in a label table 90 as shown in FIG. 10 in step S3.

The label table 90 shown in FIG.
Number of selections 92 used as label values for each of 1 to P44
And a valid flag 88 are provided. The valid flags 88 are all invalid when reading an image. The number of selections 92 is “0” in the initial state, and the value of the number of selections 92 changes from “0” to “1” when selected as a pixel in the target range by the selection processing of FIG. When the number of selections 92 is plural, the cumulative value of the number of selections is stored.

As a result, the validity flag 88 is set to bit 1 for the area in which the value of the number of selections 92 registered in the label table 90 is “1” or more, and the area becomes valid, and the target range is selected. For example, taking the color capture image 54 of FIG. 5A as an example, if an arbitrary “skin color” pixel of the hand portion in the color capture image 54 is specified in step S1 of FIG. The selection of other consecutive pixels having a predetermined tolerance ± α for the pixel is automatically performed,
It is possible to select a skin color region that is a hand portion as the target range.

Here, in the selection process of FIG. 9, the automatic selection of the target range by designating an arbitrary pixel is taken as an example. Set on the display,
Pixels included inside the region surrounded by the closed loop shape may be selected as the target range. 3. Extraction of Normal Vector Horizontal Component In the normal vector extraction unit 26 of FIG. 3, first, the color image captured by the luminance image conversion unit 28 is converted into a luminance image. FIG. 11 is a flowchart of a luminance image conversion process according to the present invention. First, in step S1, the target range is recognized by referring to the label table 90 created as shown in FIG. 10, and pixel values of the pixels in the target range are extracted from the RGB table 82 in FIG.

Subsequently, in step S2, a normalized luminance value Y having a value of 0.0 to 1.0 and a luminance value yn having a gradation value of 0 to 255 are calculated by the following equations. Y = 0.299 R + 0.587 G + 0.114 B (1) Yn = Y · 255 (2) The conversion formula to this luminance value is obtained from the RGB color space by YIQ.
This is the same as the conversion formula for the Y signal known as the conversion formula for the primary color system. In step S2, the normalized luminance value Y and the luminance value Yn
After the calculation is completed, it is registered in the luminance table 94 as shown in FIG. 12 in step S3, and the process is repeated for the pixels in the entire target range in step S4.

The luminance table 94 shown in FIG.
For each of .about.P44, a normalized luminance value Y, a luminance value Yn, and a valid flag 88 are registered as shown in, for example, a pixel P14. The valid flag 88 stores the same contents as the RGB table 82 in which the target range is selected. The conversion from the color image to the luminance image is intended to easily extract the influence of the light source on the surface of the object and accurately extract the normal vector. May be used.

FIG. 13 is a flowchart of a luminance image smoothing process performed by the luminance image smoothing unit 34 of FIG. 3 following the luminance image conversion process of FIG. In the luminance image smoothing process, in the case of a color image such as a natural image, in a luminance image converted from a color image, a natural image or the like can be recognized as the same color by human eyes, but a numerical value as a luminance value. There are parts where the values are extremely different.

Such an extreme change in the luminance value adversely affects the restored image as noise when the three-dimensional shape is restored by the processing of the present invention. Therefore, the value of the target pixel is compared with the values of the surrounding pixels, and a smoothing process for correcting an extreme value shift between the pixels is performed. That is, in the luminance image smoothing processing of FIG. 13, first, one pixel of the luminance table 94 of FIG. 12 is extracted in step S1, and a smoothing operation is performed in step S2. This smoothing operation is, for example, any of the following.

The average of the surrounding pixels is set as the pixel value of the pixel of interest. The surrounding pixels are arranged in order of size, and the pixel value of the central pixel is set as the pixel value of the pixel of interest. Subsequently, in step S3, the smoothed pixel values are stored again in the luminance table 94 in FIG.
Is repeated for all target pixels.

FIG. 14 is a flowchart of the luminance contour conversion processing performed by the luminance contour conversion unit 36 of FIG. 3 subsequent to the luminance image smoothing processing of FIG. In the luminance contour conversion processing, the smoothed image is converted into a luminance image having a contour-like pixel distribution by performing an arithmetic processing for rounding the smoothed image at an arbitrary luminance step size. That is, the brightness table 9 in FIG. 12 after the smoothing process in FIG.
4, one pixel is extracted, and in step S2, the luminance value of the pixel is rounded at a predetermined interval.

The rounding includes, for example, first rounding off, rounding off or rounding up a luminance value having a gradation value of 0 to 255. The contour pixels obtained by this rounding are used in step S3 in the contour table 10 of FIG.
In step S4, the same process is repeated for all target pixels. The contour table 100 of FIG.
Stores the contour luminance value 102 and the valid flag 88 for the pixels P11 to P14, for example, as in the pixel P14. The valid flag 88 is a luminance table 94 shown in FIG.
Contains the same contents as.

FIG. 16 is a flowchart of the luminance contour conversion processing performed by the luminance contour generator 38 of FIG. 3 subsequent to the luminance contour conversion processing of FIG. Here, the luminance contour line can be referred to as a kind of contour line connecting boundary pixels having different luminance values in the contour image generated in the processing of FIG. FIG. 17 explains the processing contents of the luminance iso-line conversion processing of FIG. FIG. 17A is an example of a contour image generated by the luminance contour conversion processing of FIG. 14, and exemplifies 45 pixels of (5 × 9). In the luminance isoline conversion processing of FIG. 16, for example, the pixel P0 having the luminance value “20” in FIG. 17A is set as the tracking start point, and the pixel P0 at the tracking start point is shown in FIG. As shown in FIG. 7, the surrounding eight pixels are examined counterclockwise (counterclockwise).

At this time, if the following condition is satisfied, the adjacent pixel is registered as the next tracking start point. Condition: The focused pixel value is the same as the tracking start point. Condition: The pixel value of the next target pixel is larger than the tracking start point. When such a condition is checked in a counterclockwise order from the pixel P1 around the pixel P0 at the tracking start point in FIG. 17B, the sixth pixel P6 satisfies the condition. Therefore, the pixel P6 is recorded as the next tracking start point. When such processing is performed on the contour pixels 104 in FIG. 17A, the luminance contour 106 can be generated as shown in FIG.

This tracking start point detection processing is terminated when the detected point of interest is detected or when the next tracking start point cannot be detected, and the processing shifts to the processing by setting the next new tracking start point. . FIG. 1 shows the conversion process of the luminance isolines.
The flowchart of No. 6 will be described as follows. First, in step S1, for example, as shown in FIG. 17A, a boundary where the luminance value changes is searched for the contour image 104 on the contour table and stored as a tracking start point.
Subsequently, the process proceeds to step S3 via step S2, and FIG.
As shown in FIG. 7 (B), attention is paid to eight adjacent pixels around the tracking start point in order counterclockwise.

When attention is paid to one of the adjacent pixels, it is first checked in step S4 whether the luminance value of the pixel of interest is the same as the tracking start point. If they are the same, it is checked in step S5 whether the luminance value of the next target pixel is larger than the tracking start point. If it is larger than the tracking start point, it is checked in step S6 whether or not the pixel is the detected pixel of interest. If it is not, the current pixel of interest is stored as the next tracking start point in step S7.

If the condition of step S4 or step S5 is not satisfied, the process returns to step S3, and the same processing is repeated focusing on the next adjacent pixel. Such processing is repeated until there is no new tracking start point in step S2. FIG. 18 is a flowchart of a horizontal component extraction process performed by the horizontal component calculation unit 40 of FIG. 3 subsequent to the luminance isoline conversion process of FIG. This horizontal component extraction processing is performed as shown in FIG.
For each pixel on the brightness contour line 106 obtained as shown in FIG. 17C in the brightness contour conversion process 6, a tangent vector to the brightness contour line of each pixel is obtained, and the tangent vector is rotated by 90 degrees. This is a process of setting the obtained vector as the horizontal component of the normal vector.

FIG. 19 illustrates the principle of the horizontal component extraction processing of FIG. FIG. 19A shows a part of a spherical object formed by a smiling face, and the micro face 70 has a normal vector 72. The horizontal component of the normal vector 72 is a horizontal component 74 obtained by projecting the normal vector 72 in parallel onto a projection plane 78 orthogonal to the light source direction 68. On the other hand, FIG. 19 (B) shows a tangent vector 1 of the luminance contour 106 passing through the minute surface 70 generated based on the luminance information of the object 66.
It is 10. When this tangent vector 110 is rotated 90 degrees in a direction in which the luminance of the object becomes dark, that is, counterclockwise, 9
The vector rotated by 0 degrees corresponds to the projection plane 78 in FIG.
Is the same as the horizontal component 74 that is the projection component of the normal vector 72 projected on the horizontal axis.

Therefore, in the present invention, as shown in FIG. 19 (B), the tangent vector 110
Is rotated 90 degrees in the direction in which the brightness decreases,
The horizontal component 74 of the normal vector 72 is calculated. FIG.
The following describes the horizontal component extraction processing with reference to the flowchart of FIG. First, in step S1, for example, FIG.
Attention is paid to a pixel on an arbitrary contour line in the luminance contour image 58 obtained as shown in FIG. Next, in step S2,
A unit vector in the tangential direction of the pixel of interest is obtained from pixels before and after the same contour line. As a method of obtaining the unit vector in the tangential direction, for example, as shown in FIG. 20, taking the pixel of interest P7 of a black circle as an example, the pixels P1 to P6 and P8 to P13 before and after located on the luminance equivalent line 106 are The tangent vector of the pixel of interest P7 is determined as the average of the sum of the directional vectors connecting the two.

Here, the case where the luminance isovalue line 106 is represented by a polygonal line is taken as an example, but if it can be converted into a curve, the center of curvature of the curved part passing through the pixel is obtained, and the tangent vector in the 90-degree direction with respect to the center of curvature is obtained. Should be obtained. Referring again to FIG. 18, if a unit vector having a magnitude of 1 in the tangential direction to the luminance contour is obtained in step S2,
In step S3, it is checked whether the target surface is a convex surface. Here, if the surface on which the minute surface 70 of the object exists is a convex surface as shown in FIGS. 19A and 19B, the process proceeds to step S4, and the unit vector in the tangential direction is set to 9 in the direction in which the luminance becomes darker.
Rotate by 0 degrees to obtain the horizontal component of the normal vector.

Here, the horizontal component of the normal vector is represented by a two-dimensional horizontal component (Nx, Ny) since it is a plane parallel to a plane 78 orthogonal to the viewing direction 68 in FIG. On the other hand, if the surface of the object is a concave concave surface, the process proceeds to step S5, in which the unit vector in the tangential direction is rotated by 90 degrees in the direction of becoming brighter, and the horizontal component (Nx, N
y).

When the horizontal component (Nx, Ny) of the normal vector is obtained in this way, in step S6, the horizontal components (Nx, Ny) of FIG.
1 is registered in the normal vector table 114 as shown in FIG. The normal vector table 114 stores an X-axis horizontal component Nx, a Y-axis horizontal component Ny, a vertical component Nz, and a valid flag 88, for example, as indicated by a pixel P14. The valid flag 88 stores the same content as the contour table 100 in FIG. Such processing of steps S1 to S6 is repeated until all target pixels are completed in step S7. 4. Extraction of Normal Vector Vertical Component FIG. 22 is a flowchart of the normal component vertical component extraction processing by the vertical component extraction unit 32 of FIG. Here, the description will be made assuming that the difference between the direction of the light source and the line of sight is negligibly small. In the vertical component extraction processing, first, in step S1, the reflection table creation unit 4 shown in FIG.
4 creates a reflection table based on the photographing conditions obtained from the photographing condition input unit 42. Focusing on the target object 66 illuminated in the line-of-sight direction 68 as shown in FIG.
A portion of the surface of the surface 6 facing the line of sight 68 is bright and therefore has a high brightness, and a portion facing the direction different from the light source is dark and has a low brightness.

Therefore, it can be considered that the luminance of each pixel on the luminance image obtained for the object 66 corresponds to the orientation of the minute surface on which the pixel exists to the light source. The conversion of the direction of the luminance value of the pixel to the light source of the minute surface on which the pixel exists is performed using the reflection table created in step S1 of FIG. FIG. 23 shows the characteristics of the reflection table created in step S1 of FIG. This reflection table has a table brightness value B on the horizontal axis and an angle θ of the surface with respect to the light source direction on the vertical axis. Here, if the table brightness value B is normalized by 0.0 to 1.0, the table brightness B is ideally 1.0 when the surface brightness is the largest θ = 90 degrees, and the angle with respect to the light source direction is ideal. When θ = 0 degrees, the table luminance value B is 0.0.

However, in practice, the table brightness value B does not become 1.0 when the surface angle θ is 90 °, but becomes, for example, 0.8 to 0.7 when θ = 90 ° as shown in FIG. Become. Further, the table brightness value B does not become 0.0 when the surface angle θ = 0, but as shown in FIG.
0.2 to 0.0. As a result, if there is a proportional relationship between the surface angle θ and the luminance value B, the characteristic in this case is a linear table characteristic 124 represented by a straight line.

The calculation of the vertical component Nz of the normal vector in proportion to the luminance value using the linear reflection table characteristic 124 is performed as follows. First, a minute surface of the object on which the angle θ with respect to the light source direction is known, that is, the luminance value B of the pixel
s is acquired by the photographing condition input unit 42. The pixel luminance value Bs in this case is desirably the pixel luminance maximum value B smax when the surface angle θ with respect to the light source direction becomes θ = 90 degrees.

When the maximum pixel luminance value Bmax of θ = 90 degrees of the object is obtained in this way, the scale coefficient K for converting into the scale of the maximum value Bmax of the table luminance value B in FIG. It is determined by the formula. K = Bmax / Bsmax (3) If the scale coefficient K can be calculated in this way, the table brightness value B can be calculated by multiplying the brightness value Bs of each pixel in the target range by the scale coefficient K. Based on the calculated luminance value B, the corresponding surface angle θ1 is determined by referring to the linear reflection table 124, and the vertical component Nz of the normal vector with the length set to 1 is determined by Nz = sinθ (4).

On the other hand, the surface angle θ of the object with respect to the light source direction
23 is not necessarily limited to a linear relationship such as the linear reflection table characteristic 124 in FIG. 23, and the change in the brightness value B with respect to the surface angle θ depends on the material of the object,
In many cases, a plurality of approximate functions are obtained depending on the type of the light source.
Therefore, by taking into account the material of the object, the type of the light source, and the like, for example, the nonlinear reflection table characteristic 1 shown in FIG.
26 can also be created.

The calculation of the vertical component Nz of the normal vector using the nonlinear reflection table characteristic 126 is performed as follows. First, the pixel luminance value Bs in the target range is extracted, and the table luminance value B is obtained by multiplying the pixel luminance value Bs by the scale coefficient K calculated by the linear reflection table characteristic 124. Next, based on the table luminance value B = K × Bs calculated from the scale coefficient K, the corresponding surface angle θ is obtained by referring to the nonlinear reflection table 126, and the vertical component N of the normal vector is obtained from the equation (4).
Calculate z.

For example, as shown in FIG.
When calculating the vertical component Nz from s1, the table brightness value B1 is obtained by multiplying the pixel brightness value Bs1 by the scale coefficient K, and the corresponding surface angle θ2 is obtained by referring to the non-linear reflection table characteristic 126 based on the table brightness value B1. , (4), the vertical component Nz of the normal vector is obtained. The process of extracting the vertical component of the normal vector using the reflection table of FIG. 23 will be described below with reference to the flowchart of FIG. If a linear or non-linear reflection table is created in step S1, in step S2 the surface angle 9
The scale coefficient K is calculated from the luminance value Bsmax of the known object at 0 degrees and the full-scale luminance value Bmax of the reflection table.

Next, in step S3, the luminance value Bs of the target pixel
Is multiplied by a scale factor K to calculate the luminance value B of the table reference pixel.
Is calculated. Subsequently, in step S4, the corresponding surface angle θ is obtained by referring to the linear or nonlinear reflection table based on the luminance value B obtained in step S3, and is substituted into equation (4) to obtain the vertical component Nz of the normal vector. In step S5, the vertical component Nz of the normal vector corresponding to the obtained surface angle θ is registered in the normal vector table. The processing from steps S3 to S5 is repeated until all the target pixels are completed in step S6.

When the difference between the direction of the light source and the direction of the line of sight is large, a linear or non-linear reflection table is created for each horizontal component of the normal vector of the surface, and the horizontal component value of the normal vector is adjusted. The vertical angle component Nz of the normal vector is calculated by obtaining the surface angle θ by referring to the reflection table. 5. Normal Vector Normalization FIG. 25 is a flowchart of a normal vector normalization process performed by the normal vector normalization unit 48 of FIG. 3 subsequent to the extraction process of the vertical component of the normal vector of FIG. In the normalization process of the normal vector, the horizontal component (Nx, Ny) of the normal vector obtained in the process of FIG. 18 is a unit vector having a size of 1, and the vertical component of the normal vector obtained in the process of FIG. Since the normal vector obtained by combining with the component (Nz) does not become a unit vector having a size of 1, the process of normalizing this to a unit normal vector having a size of 1 is performed.

FIG. 26A shows a normal vector 7 before normalization.
2, which is a composite vector of a horizontal vector 74 of a horizontal component (Nx, Ny) having a size of 1 and a vertical vector 75 of a size Nz. Therefore, the normal vector 7
2 is not a unit vector of size 1. Therefore, as shown in FIG. 26B, the horizontal component of the horizontal vector 74 is normalized based on the vertical component Nz of the vertical vector 75 so that the normal vector 72 becomes a unit vector of size 1 (Nxn, Nyn).

The normalization of the normal vector 72 of FIG. 26A to the normal vector 70 of magnitude 1 in FIG. 26D is performed according to the flowchart of FIG. First, in step S1, the target pixel is extracted from the normal vector table 114 of FIG. 21, and in step S2, based on the vertical component Nz, the size of the horizontal vector 74 to become the normal vector 72 of size 1 in FIG. HL is calculated by the following equation.

HL = √ (1−Nz ² ) (4) Next, in step S 3, a normalized horizontal component (Nxn, Nyn) that gives a horizontal vector 74 having a size HL that can be combined with a normal vector 70 of size 1 ) Is calculated by the following equation. Nxn = Nx · (HL / 1.0) (5) Nyn = Ny · (HL / 1.0) (6) Calculation of horizontal component (Nxy, Nyn) for normalizing the normal vector in this way Is obtained, this is registered in the normal vector table 114 of FIG. 21 in step S4. Valid flag 8 of normal vector table 114
8 stores the same contents as the normal line table 114 in FIG. Such processing of steps S1 to S4 is repeated until all the target pixels are completed in step S5. 6. 27 is a flowchart of the normal vector integration process performed by the integration processing unit 51 provided in the shape restoration unit 50 of FIG. 3 subsequent to the normal vector normalization process of FIG. This normal vector integration process is a process for obtaining a surface shape for creating a normal vector group obtained for each pixel that is a minute surface of the object in reverse.

FIG. 29 shows an example of the normal vector integration processing according to the present invention. In FIG. 29, a reference plane 134 orthogonal to the viewing direction is set at an arbitrary position with respect to the target object 66. First, an arbitrary pixel forming the object 66, for example, a pixel P having a normal vector 72-1 coincident with the line-of-sight direction 68
1 is set as an initial position, and the height h0 from the reference plane 134 is set as an initial value.

Next, paying attention to the pixel P2 adjacent to the pixel P1, the displacement Δh of the height in the viewing direction passing through the center 140 is calculated based on the normal vector. The displacement Δh is such that the distance between the pixels P1 and P2 on the reference plane 134 is a constant value L, and the normal vector 72 of the center point 140 of the pixel P2.
Since the vertical component Nz and the X-axis component Nx of −2 are known, the following relationship is obtained.

Nx / Nz = Δh / (L / 2) (7) When the displacement Δh is obtained from the equation (7), the following is obtained: Δh = (Nx / Nz) (P / 2) (8) Therefore, the height h of the pixel P2 from the reference plane 134 is as follows: h = h0 + Δh = h0 + (Nx / Nz) (P / 2) (9) When the height h has been obtained for the pixel P2 in this manner, the height h from the reference plane 134 is obtained for the next adjacent pixel P3 using the obtained height as a new initial value h0.

The process of calculating the height from the reference plane by the normal vector integration process will be described with reference to the flowchart of FIG. 27. First, in step S1, pixels on the object whose normal vector coincides with the line of sight are determined. As the start position, the height from the reference plane 134 set as a plane orthogonal to the line-of-sight direction as shown in FIG. 29 is set to the initial value h0. Next, in step S2, NΔh in the optical axis direction is calculated from the normal vector of the adjacent pixel, and the height h is obtained. Then, in step S3, it is stored in the height table 128 as shown in FIG. Next, in step S4, the currently calculated height h is set as the next initial value h0, and the processing in steps S1 to S3 is repeated until all target pixels are completed in step S5.

The height h0 of each pixel of the object 66 with respect to the reference plane 134 of FIG. 29, that is, the relative coordinate value Z is obtained by the integration processing of the normal vector, and the relative coordinate value Z of each pixel with respect to the reference plane 134 is obtained. The three-dimensional shape data of the target object can be obtained together with the two-dimensional coordinate values (X, Y). When the three-dimensional shape data of the object is obtained in this way, the restoration display unit 52 provided in the shape restoration unit 50 of FIG.
For example, a two-dimensional projection shape is generated from a three-dimensional shape by setting an arbitrary line-of-sight direction as shown in FIG. 6F, and the RG of each pixel obtained from the color capture image of FIG.
The restored image can be displayed using the B value.

Here, the integration processing for obtaining the three-dimensional shape data from the normal vector is performed by using the horizontal reference as shown in FIG. 27 because the normal vector is the differential value of the inclination angle of the minute surface constituting the object. In addition to the integration processing for sequentially obtaining the heights with respect to the plane, the height h from the reference plane can be obtained by integrating the normal vector groups collectively. In the three-dimensional shape restoration processing of the present invention, there is no problem with a single-color object such as a hand that can be extracted by designating a skin color as shown in FIGS. When the luminance conversion image 56 is generated as shown in FIG. 5B, the luminance values of the non-skin color parts such as eyes and mouths having different colors greatly differ from each other.
The luminance contour image 58 as shown in FIG. 4C cannot be obtained, and the normal vector cannot be generated properly.

In such a case, a three-dimensional shape restoring process excluding the flesh-colored portion is performed, and the eyes and mouth portions which are unprocessed portions after completion are interpolated from the processed peripheral height h. The three-dimensional shape of the eyes and mouth may be determined. Furthermore, the present invention is embodied as a computer-readable recording medium recording a program for executing a three-dimensional shape restoration process. The recording medium includes a removable portable storage medium such as a CD-ROM or a floppy disk, A storage device of a program provider that provides a program through a line, and a storage device of a processing device in which the program is installed.
There are memory devices such as an AM and a hard disk. The three-dimensional shape restoration program provided by the recording medium is loaded into the processing device and executed on the main memory.

The present invention is not limited to the above-described embodiment, and can be appropriately modified without impairing the object and advantages. The present invention is not limited by the numerical values shown in the embodiments.

[0090]

As described above, according to the present invention, attention is paid to the luminance information of the image data obtained by photographing the three-dimensional object, and the normal vector of the minute surface constituting the object is horizontally determined from the luminance information. The component is extracted separately from the vertical component. In particular, the vertical component is proportional to the magnitude of the luminance value, and the horizontal component is determined by rotating the tangent vector of the luminance isoline by 90 degrees. This solves the problem that the three-dimensional shape can be reconstructed only within a range where the characteristic curve in the curve expansion method can be obtained, and the three-dimensional shape can be reliably reconstructed from the entire image.

In addition, the problem that the conventional relaxation method cannot be applied unless the contour of the object is obtained, and that it takes time to repeat the calculation for each pixel even if the contour is obtained can be solved. Further, according to the present invention, it is possible to reconstruct a three-dimensional shape with uniform accuracy over the entire image area from image data that is efficiently captured with sufficient practical processing time at a processing capability of a personal computer level for the entire image.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating the principle of the present invention.

FIG. 2 is a block diagram of an apparatus configuration of the present invention.

FIG. 3 is a functional block diagram of the present invention.

FIG. 4 is an explanatory diagram of a horizontal component and a vertical component of a normal vector according to the present invention.

FIG. 5 is an explanatory diagram of a captured image, a luminance conversion image, and a luminance contour image according to the present invention.

FIG. 6 is an explanatory diagram of a horizontal component and a vertical component of a normal vector and a restored image according to the present invention.

FIG. 7 is a flowchart of a three-dimensional shape restoration process according to the present invention.

FIG. 8 is an explanatory diagram of an RGB table of a captured image.

FIG. 9 is a flowchart of a target range selection process according to the present invention.

FIG. 10 is an explanatory diagram of a label table generated by the selection processing of FIG. 9;

FIG. 11 is a flowchart of a luminance image conversion process of the present invention.

FIG. 12 is an explanatory diagram of a luminance table generated by the processing of FIG. 11;

FIG. 13 is a flowchart of a luminance image smoothing process according to the present invention.

FIG. 14 is a flowchart of luminance contour conversion processing of the present invention.

FIG. 15 is an explanatory diagram of a contour table generated in the process of FIG.

FIG. 16 is a flowchart of a luminance iso-line conversion process according to the present invention.

FIG. 17 is an explanatory diagram of the luminance iso-line conversion processing of FIG. 15;

FIG. 18 is a flowchart of horizontal component extraction processing according to the present invention.

FIG. 19 is an explanatory diagram of a principle of extracting a horizontal component.

FIG. 20 is an explanatory diagram of a method of calculating a tangent vector at a pixel on a luminance isovalue line.

FIG. 21 is an explanatory diagram of a normal line table for storing horizontal and vertical components of the normal line.

FIG. 22 is a flowchart of a vertical component extraction process according to the present invention.

FIG. 23 is an explanatory diagram of a reflection table used for vertical component extraction.

FIG. 24 is an explanatory diagram of luminance values and surface angles for generating the linear reflection table of FIG. 23;

FIG. 25 is a flowchart of normal vector normalization processing of the present invention.

FIG. 26 is an explanatory diagram for normalizing a normal vector.

FIG. 27 is a flowchart of normal vector integration processing of the present invention.

FIG. 28 is an explanatory diagram of a height table obtained by the integration processing of FIG. 27;

FIG. 29 is an explanatory diagram of the integration processing of FIG. 27;

[Explanation of symbols]

 10: Processing Device 12: Digital Camera Device 14: Video Camera Device 16: Image Scanner Device 18: Main Memory 20: 3D Accelerator 22: Display 24: Image Capture Unit 25: Target Range Selection Unit 26: Normal Vector Extraction Unit 28 : Luminance image conversion unit 30: horizontal component extraction unit 32: vertical component extraction unit 34: luminance image smoothing unit 36: luminance contour conversion unit 38: luminance contour line generation unit 40: horizontal component calculation unit 42: shooting condition input unit 44 : Reflection table creation unit 46: vertical component calculation unit 50: shape restoration unit 51: integration processing unit 52: restoration display unit 54: color capture image 56: brightness conversion image 58: brightness equivalent line image 60: normal vector horizontal Component extraction image 62: Normal vector vertical component extraction image 64: Three-dimensional shape restoration image 66: Object 68: Viewing direction 70: Fine Surface 72: Normal vector 74: Horizontal component 76: Vertical component 78: Horizontal component projection surface 80: Projected image 82: RGB table 84: R component 85: G component 86: B component 88: Valid flag 90: Label table 92: Selected cumulative number 94: Luminance table 96: Normalized luminance value 98: Luminance value 100: Contour table 102: Contour luminance value 104: Contour image 106, 106-1: Luminance equality line 110: Tangent vector 114: Normal line table 116 : X-axis horizontal component Nx 118: Y-axis horizontal component Ny 120: Vertical component Nz 124: Linear reflection table characteristic 126: Nonlinear reflection table characteristic 128: Height table 130: Height h 134: Reference plane 136: Object

Claims (16)

[Claims] An image capturing unit that captures color image data of an object having a three-dimensional shape; divides the object into minute surfaces; and extracts a normal vector of each minute surface based on luminance information. A three-dimensional shape restoring device, comprising: a normal vector extracting unit that performs a normal vector extraction; and a shape restoring unit that calculates and restores and displays three-dimensional shape data of the object based on the normal vector group. 2. The three-dimensional shape restoring device according to claim 1, wherein the normal vector extracting unit is configured to convert a color captured image into a luminance image, A horizontal component extraction unit that extracts a horizontal component of a normal vector of a minute surface; a vertical component extraction unit that extracts a vertical component of a normal vector of each minute surface based on the luminance image; A three-dimensional shape restoring device, comprising: a normal vector normalization unit that calculates a normal vector normalized by a horizontal component and a vertical component of the normal vector. 3. The three-dimensional shape restoration apparatus according to claim 2, wherein the horizontal component of the normal vector extracted by the horizontal component extraction unit is projected on a plane orthogonal to a line-of-sight direction in which the target object is photographed. The vertical component of the normal vector extracted by the vertical component extraction unit is a vector component in the line-of-sight direction of the target object. 4. The three-dimensional image restoration device according to claim 3, wherein the horizontal component extraction unit includes a luminance image smoothing unit that smoothes the luminance image, and a pixel luminance value of the smoothed luminance image. A luminance contour conversion unit that converts the luminance contour image into a luminance contour image by rounding at an arbitrary step width; a luminance contour line generating unit that generates a luminance contour line at an arbitrary step width based on the luminance contour image; A horizontal component calculator for calculating a horizontal component of a normal vector for each of the minute surfaces. 5. The three-dimensional image restoration apparatus according to claim 4, wherein the luminance contour generating unit searches for a pixel whose luminance value changes on the contour image and sets the pixel as a tracking start point. If the luminance value of the focused pixel is the same as the tracking start point and the luminance value of the next focused pixel is larger than the tracking start point, the search is performed sequentially by focusing on neighboring eight pixels around the start point in a predetermined direction. A three-dimensional shape restoring apparatus characterized in that a current pixel of interest is set as a next tracking start point, and the plurality of tracking start points are connected to generate a luminance contour line. 6. A three-dimensional image restoration apparatus according to claim 4, wherein said horizontal component calculation unit generates a tangent vector of said luminance isolines passing through said minute surface, and generates said tangent vector by 90.
A three-dimensional shape restoring device, wherein the horizontal component of the normal vector is obtained by rotating by a degree. 7. The three-dimensional image restoration apparatus according to claim 6, wherein the surface on which the minute surface exists is a convex surface.
A three-dimensional shape restoration apparatus characterized in that a tangent vector of the luminance isoline is rotated by 90 degrees in a direction in which luminance becomes dark to obtain a horizontal component of the normal vector. 8. The three-dimensional image restoration apparatus according to claim 6, wherein the surface on which the minute surface exists is concave.
A three-dimensional shape restoration apparatus, wherein a tangent vector of the luminance isoline is rotated by 90 degrees in a direction in which the luminance becomes brighter to obtain a horizontal component of the normal vector. 9. The three-dimensional image restoration device according to claim 3, wherein the vertical component extraction unit is configured to calculate a vertical component of the normal vector in proportion to a luminance value of each minute surface of the smoothed luminance image. A three-dimensional shape restoring device characterized by calculating the calculated value. 10. The three-dimensional shape restoration device according to claim 1, wherein the three-dimensional shape restoration unit generates a three-dimensional shape of the object by integrating the normalized normal vector. A three-dimensional shape restoration device comprising: an integration processing unit; and a restoration display unit that displays the three-dimensional shape as a restoration result. 11. An image capturing step of capturing color image data of an object having a three-dimensional shape, dividing the object into minute surfaces, and extracting a normal vector of each minute surface based on luminance information. A normal vector extraction process, and a three-dimensional shape restoration process of calculating and restoring and displaying three-dimensional shape data of the object based on the normal vector group,
A three-dimensional shape restoration method characterized by comprising: 12. The three-dimensional shape restoring method according to claim 11, wherein said normal vector detecting step comprises: a luminance image converting step of converting a color captured image into a luminance image; A horizontal component extraction process of extracting a horizontal component of a normal vector of the micro surface; a vertical component extraction process of extracting a vertical component of a normal vector of each of the micro surfaces based on the luminance image; A normal vector normalization step of calculating a normal vector normalized by a horizontal component and a vertical component of the normal vector. 13. The three-dimensional shape restoring method according to claim 11, wherein in the three-dimensional shape restoring step, a three-dimensional shape of the target range is generated by integrating the normalized normal vector. A method for restoring a three-dimensional shape, comprising: an integration processing step; and a restoration display step of displaying the three-dimensional shape as a restoration result. 14. An image capturing section for capturing color image data of an object having a three-dimensional shape, dividing the object into minute surfaces, and extracting a normal vector of each minute surface based on luminance information. A computer storing a three-dimensional shape restoring program, comprising: a normal vector extracting unit that performs the calculation; and a three-dimensional shape restoring unit that calculates and restores and displays three-dimensional shape data of the object based on the normal vector group. A readable recording medium. 15. A computer-readable recording medium on which a three-dimensional shape restoration program according to claim 14 is recorded, wherein said normal vector detecting unit converts a color captured image into a luminance image. A horizontal component extraction unit that extracts a horizontal component of a normal vector of each minute surface based on the luminance image, and a vertical component extraction unit that extracts a vertical component of a normal vector of each minute surface based on the luminance image A three-dimensional shape restoration program, comprising: a normal vector normalizing unit that calculates a normal vector normalized by a horizontal component and a vertical component of a normal vector of each surface of the minute area. Computer-readable recording medium on which is recorded. 16. A computer-readable recording medium having recorded thereon a three-dimensional shape restoring program according to claim 14, wherein said three-dimensional shape restoring unit integrates said normalized normal vector. A computer-readable recording recording a three-dimensional shape restoration program, comprising: an integration processing unit that generates a three-dimensional shape of a target range; and a restoration display unit that displays the three-dimensional shape as a restoration result. Medium.