JP2008090590A - Method and device for image conversion to regular image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To accurately convert a photographic image taken from an angle to an overhead regular image. <P>SOLUTION: An image input part 110 inputs the photographic image obtained by photograhing an object placed within a rectangular reference frame 35 on a stage 30 from an angle by a camera 60 as image data on an xy coordinate. A linear graph calculation part 120 converts positions (x, y) of individual pixels on the xy coordinate to a linear graph μ on a Hough transform ξρ coordinate using angle ξ and distance ρ, and a pixel value integration part 130 integrates, to each pixel located on the linear graph μ on the ξρ coordinate, the pixel value of the corresponding pixel on the xy coordinate. On the other hand, the position of the reference frame 35 on the xy coordinate is predicted based on a geographic photographing condition, and each side thereof is plotted on the ξρ coordinate. An actual accurate position of the reference frame 35 on the xy coordinate is determined from positions of pixels having the maximum pixel values of pixels near the plotted points, whereby the conversion to regular image is performed. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image conversion technique for converting a photographed image obtained by photographing a subject from an oblique direction into a regular image viewed from directly above, and in particular, high-dimensional texture data (reflection characteristics) for rendering. The present invention relates to an image conversion technique that can be used when creating data.

  Due to the improvement in computer performance, CG images are being used in various fields of industry. For example, in the design stage of buildings, furniture, automobiles, etc., many CG images are usually used. Also, various CG images of articles are indispensable for various video expressions such as product presentations and movies using computers. Furthermore, recently, there are many examples in which CG images are used instead of actual product photos in product catalogs and the like.

  Article data such as buildings, furniture, and automobile interior parts is three-dimensional data of a virtual object, whereas a CG image for presentation is usually prepared as a two-dimensional image. Therefore, when creating a CG image using 3D article data, the 3D data of the virtual object is taken into the computer, the illumination conditions, the viewpoint position, and the projection plane are set, and then the 3D virtual A rendering process for obtaining a two-dimensional projection image of the object on the projection plane is performed. This process basically simulates the phenomenon in which the illumination light from the light source determined by the illumination conditions is reflected at each part of the virtual object and travels to the viewpoint position on the computer. It can be said that the light intensity is calculated.

  Also, if you want to create an expression with a pattern or pattern pasted on the surface of a virtual object, you need to prepare two-dimensional texture data in advance and map this texture data to the surface of the object data during rendering. Done. The texture data is basically reflection characteristic data indicating a distribution of reflection characteristics, and is formally given as two-dimensional image data. By mapping this on the surface of the virtual object, it is possible to express an object having a different reflection characteristic for each part.

  Recently, a method using high-dimensional image data instead of simple two-dimensional image data as reflection characteristic data has been proposed. For example, Non-Patent Document 1 below discloses a rendering process that uses texture data represented by a six-dimensional function called BTF (Bi-directional Texture Function). This BTF is a function that defines reflection characteristics that vary depending on the incident direction and reflection direction of light, and includes two angles θL and φL that indicate the incident direction of illumination light and two angles θL and φL that indicate the emission direction of reflected light. And a total of six variables (θL, φL, θV, φV, u, v), which are coordinate values u and v in the two-dimensional uv coordinate system, become a function that defines specific reflection characteristics.

On the other hand, as a technique for expressing a texture close to an actual object, a technique is known in which the reflection characteristics of the actual object are measured and reflection characteristic data is created based on the measured values. For example, Patent Literature 1 below discloses a technique for photographing the grain of a natural tree and creating reflection characteristic data having a texture of the natural tree based on the photographed image. Thus, when actually measuring the reflection characteristics of an actual object, it is necessary to photograph the object from various directions, but in order to create reflection characteristic data of the object surface based on these captured images, It is necessary to convert the captured image taken from an arbitrary direction into a regular image viewed from directly above. Such an image conversion process can be performed by a geometric operation. For example, Patent Document 2 below discloses an image conversion processing technique called projective normalization conversion.
G. Muller, J. Meseth, M. Sattler, R. Sarlette and R. Klein 2004, Acquisition, Synthesis and Rendering of Bidirectional Texture Functions. In Eurographics 2004 STAR-State of The Art Report JP 2002-071329 A JP-A-6-274649

  As described above, in order to obtain reflection characteristic data given as high-dimensional image data such as BTF, when a method of actually measuring the reflection characteristic of an actual object is employed, it is obtained by photographing the object from an oblique direction. It is necessary to perform image conversion for converting a captured image into a regular image viewed from directly above. Such an image conversion technique is not limited to use in obtaining reflection characteristic data, and is a technique that can be widely applied to image processing in general. Conventionally, as disclosed in the above-mentioned Patent Document 2, based on geometric imaging conditions, an image conversion method using a purely geometric operation called projective normalization conversion. Has been used exclusively.

  However, when an image conversion method using such geometrical operations (projective normalization conversion based on shooting conditions, etc.) is applied to measured image data obtained by camera shooting of an actual object, accuracy There is a problem that high conversion cannot be performed. Of course, if an image conversion method using a geometric operation is used, a theoretically correct regular image should be obtained. However, even if a predetermined geometric shooting condition is set in advance and an actual object is shot under the shooting condition, actually, shooting is performed under the exact shooting condition as set. It is difficult. In other words, when photographing a real object, various factors such as an error related to the camera's relative position and orientation with respect to the object, a deviation of the optical axis of the camera, and lens aberration were involved. An error occurs between the photographed image and the photographed image predicted from the ideal photographing conditions. For this reason, when a geometric image conversion process based on ideal imaging conditions is performed on a captured image actually obtained, the obtained regular image is not accurate.

  Therefore, the present invention provides an image conversion method and apparatus capable of accurately converting a captured image obtained by photographing an actual object from an oblique direction into a regular image viewed from directly above. Objective.

(1) According to a first aspect of the present invention, a photographed image obtained by photographing a subject placed inside a reference frame composed of straight lines drawn on a plane from a predetermined direction is displayed on the plane. In the image conversion method for converting to a regular image viewed from the normal direction set at the reference point,
An image input stage for inputting a captured image or an image obtained by performing predetermined processing on the image as image data obtained by arranging pixels having predetermined pixel values in a two-dimensional xy coordinate system;
Using an expression form representing an arbitrary straight line on the xy coordinate system using a variable ξ indicating the orientation with respect to the coordinate axis and a variable ρ indicating the displacement with respect to the origin, the reference position of the pixel is set for each pixel constituting the image data. Obtain a combination of variables (ξ, ρ) for each of the straight line groups that pass through, plot the coordinate points corresponding to each variable pair (ξ, ρ) on the two-dimensional ξρ coordinate system, and configure the set of these coordinate points A line graph calculation step for obtaining a line graph μ to be obtained on the ξρ coordinate system for each pixel;
A two-dimensional pixel array for integration in which each pixel has the same initial pixel value is defined on the ξρ coordinate system, and the i-th pixel Pi constituting the image data is obtained on the two-dimensional pixel array for integration. The line graph μi is overlaid, and the pixel value of the pixel Pi is added to the pixel value of the pixel located on or near the line graph μi from the first pixel to the Nth pixel ( However, 1 ≦ i ≦ N, where N is the total number of pixels on the pixel array included in the image data)
A reference frame for predicting the position of the image of the reference frame that will appear on the photographed image arranged on the xy coordinate system based on the geometric photographing condition at the time of obtaining the photographed image by geometric calculation. A position prediction stage;
For each combination of variables (ξ, ρ) indicating each straight line constituting the image of the predicted reference frame, the corresponding coordinate points on the integrating two-dimensional pixel array are plotted as geometric prediction points ε, respectively. An actual prediction point determination stage that determines a reference position of a pixel having the maximum or minimum pixel value among pixels belonging to a predetermined neighboring region with respect to the geometric prediction point ε as an actual prediction point for the corresponding straight line. When,
On the two-dimensional xy coordinate system, draw a straight line represented by the coordinate values (ξ, ρ) of each measured predicted point, treat the position of these straight lines as the position of the correct reference frame image, and perform geometric calculation. A regular conversion stage for converting a captured image into a regular image;
Is to do.

(2) According to a second aspect of the present invention, in the image conversion method to a regular image according to the first aspect described above,
When a shot image is obtained by shooting using a reference frame consisting of a boundary line due to a contrast difference between the inside and outside of a predetermined area,
In the image input stage, a differential image obtained by performing differential processing on the captured image is input as image data.

(3) According to a third aspect of the present invention, in the image conversion method to a regular image according to the first or second aspect described above,
In the line graph calculation stage, when a perpendicular line is drawn from the origin O of the xy coordinate system to an arbitrary straight line on the xy coordinate system and the foot is set as a point F, one axis of the xy coordinate system and the line segment OF are formed. The angle is represented by ξ, the length of the line segment OF is represented by ρ, and an expression form representing a straight line by two variables ξ and ρ is used.

(4) According to a fourth aspect of the present invention, in the image conversion method to a regular image according to the first to third aspects described above,
In the pixel value integration stage, the pixel value of the pixel Pi is integrated with the pixel value of the pixel located on the line graph μ, and the pixel Pi is set to the pixel value of the pixel separated from the line graph μ by a predetermined neighborhood distance d. Are multiplied by a predetermined ratio R (a value that takes a range of 0 ≦ R ≦ 1 and decreases as the neighborhood distance d increases) corresponding to the neighborhood distance d.

(5) According to a fifth aspect of the present invention, in the image conversion method into a regular image according to the fourth aspect described above,
The predetermined ratio R is set so as to have a Gaussian distribution centered on the position of the line graph μ.

(6) According to a sixth aspect of the present invention, in the image conversion method to a regular image according to the first to fifth aspects described above,
In the line graph calculation stage, out of all N pixels, an operation for obtaining a line graph is performed only for pixels having pixel values within a predetermined range,
In the pixel value integration stage, pixel value integration processing is executed only for pixels having pixel values within a predetermined range among all N pixels.

(7) According to a seventh aspect of the present invention, in the image conversion method to a regular image according to the first to sixth aspects described above,
In the line graph calculation stage, out of all N pixels, an operation is performed to obtain a line graph only for pixels located within a predetermined consideration region,
In the pixel value integration stage, pixel value integration processing is executed only for pixels located within a predetermined consideration area among all N pixels.

(8) According to an eighth aspect of the present invention, in the image conversion method to a regular image according to the first to seventh aspects described above,
At the reference frame position prediction stage, as a geometrical imaging condition for obtaining a captured image, the angle θ between the normal line set at the reference point and the optical axis of the imaging device, and the optical axis “reference frame is drawn. Geometry based on the angle φ formed between the projected image on the plane and the predetermined reference line ζ included in the plane on which the reference frame is drawn through the reference point and the focal length of the optical system of the photographing apparatus The position of the image of the reference frame is predicted by a typical calculation.

(9) According to a ninth aspect of the present invention, in the image conversion method to a regular image according to the first to eighth aspects described above,
Predetermined deviations Δξ and Δρ are used when the coordinate value of the geometric prediction point ε is (ξε, ρε) as a predetermined neighborhood region for the geometric prediction point ε at the actual prediction point determination stage. Thus, an area indicated by coordinate values (ξ, ρ) satisfying the conditions of ξε−Δξ ≦ ξ ≦ ξε + Δξ and ρε−Δρ ≦ ρ ≦ ρε + Δρ is set.

(10) According to a tenth aspect of the present invention, in the image conversion method to a regular image according to the first to ninth aspects described above,
When there are a plurality of pixels having the maximum or minimum pixel value among the pixels belonging to the predetermined neighboring region with respect to the geometric prediction point ε at the actual measurement prediction point determination stage, the reference positions of the plurality of pixels Is determined as an actually predicted point for the corresponding straight line.

(11) According to an eleventh aspect of the present invention, in the image conversion method to a regular image according to the first to tenth aspects described above,
If the maximum pixel value in the vicinity region is less than the predetermined threshold value or the minimum pixel value exceeds the predetermined threshold value in the actual measurement prediction point determination stage, the actual measurement prediction point is determined. This is not performed, and handling that failed in image conversion to a regular image is performed.

(12) In a twelfth aspect of the present invention, in the image conversion method to a regular image according to the first to eleventh aspects described above,
When a shot image is obtained by shooting using a reference frame indicating a rectangle,
In the regular conversion stage, the image is converted into a regular image by a geometric operation based on the positions of the four vertices of the rectangle constituting the image of the reference frame.

(13) According to a thirteenth aspect of the present invention, by using the image conversion method to a regular image according to the first to twelfth aspects described above,
When illumination light is applied from a predetermined incident direction to a predetermined position on the surface to be measured of the object, the reflectance of the reflected light from the predetermined position toward the predetermined emission direction is set for each individual position on the surface to be measured. In the method for creating reflection characteristic data of an object, data defined for (m × n) cases by a combination of m incidence directions and a plurality of n emission directions is created as reflection characteristic data for a surface to be measured.
An object placement stage for placing an object having a measured surface inside a reference frame composed of straight lines drawn on a plane;
Change the positional relationship between the light source and the surface to be measured in a plurality of m, and change the positional relationship between the surface to be measured and the camera, so that the light source, the surface to be measured, and the camera have a predetermined positional relationship. A shooting condition setting stage for setting a total (m × n) by changing to a plurality of n ways,
For each of the set (m × n) shooting conditions, (m × n) object shot images are obtained by executing a shooting process of shooting an image of the measurement target surface illuminated by the light source with the camera. Object shooting stage to obtain,
Image conversion using the image conversion method to the regular image according to the first to twelfth aspects described above is performed on (m × n) object captured images, and (m × n) regular images are obtained. An image conversion stage to obtain;
A reflection characteristic data creation stage for creating reflection characteristic data of an object based on (m × n) regular images;
Is to do.

(14) According to a fourteenth aspect of the present invention, in the method for creating reflection characteristic data of an object according to the thirteenth aspect described above,
If the maximum pixel value in the neighboring region is less than the predetermined threshold value or the minimum pixel value is the predetermined threshold value in the measurement prediction point determination step in the image conversion step that converts the specific object photographed image into a regular image If it exceeds, do not determine the prediction point in actual measurement, handle the failed image conversion to a regular image related to the object captured image,
A regular image for an object-captured image that has been handled with failed image conversion is obtained by interpolation based on a regular image for another object-captured image that approximates the shooting conditions.

(15) According to a fifteenth aspect of the present invention, a photographed image obtained by photographing a subject arranged inside a reference frame composed of straight lines drawn on a plane from a predetermined direction is displayed on the plane. In an image conversion device that converts a regular image viewed from a normal direction set at a reference point,
An image input unit for inputting a captured image or an image obtained by performing predetermined processing on the image as image data obtained by arranging pixels having predetermined pixel values in a two-dimensional xy coordinate system;
Using an expression form representing an arbitrary straight line on the xy coordinate system using a variable ξ indicating the orientation with respect to the coordinate axis and a variable ρ indicating the displacement with respect to the origin, the reference position of the pixel is set for each pixel constituting the image data. Obtain a combination of variables (ξ, ρ) for each of the straight line groups that pass through, plot the coordinate points corresponding to each variable pair (ξ, ρ) on the two-dimensional ξρ coordinate system, and configure the set of these coordinate points A line graph calculation unit for obtaining a line graph μ to be calculated on the ξρ coordinate system for each pixel;
A two-dimensional pixel array for integration in which each pixel has the same initial pixel value is defined on the ξρ coordinate system, and the i-th pixel Pi constituting the image data is obtained on the two-dimensional pixel array for integration. The line graph μi is overlaid, and the pixel value of the pixel Pi is added to the pixel value of the pixel located on or near the line graph μi from the first pixel to the Nth pixel ( However, 1 ≦ i ≦ N, where N is the total number of pixels on the pixel array included in the image data)
A reference frame for predicting the position of the image of the reference frame that will appear on the photographed image arranged on the xy coordinate system based on the geometric photographing condition at the time of obtaining the photographed image by geometric calculation. A position prediction unit;
For each combination of variables (ξ, ρ) indicating each straight line constituting the image of the predicted reference frame, the corresponding coordinate points on the integrating two-dimensional pixel array are plotted as geometric prediction points ε, respectively. An actual prediction point determination unit that determines the reference position of the pixel having the maximum or minimum pixel value among the pixels belonging to the predetermined neighboring region with respect to the geometric prediction point ε as the actual prediction point for the corresponding straight line. When,
On the two-dimensional xy coordinate system, draw a straight line represented by the coordinate values (ξ, ρ) of each measured predicted point, treat the position of these straight lines as the position of the correct reference frame image, and perform geometric calculation. A regular conversion unit that converts a captured image into a regular image;
Is provided.

(16) According to a sixteenth aspect of the present invention, in the image conversion device to a regular image according to the fifteenth aspect described above,
When the line graph calculation unit draws a perpendicular line from the origin O of the xy coordinate system to an arbitrary straight line on the xy coordinate system and sets the foot as the point F, the axis of the xy coordinate system and the line segment OF are formed. The angle is represented by ξ, the length of the line segment OF is represented by ρ, and an expression form representing a straight line by two variables ξ and ρ is used.

(17) According to a seventeenth aspect of the present invention, in the image conversion device to a regular image according to the fifteenth or sixteenth aspect described above,
The pixel value accumulating unit accumulates the pixel value of the pixel Pi on the pixel value of the pixel located on the line graph μ, and the pixel Pi on the pixel value of the pixel separated from the line graph μ by a predetermined neighborhood distance d. Are multiplied by a predetermined ratio R (a value that takes a range of 0 ≦ R ≦ 1 and decreases as the neighborhood distance d increases) corresponding to the neighborhood distance d.

(18) According to an eighteenth aspect of the present invention, in the image conversion device to a regular image according to the seventeenth aspect described above,
The predetermined ratio R is set so as to have a Gaussian distribution centered on the position of the line graph μ.

(19) According to a nineteenth aspect of the present invention, in the image conversion apparatus for regular images according to the fifteenth to eighteenth aspects described above,
The line graph calculation unit performs a calculation for obtaining a line graph only for pixels having pixel values within a predetermined range among all N pixels,
The pixel value integration unit is configured to execute pixel value integration processing only for pixels having pixel values within a predetermined range among all N pixels.

(20) According to a twentieth aspect of the present invention, in the image conversion apparatus for regular images according to the fifteenth to nineteenth aspects described above,
The line graph calculation unit performs a calculation for obtaining a line graph only for pixels located within a predetermined consideration area among all N pixels,
The pixel value integration unit is configured to execute the pixel value integration processing only for pixels located in a predetermined consideration area among all N pixels.

(21) According to a twenty-first aspect of the present invention, in the image conversion apparatus for regular images according to the fifteenth to twentieth aspects described above,
The reference frame position prediction unit gives the angle θ between the normal line set at the reference point and the optical axis of the photographing apparatus, which is given as a geometric photographing condition when obtaining a photographed image, and the “reference frame of the optical axis” The angle φ formed by the projected image on the plane on which the image is drawn and the predetermined reference line ζ included in the plane on which the reference frame is drawn through the reference point and the focal length of the optical system of the photographing apparatus The position of the image of the reference frame is predicted by a geometrical operation based on it.

(22) According to a twenty-second aspect of the present invention, in the image conversion apparatus for regular images according to the fifteenth to twenty-first aspects described above,
When the predicted point determination unit in actual measurement uses the coordinate values of the geometric prediction point ε as (ξε, ρε) as the predetermined neighborhood region for the geometric prediction point ε, the predetermined deviations Δξ and Δρ are used. Thus, an area indicated by coordinate values (ξ, ρ) satisfying the conditions of ξε−Δξ ≦ ξ ≦ ξε + Δξ and ρε−Δρ ≦ ρ ≦ ρε + Δρ is used.

(23) According to a twenty-third aspect of the present invention, in the image conversion apparatus for regular images according to the fifteenth to twenty-second aspects described above,
In a case where the actually measured prediction point determination unit includes a plurality of pixels having the maximum or minimum pixel value among the pixels belonging to the predetermined neighboring region with respect to the geometric prediction point ε, the reference position of the plurality of pixels Is determined as an actually predicted point for the corresponding straight line.

(24) According to a twenty-fourth aspect of the present invention, in the image conversion apparatus for regular images according to the fifteenth to twenty-third aspects described above,
The actually measured prediction point determination unit determines the actually measured prediction point when the maximum pixel value in the vicinity region is less than the predetermined threshold value or when the minimum pixel value exceeds the predetermined threshold value. A message indicating that the image conversion to the regular image has failed is presented.

(25) A twenty-fifth aspect of the present invention uses the image conversion apparatus for regular images according to the fifteenth to twenty-fourth aspects described above,
When illumination light is applied from a predetermined incident direction to a predetermined position on the surface to be measured of the object, the reflectance of the reflected light from the predetermined position toward the predetermined emission direction is set for each individual position on the surface to be measured. Constructs an object reflection characteristic data creation device that creates, as reflection characteristic data for a surface to be measured, data defined for (m × n) combinations of m incident directions and a plurality of n exit directions. In addition, to
A shooting stage in which the upper surface forms a plane and a reference frame composed of straight lines is drawn on the plane;
A light source for illuminating an object placed on the shooting stage;
A camera for shooting an object placed on the shooting stage;
A shooting condition setting unit that sets shooting conditions by changing the positional relationship between the light source and the measured surface of the object in a plurality of m ways and changing the positional relationship between the measured surface and the camera in a plurality of n ways;
And configured so that the captured image captured by the camera is provided to the image input unit of the image conversion device,
A reflection characteristic data creation unit for creating reflection characteristic data of an object is further provided based on the regular image converted by the image conversion apparatus.

(26) According to a twenty-sixth aspect of the present invention, in the apparatus for creating reflection characteristic data of an object according to the twenty-fifth aspect described above,
A rectangle for placing an object is formed on the upper surface of the shooting stage, and the rectangular inner area and the outer area are configured to have a difference in contrast, and the reference frame is configured by the boundary line between the two areas. And
A differential processing unit that performs differential processing on the captured image captured by the camera is further provided, and the image after the differential processing is configured to be provided to the image input unit,
The regular conversion unit converts the image into a regular image by a geometric operation based on the positions of the four vertices of the rectangle constituting the image of the reference frame.

(27) According to a twenty-seventh aspect of the present invention, in the device for creating reflection characteristic data of an object according to the twenty-fifth or twenty-sixth aspect described above,
When the predicted point determination unit in actual measurement determines that the maximum pixel value in the neighborhood region is less than the predetermined threshold value or the minimum pixel value exceeds the predetermined threshold value, Outputs an error signal indicating that image conversion has failed,
The reflection characteristic data creation unit obtains a regular image that cannot be obtained from the image conversion device due to the output of an error signal by interpolation based on a regular image of another object photographed image that approximates the photographing condition. It is.

  (28) According to a twenty-eighth aspect of the present invention, a dedicated program is prepared for causing a computer to execute the image conversion method to a regular image according to the first to twelfth aspects.

  By using the image conversion method according to the present invention, it is possible to accurately convert a photographed image obtained by photographing a real object from an oblique direction into a regular image viewed from directly above. Thereby, accurate reflection characteristic data of an object such as high-dimensional texture data for rendering can be created.

  Hereinafter, the present invention will be described based on the illustrated embodiments.

<<< §1. Basic concept of rendering processing >>
The present invention is an invention related to an image conversion technique for converting a captured image obtained by shooting a subject from an oblique direction into a regular image in a state viewed from directly above. An embodiment used in the process of creating high-dimensional texture data (reflection characteristic data) for rendering will be described. Therefore, for convenience of explanation, first, the basic concept of conventional general rendering processing will be described.

  FIG. 1 is a perspective view showing the principle of general rendering processing. This example shows a shading model for obtaining the two-dimensional projection image 15 on the projection plane H when the three-dimensional virtual object 10 is observed from the viewpoint E. Here, the calculation for obtaining the projection image 15 is performed in a state where the texture data 20 prepared as a two-dimensional image is mapped to the surface of the virtual object 10.

  In order to perform such rendering processing, first, object data indicating the virtual object 10 is prepared. For example, in the illustrated example, the virtual object 10 is expressed as a set of a large number of polygons (polygons) defined on an XYZ three-dimensional coordinate system, and each part of the surface of the virtual object 10 is configured by a polygon. Has been. Therefore, the object data indicating the virtual object 10 is three-dimensional shape data including a collection of polygons, and the entity of the virtual object 10 is object data.

  On the other hand, the texture data 20 is two-dimensional image data defined on the uv two-dimensional coordinate system, and is an aggregate of a large number of pixels arranged on a two-dimensional plane. In the example shown here, a heart-shaped pattern is expressed. Of course, the pattern expressed by the texture data 20 is not necessarily an artificially drawn pattern, but may be an inherent shadow pattern of a natural material. For example, in the texture data expressing a plain carpet, an artificial pattern is not drawn, but a shadow pattern due to the fur of the carpet is expressed. After all, the texture data 20 can be referred to as reflection characteristic data indicating a distribution of reflection characteristics on a two-dimensional plane. The present invention is a technique that can be used in a process of creating such reflection characteristic data from an image of a real object such as a carpet. When performing the rendering process, the texture data 20 defined on the uv coordinate system is mapped to the surface of the virtual object 10 defined on the XYZ coordinate system, and the reflection characteristics of each part of the virtual object surface are determined. It will be.

  In the rendering process, it is necessary to set the illumination condition, the viewpoint position, and the projection plane. As illumination conditions, the type (for example, shape, size, wavelength characteristics, etc.) and position of the light source are set. In the case of the illustrated example, the light source G, the viewpoint E, and the projection plane H are set at the illustrated positions. Also, it is necessary to set a mapping mode (for example, relative position and orientation) indicating how to map the two-dimensional texture data 20 on the surface of the three-dimensional virtual object 10. On the other hand, an αβ two-dimensional coordinate system is defined on the projection plane H, and a projection image 15 (a two-dimensional image obtained as an object of rendering processing) is obtained as image data on the αβ coordinate system. become.

  Now, as shown in the figure, a sample point Q is defined on the virtual object 10, and the reflected light V from the sample point Q toward the viewpoint E is considered. The reflected light V is light obtained by reflecting the illumination light L from the light source G at the sample point Q, and is light that is observed at the position of the viewpoint E. However, “reflection” here is a broad concept that includes not only “specular reflection” but also “diffuse reflection”, and illumination light incident on the sample point Q at various angles is reflected as viewpoint E. Will head to. Accordingly, the reflected light V shown in the figure is a collection of reflected light obtained from illumination light incident from various angles.

  On the surface of the virtual object 10, a pixel P having a pixel value corresponding to the intensity of the reflected light V is defined at the intersection of the reflected light V and the projection plane H from one sample point Q toward the viewpoint E. Similarly, if pixels are defined on the projection plane H with respect to each reflected light directed from the large number of sample points to the viewpoint E, the set of the large number of pixels defined for the large number of sample points results in the three-dimensional virtual object 10. A two-dimensional projection image 15 is obtained on the projection plane H. In practice, a pixel array is defined in advance on the projection plane H with a desired resolution, and a line connecting the viewpoint E and the reference position (for example, the center point) of each pixel P is extended onto the virtual object 10. If the sample point Q is defined at the position of the intersection with the virtual object surface and the pixel value of each pixel P is calculated by the above-described method, two-dimensional data having the necessary resolution can be obtained.

  As described above, in order to obtain the pixel value of the pixel P on the projection plane H, it is necessary to obtain the intensity of the reflected light V from the sample point Q through the pixel P to the viewpoint E. The intensity of V can be basically obtained by calculation in consideration of the intensity of the illumination light L incident on the sample point Q and the light reflectance (specular reflectance or diffuse reflectance) at the sample point Q. Of course, at this time, calculation taking into account the orientation of the polygon including the sample point Q is performed, and the texture data 20 (two-dimensional image data defined on the uv coordinate system) mapped to the sample point Q is also taken into consideration. The entered operation is performed. Therefore, if the surface of the virtual object 10 is a curved surface, a projection image 15 having a shadow corresponding to the curved surface is obtained, and the texture data 20 includes information on a heart-shaped pattern as shown in the figure. For example, this heart-shaped picture is also expressed on the projected image 15 (the picture is not shown in FIG. 1).

The simplest method for calculating the intensity of the reflected light from the sample point Q is the intensity I (V) of the reflected light V from the sample point Q defined as one point on the XYZ three-dimensional coordinate system toward the viewpoint E,
I (V) = K · I (L)
It is a method to obtain by the following formula. Here, I (L) is the intensity of the illumination light L at the position of the sample point Q, and K is the reflection coefficient. The intensity I (L) of the illumination light L is determined by parameters such as the brightness of the light source G, the distance between the light source G and the sample point Q, and the incident angle of the illumination light with respect to the polygon to which the sample point Q belongs. On the other hand, the reflection coefficient K is determined by the texture data 20. First, when the texture data 20 is mapped on the surface of the virtual object 10 in a predetermined mapping manner, it is determined which coordinate of the texture data 20 the position of the sample point Q corresponds to, and the corresponding coordinates (u, v) A pixel T (u, v) located at is determined. The value defined as the pixel value of the corresponding pixel T (u, v) becomes the reflection coefficient K at the position of the sample point Q.

  When a color image is used as the texture data 20, a predetermined reflection coefficient is defined for each primary color. That is, three coefficients Kr, Kg, and Kb are defined as pixel values of one pixel T (u, v) defined on the uv coordinate system. Here, the coefficient Kr is the reflectance for the illumination light having the wavelength component of the primary color R (red), the coefficient Kg is the reflectance for the illumination light having the wavelength component of the primary color G (green), and the coefficient Kb is the primary color B. It is a reflectance with respect to illumination light having a wavelength component of (blue).

When mapping the texture data 20 composed of such a color image, the intensity of the reflected light from the sample point Q is calculated for each wavelength component of the light. For example, the intensity Ir (V) of the red wavelength component of the reflected light V is calculated using the intensity Ir (L) at the position of the sample point Q of the red wavelength component of the illumination light L and the reflection coefficient Kr.
Ir (V) = Kr · Ir (L)
It can be calculated by the following formula.

<<< §2. Model using high-dimensional image as texture >>
In §1, an example using a two-dimensional image as the texture data 20 is shown. In such a model for mapping a two-dimensional image, the reflection characteristic of the sample point Q is handled as not depending on the incident angle of illumination light or the reflection angle of reflected light, but a three-dimensional structure such as a pile ground is used. Strictly speaking, when the fiber sheet is stuck on the object surface, it must be handled in consideration of the anisotropic reflection characteristics depending on the incident angle of the illumination light and the reflection angle of the reflected light.

  As described above, a model called BRDF (Bi-directional Reflection Distribution Function) has been proposed as a basic model for handling with different reflection characteristics depending on the angle of incidence and reflection direction of light. . FIG. 2 is a perspective view for explaining the principle of the BRDF model. As shown in the figure, a normal line N is set at the position of the sample point Q on the virtual object surface, and an orthogonal plane W (hereinafter referred to as a reference plane W) including the sample point Q and orthogonal to the normal line N is defined. An angle between the normal N and the incident illumination light L is θL, a projection image L ′ of the incident illumination light L on the reference plane W, and a predetermined reference line ζ passing through the sample point Q on the reference plane W Is defined as a combination of an angle θL and an angle φL. Similarly, the angle between the normal N and the reflected light V is θV, the angle between the projected image V ′ of the reflected light V on the reference plane W and the reference line ζ is φV, and the direction of the reflected light V is the angle. It is defined by a combination of θV and angle φV. In other words, the angles θL and θV correspond to the elevation angles of the incident illumination light and the reflected light (the original “elevation angle” means an angle with respect to the horizontal plane, so strictly speaking, the angles θL and θV are “90 ° −elevation angle”. However, here, for the sake of convenience, this is simply referred to as “elevation angle”). The angles φL and φV correspond to the azimuth angles of incident illumination light and reflected light.

Here, when the intensity of the incident illumination light L at the sample point Q is I (L), the intensity I (V) of the reflected light V from the sample point Q is given as a function of θL, φL, θV, φV. Using the reflection coefficient K (θL, φL, θV, φV),
I (V) = K (θL, φL, θV, φV) · I (L)
The BRDF model is obtained by the following formula. The BRDF model is characterized in that the reflection coefficient K is given as a function of parameters “θL, φL” indicating the direction of the incident illumination light L and parameters “θV, φV” indicating the direction of the reflected light V.

A BTF (Bi-directional Texture Function) model described in the above-mentioned Non-Patent Document 1 or the like is obtained by adding the concept of texture mapping to the BRDF model. In the BTF model, coordinate values (u, v) on the two-dimensional uv coordinate system are further added as parameters to the four angle parameters used in the BRDF model, and the incident illumination light L at the sample point Q When the intensity is I (L), the intensity I (V) of the reflected light V from the sample point Q is
I (V) = K (θL, φL, θV, φV, u, v) · I (L)
It is given by That is, the intensity I (V) of the reflected light V shown in FIG. 2 relates to the angle from which direction the incident illumination light L is applied to the sample point Q and from which direction the reflected light V is emitted from the sample point Q. This is a factor that is determined depending on the parameter and also determined depending on the position of the sample point Q.

  FIG. 3 is a plan view showing the structure of the 6-dimensional texture data 20 used in this BTF model. As described above, in this BTF model, the reflection coefficient K is a parameter defined as a six-dimensional function K (θL, φL, θV, φV, u, v). This means 6-dimensional image data composed of a set of a large number of pixels arranged in a 6-dimensional space, and each pixel has a pixel value corresponding to a specific reflection coefficient K. However, for convenience of explanation, the 6-dimensional texture data 20 is illustrated as a two-dimensional pixel array on the uv plane as in the example shown in FIG. In this case, each pixel T (u, v) on the two-dimensional pixel array has a reflection coefficient K (θL, φL, θV, φV) determined as a function of four parameters “θL, φL, θV, φV”. Is defined respectively.

  Here, the reflection coefficient K (θL, φL, θV, φV) can be defined as some function expression including θL, φL, θV, and φV as variables, but in practice, it is defined as a numerical table. The embodiment described here is also premised on the handling defined as a numerical table. Thus, when the reflection coefficient K is defined as a numerical table, the angle parameters θL, φL, θV, and φV cannot be handled as continuous quantities, and thus must be handled in a discrete manner.

  FIG. 4 shows an example of a discrete definition of the angle θL. As described above, the angle θL indicates an angle with respect to the normal line N of the illumination light L incident on the sample point Q. In FIG. 4, illumination lights L <b> 1 to L <b> 6 that are incident on the sample point Q at six elevation angles are depicted. The incident angles θL1 to θL6 of the illumination lights L1 to L6 are 0 °, 15 °, 30 °, 45 °, 60 °, and 75 ° as illustrated. With respect to the incident angle (elevation angle), if six angles are discretely defined at intervals of 15 ° as described above, texture expression with a certain degree of accuracy can be achieved in practice.

  On the other hand, FIG. 5 shows an example of a discrete definition of the angle φL. As described above, the angle φL is an angle formed between the projection image L ′ of the illumination light L incident on the sample point Q on the reference plane W and the reference line ζ, and indicates the azimuth angle of the illumination light L. . In FIG. 5, the azimuth angle of illumination light incident from 12 directions with respect to the sample point Q is drawn (the reference line ζ is defined at a position of 0 °). 5 corresponds to the reference plane W, and azimuth angles in the range of 0 ° to 360 ° are defined on the reference plane W every 30 °. With regard to the azimuth angle, if 12 angles are defined discretely at intervals of 30 ° in this way, texture expression with a certain degree of accuracy can be achieved in practice.

  As described above, the discrete definition examples of the angles θL and φL related to the incident illumination light L have been shown, but the same definitions can be made for the angles θV and φV related to the reflected light V. That is, regarding the angle θV, six discrete definitions of 0 °, 15 °, 30 °, 45 °, 60 °, and 75 ° are made, and the angle φV is an azimuth angle in the range of 0 ° to 360 °. May be defined every 30 °.

  If such a discrete definition is performed, the angles θL and θV each have six values, and the angles φL and φV each have twelve values. Therefore, the combination of these four angles is 6 That is, × 12 × 6 × 12 = 5184 ways. Therefore, the reflection coefficient K defined for one pixel T (u, v) of the texture data 20 shown in FIG. 3 takes 5184 values depending on the combination of the respective angle values.

  After all, the texture data 20 shown in FIG. 3 is formally expressed as two-dimensional image data defined on the two-dimensional uv coordinate system, but actually, the six parameters θL, φL, θV are represented. , ΦV, u, v, it is 6-dimensional image data (reflection characteristic data) having a pixel value (reflectance) defined by.

  The rendering process using the BTF model using the high-dimensional texture data 20 as shown in FIG. 3 is performed as follows. First, as shown in FIG. 1, the texture data 20 is mapped on the surface of the three-dimensional virtual object 10 to determine a pixel T (u, v) corresponding to the position of the sample point Q. Next, by taking into account the position of the light source G and the orientation of the polygon to which the sample point Q belongs, the angles θL and φL of the illumination light L incident on the sample point Q, and the angle of the reflected light V emitted from the sample point Q θV and φV are determined. Then, a reflection coefficient corresponding to a combination of specific angles θL, φL, θV, and φV is obtained with reference to the reflection characteristics defined for the pixel T (u, v) shown in FIG. In the case of a color image, reflection coefficients Kr, Kg, and Kb are obtained for each of the three primary colors RGB. Finally, the intensity value for each primary color component of the reflected light V is obtained based on these reflection coefficients Kr, Kg, and Kb, and the pixel value of the pixel P on the projection plane H may be determined.

<<< §3. Method of creating reflection characteristic data by actual measurement >>>
Next, a method for creating the reflection characteristic data expressed as a high-dimensional image as described in §2 by actually measuring an actual object will be described. For example, in the rendering process shown in FIG. 1, when it is desired to represent a state in which a fiber sheet having a three-dimensional structure such as a pile ground is attached to the surface of the object data 10 using a BTF model, the texture data 20 to be mapped to the surface is used as the texture data 20. Therefore, it is necessary to prepare reflection characteristic data expressed as a six-dimensional image as shown in FIG. In order to create such high-dimensional reflection characteristics data based on measurements based on real objects, for example, a real pile fabric is prepared, and the reflection characteristics of the surface of the fabric are measured with various geometrical images. The reflection characteristic data may be created based on a plurality of photographed images obtained by photographing under the conditions.

  As specific shooting conditions, a positional relationship between the three sources of the light source, the object, and the camera may be set. That is, one shooting condition is a condition indicating that the three parties have a predetermined positional relationship. More specifically, one imaging condition is defined by a combination of the positional relationship between the light source and the object and the positional relationship between the camera and the object.

  FIG. 6 is a perspective view showing an example of an imaging system for performing imaging under various imaging conditions. In this example, a state in which a sheet-like object 40 (for example, a piece of cloth such as a pile fabric) is placed on the upper surface (plane) of the imaging stage 30 is shown. In order to photograph the upper surface W (surface to be measured) of the object 40, a light source 50 and a camera 60 are arranged as shown in the figure. Here, a point light source is used as the light source 50. In addition, in order to geometrically define the positional relationship between the light source 50, the object 40, and the camera 60, a representative point 51 is defined in the light source 50, and a reference point S is set at the center position of the upper surface W of the object 40. Defined. A point light source is a light source that theoretically generates light as a spherical wave from a geometric point. However, since it is a light emitter having a finite volume, a representative point 51 is defined therein. I will handle it. Both the representative point 51 and the reference point S are geometrical points for indicating a reference for position definition, and can be theoretically defined at arbitrary positions, but in practice, they are defined at respective center positions. It is preferable to do this. Therefore, in the illustrated example, the representative point 51 is defined at the center position of the light source 50, and the reference point S is defined at the center position of the upper surface W of the object 40.

  Here, the camera 60 is arranged in such an orientation that the reference point S is positioned on the optical axis. In addition, illumination light traveling from the representative point 51 toward the reference point S is referred to as reference illumination light Ls, and reflected light emitted from the reference point S in the optical axis direction of the camera 60 is referred to as reference reflected light Vs. Further, at the position of the reference point S, a normal line N to the upper surface W of the object 40 is set, this upper surface W is called a reference surface W, and a projection image of the reference illumination light Ls on this reference surface W is called Ls ′. The projected image of the reference reflected light Vs on the reference surface W will be referred to as Vs ′.

  With such a definition, the positional relationship between the light source 50 and the object 40 is such that the angle θLs (incidence angle of the reference illumination light Ls) formed by the reference illumination light Ls and the normal N incident on the reference point S, and the reference It can be defined by an angle φLs (an azimuth angle of the reference illumination light Ls) formed between the projection image Ls ′ of the illumination light Ls on the reference plane W and the reference line ζs. Similarly, the positional relationship between the object 40 and the camera 60 is that an angle θVs (a reflection angle of the reference reflected light Vs) formed by the reference reflected light Vs emitted from the reference point S in the optical axis direction of the camera 60 and the normal line N, and It can be defined by the angle φVs (azimuth angle of the reference reflected light Vs) formed by the projected image Vs ′ of the reference reflected light Vs on the reference surface W and the reference line ζs.

  This definition is common to the definitions of the illumination light L and the reflected light V for the sample point Q in the BTF model shown in FIG. However, the example shown in FIG. 2 is for showing the relationship between the illumination light L and the reflected light V at the position of the sample point Q on the three-dimensional virtual object 10 when performing the rendering process as shown in FIG. On the other hand, the example shown in FIG. 6 uses the reference incident light Ls incident on the reference point S defined on the real object 40 and the reference reflected light Vs reflected therefrom as a reference, and the light source 50 , The object 40 and the camera 60 to indicate the positional relationship of the three parties.

  In the case of the example shown in FIG. 6, the positional relationship between the light source 50 and the object 40 can be changed to m = (a × b) by changing the angle θLs to a and the angle φLs to b. it can. Similarly, by changing the angle θVs to c and the angle φVs to d, the positional relationship between the object 40 and the camera 60 can be changed to a plurality of n = (c × d) ways. After all, as the positional relationship between the light source 50, the object 40, and the camera 60, the positional relationship between the light source 50 and the object 40 is changed in a plurality of m ways, and the positional relationship between the object 40 and the camera 60 is changed to a plurality of n. By changing the way, it is possible to perform a total (m × n) setting. If each setting is regarded as a unique shooting condition, the total (a × b × c × d) shooting conditions can be set. Can be set.

In the embodiment shown here, the angle range of 0 to 90 ° is equally divided by α, and a = c = α, so that the θLs angle value and the θVs angle value are respectively set to α and 0 to 360 °. The angle range is divided equally into β and b = d = β, so that the angle value of φLs and the angle value of φVs are respectively set in β ways, and a total of α ² × β ² shooting conditions are set. ing.

  Specifically, for example, when α = 6 (a = 6, c = 6), the incident angle θLs and the reflection angle θVs are 0 °, 15 °, 30 as in the example shown in FIG. Six settings of °, 45 °, 60 °, and 75 ° can be performed. When β = 12 (b = 12, d = 12), the azimuth angle φLs and the azimuth angle φVs are as shown in FIG. Similarly to the example shown in FIG. 12, twelve settings of 0 °, 30 °, 60 °,..., 300 °, 330 ° can be performed. In this case, the positional relationship between the light source 50 and the object 40 is 72 (m = 6 × 12), and the positional relationship between the object 40 and the camera 60 is 72 (n = 6 × 12), for a total of 5184. The shooting conditions are set.

  Finally, in the case of the example shown in FIG. 6, the shooting conditions are such that the light source 50, the object 40, and the camera 60 have a predetermined positional relationship, the positional relationship between the light source 50 and the object 40 is changed in a plurality of m ways, By changing the positional relationship between the object 40 and the camera 60 in a plurality of n ways, a total (m × n) ways can be set.

  If the image of the object 40 illuminated by the light source 50 is captured by the camera 60 for each of the (m × n) imaging conditions set in this way, a total of (m × n) captured images can be obtained. it can. In the case of the above-described example, a total of 5184 shot images are obtained based on a total of 5184 shooting conditions. In practice, by using a camera having a function of capturing a color image as the camera 60, a color image composed of a set of primary color planes of the three primary colors RGB is captured.

  As described above, when photographing is performed under various geometric conditions, the shape and size of the object image on the photographed image become various. For example, if the upper surface (surface to be measured) of the object 40 is a rectangular plane, this surface to be measured is viewed from the direction of the normal N that stands at the position of the reference point S (the center point of the rectangle) (FIG. 6). In the case where the image is taken at a specific rotational position (in FIG. 6, at a rotational position where the angle φVs = 0 ° or 180 °) from the direction where the angle θVs = 0, as shown in FIG. In addition, a captured image of a regular rectangle (a rectangle having four sides parallel to each side of the rectangle constituting the contour of the image) is obtained. However, under the shooting conditions in which the angle φVs is changed, as shown in FIG. 7B, a rotated rectangular image is obtained. Further, under the shooting conditions in which the angles θVs and φVs take arbitrary values. As shown in FIGS. 7C to 7F, images having various shapes and sizes can be obtained.

  However, the texture data 20 (reflection characteristic data) to be created here is composed of an aggregate of pixels T (u, v) arranged in a regular matrix on the uv coordinate system as shown in FIG. Must have been. Therefore, by using a geometric image conversion method called projective normalization conversion disclosed in the above-mentioned Patent Document 2 and the like, each of the individual images shown in FIGS. Are converted into regular images shown in FIGS. 8A to 8F, respectively. Such conversion processing into a regular image can be performed based on the angles θVs and φVs and the focal length of the camera 60.

  The regular images shown in FIGS. 8A to 8F geometrically correspond to images obtained by viewing the object 40 shown in FIG. However, the regular images shown in FIGS. 8A to 8F are different from each other. For example, the regular image shown in FIG. 8 (c) geometrically corresponds to an image of the object 40 viewed from the direction of the normal N, and the number of vertical and horizontal pixels is the regular image shown in FIG. 8 (a). Is exactly the same. However, the regular image shown in FIG. 8 (c) is not an image obtained by photographing the object 40 from the direction of the normal N, but an image obtained by photographing from an oblique direction. The individual pixel values are different from the individual pixel values of the regular image shown in FIG. Thus, in this application, “regular image viewed from the normal direction” means “an image having the same pixel arrangement as the image viewed from the normal direction in a geometric sense”, It does not mean an image obtained by photographing from (for example, FIG. 8 (a)).

  Now, when (m × n) regular images are finally obtained by performing such image conversion processing on each captured image, reflection characteristic data is created based on these regular images. be able to. The finally created reflection characteristic data is obtained by assigning four parameters “θL, φL, etc. to each pixel T (u, v) on the two-dimensional pixel array as in the texture data 20 shown in FIG. The reflection coefficients K (θL, φL, θV, φV) defined as functions of “θV, φV” are respectively defined. On the other hand, the data obtained by the image conversion process is data of 5184 regular images in the above example. Therefore, reflection characteristic data as shown in FIG. 3 may be created based on the data of the 5184 regular images.

  However, this process should not be a process involving substantial operations, but simply a process of organizing data. For example, the reflection coefficient K (θL, φL, θV, φV) that is defined as a function of the angles “θL, φL, θV, φV” is defined for the pixel T (u, v) of the texture data 20 shown in FIG. Yes. Therefore, the angles “θLs, φLs, θVs, φVs” set as the imaging conditions are regarded as the angles “θL, φL, θV, φV”, respectively, and the specific imaging conditions of the angles “θLs, φLs, θVs, φVs” are used. The pixel value of the pixel at the position of the coordinate (u, v) on the regular image is obtained with respect to the angle “θL, φL, θV, φV” for the pixel T (u, v) of the texture data 20 shown in FIG. A process defined as a reflection coefficient K (θL, φL, θV, φV) may be performed.

  More specifically, it is as follows. For example, it is assumed that a specific regular image is obtained based on a photographed image photographed under specific photographing conditions of “θLs = 45 °, φLs = 120 °, θVs = 15 °, φVs = 300 °”. It is assumed that the pixel value of the pixel at the position of the coordinate (u, v) on this specific regular image is P (u, v). In this case, the reflection coefficient K (45 °) for the angles “θL = 45 °, φL = 120 °, θV = 15 °, φV = 300 °” for the pixel T (u, v) of the texture data 20 shown in FIG. , 120 °, 15 °, 300 °) is defined as P (u, v). Of course, a total of 5184 reflection coefficients K are defined for this pixel T (u, v) in accordance with variations in the combination of individual angles, but these reflection coefficients K were obtained. This is the pixel value of the pixel at the position of coordinates (u, v) on 5184 regular images.

  The reflection characteristic data obtained in this way, when the illumination light is applied to the predetermined position of the object 40 from the predetermined incident direction, the reflectance of the reflected light directed from the predetermined position toward the predetermined emission direction is determined as the individual position. In other words, the reflection characteristic data is defined for (m × n) cases each of which is a combination of a plurality of m incident directions and a plurality of n exit directions.

  Note that when a color image is taken, a regular image composed of a set of primary color planes of the three primary colors RGB may be obtained by executing a conversion process to a regular image for each primary color plane.

<<< §4. Basic Principle of Conversion in the Present Invention >>
As described above, as a technique for converting the individual captured images shown in FIGS. 7 (a) to (f) into regular images shown in FIGS. 8 (a) to (f), respectively, A geometric calculation method (projective normalization conversion) based on geometric imaging conditions (angles θVs, φVs, focal length of the camera 60) has been used. However, even if such a geometric calculation method is applied to measured image data obtained by camera shooting of an actual object, conversion with high accuracy cannot be performed. The reason for this is that, as described above, even if a predetermined geometric imaging condition is set in advance and an actual object is imaged under the imaging condition, in practice, an accurate imaging as set This is because it is difficult to perform shooting under conditions. That is, in the photographing system shown in FIG. 6, factors such as an error related to the relative position and orientation of the camera 60 with respect to the object 40, a deviation of the optical axis of the camera 60, and lens aberration are involved. The actual captured image shown in (f) does not match the ideal captured image obtained under geometric imaging conditions (angles θVs, φVs, focal length of the camera 60).

  Therefore, in the present invention, basically another approach is taken to realize a more accurate conversion to a regular image. The basic principle of the conversion in the present invention is a process of converting the photographed image into a regular image using a clue left on the actually obtained photographed image instead of using the geometric photographing condition as a clue. There is to do.

  For example, suppose that in the imaging system shown in FIG. 6, the object 40 placed on the imaging stage 30 is a planar object having a rectangular outline. In this case, on the photographed image obtained by photographing the object 40 from an oblique direction, for example, as shown in FIG. It should be. On the other hand, when the object 40 is photographed from the direction of the normal N (directly above), it is known in advance that a regular square ABCD as shown in FIG. To do. Then, to convert the square ABCD shown in FIG. 9 (a) into the square ABCD shown in FIG. 9 (b), a known geometric calculation method (for example, projective normalization conversion described above) is used. Using this calculation method, the corresponding point Q in the regular quadrilateral ABCD shown in FIG. 9 (b) can be obtained for any point Q in the quadrilateral ABCD shown in FIG. 9 (a). .

  In this way, if there is a rectangle that functions as a reference frame on the object to be imaged, a conventional geometric calculation method is applied using the shape of the image of the reference frame in the captured image as a clue. Thus, the captured image can be converted into a regular image. Of course, the point that a geometric transformation method such as projective normalization transformation is used is the same as the conventional image transformation method described in §3. On the other hand, in the present invention, conversion is performed using the shape of the image of the reference frame shown in the actually obtained captured image as a clue. When a process for converting a specific captured image into a regular image is performed, an image of the reference frame that appears on the captured image is used, so that a very accurate conversion process can be performed. In other words, in the present invention, even when shooting is not performed under ideal shooting conditions as set during shooting, there is no influence.

  However, it is not preferable to use the object 40 itself to be measured as the reference frame on the object to be photographed. In the above-described example, the example in which the object 40 is a planar object having a rectangular outline is shown. However, in practice, the object 40 to be measured has various forms, and the rectangular outline is not necessarily used. It does not necessarily have. Therefore, in practice, it is preferable to form a rectangular reference frame on the upper surface of the imaging stage 30 on which the object 40 is placed.

  FIG. 10 is a plan view showing an example of the imaging stage 30 used for utilizing the image conversion according to the present invention. As illustrated, a rectangular reference frame 35 is drawn on the upper surface of the imaging stage 30. The reference frame 35 includes four sides 35A, 35B, 35C, and 35D, and an object 40 that is a subject is placed inside the reference frame 35 as illustrated. FIG. 11 is a plan view showing an example of a photographed image obtained by photographing using the photographing stage 30 shown in FIG. 10 (the contour image of the photographing stage 30 is not shown). This example is an example of photographing from obliquely above the photographing stage 30, and the image of the reference frame 35 constitutes a distorted square ABCD. Of course, the shape of the image of the object 40 is also distorted.

  The purpose of converting the captured image as shown in FIG. 11 into a regular image is to correct the distortion of the image of the object 40. As a clue to such correction, the image of the reference frame 35 is used. The distorted rectangular ABCD is used. That is, if the image of the reference frame 35 (regular rectangle) is confirmed in advance on the captured image obtained when the imaging stage 30 shown in FIG. FIG. 11 shows an image obtained by the same geometric technique (for example, projective normalization conversion) as that obtained by converting the distorted quadrangle ABCD shown in FIG. 9 (a) into the regular quadrangle ABCD shown in FIG. 9 (b). Images can be converted into regular images.

  This is the basic principle of image conversion processing into a regular image in the present invention. However, in order to actually perform image conversion processing based on this principle, there is an important problem that must be solved. That is recognition of the image of the reference frame 35 on the photographed image as shown in FIG. In FIG. 11, for convenience of explanation, the image of the reference frame 35 is shown as a simple quadrilateral ABCD. However, in order to perform image conversion processing to a regular image as an operation using a computer, each side constituting the quadrilateral ABCD 35A, 35B, 35C, and 35D need to be recognized as straight lines by the computer.

  When digital image processing using a computer is performed, a photographed image is handled as data composed of an array of a large number of pixels. Therefore, no matter how precise the reference frame 35 is drawn on the shooting stage 30 shown in FIG. 10, the image of the reference frame 35 is only a collection of pixels on the shot image as shown in FIG. Absent. Therefore, in order for the computer to recognize the geometric quadrangle ABCD, it is necessary to perform a straight line recognition process by extracting pixels constituting the image of the reference frame 35 based on some algorithm.

  Moreover, the process of recognizing the quadrilateral ABCD from the photographed image as shown in FIG. 11 is actually a process with difficulty. This is because the image of the reference frame 35 shown on the photographed image is not necessarily a continuous line. As can be seen from the imaging system shown in FIG. 6, an actual captured image is captured under conditions in which the relative positions of the object 40, the light source 50, and the camera 60 are variously changed. There is a possibility that the reference frame 35 drawn on the photographing stage 30 is unclearly reflected by the reflection of light. In fact, the image of the reference frame 35 on the photographed image often becomes a continuous line. In the case of the example described in §2, a total of 5184 images shot under various shooting conditions are obtained. For each of these shot images, the form of the image of the reference frame 35 and how the lines are broken Will be different.

  Eventually, in order to put the basic principle of the image conversion process into the regular image in the present invention into practical use, each side 35A, 35B, 35C, 35D constituting the image of the reference frame 35 on each photographed image, It is necessary to solve the specific problem of recognizing a straight line on a computer. The present invention has succeeded in solving this problem by combining a straight line recognition method called Hough transform and a conventional image conversion method based on geometric imaging conditions. Hereinafter, this method will be described in detail.

<<< §5. Basic principle of Hough transform >>>
Here, the basic principle of the Hough transform will be briefly described. The Hough transform is a method for extracting lines from incomplete image information. For example, “Image Processing <Image Processing Standard Text Book>” Supervision: Image Processing Standard Text Book Editorial Committee, published Place: Foundation for Image Information Education, Date of Issue: February 25, 1997, 187-190 pages, and so on.

  Now, it is assumed that a two-dimensional xy coordinate system as shown in FIG. 12 (a) is defined, and one straight line La as shown exists on this coordinate. In general, an arbitrary straight line on the xy coordinate system can be expressed using a variable ξ indicating the orientation with respect to the coordinate axis and a variable ρ indicating the displacement with respect to the origin. In the case of the illustrated example, the straight line La is represented using a variable ξa indicating the orientation with respect to the coordinate axis x and a variable ρa indicating the displacement with respect to the origin O. That is, in the example shown in FIG. 12A, when the perpendicular line is drawn from the origin O of the xy coordinate system to the straight line La and the foot is set to the point F, the one axis x of this xy coordinate system and the line segment OF. Is represented by ρa, the length of the line segment OF is represented by ρa, and an expression form representing the straight line La by two variables ξa and ρa is used.

  Thus, since the combination of the two variables (ξ, ρ) uniquely corresponds to one straight line on the xy coordinates, as shown in FIG. 12 (b), the two-dimensional ξρ coordinates. If a system is defined, an arbitrary point λ on the ξρ coordinate corresponds to a specific straight line on the xy coordinate. In the illustrated example, one point λa (ξa, ρa) on the ξρ coordinate shown in FIG. 12B corresponds to a straight line La on the xy coordinate shown in FIG.

  Accordingly, as shown in FIG. 13A, a point P is taken on the xy coordinates, and three straight lines La, Lb, and Lc passing through the point P are considered. Then, as shown in FIG. 13B, points λa, λb, and λc can be plotted on the ξρ coordinate corresponding to these three straight lines La, Lb, and Lc, respectively. Since the straight line passing through the point P on the xy coordinates is not limited to three but actually exists infinitely, if the corresponding points are plotted on the ξρ coordinate for the infinite number of straight lines, An infinite number of corresponding points are plotted, and a single line graph can be drawn by the infinite number of corresponding points. That is, when a point P is defined on the xy coordinates as shown in FIG. 14 (a) and an infinite number of straight lines passing through the point P is considered, the straight line group is as shown in FIG. 14 (b). , Ξρ is represented as a single line graph μ drawn on the coordinates.

  Next, consider the case where there are four points P1, P2, P3, and P4 on the xy coordinates as shown in FIG. In this case, an infinite number of straight lines passing through the point P1 can be expressed as one line graph μ1 on the ξρ coordinate, as shown in FIG. 15 (b). Similarly, an infinite number of straight lines passing through the point P2 are represented as a line graph μ2, an infinite number of straight lines passing through the point P3 are represented as a line graph μ3, and an infinite number of straight lines passing through the point P4 are represented as a line graph μ4. The Here, as shown in FIG. 15B, when the four line graphs μ1 to μ4 pass through the common intersection γ, the intersection γ corresponds to one of the straight lines passing through the point P1, and the point P2 Corresponds to one of the straight lines passing through the point P3, and corresponds to one of the straight lines passing through the point P4. Consequently, the straight line indicated by the broken line in FIG. L (a straight line passing through all four points P1 to P4).

  Thus, if the Hough transform method is used, a straight line L passing through these four points can be determined based on the four points P1 to P4 on the xy coordinates as shown in FIG. That is, for each of the four points P1 to P4, as shown in FIG. 15 (b), line graphs μ1 to μ4 indicating an infinite number of straight lines passing through each point are obtained on the ξρ coordinate, respectively, and ξ of the intersection γ is obtained. If the coordinate value and the ρ coordinate value are obtained, the straight line L shown in FIG. 15 (a) can be uniquely determined. Therefore, even when a sharp and blurred line is given on the xy coordinates, it can be restored to a complete straight line.

<<< §6. Basic procedure of a method for creating reflection characteristic data according to the present invention >>>
Next, the basic procedure of the method for creating reflection characteristic data according to the present invention will be described with reference to the flowchart of FIG. The outline of the method shown in FIG. 16 has already been described in §3. The purpose of this method is to apply illumination light to a predetermined position on a surface to be measured of an actual object from a predetermined incident direction. When the reflectance of reflected light traveling from a position toward a predetermined emission direction is (m × n) depending on a combination of a plurality of m incident directions and a plurality of n emission directions for each position of the surface to be measured Is to create the reflection characteristic data for the surface to be measured.

  First, in the object placement stage of step S1, an object having a surface to be measured is placed inside a reference frame constituted by straight lines drawn on a plane. Specifically, as shown in FIG. 10, if an imaging stage 30 on which a rectangular reference frame 35 is drawn is prepared, and an object 40 having a measurement surface is placed inside the reference frame 35. Good.

  In the subsequent shooting condition setting stage of step S2, in the shooting system as shown in FIG. 6, the shooting conditions such that the light source 50, the surface to be measured (object 40), and the camera 60 are in a predetermined positional relationship are The total (m × n) is obtained by changing the positional relationship between the surface to be measured (object 40) in a plurality of m ways and changing the positional relationship between the surface to be measured (object 40) and the camera 60 in a plurality of n ways. ) Work to set as follows. In the case of the example described in §3, regarding the angles θV and θL, as shown in FIG. 4, six discrete definitions of 0 °, 15 °, 30 °, 45 °, 60 °, and 75 ° are performed. Regarding φV and φL, as shown in FIG. 5, azimuth angles in the range of 0 ° to 360 ° are defined in total 12 ways every 30 °, and m = 72, n = 72, and 5184 in total. Set the shooting conditions.

  In the object photographing stage of the next step S3, the surface to be measured on the object 40 illuminated by the light source 50 for each of the set (m × n) photographing conditions using the photographing system shown in FIG. (M × n) object captured images are obtained by executing a shooting process of shooting the image of the above with the camera 60. The object photographed image thus obtained is an image photographed from various directions as shown in FIG. 7, for example.

  Then, in the image conversion stage of step S4, image conversion is performed on the obtained (m × n) object photographed images, and (m × n) regular images are obtained. For example, an object photographed image as shown in FIG. 7 is converted into a regular image as shown in FIG. The process of the image conversion stage in step S4 is a process that is an essential feature of the present invention, and will be described in detail separately in §7.

  Finally, in the reflection characteristic data creation stage of step S5, the reflection characteristic data of the object is created based on (m × n) regular images. This is a process of organizing the pixel values of (m × n) regular images and creating data in the form of texture data 20 as shown in FIG. 3 as described in §3. Note that steps S4 and S5 in each procedure shown in FIG. 16 are actually executed by a computer.

<<< §7. Basic procedure of image conversion method to regular image according to the present invention >>
Here, the basic procedure of the image conversion stage in step S4 in the flowchart of FIG. 16, that is, the basic procedure of the image conversion method to a regular image that is a feature of the present invention will be described in detail. FIG. 17 is a flowchart showing the detailed procedure of this image conversion stage, and is obtained by photographing a subject placed inside a reference frame composed of straight lines drawn on a plane from a predetermined direction. This is a processing procedure for converting an image into a regular image viewed from a normal direction set at a reference point on the plane. The processing shown in each step of FIG. 17 is practically executed by a computer.

  First, step S11 is processing in an image input stage, and an image captured in the object imaging stage in step S3 in FIG. 16 is input as image data in which pixels having predetermined pixel values are arranged. become. If a digital camera is used as the camera 60 in the photographing system of FIG. 6, the image data output from this digital camera may be directly taken into the computer.

  Here, for convenience, it is assumed that the captured image input in step S11 is composed of a pixel array defined on the two-dimensional xy coordinate system, and more specifically, a simple pixel array as shown in FIG. The following description will be given on the assumption that a captured image consisting of is input. The captured image shown in FIG. 18 is data in which a large number of pixels are arranged in a vertical and horizontal matrix on an xy coordinate system. In order to simplify the description here, each pixel has “0” or “1”. It is assumed that one of the pixel values of “is given. That is, the photographed image shown in FIG. 18 is a binary image, the hatched pixels in the figure are pixels having a pixel value “1” (hereinafter referred to as black pixels), and the other pixels are pixel values. A pixel having “0” (hereinafter referred to as a white pixel). Here, it is assumed that the photographed image shown in FIG. 18 is composed of a total of N pixels, and the pixels P1, P2, P3,. For black pixels, the outline of each pixel is shown. However, for white pixels, the figure is complicated, so the white pixels P1, P2, P3 in the upper left corner and the white pixel PN in the lower right corner are excluded. The outline of each white pixel is omitted (in FIG. 18, white pixels are all arranged in the white background portion that is the background of the black pixel).

  Here, as will be described later, in order to describe an example in which a straight line constituting the reference frame 35 is recognized from an image of the reference frame 35 drawn on the shooting stage 30, the captured image shown in FIG. The image shows only a part of the straight line portion of the photographed image of the reference frame 35. If the black pixels in FIG. 18 are connected, the existence of one straight line will be recognized. In order to recognize one straight line (straight line constituting the reference frame 35) based on such a discontinuous black pixel distribution, the principle of the Hough transform described in §5 is used. Of course, the actual captured image is a complex image including the images of the four sides 35A, 35B, 35C, and 35D constituting the reference frame 35 and the image of the object 40, as in the example shown in FIG. Since it has a pixel configuration, FIG. 18 is an example in which only a part of a captured image actually obtained is illustrated.

  Subsequently, in step S12, a two-dimensional pixel array for integration is defined. Here, the two-dimensional pixel array for integration is a pixel array used when pixel value integration processing is performed in step S15 described later, and is a pixel array defined on the ξρ coordinate system. Each pixel has an initial pixel value of 0. This step S12 is a process that should be called a preparation stage of the pixel value integration stage executed in step S15. Note that the size of the two-dimensional pixel array for integration defined here may be appropriately set in consideration of necessary accuracy for the variables ξ and ρ for determining a straight line.

  In step S13, the parameter i is set to the initial value 1, and the processes in steps S14 and S15 are repeatedly executed through step S16. At this time, the parameter i is incremented and updated by 1 in step S17 each time. Here, the parameter i is a parameter indicating a pixel on the photographed image shown in FIG. 18, and when i = 1, the first pixel P1, when i = 2, the second pixel P2,. When i = N, this indicates the Nth pixel PN.

  Step S14 is a line graph calculation stage process for calculating the line graph μ on the ξρ coordinate by using the Hough transform described in §5. As shown in FIG. 12, an arbitrary straight line La on the xy coordinate system can be expressed by using a variable ξa indicating the orientation with respect to the coordinate axis and a variable ρa indicating the displacement with respect to the origin O, and one point on the ξρ coordinate system. It can be expressed as λa. If such an expression format is adopted, as shown in FIG. 14, a group of straight lines passing through one arbitrary point P on the xy coordinate system can be represented as one line graph μ on the ξρ coordinate system. Therefore, in step S14, for one pixel constituting the image data input in step S11, a combination of variables (ξ, ρ) for each of the straight line groups passing through the reference position of the pixel is obtained, and two-dimensional ξρ is obtained. A process is performed in which coordinate points corresponding to each variable pair (ξ, ρ) are plotted on the coordinate system, and a line graph μ composed of a set of these coordinate points is obtained on the ξρ coordinate system.

  That is, the process of step S14 can be said to be a process of obtaining the line graph μi on the ξρ coordinate system for the i-th pixel Pi, if described using the parameter i. FIG. 19 is a plan view showing an example of the line graph μi thus obtained on the ξρ coordinate. The line graph μi on the ξρ coordinate system shown in FIG. 19 is a graph showing a straight line group passing through the reference position of the i-th pixel Pi on the xy coordinate system shown in FIG. 18 (the line shown in FIG. 14). Same as the relationship between graph μ and point P). In the case of the embodiment described here, since the center point position of the pixel is adopted as the “pixel reference position”, the line graph μi shown in FIG. 19 is the center of the pixel Pi shown in FIG. This is a graph representing a group of straight lines passing through the points. Of course, as the “pixel reference position”, it is not always necessary to use the center point position. For example, the position of the upper left corner of the pixel may be used as the reference position.

  In the next step S15, the pixel Pi (as shown in FIG. 18) is set to the pixel value of the pixel (pixel on the integrating two-dimensional pixel array defined on the ξρ coordinate system) obtained in step S14. This is a pixel value integration stage process of integrating the pixel values of the i-th pixel arrayed on the xy coordinate system. As already described, in the preparation stage of step S12, a two-dimensional pixel array for integration in which each pixel has an initial pixel value of 0 is defined on the ξρ coordinate system. Further, in the line graph calculation stage of step S14, a line graph μi as shown in FIG. 19 is obtained on the ξρ coordinate system. In the process of step S15, the line graph μi obtained for the i-th pixel Pi constituting the image data input in step S11 is overlaid on the integrating two-dimensional pixel array prepared in step S12. This is a process of adding the pixel value of the pixel Pi to the pixel value of the pixel located on μi.

  FIG. 20 is a plan view showing pixels (hatched portions) located on the line graph μi on the ξρ coordinate shown in FIG. The process of step S15 is a process of adding the pixel value of the pixel Pi shown in FIG. 18 to the pixel value of the pixel shown by hatching in FIG. In the case of the example shown here, the pixel Pi shown in FIG. 18 is a black pixel having the pixel value “1”, and therefore the pixel value “1” is included in the pixel values of all the pixels hatched in FIG. Will be accumulated. In other words, only the pixel value of the pixel shown by hatching in FIG. 20 among the pixels constituting the integrating two-dimensional pixel array defined on the ξρ coordinate system is increased by “1”.

  As described above, the processes in steps S14 and S15 are repeatedly executed through steps S16 and S17 while increasing the parameter i by one. FIG. 21 is a plan view showing a process of integrating the pixel values of the pixel P (i + 1) shown in FIG. The line graph μi shown in FIG. 20 is a graph showing a straight line group passing through the reference position of the pixel Pi shown in FIG. 18, whereas the line graph μ (i + 1) shown in FIG. 21 is the pixel P shown in FIG. Since it is a graph showing a group of straight lines passing through the reference position of (i + 1), the pixels to be accumulated on the ξρ coordinates (pixels shown by hatching in FIGS. 20 and 21) are different. However, in both FIG. 20 and FIG. 21, the hatched pixels (pixels located near the intersections of the line graphs μi and μ (i + 1)) are the reference positions of the pixels Pi and the pixels P shown in FIG. It corresponds to a straight line passing through both of the reference positions (i + 1).

  Since the pixels P1, P2, P3, etc. shown in FIG. 18 are white pixels having a pixel value “0”, the pixels located on the line graphs μ1, μ2, μ3 on the ξρ coordinate system corresponding to these pixels. Therefore, the pixel value integrated in step S15 is “0”. Therefore, the processes in steps S14 and S15 are meaningless if the parameter i corresponds to a white pixel (for example, when i = 1, 2, 3,...). After all, theoretically, the processes in steps S14 and S15 are repeatedly executed a total of N times (from the first pixel P1 to the Nth pixel PN shown in FIG. 18) until the parameter i reaches N in step S16. However, in practice, it is sufficient to perform the processing of steps S14 and S15 only for black pixels having a pixel value “1” among all N pixels, and white processing having a pixel value “0”. There is no need to do this for the pixels.

  Thus, when the parameter i reaches N in step S16, the process proceeds from step S16 to step S18, and the reference frame position prediction step is executed. In this reference frame position prediction step, the position of the image of the reference frame that will appear on the photographed image arranged on the xy coordinate system is determined based on the geometric photographing condition at the time of obtaining the photographed image. This is a process of predicting by a typical calculation. The captured image input as image data in step S11 is an image captured under specific geometric imaging conditions as described in §3. Theoretically, this imaging condition is given accurately. If so, it is possible to convert the photographed image into a regular image by a geometric calculation based only on the photographing condition.

Specifically, in the case of the embodiment described here, photographing is performed using a photographing system as shown in FIG. 6 (however, a rectangular shape as shown in FIG. 10 is formed on the photographing stage 30. As a geometric imaging condition for obtaining a captured image,
(1) An angle θ (θVs in FIG. 6) between the normal N set at the reference point S and the optical axis of the photographing apparatus (camera 60 in FIG. 6),
(2) The projected image (Vs ′ in FIG. 6) onto the “plane on which the reference frame is drawn” (the upper surface of the imaging stage 30 in FIG. 6) and the reference point S of this optical axis passes through the “reference frame is drawn. Angle φ (φVs in FIG. 6) formed with a predetermined reference line ζ (ζs in FIG. 6) included in
(3) the focal length of the optical system of the imaging device (camera 60 in FIG. 6);
It is possible to give Therefore, in the reference frame position prediction stage in step S18, by a geometric calculation based on the above geometric imaging conditions (actually, in addition to this, the imaging magnification of the camera, the pixel pitch on the CCD element, etc.) The calculation is performed in consideration of the photographing condition on the camera side.) It is possible to predict the position of the image of the reference frame 35 that will appear on the photographed image input as image data in step S11.

  As shown in FIG. 10, this geometric calculation is performed based on the position and shape of the reference frame 35 when the imaging stage 30 is viewed from directly above, as shown in FIG. This is a calculation for predicting the position and shape of the reference frame 35 when shooting from the direction (direction indicated by the shooting conditions), and is a calculation based on a known method generally called projective normalization conversion. However, since the actual shooting condition causes an error with respect to the ideal shooting condition, the image of the reference frame obtained in the reference frame position prediction step in step S18 does not necessarily match the image of the correct reference frame. As described above, since the process of converting a captured image into a regular image has been performed on the premise that the image of the reference frame obtained in the reference frame position prediction stage is a correct image of the reference frame. The conversion process could not be performed. In the present invention, the image of the reference frame obtained in step S18 is handled as a predicted position where the reference frame will appear.

  In the subsequent step S19, the parameter j is set to the initial value 1, and the processes in steps S20 and S21 are repeatedly executed through step S22. At this time, the parameter j is incremented and updated by 1 in step S23 each time. Here, the parameter j is a parameter indicating a straight line constituting the reference frame. In the case of the embodiment described here, since the reference frame 35 is rectangular as shown in FIG. 10, the reference frame 35 is composed of a total of four straight lines and takes values j = 1, 2, 3, and 4. For example, when j = 1, side 35A is shown, when j = 2, side 35B is shown, when j = 3, side 35C is shown, and when j = 4, side 35D is shown. In addition, the determination criterion M in step S22 is M = 4 in this case.

  Steps S20 and S21 are processes in the actual measurement prediction point determination stage, and here, for the combinations of variables (ξ, ρ) indicating the respective straight lines constituting the image of the reference frame 35 predicted in step S18, integration is performed. The corresponding coordinate points on the two-dimensional pixel array are plotted as the geometric prediction points ε, and the pixel value of the pixel belonging to the predetermined neighborhood region for each geometric prediction point ε is the largest. A process of determining the reference position as an actually estimated point for the corresponding straight line is performed.

  Before explaining the meaning of the process in the prediction point determination stage in actual measurement, first, the meaning of the pixel value of each pixel constituting the integrating two-dimensional pixel array will be explained. The integrating two-dimensional pixel array is a pixel array defined on the ξρ coordinate system in step S12, and the pixel values of all the pixels are set to 0 in the initial state. However, when the processes in steps S13 to S16 are repeated, the pixel value of a specific pixel gradually increases. Therefore, let us look back at when the pixel value of each pixel increases.

  For example, in the integration process for the i-th pixel Pi shown in FIG. 18, the pixel value of each pixel shown by hatching in FIG. 20 increases by one. Here, since each pixel shown by hatching in FIG. 20 is a pixel on the line graph μi, this indicates that the pixel group on the “ξρ coordinate” indicates a straight line group passing through the reference position of the “pixel Pi on the xy coordinate”. It means “pixel”. After all, in step S15, the pixel on the ξρ coordinate on which the pixel value “1” of the specific black pixel on the xy coordinate is integrated is a pixel indicating a straight line passing through the reference position of the specific black pixel. If there is a common straight line (straight line on the xy coordinates) passing through the reference positions of all the black pixels shown in FIG. 18, pixels on the ξρ coordinate corresponding to such a straight line are all pixels on the ξρ coordinate. That is, the pixel having the maximum pixel value among the pixels. This is the same principle that the point γ in FIG. 15 (b) corresponds to the straight line L in FIG. 15 (a).

  Now, when the pixel value integration processing has been completed for all N pixels (all pixels of the captured image) on the xy coordinate system shown in FIG. 18 (that is, when i = N is determined in step S16). Suppose that the pixel having the maximum pixel value on the integrating two-dimensional pixel array defined on the ξρ coordinates is a pixel γ as shown in FIG. In this case, it can be said that the straight line on the xy coordinate corresponding to the pixel γ is the straight line having the highest possibility of a straight line connecting all the black pixels shown in FIG. For example, when the coordinate value of the reference position (for example, the center position) of the pixel γ is (ξγ, ργ), a straight line L indicated by a combination of variables (ξγ, ργ) is drawn on the xy coordinates. As shown in FIG. 23, this straight line L is a straight line passing over most black pixels shown in the figure. In other words, the maximum pixel value of the pixel γ shown in FIG. 22 is obtained as an integration result of the pixel values of the black pixels located on the straight line L in FIG.

  As described above, the pixel value of each pixel constituting the integrating two-dimensional pixel array on the ξρ coordinate indicates how many black pixels constituting the photographed image the straight line on the xy coordinate system corresponding to the pixel passes through. It will be the parameter to show. Therefore, the straight line represented by the coordinate value (ξγ, ργ) of the pixel γ having the maximum pixel value means the straight line having the highest possibility of the straight line formed by connecting the black pixels in the photographed image on the xy coordinate. It has. Thus, as shown in FIG. 22, if the pixel γ having the maximum pixel value is determined on the two-dimensional pixel array for integration, the coordinate value (ξγ, ργ) is used to determine the xy coordinate as shown in FIG. A straight line L can be determined.

  If the straight line L is determined by such a method, even if the straight line constituting the image of the reference frame 35 is constituted by discontinuous black pixels on the photographed image, this is expressed as a geometric straight line. It becomes possible to recognize as. In addition, the image of the reference frame 35 directly recognized from the information on the photographed image as described above is an index indicating an accurate position that does not include an error like the image of the reference frame predicted based on the geometric photographing condition. Can be used.

  However, it is practically problematic to determine the straight line constituting the image of the reference frame 35 only by the method of “determining the pixel γ having the maximum pixel value on the two-dimensional pixel array for integration”. The first problem is that since the reference frame 35 is composed of a plurality of straight lines, it is necessary to distinguish the plurality of straight lines from each other. The example shown in FIG. 18 shows a photographed image made up of a set of black pixels showing one straight line for convenience of explanation, and therefore only one straight line L is determined as shown in FIG. It was enough. However, since the actual reference frame 35 is composed of four straight lines, it is necessary to determine four straight lines as images of the reference frame 35 for one photographed image.

  The second problem is that only the image of the reference frame 35 is not shown on the photographed image. In actual photographing, as shown in FIG. 10, an object 40 as a subject is placed inside a reference frame 35 drawn on a photographing stage 30, and the whole is photographed with a camera. Therefore, the captured image includes not only the reference frame 35 but also the object 40. Therefore, if the processes in steps S13 to S16 are repeatedly executed for all the pixels on the captured image, a linear element on the object 40 is also detected. In practice, since photographing is performed from various photographing directions under various illumination conditions, reflections and shadows are generated in unintended portions on the photographing stage 30, and these are erroneously generated by the reference frame 35. May be recognized as part of

  It can be said that the reference frame position prediction stage performed in step S18 is a process of preparing for solving such a problem. As described above, the position of the image of the reference frame predicted in the reference frame position prediction stage usually includes an error related to the shooting conditions, and thus is not correct. However, the position of the image of the reference frame predicted here does not greatly deviate from the position of the image of the original reference frame. That is, even if the original image of the reference frame deviates from the position predicted in step S18, it can be considered that the image is located within a predetermined range in the vicinity thereof.

  Therefore, in step S20, geometric prediction points ε are plotted on the ξρ coordinate system for the jth straight line constituting the reference frame 35. That is, since the position of the image of the reference frame 35 composed of four straight lines on the xy coordinate system is already predicted in step S18, the jth straight line (for example, the four straight lines (for example, 11), the angle ξε with respect to the coordinate axis x and the length ρε of the perpendicular OF drawn from the origin O to the straight line are obtained, and the geometric prediction point ε (ξε, ρε) on the ξρ coordinate. Can be plotted. FIG. 24 is a plan view showing an example of the geometric prediction point ε plotted in this way.

  Next, in step S21, a process is performed in which a pixel having the maximum pixel value is searched for in the vicinity of the plotted geometric prediction point ε, and the reference position of the pixel is determined as an actually measured prediction point γ. Specifically, it is only necessary to compare the pixel values of pixels whose reference positions (for example, center positions) of the individual pixels fall within the vicinity region and determine the pixel having the maximum pixel value.

  In FIG. 24, a region surrounded by an alternate long and short dash line is set as a region near the geometric prediction point ε. That is, when the coordinate value of the geometric prediction point ε is (ξε, ρε), the following conditions are used: ξε−Δξ ≦ ξ ≦ ξε + Δξ, ρε−Δρ ≦ ρ ≦ ρε + Δρ, using predetermined deviations Δξ and Δρ. A region indicated by the coordinate values (ξ, ρ) to be satisfied is set as a neighborhood region for the geometric prediction point ε. Here, as the deviations Δξ and Δρ, preferable values may be appropriately set to solve the above-described two problems. Of course, the neighboring area of the geometric prediction point ε is not necessarily a rectangular area as shown in the figure. For example, a circle having a predetermined radius around the geometric prediction point ε is set as the neighboring area. Is also possible.

  In the example shown in FIG. 24, for example, the pixel having the largest pixel value among all the pixels is the pixel γ1, the pixel having the second largest pixel value is the pixel γ2, and the third largest pixel value is obtained. Assuming that the pixel having the pixel γ3 is the pixel γ3 having the maximum pixel value among the pixels belonging to the neighboring region is extracted in step S21, the reference position (ξγ, ργ) of the pixel γ3 is predicted by actual measurement. Will be determined as a point. Here, the geometric prediction point ε (ξε, ρε) obtained based on the geometric imaging conditions is shifted from the predicted point (ξγ, ργ) by the distance shown in the figure. The deviation corresponds to a deviation between the set theoretical photographing condition and the actual photographing condition when photographing is actually performed. Subsequently, as in the example shown in FIG. 23, if a straight line L having an angle ξγ corresponding to the actually estimated point (ξγ, ργ) and a displacement ργ from the origin is obtained on the xy coordinate system, the straight line L is This corresponds to an image of the jth straight line constituting the reference frame 35.

  In the example shown in FIG. 24, the pixel γ1 having the largest pixel value and the pixel γ2 having the second largest pixel value are greatly deviated from the geometric prediction point ε. The accumulated pixel value is large as a result of an influence such as a pixel corresponding to another straight line that is not the jth straight line, a pixel corresponding to a straight line constituting the object 40, or an unintentional shadow at the time of photographing. It is thought that the pixel has become.

  After all, the process of the step of determining the predicted point in actual measurement in steps S20 and S21 includes a straight line recognition method (a method for obtaining a pixel having the maximum pixel value on the ξρ coordinate) based on the photographed image and a geometric photographing condition. In combination with a conventional image conversion method that uses as a clue, the position of a straight line obtained by the conventional image conversion method is plotted as a geometric prediction point ε on the ξρ coordinate, and the geometric prediction point ε It can be said that the position of the pixel having the maximum pixel value in the vicinity region is determined as an actually measured prediction point, and a straight line image constituting the target jth reference frame 35 is obtained on the photographed image.

  In this way, the procedure of steps S20 and S21 is repeatedly executed while increasing the parameter j by 1, and the straight line images constituting the reference frame 35 are sequentially obtained one by one on the photographed image. In the example, when M = 4) is obtained, the process proceeds to step S24 via step S22.

  The process of the last step S24 draws the straight line represented by the coordinate value (ξ, ρ) of each actually estimated point on the two-dimensional xy coordinate system, and the position of these straight lines is the correct position of the image of the reference frame 35. The captured image input in step S11 is converted into a regular image by geometric calculation. That is, as shown in FIG. 9 (a), if the correct image of the reference frame 35 can be recognized as a quadrilateral ABCD on the photographed image in the xy coordinate system, a known geometric calculation method such as the projective normalization conversion described above. Thereby, the process which converts the said picked-up image into a regular image can be performed. On the regular image thus obtained, the image of the reference frame 35 is a regular quadrilateral ABCD as shown in FIG.

<<< §8. More practical embodiment >>
In §7 described above, the basic concept of the image conversion method into a regular image, which is a feature of the present invention, has been described using a simple example model. Here, some more practical embodiments of the present invention will be described.

(1) Case where Captured Image is Gradation Image In the basic embodiment described so far, a simple example in which the captured image input in step S11 in the flowchart of FIG. 17 is a binary image is shown. For example, the photographed image shown in FIG. 18 is a binary image composed of two types of pixels, a black pixel shown hatched in the figure and other white pixels. However, in photographing for the purpose of obtaining reflection characteristic data of an object, a photographed image having gradation is usually used. As described above, even when the captured image is a gradation image, it is possible to handle the basic embodiment as it is.

  For example, in the case of a gradation image having four gradation values (2-bit gradation image), the pixel value of each pixel is “0”, “1”, “2”, “3”. It will increase to 4 types. Here, a pixel having a pixel value “0” is a white pixel, a pixel having a pixel value “1” is a light pixel, a pixel having a pixel value “2” is a dark pixel, and a pixel value is “3”. The pixel is called a black pixel, and a straight line appearing on the photographed image is expressed by a combination of a black pixel, a dark pixel, and a light pixel on a background made up of white pixels as in the example shown in FIG. Let's consider the case. Actually, when the reference frame 35 drawn on the shooting stage 30 is photographed by a digital camera or the like having a predetermined resolution, the image of the reference frame 35 on the photographed image having gradation has several pixel values. It is generally expressed as a collection of pixels having

  In the case of the example shown in FIG. 25, each black pixel indicates the most probable position of the straight line image constituting the reference frame 35, and each dark pixel indicates a position with a slightly low accuracy, Each light pixel is considered to indicate a position with lower accuracy. Further, it can be considered that the white pixel indicates a position where the straight line image forming the reference frame 35 does not exist. Therefore, even when the photographed image is not a binary image but a gradation image, there is no problem if the pixel values of the individual pixels are integrated with the pixel values of the pixels on the integrating two-dimensional pixel array.

  In the above-described example using the binary image (example in FIG. 18), the pixel value “1” is integrated in the integration process for black pixels, and the pixel value “0” is integrated in the integration process for white pixels. (Substantially the same as not performing integration processing for white pixels). On the other hand, when the gradation image shown in FIG. 25 is used, the pixel value “3” is integrated in the integration process for black pixels, and the pixel value “2” is integrated in the integration process for dark pixels. In the integration process for pixels, the pixel value “1” is integrated, and in the integration process for white pixels, the pixel value “0” is integrated (substantially, no integration process is performed for white pixels). the same). This is because there is a possibility that a straight line image constituting the reference frame 35 exists with a certain degree of probability at the position of the dark pixel or the position of the light pixel. Therefore, it is better to integrate the pixel values according to the probability, This approach is based on the idea that more accurate results will be obtained.

  Of course, in practice, a gradation image having more gradation steps is generally used as a captured image. For example, in the case of an 8-bit gradation image, 256 pixel values from 0 to 255 are defined for each pixel. In this case, the straight line image constituting the reference frame 35 will be expressed as an aggregate of pixels having pixel values within the range of 240 to 255, for example. In this case, it is considered that the pixel having the pixel value “255” indicates the most probable position of the straight line image forming the reference frame 35, but the pixel having the pixel value “240” It can be considered that a certain position of the straight line image constituting the frame 35 is shown. Therefore, even in such a case, if the handling is performed such that the pixel values are integrated as they are, the position of the straight line image constituting the reference frame 35 can be accurately determined in the same manner as the basic embodiment described so far. It is possible to determine.

(2) Variation of reference frame In the basic embodiment described so far, an example in which a rectangular reference frame 35 is drawn on the upper surface of the photographing stage 30 as shown in FIG. 10 is shown. It was. However, the reference frame used in the present invention is not necessarily a rectangular figure having four sides. The role of the reference frame in the present invention is shown in FIG. 9 (b) based on the geometric shape of the quadrilateral ABCD obtained as an image when taken from an oblique direction, as shown in FIG. 9 (a). It is to provide a clue for specifying a geometric transformation (projective normalization transformation or the like) for returning to a regular quadrangle. Since such geometric transformation can be performed if the positions of the four vertices A, B, C, and D of the quadrangle are specified, the reference frame 35 does not necessarily have to be a graphic quadrangle.

  For example, the reference frame 35 shown in FIG. 10 is a rectangular frame having four sides of 35A, 35B, 35C, and 35D, but instead, a U-shaped frame having three sides of 35A, 35B, and 35C is used as a reference. It may be used as a frame. In this case, the fourth side 35D is not drawn, but it is possible to specify the four vertices A, B, C, and D by a U-shaped frame. Similarly, a figure composed of only two line segments 35A and 35C may be used as the reference frame. Even in this case, the four vertices A, B, C, and D can be specified as the end points of the two line segments 35A and 35C.

  Further, the reference frame does not necessarily need to be configured by a figure drawn with lines. For example, the reference frame can be configured by a boundary line due to a contrast difference between the inside and the outside of a predetermined area. FIG. 26A is a plan view showing an example of the reference frame 35 constituted by such a boundary line. In this example, a rectangular region 38 is formed on the upper surface of the shooting stage 30. Here, the rectangular region 38 is formed as a region in which a contrast difference occurs between the inside and the outside. Specifically, for example, a black rectangular region 38 may be formed on the imaging stage 30 having a white upper surface. Then, a rectangular reference frame 35 can be formed as a boundary line between the black region (the hatched portion in the figure) and the surrounding white region.

  However, when the reference frame is configured by the boundary line due to the contrast difference as described above, the image of the reference frame cannot be obtained as an aggregate of black pixels, for example, as shown in FIG. Therefore, the straight line constituting the reference frame can be grasped only as a boundary line between the area composed of the set of black pixels and the area composed of the set of white pixels. Therefore, in this case, when inputting the image data in step S11 of FIG. 17, it is necessary to perform a process of extracting the image of the reference frame 35 formed by the boundary line.

  Actually, the process of extracting the image of the reference frame 35 can be performed by a differentiation process on the captured image. FIG. 26B is a graph showing changes in pixel values when the imaging stage 30 shown in FIG. 26A is scanned in the horizontal direction in the figure. Here, an example in which the white pixel value is 0 and the black pixel value is 255 is shown. As shown in the figure, the pixel value of the portion corresponding to the black rectangular region 38 is 255, and the pixel value of the portion corresponding to the white region on both sides thereof is 0. Of course, since the actual captured image is captured with the article to be a subject placed inside the rectangular area 38, the pixel value distribution of the actually captured image does not look like this graph. The pixel value is abruptly changed at the boundary portion of the rectangular area 38.

  FIG. 26 (c) is a graph obtained by differentiating the graph of FIG. 26 (b). By performing differentiation, an abrupt change in pixel value appears as an abrupt increase in the differential value, and the boundary portion of the rectangular area 38 can be detected. Since the differential values in FIG. 26 (c) are shown as absolute values, the boundary portions are all represented as lines extending upward. Eventually, as shown in FIG. 26 (a), when the reference frame is constituted by the boundary line due to the contrast difference between the inside and the outside of the predetermined area, it is obtained by performing a differentiation process on the captured image at the image input stage. If the obtained differential image is input as image data, the basic embodiment described so far can be applied as it is.

  As described above, the image data input in step S11 of FIG. 17 is not necessarily the captured image itself, and the captured image may be subjected to predetermined processing (for example, the above-described differential processing or other filtering processing may be performed). It may be an image subjected to.

(3) Pixels on the ξρ coordinate that are subject to pixel value accumulation In the case of the basic embodiment described so far, at the line graph calculation stage of step S14 in the flowchart of FIG. A line graph .mu.i as shown in FIG. 19 is obtained on the .xi..rho. Coordinate corresponding to the pixel Pi, and at the subsequent pixel value integration stage of step S15, as shown in FIG. 20, the two-dimensional pixel array for integration on the .xi..rho. The line graph μi is overlaid on top of each other, and the pixel value of the pixel Pi is added to the pixel value of the pixel (the pixel shown by hatching in FIG. 20) located on the line graph μi.

  However, the pixel on the ξρ coordinate that is the pixel value integration target in the pixel value integration stage of step S15 is not necessarily limited to the pixel located on the line graph μi, and is located on or near the line graph μi. It does not matter even if it extends to the pixel to do. For example, in the example shown in FIG. 20, the pixels to be accumulated (hatched pixels) are all limited to the pixels through which the line graph μi passes through a part of the pixels. The pixels to be accumulated may be further expanded to include pixels within a predetermined distance range with respect to the line graph μi.

  FIG. 27 is a plan view for explaining a modification in which pixel values are also integrated for pixels Td in the vicinity of the line graph μi on the ξρ coordinate. For example, the distance d between the reference position (center position) of the pixel Td and the line graph μi (the distance from the closest point on μi) is defined as “the distance between the pixel Td and the line graph μi”, and this distance d is determined in advance. If the pixel Td that is equal to or less than the set threshold value is included in the pixel that is subject to pixel value accumulation, the distance d to the line graph μi as well as the hatched pixels in FIG. The pixel value of the pixel Pi is also integrated with respect to neighboring pixels in which is below a predetermined threshold.

  In addition, when the pixel to be accumulated in pixel values is extended to a pixel located in the vicinity of the line graph μi in this way, the pixel value to be accumulated is differentiated according to the degree of proximity to the line graph μi. Is preferred. For example, in the example shown in FIG. 27, the hatched pixels are all pixels on the line graph μi, whereas the pixel Td is a pixel located at a distance d from the line graph μi. When pixel values related to the line graph μi (pixel values of the pixel Pi) are integrated, it is more reasonable to integrate the latter in consideration of the contribution ratio of the pixel values.

  Therefore, in practice, the pixel value of the pixel Pi is added as it is to the pixel value of the pixel (hatched pixel in FIG. 27) located on the line graph μi, and only a predetermined neighborhood distance d from the line graph μi. The pixel value of the distant pixel is multiplied by the pixel value of the pixel Pi by a predetermined ratio R corresponding to the neighborhood distance d (a value that takes a range of 0 ≦ R ≦ 1 and decreases as the neighborhood distance d increases). What is necessary is just to integrate a value. For example, when the pixel value of the pixel Pi is 255, the pixel value 255 is added as it is to each pixel shown by hatching in FIG. 27, and the pixel value Td is located at a position separated by the neighborhood distance d. On the other hand, a value “255 × R” multiplied by a predetermined ratio R (0 ≦ R ≦ 1) determined according to the neighborhood distance d may be integrated.

  The predetermined ratio R may be defined by any method as long as the neighborhood distance d becomes smaller. However, the inventor of the present application is that R is close to 1 while the neighborhood distance d is very small. We believe that it is preferable to define the relationship between R and d by a function in which R decreases rapidly when the neighborhood distance d increases to some extent. Therefore, in practice, as shown in FIG. 28, a function having a Gaussian distribution around the position of the line graph μi is defined, and a predetermined ratio R with respect to an arbitrary neighborhood distance d is determined based on this function. That's fine.

(4) Variation of Actually Predicted Point Determination Method In the case of the basic embodiment described so far, in the actually estimated point determination step of step S21 in the flowchart of FIG. A pixel having the “maximum pixel value” was searched, and the reference position of the pixel was determined as a predicted point in actual measurement. However, the pixel having the “maximum pixel value” is searched for on the image on the xy coordinates, “the pixel value of the pixel constituting the image of the reference frame is higher than the pixel value of the background pixel. This is because it is assumed that the setting “large” is made.

  For example, in the case of the example shown in FIG. 18, description has been made that the pixel value of the black pixel constituting the image of the reference frame is “1” and the pixel value of the white pixel as the background is “0”. Also in the example shown in FIG. 25, the pixel values of the black pixel, the dark pixel, and the light pixel that constitute the image of the reference frame are “3”, “2”, and “1”, respectively. The pixel value of “0” is described as “0”. That is, the embodiments described so far are based on the premise that “the pixel value of the image portion of the reference frame is larger than the pixel value of the background portion”.

  However, in carrying out the present invention, it is not always necessary to take such a premise. For example, when the image on the xy coordinates is a binary image, the pixel value of the pixel constituting the image of the reference frame is set to “0”, and the pixel value of the pixel in the background portion is set to “1”. It doesn't matter. In the case of a gradation image having a pixel value of 0 to 255, the pixel value of the pixel constituting the image of the reference frame is “about 0 to 50”, and the pixel value of the background portion pixel is “200 to 255”. An image distributed in “degree” may be used. However, when processing is performed by inputting “an image in which the pixel value of the image portion of the reference frame is smaller than the pixel value of the background portion” on the xy coordinate system in this way, The pixel value integrated becomes smaller as the pixel corresponds to “a straight line passing through pixels constituting the image of the reference frame on the coordinates”. Therefore, in the actually predicted point determination step in step S21, a pixel having “minimum pixel value” is searched for in place of the “maximum pixel value” in the region near the geometric prediction point ε, and the reference position of the pixel is actually measured. It is necessary to determine the upper prediction point.

  In the above-described embodiment, the two-dimensional pixel array for integration is defined as a collection of pixels having the initial pixel value 0 in step S12 in FIG. 17, but the initial pixel value is not necessarily set to 0. do not have to. As described above, in determining the predicted point in actual measurement, the pixel having the “maximum pixel value” or the “minimum pixel value” is searched from the constituent pixels of the integrating two-dimensional pixel array defined on the ξρ coordinate. Therefore, it is only necessary to prepare an environment in which pixel values can be compared between pixels. Accordingly, the initial pixel values of the individual pixels constituting the integrating two-dimensional pixel array may be set to any values as long as all the initial pixel values are the same.

  It should be noted that in the actual measurement prediction point determination step in step S21, among the pixels belonging to the predetermined neighborhood region with respect to the geometric prediction point ε, a plurality of pixels having “maximum pixel value” or “minimum pixel value” existed. In this case, the center-of-gravity position of the reference position of the plurality of pixels may be determined as the actually predicted point for the corresponding straight line. For example, in the example shown in FIG. 24, the case where the pixel γ3 in the vicinity region indicated by the alternate long and short dash line is a pixel having the “maximum pixel value” has been described. Here, if there are two pixels γ3 and γ4 as the pixels having the “maximum pixel value”, if both of the reference positions are measured prediction points, two straight lines are formed on the xy plane. Will be defined. In such a case, the center-of-gravity position between the reference position of the pixel γ3 and the reference position of the pixel γ4 may be determined as an actually estimated point.

(5) Pixel on xy coordinates to be accumulated pixel values The processing of steps S13 to S17 in the flowchart of FIG. 17 is a procedure that is repeatedly executed while incrementing the parameter i by 1, and its meaning is For each of all N pixels constituting the image input on the xy coordinates, a line graph is obtained on the ξρ coordinate, and pixels on the ξρ coordinate located on or near the line graph are integrated. The pixel value is integrated as the target pixel. Therefore, according to the flowchart, the processes of steps S14 and S15 are executed for all N pixels from the first pixel on the xy coordinates to the Nth pixel. become.

  However, in practice, there are cases where the processes of steps S14 and S15 do not necessarily have to be executed for all N pixels. For example, for a pixel having a pixel value “0”, the pixel value of the pixel to be integrated is “0” even if the pixel values are integrated by executing steps S14 and S15. This is meaningless processing. Therefore, practically, in such a case, the processes of steps S14 and S15 can be omitted.

  Further, even if it is not a pixel having the pixel value “0”, there is a pixel in which the processes of steps S14 and S15 can be omitted in practice. For example, a gradation image composed of a collection of pixels having pixel values in the range of 0 to 255 exists on the xy coordinates, and the pixel values in the range of pixel values 0 to 50 are almost certainly about the pixels in the background portion. In the case where it is predicted that the pixel value of the pixel value is within the range of 0 to 50, there is no significant problem even if the processing of steps S14 and S15 is omitted for pixels having pixel values in the range of pixel values 0 to 50.

  In consideration of such a case, the line graph calculation stage in step S14 in FIG. 17 only needs to execute a calculation for obtaining a line graph only for pixels having pixel values within a predetermined range among all N pixels. In the pixel value integration step of step S15, it is only necessary to perform the pixel value integration processing only for pixels having pixel values within a predetermined range among all N pixels.

  It should be noted that as a factor to be considered when determining a pixel on the xy coordinate that is a pixel value integration target, the position of the pixel can be used instead of the pixel value of the pixel. For example, the reference frame position prediction stage in step S18 can be performed prior to step S13. In such a case, at the time when the processes in steps S14 and S15 are executed, the reference frame position prediction stage is based on geometric imaging conditions. Thus, the reference frame position has already been predicted. Of course, the reference frame position predicted here is not accurate, but it is considered that the actual reference frame position exists in a region near the predicted reference frame position. Of the pixels that are defined as “consideration regions” and are arranged on the xy coordinates, only pixels located within the “consideration region” are subject to pixel value accumulation, and other pixels are processed in steps S14 and S15. Even if the processing is omitted, no major trouble occurs.

  For example, in the case of the example shown in FIG. 18, in the basic embodiment described above, all N pixels are targeted for integration of pixel values, but it is possible to narrow down the pixels targeted for integration to a part thereof. . That is, if the predicted position of the reference frame is obtained by the reference frame position prediction step in step S18, it will be close to the straight line connecting the black pixels shown in the figure. Therefore, if a “consideration area” is defined in the vicinity of the predicted position of the reference frame, it will be an elongated area including the black pixels shown in the figure. The processes in steps S14 and S15 are executed only for the pixels in the elongated “consideration region”, and are omitted for the other pixels.

  In short, in the line graph calculation stage in step S14, a calculation for obtaining a line graph is performed only for pixels located within a predetermined consideration area among all N pixels, and in the pixel value integration stage in step S15, all N pixels are calculated. It is only necessary that the pixel value integration process is executed only for pixels located within a predetermined consideration area among the pixels.

(6) Handling of failure in determination of predicted point in actual measurement In consideration of the contents of the modification examples described so far, in step S21 of FIG. 17, the pixel belonging to the predetermined neighborhood region for the geometric predicted point ε The reference position of the pixel having the maximum or minimum pixel value is determined as a predicted point in actual measurement. However, in practice, when the actually estimated point is determined by such a method, a point corresponding to a straight line completely unrelated to the straight line image constituting the reference frame is erroneously detected as the actually measured predicted point. Is likely to occur. In particular, for the purpose of creating reflection characteristic data as described in §3, when a large number of captured images are obtained under various shooting conditions, the influence of the reflected image of illumination light is affected in some captured images. Accordingly, there is a possibility that erroneous detection occurs in step S21.

  For example, in the imaging system shown in FIG. 6, if the combination of the incident angle θLs and azimuth angle φLs of the illumination light Ls and the incident angle θVs and azimuth angle φVs of the reflected light Vs satisfies a specific condition, it is on the imaging stage 30. Due to the specular reflection, the image of the light source 50 may be directly formed on the imaging surface of the camera 60. On the photographed image obtained under such conditions, the image of the light source 50 appears as a pure white reflection area, and the image of the reference frame becomes so-called whiteout and cannot be recognized. However, if the procedure shown in the flowchart of FIG. 17 is performed on such a photographed image, a predicted point for actual measurement is determined in step S21 on the algorithm. As long as the image of the original reference frame is not shown, the actually estimated points determined in this way are completely unrelated to the straight lines constituting the original reference frame.

  This is because, as long as the predicted point of measurement is determined by the algorithm of “pixel having the maximum or minimum pixel value”, the part that can recognize the presence of any linear pattern in the pure white reflection area is the estimated point of measurement. It is because it will be recognized. In the case of the example shown in FIG. 24, for example, many pixels in the region surrounded by the alternate long and short dash line have a pixel value of “about 0 to 100”, but only the pixel γ3 has the maximum pixel value “10000”. Thus, in general, the maximum pixel value is a considerably prominent value compared to other pixel values. The higher the degree of protrusion, the more accurate straight line detection has been performed. However, when the image of the reference frame is overexposed, all the pixels in the area surrounded by the alternate long and short dash line shown in FIG. 24 may have a pixel value of “about 0 to 100”, for example. Will occur. Even in such a situation such as “acorn comparison”, the pixel γ3 having the maximum pixel value “101” is extracted, for example, as long as the algorithm described above is followed, and the reference position is determined as a predicted point in actual measurement. become. Of course, the actually measured prediction point determined under such circumstances does not indicate a straight line image forming the reference frame.

  In order to avoid such erroneous detection, “when the maximum pixel value in the neighboring region is less than a predetermined threshold” or “the minimum pixel value is not predetermined at the actual prediction point determination stage of step S21. In the case where the threshold value is exceeded, it is only necessary to handle the failure in image conversion to a regular image without determining a predicted point in actual measurement. For example, in the case where only the pixel γ3 generally takes the maximum pixel value “10000” as in the example shown in FIG. 24 described above, for example, the threshold value is set to 5000, and “ In the case where the maximum pixel value in the neighborhood region is less than 5000, it is determined that correct detection has not been performed, the prediction point in actual measurement is not determined, and the image conversion to the regular image has failed. You just have to do it. By doing so, it is possible to avoid erroneous detection in which the pixel γ3 having the maximum pixel value “101” is extracted and the reference position is determined as a predicted point in actual measurement.

  Note that, from the viewpoint of converting a single photographed image into a regular image, as described above, if determination of a predicted point in actual measurement fails, conversion to a regular image cannot be performed, resulting in complete failure. However, if a large number of captured images are obtained under various shooting conditions for the purpose of creating reflection characteristic data as described in §3, some of the captured images are converted into regular images. Even if this conversion fails, this can be interpolated.

  That is, in the measurement prediction point determination step in the image conversion step of converting a specific object photographed image into a regular image, if the maximum pixel value in the neighboring region is less than a predetermined threshold value or the minimum pixel value is not predetermined. When the threshold value is exceeded, as described above, the actual prediction point is not determined, the image conversion to the regular image related to the object photographed image is failed, and the image conversion is failed. The regular image for the object photographed image that has been subjected to the above may be obtained by performing interpolation based on the regular image for another object photographed image with similar photographing conditions.

  FIG. 29 is a plan view illustrating an interpolation method when image conversion to a regular image fails for a specific captured image in the reflection characteristic data creation process described in §3. In this figure, the elevation angle θVs of the reflected light Vs in the imaging system shown in FIG. 6 is changed in six ways from θV1 = 0 ° to θV6 = 75 °, and the azimuth angle φVs is changed from φV1 = 0 ° to φV12 = 330 °. The result of changing in 12 ways is shown in tabular form. For example, in column E in this table, the result (image converted to a regular image) when the elevation angle of the reflected light Vs is set to θV3 = 30 ° and the azimuth angle is set to φV5 = 120 ° is arranged. It will be. Of course, actually, the illumination light Ls is actually measured by changing the elevation angle θLs to 6 ways and the azimuth angle φLs to 12 ways, and therefore, in the column E, 72 variations of the illumination light Ls are provided. The 72 regular images shown are arranged.

  Here, for example, when shooting is performed under specific shooting conditions of θV3 = 30 °, φV5 = 120 °, θL4 = 45 °, and φL11 = 300 °, the above-described reference frame image whiteout phenomenon occurs, and the captured image is displayed. Let's suppose that regular conversion failed for. In this case, the failed regular images are eight regular images having the same elevation angle and azimuth angle with respect to the illumination light Ls in the adjacent fields A, B, C, D, F, G, H, and I (that is, photographing conditions). Can be obtained by an interpolation process based on a regular image of another object photographed image that approximates.

  More specifically, a regular image obtained as a result under specific photographing conditions of θV2 = 15 °, φV4 = 90 °, θL4 = 45 °, φL11 = 300 ° is extracted from the column A, and the column B Is taken out as a regular image obtained as a result of specific imaging conditions of θV3 = 30 °, φV4 = 90 °, θL4 = 45 °, φL11 = 300 °, and so on, from column I, θV4 = 45 A regular image obtained as a result under specific imaging conditions of °, φV6 = 150 °, θL4 = 45 °, and φL11 = 300 ° is extracted, and interpolation processing based on the eight regular images thus extracted (for example, 8 (Interpolation processing in which the average value of the pixels at the same position in the image is used as the pixel value of the pixel at the same position in the interpolated image), θV3 = 30 °, φV5 = 120 °, θL4 = 45 °, φL11 = 300 ° An interpolated image that can be arranged in the column E can be obtained as a substitute for the regular image obtained under the shooting conditions.

  Of course, the number of regular images used for interpolation is not necessarily eight. For example, four regular images in columns B, D, F, and H adjacent in the vertical and horizontal directions may be used, or adjacent to each other diagonally. Four regular images in the columns A, C, G, and I may be used.

<<< §9. Apparatus for creating reflection characteristic data of an object according to the present invention >>
Finally, the basic configuration of the object reflection characteristic data creation apparatus according to the present invention will be described with reference to the block diagram shown in FIG. This device is a device that can be used to implement the procedure of creating the reflection characteristic data shown in the flowchart of FIG. 16, and when illuminating light is applied from a predetermined incident direction to a predetermined position of the surface to be measured of the object, The reflectance of the reflected light traveling from the predetermined position toward the predetermined emission direction is (m × n) depending on a combination of a plurality of m incidence directions and a plurality of n emission directions for each position of the surface to be measured. The data defined for the above case has a function of creating the reflection characteristic data for the surface to be measured.

  The main components of this apparatus are an imaging stage 30, a light source 50, a camera 60, an imaging condition setting unit 210, an image conversion device 100, a differential processing unit 220, and a reflection characteristic data creation unit 230, as shown in the figure. The imaging stage 30 has a flat upper surface, and a reference frame 35 composed of straight lines is drawn on this plane. In the example shown in the drawing, a rectangle for placing an object is formed on the upper surface of the imaging stage 30, and the rectangular inner region 38 and the outer region are configured so that a difference in contrast occurs between the two regions. The reference frame 35 is configured by the boundary line.

  The light source 50 has a function of illuminating an object placed on the shooting stage 30, and the camera 60 has a function of shooting an object placed on the shooting stage 30. In the illustrated example, a digital camera is used as the camera 60, and the captured image is output as digital image data.

  The imaging condition setting unit 210 changes the positional relationship between the light source 50 and the surface to be measured of the object placed on the imaging stage 30 in a plurality of m ways, and the positional relationship between the surface to be measured and the camera 60. This is a component having a function of setting a desired photographing condition by changing the number n in plural ways. Such various imaging conditions are as described in Section 3 with reference to the imaging system of FIG. Specifically, the imaging condition setting unit 210 includes a mechanism that changes the relative positions of the light source 50, the camera 60, and the imaging stage 30 in a plurality of variations, a power source such as a motor that drives the mechanism, and the like. It can be configured by a computer to be controlled. The geometric imaging conditions set by the imaging condition setting unit 210 are given to the image conversion apparatus 100 as digital data.

  The captured image captured by the camera 60 is provided as digital image data to the differential processing unit 220, where differential processing is performed, and the differential image is input to the image input unit 110 in the image conversion apparatus 100. The contents of the differentiation process performed by the differentiation processing unit 220 and the purpose thereof are as described in §8 (2) with reference to FIG. If the reference frame formed on the photographing stage 30 is a pattern made of a solid line, the differentiation processing section 220 can be omitted because it is not necessary to perform differentiation processing. In this case, the captured image obtained by the camera 60 is directly input to the image input unit 110.

  The image conversion apparatus 100 is an apparatus having a function of executing the procedure shown in the flowchart of FIG. 17, and subjects that are arranged inside a reference frame constituted by straight lines drawn on the upper surface of the photographing stage 30. This is a device that performs processing for inputting a photographed image obtained by photographing from a predetermined direction from the camera 60 and converting it into a regular image. The reflection characteristic data creation unit 230 is a component that creates reflection characteristic data of an object based on the regular image converted by the image conversion apparatus 100.

  The image conversion apparatus 100 includes an image input unit 110, a line graph calculation unit 120, a pixel value integration unit 130, an actually measured predicted point determination unit 140, a regular conversion unit 150, and a reference frame position prediction unit 160, as illustrated. Yes. However, the image conversion apparatus 100 is actually an apparatus that should be realized by incorporating a dedicated program into a computer. In practice, each of the above components is a combination of computer hardware and software. It will be realized by combination.

  The image input unit 110 captures a captured image obtained from the camera 60 or an image obtained by performing a predetermined processing on the captured image (in the illustrated example, a differential image subjected to differential processing by the differential processing unit 220). This is a component that is input as image data formed by arranging pixels having predetermined pixel values in the two-dimensional xy coordinate system, and executes step S11 “image input stage” in FIG. In practice, the image input here is developed on the memory of the computer.

  The line graph calculation unit 120 is a component that executes step S14 “line graph calculation stage” of FIG. 17, and represents an arbitrary straight line on the xy coordinate system, a variable ξ indicating the orientation with respect to the coordinate axis, and a variable indicating the displacement with respect to the origin. For each pixel constituting the image data input by the image input unit 110 using an expression format expressed using ρ, a combination of variables (ξ, ρ) for each of a group of straight lines passing through the reference position of the pixel is set. A process of obtaining the coordinate point corresponding to each variable pair (ξ, ρ) on the two-dimensional ξρ coordinate system and obtaining a line graph μ composed of a set of these coordinate points on the ξρ coordinate system for each pixel. I do.

  In the case of the embodiment described here, the line graph calculation unit 120 drops a perpendicular line from the origin O of the xy coordinate system to an arbitrary straight line La on the xy coordinate system, as shown in FIG. , The angle between one axis x of the xy coordinate system and the line segment OF is denoted by ξ, the length of the line segment OF is denoted by ρ, and the expression form expressing the straight line La by two variables ξ and ρ Used.

  The pixel value integration unit 130 is a component that executes Step S12 “Preparation for Pixel Value Integration Stage” and Step S15 “Pixel Value Integration Stage” in FIG. That is, a two-dimensional pixel array for integration in which each pixel has the same initial pixel value (for example, initial pixel value 0) is defined on the ξρ coordinate system, and the image input unit 110 is placed on the two-dimensional pixel array for integration. Is overlapped with the line graph μi obtained by the line graph calculation unit 120 for the i-th pixel Pi constituting the image data input on the xy coordinates, and the pixel value of the pixel located on or near the line graph μi is overlapped. Then, a process of integrating the pixel values of the pixel Pi is executed. Actually, the two-dimensional pixel array for integration is developed on the memory of the computer, and the integrated value is stored on the memory.

  Note that, as described in §8 (3), when the pixel value is integrated by the pixel value integrating unit 130, the pixel value of the pixel Pi is integrated with the pixel value of the pixel located on the line graph μi, and the line The pixel value of a pixel that is separated from the graph μi by a predetermined neighborhood distance d takes a predetermined ratio R (0 ≦ R ≦ 1) corresponding to the neighborhood value d of the pixel value of the pixel Pi, and the neighborhood distance d is large. A value obtained by multiplying a value that becomes smaller may be integrated. At this time, the predetermined ratio R may be set so as to have a Gaussian distribution centered on the position of the line graph μ.

  The reference frame position prediction unit 160 is a component that executes Step S18 “reference frame position prediction stage” of FIG. That is, with respect to the image input by the image input unit 110, the image of the reference frame that will appear on the captured image arranged on the xy coordinate system based on the geometric imaging condition given from the imaging condition setting unit 210. The process of predicting the position of is performed by geometric calculation. Specifically, as a geometric imaging condition for obtaining a captured image using the imaging system of FIG. 6, an angle θ formed between the normal N set at the reference point S and the optical axis of the camera 60, this light The angle φ formed by the projected image of the axis onto the “plane on which the reference frame is drawn” and the predetermined reference line ζ included in the “plane on which the reference frame is drawn” passing through the reference point S, the camera 60 and the reference point S Is given as a geometric imaging condition from the imaging condition setting unit 210, and a process of predicting the position of the image of the reference frame is executed by geometric calculation.

  The actually measured predicted point determination unit 140 is a component that executes steps S20 and S21 “actually predicted point determination stage” of FIG. That is, with respect to combinations of variables (ξ, ρ) indicating the straight lines constituting the image of the reference frame predicted by the reference frame position prediction unit 160, the corresponding coordinate points on the integrating two-dimensional pixel array are geometrically determined. Plotted as a target prediction point ε, and the reference position of the pixel having the maximum or minimum pixel value among the pixels belonging to the predetermined neighborhood region for each geometric prediction point ε is measured on the corresponding straight line. And decide. In FIG. 24, when the coordinate value of the geometric prediction point ε is (ξε, ρε) as a predetermined neighborhood region with respect to the geometric prediction point ε, a predetermined deviation Δξ, Δρ is used, and ξε An example using an area indicated by coordinate values (ξ, ρ) satisfying the conditions of −Δξ ≦ ξ ≦ ξε + Δξ and ρε−Δρ ≦ ρ ≦ ρε + Δρ is shown.

  In the actual measurement prediction point determination unit 140, as described in §8 (4), the pixel having the maximum or minimum pixel value among the pixels belonging to the predetermined neighborhood region with respect to the geometric prediction point ε. When there are a plurality of pixels, the barycentric position of the reference position of the plurality of pixels is determined as an actually predicted point for the corresponding straight line.

  The regular conversion unit 150 is a component that executes Step S24 “regular conversion stage” of FIG. That is, on the two-dimensional xy coordinate system, straight lines represented by the coordinate values (ξ, ρ) of the actually measured predicted points are drawn, the positions of these straight lines are handled as the positions of the correct reference frame images, A process of converting a captured image obtained from the camera 60 into a regular image by calculation is performed. As described above, such regular transformation can be performed by a geometric operation based on the positions of the four vertices of a rectangle constituting the image of the reference frame. The regular image obtained by the conversion is given to the reflection characteristic data creation unit 230.

  In theory, the line graph calculation processing by the line graph calculation unit 120 and the pixel value integration processing by the pixel value integration unit 130 are all N pixels of the image on the xy coordinates input by the image input unit 110. However, in practice, as described in §8 (5), among all N pixels, only pixels having pixel values within a predetermined range, or all N pixels If it is executed only for the pixels located within the predetermined consideration area, the calculation burden can be reduced.

  Further, as described in §8 (6), the actually measured prediction point determination unit 140 determines that the maximum pixel value in the vicinity region is less than the predetermined threshold value, or the minimum pixel value has the predetermined threshold value. In the case of exceeding, it is preferable to provide a function of presenting a message that the image conversion to the regular image has failed without determining the predicted point in actual measurement. In this case, specifically, an error signal indicating that the image conversion to the regular image has failed is output from the image conversion apparatus 100 to the reflection characteristic data creation unit 230. If there is a regular image that is not output from the image conversion apparatus 100 because the error signal is output, the reflection characteristic data creation unit 230 uses the method described in §8 (6) to obtain the shooting condition. It is sufficient to provide a function of performing interpolation based on a regular image for another object photographed image that approximates.

It is a perspective view which shows the concept of the general rendering process using texture data. It is a perspective view for demonstrating the principle of a BRDF model and a BTF model. It is a top view which shows the structure of 6-dimensional texture data used with a BTF model. It is a front view which shows an example of the discrete incident angle definition in a BTF model. It is a front view which shows an example of the discrete azimuth | direction angle definition in a BTF model. It is a perspective view which shows the structure of the imaging | photography system for producing the 6-dimensional texture data used with a BTF model. It is a top view which shows some examples of the picked-up image image | photographed with the imaging | photography system shown in FIG. It is a top view which shows the regular image obtained by performing image conversion with respect to the picked-up image shown in FIG. It is a top view which shows an example of the process which converts a picked-up image into a regular image based on a geometric method. It is a top view which shows an example of the imaging | photography stage 30 used in order to utilize the image conversion which concerns on this invention. It is a top view which shows an example of the picked-up image acquired by imaging | photography using the imaging | photography stage 30 shown in FIG. It is a top view which shows the relationship between xy coordinate system and ξρ coordinate system for Hough transform. It is a top view which shows the correspondence of each straight line on xy coordinate system, and each point on the ξρ coordinate system for Hough transform. It is a top view which shows the correspondence of the straight line group which passes along 1 point P on an xy coordinate system, and the line graph micro on the ξρ coordinate system for Hough transform. It is a top view which shows the correspondence of the straight line L which passes through several points P1-P4 on xy coordinate system, and the intersection (gamma) of line graphs μ1-μ4 on the Hough transform ξρ coordinate system. It is a flowchart which shows the basic procedure of the preparation method of the reflection characteristic data of the object based on this invention. It is a flowchart which shows the basic procedure of the image conversion method (step S4 stage of the flowchart of FIG. 16) to the regular image which concerns on this invention. FIG. 18 is a plan view showing a part of a pixel array on an xy coordinate constituting the image data input in step S11 of FIG. FIG. 19 is a plan view showing a line graph μi on the ξρ coordinate corresponding to the i-th pixel Pi on the xy coordinate shown in FIG. 18. FIG. 20 is a plan view showing pixels (hatched portions) located on a line graph μi on the ξρ coordinate shown in FIG. 19. FIG. 19 is a plan view showing pixels (hatched portions) located on a line graph μ (i + 1) on the ξρ coordinate corresponding to the (i + 1) th pixel P (i + 1) shown in FIG. 18. It is a top view which shows the pixel (gamma) which has the largest pixel value calculated | required after performing the integration process of a pixel value about each pixel on (xi) ρ coordinate. FIG. 23 is a plan view showing a straight line L drawn on xy coordinates corresponding to the position (ξγ, ργ) of the pixel γ shown in FIG. 22. It is a top view which shows a mode that the pixel (gamma) 3 which has the largest pixel value among the pixels which belong to the vicinity area | region about the geometric prediction point (epsilon) is determined after performing the integration process of each pixel value on (xi) coordinate. is there. FIG. 18 is a plan view showing a part of the pixel array on the xy coordinates constituting the image data when the image data input in step S11 of FIG. 17 is a gradation image. FIG. 6 is a diagram illustrating processing for extracting a reference frame 35 formed by a boundary line when a rectangular region in which a contrast difference occurs between the inside and the outside is formed on the upper surface of the imaging stage 30. It is a top view explaining the modification which accumulate | stores a predetermined pixel value also about the pixel Td of the vicinity of the line graph microi on (xi) coordinate. It is a graph which shows the Gaussian distribution curve used when determining the pixel value which should be integrated | accumulated to the pixel Td shown in FIG. It is a top view which shows the interpolation method when the image conversion to a regular image fails. It is a block diagram which shows the structure of the preparation apparatus of the reflection characteristic data of the object which concerns on this invention.

Explanation of symbols

10: Object data (3D virtual object)
15: Two-dimensional projection image 20: Texture data (original texture data)
30: Shooting stage 35: Reference frames 35A to 35D: Four sides of reference frame 38: Rectangular area 40: Subject (object)
50: Light source 51: Representative point 60: Camera 100: Image conversion device 110: Image input unit 120: Line graph calculation unit 130: Pixel value integration unit 140: Predicted point determination unit 150: Regular conversion unit 160: Reference frame position Prediction unit 210: photographing condition setting unit 220: differentiation processing unit 230: reflection characteristic data creation unit A to D: four vertices of rectangle d: neighborhood distance (distance between line graph μ and reference position of pixel T)
E: View point F: Vertical foot G: Light source H: Projection plane I (L): Illumination light intensity I (V): Reflected light intensity K: Reflection coefficient K (θL, φL, θV, φV): Reflection coefficient K (θL, φL, θV, φV, u, v): reflection coefficient L: illumination light / straight line L1 to L6 on the xy coordinate: illumination light La to Lc: straight line L ′ on the xy coordinate: projected image of the illumination light Ls: reference illumination light Ls ′: projection image of reference illumination light N: normal O: origin of xy coordinate system P: pixels on projection plane / points P1 to P4, Pi, P (i + 1), PN on xy coordinates : Point on the xy coordinate / pixel Q on the xy coordinate Q: sample point / one point R on the image R: ratio of multiplying the pixel value S: reference points S1 to S24: each step Td in the flowchart: pixel T (on the ξρ coordinate u, v): pixels u, v on the uv coordinates, each coordinate axis V of the two-dimensional coordinate system defining the texture V: reflected light V ': projected reflected light Vs: Reference reflected light Vs ′: Projected image of reference reflected light W: Reference plane / surface to be measured X, Y, Z: Each coordinate axis α, β: Three-dimensional coordinate system on the projection plane H Coordinate axis γ: intersection of a plurality of line graphs / pixels γ1, γ2, and γ3 having maximum pixel values: pixels θL on the ξρ coordinate: incident angles θL1 to θL6 of illumination light L: incident angles θLs of illumination light: reference illumination light Ls Incident angle θV: reflection angle θVs of reflected light V: reflection angles ζ, ζs of reference reflected light Vs: reference lines λa, λb, λc: points on coordinate Φρ: azimuth angles φL1 to φL12 of illumination light L: illumination Azimuth angle φLs of light L: Azimuth angle φV of reference illumination light Ls: Azimuth angle φVs of reflected light V: Azimuth angles of reference reflected light Vs μ, μ1 to μ4, μi, μ (i + 1): Line graph on ξρ coordinate ξ, ρ: Each coordinate axis of the coordinate system for Hough transformation ξa: Angle formed by the x axis and the perpendicular OF ρa: Length of the perpendicular OF ξγ: X axis Coordinate values on the angle / xi] axis of the perpendicular OF ργ: coordinate value on the length / [rho axis of a perpendicular OF ξε: ξ coordinate values on the coordinate axis Roipushiron: coordinates Δξ on [rho axes, [Delta] [rho]: deviation

Claims (28)

A photographed image obtained by photographing a subject placed inside a reference frame composed of straight lines drawn on a plane from a predetermined direction is viewed from a normal direction set up on a reference point on the plane. An image conversion method for converting to a regular image,
An image input step of inputting the captured image or an image obtained by performing a predetermined processing on the captured image as image data obtained by arranging pixels having a predetermined pixel value in a two-dimensional xy coordinate system;
Using an expression form that represents an arbitrary straight line on the xy coordinate system using a variable ξ indicating an orientation with respect to a coordinate axis and a variable ρ indicating a displacement with respect to the origin, a reference of the pixel for each pixel constituting the image data A combination of variables (ξ, ρ) is obtained for each of the straight line groups passing through the position, and coordinate points corresponding to each variable pair (ξ, ρ) are plotted on a two-dimensional ξρ coordinate system. A line graph calculation stage for obtaining a line graph μ constituted by the ξρ coordinate system for each pixel;
An accumulation two-dimensional pixel array in which each pixel has the same initial pixel value is defined on the ξρ coordinate system, and the i-th pixel Pi constituting the image data is formed on the accumulation two-dimensional pixel array. A process of superimposing the obtained line graph μi and adding the pixel value of the pixel Pi to the pixel value of a pixel located on or near the line graph μi is performed from the first pixel to the Nth pixel. A pixel value integrating step that is repeatedly executed up to pixels (where 1 ≦ i ≦ N, N is the total number of pixels on the pixel array included in the image data);
Based on the geometric imaging conditions at the time of obtaining the captured image, the position of the image of the reference frame that will appear on the captured image arranged on the xy coordinate system is predicted by geometric calculation. A reference frame position prediction stage to perform,
For the combinations of variables (ξ, ρ) indicating the respective straight lines constituting the predicted image of the reference frame, the corresponding coordinate points on the integrating two-dimensional pixel array are plotted as geometric prediction points ε, respectively. , The actual prediction point that determines the reference position of the pixel having the maximum or minimum pixel value among the pixels belonging to the predetermined neighborhood region for each geometric prediction point ε as the actual prediction point for the corresponding straight line The decision stage;
On the two-dimensional xy coordinate system, straight lines represented by the coordinate values (ξ, ρ) of the actually measured predicted points are drawn, the positions of these straight lines are handled as the positions of the correct reference frame images, A regular conversion step of converting the captured image into the regular image by calculation;
A method for converting an image into a regular image. The image conversion method according to claim 1,
When a shot image is obtained by shooting using a reference frame consisting of a boundary line due to a contrast difference between the inside and outside of a predetermined area,
A method for converting an image into a regular image, wherein a differential image obtained by subjecting a captured image to differential processing is input as image data in an image input stage. The image conversion method according to claim 1 or 2,
In the line graph calculation stage, when a perpendicular is drawn from the origin O of the xy coordinate system to an arbitrary straight line on the xy coordinate system and the foot is set as a point F, one axis of the xy coordinate system and the line segment OF A method of converting an image into a regular image, in which the angle formed by ξ is represented by ξ, the length of the line segment OF is represented by ρ, and an expression form representing the straight line by two variables ξ and ρ is used. In the image conversion method in any one of Claims 1-3,
In the pixel value integration stage, the pixel value of the pixel Pi is integrated with the pixel value of the pixel located on the line graph μ, and the pixel Pi is set to the pixel value of the pixel separated from the line graph μ by a predetermined neighborhood distance d. A regular image obtained by multiplying the pixel value of the pixel value by a predetermined ratio R corresponding to the neighborhood distance d (a value that takes a range of 0 ≦ R ≦ 1 and decreases as the neighborhood distance d increases). Image conversion method. The image conversion method according to claim 4, wherein
A method for converting an image into a regular image, wherein the predetermined ratio R is set so as to have a Gaussian distribution centered on the position of the line graph μ. In the image conversion method according to any one of claims 1 to 5,
In the line graph calculation stage, out of all N pixels, an operation for obtaining a line graph is performed only for pixels having pixel values within a predetermined range,
A method for converting an image into a regular image, wherein pixel value integration processing is executed only for pixels having pixel values within a predetermined range among all N pixels in the pixel value integration stage. In the image conversion method in any one of Claims 1-6,
In the line graph calculation stage, out of all N pixels, an operation is performed to obtain a line graph only for pixels located within a predetermined consideration region,
A method for converting an image into a regular image, wherein pixel value integration processing is executed only for pixels located within a predetermined consideration area among all N pixels in the pixel value integration stage. In the image conversion method in any one of Claims 1-7,
In the reference frame position prediction stage, as a geometric imaging condition for obtaining a captured image, an angle θ formed between a normal line set at a reference point and the optical axis of the imaging apparatus, and the “reference frame drawn by the optical axis” The angle φ formed between the projected image on the `` planar plane '' and the predetermined reference line ζ included in the `` plane on which the reference frame is drawn '' passing through the reference point, and the focal length of the optical system of the photographing apparatus, A method for converting an image into a regular image, wherein the position of the image of the reference frame is predicted by a geometric calculation based on the method. In the image conversion method in any one of Claims 1-8,
Predetermined deviations Δξ and Δρ are used when the coordinate value of the geometric prediction point ε is (ξε, ρε) as a predetermined neighborhood region for the geometric prediction point ε at the actual prediction point determination stage. Thus, an image conversion method for converting to a regular image is characterized in that an area indicated by coordinate values (ξ, ρ) satisfying the conditions of ξε−Δξ ≦ ξ ≦ ξε + Δξ and ρε−Δρ ≦ ρ ≦ ρε + Δρ is set. In the image conversion method in any one of Claims 1-9,
When there are a plurality of pixels having the maximum or minimum pixel value among the pixels belonging to the predetermined neighboring region with respect to the geometric prediction point ε at the actual measurement prediction point determination stage, the reference positions of the plurality of pixels A method of converting an image into a regular image, wherein the position of the center of gravity is determined as an actually estimated point for the corresponding straight line. In the image conversion method in any one of Claims 1-10,
If the maximum pixel value in the vicinity region is less than the predetermined threshold value or the minimum pixel value exceeds the predetermined threshold value in the actual measurement prediction point determination stage, the actual measurement prediction point is determined. A method for converting an image into a regular image, characterized in that the image conversion into a regular image is not performed. In the image conversion method in any one of Claims 1-11,
When a shot image is obtained by shooting using a reference frame indicating a rectangle,
A method for converting an image into a regular image, wherein the image is converted into a regular image by a geometric operation based on the positions of four vertices of a rectangle constituting the image of the reference frame in the regular conversion stage. By using the image conversion method according to claim 1,
When illumination light is applied from a predetermined incident direction to a predetermined position of the surface to be measured of the object, the reflectance of reflected light from the predetermined position toward the predetermined emission direction is determined for each position of the surface to be measured. Creation of reflection characteristic data of an object in which data defined for (m × n) cases by a combination of a plurality of m incidence directions and a plurality of n emission directions are created as reflection characteristic data for the surface to be measured. A method,
An object placement stage for placing an object having a measured surface inside a reference frame composed of straight lines drawn on a plane;
An imaging condition in which a light source, the surface to be measured, and the camera have a predetermined positional relationship is changed in a plurality of m in a positional relationship between the light source and the surface to be measured, and the surface to be measured and the camera And a shooting condition setting stage for setting a total (m × n) as a result of changing the positional relationship to n in plural ways,
For each of the set (m × n) shooting conditions, (m × n) (m × n) sheets are obtained by executing a shooting process of shooting an image of the measurement target surface illuminated by the light source with the camera. An object shooting stage to obtain an object shooting image;
Image conversion using the image conversion method according to any one of claims 1 to 12 for the (m × n) object photographed images to obtain (m × n) regular images Stages,
A reflection characteristic data creating step of creating reflection characteristic data of an object based on the (m × n) regular images;
A method for creating reflection characteristic data of an object characterized by comprising: The method of creating reflection characteristic data according to claim 13,
If the maximum pixel value in the neighboring region is less than the predetermined threshold value or the minimum pixel value is the predetermined threshold value in the measurement prediction point determination step in the image conversion step that converts the specific object photographed image into a regular image If it exceeds, do not determine the prediction point in actual measurement, handle the failed image conversion to a regular image related to the object captured image,
Creation of reflection characteristic data of an object characterized by obtaining a regular image of an object photographed image that has been handled with failed image conversion by interpolation based on a regular image of another object photographed image that approximates the photographing condition Method. A photographed image obtained by photographing a subject placed inside a reference frame composed of straight lines drawn on a plane from a predetermined direction is viewed from a normal direction set up on a reference point on the plane. An image conversion device for converting to a regular image,
An image input unit that inputs the captured image or an image obtained by performing a predetermined processing on the captured image as image data in which pixels having a predetermined pixel value are arranged in a two-dimensional xy coordinate system;
Using an expression form that represents an arbitrary straight line on the xy coordinate system using a variable ξ indicating an orientation with respect to a coordinate axis and a variable ρ indicating a displacement with respect to the origin, a reference of the pixel for each pixel constituting the image data A combination of variables (ξ, ρ) is obtained for each of the straight line groups passing through the position, and coordinate points corresponding to each variable pair (ξ, ρ) are plotted on a two-dimensional ξρ coordinate system. A line graph calculation unit for obtaining a line graph μ configured by the above-mentioned ξρ coordinate system for each pixel;
An accumulation two-dimensional pixel array in which each pixel has the same initial pixel value is defined on the ξρ coordinate system, and the i-th pixel Pi constituting the image data is formed on the accumulation two-dimensional pixel array. A process of superimposing the obtained line graph μi and adding the pixel value of the pixel Pi to the pixel value of a pixel located on or near the line graph μi is performed from the first pixel to the Nth pixel. A pixel value integrating unit that repeatedly executes up to pixels (where 1 ≦ i ≦ N, N is the total number of pixels on the pixel array included in the image data);
Based on the geometric imaging conditions at the time of obtaining the captured image, the position of the image of the reference frame that will appear on the captured image arranged on the xy coordinate system is predicted by geometric calculation. A reference frame position prediction unit to perform,
For the combinations of variables (ξ, ρ) indicating the respective straight lines constituting the predicted image of the reference frame, the corresponding coordinate points on the integrating two-dimensional pixel array are plotted as geometric prediction points ε, respectively. , The actual prediction point that determines the reference position of the pixel having the maximum or minimum pixel value among the pixels belonging to the predetermined neighborhood region for each geometric prediction point ε as the actual prediction point for the corresponding straight line A decision unit;
On the two-dimensional xy coordinate system, straight lines represented by the coordinate values (ξ, ρ) of the actually measured predicted points are drawn, the positions of these straight lines are handled as the positions of the correct reference frame images, A regular conversion unit that converts the captured image into the regular image by calculation;
A device for converting an image into a regular image, comprising: The image conversion device according to claim 15, wherein
When the line graph calculation unit draws a perpendicular line from the origin O of the xy coordinate system to an arbitrary straight line on the xy coordinate system and sets the foot as a point F, one axis of the xy coordinate system and the line segment OF An image conversion apparatus for converting a regular image into a regular image, in which an angle formed by ξ is represented by ξ, a length of the line segment OF is represented by ρ, and the straight line is represented by two variables ξ and ρ. The image conversion apparatus according to claim 15 or 16,
The pixel value accumulating unit accumulates the pixel value of the pixel Pi on the pixel value of the pixel located on the line graph μ, and the pixel Pi on the pixel value of the pixel separated from the line graph μ by a predetermined neighborhood distance d. A regular image obtained by multiplying the pixel value of the pixel value by a predetermined ratio R corresponding to the neighborhood distance d (a value that takes a range of 0 ≦ R ≦ 1 and decreases as the neighborhood distance d increases). To image converter. The image conversion apparatus according to claim 17, wherein
A device for converting an image into a regular image, wherein the predetermined ratio R is set so as to have a Gaussian distribution centered on the position of the line graph μ. The image conversion device according to any one of claims 15 to 18,
The line graph calculation unit performs a calculation for obtaining a line graph only for pixels having pixel values within a predetermined range among all N pixels,
An image conversion device for a regular image, wherein the pixel value integration unit executes pixel value integration processing only for pixels having pixel values within a predetermined range among all N pixels. The image conversion apparatus according to any one of claims 15 to 19,
The line graph calculation unit performs a calculation for obtaining a line graph only for pixels located within a predetermined consideration area among all N pixels,
An image conversion apparatus for a regular image, wherein the pixel value integration unit executes pixel value integration processing only for pixels located within a predetermined consideration region among all N pixels. The image conversion device according to any one of claims 15 to 20,
The reference frame position prediction unit gives an angle θ between the normal line set at the reference point and the optical axis of the imaging apparatus, which is given as a geometric imaging condition for obtaining a captured image, and the “reference of the optical axis” The angle φ formed between the projection image on the “plane on which the frame is drawn” and the predetermined reference line ζ included in the “plane on which the reference frame is drawn” passing through the reference point, and the focal point of the optical system of the photographing apparatus An image conversion apparatus for a regular image, wherein the position of an image of a reference frame is predicted by a geometric calculation based on the distance. The image conversion device according to any one of claims 15 to 21,
When the predicted point determination unit in actual measurement uses the coordinate values of the geometric prediction point ε as (ξε, ρε) as the predetermined neighborhood region for the geometric prediction point ε, the predetermined deviations Δξ and Δρ are used. Thus, an image conversion device to a regular image, characterized in that a region indicated by coordinate values (ξ, ρ) satisfying the conditions of ξε−Δξ ≦ ξ ≦ ξε + Δξ and ρε−Δρ ≦ ρ ≦ ρε + Δρ is used. The image conversion apparatus according to any one of claims 15 to 22,
In a case where the actually measured prediction point determination unit includes a plurality of pixels having the maximum or minimum pixel value among the pixels belonging to the predetermined neighboring region with respect to the geometric prediction point ε, the reference position of the plurality of pixels A device for converting an image into a regular image, wherein the center of gravity position is determined as an actually measured predicted point for the corresponding straight line. The image conversion apparatus according to any one of claims 15 to 23,
The actually measured prediction point determination unit determines the actually measured prediction point when the maximum pixel value in the vicinity region is less than the predetermined threshold value or when the minimum pixel value exceeds the predetermined threshold value. A device for converting an image into a regular image, characterized in that a message indicating that the image conversion into the regular image has failed is presented. An image conversion apparatus according to any one of claims 15 to 24,
When illumination light is applied from a predetermined incident direction to a predetermined position of the surface to be measured of the object, the reflectance of reflected light from the predetermined position toward the predetermined emission direction is determined for each position of the surface to be measured. Creation of reflection characteristic data of an object in which data defined for (m × n) cases by a combination of a plurality of m incidence directions and a plurality of n emission directions are created as reflection characteristic data for the surface to be measured. A device, further,
A shooting stage in which the upper surface forms a plane and a reference frame composed of straight lines is drawn on the plane;
A light source for illuminating an object placed on the shooting stage;
A camera for photographing an object placed on the photographing stage;
Shooting conditions for setting shooting conditions by changing the positional relationship between the light source and the surface to be measured of the object in a plurality of m and changing the positional relationship between the surface to be measured and the camera in a plurality of n ways A setting section;
Comprising a photographed image photographed by the camera and provided to an image input unit of the image conversion device,
An apparatus for creating reflection property data of an object, further comprising a reflection property data creation unit that creates reflection property data of an object based on the regular image converted by the image conversion device. The reflection characteristic data creating apparatus according to claim 25,
A rectangle for placing an object is formed on the upper surface of the imaging stage, and the inner area and the outer area of the rectangle are configured to have a difference in contrast, and the reference frame is defined by the boundary line between the two areas. Configured,
A differential processing unit that performs differential processing on the captured image captured by the camera is further provided, and the image after the differential processing is configured to be provided to the image input unit,
An apparatus for generating reflection characteristic data of an object, wherein the regular conversion unit performs conversion into a regular image by a geometric calculation based on the positions of four vertices of a rectangle constituting the image of the reference frame. In the reflection characteristic data creation device according to claim 25 or 26,
When the predicted point determination unit in actual measurement determines that the maximum pixel value in the neighborhood region is less than the predetermined threshold value or the minimum pixel value exceeds the predetermined threshold value, Outputs an error signal indicating that image conversion has failed,
The reflection characteristic data creation unit obtains a regular image that cannot be obtained from the image conversion apparatus because the error signal is output by interpolation based on a regular image of another object photographed image that approximates the photographing condition. A device for creating reflection characteristic data of an object.   The program for making a computer perform the image conversion method in any one of Claims 1-12.