JP7088530B2 - 3D method and device for projecting measurement result-related information on the surface of an object to be measured - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Law Announced at the 22nd Intelligent Mechatronics Workshop held at the University of Yamanashi Kofu Campus on August 27, 2017.

Application of Article 30, Paragraph 2 of the Patent Law Presented at the 2017 Hokuriku Shinetsu Branch Academic Lecture Meeting held at the University of Toyama Gofuku Campus on November 25, 2017.

The present invention relates to a method and an apparatus for projecting onto the surface of a three-dimensional measurement target object.

Non-contact 3D shape measurement is in widespread demand in the manufacturing industry, medical care, civil engineering, and clothing fields. Especially for objects with large curved surfaces such as sheet metal processed products and stamped products, 3D measurement by pattern projection, which can measure 3D distribution without contact, is effective as a method for measuring and inspecting in a short time. Is.

At the actual inspection site, it takes a lot of labor for the worker to make a correspondence between the 3D measurement result and the actual surface. For example, when used for inspection of curved surface shape in a large sheet metal processing product, even if measurement data of three-dimensional shape is obtained and a defective part of the shape is detected, the defective part displayed on the monitor screen of the inspection device. However, it is necessary to recognize which part in the real thing. In particular, in the case of an object having few irregularities or other major features in shape, it becomes difficult to identify which part of the object the defective portion is. Therefore, there is a demand for a method that can identify the position of a defect on an object in a short time.

As a conventional method, an apparatus and a method shown in Patent Document 1 have been developed. This creates an optical system that aligns the optical axes of the measurement camera and the projection projector on a pixel-by-pixel basis, and projects an image showing the measurement results from the projection projector onto the surface of the target object. There is a one-to-one correspondence between the camera pixel being measured and the pixel of the projector for projection. Therefore, by projecting an image created by storing the brightness value and color information projected on the same pixel with respect to the measurement result obtained for each pixel of the camera from the projector for projection, the surface of the object can be dealt with. Measurement results and evaluation results can be projected for each point.

In Patent Document 2, three-dimensional measurement of a target three-dimensional object is performed by pattern projection using a camera and a projector, and geometric deformation processing of the projection pattern is performed based on the obtained point group data. The method of image projection is shown.

Further, in the technique of projection mapping for a three-dimensional object, a method of performing projection mapping not only for a fixed object such as a building but also for a dynamically changing object has been proposed. For example, Non-Patent Document 1 discloses a method of image projection in which the position and orientation of a target object are estimated based on the point cloud data obtained by three-dimensional measurement, and the projection pattern is obtained accordingly.

Japanese Unexamined Patent Publication No. 2001-066158 Japanese Unexamined Patent Publication No. 2015-173430

Takuya Takanohashi, Naoki Hashimoto, Mie Sato, A Study on Interactive Image Projection by Object Tracking Using 3D Point Cloud, Journal of the Institute of Image Information and Television Engineers, Vol. 69, No. 6, J213-J216 (2015).

However, the technique disclosed in Patent Document 1 requires a special optical system because it is necessary to align the optical axes of the camera for measurement and the projector for projection, which is expensive. In addition, fine adjustment is required, which is troublesome.

Further, in the technique disclosed in Patent Document 2, since the coordinate conversion process is performed after the projected coordinate conversion data is generated every time the three-dimensional measurement of the target object is performed, the target object changes its position or is deformed. If this happens, there is a problem that it takes time to perform the subsequent steps.
Further, the technique disclosed in Non-Patent Document 1 cannot be applied to an object having an arbitrary shape because the shape data of the target object needs to be registered in advance.

Therefore, an object of the present invention is to project the measurement result-related information onto the surface of the three-dimensional measurement target object according to the shape, position, and orientation of the measurement target object. It is to provide a method and a device capable of projecting to the correct position on the surface of an object.

The invention according to claim 1 of the present application is
A measurement projector that projects a grid pattern onto the surface of an object, a projection projector that projects brightness values and color information corresponding to the position of the surface of the object onto the object, and a grid pattern projected from the measurement projector. In a projection mapping method using a projection mapping device comprising a camera for photographing and fixing the positional relationship between the camera, the measurement projector, and the projection projector.
The first step of measuring the three-dimensional shape of the object using the measurement projector,
Based on the coordinate information obtained by measuring the three-dimensional shape of the object from the first step, the corresponding coordinates of the projection projector are stored corresponding to the coordinate information of the camera and the object. The second step to refer to the existing table and
In the second step, the third step of obtaining the coordinates of the pixels of the projection projector corresponding to the pixels of the camera, and
The fourth step of projecting the luminance value and the color information corresponding to the position of the surface of the object from the pixels of the projection projector obtained in the third step onto the surface of the object is included.
The table is created in advance based on the relationship between the phase of the grid pattern projected on the reference planes installed at a plurality of positions from each of the measurement projector and the projection projector and the positions of the reference planes . There,
The vertical direction of the reference plane is defined as the z-axis direction, and the horizontal and vertical directions are defined as the xy-axis direction. The reference plane can be translated in the z-axis direction, and the grid pattern for obtaining the x and y coordinates is fixed. The surface or the reference surface for displaying the pattern for obtaining the x and y coordinates is moved to a plurality of positions in the z-axis direction, and the reference surface is photographed at each of the plurality of positions.
For each coordinate of the captured image taken at each of the above positions,
(1) Obtain the x, y coordinates from the pattern for obtaining the x, y coordinates fixed on the reference plane, or the pattern for obtaining the x, y coordinates displayed on the reference plane.
(2) A grid pattern is projected from the measurement projector onto the reference plane to obtain the phase θ0 of the projected grid pattern.
(3) The grid patterns in the x-axis direction and the y-axis direction are projected from the projection projector onto the reference plane to obtain the phase values (Φip, Φjp) of the grids in the x-axis direction and the y-axis direction, and the phase values are used. Based on this, the coordinates (ip, jp) of the projected image of the projection projector are obtained.
In a three-dimensional table having pixels (ic, jc) and z-coordinates of the camera as elements, the table in which the coordinates (ip, jp) of the projected image of the projection projector are stored as element values is used.
It is a projection mapping method.
The invention according to claim 2 is
In the second step, the z coordinate and the camera coordinate obtained as the shape measurement result are input to the table created as in claim 1.
In the third step, the coordinates of the projected image of the corresponding projection projector are obtained by the second step.
In the fourth step, a projected image is generated by storing the value of the pattern to be projected on the pixels of the projected image having the coordinate values, and the projected image is projected onto the object.
The projection mapping method according to claim 1.
The invention according to claim 3 is
A measurement projector that projects a grid pattern onto the surface of an object, a projection projector that projects brightness values and color information corresponding to the position of the surface of the object onto the object, and a grid pattern projected from the measurement projector. In a projection mapping device equipped with a camera for photographing and fixing the positional relationship between the camera, the measurement projector, and the projection projector.
A three-dimensional shape measuring means for measuring the three-dimensional shape of the object using the measuring projector,
Based on the coordinate information obtained by measuring the three-dimensional shape of the object by the three-dimensional shape measuring means, the corresponding coordinates of the projection projector are stored corresponding to the coordinate information of the camera and the object. Reference means to refer to the table and
Pixel information acquisition means for obtaining the coordinates of the pixels of the projection projector corresponding to the pixels of the camera by the reference means.
A projection means for projecting a luminance value or color information corresponding to a position on the surface of the object from the pixels of the projection projector obtained by the pixel information acquisition means onto the surface of the object is provided.
The table is created in advance based on the relationship between the phase of the grid pattern projected on the reference planes installed at a plurality of positions from each of the measurement projector and the projection projector and the positions of the reference planes. There,
The vertical direction of the reference plane is defined as the z-axis direction, and the horizontal and vertical directions are defined as the xy-axis direction. The reference plane can be translated in the z-axis direction, and the grid pattern for obtaining the x and y coordinates is fixed. The surface or the reference surface for displaying the pattern for obtaining the x and y coordinates is moved to a plurality of positions in the z-axis direction, and the reference surface is photographed at each of the plurality of positions.
For each coordinate of the captured image taken at each of the above positions,
(1) Obtain the x, y coordinates from the pattern for obtaining the x, y coordinates fixed on the reference plane, or the pattern for obtaining the x, y coordinates displayed on the reference plane.
(2) A grid pattern is projected from the measurement projector onto the reference plane to obtain the phase θ0 of the projected grid pattern.
(3) The grid patterns in the x-axis direction and the y-axis direction are projected from the projection projector onto the reference plane to obtain the phase values (Φip, Φjp) of the grids in the x-axis direction and the y-axis direction, and the phase values are used. Based on this, the coordinates (ip, jp) of the projected image of the projection projector are obtained.
In a three-dimensional table having pixels (ic, jc) and z-coordinates of the camera as elements, the table in which the coordinates (ip, jp) of the projected image of the projection projector are stored as element values is used.
It is a projection mapping device .

According to the present invention, when the measurement result-related information is projected on the surface of the three-dimensional measurement target object, the measurement result-related information is transmitted to the surface of the three-dimensional measurement target object according to the shape, position, and orientation of the measurement target object. A method and device capable of projecting to the correct position can be provided.

It is a figure explaining the projection using the projector for measurement in the relationship between the point on the reference plane and the phase of the projection exercise. It is a figure explaining the projection using the projector for mapping in the relationship between the point on the reference plane and the phase of the projection exercise. It is a figure which shows the relationship between the photographing line L of 1 pixel, and 3D coordinates (x coordinate and y coordinate, z coordinate) on a reference plane. It is a figure explaining the creation of a conversion table. It is a figure which shows the relationship between the phase and the height obtained by the reference plane photography, and is the figure which shows the relationship between a phase and an x coordinate. It is a figure which shows the relationship between the phase and the height obtained by the reference plane photography, and is the figure which shows the relationship between a phase and a y coordinate. It is a figure which shows the relationship between the phase and the height obtained by the reference plane photography, and is the figure which shows the relationship between a phase and a z coordinate. It is a figure explaining the created phase-height relation table and its reference. It is a conceptual diagram explaining the conversion table of a phase and a 3D coordinate (x coordinate, y coordinate, z coordinate). It is a figure explaining the liquid crystal display used for the reference plane attached to a moving stage. It is a figure explaining the state of the projection of the grid from the projector for measurement to the reference plane, and the state of photography. It is a figure explaining the state of displaying a grid in the x direction on a reference plane, and taking a picture. It is a figure explaining the state of displaying a grid in the y direction on a reference plane, and taking a picture. It is a figure explaining the state (x direction) of the projection of the grid from the projector for mapping to the reference plane. It is a figure explaining the state (x direction) of the projection of the grid from the projector for mapping to the reference plane. It is a figure explaining the measurement image and the state of measurement. It is a figure explaining the image to be mapped and the state of measurement. It is a figure which shows the relationship between the coordinates (ic, jc) of a camera, and z coordinates. It is a figure which shows the relationship between the coordinates (ic, jc) of a 1-pixel camera and z-coordinate, and is the figure which explains the relationship between z-coordinate and projection pixel ip. It is a figure which shows the relationship between the coordinates (ic, jc) of a 1-pixel camera and z-coordinate, and is the figure which explains the relationship between z-coordinate and projection pixel jp. It is a phase connection image of the image projected in a grid by a mapping projector, and is the figure explaining the projection pixel ip. It is a phase connection image of the image projected in a grid by a mapping projector, and is the figure explaining the projected image jp. It is a figure explaining the flow of shape measurement. It is a figure which shows one configuration example of a measuring device. It is a figure which shows the arrangement of a projector, a camera, and an LED device. It is a figure explaining the state of projecting a grid pattern. It is a figure explaining how the measurement result is projected onto the target object. It is a figure explaining the state of projecting a grid pattern on the surface of the object to be measured from the projector for measurement. It is a figure explaining how the measurement result is projected onto the object to be measured.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
1. 1. Measurement principle Here, the measurement principle used for 3D measurement and 3D measurement mapping will be described. In 3D measurement mapping, the whole space tableization method is used, and even if the optical axis of the projector used for measurement and the projector for mapping measurement result-related information such as measurement results and evaluation results on an object are different, the phase is different. By referring to the table of 3D coordinates, the correspondence can be obtained without complicated calculation.

First, in 1.1, the shooting of the reference plane and the table of phase and coordinates will be described in the method of creating a full-space table for creating a table that converts the phase and three-dimensional coordinates. In 1.2, a method of creating a table using a liquid crystal display will be described. Finally, in 1.3, the conversion between the phase and the three-dimensional coordinates will be described.

Known techniques can be used for the phase shift method, the light source switching phase shift method, the measurement depth expansion method, and the overlap method. Here, the vertical direction of the reference plane used to create the table is defined as the z coordinate (height / depth) direction, and the horizontal and vertical direction is defined as the xy coordinate direction.

1.1 Total space table creation method 1.1.1 Imaging of reference plane The x-axis is the direction orthogonal to the cosine wave-shaped grid pattern to be projected. At this time, FIGS. 1A and 1B show the relationship between the point on the reference plane and the phase of the projection grid. FIG. 1A shows the case of projection using a projector for measurement, and FIG. 1B shows the case of projection using a projector for mapping.

The positional relationship between the camera, the measurement projector, and the mapping projector is fixed, and the parallax is arranged in the x-axis direction. The reference plane is installed perpendicular to the z-axis and can be translated in the z-axis direction. The grid pattern is projected from the projectors (measurement projector and mapping projector) onto the reference plane.

It is assumed that any one pixel of the camera captures a point on the straight line L shown in FIGS. 1A and 1B. Any one pixel described above will photograph points P ₀ , P ₁ , P ₂ ... P _N , respectively, according to the reference planes R ₀ , R ₁ , R ₂ ... _RN . The phase at each point can be obtained from the grid projected by the phase shift method. Specifically, the position of the reference plane is moved by Δz, and the phase distribution of the reference plane at each z is obtained. At this time, the height of the reference plane is z _i (i = 0, 1, 2, ..., N) from the lowest. Here, N + 1 is the number of times the reference plane is moved. The solid line drawn from the projectors (measurement projector and mapping projector) in the figure to the reference plane indicates a portion of the projection grid having a high luminance value.

FIG. 2 shows the relationship between the shooting line L of one pixel and the three-dimensional coordinates (x-coordinate, y-coordinate, z-coordinate) on the reference plane. For points P ₀ , P ₁ , P ₂ ... P _N on each reference plane photographed by one pixel, x-coordinates and y-coordinates are obtained from a two-dimensional pattern fixed to the reference plane. Obtain the z coordinate from the position of the reference plane. The two-dimensional pattern fixed to the reference plane needs a pattern in which the x-coordinate and the y-coordinate can be obtained from the pattern.

When a two-dimensional grid is used as a two-dimensional pattern, it is possible to obtain the x-coordinate and y-coordinate at each point by obtaining the phase values of the grid in the x and y directions, respectively, and then making a phase connection. can. By using a liquid crystal display with a light diffusing plate attached to the surface of the reference surface, the lattice pattern can be displayed on the reference surface, and the phase distribution is obtained from the captured image by the phase shift method. By doing so, the x-coordinate, y-coordinate, and z-coordinate with respect to the phase θ of the projection grid can be obtained for each pixel for each position of the reference plane.

1.1.2 Creating a table Next, create a conversion table between the z coordinate and the phase. The outline of the conversion table creation is shown in FIG. The phase at the z coordinate between the actually measured two points is linearly interpolated, and the table element is created so that the interval Δθ is kept constant at the interpolated place. The elements are numbered 0, 1, 2, ..., i ..., N from the smallest phase value, and each phase value is θ _i (i = 0, 1, 2, ..., N). Here, N + 1 is the total number of elements in the table to be created. There is a relation of the equation 1 between N and Δθ.

First, as shown in FIGS. 4A, 4B, and 4C, the x-coordinate x _P corresponding to the phase θ _P is the front and rear phases θ _i and θ _{i + 1} having the phase θ _P in between, and the corresponding x. It can be obtained by the equation 2 using the coordinates x _i and x _{i + 1} . Similarly, the y coordinate can be obtained by the equation 3 using y _i and y _{i + 1} , and the z coordinate can be obtained by the equation 4 using z _i and z _{i + 1} .

Here, θ = θ _P. If θ _{i + 1} <θ _i , add 2π to θ _{i + 1} if θ _i <θ ≤ 2π, and subtract 2π from θ _i if 0 ≤ θ <θ _{i + 1} . In, the number 2, the number 3, and the number 4 are used. In addition, a table of x-coordinate, y-coordinate, and z-coordinate corresponding to the phase θ at equal intervals is created in advance by using the equations 2, 3, and 4.

In FIGS. 4A, 4B, and 4C, θ _{N + 1} does not exist among the phases θ _N and θ _{N + 1} before and after the phase shown as K for the phase of the section. .. Therefore, the relationship between the phase and the three-dimensional coordinates is obtained by using the equations 2, 3, and 4 with i = N-1. That is, the calculation is performed using the values of the phase and coordinates on the reference plane immediately before. By doing so, three-dimensional coordinates can be obtained for all phases from 0 to 2π. Further, even when the phase connection of the projection grid is performed, it is possible to obtain the phase after the phase connection and the three-dimensional coordinates of the point by the same idea.

1.1.3 Reference of table When measuring the shape, refer to the created relationship table of phase and height, and obtain the height distribution from this. FIG. 5 shows a relationship table between the phase and the height in one created pixel. Here, the phase θ of the measurement target is obtained in the same manner as when the phase of the reference plane is obtained. As shown in Equation 5, the quotient n obtained by dividing θ by Δθ is the same as the number of the table element to be referred to, so that the z coordinate z _n in that pixel can be obtained. By doing this for all pixels, the height distribution of the measurement target can be calculated.

When creating the x-coordinate table, the relationship between the phase and the x-coordinate was converted from the relationship between the height and the x-coordinate obtained at the beginning by using the phase-z-coordinate relationship table. This eliminates the need to calculate the z-coordinate to calculate the x-coordinate. Also, for the y-coordinate, a table for obtaining the y-coordinate from the phase can be created by the same method.

Here, the method actually used for the conversion from the phase to the three-dimensional coordinates (x coordinate, y coordinate, z coordinate) will be described. FIG. 6 shows a conceptual diagram of a conversion table of phase and three-dimensional coordinates (x-coordinate, y-coordinate, z-coordinate) having m number of table elements in one pixel. The Index number in the figure is the value of the reference table element number n obtained from the equation 5. All of these are held in memory as an array in pixel units in their respective order, and when referencing, the result is obtained by reading the data stored in the nth place.

As described above, in the method according to the present invention, the height distribution of the object to be measured is calculated based on the phase value distribution of the reference plane obtained in advance. As a result, it is possible to avoid a calculation error due to the projection lattice not becoming a sine wave and an error due to the influence of the distortion aberration of the camera lens. Furthermore, since only the table reference processing is performed, it is not necessary to calculate the height for each pixel, and high-speed analysis becomes possible.

1.2 Method of creating a table using a liquid crystal display A method of drawing a two-dimensional grid pattern on a flat plate is also effective, but when a grid projection is performed by drawing a light and shade pattern on the reference plane, it is within the reference plane. The accuracy of the phase of the projection grid may vary depending on the position of.

Therefore, here, a reference plate will be described in which the phase values in the x-direction and the y-direction can be obtained with high accuracy by using a liquid crystal display, and the phase of the projection grid can also be obtained with high accuracy.

This reference plate can accurately obtain the x-coordinate and y-coordinate of each pixel by performing phase analysis by the phase shift method. Further, the accuracy of the phase of the projection grid does not vary depending on the position in the reference plane with respect to the projection grid. The relationship between the three-dimensional coordinates and the phase of the grid projected from the projector for measurement is calibrated using the whole space tableization method.

FIG. 7 shows a liquid crystal display used as a reference surface attached to the moving stage. A light diffuser is attached to the surface of the liquid crystal display, and the image displayed on the liquid crystal display is projected from the back onto the light diffuser by the illumination light from the backlight light source. The liquid crystal display is mounted on a moving stage that moves in the z direction so that the display surface is perpendicular to the z axis. The position of the moving stage is controlled by a moving stage control signal from a computer.

The procedure for acquiring table data is shown. FIG. 8 shows how the grid is projected from the projector for measurement. When photographing a grid projected from a projector for measurement, a black pattern is displayed on the liquid crystal panel as a reference plane display image. By doing so, the illumination light from the backlight source is blocked. A grid image is projected from the projector for measurement and photographed with a camera. The projection grid projected from the projector for measurement is projected on the surface of the light diffusing plate. Therefore, the surface of the light diffusing plate serves as a reference surface. Use the moving stage to move the position of the reference plane to z ₀ , z ₁ , z ₂ … z _N. Then, by performing the phase analysis of the projection grid at each position, it is possible to obtain the phases θ ₀ , θ ₁ , θ ₂ ... θ _N for each position for each pixel of the image captured by the camera.

Next, FIGS. 9A and 9B show how the grid images in the x-direction and the y-direction with respect to the reference plane are displayed, respectively. By inputting a black pattern as a projected image to the projectors for measurement and mapping, the projected light from both projectors is blocked.

A grid image in the x-direction or the y-direction is displayed on the liquid crystal panel as a reference plane display image. The illumination light emitted from the backlight source projects the lattice image displayed on the liquid crystal panel onto the light diffusing plate attached to the liquid crystal panel from the back surface. The projected image is transmitted while scattering inside the light diffusing plate, and is displayed on the surface of the light diffusing plate which is the reference surface. Therefore, the camera can capture the grid image displayed on the reference plane.

Use the moving stage to move the position of the reference plane to z ₀ , z ₁ , z ₂ … z _N. Then, at each position, the phase analysis of the grid in the x-direction and the y-direction is performed, and the phase connection is performed with respect to the phase. Furthermore, by converting the phase-connected phase to x-coordinate or y-coordinate, x-coordinate x ₀ , x ₁ , x ₂ … x _N and y for each position of the image taken by the camera. Coordinates y ₀ , y ₁ , y ₂ … y _N can be obtained.

FIGS. 10A and 10B show how the grids in the x-direction and the y-direction are projected from the projector for mapping, respectively. Similar to the projection using the projector for measurement shown in FIG. 8, when the grid projected from the projector for mapping is photographed, the black pattern is displayed as a reference plane display image on the liquid crystal panel. The position of the reference plane is moved to z ₀ , z ₁ , z ₂ … z _N using the moving stage, and the phase analysis of the projection grid in the x and y directions is performed at each position, and the image is taken by the camera. For each pixel of the image, the x-coordinate phases Φ _i0 , Φ _i1 , Φ _i2 … Φ _iN and the y-coordinate phases Φ _j0 , Φ _j1 , Φ _j2 … Φ _jN can be obtained. By doing so, the x-coordinate phase Φip, y-coordinate phase Φjp and three-dimensional coordinates (x-coordinate and y-coordinate, z) of the grid projected from the mapping projector at the position of each reference plane in each pixel. Coordinates) are obtained respectively.

The phases Φip and Φjp can be obtained only at the position of the reference plane, but the correspondence with the three-dimensional coordinates can be accurately obtained by reducing the distance between the reference planes as necessary. The phase values Φip and Φjp are converted into ip pixels and jp pixels of the coordinates for mapping, respectively. Assuming that the grid pitches in the x and y directions projected by the mapping projector are Pip and Pjp, the phase and coordinates can be converted by the equations 6 and 7.

1.3 Coordinate conversion When the image obtained by 3D measurement is projected by a projector for mapping, the optical axis is different, so the mapping shifts. In order to perform projection mapping, it is necessary to convert each measured 3D coordinate (x coordinate, y coordinate, z coordinate) into coordinates for projection from the projector. By the above calibration method, the relationship between the _{ip pixel and j p pixel} of the mapping coordinates and the z coordinate z ₀ , z ₁ , z ₂ … z _N can be obtained for each pixel.

FIG. 11 shows a measurement image and a state of measurement. The measurement image shows the three-dimensional measurement of the object in the figure. A certain coordinate value (x _a , y _a , z _a ) on the Object is stored in the coordinates (i _c , j _c ) of the measured image taken by the camera.

FIG. 12 shows how the coordinates of the measured image are converted into the coordinates for mapping. The Projection image in the figure represents the image projected by the projector for mapping. The coordinate values (x _a , y _a , z _a ) stored in the coordinates (i _c , j _c ) of the above measured image are stored in the coordinates (i _p , j _p ) of the mapping image.

In order to convert the measured 3D coordinates (x-coordinate, y-coordinate, z-coordinate) into the coordinates for projection from the projector, refer to the table created by the all-space tableization method. FIG. 13 shows the relationship between the pixels (i _c , j _c ) of the camera and the z coordinate.

The coordinates of the corresponding mapping image are stored for each pixel of the three-dimensional measurement image at the shooting position z of each reference plane. The coordinates for mapping (i _p , j _p ) can be obtained from the z coordinates acquired by the three-dimensional measurement and the coordinates (i _c , j _c ) of the measured image.

Three-dimensional coordinates are obtained for each pixel by the whole space tableization method. The relationship between the z coordinate in one pixel ( _{ic, j c )} in FIGS. 14A and 14B and the ip coordinate and the jp coordinate of the projection grid image of the projector for mapping is shown, respectively. Let z _a be the value of the z coordinate in the coordinates (i _c , j _c ). From the measured coordinates z _a , the ip coordinates and jp coordinates of the projection grid image of the projector for mapping taken at the position of z _a based on these tables can be obtained as (ipa, jpa). ..
In the above explanation, the case where the liquid crystal display is used as the reference plane is shown, but the calibration method using other means capable of reading the x and y coordinates can also be applied. For example, it can be applied to a flat plate as a reference plane on which a two-dimensional grid pattern, a dot pattern, a checker pattern, or the like is drawn.
Furthermore, even when x and y coordinate information cannot be obtained from the reference plane, this method uses the projection grid of the projector for mapping to the camera coordinates (i _c , j _c ) and z coordinates. Table the relationship between the ip coordinates and jp coordinates of the image. Therefore, it can be applied even in the case of a three-dimensional measuring device (height distribution measuring device) that does not obtain x and y coordinates. It can also be applied to a three-dimensional measuring device that simply calculates x and y coordinates from camera coordinates (i _c , j _c ).

15A and 15B show phase connection images of images projected in a grid by a mapping projector. Coordinate values for mapping are stored for each pixel in the phase connection image. The projection pixel i _pa corresponding to z _a can be obtained from FIG. 14A, and the projection pixel j _pa can be obtained from FIG. 14B.

A mapping image can be obtained by performing coordinate conversion on the pixels for which the z-coordinate of the measured image is obtained. The measurement result can be projected at the correct position on the object by projecting the mapping image obtained by putting the height distribution in each converted coordinate with the projector for mapping.

FIG. 16 is a diagram illustrating a flow from reference plane photography to table creation and object shape measurement. In the measurement procedure, first, a reference plane is placed in a range predicted to be measurable by simulation. At that position, the phase is obtained by the light source switching phase shift method. Each time the light source is switched, a picture is taken, and the phase value of each pixel is obtained from the obtained three or more images. This operation is repeated at each position where the reference plane is moved by Δz in the z direction. By repeating this n times until the reference plane reaches the position of z _n-1 , the relationship between the z coordinate and the phase of each pixel can be obtained. This is the calibration work. From the relationships obtained above, a transformation table is created by interpolating the relationship between the phase and the z coordinate. This table is temporarily stored in the memory inside the PC. This is the table creation work.

Next, the measurement object is placed in the range where the reference plane is photographed, and the light sources are turned on in order the same number of times as at the time of calibration by the light source switching phase shift method to perform imaging, and the phase distribution of the measurement object is calculated. The z-coordinate distribution of the measurement object is calculated by comparing the phase values of each pixel of this phase distribution with the table. This is the shape measurement work. As long as there is no change in the experimental environment, it is possible to perform measurement using the same table.
FIG. 17 is a diagram showing the configuration of the measuring device. FIG. 18 is a diagram showing an arrangement of a projector, a camera, and an LED device.

The three-dimensional measuring device shown in FIG. 17 includes a grid plate, a line LED device, a camera, and a personal computer. The line LED device is composed of eight lines of light sources arranged in a line. The line LED includes an LED lighting circuit and a drive circuit.

Four LED devices are used, and two sets are installed in two stages so that the distance of the light source changes with respect to the grid plate. This is to apply the measurement depth expansion method. The method for expanding the measurement depth is a known technical means.

A DLP type projector is incorporated in the three-dimensional measuring device. This projector is used for projection mapping onto an object. In addition, a photodiode is inserted inside the projector, and the output of the light source of the projector is sent to the personal computer.

As described above, when the projection mapping device according to the present invention projects the distribution of measurement results and evaluation results onto the surface of the object to be measured as the distribution of brightness values and color information, it depends on the shape of the object to be measured. By appropriately determining the position of the image projected by the projector, the measurement result and the evaluation result can be projected to the correct position on the surface of the object even if the shape of the object changes. Moreover, the processing can be performed in a very short time. In addition, by using near-infrared light or light with an inconspicuous wavelength for the projection for measurement, the operator does not have to see the projected light for three-dimensional measurement, or almost without worrying about it. Only the measurement results and evaluation results projected on the object can be seen. This enhances the workability of the worker.

Next, with reference to FIGS. 19 to 22, a three-dimensional shape measuring method using a plurality of grid projection devices (a plurality of projectors) and an embodiment of the device will be described. FIG. 19 shows how a grid pattern is projected from a measurement projector onto a measurement target object. FIG. 20 shows how the distribution of the evaluation results created based on the measurement results is projected from the projection projector. First, a grid image is projected from the measurement projector shown in FIG. 19 onto the measurement target object. The camera captures a grid image projected on the object to be measured. Next, the personal computer (PC) refers to the above-mentioned conversion table and performs processing so that the projected image is projected on the corresponding coordinates of the measurement target object measured by using the measurement projector.

Further, the method according to the present invention can construct a similar device even when the measurement projector and the projection projector are the same. The state of measurement and projection in that case is shown below. FIG. 21 shows how a grid pattern is projected from a measurement projector onto an object to be measured. FIG. 22 shows how the distribution of the evaluation results created based on the measurement results is projected as the distribution of the luminance value and the color information from the projection projector which is also used as the measurement projector. By controlling the projector for both measurement and projection by the personal computer PC, the projection mapping according to the present invention can be carried out.

As described above, the projection mapping apparatus according to the present invention immediately projects the measurement result and the distribution of the evaluation result generated based on the measurement result on the actual surface of the measurement object immediately after performing the three-dimensional measurement. be able to. That is, by repeatedly executing a series of processes, it is possible to project the measurement result and the distribution of the evaluation result generated based on the measurement result on the actual surface of the measurement object.

An image for projection can be created by the method shown above, but when the number of pixels captured by the camera is small, the number of data points mapped to the projected image is small, and there is no data in the image. Appears, and a missing point appears in the image projected on the surface of the target object.

This can be solved by (1) using the data value of a certain point of data in the vicinity and generating the data of that point by interpolation, and (2) for the image taken by the camera. This can be solved by means of performing a process of increasing the number of pixels of the image and generating a projected image using the obtained image having a large number of pixels. By combining (1) and (2), it is possible to generate a projected image in which the amount of gaps is reduced or eliminated.

As described above, according to the present invention, when the measurement result-related information is projected on the surface of the three-dimensional measurement target object, the measurement result-related information is measured in the three-dimensional manner according to the shape, position, and orientation of the measurement target object. A method and device capable of projecting to the correct position on the surface of an object can be provided.
Further, since the present invention obtains the coordinates of the corresponding pixels by referring to the table, it is possible to generate an image to be projected in a very short time.
Furthermore, the present invention uses near-infrared light for projection for measurement, so that the operator does not see the projected light for three-dimensional measurement, and the measurement result and evaluation result projected on the measurement object. Can only be seen. This enhances workability by workers and robots equipped with visual sensors.
Further, the present invention can project the distribution of the three-dimensional measurement result and the corresponding evaluation result on the corresponding real object in real time.

L shooting line PC personal computer

Claims (3)

A measurement projector that projects a grid pattern onto the surface of an object, a projection projector that projects brightness values and color information corresponding to the position of the surface of the object onto the object, and a grid pattern projected from the measurement projector. In a projection mapping method using a projection mapping device comprising a camera for photographing and fixing the positional relationship between the camera, the measurement projector, and the projection projector.
The first step of measuring the three-dimensional shape of the object using the measurement projector,
Based on the coordinate information obtained by measuring the three-dimensional shape of the object from the first step, the corresponding coordinates of the projection projector are stored corresponding to the coordinate information of the camera and the object. The second step to refer to the existing table and
In the second step, the third step of obtaining the coordinates of the pixels of the projection projector corresponding to the pixels of the camera, and
The fourth step of projecting the luminance value and the color information corresponding to the position of the surface of the object from the pixels of the projection projector obtained in the third step onto the surface of the object is included.
The table is created in advance based on the relationship between the phase of the grid pattern projected on the reference planes installed at a plurality of positions from each of the measurement projector and the projection projector and the positions of the reference planes . There,
The vertical direction of the reference plane is defined as the z-axis direction, and the horizontal and vertical directions are defined as the xy-axis direction. The reference plane can be translated in the z-axis direction, and the grid pattern for obtaining the x and y coordinates is fixed. The surface or the reference surface for displaying the pattern for obtaining the x and y coordinates is moved to a plurality of positions in the z-axis direction, and the reference surface is photographed at each of the plurality of positions.
For each coordinate of the captured image taken at each of the above positions,
(1) Obtain the x, y coordinates from the pattern for obtaining the x, y coordinates fixed on the reference plane, or the pattern for obtaining the x, y coordinates displayed on the reference plane.
(2) A grid pattern is projected from the measurement projector onto the reference plane to obtain the phase θ0 of the projected grid pattern.
(3) The grid patterns in the x-axis direction and the y-axis direction are projected from the projection projector onto the reference plane to obtain the phase values (Φip, Φjp) of the grids in the x-axis direction and the y-axis direction, and the phase values are used. Based on this, the coordinates (ip, jp) of the projected image of the projection projector are obtained.
In a three-dimensional table having pixels (ic, jc) and z-coordinates of the camera as elements, the table in which the coordinates (ip, jp) of the projected image of the projection projector are stored as element values is used.
Projection mapping method. In the second step, the z coordinate and the camera coordinate obtained as the shape measurement result are input to the table created as in claim 1.
In the third step, the coordinates of the projected image of the corresponding projection projector are obtained by the second step.
In the fourth step, a projected image is generated by storing the value of the pattern to be projected on the pixels of the projected image having the coordinate values, and the projected image is projected onto the object.
The projection mapping method according to claim 1. A measurement projector that projects a grid pattern onto the surface of an object, a projection projector that projects brightness values and color information corresponding to the position of the surface of the object onto the object, and a grid pattern projected from the measurement projector. In a projection mapping device equipped with a camera for photographing and fixing the positional relationship between the camera, the measurement projector, and the projection projector.
A three-dimensional shape measuring means for measuring the three-dimensional shape of the object using the measuring projector,
Based on the coordinate information obtained by measuring the three-dimensional shape of the object by the three-dimensional shape measuring means, the corresponding coordinates of the projection projector are stored corresponding to the coordinate information of the camera and the object. Reference means to refer to the table and
Pixel information acquisition means for obtaining the coordinates of the pixels of the projection projector corresponding to the pixels of the camera by the reference means.
A projection means for projecting a luminance value or color information corresponding to a position on the surface of the object from the pixels of the projection projector obtained by the pixel information acquisition means onto the surface of the object is provided.
The table is created in advance based on the relationship between the phase of the grid pattern projected on the reference planes installed at a plurality of positions from each of the measurement projector and the projection projector and the positions of the reference planes. There,
The vertical direction of the reference plane is defined as the z-axis direction, and the horizontal and vertical directions are defined as the xy-axis direction. The reference plane can be translated in the z-axis direction, and the grid pattern for obtaining the x and y coordinates is fixed. The surface or the reference surface for displaying the pattern for obtaining the x and y coordinates is moved to a plurality of positions in the z-axis direction, and the reference surface is photographed at each of the plurality of positions.
For each coordinate of the captured image taken at each of the above positions,
(1) Obtain the x, y coordinates from the pattern for obtaining the x, y coordinates fixed on the reference plane, or the pattern for obtaining the x, y coordinates displayed on the reference plane.
(2) A grid pattern is projected from the measurement projector onto the reference plane to obtain the phase θ0 of the projected grid pattern.
(3) The grid patterns in the x-axis direction and the y-axis direction are projected from the projection projector onto the reference plane to obtain the phase values (Φip, Φjp) of the grids in the x-axis direction and the y-axis direction, and the phase values are used. Based on this, the coordinates (ip, jp) of the projected image of the projection projector are obtained.
In a three-dimensional table having pixels (ic, jc) and z-coordinates of the camera as elements, the table in which the coordinates (ip, jp) of the projected image of the projection projector are stored as element values is used.
Projection mapping device.

Images