JP2022135340A - Three-dimensional measuring device, computer program, manufacturing system, and method for manufacturing article - Google Patents

Abstract

To provide a three-dimensional measuring device that can perform three-dimensional measurement with high accuracy with a simple configuration.SOLUTION: A three-dimensional measuring device has: a near infrared light illumination unit for illuminating a measurement object with near infrared light; an image pickup device in which adjacent pixels have different wavelength sensitivities to visible light and each pixel has sensitivity characteristics to the near infrared light; a color image signal generation unit that processes a first imaging signal obtained from the image pickup device with the measurement object not illuminated by the near infrared light illumination unit to generate a color image signal; and a three-dimensional data calculation unit that calculates three-dimensional data from a second imaging signal obtained from the image pickup device with the measurement object illuminated by using the near infrared light illumination unit.SELECTED DRAWING: Figure 1

Description

The present invention relates to a three-dimensional measuring device using near-infrared light and the like.

Machine vision is used in various fields, including automatic picking and inspection of parts on production lines, and AGVs (automated guided vehicles). It is mainly used for object recognition and image processing for inspection, and for that purpose a color image and three-dimensional data of the object to be measured are required.

Japanese Patent Application Laid-Open No. 2002-200002 discloses a three-dimensional measurement system capable of acquiring conventional color images and three-dimensional data. In Patent Literature 1, stereo matching is performed using a special imaging device to obtain a color image and three-dimensional data. Here, the special imaging device is an imaging device in which rows in which RGB light receiving elements are repeatedly arranged and rows in which light receiving elements that respond to the entire visible light are repeatedly arranged are alternately arranged in the column direction. By using such an image pickup device, it is possible to provide an image pickup device capable of acquiring color information that ensures distance measurement accuracy equal to or higher than that obtained when using an image pickup device of a monochrome camera.

In addition, in the 3D measurement system, four infrared cameras (left and right), infrared pattern lighting, and an RGB sensor are arranged. Some are acquiring images.

JP 2015-50494 A

However, in the three-dimensional measurement system described in Patent Document 1, there is a problem that the device becomes expensive because a special imaging device is used. In addition, since rows in which the RGB light receiving elements are repeatedly arranged and rows in which the light receiving elements that respond to the entire visible light are repeatedly arranged are alternately arranged in the column direction, the resolution in the column direction is degraded. There is

Further, in the three-dimensional measurement system in which the two infrared cameras on the left and right, the infrared pattern illumination, and the RGB sensor are arranged as described above, the two infrared cameras and the RGB sensor are separate sensors. Therefore, there is a problem that the three-dimensional data as the depth map and the optical axis of the color image are different, which complicates the subsequent image processing.
SUMMARY OF THE INVENTION It is an object of the present invention to provide a three-dimensional measuring apparatus or the like capable of solving the above-described problems and performing highly accurate three-dimensional measurement with a simple configuration.

In order to achieve the object, the three-dimensional measurement system of the present invention
a near-infrared light illumination unit for illuminating the measurement object with near-infrared light;
an imaging device in which adjacent pixels have different wavelength sensitivities to visible light and each pixel has a sensitivity characteristic to the near-infrared light;
a color image signal generation unit that processes a first imaging signal obtained from the imaging device in a state where the measurement object is not illuminated by the near-infrared light illumination unit to generate a color image signal;
a three-dimensional data calculation unit that calculates three-dimensional data from a second imaging signal obtained from the imaging element while the measurement object is illuminated using the near-infrared light illumination unit;
characterized by having

According to the present invention, it is possible to realize a three-dimensional measurement apparatus or the like capable of performing three-dimensional measurement with high accuracy with a simple configuration.

1 is a block diagram showing a configuration example of a three-dimensional measurement system according to a first embodiment; FIG. FIG. 4 is a diagram showing an example of an illumination pattern of a near-infrared pattern illumination unit 112; 4 is a diagram showing an example of a color filter pattern of an RGB sensor 122; FIG. FIG. 3 is a diagram showing spectral sensitivity characteristics of a general Bayer array RGB sensor; 4 is a flow chart for explaining the operation of the first embodiment; FIG. 10 is a diagram showing an example of a near-infrared light pattern that changes over time in Example 2; 9 is a flow chart for explaining the operation of the second embodiment; FIG. 4 is a diagram showing the concept of pattern matching between single images using sub-windows and the concept of pattern matching between multiple images using intensity information in the direction of the time axis; 10 is a flow chart for explaining the operation of the third embodiment; 14 is a flowchart for explaining the operation of Example 4; 14 is a flow chart for explaining the operation of the fifth embodiment; FIG. 11 is a block diagram showing the configuration of a three-dimensional measurement system according to a second embodiment; FIG. 1 is a conceptual diagram of a known random pattern ranging principle; FIG. FIG. 12 is a flowchart for explaining the operation of the sixth embodiment; FIG. FIG. 4 is a diagram showing an example of multiple line patterns in the spatial encoding method; FIG. 11 is a flow chart for explaining the operation of the seventh embodiment; FIG. FIG. 11 is a block diagram showing a configuration example of a three-dimensional measurement system according to a third embodiment; FIG. FIG. 13 is a flowchart for explaining the operation of the eighth embodiment; FIG. It is the figure which showed the three-dimensional measurement system of 4th Embodiment. FIG. 12 is a flow chart for explaining the operation of the ninth embodiment; FIG. It is a figure which shows the structural example of the manufacturing system of 5th Embodiment.

Preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings. There are four embodiments shown below. In the first embodiment, two elements, a compound eye camera and near-infrared pattern illumination, are the main components, and in the second embodiment, two elements, a monocular camera and near-infrared pattern illumination, are the main components. . In the third embodiment, two elements, a compound-eye camera and near-infrared uniform illumination, are main components, and in the fourth embodiment, three elements, a compound-eye camera, near-infrared pattern illumination, and near-infrared uniform illumination, are main components. Each embodiment is described in detail below.
<First Embodiment>

FIG. 1 is a block diagram showing the configuration of the three-dimensional measurement system of the first embodiment.
In the first embodiment, two elements, the compound eye camera and the near-infrared pattern illumination, are the main components. Five examples mainly having different imaging conditions will be described in detail with reference to FIG.

A three-dimensional measurement system 100 according to the first embodiment shown in FIG. 1 is a device for acquiring a color image and three-dimensional data of an object 101 to be measured. have. The projection unit 110 and the imaging unit 120 are arranged facing the measurement object 101 . The projection unit 110 has a white uniform illumination unit 111 and a near-infrared pattern illumination unit 112 . Here, the white uniform illumination unit 111 functions as a visible light illumination unit for illuminating the measurement object with visible light, and the near-infrared pattern illumination unit 112 illuminates the measurement object with near-infrared light. It functions as a near-infrared light illumination unit for

The two imaging units 120 each have an imaging lens 121 and an RGB sensor 122 and are arranged at symmetrical positions with the projection unit 110 interposed therebetween. When one of the two imaging units 120 is a right-eye camera and the other is a left-eye camera, a stereo matching method using parallax between the right-eye camera and the left-eye camera is used to calculate a depth map as three-dimensional data. Further, the calculation output unit 130 includes a CPU as a computer, a memory storing computer programs, and the like, and includes an image generation unit 131, a data storage unit 132, a color image output unit 133, a three-dimensional data calculation unit 134, and a three-dimensional data output unit. 135. Further, in this embodiment, it is assumed that there is ambient light 102 such as a fluorescent lamp or an LED.

Next, the near-infrared pattern illumination section 112 will be described. FIG. 2 is a diagram showing an example of an illumination pattern of the near-infrared pattern illumination section 112. As shown in FIG. In this embodiment, the near-infrared pattern illumination unit 112 illuminates the measurement object with a predetermined pattern of near-infrared light, and the illumination pattern is random illumination. Here, it is not necessary to know the illumination distribution of the pattern. Also, as a method of creating illumination, a projector may be used, a plurality of light sources arranged in an array may be used, and a CGH (computer-generated hologram) or DOE (diffractive optical element) may be used as a laser light source. Arranged ones may be used. Also, these lights may be used in combination. As for the wavelength, an illumination pattern with spectral characteristics of 780 nm to 1000 nm is irradiated.

Next, an RGB sensor as an imaging device will be described. FIG. 3 is a diagram showing an example of the color filter pattern of the RGB sensor 122. As shown in FIG. The color filter pattern shown in FIG. 3 is generally called a Bayer array. One of the R, G, and B color filters that transmit the visible light region is arranged above each pixel of the plurality of two-dimensionally arranged pixels that constitute the RGB sensor 122 . That is, as shown in FIG. 3, focusing on arbitrary 2×2=4 pixels, red (R) is arranged at a ratio of 1 pixel, green (G) at 2 pixels, and blue (B) at a ratio of 1 pixel. there is

Next, spectral characteristics of the RGB sensor 122 will be described. FIG. 4 is a diagram showing spectral sensitivity characteristics of an RGB sensor having a general Bayer array, where the horizontal axis represents wavelength (unit: nm) and the vertical axis represents transmittance. Also, the solid line is the spectral sensitivity characteristic of the red pixel, the dashed line is the green pixel, and the one-dot chain line is the blue pixel. In the wavelength range of 400 nm to 780 nm, the transmittance corresponds to each color, but it is known that the RGB transmittances are substantially the same in the wavelength range of 780 nm to 1000 nm. That is, the RGB sensor 122 is an imaging device in which adjacent pixels have different wavelength sensitivities to visible light, and each pixel has sensitivity characteristics to near-infrared light. Although it is desirable that the sensitivity characteristics to near-infrared light are substantially the same, they may be different. This is because the difference in sensitivity can be corrected by a coefficient and matched.
In the present embodiment, such sensitivity characteristics of the image sensor to near-infrared light are utilized.

Next, the imaging operation and the process of outputting a color image and three-dimensional data from the data obtained by each imaging will be described in detail with reference to FIG.
FIG. 5 is a flow chart for explaining the operation of the embodiment, and the processing of each step in FIG. 5 is performed by executing the computer program stored in the memory by the CPU in the calculation processing unit.

In Example 1, imaging is performed twice. The first imaging (first imaging) is mainly for obtaining a color image. In step S1 in FIG. 1 is imaged. At this time, the measurement object 101 is illuminated with the uniform white illumination light from the uniform white illumination unit 111 and the ambient light 102, and the reflected light is focused on the respective RGB sensors 122 via the imaging lenses 121 in the two imaging units. image.

The data (first imaging signal) acquired by the RGB sensor 122 by the first imaging in step S1 of FIG. 5 is transferred to the image generator 131 in step S2.
In step S<b>2 , the image data of one of the two imaging units 120 is supplied to the image generating unit 131 . Here, one of the imaging units 120 preselected from the right-eye camera or the left-eye camera is called a reference camera, and the image data supplied to the image generation unit 131 in step S2 is the RGB pixels of the reference camera is point-sequential image data from . That is, the R, G, and B signals are discretely input to the image generator 131 in a dot-sequential manner by the RGB filters arranged in a mosaic pattern.

In the image generator 131, general demosaicing processing is performed in step S3. That is, the dot-sequential color signals are once separated for each color, and the discrete RGB signals are continuously synchronized by performing interpolation, sample-holding, or the like on pixel regions having no data from surrounding pixel information. R signal, G signal, and B signal. Such synchronization processing is called demosaicing processing. The R signal, G signal, and B signal that have been demosaiced and synchronized are sent to the color image output unit 133 in step S4.

Then, in step S5, it is processed as a color image and output as a display image for visual recognition or a recorded image. In this way, in steps S1 to S5, the measurement object is not illuminated by the near-infrared light illumination unit, and the measurement object is illuminated by at least one of visible light and ambient light. The first imaging signal is processed to generate a color image signal. Further, steps S1 to S5 function as a color image signal generator.

On the other hand, the RGB dot-sequential image data from the left-eye camera and the RGB dot-sequential image data from the right-eye camera supplied to the image generation unit 131 in step S2 are converted into data in step S6. Each of them is stored in the storage unit 132 .
This is because, in the first embodiment, the subtraction data of both cameras are essential for stereo matching, and the subtraction data are different because the right eye and the left eye see differently.

Next, in step S7, a second imaging signal is acquired by performing a second imaging (second imaging). The second imaging is for obtaining three-dimensional data, and images are captured while simultaneously illuminating the measurement object 101 with the white uniform illumination unit 111 and the near-infrared pattern illumination unit 112 . At this time, the white uniform illumination light from the white uniform illumination unit 111, the near-infrared pattern from the near-infrared pattern illumination unit 112, and the ambient light 102 are reflected by the measurement object 101, and the reflected light is used for two imaging. An image is formed on each RGB sensor 122 via an imaging lens 121 in the unit.

The image data (second imaging signals) respectively captured by the two RGB sensors 122 in the second imaging are sent to the image generator 131 as image data from the right-eye camera and image data from the left-eye camera in step S8. Sent.
In step S9, the image generation unit 131 subtracts the RGB image data before demosaicing, stored in the data storage unit 132, from the image data from the right-eye camera and the right-eye camera acquired in the second imaging. do. Here, step S9 functions as a calculation unit that calculates the second imaging signal and the first imaging signal.

That is, point-sequential image data acquired under the first imaging condition from point-sequential image data acquired in the second imaging (image data under uniform white illumination + near-infrared pattern illumination + ambient light). Subtract (white uniform illumination + image data under ambient light).

As a result of such subtraction, it is possible to obtain data equivalent to that obtained when only the near-infrared light pattern is irradiated by the near-infrared pattern illumination unit 112 . Here, if the near-infrared light is set within the range of 780 nm to 1000 nm for the spectral sensitivity characteristics of the RGB sensor as shown in FIG. Become. That is, the RGB sensor 122 provides image data with a resolution equivalent to that of a monochrome sensor without color filters.

Therefore, by calculating the three-dimensional data, which will be described later, it is possible to obtain a depth map, that is, three-dimensional data, which has a high resolution equivalent to that obtained by a monochrome sensor.
The data as the subtraction result described above is transferred to the three-dimensional data calculation unit 134 in step S10.

In the three-dimensional data calculation unit 134, stereo matching calculation is performed in step S11, and the amount of parallax between the right-eye camera and the left-eye camera is converted into the distance from the camera and calculated. Specifically, in a random pattern as shown in FIG. 2, a sub-window of an appropriate size is defined, and sub-windows whose patterns match between the right-eye camera and the left-eye camera are extracted using pattern matching.

The difference between the positions of the sub-windows with matching patterns is the parallax. If the parallax is large, it is located in front, and if the parallax is small, it is located in the back. As a result, three-dimensional data can be acquired as a depth map of the reference camera. Since this calculation method is known as a three-dimensional data calculation method, detailed description thereof will be omitted.

Here, if the window size of the sub-window is large, the processing speed takes a long time, but even if noise is added to the pattern, the accuracy of pattern matching increases, so data jumps can be suppressed. On the other hand, if the window size is small, the processing speed is increased, but the processing becomes vulnerable to noise, resulting in data jumps and the like. Since the resolution for the window size is sacrificed, the three-dimensional data is passed through a low-pass filter.

The acquired three-dimensional data is transferred to the three-dimensional data output unit 135 in step S12, and output as three-dimensional data to an external PC or the like in step S13. In this way, steps S7 to S13 are three-dimensional data for calculating three-dimensional data from the second imaging signal obtained from the imaging device while the measurement object is illuminated using the near-infrared light illumination unit. It functions as a calculator.
The color image obtained through the color image output unit 133 and the three-dimensional data obtained through the three-dimensional data output unit 135 may be displayed on a display (not shown) or It is used for object recognition processing and the like in an external PC or the like shown in the figure.

As described above, in this embodiment, the three-dimensional data obtained using the near-infrared pattern light is obtained by the RGB sensor for color images. Axes are aligned. If an ordinary color image and an image for three-dimensional data obtained using near-infrared pattern light are obtained by different sensors, then post-processing such as position and orientation conversion must be performed, and the data handling becomes very complicated. In particular, it is generally unavoidable for 3D measurement that data missing areas occur in the depth map. Once acquired, handling of the data is simplified.

Even if the color image sensor and the three-dimensional measurement sensor are different, the influence can be reduced by bringing the sensors close to each other. However, for example, in three-dimensional measurement for industrial use, high accuracy is essential, so the lens of each sensor must be enlarged and the sensors must be spaced apart. In that respect, according to this embodiment, the above problems can be fundamentally solved. In addition, since a general and mass-produced RGB sensor can be used, the device can be constructed at a low cost.

The second embodiment is a modification of the first embodiment, and is characterized in that the near-infrared pattern illumination unit 112 uses not one imaging condition but a plurality of imaging conditions that change with time.
FIG. 6 is a diagram showing an example of a near-infrared light pattern that changes with time in Example 2. FIG. In this embodiment, while sequentially projecting four types of predetermined patterns of near-infrared light, imaging is performed four times (second imaging to fifth imaging). The number of near-infrared patterns is not limited to four, and the number of times of imaging is not limited to four. As the number of imaging conditions increases, the resolution of three-dimensional data improves, but the processing time including imaging increases.

In Example 2, the total number of times of imaging is five, including the first imaging for obtaining a color image. Under the first imaging condition, imaging is performed while illuminating the measurement object 101 with the uniform white illumination unit 111 . This is equivalent to Example 1. Under the second to fifth imaging conditions, imaging is performed while the uniform white illumination unit 111 and the near-infrared pattern illumination unit 112 are illuminated. Here, in the second to fifth imaging, the intensity of the white uniform illumination unit 111 is the same, and the near-infrared pattern illumination unit 112 emits four patterns indicated by times t0, t1, t2, and t3 in FIG. switch sequentially.

A flow of various calculations using data of each imaging condition will be described below with reference to FIG.
FIG. 7 is a flow chart for explaining the operation of the second embodiment, and the processing of each step in FIG. 7 is performed by executing the computer program stored in the memory by the CPU in the calculation processing unit. In FIG. 7, the steps having the same names as those in FIG. 5 are the same processes as in FIG.

In step S1 in FIG. 7, the first imaging is performed while illuminating the measurement object 101 with the ambient light and the white uniform illumination unit 111 . After steps S2 and S6, in step S14, the N-th imaging (initial N=2) is performed. At this time, illumination is performed with the near-infrared pattern of t0 in FIG. The data obtained by the second imaging in step S14 is sent to the image generation unit 131 in step S15, and the image data corresponding to the near-infrared pattern at t0 in FIG. The data obtained in the first imaging are subtracted.

Then, in step S17, it is determined whether N=5, and if No, 1 is added to N in S18, and the process returns to step S14. conduct. These operations are repeated until Yes in step S17. As a result, the image data of the patterns t0 to t3 (near-infrared patterns N=2 to 5) captured by the second to fifth imaging are finally obtained from the first image data stored in the data storage unit 132. are subtracted from the data obtained under the imaging conditions of , and four types of subtraction data are obtained and stored. In this description, for the sake of simplicity, the imaging in S14, the transfer in S15, and the subtraction in S16 are repeated as a series of processes, but each process may be repeated four times. The same applies to the following examples.

If Yes in step S17, the process proceeds to step S19, and the four types of subtraction data stored in step S9 are transferred to the three-dimensional data calculation unit 134. FIG. Based on these four types of subtraction data, stereo matching is performed in step S20, transferred to the three-dimensional data output unit 135 in step S21, and output to the outside as three-dimensional data in step S22.

Here, the method of stereo matching in step S20 is different from that of the first embodiment. That is, in step S11 of Example 1, the parallax between one image of the left-eye camera and one image of the right-eye camera was calculated by performing pattern matching. However, in step S20, since there are four left-eye camera images and four right-eye camera images, matching is performed using data in which intensity information of pixels at the same positions are arranged in the time axis direction.

FIG. 8A is a conceptual diagram of pattern matching using sub-windows between single images shown in the first embodiment, and FIG. FIG. 4 is a conceptual diagram of pattern matching using intensity information; In the method shown in FIG. 8(A) and shown in the first embodiment, a portion having the same pattern in the subwindow in the data of the left-eye camera and the right-eye camera is extracted by pattern matching.

On the other hand, in the method of the second embodiment shown in FIG. 8B, the four images are arranged in the direction of the time axis, and the pixel-by-pixel data obtained by skewering the images in the direction of the time axis becomes the same for the right-eye camera and the left-eye camera. Parts are extracted by pattern matching. Since the sub-window is used in the first embodiment, the three-dimensional data is passed through a low-pass filter. However, since the sub-window is not used in the present embodiment, the resolution can be improved.

Embodiment 3 is a modification of Embodiments 1 and 2, in which the uniform white illumination unit 111 is turned off when illumination is performed by the near-infrared pattern illumination unit 112 so that the RGB sensor 122 is not saturated. Instead, the white uniform illumination unit 111 and the near-infrared pattern illumination unit 112 are both turned off, and a third image capturing is performed using only the ambient light 102 .

Also, for the sake of simplification of explanation, here, it is assumed that imaging using the near-infrared pattern illumination unit 112 is performed once. However, as described in Embodiment 2, imaging using the near-infrared pattern illumination unit 112 may be performed a plurality of times, and matching using temporal data may be performed to acquire three-dimensional data with higher resolution. .
Similarly, in the examples after the example 3 of the first embodiment, while switching the near-infrared pattern, the imaging may be performed a plurality of times with a time lag, but the explanation is omitted.

FIG. 9 is a flowchart for explaining the operation of the third embodiment, and each step of FIG. 9 is processed by executing a computer program stored in the memory by the CPU in the calculation processing unit. In FIG. 9, the steps with the same names as those in FIG. 5 are the same processes as in FIG.

In step S<b>1 in FIG. 9 , the first imaging is performed while the measurement object 101 is illuminated by the ambient light and white uniform illumination unit 111 . After step S2, in step S23, imaging is performed under the second imaging conditions (near-infrared pattern + ambient light), and the data obtained in the second imaging is transferred to the image generation unit 131 in step S24. , is stored in the data storage unit 132 in step S25.

Next, in step S26, the white uniform illumination unit 111 and the near-infrared pattern illumination unit 112 are both turned off, and illumination is performed only with the ambient light 102, and the third imaging is performed. The image data obtained by the third imaging is sent to the image generation unit 131 in step S27, and is subtracted from the data obtained by the second imaging stored in the data storage unit 132 in step S28.

Thereafter, it is transferred to the three-dimensional data calculation unit 134 in step S10, stereo matching is performed in step S11 based on the subtraction data obtained in step S28, transferred to the three-dimensional data output unit 135 in step S12, and transferred to the three-dimensional data output unit 135 in step S13. It is output to the outside as three-dimensional data.

In this way, the data acquired in the first imaging is data captured under uniform white illumination + ambient light, and the data acquired in the second imaging is data captured under near-infrared pattern illumination + ambient light. The data and the data acquired in the third imaging are data captured under ambient light. In addition, the data acquired by the first imaging is used for color images, and by subtracting the data acquired by the third imaging from the data acquired by the second imaging, only the near-infrared pattern illumination is obtained. is calculated, and three-dimensional data is calculated from this.

As described above, the third embodiment is characterized in that the white uniform illumination unit 111 and the near-infrared pattern illumination unit 112 are not illuminated at the same time to prevent saturation of the RGB sensor 122 . That is, in the case of Example 1 or 2, when the white uniform illumination unit 111 and the near-infrared pattern illumination unit 112 are illuminated simultaneously, there is a possibility that some or all of the pixels of the RGB sensor 122 are saturated.

In that case, matching between the right-eye camera and the left-eye camera becomes impossible. On the other hand, in Example 3, the lighting timings of the uniform white illumination unit 111 and the near-infrared pattern illumination unit 112 are shifted, and the imaging using only the ambient light 102 is separately performed to reduce the possibility of saturation. .

Example 4 is a modification of Example 3. When it is detected that the ambient light is sufficiently bright, processing is performed without using the uniform white illumination unit 111 . For the color image, data obtained under only ambient light 102 is used. Since white uniform illumination is not required, power can be saved or the device can be made smaller.
FIG. 10 is a flow chart for explaining the operation of the fourth embodiment, and the processing of each step in FIG. 10 is performed by executing the computer program stored in the memory by the CPU in the calculation processing unit. In FIG. 10, the steps having the same names as those in FIG. 5 are the same processing as in FIG.

In FIG. 10, in step S29, both the white uniform illumination unit 111 and the near-infrared pattern illumination unit 112 are turned off, and the first imaging is performed using only the ambient light 102. FIG. The obtained image data is stored in the data storage unit 132 in step S6 via step S2. In step S<b>30 , the second imaging is performed while the ambient light and illumination by the near-infrared pattern illumination unit 112 are applied. Subsequent processing is the same as in FIG.

According to Example 4, the data acquired in the first imaging is image data under ambient light, and the data acquired in the second imaging is image data under near-infrared pattern illumination + ambient light. . By subtracting the data acquired in the first imaging from the data acquired in the second imaging, the data of only the near-infrared pattern illumination is calculated, and the three-dimensional data is calculated from this. Other calculation steps are the same as in the first embodiment.

Example 5 is a modification of Example 3, and is an example in a situation where there is no ambient light 102 or almost no ambient light. In this case, since the ambient light 102 does not exist, the calculation for subtracting the ambient light 102 is unnecessary.
FIG. 11 is a flowchart for explaining the operation of the fifth embodiment, and each step of FIG. 11 is processed by executing a computer program stored in the memory by the CPU in the calculation processing unit. In FIG. 11, the steps having the same names as those in FIG. 5 are the same processing as in FIG.

In step S31 of FIG. 11, the first imaging is performed while illuminating the measurement object 101 only with the uniform white illumination unit 111. FIG. Further, in step S32, the second imaging is performed while only the near-infrared pattern illumination unit 112 is used for illumination.
The data acquired by the first imaging is image data illuminated only by uniform white illumination, and the data acquired by the second imaging is image data illuminated only by near-infrared pattern illumination.

The data obtained by the first imaging is used for color images as in the first embodiment. On the other hand, since the data acquired under the second imaging condition is image data obtained only by near-infrared pattern illumination, the subtraction processing in S9 of FIG. 5 is unnecessary. Therefore, through steps S8 and S10, three-dimensional data is calculated and output in steps S11 to S13.
<Second embodiment>

FIG. 12 shows the three-dimensional measurement system of the second embodiment.
In the second embodiment, only one imaging unit is provided. Instead, in the first embodiment, an unknown random pattern was used for the near-infrared pattern illumination, but a monocular camera needs to use a known pattern. Processing other than calculation of three-dimensional data is the same as that of the first embodiment. That is, 700 to 735 in FIG. 12 are the same as 100 to 135 in FIG. 1, so the description is omitted. By using a monocular camera, the size of the device can be reduced. Two or less examples with mainly different imaging conditions will be described in detail.

In Example 6, the compound eye camera of Example 1 is replaced by a monocular camera, and the near-infrared pattern illumination is a known random pattern.
Specifically, in the first imaging, the measurement object 701 is imaged while the white uniform illumination unit 711 is illuminating the measurement object 701 . In the second imaging, imaging is performed while illuminating the white uniform illumination unit 711 and the near-infrared pattern illumination unit 712 . Here, the illumination pattern of the near-infrared pattern illumination unit 712 is a known random pattern, unlike the first embodiment.

A known random pattern will now be described. If the random pattern is known, even a monocular camera can grasp where the partial area of the random pattern is on the image by pattern matching. FIG. 13 is a conceptual diagram of the principle of distance measurement using a known random pattern. As shown in FIG. 13A, if there is no measurement target and there is a floor behind the measurement range, it is possible to know in advance how a known random pattern will be imaged by the RGB sensor. On the other hand, as shown on the right side of FIG. 13, when there is an object to be measured and the known random pattern image changes, the distance from the floor surface can be measured from the amount of change.

FIG. 14 is a flow chart for explaining the operation of the sixth embodiment, and the processing of each step in FIG. 14 is performed by executing the computer program stored in the memory by the CPU in the calculation processing unit. In FIG. 14, the steps with the same names as those in FIG. 5 are the same processes as in FIG. 5, and 731 to 735 in FIG. 14 are the same as 131 to 135 in FIG. 1, so the description will be omitted or simplified.

In step S33 of FIG. 14, the second imaging is performed under uniform white illumination and ambient light illumination, while performing near-infrared pattern illumination with a known random pattern. After that, through steps S8 to S10, in step S34, the three-dimensional data calculation unit 134 calculates three-dimensional data using a known random pattern. After that, the three-dimensional data is output to the outside through steps S12 and S13.

In Example 7, the compound eye camera of Example 2 is replaced by a monocular camera, and the pattern of near-infrared pattern illumination is a known plurality of line patterns instead of the random pattern. This method is generally called a spatial encoding method.
FIG. 15 is a diagram showing an example of multiple line patterns in the spatial encoding method. It is known that high-precision three-dimensional measurement can be performed by implementing a spatial encoding method. Since this is generally well known and publicly known, further explanation is omitted.

FIG. 16 is a flow chart for explaining the operation of the seventh embodiment, and the processing of each step in FIG. 16 is performed by executing the computer program stored in the memory by the CPU in the calculation processing unit. In FIG. 16, the steps with the same names as those in FIG. 7 are the same processes as in FIG. 7, and 731 to 735 in FIG. 16 are the same as 131 to 135 in FIG. 1, so the description will be omitted or simplified.

A specific imaging operation is the same as that of the second embodiment, and in step S1, the first imaging is performed while illuminating the measurement object 701 with ambient light and the uniform white illumination unit 711 . After that, through steps S2 and S6, in steps S35 and S15 to S18, the second to fifth imaging is performed while illuminating the measurement object with the ambient light, the uniform white illumination unit 711, and the near-infrared pattern illumination unit 712. . At that time, the illumination pattern of the near-infrared pattern illumination unit 712 is used by sequentially switching between four known line patterns (line patterns N=2 to 5) indicated by t1, t2, and t3 from time t0 in FIG. do.
After that, through step S19, in step S36, the three-dimensional data calculation unit 134 calculates three-dimensional data by a spatial encoding method using a known line pattern. After that, the three-dimensional data is output to the outside through steps S21 and S22.

As described above, according to the seventh embodiment, since a monocular camera can be used, the device can be made more compact, and high-precision three-dimensional measurement can be performed.
The method using known random patterns and a plurality of known line patterns described in Examples 6 and 7 can also be applied to a three-dimensional measurement system using a compound eye camera as in Embodiment 1. omitted.
<Third Embodiment>

Next, FIG. 17 is a block diagram showing a configuration example of a three-dimensional measurement system according to the third embodiment. 3rd Embodiment changes the near-infrared pattern illumination part 112 of 1st Embodiment into the near-infrared uniform illumination part 1013. FIG. That is, in this embodiment, the near-infrared illumination unit has a near-infrared uniform illumination unit for illuminating the measurement object with uniform near-infrared light. Note that 1000 to 1035 in FIG. 17 are the same as 100 to 135 in FIG. 1, so the description is omitted.

In this embodiment, stereo matching is performed using near-infrared uniform illumination. First, the difference between stereo matching using near-infrared pattern illumination and stereo matching when using near-infrared uniform illumination explain. When using near-infrared pattern illumination, since the right-eye camera and left-eye camera are matched using a part of the pattern as a mark, the pattern illumination can be performed actively even where there are no features on the object side, such as the flat surface of the measurement object. By giving information by, it is possible to measure with high accuracy. On the other hand, if the object to be measured is made of plastic, the pattern may blur due to the penetration of light into the inside of the object.

In that case, the object is illuminated evenly with uniform illumination (i.e., even if the light is blurred, there is no effect because it is blurred as a whole), and stereo matching is performed using the shape and structure of the part as a mark to determine the object to be measured. Edges and the like can be measured more clearly. However, when near-infrared uniform illumination is used, the mark becomes the shape, structure, etc. of the object to be measured, making it difficult to measure the flat surface of the object to be measured.
Therefore, the present embodiment is particularly effective for a measurement object for which light is blurred under pattern illumination, and enables highly accurate three-dimensional measurement.

Example 8 is a modification of Example 1 in which near-infrared pattern illumination is changed to near-infrared uniform illumination.
FIG. 18 is a flowchart for explaining the operation of the eighth embodiment, and each step of FIG. 18 is processed by the CPU in the calculation processing unit executing the computer program stored in the memory. In FIG. 18, the steps having the same names as those in FIG. 5 are the same processes as in FIG. 5, and 1031 to 1035 in FIG. 18 are the same as 131 to 135 in FIG. 1, so the description will be omitted or simplified.

In step S<b>1 , the first imaging is performed while illuminating the measurement object 1001 with ambient light and the white uniform illumination unit 1011 . Through steps S2 and S6, the second imaging is performed while illuminating the measurement object 1001 with ambient light, the white uniform illumination unit 1011, and the near-infrared uniform illumination unit 1013 in step S37. Other processes are the same as those in the first embodiment, and description thereof is omitted.

According to the eighth embodiment, it is possible to perform highly accurate three-dimensional measurement of a measurement object that would be blurred by pattern illumination.
The above is the description of the third embodiment. The near-infrared uniform illumination described in the third embodiment can also be used in Examples 3, 4, and 5 of the first embodiment.
<Fourth Embodiment>

FIG. 19 shows a three-dimensional measurement system according to the fourth embodiment. In FIG. 19, 1113 is a near-infrared uniform illumination unit, and 1100 to 1135 other than that are the same as 100 to 135 in FIG. 1, so description thereof is omitted.
In the fourth embodiment, three-dimensional data is acquired using a compound eye camera, and more accurate three-dimensional data is acquired by calculating from images using near-infrared pattern illumination and near-infrared uniform illumination. making it possible. That is, in the first to third embodiments, the near-infrared light illumination unit illuminates the measurement target with only one of the predetermined pattern of near-infrared light and uniform near-infrared light. In the embodiment, the object to be measured is illuminated using both near-infrared light of a predetermined pattern and uniform near-infrared light.

Example 9 is a combination of Example 1 and Example 8. With regard to near-infrared illumination, near-infrared pattern illumination and near-infrared uniform illumination are respectively irradiated for imaging, and three-dimensional data is calculated. is synthesized to
FIG. 20 is a flow chart for explaining the operation of the ninth embodiment, and the processing of each step in FIG. 20 is performed by executing the computer program stored in the memory by the CPU in the calculation processing unit. 20, the steps with the same names as those in FIGS. 1 and 18 are the same processes as in FIGS. 1 and 18, and 1131 to 1135 in FIG. 20 are the same as 131 to 135 in FIG. Simplify.

From step S11, stereo matching using infrared patterns is performed in the same manner as in FIG. 5, and the result is stored.
Next, in step S37, the third imaging is performed while illuminating the measurement object 1101 with ambient light, the white uniform illumination unit 1111, and the near-infrared uniform illumination unit 1113 in the same manner as in FIG.

Thereafter, stereo matching is performed in steps S38 to S41 in the same manner as in steps S8 to S11 of FIG. 18, and the results are saved. That is, at this stage, the three-dimensional data calculation unit 1134 has two types of three-dimensional data: the three-dimensional data calculated by the near-infrared pattern illumination and the three-dimensional data calculated by the near-infrared uniform illumination.
Next, in step S42, the three-dimensional data obtained in step S11 and the three-dimensional data obtained in step S41 are selected, whichever is more reliable, and the final three-dimensional data is created. Then, it is transferred to the three-dimensional data output unit 1135 in step S43, and is output to the outside as three-dimensional data in step S44.

In the present embodiment, if one of the three-dimensional data, which is the depth map, is missing due to noise or the like, the other three-dimensional data can be complemented, so the missing can be reduced. . Moreover, even for a pixel in which both types of three-dimensional data exist, it is possible to select the three-dimensional data with higher reliability for each pixel by using gradient information and the like with adjacent pixels on the depth map.

For example, data calculated by near-infrared pattern illumination is used for data of a flat portion of the measurement object (pixels with a small gradient to adjacent pixels on the depth map). Then, for the data of the edge part of the object to be measured (the pixel having a large gradient with respect to the adjacent pixel on the depth map), a selection is made such as using the data calculated by the near-infrared uniform illumination. Thereby, highly reliable and highly accurate three-dimensional measurement data can be acquired for the entire measurement object.
In addition, although Example 9 was described as a synthesis of Examples 1 and 8, Example 8 was synthesized with Examples 2, 3, 4, and 5 of Embodiment 1 in the same manner as in Example 9. Embodiments may be constructed.
<Fifth Embodiment>

FIG. 21 is a diagram showing a configuration example of a manufacturing system according to the fifth embodiment. The three-dimensional measuring device 1100 is attached to the tip of a robot arm 1300 (gripping device) and used, and functions as a part of the manufacturing system. The projection unit 1110 of the three-dimensional measurement apparatus 1100 projects a near-infrared pattern or near-infrared uniform light onto the measurement object 1101 placed on the support table 1350, and the imaging unit 1120 captures the image to obtain an image.

Then, the three-dimensional measurement apparatus 1100 calculates three-dimensional data of the object to be measured 1101 by performing control such as that shown in the flowchart of FIG.
The control unit 1310 obtains the position, shape, and orientation of the measurement object 1101 based on the three-dimensional data of the measurement object obtained from the three-dimensional measurement device 1100 . Then, based on the obtained position, shape and posture, a drive command is sent to the robot arm 1300 to control the arm of the robot arm 1300 and the robot hand at the tip in multiple axes.

The robot arm 1300 holds (grips) the measurement object 1101 with a robot hand or the like (gripping portion) at its tip, and performs movement control processing such as translation and rotation and posture control processing. Furthermore, by performing an assembling process of assembling the measurement object 1101 to other parts by the robot arm 1300, it is possible to manufacture an article composed of a plurality of parts, such as an electronic circuit board or a machine. Further, by processing the moved measurement object 1101, an article can be manufactured.

The control unit 1310 has an arithmetic device such as a CPU as a computer and a storage device such as a memory storing a computer program for controlling the measurement object 1101 . Note that the controller 1310 for controlling the robot arm 1300 may be provided outside the manufacturing system of FIG. Also, the measurement data measured by the three-dimensional measurement device 1100 and the obtained image may be displayed on the display unit 1320 .

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist thereof. Also, in the explanation, trade-offs such as speeding up of processing time, improvement of resolution and measurement accuracy, and cost of apparatus configuration have been explained. By appropriately combining these for each measurement object, it is possible to provide a three-dimensional measurement system that enables three-dimensional measurement at the lowest cost and with the highest efficiency.
It should be noted that a computer program that implements the functions of the above-described embodiments may be supplied to a three-dimensional measurement system or the like via a network or various storage media for part or all of the control in this embodiment. A computer (or CPU, MPU, etc.) in the three-dimensional measurement system or the like may read and execute the program. In that case, the program and the storage medium storing the program constitute the present invention.

100 Three-dimensional measurement system 101 Measurement object 112 Near-infrared pattern illumination 122 RGB sensor 130 Calculation output unit

Claims (18)

a near-infrared light illumination unit for illuminating the measurement object with near-infrared light;
an imaging device in which adjacent pixels have different wavelength sensitivities to visible light and each pixel has a sensitivity characteristic to the near-infrared light;
a color image signal generation unit that processes a first imaging signal obtained from the imaging device in a state where the measurement object is not illuminated by the near-infrared light illumination unit to generate a color image signal;
a three-dimensional data calculation unit that calculates three-dimensional data from a second imaging signal obtained from the imaging element while the measurement object is illuminated using the near-infrared light illumination unit;
A three-dimensional measuring device comprising: 2. The three-dimensional measurement apparatus according to claim 1, wherein the color image signal generating section performs demosaicing processing on the first imaging signal to generate the color image signal. 2. The three-dimensional measurement apparatus according to claim 1, wherein the three-dimensional data calculation section has a calculation section for calculating the second imaging signal and the first imaging signal. The color image signal generation unit performs demosaicing processing on the first imaging signal to generate the color image signal, and the computing unit performs demosaicing processing on the first imaging signal and the first imaging signal before the demosaicing processing. 4. The three-dimensional measuring apparatus according to claim 3, wherein the imaging signal of . 5. The three-dimensional measurement apparatus according to claim 4, wherein the calculation unit subtracts the first image signal before the demosaicing process from the second image signal. 2. The three-dimensional measurement apparatus according to claim 1, wherein the near-infrared light illumination unit includes a near-infrared pattern illumination unit that illuminates the measurement object with the near-infrared light of a predetermined pattern. 7. The three-dimensional measurement apparatus according to claim 6, wherein the near-infrared light illumination unit sequentially projects the near-infrared light of the predetermined pattern onto the measurement object. 7. The three-dimensional measuring device according to claim 6, wherein said predetermined pattern includes a random pattern. 7. The three-dimensional measuring device according to claim 6, wherein said predetermined pattern includes a line pattern. 2. The three-dimensional measurement apparatus according to claim 1, wherein the near-infrared light illumination unit includes a near-infrared uniform illumination unit for illuminating the measurement object with the uniform near-infrared light. 2. The three-dimensional measuring apparatus according to claim 1, further comprising a visible light illumination unit for illuminating the object to be measured with visible light. 12. The three-dimensional measurement apparatus according to claim 11, wherein the visible light illumination unit illuminates the measurement object with white light. 2. The three-dimensional measurement according to claim 1, wherein the near-infrared light illumination unit illuminates the object to be measured with at least one of the near-infrared light having a predetermined pattern and the uniform near-infrared light. Device. The color image signal generation unit obtains the 2. The three-dimensional measuring apparatus according to claim 1, wherein the color image signal is generated by processing the obtained first imaging signal. 2. The three-dimensional measurement apparatus according to claim 1, wherein the near-infrared light has a wavelength within a range of 780 nm to 1000 nm. A computer program for controlling each part of the three-dimensional measuring device according to any one of claims 1 to 15 by a computer. A three-dimensional measurement device according to any one of claims 1 to 15,
and a robot that holds and moves the object to be measured based on the three-dimensional data of the object to be measured obtained by the three-dimensional measuring device. A step of calculating the three-dimensional data of the measurement object using the three-dimensional measurement device according to any one of claims 1 to 15;
and a step of manufacturing an article by processing the object to be measured based on the three-dimensional data.

Images