WO2017090111A1

WO2017090111A1 - Three-dimensional image measurement device and method

Info

Publication number: WO2017090111A1
Application number: PCT/JP2015/083036
Authority: WO
Inventors: 幸康堂前; 関　真規人; 鹿井　正博; 奥田　晴久
Original assignee: 三菱電機株式会社
Priority date: 2015-11-25
Filing date: 2015-11-25
Publication date: 2017-06-01
Also published as: CN108369089B; CN108369089A; JPWO2017090111A1; DE112015007146T5; JP6038415B1

Abstract

A three-dimensional image measurement device for generating distance image data in which distances from a sensor to an object are represented by pixel values from pattern image data obtained by using a sensor to photograph a pattern projected by a light projector onto an object to be photographed, wherein a control means is provided for supplementing pixels having unknown pixel values in distance image data obtained from pattern image data photographed by the sensor with pixel values of a distance image obtained from pattern image data photographed by the sensor under different photography conditions. The control means achieves different photography conditions by changing at least one from among the exposure time of the sensor, the aperture value of the sensor, the gain of the sensor, the light projection intensity of the light projector, the light projection direction of the light projector, the ambient light position at the time of photography, and the ambient light direction at the time of photography.

Description

3D image measuring apparatus and method

The present invention is a three-dimensional image measurement that measures a three-dimensional shape of an object by projecting a pattern onto the object, imaging the reflected light, and analyzing it using a three-dimensional image measurement method such as a spatial coding method. The present invention relates to an apparatus and a method.

A pattern image, which is a reflected light image from a measurement object on which a predetermined pattern is projected, is obtained by photographing the reflected light of the pattern projected on the object with a sensor such as a camera. When the positional relationship between the projector that projects the pattern and the sensor that captured the pattern image is known, the distance from the sensor to the object can be restored by analyzing the pattern image based on the principle of triangulation. In general, this includes a three-dimensional image measurement method called an active stereo method, which includes a spatial encoding method, a structured light method, a light cutting method, a spot light projection method, and a slit light projection method. .

In order to measure the distance of the object by this three-dimensional image measurement method, the pixel value obtained when the reflected light of the pattern projected onto the object is received by the sensor element and the other elements that do not receive the reflected light. A clear difference is required between the pixel values. These pixel values are determined depending on various factors such as ambient light other than the projected pattern, diffuse reflectance on the surface material of the object, light intensity of the projected pattern, and light receiving sensitivity of the sensor.

When projecting a pattern onto a black object with a high diffuse reflectance, only reflected light with a weak intensity is returned, making it difficult to measure the pattern. In this case, it is necessary to increase the amount of light received by the sensor element. Increasing the exposure time is one method for increasing the amount of received light. However, if the exposure time is lengthened, this time, it will be strongly influenced by ambient light. For example, when a pattern is projected onto a white object or an object having a low diffuse reflectance, reflection of ambient light or pattern light returns as reflected light from a portion where the pattern is not projected. In this case also, it is difficult to measure the pattern. This is an example showing that it is difficult to simultaneously measure a black object with a high diffuse reflectance and a white object or an object with a low diffuse reflectance, but the amount of light received by the sensor element by a method other than the exposure time. Even if the number is increased, the same problem occurs.

As a method for measuring a three-dimensional image of a scene in which a plurality of objects having different surface states are mixed, a pattern image (multiple shutter speed) obtained by combining pattern images taken at different exposure times. There is a method of restoring the distance from (image) (Patent Document 1).

Japanese Patent No. 4889373

In such a three-dimensional image measurement method, three-dimensional image measurement is performed by a spatial encoding method using a multiple exposure pattern image obtained by synthesizing a short exposure time pattern image and a long exposure time pattern image. Since the continuity between pixels deteriorates due to insufficient range of the light receiving element of the camera, there is a problem that pixels having inaccurate values are generated, that is, erroneous measurement occurs.

An object of the present invention is to solve the above-described problems, and in a scene where a plurality of objects having different surface states exist, a three-dimensional image measurement apparatus capable of performing stable measurement with less measurement loss while suppressing erroneous measurement. Is to provide.

A three-dimensional image measurement apparatus according to an aspect of the present invention includes:
A three-dimensional image measuring device that generates distance image data representing a distance from the sensor to the object as a pixel value from pattern image data obtained by photographing a pattern projected onto the object to be photographed by the projector. In
Control means for complementing a pixel whose pixel value is unknown in distance image data obtained from pattern image data photographed by the sensor with a pixel value of a distance image obtained from pattern image data photographed by the sensor under different photographing conditions It is characterized by that.

Therefore, according to the three-dimensional image measurement apparatus according to the present invention, it is possible to perform stable measurement with few measurement omissions while suppressing erroneous measurement in a scene where there are a plurality of objects having different surface states.

It is a block diagram which shows the hardware constitutions of the three-dimensional image measuring device concerning Embodiment 1 of this invention. It is a block diagram which shows the schematic function of the information processing apparatus 3 of FIG. It is a figure which shows the camera image 21 when a measurement target object is image | photographed with the three-dimensional image measuring device of FIG. It is a figure which shows the distance image 22 when a measurement target object is image | photographed with the three-dimensional image measuring device of FIG. It is a figure which shows the example of the pattern image containing the positive pattern image 23 and the negative pattern image 24 which are used by the general space coding method. It is a figure of the example of an image which shows that the imaging | photography result of a measurement object changes with the difference in exposure time, Comprising: It is a figure of the example of an image of the image 25 in the case of the 1st darkness. It is a figure of the example of an image which shows that the imaging | photography result of a measurement object changes with the difference in exposure time, Comprising: It is a figure of the example of an image of the image 26 in the case of the 2nd darkness. It is a figure of the example of an image which shows that the imaging | photography result of a measurement object changes with the difference in exposure time, Comprising: It is a figure of the example of an image of the image 27 in the case of the 3rd darkness. It is a figure of the example of an image which shows that the imaging | photography result of a measurement object changes with the difference in exposure time, Comprising: It is a figure of the example of the image 28 when it is the 4th dark, ie, the case where it is the first thin. It is a figure which shows the image example of the camera image 41 of the pattern image with small light reception amount of a pixel. It is a figure which shows the image example of the distance image 42 obtained from the camera image 41 of FIG. 6A. It is a figure which shows the image example of the camera image 43 of the pattern image with comparatively large light reception amount of a pixel. It is a figure which shows the image example of the distance image obtained from the camera image 43 of FIG. 7A. It is a measurement result of the three-dimensional image measurement apparatus of FIG. 1 and is a diagram illustrating an image example of a camera image of an object. It is a measurement result of the three-dimensional image measurement apparatus of FIG. 1, and is a diagram illustrating an image example of an original distance image based on photographing with a relatively short exposure time. It is a measurement result of the three-dimensional image measurement apparatus of FIG. 1, and is a diagram illustrating an image example of a distance image calculated by Expression (2). It is a measurement result of the three-dimensional image measurement apparatus of FIG. 1, and is a diagram illustrating an image example of a distance image calculated by Expression (2) and Expression (3). It is a flowchart which shows the three-dimensional image measurement process performed by the information processing apparatus 3 of the three-dimensional image measurement apparatus of FIG. 10 is a flowchart showing measurement reliability evaluation processing (step S9) which is a subroutine of FIG. It is a flowchart which shows the distance image composition process (step S10) which is a subroutine of FIG. It is a block diagram which shows the hardware constitutions of the three-dimensional image measuring device concerning Embodiment 2 of this invention. It is a block diagram which shows the hardware constitutions of the three-dimensional image measuring device concerning Embodiment 3 of this invention. It is a flowchart which shows the three-dimensional image measurement process performed by the information processing apparatus 3 of the three-dimensional image measurement apparatus of FIG. It is a block diagram which shows the hardware constitutions of the three-dimensional image measuring device concerning Embodiment 4 of this invention.

Embodiment 1 FIG.
FIG. 1 is a block diagram showing a hardware configuration of the three-dimensional image measurement apparatus according to the first embodiment of the present invention. In FIG. 1, a camera 1 that images a three-dimensional object and outputs image data, and a projector 2 that projects a predetermined amount of light onto the three-dimensional object are respectively interface circuits 36 of the information processing apparatus 3. , 37. The information processing apparatus 3 includes a control unit such as a computer or a digital computer. Specifically, a central processing unit (hereinafter referred to as a CPU) 31 that executes a three-dimensional image measurement process, and a three-dimensional image measurement process. Read-only memory (hereinafter referred to as ROM) 32, random access memory (hereinafter referred to as RAM) 33 that temporarily stores input / output data and input / output signals, and a mouse, keyboard, gesture, for example. An input device 34 that includes a recognition camera, a wearable acceleration sensor, and the like and receives an operator's operation, and a display device 35 that presents information to the operator, such as a display, are provided. It is connected and configured.

The interface circuit 36 performs conversion of communication data and signals executed between the camera 1 and the CPU 31. Further, the interface circuit 37 performs conversion of a command signal from the CPU 1 to the projector 2 and the like. Specifically, the CPU 1 outputs a pattern projection command signal for instructing to project a predetermined pattern, which will be described in detail later, to the projector 2 via the interface circuit 37. In addition, the CPU 31 generates a shooting command signal and outputs it to the camera 1 via the interface circuit 36. In response to this, the camera 1 outputs pattern image data which is image data obtained by shooting a predetermined pattern and its high-frequency pattern. The image data is output to the CPU 1 via the interface circuit 36.

The three-dimensional image measurement apparatus according to the present embodiment synthesizes distance images (images in which the distance from the camera 1 to the measurement object is represented by pixel values) obtained from pattern images photographed under different photographing conditions. is there. At that time, a pixel with an unknown pixel value of a distance image obtained from a pattern image photographed with a relatively small amount of received light can be obtained from a pattern image photographed with a relatively large amount of received light. Interpolation is performed using pixel values of the distance image. In this case, the composition method is also changed between pixels with high and low reliability of measurement of high-frequency components in the pattern.

The camera 1 is a digital camera for photographing the pattern of the three-dimensional object projected by the projector 2 and outputting the image data captured to the information processing apparatus 3. However, when the camera 1 has a communication function for receiving a shooting command from the information processing device 3 and returning the shot pattern image to the information processing device 3, the camera 1 is a pinhole camera, a range finder camera, or a view camera. There may be. When a pattern image can be taken, a light field camera provided with a microlens array for estimating the direction of light entering the lens may be used.

The projector 2 receives a pattern projection command from the information processing device 3 and irradiates a predetermined pattern to a three-dimensional object. The projector 2 is a CRT projector, a liquid crystal projector, a digital light processing projector, a reflective liquid crystal element projector, A reflective display element projector using a diffraction phenomenon, a laser, a diffraction grating, a polygon mirror, and a driving device that changes a laser and a laser irradiation position may be used.

FIG. 2 is a block diagram showing a schematic function of the information processing apparatus 3 of FIG. In FIG. 2, the information processing apparatus 3
(1) A projection pattern control unit 10 that outputs a pattern projection command signal for projecting a predetermined pattern to the projector 2;
(2) An image acquisition control unit 11 that outputs a shooting command signal to the camera 1 at a timing synchronized with a pattern projection command signal from the projection pattern control unit 10;
(3) a distance restoration unit 12 that restores the distance between the camera 1 and the object as distance image data of digital information from a pattern image captured by the camera 1;
(4) From each of the distance image data restored by the distance restoration unit 12 and the high-frequency pattern image data photographed by the camera 1 (images obtained by photographing patterns having fine features in a plurality of types of patterns projected by the projector 2), each pixel A measurement reliability evaluation unit 13 for evaluating the reliability of the distance estimation of
(5) From a plurality of distance image data restored by the distance restoration unit 12 and a measurement reliability map of the distance image data created by the measurement reliability evaluation unit 13, a pixel whose pixel value is unknown, an actual camera, and an object Distance image composition unit 14 for compositing a plurality of distance images so that the number of pixels having inaccurate pixel values with respect to the distance up to
And is configured.

The projection pattern control unit 10 and the image acquisition control unit 11 are controlled using a synchronization signal, and synchronize the timing of the pattern projection command signal and the imaging command signal. Thereby, the increased data can be obtained for the synchronized pattern image.

FIG. 3A is a diagram showing a camera image 21 when a measurement target is photographed by the three-dimensional image measurement apparatus of FIG. FIG. 3B is a diagram showing a distance image 22 when a measurement object is photographed by the three-dimensional image measurement apparatus of FIG. In the camera image 21 of FIG. 3A, the intensity of light reflected by the reflectance of the material of the object is reflected in the pixel value, so that the black object is black and the white object is white. On the other hand, in the distance image 22 of FIG. 3B corresponding to the camera image 21, since the distance from the camera 1 to the object is expressed as a pixel value, the object close to the camera 1 is bright and the distant object is displayed black. .

FIG. 4 is a diagram showing an example of a pattern image including a positive pattern image 23 and a negative pattern image 24 used in a general spatial coding method. That is, FIG. 4 shows an image of a plurality of binary patterns used in a general spatial coding method. The slit pattern in the vertical direction is irradiated, and the irradiated portion is displayed bright and the non-irradiated portion is displayed dark. As shown in FIG. 4, a plurality of slit patterns having different frequency bands are projected. Depending on the thickness of the slit, a thin slit pattern is called a high-frequency pattern image, and a thick slit pattern is called a low-frequency pattern image. In addition, in order to suppress measurement errors in the spatial coding method, a pair of slit patterns whose binary patterns are opposite in polarity (that is, image data of a pair obtained by reversing the light and dark of the photographing pattern) is used. Here, a pair of the paired pattern images is referred to as a positive pattern image 23, and the other pair is referred to as a negative pattern image 24.

The distance restoration unit 12 restores the distance including the distance from the camera 1 to the object from the pattern image obtained by the image acquisition control unit 11. Although it is possible to express the distance restoration result as a point cloud that is a set of point clouds for a three-dimensional object, it is possible to utilize image processing with a short processing time by expressing it as distance image data. .

The measurement reliability evaluation unit 13 quantifies the possibility that the pixel value is unknown or inaccurate in each pixel of the distance image from the camera 1 to the object restored by the distance restoration unit 12. For this purpose, high-frequency pattern image data is used among the pattern images photographed by the image acquisition control unit 11. Since the high-frequency pattern is a thin pattern, the pattern tends to disappear according to the intensity change of the reflected light of the ambient light or the pattern light itself. In particular, when the intensity of the reflected light is strong, not only the pixels that directly receive the reflected light are saturated but also the surrounding pixels are saturated, leading to erroneous measurement over a wide range. Since the high frequency pattern is easily affected by fluctuations in reflected light, analyzing its state is suitable for evaluation of measurement reliability.

Here, the measurement reliability map R _PN is defined as follows.

(1)

Here, I ^P _High is high frequency positive pattern image data, and I ^N _High is high frequency negative pattern image data. The value of each pixel in the high-frequency positive pattern image data and the high-frequency negative pattern image data inherently has an extreme difference, but no difference occurs depending on the influence of ambient light and the intensity of reflected light of the pattern light. This is equivalent to the fact that a high frequency pattern could not be observed. That is, when the value of the measurement reliability map _RPN is 0 or lower than a predetermined threshold (for example, a positive value near 0, for example, 0.01), it can be evaluated that the measurement reliability is extremely low.

The distance image composition unit 14 synthesizes distance images photographed under different photographing conditions, and corrects a pixel whose pixel value is unknown under each photographing condition or a pixel value different from the actual distance between the camera 1 and the object. I do. The different shooting conditions are that a pattern image is shot by changing the amount of received light of the camera element, and the distance image is restored. In the apparatus configuration of FIG. 1, the information processing apparatus 3 has an exposure time of the camera 1 (for example, using a first exposure time and a second exposure time longer than the first exposure time), an aperture of the camera 1. By changing the value, the gain of the camera 1, the light projecting intensity of the projector 2, and the light projecting direction of the projector 2, different shooting conditions can be realized.

FIG. 5A is a diagram of an image example showing that the photographing result of the measurement object varies depending on the exposure time, and is a diagram of an image example of the image 25 in the first darkest case. FIG. 5B is an image example showing that the measurement result of the measurement object varies depending on the exposure time, and is an image example of the image 26 in the second darkest case. Furthermore, FIG. 5C is a diagram of an image example showing that the photographing result of the measurement object varies depending on the difference in exposure time, and is a diagram of an image example of the image 27 in the third darkest case. Furthermore, FIG. 5D is a diagram of an image example showing that the photographing result of the measurement object varies depending on the exposure time, and is a diagram of an image example of the image 28 when it is the fourth darkest, that is, when it is the thinnest first. It is. That is, FIGS. 5A to 5D are images of images obtained by shooting the same object under different exposure conditions under different shooting conditions. The image in FIG. 5A has the shortest exposure time, and the exposure time becomes longer in the order of FIGS. 5B, 5C, and 5D. In an image with a relatively short exposure time, the difference between the black object and the surroundings is difficult to distinguish, and in an image with a long exposure time, the difference between the white object and the surroundings is difficult to distinguish.

FIG. 6A is a diagram illustrating an image example of the camera image 41 of a pattern image with a small amount of light received by pixels. FIG. 6B is a diagram illustrating an image example of the distance image 42 obtained from the camera image 41 of FIG. 6A. Furthermore, FIG. 7A is a diagram illustrating an image example of the camera image 43 of a pattern image having a relatively large amount of light received by pixels. FIG. 7B is a diagram showing an example of a distance image obtained from the camera image 43 of FIG. 7A.

In the distance image 42 in FIG. 6B, a small black circle indicates a pixel whose pixel value is unknown. When the exposure time is short, the pixel value of the white object is easily obtained accurately, but the pixel value of the black object is likely to be unknown. A portion such as a texture in the distance image 44 in FIG. 7B indicates an incorrect pixel value. When the exposure time is long, the pixel value of the black object can be accurately obtained, but the pixel value of the white object tends to be inaccurate. Interpolation with neighboring pixels can be applied to pixels whose pixel values are unknown, but pixels with inaccurate pixel values cannot be interpolated, and serious problems such as robot collision are likely to occur in actual applications. Therefore, high-quality distance image data is obtained by synthesizing the distance images by the following strategy method.

The distance image synthesizing unit 14 generates and outputs distance image data I _fusion (x, y) synthesized by synthesizing distance images as in the following equation.

(2)

Here, I _Low indicates distance image data with a low light reception amount of pixel values based on a pattern image photographed under a certain photographing condition. I _High refers to distance image data with a high received light amount of pixel values based on pattern images shot under different shooting conditions. When the distance image is acquired twice while changing the exposure time, the distance image data based on shooting with a short exposure time corresponds to I _Low , and the distance image data based on shooting with a long exposure time corresponds to I _High . Here, (x, y) means a two-dimensional position of a pixel in an image.

In a pattern image with a low amount of received light, the pattern disappears, so that the pixel value of the distance image data that can be calculated tends to be unknown. However, since the influence of ambient light is also suppressed, light reception noise (hereinafter referred to as noise) is reduced, and the pixel value for which the pixel value of the distance image data can be calculated tends to be accurate. On the other hand, a pattern image with a high amount of received light is strongly affected by ambient light, so that noise increases and pixel values of distance image data tend to be inaccurate. That is, the distance image data I _Low has little noise, but there are many places where the pixel value is unknown. The distance image data I _High is noisy. However, there is a possibility that the pixel value of a pixel whose pixel value is unknown in the distance image data I _Low is obtained from the distance image data I _High . Therefore, as in Equation (2), the distance image data _{I Low} as a _reference, the _{I Low} pixel values unknown pixel (equation _{(2), I Low (x} , y) = 0 is unknown pixel value Is equivalent to a pixel value of the distance image data I _High , and high-quality distance image data I _fusion (x, y) can be obtained.

In this case, if the pixel value of the distance image data I _High of the pixel whose pixel value of the distance image data I _Low is unknown is incorrect, the result of the synthesized distance image will also be incorrect. Therefore, based on the measurement reliability map R _PN obtained from the high-frequency positive pattern image data I ^P _High obtained when the distance image data I _High is calculated and the high-frequency negative pattern image data I ^N _High , the following is performed. The distance image data I _fusion (x, y) is synthesized.

(3)

(4)

That is, as in the equation (2), the distance image data I _Low is used as a reference, and the pixel value of the distance image data I _Low is unknown (in the equation (2), I _Low (x, y) = 0 The pixel value of the distance image data I _High is complemented with the pixel value of the distance image data I _High. However, when the value of the measurement reliability map _RPN is 0, the pixel value is 0 (pixel value is unknown). . As a result, high-quality distance image data I _fusion (x, y) with further reduced noise can be obtained.

In Expression (3), the case is classified according to whether the value of the measurement reliability map _RPN is 0 or not, but a predetermined threshold value (for example, a value near 0, for example, 0.01) ) May be defined and the measurement reliability map _RPN may be classified according to whether the value is greater than or less than the threshold value. In this case, the pixel value is set to 0 when the value is less than the threshold value.

FIG. 8A is a measurement result of the three-dimensional image measurement apparatus in FIG. 1 and is a diagram illustrating an example of a camera image of an object. FIG. 8B is a measurement result of the three-dimensional image measurement apparatus in FIG. 1 and shows an example of an original distance image based on photographing with a relatively short exposure time. Further, FIG. 8C is a diagram showing a measurement result of the three-dimensional image measurement apparatus of FIG. 1 and showing an example of a distance image calculated by Expression (2). Further, FIG. 8D is a measurement result of the three-dimensional image measurement apparatus of FIG. 1, and is a diagram showing an example of the distance image calculated by the equations (2) and (3).

Here, in FIGS. 8A to 8D, the left image is a black object, and the right image is an object having a white portion and a metal portion. In the original distance image shown in FIG. 8A, many pixels with unknown pixel values are generated. However, in the results of Expressions (2), (3), and (4) in FIGS. Are complemented with exact values. It shows remarkable results especially for black objects. In the result of FIG. 8C by Expression (2), pixel values that are completely inaccurate are mixed in a part of a white object or a metal part. However, in the result of FIG. 8D according to the equations (3) and (4), it can be seen that most of them are removed.

In addition, even when Expression (3) and Expression (4) are used (FIG. 8D), a pixel with an unknown pixel value may remain. In that case, interpolation may be performed from the peripheral pixel values. Any method may be used for the interpolation, and a bicubic method or a B-spline method may be used. Further, different distance conditions may be used to synthesize repeated distance images using Equation (2) or Equation (3) and Equation (4).

FIG. 9 is a flowchart showing a 3D image measurement process executed by the information processing apparatus 3 of the 3D image measurement apparatus of FIG. FIG. 10 is a flowchart showing measurement reliability evaluation processing (step S9) which is a subroutine of FIG. Further, FIG. 11 is a flowchart showing distance image synthesis processing (step S10) which is a subroutine of FIG.

In the three-dimensional image measurement process of FIG. 9, a pattern image is photographed by irradiating with a projection pattern twice with different photographing parameters P1, P2 (steps S1, S5) (steps S2-S3, S6-S7). Here, the parameters are the exposure time of the camera 1, the aperture value of the camera 1, the gain of the camera 1, the light projection intensity of the projector 2, the light projection direction of the projector 2, the intensity of ambient light at the time of shooting, and the environment at the time of shooting. It corresponds to the change of the direction of light. The shooting parameter P1 in FIG. 9 is a parameter setting in which the brightness value of the shot image becomes dark overall. If the exposure time of the camera 1 is short, the exposure time of the camera 1 is set short. From there, the projection pattern is irradiated from the projector 2, and the pattern image is taken while the camera 1 is synchronized. A distance image is created from the captured pattern image. Next, parameters are set so that the brightness value of the captured image becomes bright overall, and pattern projection, shooting, and distance image creation are performed. Based on the two shootings, the measurement reliability is evaluated (step S9), and the distance images are synthesized (step 10).

In the measurement reliability evaluation process of FIG. 10, the absolute difference between the high-frequency negative pattern image and the high-frequency positive pattern image is calculated and the absolute difference between the high-frequency negative pattern image and the high-frequency positive pattern image is acquired. Difference image data having a pixel value of the value is generated (step S11). The following steps S13 to S15 are performed using one unselected pixel as the selected pixel to be processed. If the difference absolute value of the selected pixel is 0 (YES in step S13), the measurement reliability value of the selected pixel is set to 0 and the process proceeds to step S16. On the other hand, if the difference absolute value of the selected pixel is 1 (NO in step S13), the measurement reliability value of the selected pixel is set to 1 (step S15), and the process proceeds to step S16. The processing from step S12 to S14 is repeated for all pixels of the processing target image (step S16), and if the result of calculating the measurement reliability for all the pixels is obtained (YES in step S16), a measurement reliability map is generated. (Step S17), the process returns to the original routine.

In the distance image compositing process of FIG. 11, first, an unselected pixel is set as a selected pixel to be processed (step S21), and the following steps S22 to S26 are performed. Here, if the pixel value of the distance image data (imaging parameter P1) is 0 (YES in step S22), the process proceeds to step S23, but if it is not 0 (NO in step S22; In step S24, the pixel value of the distance image data (imaging parameter P1) is added to the pixel value of the synthesized distance image data in step S24. Then, the process proceeds to step S27. If the pixel value of the measurement reliability map is 0 (YES in step S23), 0 is substituted into the pixel value of the synthesized distance image data, and the process proceeds to step S27. If not, the composition is not 0 (NO in step S23). The pixel value of the distance image data (imaging parameter P2) is substituted into the pixel value of the distance image data thus obtained, and the process proceeds to step S27. The processing from step S21 to S26 is repeated for all pixels of the processing target image (step S27), and if the result of combining the distance image data is obtained for all pixels (YES in step S27), the combined distance image data is generated. And output (step S28), the process returns to the original routine.

As described above, in the distance image composition processing of FIG. 11, the distance image data of the shooting parameter P1 is obtained based on the distance image data obtained from the shooting of the shooting parameter P1 adjusted so that the overall luminance value becomes dark. If the pixel value is not 0 (meaning that the distance value is unknown) (NO in step S22), the pixel value of the distance image data of the shooting parameter P1 is set as the pixel value of the synthesized distance image data (step). S24). If the pixel value of the distance image data of the shooting parameter P1 is 0 (YES in step S22), the measurement reliability map created using the image data of FIG. 4 is referred to. If the pixel value of the corresponding pixel is not 0 in the measurement reliability map (NO in step S23), the pixel value of the distance image data of the shooting parameter P2 is set as the pixel value of the synthesized distance image (step S26). . If the pixel value of the corresponding pixel is 0 (YES in step S23), the corresponding pixel value of the synthesized distance image is set to 0 (step S25).

For pixels with a pixel value of 0 in the measurement reliability map, the same procedure is used for the distance image data captured based on the combined distance image and captured with different shooting parameters P3 (different from the shooting parameters P1 and P2). A distance image with fewer missing measurements may be created by combining the images.

As described above, it is possible to obtain synthesized distance image data I _fusion with few pixels with unknown pixel values and little noise, and high quality even in difficult measurement environments such as scenes where white and black objects are mixed A range image can be obtained.

Embodiment 2. FIG.
FIG. 12 is a block diagram showing a hardware configuration of the three-dimensional image measuring apparatus according to the second embodiment of the present invention. 12, the three-dimensional image measurement apparatus according to the second embodiment is different from the three-dimensional image measurement apparatus according to the first embodiment in FIG. 1 in the following points.
(1) The information processing apparatus 3 further includes an interface circuit 38 connected to the bus 30.
(2) The mobile device 4 further includes a movable device 4 connected to the interface circuit 38 and moving an object to be photographed, for example, in three dimensions.
Hereinafter, the differences will be described in detail.

In FIG. 12, the CPU 31 of the information processing device 3 transmits a predetermined movement command signal to the movable device 4 via the interface circuit 38, and in response to this, the movable device 4 sends a response signal (for example, an ACK signal) to the CPU 31. Reply to The movable device 4 moves the object to be photographed in, for example, a three-dimensional manner based on the movement command signal, so that the exposure time of the camera 1, the aperture value of the camera 1, the gain of the camera 1, the light projection intensity of the projector 2, Not only can the projection direction of the projector 2 be changed, but also the intensity of ambient light during shooting and the direction of ambient light during shooting can be changed, so that shooting conditions can be changed more flexibly.

Here, ambient light is light emitted from other than the projector 2, and includes sunlight, electric light, thermal radiation including flame, and luminescence. The intensity of the ambient light can be changed by changing the position of the oblique light means. The oblique light means is a bag or a cover, and a movable device including an industrial robot may be used.

The movable device 4 is, for example, an actuator. The movable device 4 itself may function as a shielding device that shields ambient light from the camera and the projector. Moreover, the movable device 4 may be composed of a plurality of actuators. In that case, ambient light from a plurality of light sources can be blocked. The movable device 4 may be a multi-axis movable device such as a robot arm composed of a plurality of actuators. A hand such as a gripper may be attached to the tip. In this case, the robot arm may change the posture, and the hand may block the light source of ambient light. Alternatively, the ambient light source may be blocked by holding a shielding object such as a peripheral plate or translucent ground glass by a robot arm and changing the posture of the shielding object. Alternatively, the direction of ambient light may be changed by holding a mirror. Alternatively, the ambient light illumination may be turned on and off by operating a switch that controls the light source of the ambient light with a robot arm.

As described above, the movable device 4 may have any configuration as long as the change in the intensity and direction of the ambient light is realized. Even with a snake-shaped robot or an actuator that repeatedly expands or contracts by air, a similar action to ambient light is realized, so that it is obvious that the movable device 4 can be configured.

Also in this embodiment, the same three-dimensional image measurement process as in FIG. 9 is executed. Here, in the adjustment of the shooting parameters P1 and P2 in steps S1 and S5 of FIG. 9, by controlling the ambient light, the brightness value of the image can be adjusted to become darker or brighter as a whole. . Further, when the projector 2 is visible light, it is strongly influenced by environmental light. Therefore, in a state where the ambient light is blocked by the movable device 4 or in a state where the ambient light is constant, the exposure time of the camera 1, the aperture value of the camera 1, the gain of the camera 1, the light projection intensity of the projector 2, By changing the projection direction of the projector 2 and adjusting the shooting parameters, it is possible to create a high-quality distance image with less noise.

Embodiment 3 FIG.
FIG. 13 is a block diagram showing a hardware configuration of the three-dimensional image measurement apparatus according to the third embodiment of the present invention. 13, the three-dimensional image measurement apparatus according to the third embodiment is different from the three-dimensional image measurement apparatus according to the first embodiment in FIG. 1 in the following points.
(1) The information processing apparatus 3 further includes an interface circuit 39.
(2) The communication device 5 further connected to the interface circuit 39 is further provided.
(3) A communication device 6 connected to the communication device 5 via the communication line 9 is further provided.
(4) It further includes an illumination device 7 that is connected to the communication device 6 and illuminates the object.
Hereinafter, the difference will be described in detail.

In FIG. 13, the interface circuit 39 of the information processing device 3 is connected to the lighting device 7 through the communication device 5, the communication line 9, and the communication device 6 to form a command signal communication path. The CPU 1 of the information processing device 3 controls the operation of the lighting device 7 by transmitting a lighting command signal to the lighting device 7 via the command signal communication path, and in response to this, the lighting device 7 A response signal (for example, ACK signal) corresponding to this is transmitted to the CPU 31 via the command signal communication path. Thereby, similarly to Embodiment 2, also in this Embodiment, the ambient light with respect to a target object can be controlled.

Here, the communication device 5 and the communication device 6 are respectively a wireless communication device or a wired communication device. The wireless communication device is, for example, a wireless router device. The wired communication device is a communication device having a communication port such as an Ethernet (registered trademark) port, a USB port, and a serial port.

The illuminating device 7 can change one or more of the intensity and orientation of the irradiation with respect to the object based on the illumination command signal from the CPU 1 of the information processing device 3, and the on / off time can be set by pulse width modulation. It includes at least one of an intensity control unit that makes it possible to adjust the intensity, such as control, and a movable unit that changes the direction of the intensity light.

FIG. 14 is a flowchart showing a three-dimensional image measurement process executed by the information processing apparatus 3 of the three-dimensional image measurement apparatus in FIG. The three-dimensional image measurement process of FIG. 14 differs from the three-dimensional image measurement process of FIG. 9 in the following points.
(1) In place of step S1, the process includes a process of step S31 “weakening the irradiation intensity of the illumination device 7 so as to be equal to or less than a predetermined threshold value Th1, or setting the posture in such a manner”.
(2) In place of step S5, the process includes the process of step S32 that “intensifies the irradiation intensity of the illumination device 7 to be equal to or higher than a predetermined threshold Th2 (> Th1) or sets the posture in such a manner”
Hereinafter, the difference will be described in detail.

14, based on the illumination command signal from the CPU 31 of the information processing apparatus 3, the intensity and / or posture of the illumination apparatus 7 is changed by controlling the intensity control unit or the movable part of the illumination apparatus 7. At this time, the luminance value of the entire image of the target object becomes dark and the illumination intensity is weakened so as to be equal to or lower than a predetermined threshold value Th1, or the illumination posture is changed so as not to irradiate directly in the shooting direction of the camera. To set.

14, based on the illumination command signal from the CPU 31 of the information processing device 3, the intensity and / or posture of the illumination device 7 is changed by controlling the intensity control unit or the movable unit of the illumination device 7. At this time, the overall luminance value of the image of the object becomes brighter, and the intensity is increased by increasing the irradiation intensity to be equal to or higher than a predetermined threshold value Th2 (> Th1) or directly in the shooting direction of the camera 1. Change the posture so that

As described above, according to the present embodiment, the ambient light can be controlled using the illumination device 7, and a high-quality distance image can be created.

Embodiment 4 FIG.
FIG. 15 is a block diagram showing a hardware configuration of the three-dimensional image measuring apparatus according to the fourth embodiment of the present invention. The three-dimensional image measurement apparatus according to the fourth embodiment in FIG. 15 is different from the three-dimensional image measurement apparatus according to the second embodiment in FIG. 12 in the following points.
(1) Instead of the movable device 4, a movable control device 8 having a function of controlling the operations of the camera 1 and the projector 2 as well as the function of the movable device 4 is provided.
Hereinafter, the difference will be described in detail.

In FIG. 15, the CPU 31 of the information processing device 3 transmits a predetermined movement command signal to the movable control device 8 via the interface circuit 38, and the movable control device 8 responds to the response signal (for example, ACK signal). Is returned to the CPU 31. The movable control device 8 changes the positions and postures of the camera 1 and the projector 2 using the imaging command signal and the projection command signal based on the movement command signal from the CPU 31 of the information processing device 3. In response to this, the camera 1 returns a response signal (for example, ACK signal) to the movable control device 8, and the projector 2 returns a response signal (for example, ACK signal) to the movable control device 8. The movable control device 8 may have a configuration called a hand eye in which the robot arm, the camera 1 and the projector 2 are mounted on their hands. In addition, the movable control apparatus 8 may be comprised with the some actuator so that each position and attitude | position of the camera 1 and the light projector 2 can each be changed independently.

When objects to be photographed include objects with different surface states such as black objects and white objects, the distance image measurement value unknown and inaccurate measurement are suppressed by the distance image synthesis method proposed in the first embodiment. be able to. On the other hand, there may be cases where the measurement value is unknown or inaccurate due to the positional relationship between the camera 1 and the projector 2 and the object, and the light of the projector 2 cannot be observed due to the shielding as viewed from the camera 1. In such a case, moving the camera 1 and the projector 2 with the movable control device 8 can further eliminate unknown measurement values and inaccurate measurement. At this time, in the measurement reliability map generated by the measurement reliability evaluation process of FIG. 10, the line-of-sight direction is maintained so as to keep an eye on the area where the pixel value of the measurement reliability map is low and the reliability is low. By changing only the irradiation direction, it is possible to efficiently re-measure the location where the measurement value is unknown or inaccurate due to the light of the projector 2 being blocked. At that time, by giving priority to a large area in which the pixel value of the measurement reliability map is 0, it is possible to re-measure pixels with unknown measurement values and inaccurate measurement more efficiently.

As described above, according to the present embodiment, by providing the movable control device 8, the measurement value is unknown and inaccurate due to different surface conditions, and the positional relationship between the camera 1 and the projector 2 and the target object. A high-quality distance image that efficiently suppresses both unknown and inaccurate measurement values can be generated.

Modified example.
In the above embodiment, the camera 1 is used. However, the present invention is not limited to this, and may be a sensor such as a camera, a CCD sensor, or an imaging sensor that can capture an image of an object.

1 camera, 2 projector, 3 information processing device, 4 movable device, 5 communication device, 6 communication device, 7 illumination device, 8 movable control device, 9 communication line, 10 projection pattern control unit, 11 image acquisition control unit, 12 distance Restoration unit, 13 Measurement reliability evaluation unit, 14 Distance image composition unit, 15 Projection plane, 21 Camera image, 22 Distance image, 23 Positive pattern image, 24 Negative pattern image, 25-28 Measurement image, 30 Bus, 31 Central calculation Processing device (CPU), 32 read only memory (ROM), 33 random access memory (RAM), 34 input device, 35 display device, 36-39 interface circuit, 41 camera image, 42 distance image, 43 camera image, 44 distance image.

Claims

A three-dimensional image measuring device that generates distance image data representing a distance from the sensor to the object as a pixel value from pattern image data obtained by photographing a pattern projected onto the object to be photographed by the projector. In
Control means for complementing a pixel whose pixel value is unknown in distance image data obtained from pattern image data photographed by the sensor with a pixel value of a distance image obtained from pattern image data photographed by the sensor under different photographing conditions A three-dimensional image measuring apparatus characterized by that.
The control means includes an exposure time of the sensor, an aperture value of the sensor, a gain of the sensor, a light projection intensity of the projector, a light projection direction of the projector, a position of ambient light at the time of shooting, and a position of ambient light at the time of shooting. The three-dimensional image measurement apparatus according to claim 1, wherein different photographing conditions are realized by changing at least one of the directions.
The control means includes
Generates a pattern image captured by the sensor under the different imaging conditions, and a positive pattern image data and a negative pattern image data, which are paired image data obtained by reversing the brightness of the pattern using a spatial encoding method. And
Generating difference image data having a pixel value of a difference absolute value obtained from a difference between the positive pattern image data and the negative pattern image data;
Only when the pixel value of the difference image data is not 0 or higher than a predetermined threshold value, a pixel whose pixel value is unknown in the distance image obtained from the pattern image captured by the sensor is determined under the different conditions. 3. The three-dimensional image measurement apparatus according to claim 2, wherein interpolation is performed using a pixel value of a distance image obtained from a pattern image photographed by the sensor.
4. The three-dimensional image measurement apparatus according to claim 3, wherein the generated positive pattern image and negative pattern image include a high frequency band pattern.
The control means controls at least one of each position and orientation of the sensor and the projector based on measurement reliability obtained from a difference between the generated positive pattern image and negative pattern image. The three-dimensional image measurement apparatus according to claim 4.
The control means is a pattern image in which a pixel whose pixel value is unknown in a distance image obtained from a pattern image photographed at a predetermined first exposure time is photographed at a second exposure time longer than the first exposure time. The three-dimensional image measurement apparatus according to claim 4, wherein interpolation is performed using a pixel value of a distance image obtained from the above.
The three-dimensional image measurement apparatus includes a movable device connected to the control unit and configured to realize the different photographing conditions by moving the object based on a command signal from the control unit. The three-dimensional image measurement apparatus according to any one of claims 1 to 6.
The three-dimensional image measurement device is an illumination device that is connected to the control means and illuminates the object, and changes illumination conditions for illuminating the object based on a command signal from the control means. The three-dimensional image measurement apparatus according to any one of claims 1 to 6, further comprising an illumination device that realizes the different photographing conditions.
The three-dimensional image measuring device is a movable control device that is connected to the control means, the sensor, and the projector and moves an object, and based on a command signal from the control means, the sensor and the projector. The three-dimensional image measurement according to any one of claims 1 to 6, further comprising an illumination device that realizes the different photographing conditions by changing at least one of a position and a posture. apparatus.
Control means for generating distance image data in which the distance from the sensor to the object is represented by a pixel value from pattern image data obtained by photographing the pattern projected onto the object to be photographed by the projector. In a 3D image measurement method for a 3D image measurement apparatus,
The control means complements a pixel whose pixel value is unknown in the distance image data obtained from the pattern image data photographed by the sensor with a pixel value of the distance image obtained from the pattern image data photographed by the sensor under different photographing conditions. A three-dimensional image measurement method comprising the step of:
In the step of complementing, the control means includes an exposure time of the sensor, an aperture value of the sensor, a gain of the sensor, a light projection intensity of the light projector, a light projection direction of the light projector, a position of ambient light at the time of photographing, The three-dimensional image measurement method according to claim 10, wherein different shooting conditions are realized by changing at least one of the directions of ambient light during shooting.
In the step of complementing, the control means
Generates a pattern image captured by the sensor under the different imaging conditions, and a positive pattern image data and a negative pattern image data, which are paired image data obtained by reversing the brightness of the pattern using a spatial encoding method. And
Generating difference image data having a pixel value of a difference absolute value obtained from a difference between the positive pattern image data and the negative pattern image data;
Only when the pixel value of the difference image data is not 0 or higher than a predetermined threshold value, a pixel whose pixel value is unknown in the distance image obtained from the pattern image captured by the sensor is determined under the different conditions. The three-dimensional image measurement method according to claim 11, wherein interpolation is performed using a pixel value of a distance image obtained from a pattern image photographed by the sensor.
13. The three-dimensional image measurement method according to claim 12, wherein the generated positive pattern image and negative pattern image include a high frequency band pattern.
In the complementing step, the control means determines at least one of the positions and orientations of the sensor and the projector based on the measurement reliability obtained from the difference between the generated positive pattern image and negative pattern image. The three-dimensional image measurement method according to claim 13, wherein control is performed.
In the complementing step, the control means applies a second exposure longer than the first exposure time to a pixel whose pixel value of the distance image obtained from the pattern image taken at the predetermined first exposure time is unknown. The three-dimensional image measurement method according to claim 13, wherein interpolation is performed using pixel values of a distance image obtained from a pattern image photographed over time.