JP6346427B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to a distance measurement technique.

  2. Description of the Related Art A distance measuring device that projects pattern light onto an object using a projection unit such as a projector and measures the distance to the object based on an image captured by an imaging unit such as a camera is known. Furthermore, a recognition processing device is known that estimates the position / posture of a plurality of stacked objects based on the distance measurement result of the distance measurement device. By outputting the estimation result of the position / orientation of the object by the recognition processing device to the robot control device, it is possible to construct a system that automatically picks up the stacked objects by the robot.

  Assuming a scene in which many objects are piled up, the distance from the device to the object changes according to the height of the pile. When the distance from the apparatus to the object changes, the illuminance of the projection pattern light and the brightness of the captured image change. Specifically, the brightness of the captured image becomes darker in proportion to the square of the distance from the apparatus to the object.

  Furthermore, a drop in the amount of peripheral light occurs depending on the position of the object in the captured image and the position in the projection range, and the brightness of the captured image changes. That is, it is brighter near the center of the screen and darker as it goes to the periphery. For example, the captured image becomes dark based on the cosine fourth law.

  As described above, the image luminance gradation value in the captured image of the pattern light projected from the projection unit varies depending on the distance to the object and the position in the captured image. When the captured image becomes dark, the influence of noise of the image sensor increases and the accuracy of distance measurement decreases. The decrease in the distance measurement accuracy also reduces the accuracy in estimating the position / orientation of the object by the recognition processing device. Therefore, in the picking system, a failure such as failure to pick the object by the robot may occur.

  In order to accurately measure the distance regardless of the distance and the position in the captured image, the brightness adjustment of the pattern light and the exposure amount adjustment of the imaging unit are appropriately performed according to the distance and the position in the screen. There is a need. If these adjustments are inappropriate, the pattern light in the captured image is saturated or blackened, and distance measurement becomes impossible. Moreover, if the adjustment is insufficient, even if distance measurement is possible, the measurement accuracy decreases.

  Patent Document 1 discloses a method of disposing an element having a low transmittance at the central portion and a higher transmittance toward the periphery so as to cancel the influence of a decrease in peripheral light amount. Patent Document 2 discloses a range finder device that controls the light output of the light source unit based on the measurement result of the approximate distance.

JP 2000-292133 A JP 2000-241131 A

  In the method disclosed in Patent Document 1, since the illumination intensity is constant with respect to an object placed at the same distance, it is possible to suppress deterioration in accuracy due to a decrease in peripheral light amount. On the other hand, the illuminance is not constant for objects having different depths. Therefore, the method disclosed in Patent Document 1 causes accuracy degradation.

  In the method disclosed in Patent Document 2, although the exposure amount of the entire screen is adjusted based on the approximate distance, an optimal exposure amount adjustment can be realized for an object placed at a specific position in the projection area. I can't say.

  This invention is made | formed in view of such a problem, and provides the technique for implementing the distance measurement with respect to each target object with higher precision.

According to one aspect of the present invention, an acquisition unit that acquires a captured image obtained by capturing an image of a plurality of objects on which a pattern is projected by a projection unit;
Derivation means for deriving distances from the imaging means to each of the plurality of objects using the captured image;
Of the first parameter for controlling the brightness and darkness of the pattern projected from the projection means and the second parameter for controlling the exposure amount of the imaging means, the adjustment amount of the adjustment target is obtained from the imaging means. And adjusting means for adjusting the adjustment target in accordance with the calculated adjustment amount, based on the distance to each of the plurality of objects .

  According to the configuration of the present invention, distance measurement for each object can be performed with higher accuracy.

The block diagram which shows the function structural example of the target object picking system 8. FIG. The figure which shows the projection pattern (pattern light) in a 4-bit spatial encoding method. The figure which shows 4 bit Gray code. FIG. 3 is a schematic diagram illustrating a relationship between an imaging unit 20, a projection unit 30, and an object 5. The figure explaining the specific precision of a pattern position. The figure explaining the change of the height of the several components piled up. The figure explaining the relationship between the luminance value in a captured image, and the distance to an imaging target. 5 is a flowchart of processing performed by the image processing unit 40. The figure explaining the process of step S106. 5 is a flowchart of processing performed by the image processing unit 40. The figure explaining the triangulation correction amount Czt. The block diagram which shows the function structural example of the target object picking system 8. FIG. The figure explaining the relationship between the position and orientation of an object and the angle of view θ relative to the object. 5 is a flowchart of processing performed by the image processing unit 40. The flowchart which shows the detail of the process in step S300. The flowchart which shows the detail of the process in step S310. The flowchart which shows the detail of the process in step S102. FIG. 3 is a block diagram illustrating an example of a hardware configuration of an apparatus applicable to the image processing unit 40.

  Preferred embodiments of the present invention will be described below with reference to the accompanying drawings. The embodiment described below shows an example when the present invention is specifically implemented, and is one of the specific examples of the configurations described in the claims.

[First Embodiment]
First, a functional configuration example of the object picking system 8 according to the present embodiment will be described with reference to the block diagram of FIG. The object picking system 8 includes an image processing apparatus (distance measuring apparatus) 1, an object position / orientation recognition unit 7 that recognizes the position / orientation of the object 5 using a distance measurement result obtained by the image processing apparatus 1, and a result of the recognition. A robot controller 62 and a robot 61 are used to control the robot 61 using the robot 61. The robot 61 is for picking the object 5.

  The image processing apparatus 1 controls the projection unit 30 that projects pattern light onto the object 5, the imaging unit 20 that images the object 5 onto which the pattern light is projected, the projection unit 30 and the imaging unit 20, and the spatial code And an image processing unit 40 that measures the distance to the object 5 based on the conversion method.

  First, the projection unit 30 will be described. As shown in FIG. 1, the projection unit 30 includes a light source 31, an illumination optical system 32, a display element 33, a projection stop 34, and a projection optical system 35.

  The light source 31 is various light emitting elements such as a halogen lamp and an LED. The illumination optical system 32 is an optical system having a function of guiding light emitted from the light source 31 to the display element 33. For example, an optical system suitable for uniform luminance such as Koehler illumination or a diffusion plate is used. The display element 33 is an element having a function of spatially controlling the transmittance or reflectance of light from the illumination optical system 32 according to the supplied pattern. For example, the display element 33 is a transmissive LCD, a reflective LCOS. DMD or the like is used. Regarding the supply of the pattern to the display element 33, a plurality of types of patterns held in the projection unit 30 may be sequentially supplied to the display element 33, or may be held in an external device (for example, the image processing unit 40). A plurality of types of patterns may be sequentially acquired and supplied to the display element 33. The projection optical system 35 is an optical system configured to form an image of light (pattern light) guided from the display element 33 at a specific position of the object 5. The projection stop 34 is used to control the F number of the projection optical system 35.

  Next, the imaging unit 20 will be described. As illustrated in FIG. 1, the imaging unit 20 includes an imaging optical system 23, an imaging aperture 22, and an imaging element 21. The imaging optical system 23 is an optical system configured to image a specific position of the object 5 on the imaging element 21. The imaging aperture 22 is used to control the F number of the imaging optical system 23. The imaging element 21 is various photoelectric conversion elements such as a CMOS sensor and a CCD sensor. The analog signal photoelectrically converted by the imaging device 21 is sampled and quantized by a control unit (not shown) in the imaging unit 20 and converted into a digital image signal. Further, the control unit generates an image (captured image) in which each pixel has a luminance gradation value (density value, pixel value) from the digital image signal, and appropriately stores the captured image in a memory or an image in the image capturing unit 20. The data is sent to the processing unit 40.

  The imaging unit 20 images the object 5 every time the projection unit 30 projects different pattern light. That is, the imaging unit 20 and the projection unit 30 operate in synchronization, and as a result, the imaging unit 20 can capture images of the object 5 onto which different pattern lights are projected.

  Next, the image processing unit 40 will be described. As shown in FIG. 1, the image processing unit 40 includes an adjustment unit 41, an approximate distance calculation unit 42, a distance calculation unit 43, a pattern light brightness control unit 44, and a captured image exposure amount control unit 45.

  The adjustment unit 41 determines parameters for controlling the brightness and darkness of the captured image that the imaging unit 20 captures next time. The “parameters for controlling the brightness and darkness of the captured image captured by the image capturing unit 20 next time” include a parameter for the image capturing unit 20 and a parameter for the projection unit 30, both of which may be determined, or one of which may be determined. May be.

  The distance calculation unit 43 acquires a captured image captured by the imaging unit 20 each time the projection unit 30 projects each of a plurality of types of pattern light. Then, the distance calculation unit 43 uses the spatial code decoded from the acquired captured image and the pattern position specified for the acquired captured image, based on the principle of triangulation, the imaging unit 20 (imaging optical system 23). The distance from the object 5 to the object 5 is measured (derived). The approximate distance calculation unit 42 obtains an approximate distance from the distance calculated by the distance calculation unit 43.

  When the adjustment unit 41 determines “a parameter for the projection unit 30”, the pattern light brightness control unit 44 controls the brightness of the pattern light projected from the projection unit 30 according to the parameter. There are, for example, the following three methods for controlling the brightness of the pattern light.

  The first method is to control the light emission luminance of the light source 31. In the case of a halogen lamp, the emission luminance increases as the applied voltage is increased. In the case of an LED, the emission brightness increases as the current flowing is increased. In such a case, however, the “parameter for the projection unit 30” is a parameter for controlling the light emission luminance of the light source 31.

  The second method is to control the display gradation value of the display element 33. In the case of an LCD, increasing the display gradation value increases the transmittance, resulting in brighter pattern light. In the case of LCOS, when the display gradation value is increased, the reflectance increases, and as a result, the pattern light becomes brighter. In the case of DMD, if the display gradation value is increased, the number of times of ON per frame increases, and as a result, the pattern light becomes brighter. In such a case, however, the “parameter for the projection unit 30” is a parameter for controlling the display gradation value of the display element 33.

  The third method is to control the projection diaphragm 34. By opening the aperture, the pattern light becomes brighter. On the contrary, the pattern light becomes dark by reducing the aperture. When controlling the projection stop 34, since the depth of field of the projection optical system also changes, it is necessary to consider its influence. Specifically, the resolution of the pattern light is evaluated on each projection stop based on an index such as the contrast of the image luminance waveform, and it is preferable to confirm whether or not a significant deterioration in accuracy occurs when performing distance measurement. In such a case, however, the “parameter for the projection unit 30” is a parameter for controlling the projection stop 34.

  When the adjustment unit 41 determines “a parameter for the image capturing unit 20”, the captured image exposure amount control unit 45 controls the exposure amount at the time of image capturing in the image capturing unit 20 according to the parameter. Controls the luminance gradation value of the image. There are, for example, the following three methods for controlling the exposure amount of the imaging unit 20.

  The first method is to control the exposure time (shutter speed) of the image sensor 21. When the exposure time is lengthened, the luminance gradation value of the captured image increases, and conversely, when the exposure time is shortened, the luminance gradation value decreases. In such a case, however, the “parameters for the image capturing unit 20” are parameters for controlling the exposure time (shutter speed) of the image sensor 21.

  The second method is to control the imaging diaphragm 22. By opening the aperture, the brightness gradation value of the captured image increases, and conversely, by reducing the aperture, the brightness gradation value of the captured image decreases. When controlling the imaging diaphragm 22, the depth of field of the imaging optical system also changes, and it is necessary to consider its influence. Specifically, in each imaging stop, it is preferable to evaluate the resolving power of the pattern light based on an index such as the contrast of the image luminance waveform, and confirm whether or not there is a significant deterioration in accuracy when performing distance measurement. However, in such a case, the “parameters for the imaging unit 20” are parameters for controlling the imaging diaphragm 22.

  The third method is a method in which the same pattern is projected a plurality of times to capture an image, and the image brightness value is added (the captured image is synthesized each time the pattern is projected). In the method of extending the exposure time, image saturation occurs due to the dynamic range of the sensor. On the other hand, as described above, if the same pattern is projected a plurality of times and each image obtained by imaging each time of the projection is added to generate one composite image, while preventing image saturation, The amount of exposure can be increased. In the composite image obtained by such a method, the noise component of the image sensor is reduced. Specifically, assuming that the noise amount of the image sensor at the time of single image capturing is σb, the noise amount when N images are captured and added is σb / √N. Although it is described that the image is captured a plurality of times and added, the same effect can be obtained by a method of calculating an average image of images captured a plurality of times.

  The object 5 is a set of objects stacked at random.

  The object position / orientation recognition unit 7 uses the distance measurement result of the image processing apparatus 1 to detect the position / orientation of the object 5 in the coordinate system (robot coordinate system) handled by the robot 61 (actually, the object 5 is imaged). Recognize each object) that can be observed from the unit 20. Various known recognition methods can be employed as the position and orientation recognition method. For example, a method of recognizing the position and orientation of an object by matching processing of 3D point cloud data based on a distance measurement result and a CAD model may be employed.

  The object position / posture recognition unit 7 may be a unit in the image processing apparatus 1 or may be a separate device from the image processing apparatus 1. In any case, any form can be adopted as long as the position and orientation of the object 5 can be recognized from the distance measurement result by the image processing apparatus 1 and the recognized position and orientation can be sent to the robot controller 62. good.

  The robot control unit 62 controls the robot 61 in accordance with the position and orientation recognized by the object position and orientation recognition unit 7, and causes the robot 61 to pick an object that has recognized the position and orientation. The picked object is carried to the next process. As the object is picked, the height of the pile gradually decreases. Therefore, the distance from the imaging unit 20 also increases.

  In the present embodiment, as described above, distance measurement is performed based on the spatial encoding method, and there are a plurality of derivatives in this spatial encoding method. In the present embodiment, the method disclosed in JP 2011-47931 A is used. The principle will be briefly described below.

  FIG. 2 shows a projection pattern (pattern light) in the 4-bit spatial encoding method. The projection pattern (all lighting pattern) shown in FIG. 2 (a) and the projection pattern (all lighting pattern) shown in FIG. 2 (b) are used to detect a shadow area and determine a light / dark judgment threshold. Is called. The projection patterns (1 to 4 bit positive patterns, respectively) in FIGS. 2C to 2F are used to determine the spatial code. The projection pattern (4-bit negative pattern) shown in FIG. 2G is used for specifying the pattern position with high accuracy together with the 4-bit positive pattern.

In the case of 4 bits, the number of spatial codes is 16. If the number of bits of the projection pattern is N, the number of spatial codes is ^2N . The projection pattern of 1 to 4 bits shown in FIG. 2 determines the arrangement of light and dark (black and white) patterns based on a code called a Gray code. FIG. 3 shows a 4-bit gray code. Since the gray code is a pattern in which the Hamming distance between adjacent codes is 1, an error can be minimized even if the code determination is wrong. For this reason, it is often used as a projection pattern of the spatial encoding method.

  Next, a method for decoding a spatial code from a captured image will be described. The 1-bit to 4-bit projection patterns shown in FIG. 2 are projected onto the measurement object, and an image of the measurement object on which each projection pattern is projected is captured. Then, from the luminance gradation value of each pixel in each captured image obtained by this imaging, whether the pixel is a portion where the bright portion of the projection pattern is irradiated or a portion where the dark portion is irradiated Determine if there is. As a determination criterion for this determination, an average value of luminance gradation values of all lighting patterns and all lighting patterns can be used. That is, for each pixel in the captured image, if the luminance gradation value of the pixel is equal to or greater than the average value, the pixel is determined to be a bright portion, and if the pixel is less than the average value, the pixel is determined to be a dark portion. A bit value “1” is assigned to a pixel determined to be a bright portion, and a bit value “0” is assigned to a pixel determined to be a dark portion. By performing such processing on each of the captured images of the projection pattern of 1 to 4 bits, a bit value for 4 bits (gray code for 4 bits) for each pixel in the captured image. Will be given. By converting the 4-bit gray code into a spatial code, the emission direction from the projection unit can be uniquely specified. As shown in FIG. 3, for example, when a code “0110” is assigned to a certain pixel, the gray code is 6. When this gray code is converted into a spatial code, 4 is obtained.

  As shown in FIG. 4, the incident direction of light from the pixel position of the captured image to the imaging unit (more specifically, the main point position of the imaging unit) is determined. When the spatial code of the pixel position is determined, the emission direction from the projection unit is specified. When the two directions are determined, distance measurement with respect to the measurement object can be performed based on the principle of triangulation. The above is the principle of distance measurement based on the spatial coding method.

  In the spatial encoding method, errors can be reduced by using together pattern light (in the case of FIG. 2, the 4-bit negative pattern of (g)) in which the brightness of the projection pattern is inverted. A method for reducing the error will be described with reference to FIG.

  Since the emission direction from the projection unit has a width corresponding to one space code, a distance measurement error is generated by the amount indicated by ΔZ. In order to reduce the error, a negative pattern (FIG. 2) in which the lightness and darkness of the projection pattern of the least significant bit (the fourth bit in FIG. 2) (the 4-bit positive pattern of (f) in FIG. 2) is inverted is shown. In the case of 2, (g) 4-bit negative pattern) is further projected. Then, a luminance value waveform composed of the luminance value of each pixel composing the captured image of the object projected with the positive pattern and a luminance value waveform composed of the luminance value of each pixel composing the captured image of the object projected with the negative pattern The position of the intersection of and is the pattern position. This pattern position is used for the distance measurement. Since the intersection position is a boundary position with an adjacent space code, ideally, it does not have a width. In reality, quantization and sampling are performed when capturing as a digital image signal, and the image sensor noise is included in the luminance gradation value of the captured image, so the width is not zero, but the distance measurement error is greatly increased. Can be reduced. The above is the description of the method for reducing the distance measurement error.

  Next, the pattern position specifying accuracy will be described with reference to FIG. FIG. 5A is a graph in which the horizontal axis represents image coordinates and the vertical axis represents luminance gradation values. The image coordinates on the horizontal axis are the coordinates in the direction perpendicular to the direction of the stripes of the projection pattern in the captured image. A solid line represents a luminance value waveform (positive pattern waveform) composed of luminance values of pixels on one line along a direction perpendicular to the stripe direction of the positive pattern in the captured image of the object on which the positive pattern is projected. Indicates. A dotted line indicates a luminance value waveform (negative pattern waveform) composed of luminance values of pixels on one line along a direction perpendicular to the stripe direction of the negative pattern in the captured image of the object on which the negative pattern is projected. Indicates. Each waveform is a luminance value waveform composed of the luminance value of each pixel on the line at the same position in each captured image. FIG. 5B shows an enlarged view of each waveform in the area surrounded by the thick solid line.

  In FIG. 5B, attention is paid to four points near the intersection of the positive pattern waveform and the negative pattern waveform (all on the positive pattern waveform or on the negative pattern waveform). The luminance gradation value at the point on the positive pattern waveform is PL and the luminance gradation value at the point on the negative pattern waveform among the two points on the left side from the intersection position (both are the same image coordinates (value on the horizontal axis)). NL. Also, PR is the luminance gradation value at the point on the positive pattern waveform among the two points on the right side from the intersection position (both are the same image coordinates (value on the horizontal axis)), and the luminance gradation at the point on the negative pattern waveform The value is NR. In addition, the absolute value of the luminance gradation value difference (PL-NL) is DL, and the absolute value of the luminance gradation value difference (PR-NR) is DR.

  Then, the luminance waveform in the vicinity of the intersection is linearly approximated (between the point of the luminance gradation value PL and the point of the luminance gradation value PR, between the point of the luminance gradation value NL and the point of the luminance gradation value NR, And the intersection position C (image coordinates) of each straight line is expressed by the following (Equation 1).

C = L + DL / (DL + DR) (Formula 1)
Here, L is the image coordinate (value on the horizontal axis) of the point of the luminance gradation value PL (the point of the luminance gradation value NL). In order to simplify the description below, the intersection position C is described as a relative position with reference to the image coordinates of the point of the luminance gradation value PL (the point of the luminance gradation value NL), so that C = DL / ( DL + DR).

  Here, it is assumed that the luminance gradation values PL, PR, NL, and NR have fluctuations by the image sensor noise σ. At this time, the intersection position also fluctuates by the width ΔC. This width is the subpixel position estimation error at the intersection. The subpixel position estimation error ΔC can be obtained by the following equation.

ΔC = σ / (DL + DR) (Formula 2)
The smaller the subpixel position estimation error ΔC at the intersection, the smaller the distance measurement error. From (Equation 2), it can be seen that in order to reduce the subpixel position estimation error ΔC, the denominator (the value of DL or DR which is the difference between the luminance gradation values) may be increased. The luminance gradation value difference DL (DR) increases in proportion to the luminance gradation value. Furthermore, among the image sensor noise σ, the shot noise component often exhibits a characteristic proportional to the square root value of the luminance gradation value. Therefore, when the luminance gradation value is increased, the influence of the noise σ is relatively small. Get smaller. In order to increase the luminance gradation value, there are two methods of increasing the exposure amount in the imaging unit 20 or increasing the brightness of the pattern light.

  Both of these two methods may be executed, or only one of them may be executed. In the present embodiment, as an example, only the method for controlling the exposure amount in the imaging unit 20 (with the exposure amount as an adjustment target) is executed. Further, in the exposure amount control method, only the exposure time is controlled. However, the control of the exposure amount can be realized by a method other than the control of the exposure time, and is not limited to a specific method. Of the exposure amount control methods, the present invention can also be applied to a method for controlling the imaging aperture 22. The present invention can also be applied to the method of controlling the brightness of the pattern light, the control of the light emission luminance of the light source 31, the control of the display gradation value of the display element 33, and the control of the projection stop 34. Further, the method is not limited to a method of controlling only a single control amount, and a plurality of control amounts can be controlled simultaneously.

If many components of the noise σ are shot noise, in order to reduce the error to 1 / n, it is necessary to increase the brightness of the pattern light or the exposure amount in the imaging unit 20 by n ² times. For example, in order to halve the error, the brightness of the pattern light or the exposure amount in the imaging unit 20 is quadrupled. When the exposure amount is quadrupled, the denominator (DL + DR) is quadrupled. When the exposure amount is quadrupled, the numerator noise σ is proportional to the square root value of the image luminance gradation value, so it becomes √4, that is, doubles. Since the numerator is doubled and the denominator is quadrupled, the error is halved.

  As shown in FIGS. 6A to 6C, in the case of picking a part whose position and orientation are recognized among a plurality of parts (target object 5) stacked, the parts are removed and the height of the stack is increased. Changes. When the pile height is high, as shown in FIG. 6A, the distance from the imaging unit 20 to the imaging target is short, and when the pile height is low, as shown in FIG. The distance from the imaging unit 20 to the subject to be photographed is long. Here, the luminance value of the component shown in the image captured by the imaging unit 20 decreases in proportion to the square of the distance from the imaging unit 20 to the imaging target.

  Here, the relationship between the luminance value in the captured image by the imaging unit 20 and the distance from the imaging unit 20 to the imaging target will be described with reference to FIG. Here, as shown in FIG. 7B, it is assumed that the exposure amount of the imaging unit 20 is set to be optimal at a distance Zm (medium distance) from the imaging unit 20 to the imaging target. In FIG. 7B, the waveform of all lighting patterns, the waveform of the positive pattern, and the waveform of the positive pattern partially exceed the saturation level, but the luminance gradation value near the intersection position used for specifying the pattern position is the saturation level. Is not exceeded. Therefore, the position of the intersection can be specified, and the luminance gradation value in the vicinity of the intersection becomes a high value, so that the subpixel position estimation error can be reduced.

  Here, FIG. 7A shows a waveform of the luminance gradation value in Zn (short distance) where the distance from the imaging unit 20 to the imaging target is smaller than Zm. In the central part of this waveform, the saturation level is exceeded. In FIG. 7A, for the sake of convenience, the waveform exceeding the saturation level is illustrated, but actually, the waveform exceeding the saturation level is output to the saturation gradation value at the saturation level. In FIG. 7A, since the intersection position reaches the saturation level luminance gradation value, it is impossible to specify the intersection position.

  Next, FIG. 7C shows a waveform of a luminance gradation value at Zf (far distance) where the distance from the imaging unit 20 to the imaging target is larger than Zm. In the state of FIG. 7C, it can be said that the luminance gradation value waveforms are not all saturated and the distance measurement accuracy is low.

  That is, from the above description, in order to perform distance measurement reliably and with high accuracy in a short distance, a medium distance, and a long distance, “parameters for controlling the brightness and darkness of a captured image that the imaging unit 20 captures next time” It can be seen that it is necessary to adjust according to the distance.

  Next, processing performed by the image processing unit 40 to adjust the exposure amount (exposure time) of the imaging unit 20 will be described with reference to FIG. 8 showing a flowchart of the processing. The processing according to the flowchart of FIG. 8 includes processing for obtaining a reference exposure amount (reference exposure amount) and a distance (reference distance) measured using a captured image captured with the reference exposure amount (step S100), And a process for adjusting the exposure amount (step S110).

  First, the process in step S100 will be described. First, in step S101, the captured image exposure amount control unit 45 controls the exposure amount of the imaging unit 20 according to the currently set exposure amount parameter. The pattern light brightness control unit 44 controls the brightness of the pattern light projected from the projection unit 30 according to the currently set pattern light brightness parameter.

  In step S101, the projection unit 30 sequentially selects the projection patterns from the plurality of projection patterns one by one, irradiates the selected projection pattern to the object 5, and the imaging unit 20 5 is imaged. The types of projection patterns are all lighting patterns, all lighting patterns, positive patterns (four positive patterns (c) to (f) in the case of FIG. 2), negative patterns (negative pattern (g) in the case of FIG. 2). ). In the case of the 4-bit spatial encoding method, a 4-bit positive pattern is used as a positive pattern, and a 4-bit negative pattern is used as a negative pattern. The imaging unit 20 images the object 5 every time the selected pattern light is projected.

  In step S <b> 102, the adjustment unit 41 performs light / dark determination using the captured image captured by the imaging unit 20. Details of the processing in step S102 will be described with reference to the flowchart of FIG. The adjustment unit 41 performs light / dark determination using the four captured images In1 to In4, and outputs either Out1: dark determination or Out2: light determination as a result of the light / dark determination. Here, the picked-up image In1 is a picked-up image of the object 5 onto which the all-lighted pattern is projected, and the picked-up image In2 is a picked-up image of the object 5 onto which the all-off pattern is projected. The captured image In3 is a captured image of the target object 5 on which the positive pattern is projected, and the captured image In4 is a captured image of the target object 5 on which the negative pattern is projected.

  The light / dark determination is performed in two stages. In the first stage, rough contrast determination is performed using the captured image In1 and the captured image In2. In the second stage, detailed light / dark determination is performed using the captured image In3 and the captured image In4. In the present embodiment, it is assumed that the light / dark determination is performed in the two-stage configuration of the first stage and the second stage as described above, but the light / dark determination in the first stage is omitted, and the light / dark determination in the second stage. Only the determination may be performed.

  First, the light / dark determination in the first stage will be described.

  In step S4201, the adjustment unit 41 sets an adjustment area for the captured image In1 or the captured image In2. For example, one of the captured image In1 and the captured image In2 is displayed as a display target image on a display unit (not shown), and the user operates the operation unit (not shown) so that the target 5 shown in the display target image is displayed. This area may be designated (set) as the adjustment area. Of course, the adjustment area setting method is not limited to this, and when the area of the object 5 in the captured image In1 or the captured image In2 is determined in advance or determined by recognition processing, without user operation, You may set the area | region of the target object 5 as an adjustment area | region. It is assumed that when an adjustment area is set for one captured image, the adjustment area is also set at the same position in the other captured image. Next, in step S4202, the adjustment unit 41 counts the number of pixels in the adjustment region as “the number of adjustment region pixels X”.

  Next, in step S4203, the adjustment unit 41 detects a shadow and a shadow area (shadow area) from the adjustment area in the captured image In1 or the adjustment area in the captured image In2. For example, there are the following methods for detecting the shadow region. That is, the difference value of the luminance gradation value between pixels corresponding in position between each pixel in the adjustment region in the captured image In1 and each pixel in the adjustment region in the captured image In2 is calculated, and the difference value is calculated in advance. An area composed of pixels at pixel positions below a set threshold (shadow area threshold) is determined as a shadow area. However, with this determination method alone, the above difference value becomes small even in the case where the luminance gradation value in the adjustment area in the captured image In2 is close to the saturation level or has reached the saturation level, the shadow area It will be judged. In order to deal with such a case, if the value obtained by adding the value of the shadow region threshold to the luminance gradation value of the captured image In2 exceeds the saturation level, it is preferable to further introduce a determination formula that excludes the value from the shadow region. . Of course, the method of specifying the shadow area from the image is not limited to the above method.

  In step S4204, the adjustment unit 41 obtains the number of remaining pixels by subtracting the number of pixels in the shadow area detected in step S4203 from the number of adjustment area pixels counted in step S4202 as the number Y of pixels in the non-shadow area. The number of pixels in the non-shadow area is the number of pixels in the remaining area excluding the shadow area from the adjustment area.

  In step S4205, the adjustment unit 41 obtains a value obtained by dividing the number Y of pixels in the non-shadow area by the number X of pixels in the adjustment area (the ratio of the number Y of pixels in the non-shadow area to the number X of pixels in the adjustment area) in advance. It is determined whether or not a set threshold A is exceeded. When there are many shadow areas in the adjustment area, Y / X takes a relatively small value. However, if Y / X ≦ threshold A (threshold or lower), the shadow area occupies a large amount in the adjustment area. As a result, the adjustment unit 41 determines that “dark determination (Out1) ) "Is output.

  On the other hand, when there are many non-shadow areas in the adjustment area, Y / X takes a relatively large value. However, if Y / X> threshold A, the process advances to step S4206 to execute the second-level light / dark determination.

  Next, the second level light / dark determination will be described. In the second-level light / dark determination, the captured image In3 and the captured image In4 are used as described above. In step S4206, the adjustment unit 41 performs a process of estimating the intersection position as a pattern position within the adjustment region in the captured image In3 (captured image In4). For example, the luminance gradation value at the pixel position (x−1, y) in the captured image In3 is P1, and the luminance gradation value at the pixel position (x + 1, y) in the captured image In3 is P2. The luminance gradation value at the pixel position (x-1, y) in the captured image In4 is Q1, and the luminance gradation value at the pixel position (x + 1, y) in the captured image In4 is Q2. At this time, if “P1> Q1, P2 <Q2” (Condition 1) or “P1 <Q1, P2> Q2” (Condition 2) is satisfied, the pixel position (x, y) is the intersection position. to decide. On the other hand, if neither condition 1 nor condition 2 is satisfied, it is determined that the pixel position (x, y) is not an intersection position. In this way, by performing such a condition determination for all pixel positions (x, y) in the adjustment region in the captured image In3 (captured image In4), it is determined whether or not each pixel position is an intersection position. be able to. In step S4207, the adjustment unit 41 counts the number of intersection positions (measurement pixels) estimated in step S4206 as “measurement pixel number Z”.

  In step S4208, the adjustment unit 41 divides the count pixel number Z by the pixel number Y of the non-shadow region (the ratio of the count pixel number Z to the pixel number Y of the non-shadow region, that is, the detection amount of the measurement pixel). ) Exceeds a preset threshold value B or not. If the ratio of the number of intersection positions to the number of pixels in the non-shadow area is low, the luminance gradation values of many pixels in the adjustment area have reached the saturation level, so it is considered that there are not many intersection positions. It is done. For example, in the vicinity of the center of FIG. 7A, the luminance gradation values of the pixels at the intersection position all exceed the saturation level. As a result, these intersection positions cannot be obtained. A relatively small value. However, if Z / Y ≦ threshold B, the adjustment unit 41 outputs “brightness determination (Out2)” as the result of the lightness / darkness determination. On the other hand, if Z / Y> threshold B, since many of the intersection positions can be calculated, the process advances to step S4209.

  In step S4209, the adjustment unit 41 determines, for each intersection position, whether or not the luminance gradation value of a pixel near the intersection position is saturated. More specifically, first, a total of four points (two points on the positive pattern waveform and two points on the negative pattern waveform) close to the intersection position (in the case of FIG. 5B, each of PR, PL, NR, and NL). Reference is made to the 4) luminance gradation values having luminance gradation values. If one or more of the four points has a luminance gradation value equal to or higher than the saturation level, the pixel at the intersection is determined as a saturation measurement pixel. Such processing is performed for each obtained intersection position. In step S4210, the adjustment unit 41 counts the number of pixels determined as saturation measurement pixels in step S4209 as “saturation measurement pixel number g”.

  In step S4211, the adjustment unit 41 sets a value obtained by dividing the “saturation measurement pixel number g” by the “measurement pixel number Z” (the ratio of the saturation measurement pixel number g to the measurement pixel number Z) as a preset threshold value C. It is determined whether or not it exceeds. When g / Z> threshold C, adjustment unit 41 outputs “brightness determination (Out2)” as the result of light / dark determination, and when g / Z ≦ threshold C, the adjustment unit 41 outputs “lightness / darkness determination”. "Dark determination (Out1)" is output.

  Note that the brightness / darkness determination processing of the captured image performed in step S102 is not limited to the processing according to the flowchart of FIG. 17, and other methods may be employed.

  In step S103, when the determination result in step S102 is “bright”, the adjustment unit 41 adjusts the current exposure amount of the imaging unit 20 to a smaller exposure amount (shorter exposure time). In order to adjust the time), the parameters of the imaging unit 20 are adjusted. On the other hand, when the determination result in step S102 is “dark”, the adjustment unit 41 adjusts the current exposure amount of the imaging unit 20 to a larger exposure amount (the current exposure time is set to a longer exposure time). In order to adjust), the parameters of the imaging unit 20 are adjusted. The exposure time may be adjusted by, for example, a binary search method. Then, the adjustment unit 41 sets the adjusted parameters of the imaging unit 20 in the captured image exposure amount control unit 45, so that the captured image exposure amount control unit 45 controls the imaging unit 20 from the next time according to the adjusted parameters. It will be.

  Then, the process returns to step S101, and in step S101, imaging is performed based on the exposure time adjusted in step S103. In this way, the processes in steps S101 to S103 are repeated a plurality of times. In step S104, the adjustment unit 41 sets the exposure amount (here, the exposure time) obtained as a result of the repetition as the reference exposure amount Exb. obtain.

  In step S105, the captured image exposure amount control unit 45 controls the exposure amount of the imaging unit 20 according to the reference exposure amount Exb. The pattern light brightness control unit 44 controls the brightness of the pattern light projected from the projection unit 30 according to the currently set pattern light brightness parameter. Then, the projection unit 30 sequentially selects one by one from the plurality of projection patterns described above, irradiates the selected projection pattern to the target object 5, and the imaging unit 20 images the target object 5. And the distance calculation part 43 measures the distance with respect to each target object reflected in the captured image by performing said process using each captured image imaged by the imaging part 20. FIG.

  In step S <b> 106, the approximate distance calculation unit 42 obtains a reference distance (reference approximate distance) Zb that is a representative distance to each object shown in the captured image. A process for obtaining the reference distance Zb from the distance to each object shown in the captured image will be described with reference to FIG. FIG. 9A shows an example of the reference distance Zb when the pile height is high, and FIG. 9B shows an example of the reference distance Zb when the pile height is medium. For example, an average distance of distances to the respective objects shown in the captured image may be used as the reference distance, and a maximum value, a minimum value, a median value, or the like may be used as the reference distance.

  Next, the process in step S110 will be described. In step S111, the captured image exposure amount control unit 45 sets the currently set exposure amount (reference exposure amount Exb in the first step S111, exposure amount Ext adjusted in the previous step S116 in the second and subsequent steps S111). Accordingly, the exposure amount of the imaging unit 20 is controlled. The pattern light brightness control unit 44 controls the brightness of the pattern light projected from the projection unit 30 according to the currently set pattern light brightness parameter. Then, the projection unit 30 sequentially selects one by one from the plurality of projection patterns described above, irradiates the selected projection pattern to the target object 5, and the imaging unit 20 images the target object 5.

  In step S <b> 112, the distance calculation unit 43 performs the above-described processing using each captured image captured by the imaging unit 20, thereby measuring the distance to each object captured in the captured image.

  In step S113, the approximate distance calculation unit 42 obtains a representative distance (approximate distance) Zt of the distance to each object shown in the captured image by performing the same process as in step S106.

  In step S114, the adjustment unit 41 obtains an adjustment amount (optical correction amount) Czo of the currently set exposure amount using the reference distance Zb and the distance Zt. Here, the optical correction amount Czo is calculated based on the optical characteristic that the luminance value in the image becomes darker in proportion to the square of the distance. Specifically, the optical correction amount Czo is obtained by calculating the following equation (3).

Czo = (Zt / Zb) ² (Formula 3)
In step S115, the adjustment unit 41 adjusts the reference exposure amount Exb using the optical correction amount Czo. For example, the adjustment is performed according to the following (Equation 4).

Czo × Exb → Ext (Formula 4)
In step S116, the adjustment unit 41 sets the exposure amount Ext adjusted in step S115 in the captured image exposure amount control unit 45. Then, the process returns to step S111, and the subsequent processes are repeated.

  Thus, according to the present embodiment, the exposure amount is set small in the case where the pile height is high, and the exposure amount is set large in the case where the pile height is low. An appropriate exposure amount can be set accordingly. That is, the luminance value of the captured image can be kept substantially constant with respect to the change in the pile height.

  In the present embodiment, the distance measurement for the target object 5 is performed based on the spatial encoding method, but the distance measurement for the target object 5 may be performed by other methods. For example, a distance measurement method based on various pattern projections such as a light cutting method and a phase shift method may be adopted.

[Second Embodiment]
In this embodiment, not only the optical correction amount but also the triangulation error is taken into account to adjust the exposure amount. In the following, differences from the first embodiment will be mainly described, and unless otherwise noted, the same as the first embodiment.

  Processing performed by the image processing unit 40 to adjust the exposure amount of the imaging unit 20 will be described with reference to FIG. 10 showing a flowchart of the processing. In the flowchart of FIG. 10, the same processing steps as those shown in FIG. 8 are denoted by the same step numbers, and description thereof will be omitted.

In step S215, the adjustment unit 41 obtains a triangulation error adjustment amount (triangulation correction amount) Czt using the reference distance Zb and the distance Zt. Here, the correction amount is calculated based on the characteristic that the error of triangulation increases in proportion to the square of the distance. As shown in FIG. 11, when the distance becomes n times, the distance measurement error becomes n ² times. Therefore, in order to keep the measurement error constant, the subpixel position estimation error ΔC is reduced to 1 / n ² . There is a need to. As described above, in order to increase the subpixel estimation error ΔC by 1 / n ² times, it is necessary to increase the brightness of the pattern light or the exposure amount by n ⁴ times. Czt can be obtained according to (Equation 5) below.

Czt = (Zt / Zb) ⁴ (Formula 5)
In step S216, the adjustment unit 41 adjusts the reference exposure amount Exb using the optical correction amount Czo and the triangulation correction amount Czt. For example, it adjusts according to the following (Formula 6).

Czo × Czt × Exb → Ext (Formula 6)
In step S217, the adjustment unit 41 sets the exposure amount Ext adjusted in step S216 in the captured image exposure amount control unit 45. Then, the process returns to step S111, and the subsequent processes are repeated.

  When the ratio between the reference distance Zb and the distance Zt is large, if the exposure amount Ext = exposure time and the exposure time is multiplied by (Czo × Czt), a case where the captured image is saturated may occur. In such a case, if the exposure time is set to the optical correction amount Czo times and the number of images to be picked up is Czt times in Ext, the measurement error can be reduced without causing saturation of the picked-up images. .

Thus, according to the present embodiment, the exposure amount is set small in the case where the pile height is high, and the exposure amount is set large in the case where the pile height is low. An appropriate exposure amount can be set accordingly. That is, the luminance value of the captured image can be kept substantially constant with respect to the change in the pile height. In addition, since the error of triangulation is taken into consideration, the brightness of the captured image and the measurement error can be kept substantially constant with respect to the change in the pile height [Third Embodiment]
A functional configuration example of the object picking system 8 according to the present embodiment will be described with reference to the block diagram of FIG. 12, the same functional units as those shown in FIG. 1 are denoted by the same reference numerals, and the description relating to the functional units is omitted. The configuration shown in FIG. 12 is different from the configuration shown in FIG. 1 in that the position and orientation recognized by the object position and orientation recognition unit 7 are supplied not only to the robot control unit 62 but also to the adjustment unit 41.

  In the present embodiment, not only the light amount change due to the change in the distance to the object but also the light amount change due to the surrounding light amount drop is considered. Therefore, it is necessary for the adjusting unit 41 to grasp where the target object is located in the captured image (in other words, what is the angle of view with respect to the range in which the target object exists). Therefore, the output result from the object position / posture recognition unit 7 is used to grasp where the object is located in the captured image.

  In the following, differences from the first embodiment will be mainly described, and unless otherwise noted, the same as the first embodiment. First, the relationship between the position and orientation of an object and the angle of view θ with respect to the object will be described with reference to FIG. Here, it is assumed that the position of the object output from the object position / posture recognition unit 7 is (Xr, Yr, Zr). The coordinate origin O was made to coincide with the origin of the coordinate system handled by the image processing unit 40. In the XY plane at the depth Zr, when the distance from the Z axis to the object is r, r is calculated by the following (Expression 7).

r = √ (Xr ² + Yr ² ) (Formula 7)
The angle of view θ can be calculated by the following (Equation 8) using r and Zr.

θ = tan ⁻¹ (r / Zr) (Formula 8)
That is, the relationship between the position (Xr, Yr, Zr) of the object and the angle of view θ can be grasped from (Equation 8).

  Next, processing performed by the image processing unit 40 to adjust the exposure amount of the imaging unit 20 will be described with reference to FIG. 14 showing a flowchart of the processing (partly executed by the object position / posture recognition unit 7). Is also included). The processing according to the flowchart of FIG. 14 includes processing for obtaining the reference exposure amount Exb, the reference distance Zb, and the reference in-plane position θb (step S300), processing for adjusting the exposure amount for each object (step S310), Have

  First, details of the processing in step S300 will be described using the flowchart of FIG. Note that both the flowchart shown in FIG. 15A and the flowchart shown in FIG. 15B are flowcharts of processes applicable to step S300. In FIG. 15A and FIG. 15B, the same processing steps as those shown in FIG. 8 are denoted by the same step numbers, and description thereof will be omitted.

  First, processing according to the flowchart of FIG.

  In step S306, the captured image exposure amount control unit 45 controls the exposure amount of the imaging unit 20 in accordance with the reference exposure amount Exb. The pattern light brightness control unit 44 controls the brightness of the pattern light projected from the projection unit 30 according to the currently set pattern light brightness parameter. Then, the projection unit 30 sequentially selects one by one from the plurality of projection patterns described above, irradiates the selected projection pattern to the target object 5, and the imaging unit 20 images the target object 5. And the distance calculation part 43 measures the distance with respect to each target object reflected in the captured image by performing said process using each captured image imaged by the imaging part 20. FIG.

  Since the object position / posture recognition unit 7 recognizes the position / posture of the target object using the distances to the respective objects calculated by the distance calculation unit 43, the adjustment unit 41 has the object position / posture recognition unit 7 The position (Xr, Yr, Zr) of each recognized object is acquired.

  In step S307, the adjustment unit 41 sets Zr of the target object as the reference distance Zb for each target object shown in the captured image.

  In step S308, the adjustment unit 41 sets the reference in-plane position θb for each object in the captured image based on the above (Expression 7) and (Expression 8) as follows (Expression 9). Calculate

θb = tan ⁻¹ (√ (Xr ² + Yr ² ) / Zr) (Formula 9)
Next, processing according to the flowchart of FIG. In FIG. 15B, the approximate distance of the entire stack is set as the reference approximate distance Zb, and the center of the captured image (position at the angle of view of 0 degrees) is set as the reference angle of view θb. However, in step S308 ′, the adjustment unit 41 sets the reference in-plane position θb common to the respective objects shown in the captured image to 0.

  Next, details of the processing in step S310 will be described with reference to FIG. 16 showing a flowchart of the processing. In FIG. 16, the same processing steps as those shown in FIG. 8 are denoted by the same step numbers, and description thereof will be omitted.

  In step S313, the target position / posture recognition unit 7 recognizes the position / posture of the target using the distances calculated by the distance calculation unit 43 for each target in the captured image captured in step S111. At this time, the object position / posture recognition unit 7 sends the position of the object (candidate 1) output to the adjustment unit 41 last time to the robot control unit 62 and one object (candidate) that has not been output yet. The position of 2) is output to the adjustment unit 41. The robot controller 62 may output not only the position but also the posture.

  In the first output by the object position / posture recognition unit 7, there is no object corresponding to candidate 1, and therefore, the candidate having the smallest Zr among the positions of the recognized objects may be used as candidate 1. .

  For example, it is assumed that the object position / posture recognition unit 7 recognizes the position / posture of the target object 1, the position / posture of the target object 2, the position / posture of the target object 3, and the position / posture of the target object 4. Here, it is assumed that the Zr of the object 3 is the smallest among the Zr of the objects 1 to 4. At this time, the object position / posture recognition unit 7 outputs the position of the target object 3 as the candidate 1 (may be added a posture) to the robot control unit 62 and the target object as the candidate 2 for the first time. 1 position is output to the adjustment unit 41. In the second time, the position (position may be added) of the target object 1 as the candidate 1 is output to the robot control unit 62, and the position of the target object 2 as the candidate 2 is output to the adjustment unit 41. In the third time, the position (position may be added) of the object 2 as the candidate 1 is output to the robot control unit 62, and the position of the object 3 as the candidate 2 is output to the adjustment unit 41. In the fourth time, the position (position may be added) of the object 3 as the candidate 1 is output to the robot control unit 62, and the position of the object 4 as the candidate 2 is output to the adjustment unit 41. In the fifth time, the position of the object 4 is output to the robot controller 62.

  In this way, the position of the second candidate is output to the adjustment unit 41, the object that was the second candidate last time is set as the first candidate of the present time, and the position of the current first candidate (an attitude may be added). Is output to the robot controller 62.

  In step S314, upon receiving the second candidate position (Xr2, Yr2, Zr2), the adjustment unit 41 sets Zr2 to the approximate distance Zt.

  In step S315, the adjustment unit 41 obtains the optical correction amount Czo by calculating the above (Equation 3) using the reference distance Zb and the approximate distance Zt.

  In step S316, the adjustment unit 41 obtains the triangulation correction amount Czt by calculating the above (formula 5) using the reference distance Zb and the approximate distance Zt.

  In step S317, the adjustment unit 41 calculates the above (Equation 9) using the position (Xr2, Yr2, Zr2) of the second candidate, thereby obtaining the angle of view θr for the second candidate.

  In step S319, the adjustment unit 41 calculates the following (formula 10) using the reference in-plane position θb calculated in step S300 (or step S300 ′) and the angle of view θr calculated in step S317. Thus, the angle of view correction amount Cθ is obtained.

Cθ = cos ⁴ θb / cos ⁴ θr (Formula 10)
In general lenses, it is known that the amount of light falls based on a law called the cosine fourth law. According to this law, the amount of light falls in proportion to the cosine fourth power of the angle of view.

  In step S319, the adjustment unit 41 adjusts the reference exposure amount Exb by calculating the following (Expression 11), and obtains the exposure amount Ext for the second candidate as a result of the adjustment.

Czo × Czt × Cθ × Exb → Ext (Formula 11)
That is, every time the adjustment unit 41 receives the position of the second candidate, the adjustment unit 41 calculates Ext for the second candidate. In the present embodiment, an appropriate exposure amount can be set even if the position of the recognized object in the screen changes in addition to the height of the stack. The exposure amount is set small near the center of the screen with a small angle of view, and the exposure amount is set large at the periphery of the screen with a large angle of view.

  In this embodiment, the example in which the position in the captured image is grasped based on the position / orientation recognition result of the object has been described, but it is not always necessary to use the position / orientation recognition result of the object. For example, the area in which the target object in the captured image is captured is divided into a plurality of areas, and the approximate distance is calculated for each divided area. Then, the center position of the area with the smallest approximate distance among the divided areas may be set as the position where the object exists.

  Further, in step S318 described above, the peripheral light amount drop due to the position (view angle) in the captured image is formulated based on the cosine fourth power law, but is not limited thereto. In the case where the characteristics of the optical system are not based on the cosine fourth law, an appropriate formula can be used in accordance therewith. Alternatively, a method can be used in which the relationship between the angle of view and the brightness is tabulated based on the actual measurement data without using a polynomial, and the corresponding angle of view value is referred to according to the position of the target in the captured image. .

[Fourth Embodiment]
Each functional unit configuring the image processing unit 40 may be configured by hardware, but may be configured by software (computer program). In such a case, when this computer program is installed, an apparatus that executes the computer program can be applied to the image processing unit 40. A hardware configuration example of a device applicable to the image processing unit 40 will be described with reference to the block diagram of FIG.

  The CPU 1601 executes processes using computer programs and data stored in the RAM 1602 and the ROM 1603, thereby controlling the operation of the entire apparatus and executing the processes described above as performed by the image processing unit 40. .

  The RAM 1602 has an area for temporarily storing computer programs and data loaded from the external storage device 1606, data received from the outside via an I / F (interface) 1607, and the like. Further, the RAM 1602 has a work area used when the CPU 1601 executes various processes. That is, the RAM 1602 can provide various areas as appropriate. The ROM 1603 stores setting data and a boot program for the apparatus.

  The operation unit 1604 is configured by a keyboard, a mouse, and the like, and various instructions can be input to the CPU 1601 by the user of the apparatus. For example, the adjustment area can be specified by operating the operation unit 1604.

  The display unit 1605 is configured by a CRT, a liquid crystal screen, or the like, and can display a processing result by the CPU 1601 with an image, text, or the like. For example, it is possible to display a screen for designating an adjustment area including the captured image In1 and the captured image In2.

  The external storage device 1606 is a mass information storage device represented by a hard disk drive device. The external storage device 1606 includes a computer program and data for causing the CPU 1601 to execute the processes described above as performed by the OS (operating system) and each function unit in the image processing unit 40 illustrated in FIGS. Is saved. This data includes information described as known information in the above description. Computer programs and data stored in the external storage device 1606 are appropriately loaded into the RAM 1602 under the control of the CPU 1601 and are processed by the CPU 1601.

  The I / F 1607 is for the apparatus to communicate with an external device. For example, the imaging unit 20, the projection unit 30, and the object position / posture recognition unit 7 are connected to the I / F 1607. be able to. Each of the above parts is connected to the bus 1608.

(Other examples)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (16)

An acquisition unit that acquires a captured image obtained by imaging a plurality of objects on which a pattern is projected by the projection unit;
Derivation means for deriving distances from the imaging means to each of the plurality of objects using the captured image;
Of the first parameter for controlling the brightness and darkness of the pattern projected from the projection means and the second parameter for controlling the exposure amount of the imaging means, the adjustment amount of the adjustment target is obtained from the imaging means. An image processing apparatus comprising: an adjustment unit that obtains the distance to each of the plurality of objects and adjusts the adjustment object according to the obtained adjustment amount . Furthermore,
First means for adjusting at least one of the first parameter and the second parameter according to the brightness of the captured image;
After the adjustment by the first means, the imaging means having the second parameter set picked up the plurality of objects on which the pattern has been projected by the projection means having the first parameter set. to acquire a captured image, using the captured image the acquired determines the distance from the imaging means to the each of the plurality of objects, a second means for calculating the one distance from the distance determined the relative distance And
The adjusting unit obtains one distance from the distance derived by the deriving unit , obtains the adjustment amount based on a calculated value obtained by dividing the obtained one distance by the reference distance, and calculates the adjustment amount . The image processing apparatus according to claim 1, wherein the adjustment target is adjusted. The adjusting means includes
The image processing apparatus according to claim 2, wherein the adjustment target is adjusted by multiplying the adjustment target by a value obtained by squaring the calculation value . The adjusting means includes
A parameter for controlling the number of times of imaging included in the second parameter by multiplying a parameter for controlling the exposure time included in the second parameter by a value obtained by squaring the calculated value. The image processing apparatus according to claim 2, wherein the second parameter is adjusted by multiplying a value obtained by multiplying the calculated value to the fourth power. The first means includes
A luminance value waveform composed of luminance values of pixels constituting the first captured image of the plurality of objects on which the first pattern is projected, and a second pattern obtained by inverting light and darkness in the first pattern Finding the intersection of the luminance value waveform composed of the luminance value of each pixel constituting the second captured image of the projected plurality of objects,
The image processing apparatus according to claim 2, wherein the adjustment target is adjusted according to the number of the intersections. If the ratio of the number of intersection points to the number of pixels in the area specified as a non-shadow area in the first captured image or the second captured image is equal to or less than a threshold, the first means The image processing apparatus according to claim 5 , wherein the parameter is adjusted so that the pattern becomes darker, or the second parameter is adjusted so that the exposure amount becomes smaller. If the ratio of the number of intersection points to the number of pixels in the area specified as a non-shadow area in the first captured image or the second captured image is equal to or less than a threshold, the first means The image processing apparatus according to claim 5 , wherein the parameter is adjusted so that the pattern becomes darker, and the second parameter is adjusted so that the exposure amount becomes smaller. When the ratio of the number of intersections to the number of pixels in the area specified as a non-shadow area in the first captured image or the second captured image exceeds the threshold, the first means The ratio of the number of intersection points obtained by referring to the pixel whose pixel value is saturated with respect to the number of intersection points is obtained as a second ratio,
When the second ratio exceeds a threshold, the first parameter is adjusted so that the pattern becomes darker, or the second parameter is adjusted so that the exposure amount becomes smaller. the image processing apparatus according to any one of claims 5 to 7, characterized. When the ratio of the number of intersections to the number of pixels in the area specified as a non-shadow area in the first captured image or the second captured image exceeds the threshold, the first means The ratio of the number of intersection points obtained by referring to the pixel whose pixel value is saturated with respect to the number of intersection points is obtained as a second ratio,
When the second ratio exceeds a threshold value, the first parameter is adjusted so that the pattern becomes darker, and the second parameter is adjusted so that the exposure amount becomes smaller. the image processing apparatus according to any one of claims 5 to 7, characterized. When the ratio of the number of intersections to the number of pixels in the area specified as a non-shadow area in the first captured image or the second captured image exceeds the threshold, the first means The ratio of the number of intersection points obtained by referring to the pixel whose pixel value is saturated with respect to the number of intersection points is obtained as a second ratio,
If the second ratio is equal to or less than a threshold value, the first parameter is adjusted so that the pattern becomes brighter, or the second parameter is adjusted so that the exposure amount becomes larger. the image processing apparatus according to any one of claims 5 to 9. When the ratio of the number of intersections to the number of pixels in the area specified as a non-shadow area in the first captured image or the second captured image exceeds the threshold, the first means The ratio of the number of intersection points obtained by referring to the pixel whose pixel value is saturated with respect to the number of intersection points is obtained as a second ratio,
If the second ratio is less than or equal to a threshold value, the first parameter is adjusted so that the pattern becomes brighter, and the second parameter is adjusted so that the exposure amount becomes larger. the image processing apparatus according to any of claims 5 to 9. The adjusting means includes
12. The adjustment amount according to claim 1, wherein the adjustment amount is obtained based on any one of an average distance, a maximum value, a minimum value, and a median value of distances from the imaging unit to each of the plurality of objects. The image processing apparatus according to item 1. An acquisition means for acquiring a captured image obtained by imaging an object on which a pattern is projected by the projection means;
Derivation means for deriving a distance from the imaging means to the object and a position of the object in the captured image based on the position of the object recognized from the captured image;
In accordance with the derived distance and the derived position, a first parameter for controlling light and darkness of the pattern projected from the projection unit, and a second parameter for controlling the exposure amount of the imaging unit. Adjusting means for adjusting at least one of the parameters of
An image processing apparatus comprising: An image processing method performed by an image processing apparatus,
An acquisition step in which the acquisition unit of the image processing apparatus acquires a captured image obtained by imaging the plurality of objects on which the pattern is projected by the projection unit, by the imaging unit;
A derivation step in which a derivation unit of the image processing apparatus derives a distance from the imaging unit to each of the plurality of objects using the captured image;
Adjusting means of the image processing apparatus, a second parameter for controlling the exposure amount of the first parameter, and the image pickup means for controlling the light-dark pattern projected from the projection means, the target adjustment of the An adjustment step of obtaining the adjustment amount based on the distance from the imaging means to each of the plurality of objects, and adjusting the adjustment object according to the obtained adjustment amount . An image processing method performed by an image processing apparatus,
An acquisition step in which the acquisition unit of the image processing apparatus acquires a captured image obtained by imaging the object on which the pattern is projected by the projection unit by the imaging unit;
The derivation means of the image processing device derives the distance from the imaging means to the object and the position of the object in the captured image based on the position of the object recognized from the captured image. A derivation process to
A first parameter for controlling light and darkness of a pattern projected from the projection unit according to the derived distance and the derived position, and an adjustment unit of the image processing apparatus; and An adjustment step of adjusting at least one of the second parameters for controlling the exposure amount;
An image processing method comprising: A computer program for causing a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 13 .

Images