JP2016169989A - Parallax image generation system and parallax image generation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a highly accurate parallax image in a short time.SOLUTION: The present invention comprises: a pattern projector 3 for radiating a pattern to an object; a reference image capture unit 210 and a comparison image capture unit 220 for capturing the image of the object to which the pattern is radiated; a parallax image generation unit 230 for generating a parallax image from the captured images; a parallax image evaluation unit 430 for changing the value of a parameter associated with the pattern while evaluating the parallax image, and outputting the parallax image on the basis of the evaluation result; and an object determination unit 410 for determining at least one of the size, shape, color, and light transmissivity of the object and the distance from a stereo camera 2. The parallax image evaluation unit 430 acquires a parameter value that is predetermined as a value corresponding to the determination result of the object determination unit 410 as an evaluation criterion value, and narrows down, on the basis of the acquired evaluation criterion value, an evaluation range that is a range of parameter values in which the parallax image is evaluated.SELECTED DRAWING: Figure 7

Description

  The present invention relates to a parallax image generation system and a parallax image generation method.

  A technique for generating a parallax image (distance image) by obtaining a parallax value for each pixel by a stereo matching process using a stereo camera is known. Stereo matching processing refers to searching for corresponding pixels between an image (reference image) captured by one of two cameras with different shooting positions and an image captured by the other (comparison image). This is a process for obtaining a parallax value for each pixel.

  However, since many stereo matching processes associate pixels between the reference image and the comparison image using a local luminance change as a clue, if the texture of the object that is the subject is weak, there are pixels for which the parallax value cannot be obtained. Many occur and it is difficult to generate a highly accurate parallax image. Therefore, various techniques have been proposed in which texture is artificially imparted by irradiating a pattern as an object to facilitate the association of pixels between a reference image and a comparative image. For example, Patent Document 1 describes a technique for calculating a score by evaluating a noise amount of a parallax image, and irradiating an object with a local unique pattern in which the scaling ratio is changed so that the calculated score is optimal. ing.

  If the pattern irradiated to the object that is the subject does not match the characteristics of the object, the number of pixels (hereinafter referred to as the “effective parallax score”) that provides a highly reliable parallax value (hereinafter referred to as “effective parallax”). ") And the parallax image with high accuracy cannot be obtained. In the technique described in Patent Document 1, the local original pattern enlargement / reduction ratio is changed so that the score calculated by evaluating the noise amount of the parallax image is optimal, but the locality where the optimal score is obtained is changed. If the search for the enlargement / reduction ratio of the target unique pattern is performed brute-force, it takes a long time until a parallax image with high accuracy is finally obtained.

  In order to solve the above-described problem, the present invention provides a pattern irradiating unit that irradiates a pattern with an object, an imaging unit that images the object irradiated with the pattern, and a parallax from an image captured by the imaging unit. Generation means for generating an image, evaluation means for evaluating the parallax image generated by the generation means while changing a value of a parameter related to the pattern, and outputting the parallax image based on an evaluation result; Determination means for determining at least one of the size, shape, color, light transmittance, and distance from the imaging means of the object, and the evaluation means is predetermined as a value corresponding to the determination result of the determination means. The obtained parameter value is obtained as an evaluation reference value, and the parameter value range for evaluating the parallax image based on the obtained evaluation reference value. Carry out the value range of refinement.

  According to the present invention, there is an effect that a highly accurate parallax image can be generated in a short time.

FIG. 1 is a diagram for explaining the principle of calculating a distance to an object using a stereo camera. FIG. 2 is a diagram for explaining the outline of the stereo matching process. FIG. 3 is a diagram illustrating the relationship between the cost value of the candidate pixel and the shift amount. FIG. 4 is a diagram for explaining the relationship between the light transmittance of an object and the pattern granularity. FIG. 5 is a schematic configuration diagram of the picking system according to the embodiment. FIG. 6 is a block diagram illustrating a hardware configuration example of the picking system according to the embodiment. FIG. 7 is a block diagram illustrating a functional configuration example of the stereo camera and the information processing apparatus. FIG. 8 is a block diagram illustrating a detailed configuration example of the parallax image generation unit. FIG. 9 is a diagram illustrating a method for calculating integer parallax and decimal parallax. FIG. 10 is a diagram illustrating an example of a correspondence table stored in the evaluation reference value storage unit. FIG. 11 is a diagram illustrating an example of narrowing down the evaluation range. FIG. 12 is a flowchart for explaining the flow of operations by the picking system. FIG. 13 is a flowchart illustrating an example of the object determination process. FIG. 14 is a flowchart for explaining an example of the pattern optimization process. FIG. 15 is a flowchart illustrating an example of the search direction determination process. FIG. 16 is a diagram for explaining the narrowing down of the evaluation range using the bisection method. FIG. 17 is a flowchart for explaining pattern optimization processing according to the second embodiment. FIG. 18 is a schematic configuration diagram of the picking system according to the third embodiment.

  Hereinafter, embodiments of a parallax image generation system and a parallax image generation method according to the present invention will be described in detail with reference to the accompanying drawings.

  The parallax image generation system of the present embodiment captures an object irradiated with a pattern with a stereo camera and generates a parallax image. At this time, in order to obtain a highly accurate parallax image having a large number of effective parallax points, the parallax image is evaluated while changing the value of the parameter of the pattern irradiated to the object as the subject, and the evaluation result is a predetermined reference A parallax image satisfying the above condition is output. Here, the pattern parameter is a parameter related to one or more of the size, shape, color, density, and brightness of the texture included in the pattern, such as a pattern enlargement / reduction ratio and contrast. Hereinafter, a process of searching for a parameter value of a pattern from which a parallax image whose evaluation result satisfies a predetermined criterion is obtained is referred to as a “pattern optimization process”. In this embodiment, in order to perform this pattern optimization process efficiently in a short time, at least one of the size, shape, color, light transmittance, and distance from the stereo camera of the object that irradiates the pattern is determined. A parameter value predetermined as a value corresponding to the determination result is acquired as an evaluation reference value. Then, based on the acquired evaluation reference value, the evaluation range that is the range of the parameter value for evaluating the parallax image is narrowed down. Thereby, an accurate parallax image can be generated in a short time.

[Principles of ranging]
First, the principle of distance measurement applied in this embodiment will be described with reference to FIG. FIG. 1 is a diagram for explaining the principle of calculating a distance to an object using a stereo camera. Here, the parallax value for the object in the stereo camera is obtained by stereo matching processing, and the distance from the stereo camera to the object is measured by this parallax value. In the following, in order to simplify the description, it is assumed that matching is performed in units of pixels instead of matching a predetermined area composed of a plurality of pixels.

  The stereo camera shown in FIG. 1 has two imaging units 10a and 10b arranged in parallel equiposition. The imaging units 10a and 10b respectively include lenses 11a and 11b that form an image of an object E serving as a subject on an image sensor. Here, a captured image (luminance image) captured by the imaging unit 10a is referred to as a reference image Ia, and a captured image (luminance image) captured by the imaging unit 10b is referred to as a comparative image Ib.

In FIG. 1, the point S on the object E in the three-dimensional space is mapped to a position on a straight line parallel to a straight line connecting the lens 11a and the lens 11b in each of the reference image Ia and the comparison image Ib. A position where the point S on the object E is mapped in the reference image Ia is a point Sa (x, y), and a position where the point S on the object E is mapped in the comparison image Ib is a point Sb (X, y). . At this time, the parallax value dp is expressed by the following equation (1) using the coordinate value of the point Sa (x, y) in the reference image Ia and the coordinate value of the point Sb (X, y) in the comparison image Ib. It is expressed in
dp = X−x (1)

  In FIG. 1, the distance between the point Sa (x, y) in the reference image Ia and the intersection of the perpendicular line dropped from the lens 11a on the imaging surface is Δa, and the point Sb (X, y) in the comparative image Ib and the lens If the distance from the intersection of the perpendicular line dropped from 11b to the imaging surface is Δb, the parallax value dp can also be expressed as dp = Δa + Δb.

By using the parallax value dp, the distance Z between the stereo camera and the object E can be calculated. The distance Z is a distance from a straight line connecting the focal position of the lens 11a and the focal position of the lens 11b to a point S on the object E. As shown in FIG. 1, using the focal length f of the lenses 11a and 11b, the baseline length B which is the length between the lenses 11a and 11b, and the parallax value dp, the distance Z Can be calculated.
Z = (B × f) / dp (2)
As can be seen from Equation (2), the distance Z decreases as the parallax value dp increases, and the distance Z increases as the parallax value dp decreases.

[Stereo matching processing]
Next, an outline of stereo matching processing for obtaining corresponding pixels in the comparison image Ib with respect to the reference image Ia will be described. Here, the corresponding pixel refers to a pixel in the comparison image Ib that is most similar to a pixel in the reference image Ia (hereinafter referred to as “reference pixel”). In the stereo matching process, for each corresponding pixel candidate in the comparison image Ib, a cost value C representing the degree of dissimilarity with respect to the reference pixel is calculated, and a candidate having the minimum cost value C is obtained as the corresponding pixel. If the corresponding pixel corresponding to the reference pixel is uniquely determined, for example, the highly reliable parallax value dp can be calculated for the pixel by the above equation (1). The parallax image is an image in which the reference pixel is represented by a parallax value dp. Therefore, as the number of reference pixels (effective parallax points) for which a highly reliable parallax value dp (effective parallax) can be calculated, the parallax image becomes more accurate. For reference pixels for which effective parallax cannot be calculated, a parallax image is configured with a parallax value dp of “0”, for example.

  FIG. 2 is a diagram for explaining the outline of the stereo matching process. FIG. 2A is a conceptual diagram showing an example p (x, y) of the reference pixel p in the reference image Ia, and FIG. 2B is a reference pixel p (x, y shown in FIG. 2A. ) Is a conceptual diagram illustrating an example q (x + d, y) of a corresponding pixel candidate (hereinafter referred to as “candidate pixel”) in the comparison image Ib corresponding to FIG. As shown in FIG. 2, the luminance value of each candidate pixel q (x + d, y) on the epipolar line EL in the comparison image Ib corresponding to the reference pixel p (x, y) is obtained and the reference pixel p (x, y) is obtained. ), The cost value C (p, d) of each candidate pixel q (x + d, y) is calculated. d is a shift amount (shift amount) of the candidate pixel q with respect to the reference pixel p, and represents a shift amount in pixel units here. That is, in FIG. 2, the candidate pixel q (x + d, y) and the reference pixel p are sequentially shifted pixel by pixel within a predetermined range (for example, 0 <d <25). A cost value C (p, d) representing the dissimilarity of the luminance value with (x, y) is calculated.

  As described above, since the imaging units 10a and 10b are arranged in parallel equivalence, the corresponding pixel in the comparison image Ib corresponding to the reference pixel p in the reference image Ia exists on the epipolar line EL shown in FIG. Will do. Therefore, in order to obtain the corresponding pixel in the comparison image Ib, it is only necessary to search for a pixel on the epipolar line EL in the comparison image Ib.

  The cost value C calculated in this way can be represented by a graph as shown in FIG. 3 in relation to the shift amount d. In the example of FIG. 3, the cost value C is “0” when the shift amount d = 5, 12, and 19, and thus the minimum value cannot be obtained. For example, when an object having a weak texture is used as a subject, there are many candidate pixels q having the same luminance value as the reference pixel p, and thus it is difficult to obtain the minimum value of the cost value C in this way. As a result, the corresponding pixel corresponding to the reference pixel p cannot be uniquely identified (a mismatch occurs), and the highly reliable parallax value dp cannot be calculated.

  Such a problem is improved by artificially imparting a texture by irradiating an object as a subject with a pattern. Here, “texture” refers to a pattern, pattern, pattern, color, dot, or the like that can appear as the brightness of each pixel on the image, for example. The pattern irradiated on the object is preferably a pattern having a plurality of brightness values or a two-dimensional random number pattern. However, a pattern having a repetition period is not preferable because the cost value C may be repeated.

  Further, it has been found that the optimum pattern for irradiating an object differs depending on the size, shape, color, light transmittance, distance from the stereo camera, and the like of the object. For example, when the light transmittance of an object is taken as an example, when an object having a high light transmittance is irradiated with a pattern with too fine particle size (pattern with a too small texture size), the light diffusely reflected on the surface of the object The texture may disappear due to the influence of interference with light that is transmitted through the object and diffusely reflected on another surface. As a result, similar luminance values are often arranged in the search direction in the stereo matching process, and the number of effective parallax points is reduced, thereby reducing the accuracy of the parallax image. Also, for example, when an object with low light transmittance is irradiated with a pattern with too high contrast, the luminance value of a pixel corresponding to a texture with high brightness may be saturated due to the large amount of reflected light on the object surface. is there. As a result, the number of pixels from which a correct luminance value cannot be obtained increases, the number of effective parallax points decreases, and the accuracy of the parallax image decreases.

  FIG. 4 is a diagram for explaining the relationship between the light transmittance of an object and the pattern granularity. In FIG. 4, a luminance image (reference image or comparison) obtained when an opaque body that is an object with low light transmittance and a translucent body that is an object with high light transmittance are irradiated with a pattern with a fine particle size. (Image) and a parallax image, and the brightness | luminance image and parallax image obtained when a pattern with coarse particle size is irradiated, respectively. In the parallax image shown in FIG. 4, for convenience, pixels for which effective parallax is obtained are shown in gray, and pixels for which effective parallax is not obtained are shown in black.

  As shown in FIG. 4, when the object that is the subject is an opaque body, the number of effective parallax points in the parallax image is greater when the pattern with the finer particle size is irradiated than when the pattern with the coarser particle size is irradiated. A good parallax image can be obtained. On the other hand, when the subject object is a semi-transparent object, when a pattern with a fine grain size is irradiated, the luminance image is blurred and the texture disappears, and the number of effective parallax points in the parallax image is greatly reduced. Since the texture remains as a lump when irradiated, the number of effective parallax points increases, and an accurate parallax image can be obtained.

  As shown in the above example, the optimum pattern for obtaining a highly accurate parallax image varies depending on the transmittance of the object that is the subject. Also, the optimum pattern for obtaining a highly accurate parallax image differs depending not only on the transmittance of the object but also on the size, shape, color, distance from the stereo camera, and the like of the object. As a method of searching for an optimum pattern for an object, there is a method of searching for an optimum value of parameters related to the pattern (for example, pattern enlargement / reduction ratio and contrast). The time required to obtain a good parallax image becomes longer. Therefore, in the present embodiment, at least one of the size, shape, color, light transmittance, and distance from the stereo camera of the object that is the subject is determined, and a parameter value determined in advance as a value corresponding to the determination result Is obtained as an evaluation reference value. Then, based on the acquired evaluation reference value, the evaluation range that is the range of the parameter value for evaluating the parallax image is narrowed down. In the present embodiment, by performing such pattern optimization processing, a highly accurate parallax image can be generated in a short time.

[First Embodiment]
Next, as an example of a specific system to which the parallax image generation system according to the present invention is applied, a product assembly factory or the like automatically recognizes an object such as a part, a semi-finished product, or a finished product conveyed from a previous process. A picking system for picking an object using a robot arm will be described. In this picking system, for example, a parallax image is generated by performing imaging with a stereo camera on an object conveyed from a previous process by a belt conveyor or the like in a state of being placed flat or stacked in a conveyance tray, Picking is performed by recognizing the three-dimensional position of the object based on the parallax image. This picking system can generate a highly accurate parallax image in a short time by performing the above-described pattern optimization processing on the object to be picked, and can shorten the tact time of the picking operation. .

(overall structure)
FIG. 5 is a schematic configuration diagram of the picking system 1 of the present embodiment. As shown in FIG. 5, the picking system 1 includes a stereo camera 2, a pattern projector 3 (an example of a pattern irradiation unit), an information processing device 4, and an arm control device 5.

  The stereo camera 2 captures an object that is a subject using two image capturing units, generates a parallax image from the two captured images (luminance images) serving as the reference image and the comparison image, and outputs the parallax image. Details of the functions of the stereo camera 2 will be described later. Note that the stereo camera 2 may be configured to generate a parallax image by capturing an object with three or more image capturing units. However, in the present embodiment, the stereo camera 2 is configured to include two image capturing units.

  The pattern projector 3 irradiates a pattern on an object that is a subject and gives a texture to the object. In the present embodiment, a parallax image with high accuracy is generated by optimizing the values of the parameters (for example, the enlargement / reduction ratio and contrast) of the pattern that the pattern projector 3 irradiates the object with the pattern optimization process. The The pattern projector 3 also has a function of irradiating an object with light for object determination described later.

  The information processing apparatus 4 evaluates the parallax image generated by the stereo camera 2 and performs object recognition processing using the parallax image whose evaluation result satisfies a predetermined criterion. The information processing apparatus 4 performs the pattern optimization process as described above when evaluating the parallax image. Details of the functions of the information processing apparatus 4 will be described later.

  The arm control device 5 controls the operation of the robot arm 6, which is an articulated robot, based on the recognition result of the object by the information processing device 4, and picks the object in the transfer tray 7.

(Hardware configuration)
FIG. 6 is a block diagram illustrating a hardware configuration example of the picking system 1 of the present embodiment. As shown in FIG. 6, the stereo camera 2 includes two imaging units 10 a and 10 b and an FPGA (Field-Programmable Gate Array) 11. The FPGA 11 is equipped with a program for generating a parallax image from images captured by the two imaging units 10a and 10b.

  As illustrated in FIG. 6, the information processing apparatus 4 is configured as a general-purpose computer apparatus including a CPU 12, a ROM 13, a RAM 14, a ROM I / F 15, a RAM I / F 16, and I / Fs 17 to 19. The CPU 12 is an arithmetic device that executes arithmetic operations for realizing each function of the information processing device 4. The ROM 13 is a non-volatile memory that stores programs and data. The RAM 14 is a main memory used as a work area for the CPU 12 to execute a program. The RAM 14 is also used as an image memory that temporarily stores captured images (luminance images) and parallax images sent from the stereo camera 2 to the information processing device 4.

  The ROM I / F 15 is an interface for the CPU 12 to access the ROM 13, and the RAM I / F 16 is an interface for the CPU 12 to access the RAM 14. The I / Fs 17 to 19 are interfaces for connecting the information processing device 4 to the stereo camera 2, the pattern projector 3, and the arm control device 5. The information processing apparatus 4 may be provided with a large-capacity storage device such as a hard disk device, for example, or may be provided externally.

  The main functions of the information processing apparatus 4 are realized, for example, when the CPU 12 uses the RAM 14 as a work area and executes a program stored in the ROM 13 or the hard disk device.

(Functional configuration of stereo camera and information processing device)
FIG. 7 is a block diagram illustrating a functional configuration example of the stereo camera 2 and the information processing apparatus 4. As illustrated in FIG. 7, the stereo camera 2 includes a parallax realized by the reference image capturing unit 210 and the comparative image capturing unit 220 (imaging unit) realized by the two imaging units 10 a and 10 b described above and the FPGA 11 described above. An image generation unit 230 (generation means). The information processing apparatus 4 includes, for example, an object determination unit 410 (determination unit), a parallax image evaluation unit 430 (evaluation unit), and the like as functional components realized by the CPU 12 executing the program. A recognition processing unit 440. Further, the information processing apparatus 4 includes an evaluation reference value storage unit 420 realized by the ROM 13 or the hard disk device.

  The reference image capturing unit 210 captures an object that is a subject, and outputs a luminance image that is the reference image Ia described above as a captured image. The comparative image capturing unit 220 captures an object that is a subject from a position different from that of the reference image capturing unit 210, and outputs a luminance image that is the above-described comparative image Ib as a captured image. The luminance images from the reference image capturing unit 210 and the comparative image capturing unit 220 are input to the parallax image generating unit 230. The luminance image (reference image Ia) from the reference image capturing unit 210 is also input to the recognition processing unit 440 of the information processing device 4. Further, the luminance image obtained by the reference image capturing unit 210 capturing an object irradiated with the object determination light is input to the object determining unit 410 of the information processing device 4.

  The parallax image generation unit 230 generates a parallax image using the luminance image (reference image Ia) from the reference image imaging unit 210 and the luminance image (comparison image Ib) from the comparative image imaging unit 220. The parallax image generated by the parallax image generation unit 230 is input to the parallax image evaluation unit 430 of the information processing device 4. FIG. 8 is a block diagram illustrating a detailed configuration example of the parallax image generation unit 230. As illustrated in FIG. 8, the parallax image generation unit 230 includes filter units 231 and 232, a cost calculation unit 233, an integer parallax calculation unit 234, and a minority parallax calculation unit 235.

  The filter units 231 and 232 respectively remove noise and enhance high frequencies for the luminance image (reference image Ia) from the reference image imaging unit 210 and the luminance image (comparison image Ib) from the comparative image imaging unit 220. To perform filtering processing.

  As described above, the cost calculation unit 233 calculates the cost value C of each candidate pixel q based on the luminance value of the reference pixel p in the reference image Ia and the luminance value of each candidate pixel q in the comparison image Ib. As a calculation method of the cost value C by the cost calculation unit 233, for example, a known method such as SAD (Sum of Absolute Difference), SSD (Sum of Squared Difference), or NCC (Normalized Cross-Correlation) can be used. .

  The integer parallax calculation unit 234 and the decimal parallax calculation unit 235 calculate the parallax value dp in sub-pixel units smaller than one pixel based on the cost value C of each candidate pixel q calculated by the cost calculation unit 233. Hereinafter, the parallax value dp calculated in units of one pixel is referred to as integer parallax Δ, and the parallax value dp calculated in units of subpixels is referred to as decimal parallax δ. FIG. 9 is a diagram for explaining a method of calculating the integer parallax Δ and the decimal parallax δ.

  As shown in FIG. 9, the integer parallax calculation unit 234 searches for a shift amount d that minimizes the cost value C calculated for each candidate pixel q (that is, for each shift amount d from the reference pixel p). Then, the value of the shift amount d that minimizes the cost value C is calculated as the integer parallax Δ. In the decimal parallax calculation unit 235, the decimal parallax δ shown in FIG. 9 is calculated using the subpixel estimation method in order to pursue the parallax value dp to the decimal. For example, in the equiangular straight line method, the integer parallax Δ calculated by the integer parallax calculation unit 234 is the center, and 3 corresponding to a total of three candidate pixels of Δ−1 having a smaller value of d and Δ + 1 having a larger value of d. The decimal parallax δ is calculated using one cost value C. Further, in the parabolic fit method or the approximation method using a polynomial, the decimal parallax δ may be calculated using the above three cost values C, and Δ-2, Δ-1, Δ, Δ + 1, Δ + 2 in FIG. The decimal parallax δ may be calculated using the corresponding five cost values C. Not limited to these, the decimal parallax δ may be calculated using a desired subpixel estimation method.

  In the parallax image generation unit 230, for each reference pixel p in the reference image Ia, the cost calculation unit 233 calculates the cost value C of the candidate pixel q, and the integer parallax calculation unit 234 and the decimal parallax calculation unit 235 calculate the minority parallax δ. Is calculated. Thereby, a parallax image representing the three-dimensional information of the object is generated. The parallax image generation unit 230 includes a pre-processing unit that performs processing such as distortion correction processing on the reference image Ia input from the reference image imaging unit 210 and the comparison image Ib input from the comparative image imaging unit 220. May be. Since the distortion states of the reference image Ia and the comparison image Ib may greatly affect the result of the cost value C calculated by the cost calculation unit 233, it is effective to perform distortion correction processing before calculating the cost value C. . In addition, when the optical axis shift between the lens and the sensor in the reference image capturing unit 210 and the comparative image capturing unit 220 and the optical distortion state of the lens are different between the reference image Ia and the comparative image Ib, the distortion is corrected. It is also effective to take measures such as holding the distortion correction parameter in the preprocessing unit and reflecting the calibration result in the distortion correction parameter.

  Returning to FIG. 7, the object determination unit 410 determines at least one of the size, shape, color, light transmittance, and distance from the stereo camera 2 for the object to be picked. Various methods can be used as the determination method by the object determination unit 410. Here, as an example, an example of determination using a captured image (luminance image) of an object captured by the stereo camera 2 will be described. .

  For example, the object determination unit 410 uses the pattern projector 3 for object determination when a predetermined determination timing is reached, such as when the picking operation is started by the picking system 1 or when the number of picking times exceeds a predetermined number (picking frequency threshold). A control command for instructing the irradiation of the light is output, and a control command for instructing the stereo camera 2 to take a luminance image is output. When the luminance image of the object irradiated with the object determination light from the pattern projector 3 is output from the reference image imaging unit 210 of the stereo camera 2, the object determination unit 410 is output from the reference image imaging unit 210. By acquiring and analyzing the obtained luminance image, at least one of the size, shape, color, light transmittance, and distance from the stereo camera 2 is determined.

  For example, the object determination unit 410 can determine the shape of the object by binarizing the luminance image output from the reference image capturing unit 210 and performing edge detection, and can further determine the camera parameters (focal length) of the stereo camera 2. And the angle of view, position, etc.), the size of the object can be determined. Conversely, the distance from the stereo camera 2 to the object can also be determined from the ratio between the size known as the object size of the predetermined shape and the object size in the luminance image. Further, the color of the object can be determined using the pixel values of the luminance image of the RGB 3 channel. Note that the object determination light emitted to the object from the pattern projector 3 is, for example, uniform light having a predetermined brightness and having no brightness distribution. The determination result of the object determination unit 410 is notified to the parallax image evaluation unit 430.

  The parallax image evaluation unit 430 outputs a parallax image with high accuracy while evaluating the parallax image by the pattern optimization process. That is, the parallax image evaluation unit 430 evaluates the parallax image generated by the parallax image generation unit 230 of the stereo camera 2 while changing the value of the parameter of the pattern irradiated to the object from the pattern projector 2, and the evaluation result Outputs a parallax image satisfying a predetermined criterion. As a method of reflecting the parameter value in the pattern irradiated from the pattern projector 2, for example, a pattern prepared for each parameter value may be irradiated from the pattern projector 2, or a parallax image evaluation unit may be used. The pattern projector 430 may generate a pattern according to the parameter value and irradiate it from the pattern projector 2. Moreover, the method of adjusting the optical system and position of the pattern projector 2 according to the value of a parameter may be used. As an example of the parallax image evaluation method, a method for evaluating the effective parallax score of the parallax image will be described here.

  As described above, the number of effective parallax points is the number of pixels from which a highly reliable parallax value (effective parallax) is obtained. The effective parallax is a parallax value of a pixel whose correspondence between the reference image Ia and the comparison image Ib can be uniquely identified, and the correspondence between the parallax values of each pixel included in the parallax image cannot be uniquely identified. It excludes the parallax value (invalid parallax) determined to be a parallax value or an abnormal value (noise) of a pixel. For example, the parallax image evaluation unit 430 detects an object region in which an object to be picked is displayed from the parallax image generated by the parallax image generation unit 230 of the stereo camera 2, and the total number of pixels in the object region and the object region The number of effective parallax points at is counted. Then, the ratio of the number of effective parallax points to the total number of pixels in the object region is calculated as an effective parallax evaluation value, and a parallax image whose effective parallax evaluation value exceeds a predetermined threshold is output. Note that the number of invalid parallaxes (invalid parallax points) is counted instead of the effective parallax points in the object region, and the reciprocal of the ratio of the invalid parallax points to the total number of pixels in the object region is calculated as an effective parallax evaluation value. May be.

  The parallax image evaluation unit 430 uses the determination result of the object determination unit 410 to perform the pattern optimization process as described above efficiently in a short time, and changes the parameter value of the pattern when evaluating the parallax image. Narrow down the evaluation range that is the range of. Specifically, the parallax image evaluation unit 430 acquires, from the evaluation reference value storage unit 420, a parameter value determined in advance as a value corresponding to the determination result of the object determination unit 410 as the evaluation reference value, and the acquired evaluation reference The evaluation range is narrowed down based on the value.

  FIG. 10 is a diagram illustrating an example of a correspondence table stored in the evaluation reference value storage unit 420. As shown in FIG. 10, the evaluation reference value storage unit 420 stores a correspondence table in which object characteristics determined by the object determination unit 410 are associated with pattern parameter values. When the parallax image evaluation unit 430 receives the determination result of the object determination unit 410, the parallax image evaluation unit 430 refers to the correspondence table stored in the evaluation reference value storage unit 420 and acquires the parameter value corresponding to the determination result of the object determination unit 410. .

  10A shows a correspondence table in which the light transmittance of an object is associated with the pattern enlargement / reduction ratio and contrast, and FIG. 10B shows the object size, the pattern enlargement / reduction ratio and contrast. Shows a correspondence table in which However, in the evaluation reference value storage unit 420, in addition to the light transmittance and size, other characteristics of the object (shape, color, distance from the stereo camera 2, etc.) determined by the object determination unit 410, and pattern parameters A correspondence table in which the values are associated with each other is also stored. The pattern scaling ratio is a parameter related to the pattern granularity (texture size), and the contrast is a parameter related to the pattern lightness distribution (texture lightness). In the correspondence table stored in the evaluation reference value storage unit 420, values of parameters other than the enlargement / reduction ratio and contrast (for example, texture frequency (appearance frequency), texture brightness level (multi-brightness), etc.) It may be associated with the characteristic of the object determined by the determination unit 410.

  The correspondence table stored in the evaluation reference value storage unit 420 is created by performing an evaluation experiment or the like in advance and is stored in the evaluation reference value storage unit 420. In the evaluation experiment, for example, a parallax image is generated for various objects, the number of effective parallax points is evaluated, and a parameter value when a good evaluation result is obtained is obtained. A correspondence table is created by associating the parameter values obtained in this evaluation experiment with the size, shape, color, light transmittance, distance from the stereo camera 2, etc. of the object, and stored in the evaluation reference value storage unit 420. .

  Here, a specific example of narrowing down the evaluation range by the parallax image evaluation unit 430 will be described. FIG. 11 is a diagram illustrating an example of narrowing down the evaluation range. In FIG. 11, the horizontal axis indicates the values of pattern parameters (for example, the enlargement / reduction ratio and contrast), and the vertical axis indicates the effective parallax evaluation value of the parallax image. Further, P0 in the figure is the current value of the parameter value (the parameter value at the determination timing when the determination by the object determination unit 410 is performed), and P1 is the evaluation reference value acquired from the evaluation reference value storage unit 420 It is.

  When the parallax image evaluation unit 430 obtains the evaluation reference value P1 from the evaluation reference value storage unit 420, first, the parallax image evaluation unit 430 selects a range that does not include the evaluation reference value P1 with the current value P0 as an end point out of all possible ranges of parameter values. By excluding from the evaluation range, the evaluation range is narrowed down. In the example shown in FIG. 11A, the evaluation range is narrowed down to the range R1 on the left side of the current value P0 by excluding the range on the right side of the current value P0 from the evaluation range.

  Next, the parallax image evaluation unit 430 changes the parameter value to the evaluation reference value P1, evaluates the parallax image generated by the parallax image generation unit 230 of the stereo camera 2 at this time, and sets the effective parallax evaluation value. calculate. Then, it is determined whether or not the calculated effective parallax evaluation value exceeds the threshold value VTh. If the effective parallax evaluation value exceeds the threshold value VTh, the parallax image is output and the evaluation ends. On the other hand, if the effective parallax evaluation value is equal to or less than the threshold value VTh, the parameter value is changed within the evaluation range. At this time, the parallax image evaluation unit 430 further narrows down the evaluation range.

  The narrowing down of the evaluation range is performed by changing the effective parallax evaluation value of the parallax image generated by the parallax image generation unit 230 of the stereo camera 2 when the parameter value is changed from the evaluation reference value P1 before the change (that is, the parameter value). Is based on whether or not the effective parallax evaluation value is higher than the evaluation reference value P1). Specifically, the parallax image evaluation unit 430 holds an effective parallax evaluation value when the parameter value is the evaluation reference value P1. The parameter value is decreased by a predetermined amount from the evaluation reference value P1, and the effective parallax evaluation value of the parallax image generated by the parallax image generation unit 230 of the stereo camera 2 at this time is calculated and held. Compare with the value. As a result of this comparison, if the effective parallax evaluation value is improved, the evaluation range is narrowed down to a parameter value range smaller than the evaluation reference value P1, and if the effective parallax evaluation value is reduced, the evaluation reference value P1. The evaluation range is narrowed down to the range of parameter values that are also larger. The direction of the narrowed down evaluation range from the evaluation reference value P1 is hereinafter referred to as “search direction”. In the example shown in FIG. 11B, since the effective parallax evaluation value is improved when the parameter value is decreased from the evaluation reference value P1, the evaluation range is narrowed down to the range R2 on the left side of the evaluation reference value P1. It is. The direction in which the parameter value is decreased (the direction toward the left side in the figure) is the search direction.

  Thereafter, the parallax image evaluation unit 430 calculates the effective parallax evaluation value of the parallax image generated by the parallax image generation unit 230 of the stereo camera 2 while changing the parameter value by a predetermined amount in the search direction within the evaluation range. . Then, when the effective parallax evaluation value exceeds the threshold value VTh, the parallax image is output, and the evaluation ends. Here, the evaluation is finished when the effective parallax evaluation value exceeds the threshold value VTh. However, the parameter value that maximizes the effective parallax evaluation value is searched, and the effective parallax evaluation value becomes maximum. The parallax image may be output to end the evaluation. In this case, for example, the effective parallax evaluation value calculated from time to time while changing the parameter value by a predetermined amount in the search direction within the evaluation range is the effective parallax value immediately before the parameter value that has changed from improvement to reduction. Can be regarded as a parameter value that maximizes, and the parallax image at that time may be output to end the evaluation. The parallax image output from the parallax image evaluation unit 430 is passed to the recognition processing unit 440. In addition, when the parallax image evaluation unit 430 does not obtain an effective parallax evaluation value exceeding the threshold value VTh even if all the narrowed evaluation ranges are evaluated, the parallax with the highest effective parallax evaluation value within the evaluation range is obtained. An image may be output.

  Returning to FIG. 7, the recognition processing unit 440 uses the third order of the object imaged by the stereo camera 2 based on the parallax image output from the parallax image evaluation unit 430 and the reference image Ia used to generate the parallax image. Recognize the shape, position and distance in the original space. The recognition result of the recognition processing unit 440 is sent to the arm control device 5. Thereby, the operation of the robot arm 6 is controlled by the arm control device 5, and the picking of the object is performed.

(Picking system operation)
Next, the operation of the picking system 1 of the present embodiment will be described with reference to FIGS. First, an outline of the operation of the picking system 1 will be described with reference to FIG. FIG. 12 is a flowchart for explaining the flow of operations by the picking system 1.

  When the operation of the picking system 1 is started, an object determination process is first performed by the object determination unit 410 (step S101). As described above, the object determination process is a process for determining at least one of the size, shape, color, light transmittance, and distance from the stereo camera 2 of the object to be picked. Note that a specific example of the object determination processing in step S101 will be described later in detail.

  Next, a pattern optimization process is performed by the parallax image evaluation unit 430 (step S102). As described above, the pattern optimization process is a process of searching for a parameter value of a pattern from which a parallax image whose effective parallax evaluation value exceeds the threshold value VTh is obtained. A specific example of the pattern optimization process in step S102 will be described later in detail.

  Next, an object recognition process is performed by the recognition processing unit 440 (step S103), and the operation of the robot arm 6 is controlled by the arm control device 5 based on the recognition result, thereby picking an object ( Step S104).

  Thereafter, it is determined whether or not the picking of all objects has been completed (step S105). If there is an object for which picking has not been completed (step S105: No), whether or not the number of picking has exceeded the picking number threshold is determined. Determination is made (step S106). If the picking count is less than the picking count threshold (step S106: No), the stereo camera 2 captures an image and generates a parallax image for the next picking target object irradiated with the pattern by the pattern projector 3. (Step S107), the processing after the object recognition processing of Step S103 is repeated.

  On the other hand, if the picking count exceeds the picking count threshold (step S106: Yes), there is a possibility that an environmental change such as a change of an object to be picked or a change in the operation state of the pattern projector 3 has occurred. Returning to S101, the process is repeated from the object determination process. If it is determined in step S105 that the picking of all objects has been completed, the operation of the picking system 1 is completed.

  Next, a specific example of the object determination process in step S101 of FIG. 12 will be described with reference to FIG. FIG. 13 is a flowchart illustrating an example of the object determination process.

  When the object determination process is started, the object determination unit 410 outputs a control command for instructing the pattern projector 3 to emit light for object determination, and also instructs the stereo camera 2 to capture a luminance image. Output a control command to Thereby, the object light is irradiated from the pattern projector 3 to the object (step S201), and the object irradiated with the object determination light is imaged by the stereo camera 2 (step S202).

  Next, the object determination unit 410 acquires the luminance image output from the reference image imaging unit 210 of the stereo camera 2, and analyzes the luminance image to thereby determine the size, shape, color, light transmittance, and stereo camera of the object. At least one of the distances from 2 is determined (step S203), and the determination result is notified to the parallax image evaluation unit 430.

  Next, a specific example of the pattern optimization process in step S102 of FIG. 12 will be described with reference to FIG. FIG. 14 is a flowchart for explaining an example of the pattern optimization process.

  When the pattern optimization process is started, the parallax image evaluation unit 430 first refers to the correspondence table stored in the evaluation reference value storage unit 420 and the parameter value corresponding to the determination result notified from the object determination unit 410 Is obtained as an evaluation reference value (step S301).

  Next, the parallax image evaluation unit 430 narrows down the evaluation range based on the evaluation reference value acquired in step S301 (step S302). Specifically, as described with reference to FIG. 11A, the parallax image evaluation unit 430 sets the current value P0 out of the entire range that can be taken by the parameter values of the pattern irradiated from the pattern projector 3 to the object. The evaluation range is narrowed down by excluding a range that does not include the evaluation reference value P1 as an end point from the evaluation range.

  Next, the parallax image evaluation unit 430 sets the value of the parameter of the pattern that the pattern projector 3 irradiates the object to the evaluation reference value (step S303). Thus, the pattern whose parameter value is set as the evaluation reference value is irradiated from the pattern projector 3 to the object, and the stereo camera 2 captures an image and generates a parallax image (step S304).

  Next, the parallax image evaluation unit 430 calculates an effective parallax evaluation value of the parallax image generated in step S304, and determines whether or not the calculated effective parallax evaluation value exceeds the threshold value VTh (step S305). If the effective parallax evaluation value exceeds the threshold value VTh (step S305: Yes), the parallax image generated in step S304 is output (step S310), and the pattern optimization process ends. On the other hand, if the effective parallax evaluation value is equal to or less than the threshold value VTh (step S305: No), the parallax image evaluation unit 430 next performs a search direction determination process (step S306). Details of a specific example of the search direction determination process will be described later.

  Next, the parallax image evaluation unit 430 determines whether or not the effective parallax evaluation value of the parallax image generated by the stereo camera 2 in the search direction determination process in step S306 exceeds the threshold value VTh (step S307). If the effective parallax evaluation value exceeds the threshold value VTh (step S307: Yes), the parallax image is output (step S310), and the pattern optimization process ends.

  On the other hand, if the effective parallax evaluation value is equal to or less than the threshold value VTh (step S307: No), the parallax image evaluation unit 430 uses the parameter value of the pattern irradiated by the pattern projector 3 to the object in the search direction determination process in step S306. A predetermined amount is changed in the determined search direction (step S308). As a result, the pattern whose parameter value is set to the evaluation reference value is irradiated from the pattern projector 3 to the object, and the stereo camera 2 captures an image and generates a parallax image (step S309).

  Thereafter, the parallax image evaluation unit 430 returns to step S307, calculates the effective parallax evaluation value of the parallax image generated in step S309, and determines whether or not the calculated effective parallax evaluation value exceeds the threshold value VTh. If the effective parallax evaluation value is less than or equal to the threshold value VTh, the processing from step S308 is repeated, and when the effective parallax evaluation value exceeds the threshold value VTh, the parallax image at that time is output (step S310), and pattern optimization processing is performed. Exit.

  Next, a specific example of the search direction determination process in step S306 of FIG. 14 will be described with reference to FIG. FIG. 15 is a flowchart illustrating an example of the search direction determination process.

  When the search direction determination process is started, the parallax image evaluation unit 430 first holds the effective parallax evaluation value (the effective parallax evaluation value calculated in step S305 in FIG. 14) when the parameter value is the evaluation reference value. (Step S401).

  Next, the parallax image evaluation unit 430 decreases the parameter value from the evaluation reference value by a predetermined amount (step S402). As a result, a pattern in which the parameter value is reduced by a predetermined amount from the evaluation reference value is irradiated onto the object from the pattern projector 3, and imaging by the stereo camera 2 and generation of a parallax image are performed (step S403).

  Next, the parallax image evaluation unit 430 calculates an effective parallax evaluation value of the parallax image generated in step S403, and the calculated effective parallax evaluation value (current effective parallax evaluation value) is a parameter value that is an evaluation reference value. It is determined whether it is larger than the effective parallax evaluation value at this time (effective parallax evaluation value held in step S401) (whether the effective parallax evaluation value has been improved) (step S404). If the effective parallax evaluation value is improved (step S404: Yes), the direction in which the parameter value is decreased is determined as the search direction (step S405). In this case, in step S307 of FIG. 14, it is determined whether or not the effective parallax evaluation value calculated in step S404 exceeds the threshold value VTh.

  On the other hand, when the effective parallax evaluation value is decreased (step S404: No), the parallax image evaluation unit 430 determines the direction in which the parameter value is increased as the search direction (step S406). Then, the parallax image evaluation unit 430 increases the parameter value by a predetermined amount from the evaluation reference value (step S407). As a result, a pattern whose parameter value is increased by a predetermined amount from the evaluation reference value is irradiated from the pattern projector 3 to the object, and imaging by the stereo camera 2 and generation of a parallax image are performed (step S408). In this case, in step S307 in FIG. 14, the effective parallax evaluation value of the parallax image generated in step S408 is calculated, and in step S307 in FIG. 14, the effective parallax evaluation value is calculated. It is determined whether or not the threshold value VTh is exceeded.

  In the example described above, the parameter value is decreased from the evaluation reference value by a predetermined amount in step S402. Alternatively, the parameter value may be increased from the evaluation reference value by a predetermined amount. In this case, the direction in which the parameter value is increased is determined as the search direction in step S405, and the direction in which the parameter value is decreased is determined as the search direction in step S406.

  As described above in detail with specific examples, the picking system 1 according to the present embodiment performs the pattern optimization processing for obtaining a highly accurate parallax image, the size of an object to be picked, At least one of shape, color, light transmittance, and distance from the stereo camera 2 is determined, and a parameter value predetermined as a value corresponding to the determination result is acquired as an evaluation reference value. Then, based on the obtained evaluation reference value, the evaluation range that is the range of the parameter value for evaluating the parallax image is narrowed down. Therefore, according to the picking system 1 of the present embodiment, pattern optimization processing for obtaining an accurate parallax image can be performed efficiently in a short time, and an accurate parallax image can be generated in a short time. The tact time required for the picking operation of the object can be shortened and the working efficiency can be improved.

  In the above embodiment, as a method in which the object determination unit 410 determines at least one of the size, shape, color, light transmittance, and distance from the stereo camera 2, the object imaged by the stereo camera 2 is determined. Although the example using a brightness | luminance image was demonstrated, the determination method by the object determination part 410 is not restricted to this. For example, an optical sensor or a laser measurement device different from the stereo camera 2 or the pattern projector 3 is incorporated in the picking system 1, and the object determination unit 410 uses the signals output from the optical sensor or the laser measurement device to detect the object. It may be configured to determine at least one of size, shape, color, light transmittance, and distance from the stereo camera 2. In addition, a user interface that receives a user operation input is provided, and the object determination unit 410 determines the size, shape, color, light transmittance, and distance from the stereo camera 2 based on the user operation input received through the user interface. It is good also as a structure which determines at least 1 of these.

[Second Embodiment]
Next, a second embodiment will be described. This embodiment is different from the first embodiment in pattern optimization processing by the parallax image evaluation unit 430. Specifically, in this embodiment, the evaluation range is narrowed down using a bisection method widely used in the field of numerical analysis as a solution algorithm for obtaining an optimal solution. In addition, since it is common with 1st Embodiment except the pattern optimization process by the parallax image evaluation part 430, below, only the pattern optimization process of this embodiment is demonstrated.

  FIG. 16 is a diagram for explaining the narrowing down of the evaluation range using the bisection method. Similarly to the example shown in FIG. 11, the horizontal axis in FIG. 16 indicates the values of pattern parameters (for example, the enlargement / reduction ratio and contrast), and the vertical axis indicates the effective parallax evaluation value of the parallax image. Further, P0 in the figure is the current value of the parameter value, and P1 is the evaluation reference value acquired from the evaluation reference value storage unit 420. In addition, P2 and P3 in the figure respectively represent parameter values corresponding to the midpoint used for narrowing down the evaluation range.

  When the parallax image evaluation unit 430 of the present embodiment obtains the evaluation reference value P1 from the evaluation reference value storage unit 420, first, the parallax image evaluation unit 430 uses the current value P0 as an end point in the entire range that the parameter value can take. The evaluation range is narrowed down by excluding ranges not included in the evaluation range. In the example shown in FIG. 16A, the evaluation range is narrowed down to the range R11 on the left side of the current value P0 by excluding the range on the right side of the current value P0 from the evaluation range.

  Next, the parallax image evaluation unit 430 changes the parameter value to the evaluation reference value P1, calculates the effective parallax evaluation value of the parallax image generated by the stereo camera 2 at this time, and calculates the calculated effective parallax evaluation value. Hold.

  Next, the parallax image evaluation unit 430 changes the value of the parameter to a value P2 corresponding to the midpoint of the evaluation range narrowed down first. At this time, the parallax image evaluation value of the parallax image generated by the stereo camera 2 is changed. calculate. Then, the parallax image evaluation unit 430 compares the effective parallax evaluation value when the parameter value is the evaluation reference value P1 and the effective parallax evaluation value when the parameter value is the value P2 corresponding to the middle point of the evaluation range. The evaluation range is further narrowed down according to the result.

  That is, if the effective parallax evaluation value when the parameter value is the evaluation reference value P1 is larger than the effective parallax evaluation value when the parameter value is the value P2 corresponding to the middle point of the evaluation range, the midpoint of the evaluation range The evaluation range is narrowed down to a range including the evaluation reference value P1 with the value P2 corresponding to. On the other hand, if the effective parallax evaluation value when the parameter value is the value P2 corresponding to the middle point of the evaluation range is greater than the effective parallax evaluation value when the parameter value is the evaluation reference value P1, the middle point of the evaluation range The evaluation range is narrowed down to a range that does not include the evaluation reference value P1 with the value P2 corresponding to. In the example shown in FIG. 16B, since the effective parallax evaluation value when the parameter value is P2 is larger than the effective parallax evaluation value when the parameter value is the evaluation reference value P1, the evaluation range is the range R12 on the left side of P2. It is narrowed down to. Note that the effective parallax evaluation value when the parameter value is P2 is held to narrow down the next evaluation range.

  Next, the parallax image evaluation unit 430 changes the parameter value to a value P3 corresponding to the midpoint of the evaluation range narrowed down last time, and calculates the effective parallax evaluation value of the parallax image generated by the stereo camera 2 at this time. To do. Then, the parallax image evaluation unit 430 compares the effective parallax evaluation value when the parameter value is P2 with the effective parallax evaluation value when the parameter value is P3, and further narrows down the evaluation range according to the result.

  That is, if the effective parallax evaluation value when the parameter value is P2 is larger than the effective parallax evaluation value when P3, the evaluation range is narrowed down to a range from P2 to P3. On the other hand, if the effective parallax evaluation value when the parameter value is P3 is larger than the effective parallax evaluation value when P2 is set, the evaluation range is narrowed down to a range not including P2 with P3 as an end point. In the example shown in FIG. 16C, since the effective parallax evaluation value when the parameter value is P2 is larger than the effective parallax evaluation value when P3, the evaluation range is narrowed down to a range R13 from P2 to P3. Note that the effective parallax evaluation value when the parameter value is P3 is retained for narrowing down the next evaluation range.

  The parallax image evaluation unit 430 repeats the narrowing down of the evaluation range as described above a predetermined number of times to bring the parameter value close to the optimum value, and outputs the latest parallax image obtained when the evaluation range is narrowed down the predetermined number of times. . In this embodiment, the evaluation range is narrowed down as described above, and then the parallax image is output and the evaluation is ended. However, as in the first embodiment, the effective parallax of the parallax image is output. When the evaluation value exceeds the threshold value VTh, the parallax image may be output to end the evaluation.

  FIG. 17 is a flowchart for explaining pattern optimization processing according to the second embodiment. The process shown in the flowchart of FIG. 17 is executed in step S102 in the flowchart of FIG. 12 instead of the pattern optimization process of the first embodiment shown in FIG.

  When the pattern optimization process is started, the parallax image evaluation unit 430 first refers to the correspondence table stored in the evaluation reference value storage unit 420 and the parameter value corresponding to the determination result notified from the object determination unit 410 Is obtained as an evaluation reference value (step S501).

  Next, the parallax image evaluation unit 430 narrows down the evaluation range based on the evaluation reference value acquired in step S501 (step S502). Specifically, as described with reference to FIG. 16A, the parallax image evaluation unit 430 sets the current value P0 out of the entire range that can be taken by the parameter values of the pattern irradiated from the pattern projector 3 to the object. The evaluation range is narrowed down by excluding a range that does not include the evaluation reference value P1 as an end point from the evaluation range.

  Next, the parallax image evaluation unit 430 sets the value of the parameter of the pattern that the pattern projector 3 irradiates the object to the evaluation reference value (step S503). As a result, the pattern with the parameter value set as the evaluation reference value is irradiated from the pattern projector 3 to the object, and the stereo camera 2 captures an image and generates a parallax image (step S504).

  Next, the parallax image evaluation unit 430 calculates an effective parallax evaluation value of the parallax image generated in step S504, and holds the calculated effective parallax evaluation value (step S505).

  Next, the parallax image evaluation unit 430 sets the value of the parameter of the pattern irradiated to the object by the pattern projector 3 to a value corresponding to the midpoint of the evaluation range narrowed down in step S502 (step S506). As a result, a pattern whose parameter value is set to a value corresponding to the midpoint of the evaluation range is irradiated onto the object from the pattern projector 3, and the stereo camera 2 captures an image and generates a parallax image (step S507).

  Next, the parallax image evaluation unit 430 calculates an effective parallax evaluation value of the parallax image generated in step S507, and determines whether or not the calculated effective parallax evaluation value is larger than the effective parallax evaluation value held in step S505. Determination is made (step S508). If the effective parallax evaluation value of the parallax image generated in step S507 is larger than the effective parallax evaluation value held in step S505 (step S508: Yes), the evaluation range is the parameter value set in step S506 (this). (The value corresponding to the middle point of the evaluation range up to) is narrowed down to a range not including the evaluation reference value (step S509). On the other hand, if the effective parallax evaluation value of the parallax image generated in step S507 is equal to or smaller than the effective parallax evaluation value held in step S505 (step S508: No), the evaluation range is set to the parameter value set in step S506. Is narrowed down to a range including the evaluation reference value (step S510).

  Thereafter, the parallax image evaluation unit 430 determines whether or not the number of times of narrowing down the evaluation range has reached a predetermined number (step S511). If the number of narrowing down of the evaluation range has not reached the predetermined number (step S511: No), the parameter value set in step S506 (a value corresponding to the middle point of the evaluation range so far) is replaced with the evaluation reference value. (Step S512), the processing from step S505 onward is repeated. On the other hand, when the number of times of narrowing down the evaluation range has reached a predetermined number (step S511: Yes), the latest parallax image at that time is output (step S513), and the pattern optimization process is terminated.

  As described above, in the present embodiment, by narrowing down the evaluation range using the bisection method, pattern optimization processing for obtaining a highly accurate parallax image can be performed efficiently in a short time. Therefore, according to the present embodiment, as in the first embodiment, a highly accurate parallax image can be generated in a short time, and the tact time required for the object picking operation can be shortened to increase work efficiency. Can do.

[Third Embodiment]
Next, a third embodiment will be described. In the present embodiment, the position of the pattern projector 3 relative to the object is handled as a parameter of the pattern that the pattern projector 3 irradiates the object. The position of the pattern projector 3 relative to the object is equivalent to the enlargement / reduction ratio described as one of the pattern parameters in the first embodiment. That is, when the position of the pattern projector 3 approaches the object, the pattern irradiated to the object is enlarged, and when the position of the pattern projector 3 moves away from the object, the pattern irradiated to the object is reduced. Become.

  FIG. 18 is a schematic configuration diagram of the picking system 1 of the present embodiment. As shown in FIG. 18, in the picking system 1 of the present embodiment, a position adjusting mechanism 9 is added to the configuration of the picking system 1 of the first embodiment (see FIG. 5). The position adjusting mechanism 9 adjusts the position of the pattern projector 3 with respect to the object to be picked. For example, an electromagnetic actuator that can linearly control the adjustment amount or a stepping motor that can be adjusted in a predetermined unit is used. Can be used.

  In the picking system 1 of the present embodiment, the parallax image evaluation unit 430 of the information processing device 4 adjusts the position of the pattern projector 3 relative to the object to be picked by outputting a control command to the position adjustment mechanism 9. . The parallax image evaluation unit 430 uses the position of the pattern projector 3 adjusted by the position adjustment mechanism 9 as a parameter of the pattern irradiated to the object from the pattern projector 3, and is the same method as in the first embodiment and the second embodiment. Thus, pattern optimization processing is performed. As a result, similar to the first embodiment and the second embodiment, a highly accurate parallax image can be generated in a short time, and the tact time required for the object picking operation can be shortened to improve work efficiency. it can.

(Supplementary explanation)
For example, the CPU 12 of FIG. 6 uses the RAM 14 as a work area for the functional components (the object determination unit 410, the parallax image evaluation unit 430, the recognition processing unit 440, etc.) of the information processing apparatus 4 described in each of the above-described embodiments. This can be realized by executing a program stored in the ROM 13 or the hard disk device. In this case, the program is recorded in a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD in a format that can be installed in the information processing apparatus 4 or an executable format. Provided. Further, the program may be provided by being stored on a computer connected to a network such as the Internet and downloaded to the information processing apparatus 4 via the network. Furthermore, the program may be configured to be provided or distributed via a network such as the Internet. Further, the program may be provided by being incorporated in advance in, for example, the ROM 13 in the information processing apparatus 4.

  In addition, the functional components of the information processing apparatus 4 described in the above-described embodiment and each modification use part or all of dedicated hardware such as ASIC (Application Specific Integrated Circuit) or FPGA. Can also be realized. In addition, the functional components of the information processing apparatus 4 described in the above-described embodiments and modifications are partly or entirely different from the information processing apparatus 4 such as the stereo camera 2 and the pattern projector 3. You may make it provide in an apparatus.

  Although specific embodiments of the present invention have been described above, the above-described embodiments show an application example of the present invention. The present invention is not limited to the above-described embodiment as it is, and can be embodied with various modifications and changes without departing from the scope of the invention in the implementation stage. For example, in the above-described embodiment, the configuration in which the reference image capturing unit 210 and the comparative image capturing unit 220 are realized by the two image capturing units 10a and 10b in the stereo camera 2 is exemplified. However, the stereo camera having the parallax image generating unit 230 is illustrated. 2 may be configured such that a separate camera is connected to the stereo camera 2 and at least one of the reference image capturing unit 210 and the comparative image capturing unit 220 is realized by the separate camera.

  In the above-described embodiment, the stereo camera 2 is described as being provided. However, the present invention is not limited to this, and a monocular camera may be provided instead of the stereo camera 2. In this case, the configuration may be such that the object as a subject is imaged a plurality of times by moving the monocular camera, and distance information to the object is obtained using the image captured a plurality of times. The structure which acquires distance information may be sufficient.

  In the above-described embodiment, the pattern enlargement / reduction ratio and contrast are exemplified as parameters whose values are changed by the pattern optimization process. However, the parameters handled in the pattern optimization process are not limited to this example. Other parameters related to one or more of the size, shape, color, density, and brightness of the texture of the image may be handled. For example, the frequency of pattern texture (appearance frequency) and the luminance level of texture (multi-brightness) are pattern parameters that affect the effective parallax evaluation value of parallax images. You may make it handle.

  For example, when the size of the object is small and the texture frequency of the pattern is low, the frequency of appearance of the texture is low, and sufficient texture is not applied to the object surface. As a result, the number of effective parallax points may be reduced. In order to irradiate the object surface with sufficient texture, the number of effective parallax points increases by increasing the appearance frequency of the texture. Therefore, when the object size is determined by the object determination unit 410 and the object size is smaller than the texture frequency of the pattern, by increasing the texture frequency value as the pattern parameter in the pattern optimization process, The appearance frequency increases and the number of effective parallax points can be increased.

  Further, for example, when the color of the object surface is a bright color such as white or yellow, the pattern of the irradiated pattern is clearly irradiated on the object surface. At this time, if the pattern is only binary white / black, an error in the stereo matching process occurs when a similar black-and-white repetition appears as a corresponding pixel of the reference image and the comparison image in the parallax image generation unit 230. There may be a case where matching occurs and the number of effective parallax points decreases. To increase the number of effective parallax points by reducing false matching in the stereo matching process, it is effective to irradiate textures with multiple brightness levels so that similar repetitions do not appear and prevent similar repetitions from appearing. It is. For this reason, when the object color is determined by the object determination unit 410 and the luminance level of the texture of the pattern is less than the color of the object, the luminance level of the texture is increased (multi-brightness), thereby performing stereo matching processing. The number of effective parallax points can be increased by preventing erroneous matching in

DESCRIPTION OF SYMBOLS 1 Picking system 2 Stereo camera 3 Pattern projector 4 Information processing apparatus 5 Arm control apparatus 9 Position adjustment mechanism 210 Reference image imaging unit 220 Comparative image imaging unit 230 Parallax image generation unit 410 Object determination unit 420 Evaluation reference value storage unit 430 Parallax image evaluation Part 440 recognition processing part

Japanese Patent No. 4473136

Claims (10)

Pattern irradiation means for irradiating the object with a pattern;
Imaging means for imaging the object irradiated with the pattern;
Generating means for generating a parallax image from a captured image captured by the imaging means;
An evaluation unit that evaluates the parallax image generated by the generation unit while changing a value of a parameter related to the pattern, and outputs the parallax image based on an evaluation result;
Determination means for determining at least one of the size, shape, color, light transmittance, and distance from the imaging means of the object,
The evaluation means acquires the parameter value predetermined as a value corresponding to the determination result of the determination means as an evaluation reference value, and evaluates the parallax image based on the acquired evaluation reference value The parallax image generation system which narrows down the evaluation range which is the range of the value of.   The evaluation means narrows down the evaluation range by excluding, from the evaluation range, a range that does not include the evaluation reference value, with the current value of the parameter as an end point, out of all possible ranges of the parameter value The parallax image generation system according to claim 1.   The evaluation means further changes the value of the parameter within the evaluation range when the evaluation result of the parallax image when the current value of the parameter is changed to the evaluation reference value does not satisfy a predetermined reference; The parallax image generation system according to claim 2.   The evaluation means further narrows down the evaluation range based on whether or not an evaluation result of the parallax image when the parameter value is changed from the evaluation reference value is improved from an evaluation result before the change. Item 4. The parallax image generation system according to Item 3.   The parallax image generation system according to claim 4, wherein the evaluation unit narrows down the evaluation range when changing the parameter value from the evaluation reference value by a bisection method.   The evaluation means evaluates the effective parallax score of the parallax image, and ends the evaluation and outputs the parallax image when the evaluation value of the effective parallax score exceeds a threshold value. The parallax image generation system described in 1.   The parallax image generation system according to claim 1, wherein the parameter is a parameter related to any one or more of a size, shape, color, density, and brightness of a texture included in the pattern. A position adjusting means for adjusting the position of the pattern irradiation means with respect to the object;
The parallax image generation system according to any one of claims 1 to 7, wherein the parameter includes a position of the pattern irradiation unit with respect to the object.   The determination unit determines at least one of a size, a shape, a color, a light transmittance, and a distance from the imaging unit based on a captured image captured by the imaging unit. The parallax image generation system according to any one of the above. A parallax image generation method executed by a parallax image generation system,
Irradiating an object with a pattern, imaging the object irradiated with the pattern with an imaging unit, and generating a parallax image from the captured image;
Determining at least one of the size, shape, color, light transmittance, and distance from the imaging means of the object and outputting a determination result;
Evaluating the parallax image generated by the generation unit while changing the value of a parameter related to the pattern, and outputting the parallax image based on an evaluation result,
When evaluating the parallax image, the parameter value predetermined as a value corresponding to the determination result is acquired as an evaluation reference value, and the parallax image is evaluated based on the acquired evaluation reference value A parallax image generation method that narrows down an evaluation range that is a range of parameter values.