JP6548076B2 - Pattern image projection apparatus, parallax information generation apparatus, pattern image generation program - Google Patents

Description

  The present invention relates to a pattern image projection device, a parallax information generation device, and a pattern image generation program.

  Conventionally, there is known a pattern image projection apparatus that projects a pattern image toward an imaging area when imaging an object in an imaging area from different viewpoints by a plurality of imaging units to generate parallax information of the object. There is.

  In Patent Document 1, the reference image obtained by imaging the subject (target object) irradiated with the light projection pattern (pattern image) from the pattern image projection apparatus from different viewpoints is matched (matching process) A three-dimensional shape measuring apparatus is disclosed that generates a distance image (disparity information) based on the disparity value between corresponding matched pixels. In this three-dimensional shape measuring apparatus, in order to reduce the occurrence of mismatching at the time of corresponding spotting, which has occurred when irradiating a subject with a light projection pattern having a periodic pattern, the size of a dot, Generate a non-periodic floodlight pattern consisting of a pattern whose line length, thickness, position, density, etc. are irregular.

  Generally, a pattern image projection apparatus projects a pattern image toward an imaging area via an optical system such as a lens, but the optical characteristics of the optical system differ depending on the point in the imaging area. For example, optical characteristics such as brightness (brightness), distortion, and contrast of the pattern image are different between a point near the optical axis center of the optical system and a point distant from the optical axis center. Such optical characteristics affect the matching accuracy of the matching processing that associates the pixels that project the same point in the imaging region among the plurality of captured images. Therefore, if the optical characteristics differ depending on the point in the imaging area, a point with relatively low matching accuracy locally occurs in the imaging area. Therefore, there may be a possibility that appropriate disparity information can not be obtained for an object at such a point.

  Similarly, also in the optical system of the imaging means in the case where an object in the imaging area on which the pattern image is projected is imaged from different points by a plurality of imaging means, the optical characteristics thereof differ depending on the point in the imaging area. Therefore, even if a uniform pattern image is projected, optical characteristics such as brightness (brightness), distortion, and contrast of the captured image differ depending on the point in the imaging region. Since such optical characteristics also affect the matching accuracy of the matching process, even when the optical characteristics of the imaging means differ depending on the point in the imaging region, a point with relatively low matching accuracy locally occurs. It will Therefore, there is a possibility that appropriate disparity information can not be obtained for an object at such a point.

  In addition, the point where the matching accuracy is relatively low is not limited to the point away from the center of the optical axis of the optical system, but may vary depending on the configuration of the optical system. Therefore, the point where the matching accuracy is relatively low, that is, the portion where appropriate parallax information can not be obtained may differ depending on the configuration of the optical system and the like.

  Moreover, even if it uses the pattern image projection apparatus of an indication disclosed to patent document 1 which projects a non-periodic light projection pattern (pattern image), the subject mentioned above can not be solved. In this light projection pattern, the pattern image portion projected on each imaging area portion having different optical characteristics with respect to the image projection means and the imaging means is not necessarily the pattern image portion optimized for each imaging area portion . Therefore, if the optical characteristics of the image projection means and the imaging means differ depending on the imaging region location, a point having a relatively low matching accuracy locally occurs, and an object present at such a point In some cases, appropriate disparity information can not be obtained.

In order to solve the problems described above, according to the present invention, when parallax information of the object is generated by the parallax information generation unit on the basis of captured images obtained by imaging an object in an imaging region from different viewpoints by a plurality of imaging units. And a pattern image projection device having an image projection unit that projects a pattern image toward an imaging region, wherein the pattern image is generated based on the number of parallax values included in the parallax information generated by the parallax information generation unit. An image generation unit is provided, and the pattern image has different pattern image portions projected onto the respective imaging region portions in accordance with the imaging region portions having different optical characteristics with respect to at least one of the image projection unit and the imaging portion. It is characterized by

  According to the present invention, even if an imaging area location with a relatively low matching accuracy exists in the imaging area, because the optical characteristics of the image projection means and the imaging means differ depending on the imaging area location, etc. The excellent effect is obtained that appropriate parallax information can be obtained for an object present in the imaging region.

It is a block diagram showing a robot picking system in an embodiment. It is explanatory drawing which shows arrangement | positioning of the principal part of the robot picking system. It is explanatory drawing which shows an example of the light quantity change with respect to the view | field angle of the camera which comprises the robot picking system, and a projector. FIG. 7 is an explanatory view showing an example of division of pattern image portions created under different pattern conditions in the pattern image in the embodiment. It is a flow chart which shows a flow of initial image pattern generation processing in an embodiment. It is explanatory drawing which shows arrangement | positioning of the camera which comprises the robot picking system, and a projector. It is an explanatory view showing an overlap state of an optical characteristic of a camera, and an optical characteristic of a projector on a pattern image projected. It is explanatory drawing which shows an example of the pattern image which has a pattern image part in which particle size and contrast differ. It is a flowchart which shows the flow of the image pattern change process in embodiment. It is explanatory drawing which shows arrangement | positioning of the camera which comprises the robot picking system in a modification, and a projector. It is explanatory drawing for demonstrating that the granularity of the pattern image projected on a workbench surface changes with the distances from a projector to a workbench surface. It is a flow chart which shows a flow of initial image pattern generation processing in a modification.

Hereinafter, an embodiment in which a parallax information generating device provided with a pattern image projecting device according to the present invention is applied to a robot picking system will be described.
In the robot picking system in the present embodiment, for example, one work (for example, a T-shaped pipe) is taken out by a robot arm from a box in which a large number of works (objects) are stacked in bulk, It is something to deliver. In this system, the inside of the box is imaged from above, and individual workpieces in the box are distinguished and recognized based on the imaged image. Then, based on the recognition result, the operation of the robot arm is controlled such that one hand is picked up from above by the hand portion of the robot arm and taken out of the box.
Note that the parallax information generation device provided with the pattern image projection device according to the present invention is not limited to the robot picking system, and for example, is equipped with an image analysis device that performs recognition processing of an object present in an imaging region based on a captured image Can be applied to other systems.

FIG. 1 is a block diagram showing a robot picking system in the present embodiment.
FIG. 2 is an explanatory view showing the arrangement of the main part of the robot picking system of the present embodiment.
The robot picking system of the present embodiment mainly includes an imaging device including a stereo camera including a camera unit 100 and an image processing unit 200, a robot arm control device including a recognition processing unit 300 and a robot arm control unit 400, It comprises a projector 600 as an image projection means and a pattern image projection apparatus provided with a personal computer (hereinafter referred to as a "personal computer") 500 for controlling the projector 600. The distance from the projector 600 to the work bench surface 601 on which the work is placed and the distance from the work bench surface 601 to the camera unit 100 are substantially equal.

  The camera unit 100 is composed of cameras 101A and 101B as two imaging means, and the configurations of the two cameras 101A and 101B are the same. The cameras 101A and 101B mainly include an optical system such as an imaging lens, an image sensor including a pixel array in which light receiving elements are two-dimensionally arranged, and analog electric signals output from the image sensor (each on the image sensor A signal processing unit generates and outputs captured image data obtained by converting the amount of light received by the light receiving element into a digital electric signal.

  The image processing unit 200 receives captured image data output from the camera unit 100 and performs various image processing. Captured image data output from each of the cameras 101A and 101B of the camera unit 100 is input to image correction units 201A and 201B that perform correction processing to convert the image into an image obtained by the ideal pinhole model. The image correction units 201A and 201B are provided for each of the cameras 101A and 101B, and are configured by an FPGA (Field-Programmable Gate Array), a memory, and the like. The image correction units 201A and 201B execute correction processing such as magnification correction, image center correction, distortion correction, and the like on the captured image data input from the cameras 101A and 101B using the FPGA. The memory stores correction parameters and the like used for the correction process. Note that, in place of the FPGA, for example, an application specific integrated circuit (ASIC) may be employed.

  The captured image data (corrected image data) after correction processing output from the image correction units 201A and 201B is sent to the parallax information calculation unit 202 as parallax information generation means. The parallax information calculation unit 202 performs processing of generating parallax image data obtained from the corrected image data obtained by imaging with each of the cameras 101A and 101B. Specifically, in order to obtain parallax image data from the corrected image data output from the image correction units 201A and 201B, the parallax value of the corresponding image portion between the two corrected images is calculated.

  The parallax value referred to here means that one of the images taken by each of the cameras 101A and 101B is a reference image and the other is a comparison image, and the comparison value on the image portion on the reference image corresponding to the same point in the imaging area The positional displacement amount of the image portion is calculated as the parallax value of the image portion. By using the principle of triangulation, it is possible to calculate the distance from the parallax value to the same point in the imaging region corresponding to the image portion. The parallax image data is a parallax image in which pixel values corresponding to parallax values calculated for each image portion on the reference image data (for example, captured image data a captured by the camera 101A) are represented as pixel values of the respective image portions It is shown. The parallax image data output from the parallax information calculation unit 202 is sent to the recognition processing unit 300 in the subsequent stage, and is also sent from the external I / F 203 to the personal computer 500.

  The recognition processing unit 300 uses the parallax image data output from the parallax information calculation unit 202 to perform processing for individually recognizing a large number of works contained in a box present in the imaging region. Specifically, for example, an edge portion where a pixel value (disparity value) changes largely on a parallax image is detected, and an image portion showing an individual work is identified from the detection result. Then, for example, the image part having the largest parallax value average among the identified image parts is identified, and the work calculated from the horizontal position of the workpiece corresponding to the identified image part and the parallax value average of the image part The distance to the point is output to the robot arm control unit 400 that controls the operation of the robot arm as a result of workpiece recognition. The work identified in this manner is the work stacked on top of the box.

  The robot arm control unit 400 can grasp the position (horizontal direction position and distance) of one workpiece to be picked based on the workpiece recognition result output from the recognition processing unit 300. The robot arm control unit 400 controls the robot arm operation so as to pick the workpiece present at the grasped position from above with the hand unit of the robot arm, take it out of the box, and deliver it to the processing device in the subsequent stage.

  The personal computer 500 controls the projector 600 in order to cause the projector 600 to project a pattern image onto a workpiece that is an object. The personal computer 500 according to the present embodiment mainly includes an operation unit including a central processing unit (CPU) and a storage device such as a read only memory (ROM), a random access memory (RAM), or a hard disk drive (HDD). It has a general function of a general-purpose personal computer such as a storage unit, a communication unit including a communication interface, and an input unit such as a keyboard and a pointing device.

  In the present embodiment, the personal computer 500 executes various computer programs stored in the storage unit by the CPU to realize various functions necessary for the operation as a pattern image projection apparatus together with the projector 600. In particular, in the first embodiment, the pattern image generation program is executed by the CPU to execute pattern image generation processing for generating a pattern image projected by the projector 600 toward the work. The personal computer 500 is provided with a parallax score calculation unit 501, a pattern image generation unit 502, and an external I / F 503 as a configuration for executing this pattern image generation process, and recognizes a workpiece as an object with high recognition accuracy. Generate a pattern image that can be

  The pattern image generation processing is configured by the parallax score calculation unit 501 and the pattern image generation unit 502 in the personal computer 500. The parallax score calculation unit 501 acquires parallax image data output from the parallax information calculation unit 202 in the image processing unit 200 via the external I / Fs 203 and 503, and a designated image portion designated in advance from the parallax image data. Count the number of effective disparity values in The pattern image generation unit 502 generates a pattern image in which the number of effective parallax values counted by the parallax score calculation unit 501 satisfies a predetermined condition.

In the present embodiment, the reason for generating a pattern image for enhancing the recognition accuracy of the workpiece as the object is as follows.
In the recognition processing by the recognition processing unit 300, as described above, an image portion that shows a work is recognized based on parallax information between captured images captured by the two cameras 101A and 101B. The recognition accuracy in such processing depends on the number of parallax values (the number of pixels on the parallax image forming the three-dimensional object area) accurately calculated for the image portion that shows the work. The calculation of the parallax value is premised on accurately specifying (matching) a corresponding image portion between both captured images (an image portion on each image corresponding to the same point in the imaging region).

  The matching process is performed, for example, for each image area in the comparison image (horizontal direction position (image horizontal direction) which is the same vertical position (image vertical direction position) as the target image area in the reference image including the corresponding image portion to be matched. The correlation calculation is performed for each of the image areas having different positions) in relation to the arrangement of a plurality of pixel values (brightness values) constituting the target image area in the reference image. Then, the image area having the highest correlation among the image areas in the comparison image in which the correlation of a predetermined level or more is recognized is specified as the corresponding image area corresponding to the target image area in the reference image. Thereby, the corresponding image portion between both captured images is specified.

  Here, the projector 600 projects the pattern image through the optical system, but the optical characteristics of the optical system are not uniform throughout the projected pattern image. The projector 600 of the present embodiment is the center of the optical axis of the optical system (hereinafter, also referred to as “the center of the pattern image”, the center of the pattern image does not indicate the central position of the pattern image). The brightness is highest at the point where the brightness decreases as the distance from the optical axis center (the center of the pattern image) decreases. That is, the brightness (brightness) of the pattern image between the vicinity of the optical axis center of the optical system of the projector 600 (the central portion O of the pattern image) and the point distant from the optical axis center (the peripheral portion of the pattern image) Is different.

  As a result, for example, when the pattern image is projected so as to obtain an optimal brightness (brightness) at the central portion O of the pattern image, that is, a brightness (brightness) for which an appropriate parallax value can be calculated for the central portion O. As shown in FIG. 3, in the peripheral portion of the pattern image, the brightness is lower than the optimal brightness (brightness). When the brightness is low, the contrast of the captured image is lowered, so that the accuracy of the correlation operation is lowered, and it becomes difficult to perform the accurate matching process. As a result, with respect to a workpiece on which the peripheral portion of the pattern image is projected, accurate matching processing can not be performed, and there is a high possibility that the parallax value can not be calculated or an incorrect parallax value is calculated. As a result, the number of correctly calculated parallax values (the number of pixels of the image portion on the parallax image corresponding to the image portion projecting the work) on the image portion projecting the work decreases, and the recognition processing unit 300 The accuracy of recognition processing in

  On the contrary, for example, when the pattern image is projected so as to obtain the optimum brightness (brightness) in the peripheral part of the pattern image, the central part O of the pattern image is brighter than the optimum brightness (brightness) It becomes. If the brightness is too high, among the light receiving elements that receive light from the central portion O, there is a light receiving element whose light reception amount exceeds a saturation value which is the upper limit value of the light quantity detection range of the light receiving element. It can be difficult to perform an accurate matching process. In this case, with respect to a workpiece on which the central portion of the pattern image is projected, accurate matching processing can not be performed, and there is a high possibility that the parallax value can not be calculated or an incorrect parallax value is calculated. As a result, the number of parallax values accurately calculated for the image portion showing the work decreases, and the recognition processing accuracy in the recognition processing unit 300 is deteriorated.

  In addition, the optical characteristics of the two cameras 101A and 101B of the camera unit 100 that capture the workpiece on which the pattern image is projected from different points are not uniform in optical characteristics over the entire imaging region. As shown in FIG. 3, in the cameras 101A and 101B of the present embodiment, the amount of light received is the largest at the optical axis center of the optical system (the center of the image sensor of each camera), and the further away from the optical axis center (the center of the image sensor) It has an optical characteristic that the amount of received light decreases. That is, the brightness (brightness) of the captured image between the vicinity of the optical axis center of the optical system of the cameras 101A and 101B (central part of the image sensor) and the point distant from the optical axis center (peripheral part of the image sensor) ) Is different.

  As a result, for example, even if an image of a workpiece on which a pattern image with uniform brightness (brightness) is projected is imaged, the brightness (brightness) is such that an appropriate parallax value can be calculated for the central portion of the captured image. Sometimes, in the peripheral area of the captured image, the brightness is lower than the optimal brightness (brightness). When the brightness of the captured image is low, the contrast of the captured image is lowered, so that the accuracy of the correlation calculation is lowered, which makes it difficult to perform accurate matching processing. As a result, accurate matching processing can not be performed for a work displayed on the peripheral portion of the captured image, and there is a high possibility that the parallax value can not be calculated or an incorrect parallax value is calculated. As a result, the number of parallax values accurately calculated for the image portion showing the work decreases, and the recognition processing accuracy in the recognition processing unit 300 is deteriorated.

  Conversely, for example, when the brightness (brightness) is such that an appropriate parallax value can be calculated for the peripheral part of the captured image, the brightness at the central part of the captured image is higher than the optimal brightness (brightness) It becomes. If the brightness of the captured image is too high, a light receiving element whose light reception amount exceeds the saturation value, which is the upper limit value of the light amount detection range of the light receiving element, is present in the light receiving element corresponding to the center of the captured image. It may be difficult to perform accurate matching processing. In this case, accurate matching processing can not be performed for the work shown in the center of the captured image, and there is a high possibility that the parallax value can not be calculated or an incorrect parallax value is calculated. As a result, the number of parallax values accurately calculated for the image portion showing the work decreases, and the recognition processing accuracy in the recognition processing unit 300 is deteriorated.

  In particular, there may be a case where an imaging area location projected on the periphery of a captured image imaged by the two cameras 101A and 101B of the camera unit 100 and an imaging area location on which the periphery of the pattern image is projected by the projector 600 match. is there. In this case, for example, when the pattern image is projected under the optimum condition at the central portion O of the pattern image and imaging is performed under the optimum condition at the central portion of the captured image, the optical characteristics of the projector 600 (the pattern image The light amount in the peripheral portion is small) and the optical characteristic of each camera 101A and 101B (the light receiving amount in the peripheral portion of the pickup image is small) is superimposed, and the pickup image portion (the pickup image It is particularly difficult to perform accurate matching processing by greatly reducing the accuracy of the correlation operation in the matching processing in the peripheral portion, and the recognition processing accuracy in the recognition processing unit 300 is particularly low.

  Therefore, in the present embodiment, in order to suppress the decrease in the recognition processing accuracy in the recognition processing unit 300 locally in the imaging region, the imaging region location where the optical characteristics of the projector 600 and the cameras 101A and 101B are different from each other The parallax image data is generated using pattern images in which each pattern image portion to be projected is created under different pattern conditions. Specifically, as shown in FIG. 4, the pattern image is divided into w total in the horizontal direction and h total in the vertical direction w × h pattern image portions, and each pattern image portion is subjected to different pattern conditions Create and generate a pattern image.

  In the present embodiment, in the initial setting before operating the robot picking system, generation processing of an initial pattern image is executed. In the present embodiment, each imaging is performed in consideration of the optical characteristics of the projector 600 (decrease in the amount of light in the peripheral portion of the pattern image) and the optical characteristics of the cameras 101A and 101B (decrease in the amount of light in the peripheral portion of the captured image). A pattern image in which the pattern conditions of the pattern image portion to be projected onto the area portion are adjusted is generated as an initial pattern image. The pattern conditions include the particle size and the contrast of the pattern, but the pattern conditions are not particularly limited as long as the recognition processing accuracy in the recognition processing unit 300 is improved. The pattern image generation unit 502 according to the present embodiment can generate patterns with different particle sizes using the spatial frequency as a parameter. Therefore, in the present embodiment, when generating the initial pattern image, the spatial frequency of the pattern image portion projected onto each imaging area is adjusted, and the pattern image portion of the granularity suitable for each optical characteristic Generate an initial pattern image with (w, h). The higher the spatial frequency, the finer the granularity of the pattern.

FIG. 5 is a flowchart showing a flow of initial image pattern generation processing in the present embodiment.
In the initial pattern image generation process, first, the pattern image generation unit 502 of the personal computer 500 generates a reference pattern image with uniform pattern conditions, and the personal computer 500 controls the projector 600 to direct this reference pattern image to the work bench surface 601 Project (S1). The pattern image at this time is one in which all pattern image portions (w, h) are created at the same spatial frequency f0 (w, h) (with the same granularity). The pattern image projected from the projector 600 is not distorted by the optical system of the projector 600 or the like.

  When a reference pattern image having a uniform pattern condition (spatial frequency) is projected, each camera 101A, 101B of the camera unit 100 performs an imaging operation (S2), and one frame of captured image data (reference image data and comparison image data) Get). The captured image data is corrected by the image correction units 201A and 201B, and the parallax information calculation unit 202 generates parallax image data (disparity information) from the corrected image data (S3). The parallax image data generated in this manner is sent to the pattern image generation unit 502 via the external I / F 203 and 503, and the pattern image generation unit 502 performs general triangulation from the received parallax image data. The distance from the work bench surface 601 to the camera unit 100 is calculated using the principle (S4).

  Subsequently, the pattern image generation unit 502 of the personal computer 500 determines relative position information of the projector 600 based on the installation position of the camera unit 100 from the received parallax image data or reference image data corresponding to the parallax image data (XYZ The relative position coordinates and the rotation angle about the Z axis are calculated (S5). Specifically, as shown in FIG. 2, the X axis and the Y axis are parallel to the work surface, the Z axis is normal to the work surface, and the installation position (reference position) of the camera unit 100 is X-coordinate, Y-coordinate, Z-coordinate, Z-axis rotation angle) = (x0, y0, z0, rz0), and the installation position of the projector 600 is (X-coordinate, Y-coordinate, Z-coordinate, Z-axis rotation angle) = (x , Y, z, rz), the relative position information of the projector 600 is (x-x0, y-y0, z-z0, rz-rz0).

  Here, the optical characteristic of the projector 600 (the decrease in the amount of light at the peripheral portion of the pattern image) is represented by the peripheral light amount coefficient Qp (w, h). The peripheral light amount coefficient Qp (w, h) is, as shown in FIG. 4, when the pattern image is divided into w total in the horizontal direction and h total in the vertical direction w × h pattern image portions, For example, the ratio of the relative amount of light of each pattern image portion to the amount of light of the pattern image portion located at the central portion O of the pattern image is shown. In this case, the peripheral light amount coefficient Qp of the pattern image portion located at the central portion O of the pattern image is 1, and the peripheral light amount coefficient Qp of the other pattern image portions is usually less than 1 and The peripheral light amount coefficient Qp takes a smaller value as the pattern image portion is farther from the center.

The peripheral light amount coefficient Qp (w, h) indicating the optical characteristics of the projector 600 has a smaller peripheral light amount coefficient Qp as the pattern image portion is farther from the center of the pattern image (optical axis of the projector). Thus, the peripheral light amount coefficient Qp (w, h) corresponding to each pattern image (w, h) of the pattern image projected on the work bench surface 601 is the pattern image (w, h) on the work bench surface 601 Using an angle θp (w, h) between an imaginary line connecting the projector 600 and the optical axis of the projector 600, the following equation (1) can be expressed.
Qp (w, h) = Ip / cos ⁴ θ p (w, h) (1)

In the present embodiment, the proportional constant Ip is a constant specified for each projector 600, and is stored in advance in the storage unit. cos ⁴ θ p (w, h) is calculated from the distance obtained in the processing step S4 described above. Therefore, the peripheral light amount coefficient Qp (w, h) of the projector 600 is calculated from the proportional constant Ip stored in the storage unit and the distance obtained in the processing step S4 (S6).

  On the other hand, the optical characteristics of the cameras 101A and 101B (the decrease in the amount of light in the peripheral portion of the captured image) are represented by the peripheral light amount coefficient Qc (w, h). The peripheral light amount coefficient Qc (w, h) is, for example, the ratio of the relative light reception amount of each pattern image portion imaged by each camera 101A, 101B to the light reception amount of the pattern image portion projected at the center of the imaged image. Is an indicator of

The peripheral light amount coefficient Qc (w, h) indicating the optical characteristics of the cameras 101A and 101B takes smaller values as the pattern image portion is farther from the center of the captured image (optical axis of the camera), as shown in FIG. The peripheral light amount coefficient Qc (w, h) corresponding to each pattern image (w, h) of the pattern image projected on the work surface 601 is the pattern image (w, h) on the work surface 601 and the camera 101A, Using an angle θc (w, h) between a virtual line connecting with 101B and the optical axes of the cameras 101A and 101B, the following equation (2) can be expressed.
Qc (w, h) = Ic / cos ⁴ θ c (w, h) (2)

In the present embodiment, the proportional constant Ic is a constant specified for each of the cameras 101A and 101B, and is stored in advance in the storage unit. cos ⁴ θc (w, h) is calculated from the distance obtained in the above-described processing step S4. Therefore, the peripheral light amount coefficient Qc (w, h) of the cameras 101A and 101B is calculated from the proportionality constant Ic stored in the storage unit and the distance obtained in the processing step S4 (S7).

  Here, in the present embodiment, the pattern of each pattern image portion (w, h) in the pattern image projected by the projector 600 is the optical characteristic of each of the cameras 101A and 101B (the amount of light received in the peripheral portion of the captured image is small). Reflect on the conditions. At this time, in the present embodiment, as shown in FIG. 6, the central portion (the central portion O ′ of the imaged image) of the imaging region imaged by each of the cameras 101A and 101B and the center of the pattern image projected by the projector 600 Part O does not match and is offset. Therefore, on the pattern image, the optical characteristics of the projector 600 as indicated by the triple concentric circles on the right side in FIG. 7 (the decrease in the amount of light at the periphery of the pattern image) and the camera 101A as indicated by the triple concentric circles on the left side in FIG. As shown in FIG. 7, the optical characteristics 101 </ b> B (the decrease in the amount of light in the peripheral portion of the captured image) are superimposed. The offset amount corresponds to the offset amount (x−x0, y−y0) between the installation position (reference position) of the camera unit 100 and the installation position of the projector 600. Therefore, the peripheral light amount coefficient Qc (w, h) indicating the optical characteristic of each camera 101A, 101B is Qc in consideration of the offset amount when reflecting it on the pattern condition of each pattern image portion (w, h). It represents as (w + (x-x0), h + (y-y0)).

From the above, the spatial frequency f1 (w, h) for determining the granularity of each pattern image portion (w, h) in the initial pattern image of the present embodiment can be obtained from the following equation (3) (S8).
f1 (w, h) = f0 (w, h) / Qp (w, h) / Qc (w + (x-x 0), h + (y-y 0)) (3)

  Thus, in consideration of the peripheral light amount coefficient Qp (w, h) which is an optical characteristic of the projector 600, and the peripheral light amount coefficient Qc (w, h) which is an optical characteristic of the cameras 101A and 101B, After calculating the spatial frequency f1 (w, h) that determines the granularity of the pattern image portion (w, h), the pattern image generation unit 502 determines each pattern image portion (w, h) according to the spatial frequency f1 (w, h). ), And generates an initial pattern image having pattern image portions different in granularity (S9).

  In the present embodiment, the particle size (spatial frequency) is adopted as the pattern condition, but another pattern condition which improves the recognition processing accuracy in the recognition processing unit 300 instead of or together with the particle size is adopted. May be For example, when contrast is adopted as the pattern condition, the pattern image generation unit 502 also determines the contrast of each pattern image portion (w, h) according to each spatial frequency f1 (w, h) calculated as described above. An initial pattern image having pattern image portions different in granularity and contrast as shown in 8 may be generated.

  The initial pattern image generated in this manner may be generated by performing a test or the like in advance and stored in the storage unit of the personal computer 500. In this case, in the initial setting before operating the robot picking system, an initial pattern image may be read from the storage unit.

  Here, the above-described initial pattern image takes into consideration the optical characteristics of the projector 600 (the decrease in the amount of light at the periphery of the pattern image) and the optical characteristics of the cameras 101A and 101B (the decrease in the amount of light at the periphery of the captured image). And adjust the granularity of the pattern image portion projected on the imaging region location (such as the vicinity of the pattern image) having a relatively low matching accuracy, and improve the matching accuracy for the imaging region location, and the appropriate parallax value To reduce the occurrence of parts that can not be obtained. Therefore, after the above-described initial setting, this initial pattern image may be used in a fixed manner.

  However, since the optical characteristics of the projector 600 and the cameras 101A and 101B may change with time or may change depending on environmental conditions such as temperature and humidity, such a change causes an appropriate parallax value to be obtained. It may happen that the effect of reducing the occurrence of the portion that can not be obtained is reduced. Further, not only changes in the optical characteristics of the projector 600 and the cameras 101A and 101B, but also other changes (such as changes in ambient illumination conditions) reduce the occurrence of portions where appropriate parallax values can not be obtained. It may happen that the effect is reduced. Therefore, in the present embodiment, after the robot picking system is operated, pattern image change processing is executed at a predetermined execution timing, and with regard to an imaging area location where the matching accuracy has fallen due to any change, Adjust pattern conditions (grain size etc.) to improve matching accuracy.

Hereinafter, pattern image change processing implemented in the present embodiment will be described.
FIG. 9 is a flowchart showing a flow of image pattern change processing in the present embodiment.
When a predetermined execution timing of the robot picking system in operation comes, first, the personal computer 500 controls the projector 600 to project the pattern image stored in the storage unit toward the work bench surface 601 (S11) The pattern image is displayed on the work on the work surface 601. As the pattern image at this time, if there is no change from the initial pattern image in the past image pattern change processing, the initial pattern image is used. Thereafter, each camera 101A, 101B of the camera unit 100 executes an imaging operation (S12) to obtain one frame of captured image data (reference image data and comparison image data). The captured image data is corrected by the image correction units 201A and 201B, and the parallax information calculation unit 202 generates parallax image data (disparity information) from the corrected image data (S13). The parallax image data generated in this manner is sent to the parallax score calculation unit 501 of the personal computer 500 via the external I / Fs 203 and 503.

  On the other hand, in the personal computer 500, a parallax image based on the parallax image data is displayed on the display unit, and the operator operates the input unit to set a designated image portion for counting the number of effective parallax values in the parallax image. (S14). The setting information of the designated image portion set in this manner is delivered to the parallax score calculation unit 501. The designated image portion is, for example, a rectangular image portion set along the inner edge of a box that accommodates a workpiece that is an object. The designated image portion for counting the number of effective parallax values is preferably set so as to include the image portion in which the object is projected, and preferably exclude as much as possible the image portion in which the object is not imaged. This is because, if an image portion other than the target (workpiece) recognized by the recognition processing unit 300 is included, the number of parallax values which is not originally required is counted, and the pattern condition of each pattern image portion is optimized. Because it is an obstacle to

  The shape of the designated image portion is not limited to a rectangle, and may be another shape such as a circle. Further, although the case where the operator of the personal computer 500 manually sets the designated image portion is described here, the image portion in which the object is projected is specified and automatically designated based on the acquired parallax image data. An image portion may be set. Further, the setting of the designated image portion is not essential, and the number of effective parallax values may be counted for the entire parallax image.

  Based on the parallax image data received through the external I / Fs 203 and 503 and the setting information of the designated image portion set in the processing step S14, the parallax score calculation unit 501 of the personal computer 500 determines the designated image on the parallax image The number of pixels of the effective parallax value (parallax score) in the part is measured (S15). The measured parallax score is stored in the storage unit in a state of being associated with the pattern image (the pattern image projected in the processing step S11) used for measuring the parallax score (S16).

  Next, the parallax score calculation unit 501 of the personal computer 500 determines whether the measured parallax score satisfies a predetermined condition. Specifically, in the present embodiment, since the total number of pixels of the specified image portion changes due to the operator's specification, the ratio of the number of parallax points to the total number of pixels of the specified image portion (hereinafter referred to as “occupancy percentage of parallax points”). Is determined whether or not the threshold value or more (S17). If the occupancy rate of the parallax score is equal to or more than the threshold (Yes in S17), the pattern image (pattern image projected in processing step S11) used for measuring the parallax score is used when the robot picking system is operated thereafter It determines as a pattern image (S20).

  On the other hand, when the occupancy rate of the parallax score is less than the threshold (No in S17), each pattern is generated under the pattern condition different from the pattern image (pattern image projected in processing step S11) used for measuring the parallax score. Another pattern image in which the image portion is created is generated (S19), and the processing steps S11 to S17 are executed again.

In the present embodiment, another pattern image generated in the processing step S19 is generated as follows.
First, with reference to the spatial frequency f1 (w, h) that determines the granularity of each pattern image portion of the pattern image projected first in the present pattern image changing process, for example, the spatial frequency f1 (w, h) Nine spatial frequencies f 2 (w, h), f 3 (w, h), f 4 (w, h), f 5 (w, h), f 1 (w, h), f 6 (w, h) , F7 (w, h), f8 (w, h), f9 (w, h) are set. Specifically, the spatial frequency f2 (w, h) to f9 (w, h) is set by changing the value of the proportional constant K within the range of 0.5 ≦ K1 ≦ 2 according to the following equation (4) Do.
f2 (w, h)-f9 (w, h) = f1 (w, h) x K (4)

  By using these spatial frequencies f2 (w, h) to f9 (w, h), the pattern image generation unit 502 of the personal computer 500 makes the granularity of each pattern image portion coarser than that of the first projected pattern image. It becomes possible to generate four types of pattern images and four types of pattern images in which the granularity of each pattern image portion is finer than that of the first projected pattern image.

  In processing step S19, among the spatial frequencies f2 (w, h) to f9 (w, h) set in this manner, according to a predetermined order (for example, an order closer to the spatial frequency of the first projected pattern image) The pattern image generation unit 502 of the personal computer 500 generates another pattern image. Then, processing steps S11 to S17 are executed, and in the processing step S17, if the occupancy rate of the parallax score becomes equal to or more than the threshold (Yes in S17), the pattern image used for measuring the parallax score is the robot picking thereafter It is determined as a pattern image used at the time of operation of the system (S20).

  On the other hand, in any pattern image of another pattern image (eight types in this case) generated in this manner, when the occupancy rate of the parallax score does not become equal to or more than the threshold (No in S17), the number of times of pattern generation is a prescribed value ( Here, it reaches 8 times) (Yes in S18). In this case, parallax scores for a total of nine types of pattern images are stored in the storage unit in a state in which they are associated with the respective pattern images in processing step S16. Therefore, among the nine types of pattern images, the pattern image having the highest parallax score is specified, and the pattern image is determined as a pattern image to be used when the robot picking system is operated subsequently (S20).

[Modification]
Next, a modified example of the initial pattern image generation process in the present embodiment will be described.
In the embodiment described above, the distance from the projector 600 to the work bench surface 601 and the distance from the work bench surface 601 to the camera unit 100 are arranged to be substantially equal, but in this modification, the projector 600 to the work bench surface This is an example in which the distances up to 601 are changed, and as shown in FIG. 10, these distances are different.

  As shown in FIG. 11, as the distance from the projector 600 to the work bench surface 601 is shorter, the size of the pattern image projected onto the work bench surface 601 is smaller (the magnification is smaller), and as the distance is longer, the work bench surface 601 The size of the pattern image projected on the top becomes large (the magnification is large). Therefore, in the pattern image projected on the work surface 601, the closer the distance from the projector 600 to the work surface 601 is, the denser the pattern per unit area becomes, and the longer the distance becomes, the rougher the pattern per unit area becomes . When the distance from the projector 600 to the workbench surface 601 is relatively changed with respect to the distance from the workbench surface 601 to the camera unit 100, the area ratio of the imaging region to the pattern image is changed. At this time, although the ratio (density) of the pattern (black and white) per unit area hardly changes, the grain size of the pattern changes.

  The granularity of the pattern affects the matching accuracy at the time of parallax value calculation. Specifically, in the case where the distance from the work bench surface 601 to the camera unit 100 is constant, the closer the distance from the projector 600 to the work bench surface 601, the higher the grain size overall, and the farther the grain size is, the farther the distance. However, as the matching accuracy is improved and the number of effective parallax values (parallax points) is increased, the recognition processing accuracy in the recognition processing unit 300 is improved. Therefore, when the distance from the projector 600 to the workbench surface 601 changes relative to the distance from the workbench surface 601 to the camera unit 100, it is desirable to generate a new pattern image according to the distance .

  Therefore, in the initial image pattern generation process in the present modification, the spatial frequency f0 (w, h) that is the pattern condition of the uniform reference pattern image of the pattern conditions generated by the pattern image generation unit 502 in the processing step S1 first. The correction is made according to the relative distance from the projector 600 to the workbench surface 601 with respect to the distance from the workbench surface 601 to the camera unit 100.

FIG. 12 is a flowchart showing the flow of initial image pattern generation processing in the present modification.
Also in the present modification, first, a reference pattern image with uniform pattern conditions is generated, and the relative position information of the projector 600 based on the distance from the work bench surface 601 to the camera unit 100 and the installation position of the camera unit 100 is calculated. (S1 to S5). Thereafter, in the present modification, the distance Z from the projector 600 to the work surface 601 based on the maximum projection distance Zmax of the projector 600 and the relative position information of the projector 600 calculated in the processing step S5, and a predetermined pattern generation coefficient The correction coefficient H is calculated from the following equation (5) from H0 which is a (constant) (S21).
H = (Zmax-Z) / H0 (5)

Then, using the correction coefficient H thus determined, the spatial frequency f0 (w, h) of the reference pattern image projected in the processing step S1 is corrected according to the following equation (6), and the corrected frequency H is corrected. The spatial frequency f'0 (w, h) is calculated (S22). Thus, the spatial frequency f'0 (w, h) of the reference pattern image corrected in consideration of the relative distance fluctuation from the projector 600 to the workbench surface 601 with respect to the distance from the workbench surface 601 to the camera unit 100 You can get it.
f'0 (w, h) = f0 (w, h) x H (6)

After that, as in the above-described embodiment, the peripheral light amount coefficient Qp (w, h) of the projector 600 and the peripheral light amount coefficient Qc (w, h) of the cameras 101A and 101B are calculated (S6, S7). A spatial frequency f1 (w, h) for determining the particle size of each pattern image portion (w, h) in the initial pattern image is obtained from the following equation (3 ') (S8').
f1 (w, h) = f'0 (w, h) / Qp (w, h) / Qc (w + (x-x0), h + (y-y0)) (3 ')

The above-described one is an example, and each mode has the unique effect.
(Aspect A)
Parallax information generation means such as the parallax information calculation unit 202 is based on captured images captured by a plurality of imaging means such as cameras 101A and 101B from different viewpoints of the object such as a work in the imaging area In a pattern image projection apparatus having an image projection means such as a projector 600 for projecting a pattern image toward an imaging region when generating the pattern image, the pattern image is an optical image of at least one of the image projection means and the imaging means. It is characterized in that pattern image portions (w, h) projected onto the respective imaging region portions are different according to the imaging region portions having different characteristics.
The optimal pattern image that can improve the matching accuracy when generating parallax information differs depending on the optical characteristics of the image projection means and the imaging means. Therefore, when the optical characteristics of the image projection means and the imaging means differ depending on the imaging area location in the imaging area, the entire pattern image projected onto the imaging area is not considered without considering the difference in the imaging area location. Even if it comprises the pattern image which consists of a regular or non-periodic pattern, the point with comparatively low matching accuracy will exist locally.
In this aspect, it is possible to project a pattern image composed of different pattern image portions in accordance with each imaging area location where optical characteristics of at least one of the image projection means and the imaging means are different, toward the imaging area. . Therefore, it is possible to project a pattern image portion suitable for the optical characteristics of the image projection unit and the imaging unit which are different for each imaging region location on each imaging region location in the imaging region. For example, for a part of the imaging area part, a pattern image part suitable for the part of the imaging area part is projected, and for the other imaging area parts, a pattern image part suitable for the part of the imaging area part Also, it is possible to project a pattern image portion suitable for the other imaging area portion. Therefore, it is possible to reduce or eliminate the imaging area location with relatively low matching accuracy in the imaging area, and it is possible to obtain appropriate parallax information for any object present in any location in the imaging area.

(Aspect B)
The aspect A is characterized by including pattern image generation means such as a pattern image generation unit 502 which generates the pattern image.
According to this, it becomes possible to set each pattern image part according to the use environment, use condition, etc. of the pattern image projection apparatus, and a pattern image suitable for use environment, use condition, etc. can be generated.

(Aspect C)
The aspect B is characterized in that the pattern image generation unit generates the pattern image based on the number of parallax values (parallax score) included in the parallax information generated by the parallax information generation unit.
According to this, it becomes possible to set each pattern image portion from the result of directly evaluating the parallax information, and a pattern image can be generated which can effectively reduce the occurrence of a portion where appropriate parallax information can not be obtained in the object. can do.

(Aspect D)
In the aspect B or C, the pattern image generation unit changes the pattern image portion according to the relative distance Z between the image projection unit and the object with respect to the distance between the imaging unit and the object. Generating an image.
As described in the above-described modification, even when the relative distance Z between the image projection unit and the object with respect to the distance between the imaging unit and the object changes, an appropriate pattern image can be generated.

(Aspect E)
In any one of the above aspects A to D, the pattern image portion projected onto each of the imaging region locations is characterized in that at least one of the particle size and the contrast of the pattern is different.
According to this, it is possible to project, toward the imaging region, a pattern image in which the granularity and contrast having a large influence on the matching accuracy are different in each pattern image portion.

(Aspect F)
In any one of the above aspects A to E, the optical characteristic of the image projection means includes the brightness of a pattern image projected by the image projection means.
According to this, the luminance of the pattern image projected by the image projection means is different depending on the imaging area portion, so that an imaging area portion having a relatively low matching accuracy is present in the imaging region, so that the object is appropriate It is possible to reduce the occurrence of a portion where no proper parallax information can be obtained.

(Aspect G)
In any one of the above-mentioned aspects A to F, the optical characteristic of the imaging means is characterized in that the imaging means includes an amount of light received from the imaging region.
According to this, when the amount of light received by the imaging unit from the imaging region varies according to the imaging region location, the imaging object region is suitable in the case where the imaging region location with relatively low matching accuracy exists in the imaging region It is possible to reduce the occurrence of a portion where no proper parallax information can be obtained.

(Aspect H)
Parallax information generation means such as the parallax information calculation unit 202 that generates parallax information of the target object based on captured images captured by a plurality of imaging units from different viewpoints of the target object in the imaging region, and a pattern toward the imaging region In a parallax information generation device having pattern image projection means such as a personal computer 500 or a projector 600 for projecting an image, the pattern image projection device according to any one of the aspects A to G is used as the pattern image projection means. It is characterized by
According to this, it is possible to reduce the occurrence of a portion where appropriate disparity information can not be obtained in the object.

(Aspect I)
A pattern projected by the image projection unit toward the imaging region when the parallax information of the object is generated by the parallax information generation unit based on captured images obtained by imaging the object in the imaging region from different viewpoints by a plurality of imaging units It is a pattern image generation program that causes a computer to function as pattern image generation means for generating an image, wherein the pattern image corresponds to an imaging area location where optical characteristics of at least one of the image projection means and the imaging means are different. It is characterized in that the pattern image portions projected onto the respective imaging area portions are different.
In this aspect, it is possible to project a pattern image composed of different pattern image portions in accordance with each imaging area location where optical characteristics of at least one of the image projection means and the imaging means are different, toward the imaging area. . Therefore, it is possible to project a pattern image portion suitable for the optical characteristics of the image projection unit and the imaging unit which are different for each imaging region location on each imaging region location in the imaging region. For example, for a part of the imaging area part, a pattern image part suitable for the part of the imaging area part is projected, and for the other imaging area parts, a pattern image part suitable for the part of the imaging area part Also, it is possible to project a pattern image portion suitable for the other imaging area portion. Therefore, it is possible to reduce or eliminate the imaging area location with relatively low matching accuracy in the imaging area, and it is possible to obtain appropriate parallax information for any object present in any location in the imaging area.

  The program can be distributed or obtained in the state of being recorded on a recording medium such as a CD-ROM. This program can also be distributed and obtained by distributing or receiving a signal transmitted by a predetermined transmission device via a transmission medium such as a public telephone line, a dedicated line, or another communication network. Is possible. At the time of this distribution, at least a part of the computer program may be transmitted in the transmission medium. That is, all the data making up the computer program need not be present on the transmission medium at one time. The signal carrying the program is a computer data signal embodied on a predetermined carrier including a computer program. In addition, the transmission method of transmitting a computer program from a predetermined transmission apparatus includes the case of continuously transmitting data constituting the program and the case of intermittently transmitting data.

100 camera unit 101A, 101B camera 200 image processing unit 201A, 201B image correction unit 202 parallax information calculation unit 300 recognition processing unit 400 robot arm control unit 500 personal computer 501 parallax score calculation unit 502 pattern image generation unit 600 projector 601 work surface

JP, 2001-91232, A

Claims (9)

An image for projecting a pattern image toward the imaging area when the parallax information of the object is generated by the parallax information generation unit based on captured images obtained by imaging an object in the imaging area from different viewpoints by a plurality of imaging units In a pattern image projection apparatus having projection means,
The pattern image generation unit generates the pattern image based on the number of parallax values included in the parallax information generated by the parallax information generation unit,
The pattern image is characterized in that the pattern image portion projected on each imaging region portion differs according to the imaging region portion where optical characteristics of at least one of the image projection means and the imaging means are different. Projection device. In the pattern image projection apparatus according to claim 1 ,
The pattern image generation means generates a pattern image in which each pattern image portion is changed according to a relative distance between the image projection means and the object with respect to the distance between the imaging means and the object. Pattern image projection device.   An image for projecting a pattern image toward the imaging area when the parallax information of the object is generated by the parallax information generation unit based on captured images obtained by imaging an object in the imaging area from different viewpoints by a plurality of imaging units In a pattern image projection apparatus having projection means,
  It has pattern image generation means for generating a pattern image in which each pattern image portion is changed according to the relative distance between the image projection means and the object relative to the distance between the imaging means and the object.
  The pattern image is characterized in that the pattern image portion projected on each imaging region portion differs according to the imaging region portion where optical characteristics of at least one of the image projection means and the imaging means are different. Projection device. In the pattern image projection apparatus according to any one of claims 1乃Itaru 3,
The pattern image projection device according to claim 1, wherein at least one of the particle size and the contrast of the pattern is different in the pattern image portion projected on each of the imaging region locations. In the pattern image projection apparatus according to any one of claims 1乃Itaru 4,
The optical characteristic of the image projection means includes the brightness of the pattern image projected by the image projection means. In the pattern image projection apparatus according to any one of claims 1乃Itaru 5,
The optical characteristic of the image pickup means includes an amount of light received by the image pickup means from an image pickup area. Parallax information generating means for generating parallax information of the object based on captured images obtained by imaging the object in the imaging area from different viewpoints by a plurality of imaging devices;
In a parallax information generating device including: pattern image projection means for projecting a pattern image toward an imaging region;
The pattern as the image projection unit, the disparity information generating apparatus characterized by using a pattern image projection apparatus according to any one of claims 1乃optimum 6. A pattern projected by the image projection unit toward the imaging region when the parallax information of the object is generated by the parallax information generation unit based on captured images obtained by imaging the object in the imaging region from different viewpoints by a plurality of imaging units It is a pattern image generation program that causes a computer to function as pattern image generation means for generating an image.
The pattern image generation unit generates the pattern image based on the number of parallax values included in the parallax information generated by the parallax information generation unit,
The pattern image is characterized in that the pattern image portion projected on each imaging region portion differs according to the imaging region portion where optical characteristics of at least one of the image projection means and the imaging means are different. Generator.   A pattern projected by the image projection unit toward the imaging region when the parallax information of the object is generated by the parallax information generation unit based on captured images obtained by imaging the object in the imaging region from different viewpoints by a plurality of imaging units It is a pattern image generation program that causes a computer to function as pattern image generation means for generating an image.
  The pattern image generation unit generates a pattern image in which each pattern image portion is changed according to a relative distance between the image projection unit and the object with respect to the distance between the imaging unit and the object.
  The pattern image is characterized in that the pattern image portion projected on each imaging region portion differs according to the imaging region portion where optical characteristics of at least one of the image projection means and the imaging means are different. Generator.