JP2017020971A - Pattern image projection device, parallax information creating device, and pattern image creating program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce generation of a part where suitable parallax information is not obtained in an object, even when an image region portion with relatively low matching accuracy is present in an image region because optical characteristics of image projection means and of imaging means differ depending on an image region portion.SOLUTION: A pattern image projection device includes image projecting means 600 for projecting a pattern image onto an imaging region upon creating parallax information of an object by parallax information creating means 202 on the basis of an image obtained by imaging the object in the imaging region by use of a plurality of imaging means 101A, 101B from different viewing points. In accordance with imaging region portions where optical characteristics of at least one of the image projecting means and the imaging means differ, the pattern image includes different pattern image portions to be projected onto the respective imaging region portions.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a pattern image projection device, a parallax information generation device, and a pattern image generation program.

  Conventionally, there has been known a pattern image projection device that projects a pattern image toward an imaging region when the object in the imaging region is imaged by a plurality of imaging units from different viewpoints to generate parallax information of the object. Yes.

  Japanese Patent Laid-Open No. 2004-26883 discloses matching (matching processing) between a reference image obtained by imaging a subject (object) irradiated with a light projection pattern (pattern image) from a pattern image projection device from different viewpoints and a reference image. And a three-dimensional shape measurement apparatus that generates a distance image (disparity information) based on a parallax value between matched corresponding pixels is disclosed. In this three-dimensional shape measuring apparatus, random numbers are used for the purpose of reducing the occurrence of mismatching at the time of matching corresponding to the case where a subject is irradiated with a light projection pattern having a periodic pattern, the dot size, A non-periodic light projection pattern is generated that is a pattern with irregular line length, thickness, position, density, and the like.

  In general, a pattern image projection device projects a pattern image toward an imaging region via an optical system such as a lens, but the optical characteristics of the optical system differ depending on the point in the imaging region. For example, optical characteristics such as brightness (luminance), distortion, and contrast of the pattern image are different between a point near the optical axis center of the optical system and a point far from the optical axis center. Such optical characteristics affect the matching accuracy of the matching process for associating pixels that show the same point in the imaging region between a plurality of captured images. For this reason, if the optical characteristics differ depending on the points in the imaging region, points with relatively low matching accuracy locally occur in the imaging region. Therefore, there is a possibility that appropriate parallax information cannot be obtained for the object at such a point.

  Similarly, in the optical system of the image pickup means when the object in the image pickup area on which the pattern image is projected is picked up from different points by a plurality of image pickup means, the optical characteristics thereof differ depending on the points in the image pickup area. Therefore, even if a uniform pattern image is projected, optical characteristics such as brightness (brightness), distortion, and contrast of the captured image differ depending on points in the imaging region. Since such optical characteristics also affect the matching accuracy of the matching process, even when the optical characteristics of the imaging means differ depending on the points in the imaging region, points with relatively low matching accuracy are locally generated. End up. Therefore, there is a possibility that appropriate parallax information cannot be obtained for the object at such a point.

  Note that the point with relatively low matching accuracy is not limited to the point away from the optical axis center of the optical system, but varies depending on the configuration of the optical system. Therefore, a point where the matching accuracy is relatively low, that is, a portion where appropriate parallax information cannot be obtained may differ depending on the configuration of the optical system.

  Moreover, even if it uses the pattern image projection apparatus disclosed by patent document 1 which projects an aperiodic light projection pattern (pattern image), the subject mentioned above cannot be solved. In this projection pattern, the pattern image portion projected to each imaging region location having different optical characteristics for the image projecting means and the imaging device is not a pattern image portion optimized for each imaging region location. . Therefore, when the optical characteristics of the image projection unit and the imaging unit differ depending on the location of the imaging region, a point with relatively low matching accuracy is generated locally, and an object existing at such a point In some cases, appropriate disparity information may not be obtained.

  In order to solve the above-described problem, the present invention generates parallax information of an object by the parallax information generating unit based on captured images obtained by capturing the object in the imaging region from different viewpoints by a plurality of imaging units. In addition, in the pattern image projection apparatus having the image projection unit that projects the pattern image toward the imaging region, the pattern image is in an imaging region where the optical characteristics of at least one of the image projection unit and the imaging unit are different. Accordingly, the pattern image portion projected on each imaging region location is different.

  According to the present invention, even if there is an imaging region location with relatively low matching accuracy in the imaging region, for example, because the optical characteristics of the image projecting means or the imaging means differ depending on the imaging region location, An excellent effect is obtained in that appropriate parallax information can be obtained for an object present in an imaging region location.

It is a block diagram showing a robot picking system in an embodiment. It is explanatory drawing which shows arrangement | positioning of the principal part of the robot picking system. It is explanatory drawing which shows an example of the light quantity change with respect to the angle of view of the camera and projector which comprise the robot picking system. It is explanatory drawing which shows the example of a division | segmentation of the pattern image part produced on different pattern conditions in the pattern image in embodiment. It is a flowchart which shows the flow of the initial image pattern production | generation process in embodiment. It is explanatory drawing which shows arrangement | positioning of the camera and projector which comprise the robot picking system. It is explanatory drawing which shows the overlap state of the optical characteristic of a camera, and the optical characteristic of a projector on the pattern image to project. It is explanatory drawing which shows an example of the pattern image with the pattern image part from which a particle size and contrast differ. It is a flowchart which shows the flow of the image pattern change process in embodiment. It is explanatory drawing which shows arrangement | positioning of the camera and projector which comprise the robot picking system in a modification. It is explanatory drawing for demonstrating that the particle size of the pattern image projected on a work table surface changes with the distance from a projector to a work table surface. It is a flowchart which shows the flow of the initial image pattern production | generation process in a modification.

Hereinafter, an embodiment in which a parallax information generation device including a pattern image projection device according to the present invention is applied to a robot picking system will be described.
In the robot picking system according to the present embodiment, for example, one workpiece (for example, a T-shaped pipe) is taken out from a box in which a large number of workpieces (objects) are stacked, and the workpiece is transferred to a subsequent processing apparatus. It is a thing to pass. In this system, the inside of the box is imaged from above, and individual workpieces in the box are distinguished and recognized based on the captured image. Based on the recognition result, the operation of the robot arm is controlled so that one workpiece is picked from above by the hand portion of the robot arm and taken out from the box.
Note that the disparity information generation device including the pattern image projection device according to the present invention is not limited to the robot picking system, and includes, for example, an image analysis device that performs recognition processing of an object existing in the imaging region based on the captured image. It can be applied to other systems.

FIG. 1 is a block diagram showing a robot picking system in the present embodiment.
FIG. 2 is an explanatory diagram showing an arrangement of main parts of the robot picking system of the present embodiment.
The robot picking system of the present embodiment mainly includes an imaging device including a stereo camera including a camera unit 100 and an image processing unit 200, a robot arm control device including a recognition processing unit 300 and a robot arm control unit 400, The image forming apparatus includes a projector 600 as an image projecting unit and a pattern image projecting apparatus including a personal computer (hereinafter referred to as “personal computer”) 500 that controls the projector 600. The distance from the projector 600 to the work table surface 601 on which the workpiece is placed and the distance from the work table surface 601 to the camera unit 100 are arranged to be substantially equal.

  The camera unit 100 is composed of two cameras 101A and 101B that are image pickup means, and the two cameras 101A and 101B have the same configuration. The cameras 101A and 101B mainly include an optical system such as an imaging lens, an image sensor configured by a pixel array in which light receiving elements are two-dimensionally arranged, and analog electric signals (each on the image sensor) output from the image sensor. A signal processing unit that generates and outputs captured image data obtained by converting a received light amount received by the light receiving element into a digital electric signal.

  The image processing unit 200 receives captured image data output from the camera unit 100 and performs various image processing. The captured image data output from each of the cameras 101A and 101B of the camera unit 100 is input to the image correction units 201A and 201B that perform correction processing to convert the captured image data into an image obtained by an ideal pinhole model. The image correction units 201A and 201B are provided for each of the cameras 101A and 101B, and are configured by an FPGA (Field-Programmable Gate Array), a memory, or the like. The image correction units 201A and 201B perform correction processing such as magnification correction, image center correction, and distortion correction on the input image data from the cameras 101A and 101B by the FPGA. The memory stores correction parameters used for the correction process. Note that, for example, an ASIC (Application Specific Integrated Circuit) may be employed instead of the FPGA.

  The captured image data after correction processing (corrected image data) output from the image correction units 201A and 201B is sent to the parallax information calculation unit 202 as parallax information generation means. The parallax information calculation unit 202 performs processing for generating parallax image data obtained from corrected image data obtained by imaging with the cameras 101A and 101B. Specifically, in order to obtain parallax image data from the corrected image data output from the image correction units 201A and 201B, the parallax value of the corresponding image portion between the two corrected images is calculated.

  The parallax value here refers to a comparison image with respect to an image portion on the reference image corresponding to the same point in the imaging region, with one of the captured images captured by the cameras 101A and 101B being a reference image and the other being a comparison image. The positional deviation amount of the image portion is calculated as the parallax value of the image portion. By using the principle of triangulation, the distance to the same point in the imaging area corresponding to the image portion can be calculated from the parallax value. The parallax image data is a parallax image in which pixel values corresponding to parallax values calculated for each image portion on reference image data (for example, captured image data a captured by the camera 101A) are represented as pixel values of the respective image portions. It is shown. The parallax image data output from the parallax information calculation unit 202 is sent to the recognition processing unit 300 at the subsequent stage and also sent from the external I / F 203 to the personal computer 500.

  The recognition processing unit 300 uses the parallax image data output from the parallax information calculation unit 202 to perform processing for individually recognizing a large number of works housed in a box existing in the imaging region. Specifically, for example, an edge portion where the pixel value (parallax value) greatly changes is detected on the parallax image, and an image portion on which each work is projected is identified from the detection result. Then, for example, the image portion having the largest parallax value average among the identified image portions is identified, and the workpiece calculated from the horizontal position of the workpiece corresponding to the identified image portion and the parallax value average of the image portion. To the robot arm control unit 400 that controls the operation of the robot arm as a workpiece recognition result. The work specified in this way is the work stacked on top in the box.

  The robot arm control unit 400 can grasp the position (horizontal position and distance) of one workpiece to be picked based on the workpiece recognition result output from the recognition processing unit 300. The robot arm control unit 400 controls the robot arm operation so that the workpiece existing at the grasped position is picked from the top by the hand unit of the robot arm, taken out from the box, and transferred to the subsequent processing apparatus.

  The personal computer 500 controls the projector 600 in order to project a pattern image from the projector 600 onto a work that is an object. The personal computer 500 according to the present embodiment mainly includes a calculation unit including a CPU (Central Processing Unit) and a storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), and an HDD (Hard Disk Drive). A general-purpose personal computer has general functions such as a storage unit, a communication unit including a communication interface, and an input unit such as a keyboard and a pointing device.

  In the present embodiment, the personal computer 500 realizes various functions necessary for the operation as the pattern image projection apparatus together with the projector 600 by the CPU executing various computer programs stored in the storage unit. In particular, in the first embodiment, a pattern image generation process for generating a pattern image projected by the projector 600 toward the work is executed by executing the pattern image generation program on the CPU. The personal computer 500 includes a parallax score calculation unit 501, a pattern image generation unit 502, and an external I / F 503 as a configuration for executing this pattern image generation processing, and recognizes a workpiece that is an object with high recognition accuracy. A pattern image that can be generated is generated.

  This pattern image generation process includes a parallax point calculation unit 501 and a pattern image generation unit 502 in the personal computer 500. The parallax point calculation unit 501 acquires the parallax image data output from the parallax information calculation unit 202 in the image processing unit 200 via the external I / Fs 203 and 503, and designates a designated image portion designated in advance from the parallax image data. The number of effective parallax values is counted. The pattern image generation unit 502 generates a pattern image in which the number of effective parallax values counted by the parallax point calculation unit 501 satisfies a specified condition.

In the present embodiment, the reason for generating a pattern image for increasing the recognition accuracy of a workpiece that is an object is as follows.
As described above, the recognition processing in the recognition processing unit 300 recognizes an image portion on which a workpiece is projected based on parallax information between captured images captured by the two cameras 101A and 101B. The recognition accuracy in such processing depends on the number of parallax values (the number of pixels on the parallax image constituting the three-dimensional object region) accurately calculated for the image portion showing the workpiece. The calculation of the parallax value is premised on accurately specifying (matching) a corresponding image portion between both captured images (an image portion on each image corresponding to the same point in the imaging region).

  For example, the matching process is performed by using each image region (horizontal position (image horizontal direction) in the comparison image having the same vertical position (image vertical position) as the target image region in the reference image including the corresponding image portion to be matched. Correlation calculation is performed on the relationship between the image areas having different positions) and the arrangement of a plurality of pixel values (luminance values) constituting the target image area in the reference image. Then, the image area having the highest correlation among the image areas in the comparison image in which a correlation of a certain level or more is recognized is identified as the corresponding image area corresponding to the target image area in the reference image. Thereby, the corresponding image part between both captured images is specified.

  Here, the projector 600 projects a pattern image via an optical system, but the optical characteristics of the optical system are not uniform over the entire projected pattern image. The projector 600 according to the present embodiment is the optical axis center of the optical system (hereinafter also referred to as “the center of the pattern image”, but the center of the pattern image here does not indicate the positional central point of the pattern image). Has an optical characteristic that the luminance is highest and the luminance decreases as the distance from the optical axis center (center of the pattern image) increases. That is, the brightness (luminance) of the pattern image between the vicinity of the optical axis center of the optical system of the projector 600 (the central portion O of the pattern image) and a point far from the optical axis center (the peripheral portion of the pattern image). Is different.

  As a result, for example, when the pattern image is projected so as to obtain optimal brightness (luminance) in the central portion O of the pattern image, that is, brightness (luminance) with which an appropriate parallax value can be calculated for the central portion O, FIG. As shown in FIG. 3, the brightness is lower than the optimum brightness (luminance) at the periphery of the pattern image. When the brightness is low, the contrast of the captured image is lowered, so that the accuracy of the correlation calculation is lowered and it is difficult to perform an accurate matching process. As a result, the workpiece on which the peripheral part of the pattern image is projected cannot be accurately matched, and the possibility that the parallax value cannot be calculated or an incorrect parallax value is calculated increases. As a result, the number of correctly calculated parallax values (the number of pixels of the image portion on the parallax image corresponding to the image portion projecting the work) is reduced for the image portion projecting the work, and the recognition processing unit 300. The recognition processing accuracy in the case will be deteriorated.

  On the other hand, for example, when a pattern image is projected so that the optimum brightness (luminance) is obtained in the peripheral portion of the pattern image, the brightness is higher than the optimum brightness (luminance) at the center portion O of the pattern image. It becomes. If the brightness is too high, among the light receiving elements that receive the light from the center O, there is a light receiving element in which the amount of received light exceeds the saturation value that is the upper limit value of the light amount detection range of the light receiving element, It can be difficult to perform an exact matching process. In this case, with respect to the work on which the center portion of the pattern image is projected, there is a high possibility that accurate matching processing cannot be performed, and the parallax value cannot be calculated or an incorrect parallax value is calculated. As a result, the number of parallax values accurately calculated for the image portion that shows the workpiece is reduced, and the recognition processing accuracy in the recognition processing unit 300 is deteriorated.

  Also, the optical characteristics of the two cameras 101A and 101B of the camera unit 100 that captures the workpiece on which the pattern image is projected from different points are not uniform over the entire imaging region. As shown in FIG. 3, the cameras 101 </ b> A and 101 </ b> B of the present embodiment receive the largest amount of received light at the optical axis center (image sensor center of each camera) of the optical system, and the farther from the optical axis center (image sensor center). The optical characteristic is that the amount of received light is reduced. That is, the brightness (brightness) of the captured image between the vicinity of the optical axis center of the optical system of the cameras 101A and 101B (center portion of the image sensor) and a point away from the optical axis center (peripheral portion of the image sensor). ) Is different.

  As a result, for example, even when a workpiece on which a pattern image with uniform brightness (luminance) is projected is captured, the brightness (luminance) is such that an appropriate parallax value can be calculated for the center of the captured image. Sometimes, the brightness of the peripheral portion of the captured image is lower than the optimum brightness (luminance). If the brightness of the picked-up image is low, the contrast of the picked-up image is lowered, so that the accuracy of the correlation calculation is lowered and it is difficult to perform accurate matching processing. As a result, the workpiece displayed in the peripheral portion of the captured image cannot be accurately matched, and the possibility that the parallax value cannot be calculated or an incorrect parallax value is calculated increases. As a result, the number of parallax values accurately calculated for the image portion that shows the workpiece is reduced, and the recognition processing accuracy in the recognition processing unit 300 is deteriorated.

  On the other hand, for example, when the brightness (brightness) at which an appropriate parallax value can be calculated for the peripheral part of the captured image is higher than the optimal brightness (brightness) at the center of the captured image It becomes. If the brightness of the captured image is too high, there is a light receiving element in the light receiving element corresponding to the center of the captured image that has a light receiving amount that exceeds the saturation value that is the upper limit of the light amount detection range of the light receiving element. Thus, accurate matching processing can be difficult. In this case, the workpiece displayed in the center of the captured image cannot be accurately matched, and the possibility that the parallax value cannot be calculated or an incorrect parallax value is calculated increases. As a result, the number of parallax values accurately calculated for the image portion that shows the workpiece is reduced, and the recognition processing accuracy in the recognition processing unit 300 is deteriorated.

  In particular, there may be a case where an imaging region portion projected on the peripheral portion of the captured image captured by the two cameras 101A and 101B of the camera unit 100 matches an imaging region portion on which the peripheral portion of the pattern image is projected by the projector 600. is there. In this case, for example, when the pattern image is projected under the optimal condition at the center O of the pattern image and the image is captured under the optimal condition at the center of the captured image, the optical characteristics of the projector 600 (the pattern image The amount of light at the peripheral portion is small and the optical characteristics of the cameras 101A and 101B (the amount of light received at the peripheral portion of the captured image is small) are superimposed, and the captured image portion corresponding to the imaging region location (of the captured image) It is particularly difficult to perform accurate matching processing by greatly reducing the accuracy of correlation calculation in the matching processing of the peripheral portion), and the recognition processing accuracy in the recognition processing unit 300 is particularly low.

  Therefore, in the present embodiment, in order to suppress a reduction in recognition processing accuracy in the recognition processing unit 300 locally in the imaging region, imaging region locations where the optical characteristics of the projector 600 and the cameras 101A and 101B are different from each other. The parallax image data is generated using the pattern images created under different pattern conditions for each pattern image portion projected on the screen. Specifically, as shown in FIG. 4, the pattern image is divided into w pattern patterns of w × h in the horizontal direction and a total of w × h pattern image sections in the vertical direction. Create a pattern image.

  In the present embodiment, an initial pattern image generation process is executed in an initial setting before operating the robot picking system. In the present embodiment, each imaging is performed in consideration of the optical characteristics of the projector 600 (a reduction in the amount of light in the peripheral part of the pattern image) and the optical characteristics of the cameras 101A and 101B (a reduction in the amount of light in the peripheral part of the captured image). A pattern image in which the pattern conditions of the pattern image portion projected on the region portion are adjusted is generated as an initial pattern image. Examples of the pattern conditions include pattern granularity and contrast. However, the pattern conditions are not particularly limited as long as the recognition process accuracy in the recognition processing unit 300 is improved. The pattern image generation unit 502 of this embodiment can generate patterns with different granularities using the spatial frequency as a parameter. Therefore, in the present embodiment, when generating the initial pattern image, the spatial frequency of the pattern image portion projected on each imaging area portion is adjusted, and the pattern image portion having a granularity suitable for each optical characteristic is adjusted. An initial pattern image having (w, h) is generated. The higher the spatial frequency, the finer the pattern granularity.

FIG. 5 is a flowchart showing the flow of initial image pattern generation processing in the present embodiment.
In the initial pattern image generation process, first, the pattern image generation unit 502 of the personal computer 500 generates a reference pattern image with uniform pattern conditions, and the personal computer 500 controls the projector 600 to direct the reference pattern image toward the work table surface 601. Project (S1). The pattern image at this time is one in which all the pattern image portions (w, h) are created with the same spatial frequency f0 (w, h) (the same granularity). Note that the pattern image projected from the projector 600 does not have distortion due to the optical system of the projector 600 or the like.

  When a reference pattern image with uniform pattern conditions (spatial frequency) is projected, each camera 101A, 101B of the camera unit 100 executes an imaging operation (S2). One frame of captured image data (reference image data and comparative image data) ) The captured image data is corrected by the image correction units 201A and 201B, and parallax image data (parallax information) is generated from the corrected image data by the parallax information calculation unit 202 (S3). The parallax image data generated in this way is sent to the pattern image generation unit 502 via the external I / Fs 203 and 503, and the pattern image generation unit 502 performs general triangulation from the received parallax image data. Using the principle, the distance from the work table surface 601 to the camera unit 100 is calculated (S4).

  Subsequently, the pattern image generation unit 502 of the personal computer 500 uses the received parallax image data or the reference image data corresponding to the parallax image data to determine the relative position information (XYZ) of the projector 600 based on the installation position of the camera unit 100. The relative position coordinates and the rotation angle around the Z axis are calculated (S5). Specifically, as shown in FIG. 2, the X axis and the Y axis are taken in parallel to the work table surface, the Z axis is taken in the normal direction of the work table surface, and the installation position (reference position) of the camera unit 100 is ( X coordinate, Y coordinate, Z coordinate, Z axis rotation angle) = (x0, y0, z0, rz0), and the installation position of projector 600 is (X coordinate, Y coordinate, Z coordinate, Z axis rotation angle) = (x , Y, z, rz), the relative position information of the projector 600 is (x−x0, y−y0, z−z0, rz−rz0).

  Here, the optical characteristic of the projector 600 (a decrease in the amount of light at the periphery of the pattern image) is represented by a peripheral light amount coefficient Qp (w, h). As shown in FIG. 4, the peripheral light quantity coefficient Qp (w, h) is obtained when the pattern image is divided into a total of w × h pattern image portions of w pieces in the horizontal direction and h pieces in the vertical direction. For example, the ratio of the relative light amount of each pattern image portion to the light amount of the pattern image portion located at the center portion O of the pattern image is shown. In this case, the peripheral light amount coefficient Qp of the pattern image portion located at the center portion O of the pattern image is 1, and the peripheral light amount coefficient Qp of the other pattern image portions is usually a value less than 1, and The peripheral light quantity coefficient Qp takes a smaller value as the pattern image part is farther from the center.

The peripheral light quantity coefficient Qp (w, h) indicating the optical characteristics of the projector 600 is as shown in FIG. 6 because the peripheral light quantity coefficient Qp takes a smaller value as the pattern image part is farther from the center of the pattern image (the optical axis of the projector). As described above, the peripheral light quantity coefficient Qp (w, h) corresponding to each pattern image (w, h) of the pattern image projected on the work table surface 601 is the same as each pattern image (w, h) on the work table surface 601. Using an angle θp (w, h) formed by an imaginary line connecting the projector 600 and the optical axis of the projector 600, it can be expressed as the following equation (1).
Qp (w, h) = Ip / cos ⁴ θp (w, h) (1)

In the present embodiment, the proportionality constant Ip is a constant specified for each projector 600 and is stored in advance in the storage unit. cos ⁴ θp (w, h) is calculated from the distance obtained in the processing step S4 described above. Therefore, the peripheral light amount coefficient Qp (w, h) of the projector 600 is calculated from the proportionality constant Ip stored in the storage unit and the distance obtained in the processing step S4 (S6).

  On the other hand, the optical characteristics of each of the cameras 101A and 101B (decrease in the amount of light at the periphery of the captured image) are represented by a peripheral light amount coefficient Qc (w, h). The peripheral light amount coefficient Qc (w, h) is, for example, the ratio of the relative received light amount of each pattern image portion captured by each camera 101A, 101B to the received light amount of the pattern image portion projected at the center of the captured image. Is shown.

Since the peripheral light quantity coefficient Qc (w, h) indicating the optical characteristics of the cameras 101A and 101B takes a smaller value as the pattern image part is farther from the center of the captured image (the optical axis of the camera), as shown in FIG. The peripheral light quantity coefficient Qc (w, h) corresponding to each pattern image (w, h) of the pattern image projected on the work table surface 601 is the same as each pattern image (w, h) on the work table surface 601 and the camera 101A, Using an angle θc (w, h) formed by an imaginary line connecting 101B and the optical axes of the cameras 101A and 101B, it can be expressed as the following equation (2).
Qc (w, h) = Ic / cos ⁴ θc (w, h) (2)

In the present embodiment, the proportionality constant Ic is a constant specified for each of the cameras 101A and 101B, and is stored in advance in the storage unit. cos ⁴ θc (w, h) is calculated from the distance obtained in the processing step S4 described above. Therefore, the peripheral light amount coefficient Qc (w, h) of the cameras 101A and 101B is calculated from the proportionality constant Ic stored in the storage unit and the distance obtained in the processing step S4 (S7).

  Here, in the present embodiment, the optical characteristics of the cameras 101A and 101B (the amount of received light in the peripheral portion of the captured image is small) are the patterns of the pattern image portions (w, h) in the pattern image projected by the projector 600. Reflect in the conditions. At this time, in the present embodiment, as shown in FIG. 6, the center of the imaging region captured by each of the cameras 101A and 101B (the center O ′ of the captured image) and the center of the pattern image projected by the projector 600 The part O does not match and is offset. Therefore, on the pattern image, the optical characteristics of the projector 600 as shown by the triple concentric circle on the right side in FIG. 7 (decrease in the amount of light at the periphery of the pattern image) and the camera 101A, as shown by the triple concentric circle on the left side in FIG. The optical characteristics of 101B (a decrease in the amount of light at the periphery of the captured image) are superimposed as shown in FIG. This offset amount corresponds to the offset amount (x−x0, y−y0) between the installation position (reference position) of the camera unit 100 and the installation position of the projector 600. Therefore, the peripheral light quantity coefficient Qc (w, h) indicating the optical characteristics of the cameras 101A and 101B is taken into account when the offset amount is taken into account when reflecting the offset amount in the pattern condition of each pattern image portion (w, h). (W + (x−x0), h + (y−y0)).

From the above, the spatial frequency f1 (w, h) for determining the granularity of each pattern image portion (w, h) in the initial pattern image of the present embodiment can be obtained from the following equation (3) (S8).
f1 (w, h) = f0 (w, h) / Qp (w, h) / Qc (w + (x−x0), h + (y−y0)) (3)

  In this way, the peripheral light quantity coefficient Qp (w, h) that is the optical characteristic of the projector 600 and the peripheral light quantity coefficient Qc (w, h) that is the optical characteristic of the cameras 101A and 101B are considered. After calculating the spatial frequency f1 (w, h) that determines the granularity of the pattern image part (w, h), the pattern image generation unit 502 calculates each pattern image part (w, h) according to each spatial frequency f1 (w, h). ) And an initial pattern image having pattern image portions having different granularities is generated (S9).

  In this embodiment, the granularity (spatial frequency) is adopted as the pattern condition. However, instead of or together with the granularity, another pattern condition that improves the recognition processing accuracy in the recognition processing unit 300 is adopted. May be. For example, when contrast is adopted as the pattern condition, the pattern image generation unit 502 also determines the contrast of each pattern image portion (w, h) according to each spatial frequency f1 (w, h) calculated as described above. An initial pattern image having pattern image portions having different particle sizes and contrasts as shown in FIG. 8 may be generated.

  The initial pattern image generated in this way may be generated by performing a test or the like in advance and stored in the storage unit of the personal computer 500. In this case, in the initial setting before operating the robot picking system, the initial pattern image may be read from the storage unit.

  Here, the above-described initial pattern image takes into consideration the optical characteristics of the projector 600 (reduction in the amount of light at the periphery of the pattern image) and the optical characteristics of the cameras 101A and 101B (reduction in the amount of light at the periphery of the captured image). By adjusting the granularity of the pattern image portion projected to the imaging area location (such as near the periphery of the pattern image) with relatively low matching accuracy, the matching accuracy for the imaging area location is improved, and an appropriate parallax value is obtained. This reduces the occurrence of parts that cannot be obtained. Therefore, after the initial setting described above, this initial pattern image may be used in a fixed manner.

  However, since the optical characteristics of the projector 600 and the cameras 101A and 101B may change over time, or may change according to environmental conditions such as temperature and humidity, an appropriate parallax value is caused by such changes. It may happen that the effect of reducing the generation of unobtainable portions is reduced. In addition, not only changes in the optical characteristics of the projector 600 and the cameras 101A and 101B, but also other changes (such as changes in ambient lighting conditions) reduce the occurrence of portions where an appropriate parallax value cannot be obtained. It can happen that the effect is reduced. Therefore, in the present embodiment, after the robot picking system is operated, the pattern image change process is executed at a predetermined execution timing, and the pattern image portion of the image pickup area portion of the image pickup area portion where the matching accuracy is reduced due to some change is detected. Adjust the pattern conditions (granularity etc.) to improve the matching accuracy.

Hereinafter, the pattern image change process performed in the present embodiment will be described.
FIG. 9 is a flowchart showing the flow of image pattern change processing in the present embodiment.
When the predetermined execution timing during operation of the robot picking system arrives, first, the personal computer 500 controls the projector 600 to project the pattern image stored in the storage unit toward the work table surface 601 (S11). Then, the pattern image is displayed on the work on the work table surface 601. As the pattern image at this time, if there is no change from the initial pattern image in the past image pattern change processing, the initial pattern image is used. Thereafter, each of the cameras 101A and 101B of the camera unit 100 executes an imaging operation (S12), and captured image data (reference image data and comparison image data) for one frame is obtained. The captured image data is corrected by the image correction units 201A and 201B, and parallax image data (parallax information) is generated from the corrected image data by the parallax information calculation unit 202 (S13). The parallax image data generated in this way is sent to the parallax point calculation unit 501 of the personal computer 500 via the external I / Fs 203 and 503.

  On the other hand, in the personal computer 500, a parallax image based on the parallax image data is displayed on the display unit, and the operator operates the input unit to set a designated image portion that counts the number of effective parallax values in the parallax image. (S14). The setting information of the designated image portion set in this way is passed to the parallax point calculation unit 501. The designated image portion is, for example, a rectangular image portion that is set along the inner edge of a box that accommodates a workpiece that is the object. The designated image portion for counting the number of effective parallax values is preferably set so as to include an image portion on which the target object is projected, and the image portion on which the target object is not projected is excluded as much as possible. This is because, when an image part other than the object (work) recognized by the recognition processing unit 300 is included, the number of parallax values that are not originally required is counted, and the pattern conditions of each pattern image part are optimized. It is because it becomes an obstacle.

  Note that the shape of the designated image portion is not limited to a rectangle, and may be another shape such as a circle. In addition, here, a case where the operator of the personal computer 500 manually sets the designated image portion will be described. However, the image portion on which the object is projected is identified and automatically designated based on the acquired parallax image data. An image part may be set. The setting of the designated image portion is not essential, and the number of effective parallax values may be counted for the entire parallax image.

  Based on the parallax image data received via the external I / Fs 203 and 503 and the setting information of the designated image part set in the processing step S14, the parallax score calculation unit 501 of the personal computer 500 performs the designated image on the parallax image. The number of pixels (number of parallax points) of effective parallax values in the portion is measured (S15). The measured number of parallax points is stored in the storage unit in a state associated with the pattern image (pattern image projected in processing step S11) used for the measurement of the number of parallax points (S16).

  Next, the parallax score calculation unit 501 of the personal computer 500 determines whether or not the measured parallax score satisfies a predetermined condition. Specifically, in this embodiment, since the total number of pixels of the designated image portion varies depending on the designation of the operator, the ratio of the number of parallax points to the total number of pixels of the designated image portion (hereinafter referred to as “disparity of the number of parallax points”). ) Is greater than or equal to a threshold value (S17). If the occupancy rate of the number of parallax points is equal to or greater than the threshold (Yes in S17), the pattern image (pattern image projected in processing step S11) used for the measurement of the number of parallax points is used when the robot picking system is subsequently operated. The pattern image is determined (S20).

  On the other hand, when the occupancy ratio of the number of parallax points is less than the threshold (No in S17), each pattern has a pattern condition different from that of the pattern image (pattern image projected in processing step S11) used for the measurement of the number of parallax points. Another pattern image in which the image portion is created is generated (S19), and the processing steps S11 to S17 are executed again.

In the present embodiment, another pattern image generated in the processing step S19 is generated as follows.
First, for example, the spatial frequency f1 (w, h) is determined with reference to the spatial frequency f1 (w, h) that determines the granularity in each pattern image portion of the pattern image initially projected in the pattern image change process. 9 levels of spatial frequencies f2 (w, h), f3 (w, h), f4 (w, h), f5 (w, h), f1 (w, h), f6 (w, h) ), F7 (w, h), f8 (w, h), and f9 (w, h) are set. Specifically, the value of the proportionality constant K is changed within the range of 0.5 ≦ K1 ≦ 2 by the following equation (4), and the spatial frequencies f2 (w, h) to f9 (w, h) are set. To do.
f2 (w, h) to f9 (w, h) = f1 (w, h) × K (4)

  By using these spatial frequencies f2 (w, h) to f9 (w, h), the pattern image generation unit 502 of the personal computer 500 has a coarser grain size than that of the pattern image projected first. It is possible to generate four types of pattern images and four types of pattern images in which the granularity of each pattern image portion is finer than that of the pattern image projected first.

  In the processing step S19, among the spatial frequencies f2 (w, h) to f9 (w, h) set in this way, according to a predetermined order (for example, an order close to the spatial frequency of the pattern image projected first). The pattern image generation unit 502 of the personal computer 500 generates another pattern image. Then, the processing steps S11 to S17 are executed, and if the occupancy rate of the number of parallax points is equal to or greater than the threshold value in the processing step S17 (Yes in S17), the pattern image used for the measurement of the number of parallax points is the robot picking thereafter It is determined as a pattern image to be used during system operation (S20).

  On the other hand, in any of the other pattern images generated in this manner (eight types here), if the occupancy rate of the number of parallax points does not exceed the threshold value (No in S17), the number of pattern generations is the specified value ( Here, it reaches 8 times (Yes in S18). In this case, the number of parallax points for a total of nine types of pattern images is stored in the storage unit in a state of being associated with each pattern image in processing step S16. Therefore, a pattern image having the highest number of parallax points is specified from the nine types of pattern images, and the pattern image is determined as a pattern image to be used when the robot picking system is subsequently operated (S20).

[Modification]
Next, a modification of the initial pattern image generation process in the present embodiment will be described.
In the above-described embodiment, the distance from the projector 600 to the work table surface 601 and the distance from the work table surface 601 to the camera unit 100 are arranged to be substantially equal to each other. In this example, the distance up to 601 varies and these distances are different as shown in FIG.

  As shown in FIG. 11, the closer the distance from the projector 600 to the work table surface 601, the smaller the size of the pattern image projected on the work table surface 601 (the smaller the magnification), and the farther the distance, the work table surface 601. The size of the pattern image projected on the screen is increased (the magnification is increased). Therefore, in the pattern image projected on the work table surface 601, the pattern per unit area becomes denser as the distance from the projector 600 to the work table surface 601 is shorter, and the pattern per unit area becomes rougher as the distance is longer. . When the distance from the projector 600 to the work table surface 601 changes relative to the distance from the work table surface 601 to the camera unit 100, the area ratio of the imaging region to the pattern image changes. At this time, the ratio (density) of the pattern (monochrome) per unit area hardly changes, but the pattern granularity changes.

  The granularity of the pattern affects the matching accuracy when calculating the parallax value. Specifically, when the distance from the work table surface 601 to the camera unit 100 is constant, the particle size is generally higher as the distance from the projector 600 to the work table surface 601 is closer, and the particle size is lower as the distance is longer. However, matching accuracy increases and the number of effective parallax values (number of parallax points) increases, and as a result, recognition processing accuracy in the recognition processing unit 300 increases. Therefore, when the distance from the projector 600 to the work table surface 601 varies relative to the distance from the work table surface 601 to the camera unit 100, it is desirable to generate a new pattern image according to the distance. .

  Therefore, in the initial image pattern generation process in the present modification, the spatial frequency f0 (w, h) that is the pattern condition of the reference pattern image having the uniform pattern condition generated first in the processing step S1 by the pattern image generation unit 502 is set. The correction is made according to the relative distance from the projector 600 to the work table surface 601 with respect to the distance from the work table surface 601 to the camera unit 100.

FIG. 12 is a flowchart showing a flow of initial image pattern generation processing in the present modification.
Also in this modified example, first, a reference pattern image with uniform pattern conditions is generated, and the relative position information of the projector 600 based on the distance from the work table surface 601 to the camera unit 100 and the installation position of the camera unit 100 is calculated. (S1 to S5). Thereafter, in the present modification, the maximum projection distance Zmax of the projector 600, the distance Z from the projector 600 to the work table surface 601 based on the relative position information of the projector 600 calculated in processing step S5, and a predetermined pattern generation coefficient The correction coefficient H is calculated from the following equation (5) from H0, which is a (constant) (S21).
H = (Zmax−Z) / H0 (5)

Then, using the correction coefficient H obtained in this way, the spatial frequency f0 (w, h) of the reference pattern image projected in the processing step S1 is corrected by the following equation (6), and the corrected The spatial frequency f′0 (w, h) is calculated (S22). Thus, the spatial frequency f′0 (w, h) of the reference pattern image corrected in consideration of the relative distance variation from the projector 600 to the work table surface 601 with respect to the distance from the work table surface 601 to the camera unit 100 is obtained. Can be obtained.
f′0 (w, h) = f0 (w, h) × H (6)

Thereafter, similar to the above-described embodiment, the peripheral light amount coefficient Qp (w, h) of the projector 600 and the peripheral light amount coefficient Qc (w, h) of the cameras 101A and 101B are calculated (S6, S7), and then the present embodiment. The spatial frequency f1 (w, h) for determining the granularity of each pattern image portion (w, h) in the initial pattern image is obtained from the following equation (3 ′) (S8 ′).
f1 (w, h) = f′0 (w, h) / Qp (w, h) / Qc (w + (x−x0), h + (y−y0)) (3 ′)

What was demonstrated above is an example, and there exists an effect peculiar for every following aspect.
(Aspect A)
The disparity information of the object is obtained by the disparity information generating means such as the disparity information calculating unit 202 based on the captured images obtained by capturing the object such as the workpiece in the imaging region from the different viewpoints by the plurality of imaging means such as the cameras 101A and 101B. In a pattern image projection apparatus having an image projection unit such as a projector 600 that projects a pattern image toward an imaging region when generating the pattern image, the pattern image is an optical component for at least one of the image projection unit and the imaging unit. The pattern image portion (w, h) projected on each imaging region location differs depending on the imaging region location having different characteristics.
The optimum pattern image that can improve the matching accuracy when generating the parallax information varies depending on the optical characteristics of the image projection unit and the imaging unit. Therefore, when the optical characteristics of the image projecting means and the imaging means differ depending on the location of the imaging area in the imaging area, the entire pattern image projected on the imaging area is not considered without considering the difference of the imaging area location. Even if the pattern image is composed of a regular or non-periodic pattern, a point with relatively low matching accuracy locally exists.
In this aspect, it is possible to project a pattern image composed of different pattern image portions in accordance with each imaging region portion having different optical characteristics with respect to at least one of the image projecting unit and the imaging unit toward the imaging region. . Therefore, it is possible to project a pattern image portion suitable for the optical characteristics of the image projecting means and the image capturing means that are different for each image capturing area location to each image capturing area location in the image capturing area. For example, a pattern image portion suitable for the part of the imaging region part is projected for a part of the imaging region part, and a pattern image part suitable for the part of the imaging region part is projected for the other imaging region part. Also, it is possible to project a pattern image portion suitable for the other imaging region portion. Therefore, it is possible to reduce or eliminate imaging region locations with relatively low matching accuracy in the imaging region, and it is possible to obtain appropriate parallax information for an object present at any location in the imaging region.

(Aspect B)
The aspect A is characterized by having pattern image generation means such as a pattern image generation unit 502 for generating the pattern image.
According to this, it becomes possible to set each pattern image part according to the use environment, use conditions, etc. of the said pattern image projection apparatus, and the pattern image suitable for the use environment, use conditions, etc. can be produced | generated.

(Aspect C)
In the aspect B, the pattern image generation unit generates the pattern image based on the number of parallax values (number of parallax points) included in the parallax information generated by the parallax information generation unit.
According to this, it becomes possible to set each pattern image part from the result of directly evaluating the parallax information, and generate a pattern image that can effectively reduce the occurrence of a part where the appropriate parallax information cannot be obtained in the object can do.

(Aspect D)
In the aspect B or C, the pattern image generation unit changes the pattern image portion according to the relative distance Z between the image projection unit and the object with respect to the distance between the imaging unit and the object. An image is generated.
As described in the above-described modification, an appropriate pattern image can be generated even when the relative distance Z between the image projection unit and the object varies with respect to the distance between the imaging unit and the object.

(Aspect E)
In any one of the aspects A to D, the pattern image portion projected on each of the imaging region portions is different in at least one of pattern granularity and contrast.
According to this, it is possible to project a pattern image whose grain size and contrast having a great influence on the matching accuracy are different in each pattern image portion toward the imaging region.

(Aspect F)
In any one of the aspects A to E, the optical characteristic of the image projecting unit includes the brightness of a pattern image projected by the image projecting unit.
According to this, the brightness of the pattern image projected by the image projecting means is different depending on the imaging region location, so that it is appropriate for the object when there is an imaging region location with relatively low matching accuracy in the imaging region. It is possible to reduce the occurrence of portions where no disparity information can be obtained.

(Aspect G)
In any one of the aspects A to F, the optical characteristic of the imaging unit includes a light reception amount received from the imaging region by the imaging unit.
According to this, when the amount of light received by the imaging unit from the imaging area varies depending on the imaging area location, it is appropriate for the object when there is an imaging area location with relatively low matching accuracy in the imaging area. It is possible to reduce the occurrence of portions where no disparity information can be obtained.

(Aspect H)
A disparity information generating unit such as a disparity information calculating unit 202 that generates disparity information of the target object based on captured images obtained by capturing the target object in the imaging region from different viewpoints by a plurality of image capturing units, and a pattern toward the imaging region In a parallax information generating device having a pattern image projecting unit such as a personal computer 500 or a projector 600 that projects an image, the pattern image projecting device according to any one of the modes A to G is used as the pattern image projecting unit. It is characterized by.
According to this, generation | occurrence | production of the part from which suitable parallax information cannot be obtained in a target object can be reduced.

(Aspect I)
A pattern projected by the image projecting unit toward the imaging region when the parallax information generating unit generates the parallax information of the target object based on the captured images captured by the plurality of imaging units from the different viewpoints. A pattern image generation program that causes a computer to function as a pattern image generation unit that generates an image, wherein the pattern image corresponds to an imaging region where optical characteristics of at least one of the image projection unit and the imaging unit are different. Thus, the pattern image portions projected on the respective imaging region portions are different.
In this aspect, it is possible to project a pattern image composed of different pattern image portions in accordance with each imaging region portion having different optical characteristics with respect to at least one of the image projecting unit and the imaging unit toward the imaging region. . Therefore, it is possible to project a pattern image portion suitable for the optical characteristics of the image projecting means and the image capturing means that are different for each image capturing area location to each image capturing area location in the image capturing area. For example, a pattern image portion suitable for the part of the imaging region part is projected for a part of the imaging region part, and a pattern image part suitable for the part of the imaging region part is projected for the other imaging region part. Also, it is possible to project a pattern image portion suitable for the other imaging region portion. Therefore, it is possible to reduce or eliminate imaging region locations with relatively low matching accuracy in the imaging region, and it is possible to obtain appropriate parallax information for an object present at any location in the imaging region.

  This program can be distributed or obtained in a state of being recorded on a recording medium such as a CD-ROM. It is also possible to distribute and obtain signals by placing this program and distributing or receiving signals transmitted by a predetermined transmission device via transmission media such as public telephone lines, dedicated lines, and other communication networks. Is possible. At the time of distribution, it is sufficient that at least a part of the computer program is transmitted in the transmission medium. That is, it is not necessary for all data constituting the computer program to exist on the transmission medium at one time. The signal carrying the program is a computer data signal embodied on a predetermined carrier wave including the computer program. Further, the transmission method for transmitting a computer program from a predetermined transmission device includes a case where data constituting the program is transmitted continuously and a case where it is transmitted intermittently.

DESCRIPTION OF SYMBOLS 100 Camera part 101A, 101B Camera 200 Image processing part 201A, 201B Image correction part 202 Parallax information calculation part 300 Recognition processing part 400 Robot arm control part 500 Personal computer 501 Parallax point calculation part 502 Pattern image generation part 600 Projector 601 Work surface

JP 2001-91232 A

Claims (9)

An image in which a pattern image is projected toward an imaging region when the parallax information generating unit generates parallax information of the target based on captured images obtained by imaging the target in the imaging region from different viewpoints by a plurality of imaging units. In the pattern image projection apparatus having the projection means,
The pattern image is characterized in that a pattern image portion projected on each imaging area location differs according to an imaging area location having different optical characteristics for at least one of the image projection means and the imaging means. Projection device. The pattern image projection apparatus according to claim 1,
A pattern image projection apparatus comprising pattern image generation means for generating the pattern image. The pattern image projection device according to claim 2,
The pattern image projection device, wherein the pattern image generation unit generates the pattern image based on the number of parallax values included in the parallax information generated by the parallax information generation unit. In the pattern image projector of Claim 2 or 3,
The pattern image generation unit generates a pattern image in which each pattern image portion is changed according to a relative distance between the image projection unit and the object with respect to a distance between the imaging unit and the object. Pattern image projection device. In the pattern image projector of any one of Claims 1 thru | or 4,
The pattern image projection device, wherein the pattern image portion projected on each of the imaging region portions is different in at least one of pattern granularity and contrast. The pattern image projection device according to any one of claims 1 to 5,
An optical characteristic of the image projecting unit includes a brightness of a pattern image projected by the image projecting unit. In the pattern image projector of any one of Claims 1 thru | or 6,
The pattern image projection apparatus according to claim 1, wherein the optical characteristic of the imaging unit includes a received light amount received by the imaging unit from an imaging region. Disparity information generating means for generating disparity information of the object based on captured images obtained by capturing the object in the imaging area from a plurality of image capturing means from different viewpoints;
In a parallax information generation device having pattern image projection means for projecting a pattern image toward an imaging region,
A parallax information generation device using the pattern image projection device according to claim 1 as the pattern image projection unit. A pattern projected by the image projecting unit toward the imaging region when the parallax information generating unit generates the parallax information of the target object based on the captured images captured by the plurality of imaging units from the different viewpoints. A pattern image generation program for causing a computer to function as a pattern image generation means for generating an image,
The pattern image is characterized in that a pattern image portion projected on each imaging area location differs according to an imaging area location having different optical characteristics for at least one of the image projection means and the imaging means. Generation program.