JP5736978B2 - Article recognition apparatus, article recognition system, program, and article recognition method - Google Patents

Description

  The present invention relates to an article recognition device, an article recognition system, a program, and an article recognition method.

In recent years, various industrial robots have been used in factory production lines and the like. In particular, since the (vertical) articulated robot has a structure similar to a human arm, it can perform various operations on behalf of a human.
For example, Patent Document 1 discloses a picking device that detects a specific part that can be gripped from an image of an article (work) captured by a camera, and grips and picks up the specific part with a robot hand. For example, in Patent Document 2, as a workpiece recognition method, a workpiece group to be recognized is photographed by a stereo camera including two cameras having different relative positions, and a pair of captured two-dimensional images (image pairs) is obtained. A method for determining a three-dimensional position and posture using the method is disclosed.
In this way, the image captured by the stereo camera is three-dimensionalized to recognize the workpiece, and the obtained position / posture information of the workpiece can be used for controlling a robot such as a picking device.

JP 7-319525 A JP-A-2-309202

However, many workpieces handled on a production line or the like use a gloss material such as metal or resin on the entire workpiece or the surface, and reflected light may enter the camera on the workpiece surface. For example, as shown in FIG. 2 of Patent Document 1 and FIG. 1 of Patent Document 2, both the camera and the lighting device are arranged above the workpiece group, and the workpieces are stacked or stacked in a random posture. In such a case, there may be a work that regularly reflects the illumination light in the direction of the camera.
When a stereo camera is used for workpiece recognition, when reflected light that is specularly reflected is incident on one camera, the illumination device is reflected in only one of the captured image pairs, and the obtained workpiece information is displayed between the images. Since they differ greatly, there are cases where three-dimensionalization is difficult. In addition, for example, even in the case of a production line in which each workpiece is handled in a fixed posture, if there are a plurality of workpieces to be recognized, the illumination light is regularly reflected in the direction of the camera depending on the position of the workpiece. There is a case.

  The main present invention for solving the above-described problem is that a camera control unit that controls a stereo camera including first and second cameras that capture an image of an article group composed of a plurality of articles to be recognized is different from the article group. An illumination control unit that controls the first to third illumination devices that illuminate from the direction and any one of the first to third illumination devices are sequentially turned on, and each image is captured by the stereo camera. An imaging processing unit that acquires first to third image pairs including a pair of two-dimensional images, and an area that includes the article from at least one two-dimensional image included in the first to third image pairs. For each of the regions, the image processing unit is configured based on the image pair in which a difference in received light amount between the first and second cameras is equal to or less than a predetermined value among the first to third image pairs. A three-dimensional processor for generating a dimension image is an article recognition apparatus characterized by having a recognition processing unit for calculating the position and orientation information to recognize the article based on the 3-dimensional image.

  Other features of the present invention will become apparent from the accompanying drawings and the description of this specification.

  According to the present invention, a plurality of articles whose surfaces are glossy can be recognized using a stereo camera.

It is a block diagram which shows the structure of the article | item recognition apparatus in one Embodiment of this invention. It is a block diagram which shows the outline of a structure of the whole article recognition system provided with the article recognition apparatus in one Embodiment of this invention. It is a top view which shows the positional relationship of the stereo camera 2 and the illuminating devices L1-L3. It is a perspective view which shows an example of the shape of a workpiece | work. It is a flowchart explaining the operation | movement of the program for making a computer implement | achieve the function (work imaging | photography process) of the imaging process part 11. FIG. It is a figure which shows the relationship between a camera and an illuminating device, and the imaged image in a workpiece | work imaging process. It is a flowchart explaining operation | movement of the program for making a computer implement | achieve the function (work recognition process) of the extraction process part 14, the three-dimensionalization process part 15, and the recognition process part 16. FIG. It is a figure explaining an example of the process in which the extraction process part 14 extracts each workpiece | work using the image I11. It is a figure which shows an example of the positional relationship in which the reflected light regularly reflected by the workpiece | work W2 injects into the camera C1, when the illuminating device L1 is lighted. It is a figure which shows an example of the positional relationship in which the reflected light regularly reflected by the workpiece | work W2 injects into the camera C2 when the illuminating device L2 is lighted. It is a figure which shows an example of the reflected light regularly reflected by the workpiece | work W2 when the illuminating device L3 is lighted in the positional relationship shown in FIG. 9 and FIG. It is a figure which shows an example of the light reception amount of the cameras C1 and C2 in the area | region containing the workpiece | work W2 in the positional relationship shown in FIG. 9 thru | or FIG.

  At least the following matters will become apparent from the description of this specification and the accompanying drawings.

=== Overall Configuration of Article Recognition System ===
Hereinafter, with reference to FIG. 2 thru | or FIG. 4, the outline of the structure of the whole article recognition system provided with the article recognition apparatus in one Embodiment of this invention mentioned later is demonstrated. 3 is a diagram showing the positional relationship between the stereo camera 2 and the illumination devices L1 to L3 as viewed from the stereo camera 2 side (upper side in FIG. 2).

  The article recognition system shown in FIG. 2 includes an article recognition apparatus 1, a stereo camera 2, and lighting apparatuses L1 to L3. As an example, eight workpieces Wn (1 ≦ n ≦ 8) is a recognition target. Further, the stereo camera 2 and the lighting devices L1 to L3 are arranged above a work table or the like on which a tray 9 for storing a work group is installed. Each work Wn has a shape shown in FIG. 4, for example, and takes a random posture in the tray 9.

  The stereo camera 2 includes cameras C1 and C2 (first and second cameras) having different relative positions with respect to the workpiece group, and a pair of (two-dimensional) images Im1 and Im2 output from the cameras C1 and C2, respectively, It is input to the recognition device 1. The article recognition apparatus 1 outputs position / posture information T [Wn] of each recognized workpiece Wn. Furthermore, the article recognition apparatus 1 also outputs control signals SC1, SC2, and SL1 to SL3 for controlling each camera and each illumination device.

  The illuminating devices L1 to L3 (first to third illuminating devices) are ring illuminating devices that can uniformly illuminate a work group in the tray 9 from the periphery in a ring shape. LEDs (Light Emitting Diodes) having good characteristics are arranged in a ring shape. Further, the illumination devices L1 to L3 are arranged at different heights so that the workpiece group can be illuminated from different directions.

  In FIGS. 2 and 3, the illumination devices L1 to L3 have larger diameters as they are arranged in the lower stage, but are not limited thereto. The illumination devices L1 to L3 may be of the same size as long as they can be arranged so as not to be shaded from each other (without blocking the illumination light) and to be illuminated from different directions when illuminating the workpiece group.

=== Configuration of article recognition apparatus ===
Hereinafter, with reference to FIG. 1, the structure of the article | item recognition apparatus in one Embodiment of this invention is demonstrated.

  The article recognition apparatus 1 shown in FIG. 1 includes an imaging processing unit 11, a camera control unit 12, an illumination control unit 13, an extraction processing unit 14, a three-dimensional processing unit 15, a recognition processing unit 16, and a storage unit 17. It is configured to include.

  Images Im 1 and Im 2 are input to the imaging processing unit 11, and the image pair is input from the imaging processing unit 11 to the extraction processing unit 14. Further, an imaging command signal IMG is input from the imaging processing unit 11 to the camera control unit 12, and control signals SC <b> 1 and SC <b> 2 are output from the camera control unit 12. Further, an illumination selection signal SEL is input from the imaging processing unit 11 to the illumination control unit 13, and control signals SL <b> 1 to SL <b> 3 are output from the illumination control unit 13.

  The region image Im1 [Wn] and Im2 [Wn] including the work Wn are input from the extraction processing unit 14 to the three-dimensional processing unit 15, and the three-dimensional image of the work Wn output from the three-dimensional processing unit 15 ( Information) Im3 [Wn] is input to the recognition processing unit 16. In addition, a three-dimensional model MDL representing the shape of the workpiece is input from the storage unit 17 to the recognition processing unit 16, and position / posture information T [Wn] of the workpiece Wn is output from the recognition processing unit 16.

=== Operation of article recognition apparatus ===
Hereinafter, the operation of the article recognition apparatus in the present embodiment will be described with reference to FIGS. 5 to 12 as appropriate.

  In the article recognition apparatus 1, the functions of the imaging processing unit 11, the extraction processing unit 14, the three-dimensional processing unit 15, and the recognition processing unit 16 include, for example, the camera control unit 12, the illumination control unit 13, and the storage unit 17. It can be realized by a computer provided. 5 and 7 show operations of a program for causing a computer to realize functions corresponding to the imaging processing unit 11, the extraction processing unit 14, the three-dimensionalization processing unit 15, and the recognition processing unit 16.

  FIG. 5 shows a function (work imaging process) corresponding to the imaging processing unit 11. When the workpiece imaging process is started (S11), the processes of S12 and S13 are performed for each lighting device Lm (1 ≦ m ≦ 3) (loop process 1).

  In the loop process 1, first, the illumination device Lm is selected, and an illumination selection signal SEL (= m) indicating the illumination device Lm is input to the illumination control unit 13. Then, under the control of the illumination control unit 13, the illumination device Lm is turned on, and the other illumination devices are turned off (S12).

  Next, the imaging command signal IMG is input to the camera control unit 12 with the lighting device Lm turned on. Then, under the control of the camera control unit 12, the cameras C1 and C2 are caused to photograph the work group in the tray 9, and the captured images Im1 and Im2 are acquired (S13).

  Then, the loop process 1 of S12 and S13 is performed on each lighting device Lm, and any one of the lighting devices L1 to L3 is sequentially turned on to obtain the image pairs captured, thereby obtaining six images. Images (three image pairs) can be acquired, and the workpiece imaging process is terminated (S14). As shown in FIG. 6, the images I11 and I12 (first image pair) captured by lighting the illumination device L1 and the images I21 and I22 captured by lighting the illumination device L2 by the workpiece imaging process. (Second image pair) Images I31 and I32 (third image pair) captured by lighting the lighting device L3 are acquired.

  FIG. 7 shows functions (work recognition processing) corresponding to the extraction processing unit 14, the three-dimensional processing unit 15, and the recognition processing unit 16. When the workpiece recognition process is started (S21), first, each workpiece Wn is extracted using at least one of the six images acquired by the workpiece imaging process (for example, the image I11) (S22).

  Specifically, for example, the edge of each workpiece Wn is extracted from the image I11, and the region including each workpiece Wn is extracted to generate a region image I11 [Wn] as shown in FIG. In some cases, part of the region image overlaps, for example, when the workpieces overlap or contact each other in the tray 9. Further, the region image I11 [Wn] is not a trimmed image I11 as shown in FIG. 8, but may be a masked portion of the image I11 other than the work Wn.

  Next, the processing from S23 to S29 is performed for each workpiece Wn (1 ≦ n ≦ 8) (loop processing 2). In the following, as an example, processing for the workpiece W2 (n = 2) will be described.

  In the loop process 2, first, m = 1 is set (S23), the images I11 and I12 picked up by turning on the lighting device L1 are selected, and region images I11 [W2] including the workpiece W2 are selected from the image pair, respectively. I12 [W2] is generated (S24). Therefore, the processing of S22 to S24 realizes a function corresponding to the extraction processing unit 14.

  Specifically, similarly to the processing of S22, for example, regions including the workpiece W2 are extracted from the images I11 and I12, and portions other than the workpiece W2 are trimmed or masked to generate region images I11 [W2] and I12 [W2 ] Is generated. Note that, in order to perform corresponding point search and three-dimensionalization in the processing of S28 described later, the region including the work W2 is extracted at the same position and size in the images I11 and I12.

  Next, in the region images I11 [W2] and I12 [W2], a received light amount difference DL between the cameras C1 and C2 is obtained and compared with a predetermined threshold value Dth (S25). Specifically, for example, the received light amount difference DL can be obtained based on the average luminance of the pixels included in each region image and the change in luminance on the epipolar line in each region image.

  Here, as an example, when the cameras C1 and C2, the illuminating devices L1 to L3, and the workpiece W2 have the positional relationship as shown in FIGS. 9 to 11, as shown in FIG. 9, the illumination from the illuminating device L1 The reflected light that is regularly reflected by the surface of the workpiece W2 enters the camera C1. Therefore, the illumination device L1 is reflected on the surface of the workpiece W2 in the area image I11 [W2], and the reflection does not occur in the area image I12 [W2]. For this reason, as shown in the case of m = 1 in FIG.

  As described above, when the illumination device L1 is reflected in one of the region images and the received light amount difference DL is larger than the threshold value Dth (S25: NO), next, m = 2 is set (S27), and the illumination device L2 is turned on. The images I21 and I22 that have been picked up are selected. Then, region images I21 [W2] and I22 [W2] each including the workpiece W2 are generated from the image pair (S24).

  When the illumination device L2 is turned on, as shown in FIG. 10, the reflected light obtained by regular reflection of the illumination light from the illumination device L2 on the surface of the workpiece W2 enters the camera C2. Accordingly, the illumination device L2 is reflected on the surface of the workpiece W2 in the area image I22 [W2], and the reflection is not generated in the area image I21 [W2]. Therefore, for example, as shown in the case of m = 2 in FIG. 12, the brightness in these area images is greatly different.

  As described above, when the illumination device L2 is reflected in one of the region images and the received light amount difference DL is larger than the threshold value Dth (S25: NO), then m = 3 (S27) and the illumination device L3 is turned on. The images I31 and I32 that have been picked up are selected. Then, region images I31 [W2] and I32 [W2] each including the workpiece W2 are generated from the image pair (S24).

  When the illumination device L3 is turned on, as shown in FIG. 11, the reflected light obtained by regular reflection of the illumination light from the illumination device L3 on the surface of the workpiece W2 does not enter either of the cameras C1 and C2. Therefore, in any of the area images I31 [W2] and I32 [W2], the illumination device L2 is not reflected on the surface of the workpiece W2. Therefore, for example, as shown in the case of m = 3 in FIG. 12, the luminances in these area images are approximately the same.

  As described above, in any region image, when the reflection of the illumination device L3 does not occur and the received light amount difference DL is equal to or less than the threshold value Dth (S25: YES), the region images I31 [W2] and I32 [W2] ] Is used to perform corresponding point search and three-dimensionalization (S28). Specifically, for example, as shown in FIG. 12, the corresponding point A ′ on the epipolar line EP2 of the region image I32 [W2] with respect to the attention points A and B on the epipolar line EP1 of the region image I31 [W2]. And B ′, respectively. Then, a three-dimensional image I33 [W2] of the workpiece W2 is generated based on the obtained parallax between images.

  As described above, out of the pair of region images when the lighting devices L1 to L3 are respectively turned on, the lighting device is not reflected and the region images I31 [W2] and I32 [W2] satisfying DL ≦ Dth are used. The three-dimensional image I33 [W2] of the workpiece W2 is generated. Therefore, the processing of S25 to S28 realizes a function corresponding to the three-dimensional processing unit 15.

  Here, the article recognition system of the present embodiment includes three illumination devices L1 to L3 that illuminate the workpiece group from different directions. Therefore, for example, as shown in FIG. 9 and FIG. 10, even when there is a combination of a lighting device and a camera in which reflection occurs in one area image, as shown in FIG. There is at least one illumination device in which no reflection occurs in the area image. Note that the process of S26 is an exception process when any one of the illumination devices L1 to L3 is turned on and there is a reflection in one area image (S26: YES), and is not essential. .

  Next, the generated three-dimensional image I33 [W2] of the workpiece W2 is collated with the three-dimensional model MDL stored in the storage unit 17, the workpiece W2 is recognized, and position / posture information T [W2] is obtained. Calculate (S29). Therefore, the process of S9 realizes a function corresponding to the control unit 17.

  Then, by performing the loop process 2 of S23 to S29 on each workpiece Wn, all the workpieces W1 to W8 can be recognized and the position / posture information T [W1] to T [W8] can be calculated respectively. The process ends (S30).

  As described above, in the article recognition apparatus 1, each of the image pairs captured by sequentially lighting any one of the illumination apparatuses L1 to L3 that illuminate a workpiece (article) group from different directions is obtained. Thus, it is possible to obtain at least a pair of region images in which there is no reflection of the lighting device and the received light amount difference DL between the cameras C1 and C2 is equal to or less than a predetermined threshold value Dth, and is generated using the pair of region images. Based on the three-dimensional image, the work Wn (1 ≦ n ≦ 8) having a glossy surface can be recognized using the stereo camera 2.

  Further, by collating a three-dimensional image generated using a pair of region images with a three-dimensional model MDL representing the shape of the workpiece, the workpiece Wn can be recognized and position / posture information T [Wn] can be calculated. it can.

  Further, the illumination device compares the received light amount difference DL with the threshold value Dth by obtaining the received light amount difference DL based on the average luminance of the pixels included in each region image and the change in luminance on the epipolar line in each region image. A pair of region images without the reflection of the image can be selected and used to generate a three-dimensional image.

  2 and 3, the lighting devices L1 to L3 are turned on or off under the control of the lighting control unit 13, and the cameras C1 and C2 are photographed with the cameras C1 and C2 under the control of the camera control unit 12. Thus, it is possible to acquire image pairs captured by illuminating the workpiece group from different directions.

  Further, as the illumination devices L1 to L3, by using ring illumination devices that uniformly illuminate the workpiece group from the periphery in a ring shape, the workpiece groups can be illuminated from different directions by being arranged at different heights. it can.

  Further, in a program for causing a computer to realize functions corresponding to at least the imaging processing unit 11, the extraction processing unit 14, the three-dimensionalization processing unit 15, and the recognition processing unit 16 of the article recognition apparatus 1, directions of work groups are different from each other. 3 is generated using a pair of region images without the reflection of the illumination device, by sequentially turning on any one of the illumination devices L1 to L3 that illuminate from 3 and acquiring each image pair captured. Based on the three-dimensional image, the work Wn having a glossy surface can be recognized using the stereo camera 2.

  In addition, by turning on or off the lighting devices L1 to L3, illuminating the work group from different directions, and photographing each with the stereo camera 2, it is possible to obtain at least a pair of region images without reflection of the lighting device. The workpiece Wn having a glossy surface can be recognized based on the three-dimensional image generated using the pair of region images.

  In addition, the said embodiment is for making an understanding of this invention easy, and is not for limiting and interpreting this invention. The present invention can be changed and improved without departing from the gist thereof, and the present invention includes equivalents thereof.

  In the above-described embodiment, the article recognition system includes the stereo camera 2 including the two cameras C1 and C2 and the three illumination devices L1 to L3, but is not limited thereto. For example, a stereo camera that includes three or more cameras and outputs a pair of two-dimensional images picked up by two cameras appropriately selected according to the position of a workpiece to be recognized can be used. Further, for example, a three-dimensional image of a workpiece may be generated based on k (k ≧ 3) two-dimensional images (image sets) captured by three or more cameras. In this case, by using (k + 1) illuminating devices, at least one image set that does not reflect the illuminating device can be obtained.

DESCRIPTION OF SYMBOLS 1 Article recognition apparatus 2 Stereo camera 9 Tray 11 Imaging process part 12 Camera control part 13 Illumination control part 14 Extraction process part 15 Three-dimensionalization process part 16 Recognition process part 17 Memory | storage part C1, C2 Camera L1-L3 Illumination apparatus W1-W8 work

Claims (7)

A camera control unit that controls a stereo camera including first and second cameras that capture an article group including a plurality of articles to be recognized;
An illumination control unit for controlling the first to third illumination devices that illuminate the article group from different directions;
Imaging processing for acquiring first to third image pairs each made up of a pair of two-dimensional images captured by the stereo camera by sequentially lighting any one of the first to third lighting devices. And
An extraction processing unit that extracts regions each including the article from at least one two-dimensional image included in the first to third image pairs;
For each region, a three-dimensional image of the article is generated based on an image pair in which a difference in received light amount between the first and second cameras is a predetermined value or less among the first to third image pairs. A three-dimensional processing unit;
A recognition processing unit that recognizes the article based on the three-dimensional image and calculates position / posture information;
An article recognition apparatus characterized by comprising: A storage unit storing a three-dimensional model representing the shape of the article;
The article recognition apparatus according to claim 1, wherein the recognition processing unit calculates the position / attitude information by collating the three-dimensional image with the three-dimensional model.   3. The article recognition apparatus according to claim 1, wherein the three-dimensional processing unit obtains the difference in the amount of received light based on luminance of pixels respectively included in the pair of two-dimensional images. The article recognition apparatus according to any one of claims 1 to 3,
The stereo camera that captures the group of articles under the control of the camera control unit and outputs the pair of two-dimensional images;
The first to third illumination devices that illuminate the group of articles under the control of the illumination control unit;
An article recognition system comprising:   5. The article recognition system according to claim 4, wherein the first to third illumination apparatuses are ring illumination apparatuses that illuminate the group of articles in a ring shape from different heights. A camera control unit that controls a stereo camera including first and second cameras that capture an article group including a plurality of articles to be recognized;
An illumination control unit for controlling the first to third illumination devices that illuminate the article group from different directions;
On a computer with
A function of sequentially turning on any one of the first to third illumination devices to obtain first to third image pairs each consisting of a pair of two-dimensional images captured by the stereo camera; ,
A function of extracting regions each containing the article from at least one two-dimensional image included in the first to third image pairs;
For each region, a three-dimensional image of the article is generated based on an image pair in which a difference in received light amount between the first and second cameras is a predetermined value or less among the first to third image pairs. Function and
A function of recognizing the article based on the three-dimensional image and calculating position / posture information;
A program to realize First and second images are captured by sequentially turning on any one of first to third illumination devices that illuminate an article group composed of a plurality of recognition target articles from different directions. Obtaining first to third image pairs each consisting of a pair of two-dimensional images captured by a stereo camera equipped with a camera;
Extracting regions each containing the article from at least one two-dimensional image included in the first to third image pairs;
For each region, a three-dimensional image of the article is generated based on an image pair in which a difference in received light amount between the first and second cameras of the first to third image pairs is a predetermined value or less. ,
An article recognition method, comprising: recognizing the article based on the three-dimensional image and calculating position / posture information.

Images